Before the American Revolution there were frequent slave uprisings, and a lot of people would run away.
The topic of slavery is like an electric fence. Touch it and people will react.
It's not a common thing for a Southern white family to go out and seek their black cousins.
Slavery in West Africa, and in Rome and in the Mediterranean, was something different than slavery in America.
If you read the memoirs of slave-owning families, you'd be hard pressed to find evidence of black people in the lives of the whites, even though for most of the time on the plantations black people outnumbered whites by a ratio of seven to one.