And it wasn't even racial, EVERYONE could be enslaved in those societies. A few Greek philosophers were slaves in their youths after having been in the losing side of wars and battles.
Should we tear down the statue of any and all Arab and Ottoman sultans and caliphs, since they had slaves?
How about African kings?