Problem with that logic is that really it breaks down to be even worse than that, slavery was the tool, the reason was money.
Cotton was a product that was very difficult to harvest, it took serious amounts of man power to produce. Paying individuals would have consumed most of the profits generated by it's production.
So yes Slavery was the biggest contention, but sadly the true reason was greed and wealth.