Don't get me wrong it was a factor, but it is not the reason the majority of southerners supported the secession. The hot issue in the south was the taxation of cotton being sold internationally. Something they where all strongly apposed to. and it was something the North wanted.
Until 1950 this was the case, however the roles reversed. Kennedy was the First modern Democrat President, and a sign that the roles had reversed. The Democrats became the party of equality and change, and the Republicans which traditionally where the northern business party started to fill up with former southern democrats. Reagan was the first Southern Democrat to be elected as a Republican. He said so much often. He called them "boll weevils".