You're absolutely right. We have the benefit of hindsight and historical documentation on this one, but I can imagine the vast majority of Southerners at the time didn't care one way or another about slavery and were much more concerned with trying to live their lives without the overreach of the federal government.