Jefferson and Madison both agreed that the Constitution was a voluntary arrangement among the states. Like any voluntary arrangement, if one party is dissatisfied, it has the right to end that arrangement. If there aren't any terms in the contract for how it needs to end, a unilateral declaration of withdrawal immediately ceases all future obligations on both sides, and then the two parties go their separate ways in peace.

Lincoln reacted to the South seceding with a war of conquest. That killed the last vestiges of the "voluntary arrangement," leaving the Constitution as an unenforced and unenforceable set of aspirations at best.

And of course, that's if the Constitution was of any authority to begin with.