"Cancelling" isn't against free speech. It's a bunch of people deciding, collectively, to use their free speech to call someone a dumbass not worth listening to because said dumbass likely said something racist or bigoted. There are no "legal" ramifications at play, and therefore no first amendment violations occurring. Twitter is not a government forum or public space, no matter how "prevalent" it or any other social media platform may have gotten.
The conservative argument seems to be "You can't use your free speech to say that my free speech calling X group a bunch of godless abominations who deserved to be executed/expelled/castrated/imprisoned is bad, that's against free speech!"
"Freedom of speech" in no way exonerates you from someone calling you or your opinion stupid, dangerous, misinformed and hateful. Because saying that they can't do that would infringe on their free speech.