It's a difference of scope

We have.... Here, chans and an ever shrinking number of places. People who come here to ridicule us at best, distract us, or entrap us have the entire internet to talk about whatever they wish

This is like letting illegal immigrants move into your house. You'll lose what little space you have to yourself if you don't get them out

If they all ban us, we have no voice.

If we ban them, they still have the rest of the "civilized" internet to have their voice