I think one of the main reasons is the same one people give for not voting:

I didn't speak up because it wouldn't have mattered.

Which is only mostly true, because hearing opposing voices DOES matter for morale purposes for those trying to resist.

But I disagree about the coward thing in some cases. There is a huge difference between discretion and responsibilities and cowardice.