To a degree yes, but I disagree it is selfish. Not all who wish to be left alone have the skills to do as you suggest. While they may have the correct beliefs and moral compass, just because you admire the stereotypical strong silent type, as were taught to in the movies, doesn't mean they would make good political leaders.

As suggested by the OP's photo, some people are good at politics, some are not, regardless of beliefs. Others are VERY good in SHTF scenarios, and can clean up when things go south. Being good at killing your opponents has its place, but it is not in day to day politics.