Only time will tell. Probably a good idea, but you'll never know how the other choice would turn out in the long run, because you only get one shot.

Surrounding yourself with like-minded intelligent people is a great ideal. But it's hard to isolate forever. Even Christ had stupid people catch up with him.

So tolerance of dummies is a good quality, but don't punish yourself for no reason. Your heart will tell you.