If there was a group of people who started off with a religious motivation, and that religious motivation causes them to be very criminal and horrible to those around them, they wont be liked. Over time (assuming they survive as a coherent grouping) the hate they are religiously compelled to express to everyone thats not them, and the flak and antagonism coming back at them as a direct consequence of their crimes will make them incredibly insular. Being insular means they develop a culture isolated from those they live amongst, as well as marry and really only associate among their own kind, so their group gets both a genetic set of properties, and eventually recessive genes that bring about negative physical and mental attributes and other flaws adding to their 'otherness'.

So its both a religious, genetic and cultural identifier. Maybe it started off with just one of these and over time enveloped all 3, or maybe all three of these were present at the start.

Using confusion over the term to conduct swindles and shakedowns, to infiltrate and subvert is just a continuation of the same crime motivations.