Given the context is it won't be safe for "them", I'd say it's that people will be harassing and stalking them. If it was crime based, it wouldn't be safe for anyone to walk down the street.

Also these people used to hide in public because the sleepy sheep had no idea who they were or what they stood for. When that changes, people will follow them whenever they are in public.

Side note, the crime is bad and it's getting worse as you said.