It is not an unreasonable concern. It is unreasonable to state that anyone is required to let you into their home when you have no proof or even suspicion that they are doing anything wrong.

• Mass surveillance is illegal.
• Unwarranted searches are illegal.
• Sobriety checkpoints are illegal.
• Entering private property without consent, and without court-granted approval or reasonable suspicion of an emergency is illegal.

I am not saying they are actively or consciously trying to do anything nefarious, but if OP doesn't want random government employees in his home, it is his right to make them first prove cause to be there. If he is worried about his plumbing, he can also just hire a private plumber of his choosing to do an inspection.