In Griswold v. Connecticut, the Supreme Court found that there is an implied right to privacy.

Tort law says that "Public disclosure of private facts: the dissemination of truthful private information which a reasonable person would find objectionable" would be violation of our right to privacy.

An agent of the government coming to a person's house would basically let everyone around know that at least one family member had not been vaccinated.

"False light: the publication of facts which place a person in a false light, even though the facts themselves may not be defamatory"

Again agent would be broadcasting or publishing the fact for everyone around to see that someone had not been vaccinated. Some nutjobs might decide to harm people in the house because of the homeowners being unvaccinated.