Government agencies and agents do not have rights. They have powers vested in them by law. Any action they take that is not specifically in accordance with enabling legislation is unconstitutional, illegal and unenforceable. The overturning of the Chevron Deference clarified this.
These powers can be revoked by legislative action and the agencies can also eliminated the same way. As a matter of fact, the legislature can withhold funding and strangle the agency almost immediately.