Something not discussed nearly enough, even around here, is that the federal government does NOT have authority to "do whatever it wants." It has LIMITED authority, per the Constitution.
The federal government has exclusive legislative jurisdiction over (a) Washington, DC, (b) the federal territories (Puerto Rico, Guam, etc.), and (c) the federal enclaves (areas such as military bases, where the State granted to the federal government exclusive jurisdiction or such jurisdiction was reserved by the federal government upon statehood).
In every other area within the States, the federal government ONLY has jurisdiction in those specific areas that the Constitution enumerates such powers (bankruptcy courts, immigration law, post offices, etc.).
In 1956, a report was published discussing federal jurisdiction:
https://www.defendruralamerica.com/files/DSJurisdictionReport1957.pdf
As the federal government attempts to encroach on areas that are not within its powers, it becomes more and more important to be reminded of the constitutional limitations.
The trick used today is the feds claim they have a "contract" with a person or business and that allows them to violate their jurisdictional limits.
It does not. The feds cannot obtain by private contract what it cannot have the authority to do via the Constitution.
Actually, it's whether or not you receive a federal benefit, not the location of where you live.
Income tax started in 1861. It is an excise tax on the benefit of a federal privilege (such as holding public office, or owning stock in a federal corporation like the railroads were back then). It's the exact same concept as a tariff, where a foreigner pays an excise tax on the benefit of doing business by crossing the border into the USA (foreign commerce).