Technically, any Constitutional sheriff with his/her deputies may arrest/remove/replace all violators of oaths or those ineligible to serve, but illegally holding office. Also, any citizen can file claims against public official surety bonds to make them ineligible for office. And then maybe notify sheriff for enforcement.

Problem is sheriffs are corrupt as well, so we may need to replace them first.