These candidates are elected by their party members at the organizing caucuses held soon after the new Congress is elected. In cases of an unexpected vacancy during a Congress a new Speaker is elected by a majority of the House from candidates previously chosen by the two parties.
Did you mean switch parties?
https://history.house.gov/People/Office/Speakers-Intro/#:~:text=These%20candidates%20are%20elected%20by,chosen%20by%20the%20two%20parties.