Seems only if the speaker is removed that both parties get a vote in the matter of speaker of the house.

https://history.house.gov/People/Office/Speakers-Intro/#:~:text=These%20candidates%20are%20elected%20by,chosen%20by%20the%20two%20parties.

These candidates are elected by their party members at the organizing caucuses held soon after the new Congress is elected. In cases of an unexpected vacancy during a Congress a new Speaker is elected by a majority of the House from candidates previously chosen by the two parties.