Other translations use brethren.

https://www.biblegateway.com/passage/?search=Matthew+13:55&version=GNV

55 Is not this the carpenter’s son? Is not his mother called Mary, and his brethren James and Joses, and Simon, and Judas?

In full context, they weren't calling into question his heritage, they knew it quite well, but rather they were calling into question his claims of being the Messiah. He did not fit the prophecy, because they were aware he was not "Of the House of David" in terms of being related to the royal family as they were expecting from the Messiah.

54 And came into his own country, and taught them in their Synagogue, so that they were astonied, and said, Whence cometh this wisdom and great works unto this man?

They were more upset that he was just some scrub carpenter's son and not a son of a king. They further were upset that a carpenter's son would have such deep understanding of scripture when they said "Whence cometh this wisdom and great works unto this man?"

They wanted to know who taught him all he knew, and refused to believe a nobody would dare speak with such authority.

I'm not saying you're wrong in him having siblings of Mary, but in context they used his perceived siblings to reject him as the Messiah. He didn't fit their preconceived notions.