It's not only on television and in movies that coroners can tell the sex of a victim (even if only the skeleton is left) by their bone structure.

For that matter, ancient human (and even more-ancient proto-human) fossils tell the same tale: male or female. Humans have gigantic brains for their body size and getting that skull through a birth canal requires a different pelvic shape with a larger opening than males have. Males also have larger and thicker skulls (females have smaller neuron size that provides brainpower parity), wider shoulders, and a bunch of other differences that don't change just because someone surgically removes their testicles (or breasts) and takes hormones.