That's NOT a problem. The reality is that the President being sworn in at a certain time isn't how the transfer of power works. The wording of the law reads that the old President's term ends at noon - at which time the new President's authority comes into effect.

Even though Biden was sworn in early, he didn't get to assume Presidential authority until noon.

This is to prevent any power vacuum between Presidents, even if only for a few moments.