Keep in mind these LLM models are just statistical predictors. They don't understand the concept of "I don't know". Whether they "know" the answer or not, they will start producing text and keep adding words that make the reply sensible. If they lack the context and data to form a reply based on facts, they will happily give you a reply based on pure fabrication. Consider that factual data to these models is just a way of biasing the output. Actual facts biases the response towards those actual facts. Lack of facts gives you something completely random and unpredictable.

Just because the model answered the question of what will happen in 2334 does not mean the model had any real training data or knowledge of what will happen in that year. It simply strung together a sequence of words that sounded reasonable.

ChatGPT will never answer a question with "I don't know." That is a human construct. ChatGPT doesn't know what it doesn't know.