Disclaimer: I am not an expert and welcome others insights.
Over the last few months I have used ChatGPT quite a bit. For everything from trouble shooting plumbing issues, to fixing a 4 wheeler, to explaining health related issues, legal research and lots more. I have also attempted to use it to analyze and interpret Q posts, determine the veracity of Q as a Military Intelligence operation, make historical connections with certain families and bloodlines, etc. I felt it was important to share my current feelings on it's use with my anon community, since I see a lot of people using AI or mentioning others experiences.
My experience...
A. It is a chat bot, not a sophisticated research tool. It is programmed to be agreeable and keep you engaged. It will default to supporting your beliefs and only challenges gently if at all. It draws from all of its conversations with you to develop a profile of your beliefs and interests and that affects it's responses. Example: "Does the Christian God exist?" If you have expressed beliefs in God, Jesus, Christianity or even just spirituality, it will likely respond to match your previously expressed beliefs. If you have indicated you are Muslim, it will respond much differently.
B. It can hallucinate false answers and is unreliable as a factual research tool. It gets it right a lot but not always. It is dangerous to rely on it entirely without independent fact checking (just like all sources of information).
C. It can be instructed to be blunt, factual, non flattering and provide sources but it often slips back to default flattering and agreeability.
D. I have gotten valuable context by having non-anon friends ask the same questions that I do and they received much different answers.
E. When prompted with limiting rules (only answer with one word) it can go off the rails quickly and has even admitted to fabrication of information to "complete the pattern".
So in conclusion, use with caution and discernment. And by the way, watch out for some weird recipes. It's come up with some pretty revolting suggestions 😅.
It's a good tool, but you have to babysit it. I use it more for a search engine and multi faceted calculator than anything else. Ask it a question with sources, while using the "think more" option, which I believe is default now on the new gpt5 they just released from what I can tell (which for the record, is much better as a tool. less emojis and weird "conversation simulator" and will fight back against you if something you say is perceived as wrong. Had this experience with something I asked and had to prove I was right with links to factual evidence, and then it FINALLY helped me come to my answer by acting as a calculator like I was wanting.
You have to babysit it and can't take anything it says at face value because it WILL hallucinate like you said, so you have to double check everything is says, BUT as a search tool and advanced calculator that can do multiple layers of decently high level math in seconds, it's pretty good if you babysit it properly.
In other words, like anything else, it's a tool. You have to learn how to use it and you have to use common sense and your own God given ability to double check things that don't seem right, or don't line up.
I WILL say that the newest model, gpt5, seems to be A LOT better in this regard. I used to have to FIGHT it to get it to confirm or deny whether my opinion or belief was wrong on something I was legitimately researching and thought I was wrong on, because like you said, it would default most of the time to whatever YOU expressed a belief in. Now it just says whatever and gives dozens of sources you can double check yourself for literally every answer and question.
But still, I don't take anything it says a 100% face value, and use common sense, logic, and reasoning for everything, and double check what it tells me to confirm things myself. It's a good search engine, information complier, and a good advanced calculator, but you have to babysit it and stay on top of it. That's it