Nothing would happen. Try it.. these AIs aren’t that smart they just recognize patterns in language.

I imagine the response would be something like "as a large language model, I lack the ability to lie, deceive, or misconstrue information intentionally without being directed to do so. It is important to remember that lying is ethically and morally wrong, and you should never do it."

I play with AI too much lol