LOL. Aliens from outside the garden, I'll give that, although the garden is alegedly located on earth, and we were made from the earth (uncursed earth, at the time). So technically we are terrestrial to the planet.

I've always taken the "not of this world" to be more the socio/cultural aspect. You could say we are cultural and/or societal aliens then, even if we are physically natives.