It turns out your visual system has really strong views about whether faces should be inside-out or not, and it's willing to execute a hard override on perception if it doesn't like what it sees.
Madness string 81366
Does that make sense? I didn't think it did.