No-one cared when there was a giant in my back-yard.
I usually say it's not offensive in order to preemptively mollify the people who don't have the mental processing power or political will to use my correct pronouns, and to head off the inevitable speech about why they got it wrong and the subsequent avoidant behavior as they subconsciously categorize me as a source of shame and/or avatar of wokeness.