We may not be responsible for the world that creates our mind, but we can take responsibility for the mind with which we create our world.
I usually say it's not offensive in order to preemptively mollify the people who don't have the mental processing power or political will to use my correct pronouns, and to head off the inevitable speech about why they got it wrong and the subsequent avoidant behavior as they subconsciously categorize me as a source of shame and/or avatar of wokeness.