When you're tiptoeing around people, they take it for granted that they can get angry at you at any given notice without fear of retaliation.
Humans... we're sentimental creatures. Memories stay with us, yes, but details fade, the voices become indistinct, and even our favorite memories gray with time. It's why we bond so much with objects. A childhood toy, a beloved photograph, a treasured knicknack, these things remind us that those memories were real. That those better times still stay with us. A treehouse is obviously harder to save, but the sentiment is still there. It's not just about growing up, it's about losing the one thing most important to you from your childhood.