I have indeed thought about it, and think your concerns are stupid.
We don't know what a large fraction of our genome is exactly supposed to DO. Sometimes it does nothing; a large genome is no guarantee of the complexity of the organism. The humble potato has a larger genome (number of base pairs in its DNA sequence) than humans do, and yet it just kind of sits there and... potates.