This is why a lot of people with lazy eyes develop monocular vision. The brain just kind of decides 'yeah, no, that's wrong' and ignores anything coming from that organ. It happens easier in children. The alternative is so wonky it'd cause severe headaches, dizziness, etc. The images are so separate and your brain trying to work with both when it's used to a single joint image is usually going to result in the brain ignoring the image you're not focusing on.