It goes fundamentally against the idea of rule of law if you're allowed to opt out of having your disputes be judged by the courts.
Humans are either hits or misses. The misses make themselves known quickly, but it takes time to know if you've found a hit. And spending that much time on every person is bothersome. As far as I'm concerned, the misses that trespass on my home to steal my treasure are better. However...