Agreed, and there is already a feature request for this. No need to start another, just add votes and comments to the existing one. (Also under discussion already is how to work around the problem.)
Control points and playing around with the options and selectivity should narrow down your choices fairly effectively in a small area, I’d start there for now.