If you don't want to grind on your mounts, you could always beef up the diameter of the bar with a few wraps of electrical tape and use a 2inch clamp. I'm not sure how well this would work for heavier accessories, but I have done this for my rearview mirror and its worked well so far.
I for one hate to start grinding on something before I can fit it up and make sure I like it.