I'm thinking your belt tension is fine if you were able to get it on. It's suppose to be rather tight, and will loosen up(slightly)in time. Go easy for 100 miles or so(this is just my stab at a guess). For the record(and as I laid out in post #15), It is recommended to ream both holes. Leaving only 3mm's of bracket material at the edge is just too thin IMO. If you're that close, and in addition to now touching the muffler, I'd guess that your flange nut when installed is hanging over the edge a bit. I tend to be a bit anal about these things(Dad was an engineer)and couldn't live with that. Symmetry is best in this application.
You have a few options. 1.)Dimple your muffler to accomodate, and go ride 2.)Find another tensioner bracket and start over doing both the holes 3.)Shim up the oversize hole and ream the other one until you get the right fit.
Edit: I just looked at the space between my pulley wheel and my stock muffler. I don't know how your frame varies from mine(if any)but the space where it is the closest on mine measures 3 mm's, and it is right at the rear muffler strap. Are you saying that yours is touching on the strap there? If so, a "dimple" as I previously mentioned would be a bit more problematic. I see that as the pulley wheel drops down, it would get closer and closer to that strap. Good luck, let us know how you work it out.