I say no, you're not too big. But try one on, if you feel it's too small go the the Uly. I also don't worry about what you look like, ride what you want to ride.
I am in Corner Brook, Newfoundland. Very little long straight aways and lots of hills with many twisties. Buell pilots dream and I have the only buell here out of a 50,000 population. When I pass by everyone has their head on a swivel. Love the Buell.
Dude I'm 6'3" 240lbs with a 36" inseam, and I can ride mine all day with no problems. Besides if you're that big no one is going to give any crap about riding a Buell.