It's got a "steep" learning curve for newer riders. It's not smooth like the 4's are at lower RPMs, like when putting around a parking lot/gas stations/etc. And these have a short wheel base, so it takes some good counter-leaning at lower speeds to keep them balanced & upright. Having said those, I don't regret it for a second, corners with very little input. (BTW: Not much difference between the XB9 & XB12, they output about the same numbers)
But it sounds like you've got quite a bit of dirt experience. Do take an MSF rider's course if you've not done much street riding. There's a lot of difference between riding on pavement and riding on dirt. (Guys I took my class with were motocross and commenting on things we were learning constantly.)
If you do get one, gear up fully when you ride. It's easy to get that "invincible" feeling when carving corners, especially since Buells are so intuitive. They're torque monsters so you can get in trouble Fast. Take an MSF, see how you feel, then maybe test ride one if you can.
Sorry for the long post, but in the end - Is it a great starter bike? It depends, like Adam said, on your maturity. Ride safe.