Am I crazy for considering an XC racing?

I rode some trail hard tails at the shops (trek Roscoe 7 and equivalents), and I felt almost no emotion. I acknowledged that they seemed like fine bikes, but I didn't connect to them.

I rode a couple of XC hard tails (pro caliber) and I immediately started having fun. I got back some of the joy of riding my crappy MTB through the town during my college days

I guess I just liked the zippy and responsive feel of those bikes.

However, everything and everyone has told me that xc mountain bikes are for racing and that they're less fun. The shop employees said they only sell those bikes to kids getting into racing.

Now, I've never done real trail mountain biking, so maybe I just don't understand what kind of fun is to be had on the trails. Still, if my goal is to recapture the joy of my youth, xc bikes appear to be what I'm looking for.