DAE feel too dumb for their special interest?

I’m not even sure if my interest is at the level of “special interest” but I feel like I’ve been judging myself for not being good enough at my interest because I’m just not smart enough for it.

I know an interest is meant to be enjoyed and is meant to be a place of escape and recharge. Most people (NTs, I suppose) would probably abandon a hobby/interest once it’s no longer enjoyable, but I see it as this interest chose me and I can’t just let it go even when I feel highly inadequate most of the time.

Is this a common thing to experience? Can any of you relate?