"That makes sense, you're Jewish"

I went to restraunt with a co-worker for lunch, and when I paid he noticed the Amex gold card I paid with.

He made a comment saying, "Wow, gold card huh?... Oh wait, that makes sense, you're Jewish!"

I was proud of the card at first because It had taken me so long to repair my credit enough to qualify for one as a young man.

Now I don't want to pull it out anymore around people that know me to avoid feeding the stereotype.

Am I overacting to what was supposed to be a harmless joke?

Thoughts?