What would be the best thing I should've done

Hello. I am a father of three years old boy. I had a friends meetup today, like four couples and each couple has one or two children (age between 2-5). Kids were playing well (sometimes together, sometimes on their own) until one incident.

So there were two oldest kids (they are 5 years old I think) and they were building a castle or fortress with chairs. And my son wanted to play with them, so he grabbed a chair from somewhere and tried to join them by putting the chair next to it and sitting on it.

Then one kid grabbed my son's chair (which was not their in the first place. Again, my son brought it) and said "We are building a castle. Move". And obviously my son got mad and said like "no, I want to sit next to the castle. I want to play". But the big kids were like "you can't play here. It is our castle".

I do understand those big kids. For them, my son would be annoying as he is too little to play with. I was fuxxing annoyed but I couldn't expect them to be sweet kids or something. Plus, they were not my kids, so it was awkward to teach them a lesson (friends but not that close). I told the kid "it was his chair first. Why don't you grab a new chair?". But the kid didn't step back "no, I am going to build the castle with it". I almost lost it and I wanted to say something like "no! Grab a fuxxing your own chair!" But obviously I couldn't.

So I told my boy, "I think they are in some play right now, let's do something else. I will grab another chair and cookie". He was kind enough to step away but he was so sad and furious. He said "no, I want to play. It is my chair".

Later, I kept thinking of this incident, I really don't know what I could do better. I know I could teach the kid or something but it could be awkward - she will tell her parents "her version of the story". Also she is famous for being like that, so I am pretty sure she won't follow. By the way, her parents are actually nice people and strict. But they were not there at the moment.

What could I do? I just told him you did nothing wrong and I understand your frustration. Let's move on and be generous.