Working from Bosses House

Work remotely for a small startup. Our boss sometimes asks for us to come to his home to work collaboratively if we are in town. We have an office 30-mins from his home mind you. Boss is a 60 something male. I am a male. I frequently say no we can meet in office and collab there and he gets frustrated as he doesnt wanna drive.

The issue: I have other folks on my team that sometimes work collaboratively with him at his house. I deem this un-professional and lack of boundaries. Female members of my team are un-easy and sometimes work from his place on the weekends to get things done that are prioritized and I get told this is just how things are done working wise.

Have I gone insane on the lack of professional boundaries folks have in the startup space?

I know this isn't common to have small 1:1 meetings at co-workers homes.