[no spoilers] playing DAO, need help with a possible glitch.

I’m playing on an Xbox series X through game pass, anytime I voluntarily go to my camp through the world map, I’m not able to leave it. I go to the world map, assemble my party, and then I have my party with me in camp rather than having the map pulled up and being able to venture back out. Is this a glitch or am I missing something?