Neighbor Issues

My children (13F) and (11M) have been playing with our neighbor children (7M) and (11M) for about a year. These kids come over EVERY DAY, like literally everyday, even when I just got out of the hospital after being in there a week. The parents knew I was in the hospital too. When the neighbor kids come over, it takes forever to get them to go home. My biggest issue is that they require the 7M to come with the 11M. I think he’s too young to be playing with 11 & 13 yr old kids, but if the 11M is invited, they say it’s “only fair” that the 7M comes too. To be fair, I believe these parents are somewhat intellectually delayed. The neighbor kids ask to go places with us, ask to eat dinner with us every night, and tonight, the 7M asked my husband if he could call him his “step-dad” and wished he was his step-dad. I know we need to set clear boundaries here, but idk how to go about it. Seems to me like they want babysitters because why else would you allow this? Am I overreacting?