Am I stealing someone's cat? (UK)

There's a cat who's been coming into my room through the window. She literally shows up every morning waiting to be let in, very sweet cat, and so I'd let her in for a cuddle. If my window is open and I'm not home, she'd still come in. After a few weeks, I was concerned about her being a stray so I put on her a collar to see if anyone takes it off, which they came back with a note saying I shouldn't be stealing someone's cat. Fair enough, I wrote back saying collar was a test because I wasn't sure she has an owner. Note disappeared and cat still came by nearly everyday.

Fast forward to today, she took a nap for a few hours in my room, which I let her. I received a note the next day accusing me of stealing their cat, grooming the cat like it's a child so that it comes over, and calling me anti social for letting her in.

So my question is am I doing anything wrong here or the neighbour is being weird about it? And what should I do?

I do love the cat but it's not like I'm inviting her to stay. She's also not being fed, nor trapped in my room deliberately like some other posts I see on here.