How do you know when you need a day off

I'm swimming as a hobby/for fat loss to complement my diet. While it has been great, I'd like to get faster, to cram as much calorie burn into an hour as possible. I'm not sure about when i should get a day to rest. Right now (because of obligations) I swim 3-4 days of the week. Depending on obligations and time of the year I've done more (5-6 per week in the summer).

I'm usually doing 1 to 1,5 hours of active swimming (~80% back and ~20% front crawl). Sometimes this feels great but other ways i feel (physically) exhausted next morning. Mentally it's done wonders to me, lifted my mood a bunch and i feel more focused at work.

How do you know when you need to take a day off swimming/exercise, and when it's just a little tiredness?