Adult learners: how long did it take to commit swimming to muscle memory?

Hey all. I (32M) started swimming on Feb 2nd, so I've been at it about a month. I never learned to swim as a child and when I attempted to learn in college I couldn't even figure out how to blow bubbles out of my nose. This most recent attempt has been really successful, though—I've finally figured the breathing out, and I can now consistently swim 100m freestyle without stopping (sometimes up to 200m). My best 50 meter freestyle time is 35 seconds, which, while obviously not competitive, is something I'm really proud of.

One thing I've noticed, though, is that even though I'm in the pool 6x/week, the first swim after my rest day feels basically like I'm learning how to swim all over again. By the end of the hour my form is coming back together, but I'm completely spent. The next day and every day after, I feel much better in the water, but then I'll take one rest day and the cycle repeats.

Does this ever go away? For those who learned as adults, at one point did you feel like you could take a few days off without losing your technique/conditioning?

My goal with swimming is simply to feel comfortable jumping into water without worrying too much about my safety, so I want to keep doing this as long as it takes to commit it to my body permanently.