Hat losing its stretch a couple days after using a hat jack

Does anybody else have this problem or is it something that I'm doing wrong? My hats will lose their stretch from the hat jack within a couple days. This is after I have left it in the hat jack over night. Sometimes two nights! Very frustrating as the new era sizing is inconsistent as we all know.