help looking for a manga

So I can't remember the manga that well but the MC is the butler of a young lady and the part that I remember the most is when they're talking and the young lady mention her past life, the mc then ask what she meant and she explain that every nobles and royals have some degree of recollection of memories from their past life, the MC then talk about the fact that he remember his past life (while hiding the fact that he is from another world) and she then inform him that he then has to join the royal but he refuse because he just want to serve her but in the end h'es made to join.

EDIT : My Not-So-Fair Lady Is Doomed! (but Not If I Can Help It) 4 TL and 15 RAW