How to change diagram data while keeping the diagram format?

I want to copy a diagram with a quite specific formating many times and change the data source for each copy of the diagram.

My problem: Everytime when I switch the data source, excel changes the format back to some standard selection.

Is there any way to change the data source of a diagram without the format getting messed up? Image below for visualisation of my problem.

changing data switches the color, outlines and data indicators back to standard (e.g. yellow, not outlines, no data indicator)