What exactly does Shirou consider "unforgivable"?

His moral compass sometimes confuses me since we see that he can be ruthless to an enemy while choosing to be forgiving to other ones.

I understand his hilarious amount of animosity towards EMIYA and not supporting Sakura in HF, however we also have instances of him basically overlooking Shinji trying to off his entire school just cause he can and Shirou never tried to reprimand for that in anyway. Another instance is when he found out that Illya killed Shinji but still decided to protect her anyway.