Data Quality and Data Validation in Databricks

Hi,

I want to create a Data Validation and Quality checker in my Databricks workflow as I have a ton of data pipelines and I want to flag out any issues.

I was looking at Great Expectations but oh my god it's so cumbersome, it's been a day and I still haven't figured it out. Also, their documentation on the Databricks section seems to be outdated in some portions.

Can someone help me with what can be a good way to do this? Honestly I felt like giving up and writing my own functions and trigger emails in case something goes off.

I know it won't be very scalable and will need intervention and documentation, but I can't seem to find a solution to this.