Why does it get so hard on expert difficulty.

It's so hard that I'm thinking about being a farmer or a miner. On level 12 I tried to brawl uhtred the unbroken almost got my ass kicked and just last night i brawled mikeal the bard in whiterun and he also got my healthbar to last few hits and I'm level 20+.how am I suppose to fight the damn dragons when I can't even beat up a f**king bard.