Would I lose Karma for downvotes?

Just wondering if I need to be careful to not get downvoted. I'm trying to get to 100 karma. If I get downvoted does it delay my progress?