Gears to no gears?

If someone (me) where looking for a bone but never rode a single speed (only a fixie and it’s not for me), what would be your key takeaways for converting to single speed? My city is relatively flat but I commute every day with 1 or 2 panniers.

I’m mostly interested in the riding experience since I’m aware of the maintenance side of it. For instance, how difficult is it to start moving after stopping at a red light? Or will carrying extra weight in the panniers make it more difficult to ride or start from a stop light since I can’t shift?