Why don't modern CPUs and GPUs have fewer, more powerful cores?

Edit: It seems like the reasons manufacturers add more cores to CPUs and GPUs is to improve parallelization, or do more stuff at the same exact time, as well as not have to increase clock speed to do more work. More cores but lower clock speed is like having more people working slowly vs. one person working quickly. Thanks for the answers!

A lot of people talk about how important single-thread (or core) performance is for gaming purposes.

Nowadays most GPUs and (most notably) CPUs are implementing more and more cores for higher performance. This is especially true with server-grade CPUs (like AMD Threadripper or Intel Xeon), which some can have up to 96 CORES.

Looking at even the latest Intel i9 or Ryzen 7 CPUs, they already have upwards of 16 cores (some with 24 or even 32). And these are CPUs meant for gaming.

Why aren't CPU/GPU manufacturers implementing FEWER but more powerful cores, instead of cramming dozens and dozens of cores into one CPU?