If you dont have a GPU, prompt ingestion is totally threaded. The more cores the better.
For generating tokens, more cores helps to a point, but then:
- You start saturating the memory bus, and performance plateaus.
- There is some overhead from the threading implementation, and too many threads hurts performance.
The ideal number of threads varys per CPU. For instance, using hyperthreaded cores or Apple/Intel e cores typically hurts performance... But not always. You just have to test and see.
The problem is memory bandwidth rather than CPU cores: "Memory bandwidth is the limiting factor in almost everything to do with sampling from transformers. Anything that reduces the memory requirements for these models makes them much easier to serve"
