Given the massive imbalance in the memory bandwidth bottleneck, I wonder why specialized hardware is the way it is. Is there some use case in which processing is the bottleneck, or at least it's more even? Are we expecting some software paradigm shift which will change the balance? Why couldn't they just make a cheaper, more rounded card which isn't heavily underutilized because of a large bottleneck?
But llama.cpp, and llms in general, are very atypically memory bound, even for AI workloads. Other models/backends will typically make more use of compute and cache.
For llms specifically, cloud hosts will batch requests to make better use of GPUs.
IMHO the current bottleneck is driven by the partial overlap between the gaming GPU market and machine learning needs. The expensive things that are needed by both (i.e. parallel calculations) are included in all cards, but expensive things that aren't needed by games (i.e. memory bandwidth) become a bottleneck unless you use expensive ML-specific niche hardware instead of consumer (gamer) GPUs.
