I have never really delved into kv cache implementation, do they run effectively separate caches per layer?
If so I can see it all dividing nicely, computation and data size wise and the only slowdown would be in search layer waiting for it's turn. If you pipelined it you could run multiple queries.
Is anyone doing best-of-n with a n stage pipeline running each query offset by one?
Each stage has its own KV for the layers it hosts. You are on the money there, when one stage is waiting it's free for more parallelism. I am planning on exploiting this for more token verification through ngram spec decoding.
If so I can see it all dividing nicely, computation and data size wise and the only slowdown would be in search layer waiting for it's turn. If you pipelined it you could run multiple queries.
Is anyone doing best-of-n with a n stage pipeline running each query offset by one?