To defend against enshittification. Why would I use a big tech controlled resource when avoidable? If I avoid consuming or building on imdb in this context, I avoid inevitable platform decay. Investing in and leveraging public goods > eventual rug pull. Try to avoid the rug pull, when efficient to do so.
You can run larger models by offloading to SSD (for weights), it's just slow so people don't do it all that much. But you can get back at least some of that performance by using either MTP (at least for dense models; not effective for sparse MoE models unless you're batching them already and have VASTLY more parallel compute than you'd know what to do with) or batching multiple requests in parallel (note, this hurts throughput for your single sessions but running more sessions in parallel still boosts your total amount of inference. This requires careful management of memory requirements for your context/KV cache, and Qwen models tend to be KV-cache heavy).
Broadly speaking, this ultimately pushes local inference towards a challenging world where you use SSD offload for weights as a matter of course; then smaller requests (or requests sharing the bulk of their context, e.g. subagent swarms) can be batched together and run quickly in aggregate, but running very large contexts will actually limit you to single-session inference and require swapping out even the KV cache itself to some external scratch SSD, further hurting your performance. Then feel free to add wide use of MTP in a probably futile effort to go back to tolerable tok/s numbers.
there's been a lot of research about reducing models by taking out layers; there's also using it to train smaller models by optimizing parameters.
I dont see most model building as anything more than a pig at a slop troth, despite the level of sophistication; they're still rarely pruning the input beyond random sampling.
Vulnhunter isn't that exciting but if you scroll down their feed you'll see "guide to common rust errors" which... also isn't very exciting. I think they just need to publish something every week.
fwiw I have seen good whitepapers from them. A while back I used one about their IVR to get exec buy in for re-doing my company's IVR into something a lot better.
Eh, Capital One has long been surprisingly progressive on open source and whatnot. They were one of the first to properly adopt OAuth to connect accounts, too, back when Plaid/Mint/etc. were mostly proxying logins.
reply