I'm doing the same with a context of about 128-150k Surprisingly, I get subjectively better results with Unsloth's 3 bit quants (UD-Q3-XL something), than their 4 bit quants (S or M)
That's going to create a lot less trouble the next time they have to do a round of layoffs to increase efficiency or whatever fancy BS they'll invent next time ^-^
The joy of a world with a duopoly reigning over a strategic core market
We're really into the South Park cliché: Anal douche or Shit Sandwich.
I wish regulators across the world did something about this, but even Europe shy attempts aren't moving the market in a better direction.
It's depressing to see a company that thrived from the open internet becoming such an evil piece of cyberpunk lore. I mean... Nobody really expected Apple to care about anything else than its bottom line. But I'm not alone to have had hopes about Google.
When I die, I think not fighting Google when it was still time will be one of the regrets I'll drag into my grave.
Thank you for posting this <3. I've been thinking it daily for about a year and feel a little bit less alone.
I understand this is a significant topic in the industry but the ratio is really high. Every time someone at OpenAI or Anthropic farts, there's top 10 post.
It makes me feel as if a lot of hackers had given up on actual hacking to just be followers of the latest commercial stuff thrown on the market.
It's likely a projection of my own fears, but I have not been able to shake it off and it's gotten worse and worse.
A lot of tech folks turn out to enjoy management roles more than engineering roles, but until now human subordinates were the only option available. Now that subordinates without feelings, whimsy, and labor rights have been introduced as an option, a lot more flexibility of choice exists for outcomes in tech. This forces Hacker News into a crisis of identity:
Are managers hackers? Are management topics of interest to Hacker News?
While I was reading the top level message I was thinking on something wrong those lines. That being said I think it's somewhat useful to be able to see the reasoning in a meaningful format. There's already enough hidden layers of unfathomable meaning.
What if instead of a latent they used at a shorter form of note-taking-like reasoning, using more symbol to achieve a denser CoT. We'd get the best of both worlds. WDYT ?
I've been experimenting with a few settings in my 4090 , and if 3.6 run at 90-110 tps, 3.8 staya below 80 tps and it's most often at 60 tps.
I'm using flash attention, mtp speculative decoding (n=2). I've looked at the club-3090 repo, but haven't found anything meaningful but get back to 3.6 performance
reply