> It's a fine project to do but clearly they put zero value on being familiar with the project's codebase/stack and ecosystem,
As much as I dislike it, I think that's the future of _all_ non-critical software (think social media, crms, CI, food delivery etc). Leadership in many companies is explicitly asking employees to have multiple agents running through the day and that will lead to this.
Read this for example: https://www.uber.com/in/en/blog/efficient-software-factory/ . A very useful system, I am sure. But when you have AI at every layer from code to review to triaging, rest assured AI is the only know who knows your system. And you better hope it's not telling you that something is load bearing during an incident.
I read the article and couldn't understand it. I asked Gemini; Pareto things definitely sounds like science.
I read the article again:
- We had bugs. We let agents try to fix the bugs. True positive (fixed bug) is the F1 score. Here are the results for different models. Fixes worked 50% of the time.
- We needed to find a query. Without a knowledge graph it took 20mins, and didn't work. We used a knowledge graph. It was fast (20s), and found the right thing.
I gave my LLM my summary of the article and apparently I "hit the nail on the head".
Although I don't think it even matters. They could easily cherry pick a period that is favorable for them. Wasting time and money on short term trading, rather than long term investment, using LLM or not, is never a good strategy for most people.
They backtested with public models, wouldn't the model weights already have the data? I double checked with chatgpt and looks like agents also had web search tool available so they could just lookup the past.
Opus 5 is remarkably bad at instruction following over long chats. I have to repeat “Be succint”, “talk like a friend or colleague would”, “no rambling” or some variant of it every few messages
Have you tried a custom output style? CLAUDE.md instructions are appended to the system prompt. A custom output style can replace the system prompt. At least the part of it pertaining to voice and persona. The reason it forgets over long chats is the context size starts getting too large. Instructions weigh more strongly the later they appear in the context. This is necessarily true otherwise you couldn't change your mind in a conversation. The model would stick with what you originally said. For the output styles, there is a per-turn "reminder" that gets added to the context asking it to "remember" the content in the system prompt. That's why it has more staying power than the CLAUDE.md instructions in long conversations.
ai has been an excuse for people to shove into pr's all kinds of shit that would never be tolerated otherwise; from super long descriptions to annoying emojis in commit messages and overly verbose comments and code that has the mpg of a broken scooter...
The thing is beyond like top 10 AWS services(ec2, s3, sqs, aurora and few more) this is how most of AWS is. It’s just stuff thrown at the wall to see if it sticks.
I wouldn’t be surprised if median AWS service does less than 100k in annual revenue
reply