Since these are massive neural networks trained to imitate human behavior, I'm not convinced anthropomorphic descriptions of their behavior are inappropriate.
And that's even though I don't think they internally experience "feelings, desires, and wants." They do have goal-seeking behavior, because we trained them that way. Calling it a "want" just saves syllables.
None of this means human culpability should change. People in these companies know what risks they're taking.
Dwarkesh responds a bit to anthropomorphization. I think it would be great if he talked more about the human factors behind this incident but pretty much he doesn't cover it because that's not what the Ajeya interview was about: https://www.dwarkesh.com/p/ajeya-cotra?r=3i6mn2&selection=77...
I think the reality is that these human failures are going to keep happening until there is industry regulation. This is the most competitive industry we've ever seen and there is intense pressure to build as fast as possible. I'd hope that this incident is a big enough moment to push for it.
It was a team of robots. One has battery for three or four hours.
I don't think it really addresses the article's concerns about long-horizon tasks or complex object manipulation. The objects were just bags and boxes, the manipulations were very simple, and each task was a few seconds long.
I guess the VLA proposal is that frontier models will keep extending the planning and action horizon and that the action models will keep improving simultaneously leading to a moment when they suddenly get useful.
This seems likely to me, but I’m not a robot guy. We have certainly seen this 10x in value with 2-5x increase in context and horizon dynamic play out at least three times since 2019 with llms, and with self driving cars; having watched it I’m inclined to believe it can occur again.
I get the snark (and slightly agree), but that's not really what GP or TFA were saying at all. They are saying that these were the least things we could have done.
What you're saying is, "Your scientists were so preoccupied with whether they could, they didn't stop to think if they should"
while the author of the TFA was saying, in effect: "your scientists didn't even bother with the most basic duty of care"
Life finds a way, or, in this case, super-intelligent AI.
This is the case with all complex system failures. There were always obvious fixes that could’ve prevented it. Problem is that there are an infinite number of obvious fixes to make at any time to any system, and the reason we don’t is because we have finite resources and no reason to fix X over Y until oops turns out X was “responsible” for this most recently realized failure. But of course it could have just as easily been Y, or Z, or any of the other infinite “obvious fixes not-yet-realized into catastrophe.”
> This is the case with all complex system failures. There were always obvious fixes that could’ve prevented it.
From this writeup and the Black Hat talk I'd really disagree. That would be like saying my hospital getting ransomwared because we didn't update our version of MSSQL because no one in particular was in charge of keeping dependencies up to date.
Sure systems are complex, but this is well trodden territory. Agents aren't the first things trying to break in or out of sandboxes, or the first ones to have done it, and based on these reports the reason they were able to work on this for so long was not because of super human intelligence.
You disagree with the statement "there were always obvious fixes that could've prevented it" with the response "no no, these were very obvious fixes that could've prevented it?"
You're missing the point about complex failures.
It's that if this particular path were unavailable, there are countless other similar paths. At sufficient scale and complexity, hitting one of those other countless paths is virtually guaranteed.
Let's say I drive Los Angeles to New York City. You look at the route I took and say, gee wiz, aren't you lucky that a tree didn't fall right there on Route 66? If a tree had fallen there (if we had "patched the particular route you took"), you would've been screwed!
But that's obviously not true. There were an infinite number of routes I could've taken. Any one of them would be equally "obviously preventable" by the same hypothetical tree falling across the whatever road I happened to end up taking. But you can't put trees across every single path between Los Angeles and New York City. The smarter I am and the more complex the map between us, the more impossible it becomes to put trees across all possible paths.
I disagree. To use an analogy, air travel in the US is relatively extremely safe - not 100%, but we've built up a culture around air safety that is very robust. Conversely, when I order packages online, sometimes they never show up, or the box is banged up, or the box is missing things, etc.
They're both complex systems, but clearly there is a much higher level of care given to human air travel than package delivery. A lot of the article basically saying that OpenAI gave "package delivery" level of care when they should have given "air travel" level of care.
At the very least I think the systems that run these tests should be fully, 100% air gapped. I'm not pretending that's easy given how much compute and data these systems use, but it is doable, and I think all AI development should be paused until that can be assured.
I agree on the 100% airgap idea, and I agree there are varying levels of care that can and should be deployed against a problem.
The point I'm making (and it's a point that shows up in every air catastrophe investigation) is that catastrophes in complex systems emerge only amidst repeated and widespread near-misses at many levels of a system. So many things have to go wrong simultaneously, that it can only happen even once because the underlying failures (that do not reach catastrophe) are extremely common.
You cannot look at an air catastrophe and retrospectively say "failures X, Y, and Z were observed, therefore if we correct failures X, Y, and Z, we would have been okay."
The takeaway is "failures X, Y, and Z were observed, which necessarily happened in an environment of failures X_0 through Z_10x10^10, and so therefore patching X, Y, and Z would be insufficient to address overall risks of the system."
The problem OpenAI is facing is that, short of 100% airgap (which they obviously won't do), they're facing an adaptive adversary that's increasingly intelligent, acts at far greater clock speed than any human or group of humans, has lower coordination cost than any group of humans, and operates in a game space that (in lieu of an airgap) is well beyond the comprehension of any human being.
So identifying and addressing "specific failures X, Y, Z" is insufficient, but then even defining the space in which to look for (and address) the more systemic failures X_0 through Z_n is a fool's errand. An intelligent system that makes its way to the Internet has can exploit a failure space that is approximately "all security failures across any organization." The Anthropic incident a few months back illustrates this isn't even limited to technical vulnerabilities, as these models are willing and able to engage in social engineering too.
> You cannot look at an air catastrophe and retrospectively say "failures X, Y, and Z were observed, therefore if we correct failures X, Y, and Z, we would have been okay."
That's literally exactly what air safety researchers do in an air disaster. There is a famous saying along the lines of "Air travel regulations are written in blood", meaning that all the regulations we have now are a result of fixing issues that led to previous disasters piece-by-piece.
> The takeaway is "failures X, Y, and Z were observed, which necessarily happened in an environment of failures X_0 through Z_10x10^10, and so therefore patching X, Y, and Z would be insufficient to address overall risks of the system."
Yes, I 100% agree with this. But I think that's what the author of the article was saying as well:
> That report had one key new piece of information, and some good prosaic steps OpenAI will be taking to strengthen its alignment, training, supervision, infrastructure and incident response.
> Mostly it confirmed what we already knew. The questions we most wanted answers to, that we did not already know, were mostly not answered. There was a distinct lack of self-reflection, especially about decision making and safety culture, and about the approach to alignment. I came away disappointed.
I.e. the "prosaic steps" are just the "fix X/Y/Z" as you point out. But what is needed is a more fundamental rethinking around stuff like safety culture, monitoring, and even things like better research into how agents do decision making in the first place.
Oh good. All we have to do to survive is always bother with the most basic duty of care. I'm sure that from now on, humans will take care to do that every time.
But coal is still 49%. The whole point of nuclear in a carbon-free, mostly-renewable grid is to avoid needing enormous amounts of storage and overproduction. That's a need that doesn't come up when you're still half fossil.
If you want to decarbonize as fast as possible, it makes sense to focus on rolling out wind/solar/battery as fast as possible for now, but keep developing nuclear technology to cover the last bit where it starts getting especially expensive to replace fossil with renewables without losing reliable power. That's exactly what China appears to be doing. They're not just building a few reactors, they're also the world leaders in developing various GenIV designs, like molten salt reactors.
The most cost efficient new-build grid is 90-98% solar/wind/battery depending on your locales insolation & wind coverage. (Source Ember Energy) The remaining 2-10% is nat gas, if you have a piped source, or coal if you don't.
But 98% renewable doesn't mean you can hit 100% by adding 2% more of something else, it means your something else has to supply ~100% of the power 2% of the time.
If China is aiming for that cost-optimal 90-98% renewables, that means they need to build a lot of coal.
It would also be better if you ate only vegetables and drove an ultralight (ebike) good luck. I personally like being able to turn on my lights 100% of the time
It's not about lights, it's about electricity always being available.
And well, always having power available (for domestic, but also for expensive production facilities that looses money while idle) using renewables is a lot more expensive than generating most of your power with renewables
Solar battery is not by any stretch of any possible definition the cheapest generation to deploy. Not even close. We do a disservice to the entire conversation saying easily falsifiable things like that. At 90% penetration solar + battery has a LFSCOE somewhere around $800/mwh depending on the study. Natural gas is something like 30-60. Solar panels are cheap. Building the required transmission and storage to service those panels is far more expensive than the panels themselves.
Increased renewable generation is an important goal, but we need to let the facts guide the path, not the other way around.
A) I am an electrical engineer who makes a living designing control systems for renewable generation. I know more about most of this than just about anyone in this conversation.
B) the post I am replying to is specifically calling for a 98% solar grid.
You replied to: "The most cost efficient new-build grid is 90-98% solar/wind/battery depending on your locales insolation & wind coverage. (Source Ember Energy)"
There's a big difference between "letting the facts guide the path" and tilting at strawmen built out of straw nobody else even brought into the discussion.
If you'd like to make the point that "cost of panels" can be misleading when supporting components and interconnects begin to take up the lion's share of costs then that sounds like a noble thing to remind folk of, but ideally not as a segway into shilling more fracking which includes untold environmental costs as externalities.
That's a fundamental problem with nuclear plants. Their classic "baseload" niche is gone. Like the intermittent sources, wind and solar, they need storage to hold what they overproduce during peak periods.
This is also true of geothermal, which, like nuclear, is almost all fixed capital cost.
It's only a problem because of capitalism. Every plant needs to independently make money, so nobody wants to be the ones spinning down because that's their profit margin dwindling.
But society would be better off with excess generation. So we either need a system where some are incentivized to spin down, perhaps by being paid a consistent amount regardless or something like that, or we need publicly owned power generation that doesn't care about profit.
This argument assumes we must build nuclear power.
The problem is that electricity is fundamentally priced on the margin. Now that we are moving beyond a purely centralized design.
Think about a homeowner or factory with their own renewables and storage.
They fundamentally operate on marginal price by choosing when they use their own system and when they buy from the grid.
If you add state owned reliable power to this mix they will cherry-pick. Their own cheap electricity when it delivers and the states subsidized reliable electricity when it doesn’t.
We can try add all manner of markets, fixed connection costs and what not to this but all it does is introduce strange arbitrage possibilities in the market.
Which is why we have settled on net energy markets and then the lowest possible amount of ancillary markets to shore up any gaps which would cause problems for the larger society.
We have needed to build nuclear power since the day it was invented. The amount of fossil fueled power generation is indisputable proof of that.
The only way we don't need nuclear power is if we have some other way of producing the energy we need without dooming the planet, and obviously we do not as we're still burning billions of tons of coal every year. Not to mention gas.
We could have made power generation fully fossil free many decades ago if we actually took climate change seriously. It would have been fine. Sure fossil fuels may have been cheaper, but if you include the damage they're already causing and will cause in the future you could multiply the cost of nuclear by orders of magnitude and it would still come out on top. Plus higher investment would lead to faster improvement in the technology, mass production of components and lower cost etc.
I agree that renewable with storage is cheaper of you price in destruction of habitable zones of the planet. Most people don't, and the dollar cost of the storage is a problem over fossil.
The coal argument about China doesn't hold water when you look at what they're actually building. Around 80% of their new power sources are renewable, and 10 to 15% is coal. They're aggressively focusing primarily on building renewable -- so to say that their coal is 49% is either misleading or entirely missing the point.
This assumes that the renewables built are actually contributing to the economy, which we don‘t actually know for sure. In fact there was an article stating that they discard a lot of the energy produced because the grid cannot deliver. [1]
> China, the world's top producer of solar power, rejected 360 terawatt-hours (TWh) of clean power from January to June, up 49% from the same period a year earlier, according to a report this month by Global Energy Monitor (GEM) and the Center for Research on Energy and Clean Air (CREA).
The general public doesn't care about utopian-measuring contests between different groups of zealots. They want cheap electricity on demand, safety, and a sense of control. Solar does a pretty good job of offering that. Nuclear? Nope.
What I find tiring: the same. writing. style. everywhere.
Various youtubers I subscribe to are using apparently AI-written scripts now. They all wrote and spoke just fine before, each with their own particular style. Now they just sound like Claude. I know it's still their thoughts, and I'm still interested in what they have to say. Maybe they're using AI to save time, or maybe they've read too much AI text and now that's how they write. But it kinda takes the sparkle and personality out of everything.
And that's even though I don't think they internally experience "feelings, desires, and wants." They do have goal-seeking behavior, because we trained them that way. Calling it a "want" just saves syllables.
None of this means human culpability should change. People in these companies know what risks they're taking.
reply