Shoutout to Toph https://x.com/VictorTaelin who's been building a programming language out of formal method Lean proof checkers, which probably would indeed have the scope you're looking for.
I do think the current state of the AI just using rigorous tests and rendering inspection loops is still far more than most programmers did for the majority of apps, but yes I think rigorous formal verification will come too. Worst case it's gonna be something like a MechanicalTurk pipeline having us humans verify narrowed scopes the AI can't confidently inspect (yet. while also training on those results for next iteration)
As for the decision stack of what makes a good end product - turtles all the way up/down. The ProductOwnerAI role decides those things, and it will likely do so with the same deep skill that programming AIs are currently hitting our profession with. Optimizing enormous breadths of concerns and simulating results is AI's main specialty.
I do think the current state of the AI just using rigorous tests and rendering inspection loops is still far more than most programmers did for the majority of apps, but yes I think rigorous formal verification will come too. Worst case it's gonna be something like a MechanicalTurk pipeline having us humans verify narrowed scopes the AI can't confidently inspect (yet. while also training on those results for next iteration)
As for the decision stack of what makes a good end product - turtles all the way up/down. The ProductOwnerAI role decides those things, and it will likely do so with the same deep skill that programming AIs are currently hitting our profession with. Optimizing enormous breadths of concerns and simulating results is AI's main specialty.