I see one challenge we don't seem to be addressing.
If Conway's Law (https://en.wikipedia.org/wiki/Conway%27s_law) holds true, we will design systems that represent communication structures in our organizations. They are nothing like a network of independent agents negotiating their "needs" in an interconnected network.
They are fairly rigid hierarchies still strongly rooted in bureaucracy principles developed some 120 years ago.
So if Melvin Conway was right, the future of these "agentic firms" may depart from the rosy view.
The implication goes further, as it touches the alignment issue. Again, we may make an assumption that the AI corporate entity (or whatever form it will have) will be aligned. But will it?
Do we see that much alignment in current organizations? The way they act in the broader context? The way they are organized internally? Again, if Conway's Law holds, why would we assume the product of this organizational mess we live in will produce perfectly (internally) aligned entities?
And I don't even touch the topic of the alignment *between* these entities. Why would we assume that all the issues plaguing our current business ecosystem (local optimizations, tragedy of the commons, etc.) would magically be solved by things we design?
"The compute bottleneck is real." - probably, it isn't. By 1960s measure, 2025 doesn't have a computer bottleneck. Everything we have is a 1960's supercomputer.
Similarly, by today's measure, 2035 won't have a bottleneck. Nvidia just announced an initiative to replace every desktop computer with a supercomputer. Literally and seriously, for roughly $3k per device, they think they can do it. This puts autonomous AI on every desk. Musk's dream of AI-driven robots in every house, might be as realistic as Henry Ford's vision of every house with an automobile. It might even be achievable within Musk's predicted time frame.
"Alignment is unsolved." - sure, I'll grant you this one, but that's a process problem that isn't obviously non-solveable.
"External control is illusion." - it has always been an illusion. CEOs and other upper management types like to think otherwise, but that doesn't mean they have EVER been correct.
"Firms may evolve into agents that no longer serve us." - depends on who "us" is and what "serve" means. Taken in the broad view, a lot of firms didn't "serve us". The East India Corporation (EIC) deliberately induced famine in India to protect its bottom line. The English government did the same thing to the Irish during the 19th century. United Health Care hasn't been serving the people who buy its insurance policies (it kills those people), but it DOES serve its shareholders, in much the same way the EIC did.
Similarly, most of the impediments to this vision boil down to energy. As long as there's enough power to move the transports and the lithograph machines, every other problem goes away. Read "With Folded Hands" and "The Midas Plague": two complementary visions on how this plays out.
In both visions:
* AI as a corporation
* AI as a sovereignty
I see one challenge we don't seem to be addressing.
If Conway's Law (https://en.wikipedia.org/wiki/Conway%27s_law) holds true, we will design systems that represent communication structures in our organizations. They are nothing like a network of independent agents negotiating their "needs" in an interconnected network.
They are fairly rigid hierarchies still strongly rooted in bureaucracy principles developed some 120 years ago.
So if Melvin Conway was right, the future of these "agentic firms" may depart from the rosy view.
The implication goes further, as it touches the alignment issue. Again, we may make an assumption that the AI corporate entity (or whatever form it will have) will be aligned. But will it?
Do we see that much alignment in current organizations? The way they act in the broader context? The way they are organized internally? Again, if Conway's Law holds, why would we assume the product of this organizational mess we live in will produce perfectly (internally) aligned entities?
And I don't even touch the topic of the alignment *between* these entities. Why would we assume that all the issues plaguing our current business ecosystem (local optimizations, tragedy of the commons, etc.) would magically be solved by things we design?
The replacement of management with so called copies is scary 😱
"The compute bottleneck is real." - probably, it isn't. By 1960s measure, 2025 doesn't have a computer bottleneck. Everything we have is a 1960's supercomputer.
Similarly, by today's measure, 2035 won't have a bottleneck. Nvidia just announced an initiative to replace every desktop computer with a supercomputer. Literally and seriously, for roughly $3k per device, they think they can do it. This puts autonomous AI on every desk. Musk's dream of AI-driven robots in every house, might be as realistic as Henry Ford's vision of every house with an automobile. It might even be achievable within Musk's predicted time frame.
"Alignment is unsolved." - sure, I'll grant you this one, but that's a process problem that isn't obviously non-solveable.
"External control is illusion." - it has always been an illusion. CEOs and other upper management types like to think otherwise, but that doesn't mean they have EVER been correct.
"Firms may evolve into agents that no longer serve us." - depends on who "us" is and what "serve" means. Taken in the broad view, a lot of firms didn't "serve us". The East India Corporation (EIC) deliberately induced famine in India to protect its bottom line. The English government did the same thing to the Irish during the 19th century. United Health Care hasn't been serving the people who buy its insurance policies (it kills those people), but it DOES serve its shareholders, in much the same way the EIC did.
Similarly, most of the impediments to this vision boil down to energy. As long as there's enough power to move the transports and the lithograph machines, every other problem goes away. Read "With Folded Hands" and "The Midas Plague": two complementary visions on how this plays out.