I think the infrastructure is more fungible than you think. Look at the old buildings pressed into service as data centers.
One of the largest data centers in Chicago started about 1920 as a printing site for catalogs and phone books (remember those?). It was purpose built -next to the railway for transportation, built to support presses weighing hundreds of tons, loads of power for the machines. If that business went away - who would want it?
Turns out the fiber optic cables a data center needs get laid next to rail lines, the floors could support massive batteries and server racks and the data center needed all the power and more.
Someday the H100 will be as obsolete as an 8080, and if its replacement needs less space and power we can turn the data center into an automated factory where robots use the extra power to weld, print or extrude the tools of the future.
Space, power and cooling will always be needed for something.
Malls depend on people. DataBank's new LGA3 in Orangeburg, NY, just had cabling crews alone of 60-80 people on two shifts working for a month or so, and that's just the low voltage cabling work, but when completed, there will be permanent onsite jobs for about eight techs, 2 per shift.
And yes, they tore down the entire Verizon datacenter building that was there and built new. But its infrastructure is more relevant than ever.
I've had the same thoughts about disruptive technologies bypassing the current AI tulip bulb mania. What will be interesting to watch is how the sunk cost fallacy plays out this time.
Very interesting piece. Even without a paradigm‐shift, ongoing gains in capability density and hardware efficiency could move a large share of inference to edge devices. This would strand large portions of hyperscalers’ GPU fleets. The crux here is the share of workloads whose compute requirements can be delivered cost effectively on-device.
I agree, but I also think the relative share of demand for frontier level capabilities could fall pretty dramatically over time. Eventually, most companions and agents and avatars may not need world breaking capabilities/thinking time to serve their purpose reliably. In that case, hyperscalers would be happy to offload the power demand to users’ devices for inference requests.
Delivered on device. How many of you have been in a room with a bunch of NVIDIA pods? Even ones just being configured not working a problem. Have you felt the dragon's breath of all the rejected heat which has NOT gone into the ethylene glycol? The rack, one rack, weighs around 3000lb and eats 100-120 KW. The NVL connections are copper, because they have to be, and the main busbar down the spine is inches thick, like something you read about in First Lensman or Galactic Patrol. The rack nodes are clustered on either side of the switch nodes because distances have to be kept that short. (For rack to rack and of course the trunks it is 400 or 800 Gbps fiber.)
Put that in your phone, or the brain of an FPV, or indeed an F-47. And yes I know that an iPhone from 2009 has more compute than everything Apollo Mission Control had at its disposal in 1969, including the black ladies. Still, that's forty years. If there is something coming along in less than 10 that will make it silly to have this buildout, indeed more than hints and whispers.
PS these pods are very generously spaced out. If you say reduced power and BTU demands by a factor of 10, they'll just cram 10x machines into the data halls.
As a Northern Virginia resident, this is a regular thought that crosses my mind. In the near term, the tax revenue is a benefit - but there are no fewer than 5 or 6 large Datacenter projects in the works at any point in time.
These points make sense if there are substantial limits to demand for AI. That is the question.
Arguments about changing compute technology may impact the products of specific companies, but almost any new technology can take advantage of high density power and cooling availability. I say that as an IC design engineer who has worked with both digital and analog computing for many years.
I think the infrastructure is more fungible than you think. Look at the old buildings pressed into service as data centers.
One of the largest data centers in Chicago started about 1920 as a printing site for catalogs and phone books (remember those?). It was purpose built -next to the railway for transportation, built to support presses weighing hundreds of tons, loads of power for the machines. If that business went away - who would want it?
Turns out the fiber optic cables a data center needs get laid next to rail lines, the floors could support massive batteries and server racks and the data center needed all the power and more.
Someday the H100 will be as obsolete as an 8080, and if its replacement needs less space and power we can turn the data center into an automated factory where robots use the extra power to weld, print or extrude the tools of the future.
Space, power and cooling will always be needed for something.
You might be right. On the other hand consider the phenomenon of dead malls that stay vacant for decades for want of alternative use cases.
Malls depend on people. DataBank's new LGA3 in Orangeburg, NY, just had cabling crews alone of 60-80 people on two shifts working for a month or so, and that's just the low voltage cabling work, but when completed, there will be permanent onsite jobs for about eight techs, 2 per shift.
And yes, they tore down the entire Verizon datacenter building that was there and built new. But its infrastructure is more relevant than ever.
I've had the same thoughts about disruptive technologies bypassing the current AI tulip bulb mania. What will be interesting to watch is how the sunk cost fallacy plays out this time.
Very interesting piece. Even without a paradigm‐shift, ongoing gains in capability density and hardware efficiency could move a large share of inference to edge devices. This would strand large portions of hyperscalers’ GPU fleets. The crux here is the share of workloads whose compute requirements can be delivered cost effectively on-device.
Thanks. I can see inference at the edge increasing, though not for frontier models.
I agree, but I also think the relative share of demand for frontier level capabilities could fall pretty dramatically over time. Eventually, most companions and agents and avatars may not need world breaking capabilities/thinking time to serve their purpose reliably. In that case, hyperscalers would be happy to offload the power demand to users’ devices for inference requests.
That's an interesting point. You might be right.
Delivered on device. How many of you have been in a room with a bunch of NVIDIA pods? Even ones just being configured not working a problem. Have you felt the dragon's breath of all the rejected heat which has NOT gone into the ethylene glycol? The rack, one rack, weighs around 3000lb and eats 100-120 KW. The NVL connections are copper, because they have to be, and the main busbar down the spine is inches thick, like something you read about in First Lensman or Galactic Patrol. The rack nodes are clustered on either side of the switch nodes because distances have to be kept that short. (For rack to rack and of course the trunks it is 400 or 800 Gbps fiber.)
Put that in your phone, or the brain of an FPV, or indeed an F-47. And yes I know that an iPhone from 2009 has more compute than everything Apollo Mission Control had at its disposal in 1969, including the black ladies. Still, that's forty years. If there is something coming along in less than 10 that will make it silly to have this buildout, indeed more than hints and whispers.
PS these pods are very generously spaced out. If you say reduced power and BTU demands by a factor of 10, they'll just cram 10x machines into the data halls.
As a Northern Virginia resident, this is a regular thought that crosses my mind. In the near term, the tax revenue is a benefit - but there are no fewer than 5 or 6 large Datacenter projects in the works at any point in time.
These points make sense if there are substantial limits to demand for AI. That is the question.
Arguments about changing compute technology may impact the products of specific companies, but almost any new technology can take advantage of high density power and cooling availability. I say that as an IC design engineer who has worked with both digital and analog computing for many years.