Why is this?: “Interrupt it because ERCOT day-ahead pricing jumps from $50 to $800/MWh, and you risk having to restart the training from scratch. Unlike bitcoin mining, you can’t resume training from where you left off. “. Is there really no way to save state?
And are any of these AI data centers pursuing nuclear? I remember there being a lot of buzz about that a few months ago but lost track of the story.
Thanks for the comment and questions. I think there is a way to save state, but that has to be baked into the process from the beginning of the training run. I could have phrased this more clearly. As for nuclear power, yes, the hyperscalers have been interested in this. I think that Microsoft signed a partnership with the Three Mile Island nuclear power plant in New York, for example.
Thanks for getting back to me! I loved this article.
I figure this isn’t a question that can be answered without inside knowledge, but I wonder how common it is for these AI companies to implement save-state functionality. I would think it would slowly become more common in locations that are exposed to frequent grid interruptions. Then again teams ignore all kinds of risks when racing toward big goals so 🤷♀️.
Wow!
Why is this?: “Interrupt it because ERCOT day-ahead pricing jumps from $50 to $800/MWh, and you risk having to restart the training from scratch. Unlike bitcoin mining, you can’t resume training from where you left off. “. Is there really no way to save state?
And are any of these AI data centers pursuing nuclear? I remember there being a lot of buzz about that a few months ago but lost track of the story.
Thanks for the comment and questions. I think there is a way to save state, but that has to be baked into the process from the beginning of the training run. I could have phrased this more clearly. As for nuclear power, yes, the hyperscalers have been interested in this. I think that Microsoft signed a partnership with the Three Mile Island nuclear power plant in New York, for example.
Thanks for getting back to me! I loved this article.
I figure this isn’t a question that can be answered without inside knowledge, but I wonder how common it is for these AI companies to implement save-state functionality. I would think it would slowly become more common in locations that are exposed to frequent grid interruptions. Then again teams ignore all kinds of risks when racing toward big goals so 🤷♀️.
Very interesting article !
Incidentally I wrote this on the related topic of AI’s Energy Appetite.
https://open.substack.com/pub/pramodhmallipatna/p/ais-energy-appetite-the-hidden-costs