Are you cooking with open source AI?
If you want to innovate you have to use open source AI
This post provides an overview of open source and closed source large language models (LLMs)1. The choice of open source vs closed source software is a tendentious and contentious one amongst technologists; this post avoids the controversy in favor of a more neutral and dispassionate discussion of these concepts as they pertain to LLMs.
If you want a flavor of the kind of emotion which this debate elicits, here’s an article about the time that Steve Ballmer called open source software a cancer.
The introduction to this post provides a mental framework for thinking about open source vs closed source, and is intended for those with no background knowledge.
Introduction: Cooking as metaphor for software
In a previous life I worked as a financial analyst at Bloomberg. If you’ve never been to a Bloomberg office, let me set the stage for you: every floor is open plan, and every floor has a large section near the stairs where you can grab snacks. If you believe Michael Bloomberg’s autobiography2, this floor design was chosen to encourage interactions amongst various employees who would otherwise never have reason to speak to each other. Innovation, Bloomberg3 alleged, arose from such spontaneous reactions.
Soon after I started working there, I found myself between meetings, munching on snacks in the snack area. (It was a great place to work if you want to gain weight.) Anyway, I overheard two people, who I believe worked in sales, arguing about open source software. Maybe they heard the term when talking to one of the field technicians who installed Bloomberg’s equipment at client sites. Maybe one of their contacts at an investment bank mentioned that open source was the hot new thing. They just didn’t understand the term4. So I interjected and said something along the lines of “It’s like cooking. If you’ve got the recipe it’s open source. If you don’t have the recipe it’s closed source.”5
Let’s continue this metaphor for a bit. Have you ever cooked a meal from scratch, without a recipe? The meal you cooked was closed source: no one other than you can modify it to their liking because no one but you knows how it’s made. On the other hand if you’ve ever used a recipe from a cookbook to cook a meal, you’ve done open source cooking: anyone can modify the recipe to their liking. Customization is king with open source. You can have any color you like as long as it is black6 is the way of closed source.
Another way to think about it: your grandmother’s famous lasagna, for which she refuses to divulge the secrets of her culinary art? That’s closed source. Sure, you can put a gun to grandma’s head and get her to cough up the details but who wants to do that? But if grandma writes her recipe down? That’s open source. You can modify Grandma’s Famous Lasagna to your heart’s content, and you don’t even need a gun!
I relate all of this gustatory detail because it’s an easy and concrete way to understand the difference between open source software and closed source software. Open source software is software that anyone can modify to fit their needs.7 Closed source software is proprietary software whose code can’t be modified by end users.8 I swear that all of this has something to do with AI, so read on.
Is open source AI better than closed source AI?
The most well-known closed source LLM is OpenAI’s series of GPTs: GPT 3.5, GPT 4, maybe GPT 4.5. In exchange for buying a subscription to closed source LLMs, you get the reliability of an enterprise-grade partner for your AI computing needs.
Closed source LLMs are controlled by organizations like OpenAI/Microsoft, Google, etc. And you might think that these large companies’ built-in distribution advantage means that no competing alternatives exist. But just as in the world of traditional operating systems, there is a thiriving community of open source alternatives to these centrally-managed and -owned closed source LLMs. Consider the following:
In the past few years, it seemed that wealthy tech companies would be able to monopolize the growing market for large language models (LLM). And recent earnings calls from big tech companies suggested they are in control. Microsoft’s announcements, in particular, show that the company has created a billion-dollar business from its AI services, including through Azure OpenAI Services and the workloads OpenAI runs on its cloud infrastructure.
However, a recently leaked internal document from Google indicates that the market share of big tech is not as secure as it seems thanks to advances in open-source LLMs. In short, the document says “We have no moat, and neither does OpenAI.” The dynamics of the market are gradually shifting from “bigger is better” to “cheaper is better,” “more efficient is better,” and “customizable is better.” And while there will always be a market for cloud-based LLM and generative AI products, customers now have open-source options to explore as well.
The leaked Google document provides a wealth of detail about why open source LLMs pose such a significant threat to the large tech companies’ AI initiatives. Further, it explains how innovation flourishes in the world of open source LLMs, in a way that it can’t with closed source LLMs. Closed source LLMs can’t innovate as quickly as can open source simply because closed source LLMs are managed to adhere to the dictates of the organization which controls them. OpenAI, Microsoft, Google, and other large tech companies all have reputations to protect. Consider the following from the leaked Google doc:
LLMs on a Phone: People are running foundation models on a Pixel 6 at 5 tokens / sec.
Scalable Personal AI: You can finetune a personalized AI on your laptop in an evening.
Responsible Release: This one isn’t “solved” so much as “obviated”. There are entire websites full of art models with no restrictions whatsoever, and text is not far behind.
Multimodality: The current multimodal ScienceQA SOTA was trained in an hour.
Open source large language models, in other words, allow you to do AI computation at the edge. I’ve argued before that imbuing the edge of computer networks with intelligence is critical. A world in which every mobile device, laptop, desktop, and industrial sensor, to say nothing of autonomous vehicle, has its own internal intelligence capabilities, will allow for a much different world.
If you want autonomous drones delivering medicine to remote locations, or if you want better rescue services, or better monitoring of remote industrial sites, or if you just want to be able to run a large language model on your personal computing device, all without having to worry about a reliable connection to a centralized server, well, you can see the benefit in computing at the edge. (And, to be fair, Google clearly recognizes this trend, with its Gemini Nano model.)
But the point here isn’t that Google is enabling AI at the edge. The point is that the open source AI community got there first, and it’s rapidly advancing the state of the art. Here’s how Ethan Mollick explains the rise of open source LLMs:
The implications of [open source LLMs] are pretty big:
The AI genie is out of the bottle. To the extent that LLMs were exclusively in the hands of a few large tech companies, that is no longer true. There is no longer a policy that can effectively ban AI or one that can broadly restrict how AIs can be used or what they can be used for. And, since anyone can modify these systems, to a large extent AI development is also now much more democratized, for better or worse. For example, I would expect to see a lot more targeted spam messages coming your way soon, given the evidence that GPT-3.5 level models works well for sending deeply personalized fake messages that people want to click on.
AI will be everywhere. I can already run the fairly complex Mistral model on my recent model iPhone, but, just a week ago, Microsoft released a model that [is] just as capable, but only a third as taxing on my poor phone. Between increasing hardware speeds (thanks, Moore’s Law) and increasingly optimized models, it is going to be trivial to run decent LLMs on almost everything. Someone already has an AI running on their watch.
One important implication which flows from Mollick’s observations is that mobile and other computing devices, such as industrial sensors, and so on, will be imbued with intelligence. The edge, in other words, becomes intelligent.
So what are the benefits and drawbacks of open source and closed source software? How do I know which kind of AI I should be rooting for? Well, the somewhat unappealing answer is…it depends. What do you want to do with the AI? The most powerful AI systems, such as OpenAI’s GPT4 and Google’s Gemini are, of course, closed source. They’re proprietary software which you can’t customize. But not all AI applications require the most advanced LLMs. Maybe a less powerful LLM is sufficient for whatever you need to do.
Consider this table:
It’s a mistake to assume that the large tech companies’ hundreds of millions of customers provide an unassailable moat. This is not to say that closed source LLMs will disappear. Rather it is to say that closed source LLMs will compete with open source LLMs for users. Further, because open source LLMs are not constrained by the mandates of their parent organization, they will innovate at a faster pace than closed source LLMs.
There’s a lot of debate about whether “open source” and “closed source” need hyphens. I’ve chosen to ignore the debate and not use hyphens in this post.
I first read his autobiography while working as a campaign staffer for his 2001 NYC mayoral campaign. I recall being interested in the book mainly because I wanted to learn more about the guy who signed my paychecks. I recall it being a worthwhile read.
This was both the company’s official stance and that of its founder, Michael Bloomberg. I joined the company when it had thousands of employees spread across the globe. I think the idea about chance encounters with other employees leading to unexpected innovations may have made sense when Bloomberg was a much smaller, more entrepreneurial company. But by the time I joined, it was a multi-billion dollar behemoth serving tens of thousands of customers across the globe. There was no innovation to be had by a chance encounter between, say, a sales guy, and a marketing executive. This does not mean that the design was a bad choice, though! There is something to be said for being able to meet people who work at the same company as you, who you would otherwise never meet. I was just skeptical that these chance encounters fomented the innovation that the company and its founder claimed they did.
I always thought it somewhat amusing that these people didn’t understand what closed source software was, in spite of working for Bloomberg, which of course creates closed source software.
I know that I am not the only person to have come up with this metaphor. I don’t know who invented it, but I am certain that it pre-dates my use of it. I don’t recall where I first heard it, or whether I invented it out of whole cloth on the spur of the moment.
An adaptation of Henry Ford’s famous quote.
Obviously the truth of this statement is contingent on the person interested in modifying the software being able to understand computer code, so the comparison with cooking is not wholly accurate.
Yes, there will always be exceptions to this—see iPhone jailbreaking, for example. But for the most part, closed source software is not manipulable by, or accessible to, the end user. Again, the comparison with cooking is not perfectly accurate, but it is sufficiently accurate for our purposes here.