When it comes to large language models, should you build or buy?

Last summer could only be described as an “AI summer,” especially with large language models making an explosive entrance. We saw huge neural networks trained on a massive corpora of data that can accomplish exceedingly impressive tasks, none more famous than OpenAI’s GPT-3 and its newer, hyped offspring, ChatGPT.

Companies of all shapes and sizes across industries are rushing to figure out how to incorporate and extract value from this new technology. But OpenAI’s business model has been no less transformative than its contributions to natural language processing. Unlike almost every previous release of a flagship model, this one does not come with open-source pretrained weights — that is, machine learning teams cannot simply download the models and fine-tune them for their own use cases.

Instead, they must either pay to use them as-is, or pay to fine-tune the models and then pay four times the as-is usage rate to employ it. Of course, companies can still choose other peer open-sourced models.

This has given rise to an age-old corporate — but entirely new to ML — question: Would it be better to buy or build this technology?

It’s important to note that there is no one-size-fits-all answer to this question; I’m not trying to provide a catch-all answer. I mean to highlight pros and cons of both routes and offer a framework that might help companies evaluate what works for them while also providing some middle paths that attempt to include components of both worlds.

Buying: Fast, but with clear pitfalls

Let’s start with buying. There are a whole host of model-as-a-service providers that offer custom models as APIs, charging per request. This approach is fast, reliable and requires little to no upfront capital expenditure. Effectively, this approach de-risks machine learning projects, especially for companies entering the domain, and requires limited in-house expertise beyond software engineers.

Projects can be kicked off without requiring experienced machine learning personnel, and the model outcomes can be reasonably predictable, given that the ML component is being purchased with a set of guarantees around the output.

Unfortunately, this approach comes with very clear pitfalls, primary among which is limited product defensibility. If you’re buying a model anyone can purchase and integrate it into your systems, it’s not too far-fetched to assume your competitors can achieve product parity just as quickly and reliably. That will be true unless you can create an upstream moat through non-replicable data-gathering techniques or a downstream moat through integrations.

What’s more, for high-throughput solutions, this approach can prove exceedingly expensive at scale. For context, OpenAI’s DaVinci costs $0.02 per thousand tokens. Conservatively assuming 250 tokens per request and similar-sized responses, you’re paying $0.01 per request. For a product with 100,000 requests per day, you’d pay more than $300,000 a year. Obviously, text-heavy applications (attempting to generate an article or engage in chat) would lead to even higher costs.

You must also account for the limited flexibility tied to this approach: You either use models as-is or pay significantly more to fine-tune them. It is worth remembering that the latter approach would involve an unspoken “lock-in” period with the provider, as fine-tuned models will be held in their digital custody, not yours.

Building: Flexible and defensible, but expensive and risky

On the other hand, building your own tech allows you to circumvent some of these challenges.

In most cases, “building” refers to leveraging and fine-tuning open sourced backbones not building from scratch (although that also has its place). This approach grants you exponentially greater flexibility for everything from modifying model architectures to reducing serving latencies through distilling and quantization.

It’s worth remembering that while purchased models might be impressive at many tasks, models trained in-house may well achieve sizable performance improvements on a specific task or domain. At scale, these models are much cheaper to deploy and can lead to the development of significantly defensible products that can take competitors much longer to replicate.

The most prominent example of this is the TikTok recommendation algorithm. Despite much of its details being publicly available in various research papers, even massive ML teams at its competitors are yet to replicate and deploy a similarly effective system.

Of course, there are no free lunches: Developing, deploying and maintaining elaborate machine learning systems in-house requires data engineering, machine learning and DevOps expertise, all of which are scarce and highly sought-after. Obviously, that requires high upfront investment.

The success of machine learning projects is also less predictable when you’re building them in-house, and some estimates put the likelihood of success at around the 20% mark. This may prolong the time-to-market.

All in all, while building looks extremely attractive in the long run, it requires leadership with a strong appetite for risk over an extended time period as well as deep coffers to back said appetite.

The middle road

That said, there are middle ground approaches that attempt to balance these positives and negatives. The first and most oft-discussed is prompt engineering.

This approach starts with buying then building a custom input template that serves to replace fine-tuning in some sense. It aims to guide the off-the-shelf model with clear examples or instructions, creating a middling level of defensibility in the form of custom prompts while retaining the benefits of buying.

Another way is to seek open source alternative backbones of closely equivalent quality and build atop them. This reduces upfront costs and lack of output predictability to some extent while retaining the flexibility offered by building. For example, GPT-J and GPT-Neo are two open source alternatives to GPT-3.

A slightly more intricate and newer approach is closed source approximation. This involves attempting to train an in-house model that aims to minimize the difference between GPT-3 and its own output, either terminally or at an earlier embedding stage. This will reduce time-to-market by leveraging GPT-3 in the short term and then transitioning to in-house systems as their quality improves in the long term to enable cost optimization and defensibility.

Still confused about which way to go? Here’s a three-question quiz:

Are you currently an AI business?

If yes, you’ll need to build to maintain defensibility.

If you’re not, buy for now and prompt engineering to tailor the model to your use cases.

If you want to be an AI business, work toward that over time: store data cleanly, start building an ML team and identify monetizable use cases.

Is your use case addressed by existing pre-trained models?

Can you simply afford to buy without putting in much additional work? If so, you should probably shell out the cash if time-to-market is a factor.

Building is not fast, easy or cheap. This is especially true if your use case is non-monetizable or you need a model for internal use.

Do you have unpredictable or extremely high request latency?

If yes, buying might not be economically feasible, especially in a consumer setting. That said, be realistic — quantify your request latency and buying costs to whatever extent possible. Building can be deceptively expensive, especially because you’ll need to hire ML engineers, buy tooling and pay for hosting.

Hopefully, this helps you kick off your journey!