How to Build a LLM from Scratch

Open AI’s release of ChatGPT in late 2002 ushered in the ‘Era of Generative Artificial Intelligence”. ChatGPT introducing the amazing powers of AI to the public. Now we have an environment where many Businesses enterprises and other Organizations are seeking to adopt Gen-AI and train their their own Large Language Models in order to remain competitive.

One of the most notable examples of this trend is Bloomberg GPT, which is a Large Language Model that was specifically built by Bloomberg to handle tasks in the Finance domain.

Even though building a Large Language Model from scratch is often not necessary for a vast majority of use cases since fine-tuning an existing LLM is relatively quick and inexpensive, it is still valuable to understand what it takes to build one since all use cases are not addressed by existing LLM’s. Moreover, some organization may have strategies to for competitive and proprietary assets.

In this section, we’ll discuss the key aspects, considerations and pro’s and con’s for building a Large Language Model from scratch.

Here are the steps we will cover:

1. Estimate of Cost (Rent Compute Hardware vs. Purchase Compute Hardware)

2. Data Curation

3. Model Architecture

4. Training the Model at Scale

5. Evaluating the Model