Why Not Build an LLM?

Retrieval-Augmented Generation (RAG) achieves better results than training a proprietary large language model due to lower initial costs, reduced maintenance requirements, access to diverse data sources, optimized utilization of human resources, and the integration of extensive pre-existing knowledge.

Companies are motivated to create proprietary large language models to achieve tailored solutions that meet specific business needs, enhance competitive differentiation, maintain control over data privacy and security, and drive innovation aligned with their strategic objectives.

BloombergGPT is a specialized transformer-based language model designed by Bloomberg to enhance NLP capabilities within the financial domain. It incorporates a large number of layers and attention heads, tailored to process and analyze vast amounts of financial data. It excels at financial sentiment analysis, named entity recognition, and market prediction.

The costs of training frontier AI models have grown dramatically, but there is limited public data on the magnitude and growth of these expenses. Epoch AI's analysis estimates training costs using three approaches accounting for hardware, energy, cloud rental, and staff expenses.

Chinchilla is a family of large language models developed by DeepMind and introduced in March 2022, named as a successor to the Gopher model family, designed to further explore the scaling laws of large language models.

Researchers trained models ranging from 70 million to over 16 billion parameters on 5 to 500 billion tokens, and found that models trained following the optimal scaling law significantly outperformed larger, undertrained models such as GPT-3 and Megatron-Turing NLG.

Detailed in Hoffmann et al.'s paper of the same name, the Chinchilla Scaling Law addresses the optimal balance between model size, number of training tokens, and compute power to maximize performance within a given compute budget.

The Pile is an 825 GiB English text corpus targeted at training large-scale language models, constructed from 22 diverse, high-quality subsets derived from academic and professional sources. Models trained on it improve significantly over both raw Common Crawl and CC-100 across all its components, while also improving downstream evaluation performance.

Benchmarks like SuperGLUE, EleutherAI LM Harness, and BIG-bench differ in their selection of scenarios and metrics, each with varying priorities and transparency. HELM aims for holistic evaluation by explicitly detailing the relationship between intended and actual evaluations through a top-down taxonomy of scenarios and metrics.

Salesforce developed the first LLM benchmark for CRM to assess the efficacy of generative AI models for business applications — evaluating LLMs for sales and service use cases across accuracy, cost, speed, and trust & safety, based on real CRM data and expert evaluations. What sets it apart: human evaluations by both Salesforce employees and actual external customers, on real-world datasets from Salesforce and customer operations.

Word2Vec, developed by Mikolov et al. at Google in 2013, is a highly influential method for learning word embeddings from large text corpora — transforming words into continuous vector representations that capture semantic relationships and syntactic properties. Words are represented as dense vectors in a high-dimensional space, typically 50 to 300 dimensions, enabling mathematical operations like dot products and vector addition.

Word embeddings are a fundamental component of LLMs, mapping words or phrases from a vocabulary to vectors of continuous numbers. These vectors capture semantic meanings and relationships, enabling models to perform NLP tasks more effectively. Words with similar meanings sit closer together in the vector space — "king" and "queen" are closer to each other than "king" and "apple."

Seq2Seq networks, introduced by Sutskever, Vinyals, and Le in 2014, transform sequences from one domain into sequences in another — making them effective for machine translation, text summarization, and dialogue systems. They typically consist of an encoder and a decoder, often implemented with RNNs, LSTMs, or more recently, transformers.

Transformer architectures surpass LSTMs and RNNs on three fronts: parallelization (self-attention processes entire sequences simultaneously, rather than one token at a time), long-range dependencies (self-attention attends to all words in a sequence at once, avoiding the vanishing-gradient problems that limit RNNs), and attention mechanisms (multi-head self-attention is inherently integrated into the architecture, not bolted on).

Vaswani et al.'s 2017 paper introduces the Transformer, a neural network architecture that relies entirely on attention mechanisms, dispensing with recurrence and convolutions entirely — a paradigm shift demonstrating how self-attention can significantly enhance the efficiency and performance of sequence transduction tasks like machine translation.

BERT, developed by Google AI in 2018, uses a transformer-based encoder-only structure, reading text bidirectionally to capture context from both sides of a word. It leverages self-attention across 12 layers (BERT Base) or 24 layers (BERT Large), each with multiple attention heads, feed-forward networks, and residual connections with layer normalization. Its open-source release significantly influenced RoBERTa, ALBERT, and DistilBERT.

CTRL, developed by Salesforce in 2019, is a 1.63-billion-parameter conditional transformer language model trained on a dataset containing control codes that govern style, content, and task-specific behavior — enabling fine-grained control over the model's output. Control codes were derived from structure that naturally co-occurs with raw text, preserving the advantages of unsupervised learning while adding explicit control, and enabling model-based source attribution over training data.

RAG is a hybrid NLP approach that combines the strengths of retrieval-based methods and generative models to enhance the quality and relevance of generated text.

Vector databases are a specialized class of databases designed to handle and query high-dimensional vector representations of data — essential for image recognition, NLP, and recommendation systems, where retrieval is based on vector similarity rather than exact match.

LangChain streamlines and enhances the operations of large language models, integrating components and tools for efficient training, fine-tuning, and deployment across diverse NLP tasks. It provides robust support for modular workflows, letting practitioners build and customize pipelines for text generation, summarization, question answering, and more.