Grounded in Data

As many are aware, standalone Large Language Models (LLMs), such as ChatGPT, operate with a fixed knowledge cutoff. This means that their pre-training is a singular event rather than a continuous learning process. For example, ChatGPT's knowledge is current only up until April 2023. Consequently, it cannot provide information on events or facts that emerged after this date.

This is where RAG plays a critical role.

Understanding Semantic Similarity

The fundamental concept behind RAG is to retrieve data from a known source—information that the LLM likely has not encountered previously—and provide it to the model in real-time. This ensures that the model has access to updated and contextually relevant information for accurate responses.

But how does this retrieval process operate?

At its core, the architecture relies on the ability to fetch semantically meaningful data pertinent to the user's prompt. This involves three main components:

1. Embedding Model
2. Retriever (often a vector database)
3. Generator (the LLM)

To facilitate this retrieval, data must first be converted into an 'embedding form,' which represents text as numerical vectors. Importantly, similar concepts will correspond to similar vectors. For instance, the words 'dog' and 'cat' might be represented as [3, -1, 2] and [2.98, -1, 2.2], respectively. These embeddings are then stored in the vector database.

When a user submits a request like "find me similar results to a 'yellow cat'," the vector database executes a 'semantic query,' extracting the closest vectors to the user's input. These vectors represent underlying concepts, leading to the retrieval of relevant information.

The final step involves constructing the LLM prompt, which includes the user’s request, the extracted content, and a set of system instructions. For example, a typical instruction might be "be concise."

RAG, in essence, is designed to deliver relevant information in real-time, enhancing the LLM's response. Its efficacy stems from the LLMs' unique ability for in-context learning, allowing them to utilize previously unseen data for accurate predictions without requiring weight training.

For a deeper exploration of in-context learning and its implications, check out my detailed analysis.

Challenges and Limitations

The seemingly perfect nature of this process has its drawbacks. Understanding the key intuitions driving the evolution of advanced AI models can be daunting. However, staying updated with the rapidly changing AI landscape and preparing for the future is essential.

Stitching Without Refinement: A Thing of the Past

To visualize the current state of RAG systems, consider a pair of mismatched trousers: while they may function for some, many would find them unwearable due to their lack of cohesion. This analogy illustrates how traditional RAG systems combine three separate components that were independently pre-trained and were never intended to operate together.

In contrast, RAG 2.0 is designed to function as a cohesive unit from the outset. Unlike its predecessor, RAG 2.0 integrates all components—pretraining, fine-tuning, and Reinforcement Learning from Human Feedback (RLHF)—simultaneously, ensuring that the entire system learns in unison.

The results speak volumes. Despite potentially using a less sophisticated standalone model than GPT-4, RAG 2.0’s approach surpasses any combinations of GPT-4 with other retrieval systems.

The crux of the matter lies in the fact that RAG 1.0 relied on separately trained components, while RAG 2.0 ensures that all elements are interconnected from the very beginning.

The Real Question Remains

While RAG 2.0 appears poised to become the standard for enterprises reluctant to share sensitive data with LLM providers, the question remains: will RAG, in any form, still be necessary?

The Rise of Extensive Context Length

Modern models, such as Gemini 1.5 and Claude 3, now boast enormous context windows—up to a million tokens in released versions and even 10 million tokens in research settings. This means these models can process remarkably lengthy sequences of text in a single prompt.

For context, the combined word count of The Lord of the Rings and Harry Potter series is well under the maximum token limit of these models. Given this capability, one must ponder: is a knowledge retriever still necessary, or can we simply input all relevant information directly?

However, there are arguments against this approach. Longer sequences can complicate the retrieval of accurate context. On the flip side, RAG’s methodology allows for the selection of only the most relevant data, making it more efficient overall.

Google's findings demonstrate that accuracy remains high, even with extensive sequences. Their research showed nearly perfect accuracy at 10 million tokens in tasks designed to test the model's retrieval capabilities.

How is this achievable?

The underlying architecture of these models, particularly the attention mechanism, enables them to maintain a global context. This ensures that all tokens in the sequence can reference each other, allowing the model to capture long-range dependencies effectively.

Ultimately, the fate of RAG may hinge not on its accuracy but on economic factors.

Cost Considerations: A Business Perspective

The current limitations of Transformer models result in skyrocketing costs associated with processing lengthy sequences. The computational expenses grow quadratically with sequence length, and memory requirements can become unmanageable.

In fields such as genomics, where processing lengthy sequences is crucial, researchers have explored alternatives to traditional attention mechanisms, employing methods like the Hyena operator to reduce costs while preserving performance.

While the terminology may seem complex, the fundamental idea is simple: various strategies exist to help AI systems interpret language more effectively. The goal is to enhance efficiency while minimizing costs.

In conclusion, as the technology evolves, we may soon witness the processing of extensive sequences at a fraction of current costs, prompting skepticism about the necessity of RAG architectures. When that day arrives, the future of RAG will be uncertain.

If you found this discussion valuable, I invite you to explore similar insights on my LinkedIn or connect with me on X. I'm eager to engage with you further.