What Is a Context Window? 

A context window refers to the maximum amount of text an LLM (Large Language Model) can consider at one time when generating responses or processing input. This determines how much prior information the model can use to make decisions or generate coherent and relevant responses.

The size of a context window varies from model to model, and is rapidly growing in size as LLM development progresses. The context window of GPT-3, at the time it was first released, was 2,049 tokens (around 1,000 words). At the time of this writing, LLM context windows typically range from a few thousand to hundreds of thousands of tokens, or even millions for some models. 

The size of the context window is limited by computational constraints and model architecture, which must be carefully designed to allow the model to consider large amounts of data while formulating responses.

This is part of a series of articles about generative models

Why Are Context Windows Important for LLMs? 

Context windows are central in defining the capability of LLMs to understand and generate coherent text over sessions. They influence a model’s ability to maintain topic consistency, understand nuanced dialogue, and produce connected responses over extended interactions.

The context window also determines the breadth and types of information the model can consider for each decision point, impacting the accuracy and relevance of the model’s outputs. For example, models with larger context windows can accept not only textual prompts but also entire documents, images, videos, or even a combination of multiple files and formats. This opens up new use cases and allows users to interact with LLMs in new ways.

Examples of Context Windows of Popular LLMs

ChatGPT 4

GPT-4, developed by OpenAI, has a large context window size compared to its predecessors. The first two versions of GPT-4 had context windows of 8,192 and 32,768 tokens. GPT-3.5 was initially released with a context windows of 4,096 tokens.

In November 2023, OpenAI introduced GPT-4 Turbo and GPT-4 Turbo with Vision, featuring a larger context window of 128K tokens. In May, 2024, OpenAI released a more advanced model, GPT-4o, which supports multi-modal inputs, but its context window remained 128K tokens.

Claude 3.5

Claude 3.5 Sonnet, the latest release in the Claude model family, has set industry benchmarks in reasoning, knowledge, and coding proficiency. It has a 200K token context window, which surpasses many of its competitors. This large context window enables the model to handle extensive and complex inputs, making it suitable for tasks that require detailed contextual understanding and continuity over long interactions.

Google Gemini

Google’s Gemini 1.5 Pro was initially released with a context window of 128,000 tokens, compared to 32,000 tokens available in Gemini 1.0. Google offered a larger context window of 1 million tokens to a select group of developers and testers. 

In June, 2024, Google announced that Gemini 1.5 Pro would provide a context window of 2 million token to all developers. At the time of the announcement, this was the largest context window ever offered by a production language model. This allows the model to process large documents (containing approx. 1 million words), entire libraries of images, large codebases, and long-form video.

LLaMA 3

Meta’s LLaMA 3 is an open source LLM which can be easily extended and fine-tuned by users. The model is available in two versions, 8B and 70B, supporting different coding and problem-solving tasks. LLaMA 3 supports a context window of 8,000 tokens, which is double the capacity of its predecessor, LLama 2. 

Command R+

Command R+ is Cohere’s latest large language model, built for conversational interactions and tasks requiring extensive context. It is optimized for complex Retrieval-Augmented Generation (RAG) workflows and multi-step tool use, making it suitable for moving from proof of concept to production in various business applications.

Command R+ supports a context window of up to 128,000 tokens. This allows the model to process and understand large amounts of text within a single prompt, enhancing its ability to maintain coherence and relevance over extended interactions. The maximum output tokens for Command R+ are 4,000.

How Does the Context Window Size Affect the Performance of an LLM?

The size of a language model’s context window affects several aspects of performance.

Coherence

A larger window allows the model to reference more of the past conversation or document, enabling it to maintain themes and ideas over longer stretches of text. This results in outputs that are consistent, logical, and fluent.

Smaller windows, while computationally less demanding, might lead to repeated information or abrupt shifts in topic since the model has less prior text to anchor its responses. They also provide the user with less options to offer knowledge or content to ‘ground’ the model’s responses, which can improve accuracy and reduce hallucinations.

Memory

A larger context window extends the memory of an LLM, allowing it to remember and utilize more information from previous interactions. This capacity to remember more substantial parts of a conversation can improve the model’s ability to provide relevant and context-aware responses.

A smaller window restricts what the model can remember, possibly leading to responses that lack context or ignore important details from earlier in the conversation.

Computational Resources

Larger context windows require more memory and processing power, which can increase the cost and complexity of deploying these models. This is because the model needs to manage and process a more extensive sequence of tokens, leading to higher computational overhead.

Smaller context windows, while less resource-intensive, limit the amount of information the model can consider at any given time. This trade-off requires a balance between the desired performance and the available computational resources. 

Long-Form Content Generation

Larger context windows improve the ability of LLMs to generate long-form content, such as detailed articles, stories, or reports. By retaining more context, the model can produce coherent and comprehensive narratives that maintain consistency over extended text.

Models with smaller context windows might struggle with long-form content generation. When working with models that have a small context window, users might need to use strategies like dividing the content into smaller sections or using external tools to manage the overall structure and flow.

Strategies to Manage the Limited Context Windows in LLMs 

Here are some of the ways that organizations can manage the size of LLM context windows.

1. Sliding Window Technique

The sliding window technique helps mitigate the constraints of limited context windows. In this method, text is processed in overlapping segments. For example, if the context window can handle 1000 tokens, the first segment might cover tokens 1-1000, the second segment 501-1500, and so on. This overlap ensures that key information from the end of one segment is available at the beginning of the next, preserving continuity and context across the entire text.

This technique is particularly useful for tasks like document summarization or long-form text generation, where maintaining coherence and context is a priority. The sliding window method allows models to handle longer texts by piecing together these overlapping segments to create a cohesive output.

2. Prompt Engineering

Prompt engineering involves designing prompts that effectively utilize the context window to produce the best possible responses from LLMs. This involves strategically structuring the input to include essential details while omitting irrelevant information. Techniques include:

• Contextual summarization: Summarizing previous parts of a conversation or document to fit within the context window.
• Key information highlighting: Ensuring that critical facts or instructions are included within the prompt to guide the model’s response.
• Segmentation: Breaking down complex queries into simpler, sequential prompts that build on each other.

3. Dynamic Prompting

Dynamic prompting involves adjusting the input prompt in real time based on the evolving context of the interaction. This approach can be useful in interactive applications, such as chatbots or virtual assistants. Techniques for dynamic prompting include:

• Incremental context building: Continuously appending relevant information from previous interactions to the prompt.
• Adaptive prompts: Modifying the prompt based on the model’s previous responses to steer the conversation or task in the desired direction.
• Context trimming: Removing less relevant parts of the prompt as new, more relevant information becomes available.

4. Chunking and Summarizing

Chunking involves dividing large texts into smaller, manageable segments that fit within the context window. Each chunk is processed independently, and summaries of these chunks are generated to capture the main points. These summaries can then be used to provide context for processing subsequent chunks. This approach includes:

• Hierarchical summarization: Summarizing chunks at multiple levels (e.g., paragraphs into sections, sections into chapters) to maintain an overarching context.
• Contextual integration: Using summaries of previous chunks as part of the prompt for processing the next chunk, ensuring continuity.

5. External Memory

External memory systems extend the model’s context window by providing a repository of information that can be accessed as needed. This can be implemented through:

• Databases and knowledge graphs: Storing relevant facts, figures, and context externally, which the model can query to supplement its internal context.
• Retrieval-Augmented Generation (RAG): Combining retrieval systems with generation models to obtain relevant information from external sources in real time.
• Persistent storage: Maintaining a log of interactions or key information that the model can reference throughout the session.

Testing and Evaluating LLM and NLP Models with Kolena

We built Kolena to make robust and systematic ML testing easy and accessible for all organizations.  With Kolena, machine learning engineers and data scientists can uncover hidden behaviors of LLM and NLP models, easily identify gaps in the test data coverage, and truly learn where and why  a model is underperforming, all in minutes not weeks. Kolena’s AI / ML model testing and validation solution helps developers build safe, reliable, and fair systems by allowing companies to instantly stitch together razor-sharp test cases from their data sets, enabling them to scrutinize AI/ML models in the precise scenarios those models will be unleashed upon the real world. Kolena platform transforms the current nature of AI development from experimental into an engineering discipline that can be trusted and automated.

Among its many capabilities, Kolena also helps with feature importance evaluation, and allows auto-tagging features. It can also display the distribution of various features in your datasets.

Reach out to us to learn how the Kolena platform can help build a culture of AI quality for your team.