Token-Efficient Prompt Design: Context Compression and Information Density

One support assistant we audited had launched on a lean 900-token system prompt, built for one product line. Eight months and four product owners later, the prompt had swollen to 4,600 tokens. API costs had tripled while ticket volume had barely moved, and nobody could say which half of it still mattered. 

The short answer to what AI tokens are: They are the sub-word units a tokenizer produces, where one token can be a whole word or phrase fragment, and providers bill every input and output token. Drawing on 150+ client engagements across 30+ industries, we show Data and AI Practitioners how to cut tokens without losing output quality.

How the Context Window Shapes Prompt Design

The context window is the shared token budget every part of a request draws from. In a large language model built on transformer architecture, that budget covers:

• The system prompt and tool definitions.
• Retrieved context from retrieval-augmented generation.
• Conversation history.
• The reserved output.

Current limits look generous. The GPT-4o context window holds 128K tokens, the Claude context window holds 200K on standard tiers, the Gemini API offers larger windows on selected models, and Meta advertises a 10 million context window for Llama 4 Scout. That generosity is the trap.

Longer windows cost latency as well as money. The attention mechanism in transformer models compares tokens with each other, so compute grows roughly quadratically with input length, and AI computing lands on your invoice while time to first token climbs.

A bigger window does not mean better answers either. Stanford research published as Lost in the Middle found that models recall the start and end of a long context far better than the middle. Bigger context windows didn't solve the token problem; they just made it more expensive to ignore.

Where Token Efficiency Happens: Model, Tokenizer, and Prompt Layers

Token efficiency lives in three layers. A team fully controls only one of them:

• Model layer: Sparse attention, KV caching, and distillation sit with the provider.
• Tokenizer layer: A BPE tokenizer counts formatting, the part most answers to what are tokens in AI leave out. Whitespace, verbose JSON keys, and decorative separators all cost tokens.
• Prompt layer: What you send and write. Nobody else has to approve changes.

Two supporting layers sit around these: The orchestration layer that sequences calls, and the context management layer that decides what each call receives. Map your prompt flow once, and you will see which layer owns each token.

Is the engineering time worth it? Run the maths against current API pricing:

• 2 million requests a month at 3,000 input tokens is 6 billion tokens.
• A 30% trim at $3 per million saves about $5,400 a month from one prompt.

Our AI services team usually starts with routing, because its savings stack with every later cut. In AI integration consulting work, it is often live within a fortnight.

AI Prompt Engineering Techniques for Higher Information Density

Information density is useful signal per token. Manual restructuring typically removes 20 to 40% of system prompt tokens with no quality regression. That is LLM token optimisation at its least glamorous and most profitable.

Cutting Redundant Instructions

System prompt bloat usually starts with one rule written three ways by three people. Merge them:

Then audit for stale rules, which linger for years because deleting them feels risky. Effort: Two to four hours per prompt, including an eval run. Look for:

• Instructions for retired features.
• Old pricing or policy text.
• Contradictions between contributors.

# Delete: Feature retired last year
If the user asks about Legacy Reports, send them to the Reports v1 tab.
 
# Delete: Contradicts the current 30-day refund policy
Refunds are only available within 14 days of purchase.

Using Structured Formats Instead of Long Prose

Instructions are easier to follow, and tokens are saved by structure. The Anthropic prompt engineering guide recommends clear sections, which is why Claude prompt engineering relies on XML-style tags. Provider docs also change faster than any prompt engineering guide 2026 PDF, so recheck them when models change.

• Test JSON against compact YAML or Markdown in the tokenizer; long keys add up.
• Apply hierarchical context organisation: Global rules first, task rules next, examples last.

# Before: Prose (≈65 tokens)
When a customer asks about a refund, you should first check whether the order
was placed within the last 30 days. If it was, you can go ahead and offer a
full refund. If it was not, politely explain that the order is outside the
refund window and offer store credit instead.
 
# After: Numbered rules (≈25 tokens)
### Refunds
1. Order 30 days old or less: Offer a full refund.
2. Order older than 30 days: Explain the window, offer store credit.
 
# Verbose JSON (≈30 tokens)
{"customer_support_response_guidelines": {"maximum_response_length_in_words": 120, "preferred_tone_of_voice": "friendly"}}
 
# Compact YAML (≈12 tokens)
support:
  max_words: 120
  tone: friendly

Choosing Fewer, Stronger Few-Shot Examples

Examples are often the most expensive block in a prompt, and one sharp example beats three similar ones. Better still, stop sending the whole library and select examples per query (Effort: Roughly one sprint, including evals):

• Run your example library through an embedding function.
• Use semantic search to pull the two or three closest matches.

# Send the 3 closest examples instead of all 25
query_vec = embed(user_query)                        # same embedding function used to index examples
examples = example_index.search(query_vec, top_k=3)  # semantic search over the example library
prompt = SYSTEM_PROMPT + format_examples(examples) + user_query

Shrinking Output With Structured Outputs

Output costs several times more than input: $15 against $3 per million on Claude Sonnet-class pricing. Most guides ignore it.

• Structured outputs with schema constraints stop essays when you need fields.
• Schema-constrained generation also blocks malformed results, such as an invalid SQL command.
• Split long text generation into structured subtasks, each with its own max_tokens cap.

{
  "type": "object",
  "properties": {
    "intent":   {"type": "string", "enum": ["refund", "delivery", "other"]},
    "priority": {"type": "integer", "minimum": 1, "maximum": 3},
    "reply":    {"type": "string", "maxLength": 400}
  },
  "required": ["intent", "priority", "reply"]
}
# Free-form reply (≈300 tokens)
"Thanks so much for reaching out! I completely understand how frustrating..."
 
# Schema-constrained reply (≈60 tokens)
{"intent": "refund", "priority": 2, "reply": "Your refund for order 4821 is approved and will arrive in 5 days."}

Volume multiplies every saving. The agentic underwriting system that cut manual underwriting by 65% for a US digital bank runs thousands of calls a day.

This is where most teams stop: They clean up once, and bloat returns within two quarters. Treat prompts like code instead, as our engineers do on every production LLM engagement:

• Token counts in every pull request.
• Evaluation workflows before merge.
• A named owner per prompt.

Context Compression Methods for Production Prompts

Context compression targets runtime content nobody edits by hand: History, retrieved chunks, and tool outputs. Prompt compression here offers the biggest savings and the biggest quality risk.

Summarising Conversation History

Session compaction, a form of context compaction, keeps long chats inside budget. The pattern is simple:

• Keep the last four to six turns verbatim.
• Fold older turns into a rolling summary.
• Pin exact values (file paths, error strings, order IDs, amounts) in a facts block.

Here is the problem: Summarisation is abstractive compression. It rewrites content, and a long support conversation can lose the one order number that mattered.

Prompt Caching for Repeated Context

Prompt caching rewards a stable prefix: Static content first, per-request content last. Each major provider prices it differently:

• Anthropic: Writes cost 1.25x base input, reads 0.1x, with a five-minute TTL extendable to one hour (Anthropic docs).
• OpenAI: Automatic caching on longer prompts; the discount varies by model.
• Gemini API: Explicit context caching for large shared prefixes.

Caching does not shrink context; it makes repetition cheaper. Production needs a stable prefix, a cache marker, and usage logging:

import anthropic
 
client = anthropic.Anthropic()
 
response = client.messages.create(
    model="claude-sonnet-5-5",
    max_tokens=600,  # cap output: output tokens cost more than input
    system=[{
        "type": "text",
        "text": STATIC_SYSTEM_PROMPT + REFERENCE_DOCS,  # stable prefix, never compressed
        "cache_control": {"type": "ephemeral"},        # 5-minute TTL by default
    }],
    messages=[{"role": "user", "content": filtered_chunks + user_query}],  # varies per call
)
 
u = response.usage
log_metrics(cache_write=u.cache_creation_input_tokens,
            cache_read=u.cache_read_input_tokens,
            uncached=u.input_tokens, output=u.output_tokens)

Those fields expose your cache hit rates. If cache reads stay at zero, something in the prefix changes between calls.

Filtering Retrieved Chunks in a RAG Pipeline

The RAG pipeline is usually the highest-yield target, because retrieved context is reliably redundant. Retrieval filtering takes three moves:

• Rerank results from your knowledge base.
• Apply a relevance threshold.
• Lower top-k: Five strong chunks beat twenty loose ones.

Comparative studies found extractive compression held or improved multi-document QA accuracy where abstractive rewriting hurt it. When relationships between documents matter, graph traversal can replace a broad top-k pull. Retrieval quality starts upstream, with your vector database choice.

Automated Prompt Compression Tools

Automated tools belong last in the sequence. Their research numbers still look impressive:

• LLMLingua: Up to 20x compression in research benchmarks.
• LLMLingua-2: Faster and task-agnostic.
• Selective Context (Li et al., 2023): Roughly 50% reduction with minimal loss.

All three rely on extractive pruning at the token level, which breaks code and exact identifiers. Savings below are indicative, so measure your own workload.

Which method fits your workload?

1. The same long prefix on every call? → Prompt caching first.
2. RAG answers padded with loose chunks? → Reranking and a lower top-k.
3. Long agent sessions breaking the budget? → Rolling summaries with pinned facts.

Our Take: Most teams reach for LLMLingua first because it feels like engineering, and that is the wrong order. Cache the stable prefix, filter retrieval, and trim history before adding a compressor. Those three steps deliver most of the LLM cost reduction with almost no quality risk.

Think your workload is too specialised for this? It rarely is. The same leaks appear in a fintech underwriting agent, a hospital scheduling assistant, AI in fashion styling tools, and AI interior design apps.

For a 400-location US retail chain, our agentic AI auto-resolved 64% of IT tickets. At that volume, every hundred tokens trimmed repeats across millions of calls.

Context Engineering vs Prompt Engineering

Ask what prompt engineering in AI is, and the short answer is writing the instructions. Context engineering, often called context management, decides which tokens enter the window at each step.

The context engineering vs prompt engineering distinction matters most in agent systems, agentic RAG, and conversational AI. In those products, the system assembles most tokens, not a person:

• A customer service agent pulls order history, policy, and tone rules each turn.
• A legal analysis agent should load relevant clauses, not whole contracts.
• A multi-turn coding session piles up tool output; a ten-step agent can carry 60,000 tokens of it.

Four patterns keep agent context optimisation under control. Each one limits what enters the window by default:

• Memory tiers: Working context, session summary, and a long-term store.
• Progressive disclosure: Short tool descriptions first, full specs only when a tool is called.
• Tool differentiation: No two tools with overlapping descriptions.
• Agent-Driven Dynamic Context: The agent requests what it needs instead of receiving everything.

For long-horizon work, including digital twins that track temporal state, a timestamped summary beats replaying every event. Our guide to agentic AI systems with Claude and LangChain applies that thinking to agent design.

Common Pitfalls: Where Token Cuts Backfire

One support bot was minified to save 30% of its tokens. Within a week, it was answering angry customers it should have ignored.

• Semantic bleeding and loss of nuance: Minification turns "Analyse the customer's mood, but do not respond if they are angry" into "Analyse customer mood. No respond angry". The model then drops the edge case and replies when it should stay silent.
• Reasoning degradation in chain-of-thought: A tight max_tokens cap or "answer in few words" removes step-by-step reasoning. Accuracy falls on maths, coding, and logic, so apply output token control to answers, not reasoning.
• Brittleness in few-shot examples: Trimming variations, edge cases, or formatting inside examples to fit a budget costs the model its schema. Expect malformed JSON or missed classification categories.
• Over-aggressive context pruning in RAG: An aggressive LLMLingua threshold drops product IDs or qualifiers such as "not available" and "never". The model then hallucinates confidently because the facts it needed are gone.

Never compress guardrails: Prompt injection is among the most common online attacks on an LLM-powered product, so keep a security service or security solution in the orchestration layer. A held-out eval set with one scalarised score catches all four failures, and our production-ready AI agent workflow guide covers LLM observability.

Conclusion

Token cost is rarely a model problem. It is a design problem that shows up on an invoice, and the teams that control it budget the context window first, densify the prompts they own, and compress what their systems generate, confirming output quality with evals after every cut.

Practised that way, AI prompt engineering becomes an operating discipline rather than a one-off tidy-up. The harder question is next quarter, when new features, new rules, and new contributors start adding tokens again. Savings that nobody measures and owns tend to disappear quietly.