Every engineering team I’ve talked to in the past six months shares the same frustration: AI coding assistants are great, until you look at the bill.

Let me give you a concrete example. We ran a React development task through a standard AI assistant setup. The task: implement a feature with proper error handling. The result? 20 minutes and 50,300 tokens consumed. For a single feature. In production, this compounds fast—multiplied across a team of ten engineers running dozens of sessions daily, you’re looking at serious API costs bleeding into your compute budget.

Then we discovered Caveman Mode.

Same task, same AI model, different prompt strategy. 14 minutes. 17,500 tokens. That’s a 30% speed improvement and 65% token reduction. In real money, that translates to roughly 3x cost savings on API calls.

This isn’t a benchmark from a controlled lab environment. This is from actual production usage on GitHub, where the project has accumulated over 50,000 stars in under a month. The pattern is clear: AI is generating far more output than we actually need, and somewhere in that bloat is a massive efficiency opportunity.

What Exactly Is Caveman Mode?

Caveman Mode is deceptively simple in concept. The prompt tells the AI to respond like a caveman—use one word when possible, use symbols when possible, never write a full sentence if a fragment suffices. Think: "function calc() { return x+y; }" instead of "Here's a function that calculates the sum of x and y by adding them together and returning the result."

Before you dismiss this as a gimmick, consider what the AI is actually doing under the hood. Large language models generate content through probability prediction—they calculate what the next token is most likely to be. When you constrain output to core content only, you’re effectively pre-filtering for the AI, reducing the search space across low-probability paths.

Here’s the key insight: every token the AI generates has a cost. Not just the obvious API cost—there’s also context window overhead. A 500-token response consumes more context than a 150-token response, which means every subsequent conversation turn has to process and store that additional baggage. Shorter outputs compound their savings across the entire conversation history.

The Real Numbers

Let me break down what we observed across multiple React development scenarios:

The error boundary scenario hit 87% reduction—the AI essentially stopped explaining itself and just output the code with minimal commentary. But here’s the catch: this only works because error boundary code is structurally predictable. The AI knows exactly what shape the output should take, so compression doesn’t lose information.

Where It Breaks Down

I’ve been burned by over-applying Caveman Mode. Here are the scenarios where it actively hurts:

Language and literary tasks: Ask an AI to translate classical Chinese text—like the Dao De Jing—under Caveman constraints, and you get three words: “Dao ke dao.” Technically accurate. Completely useless. The compression destroys the contextual richness that translation requires.

Complex debugging: When a bug has multiple cascading causes, the AI needs to explain the causal chain. Force compression, and you get fragments that miss critical connections. I spent more time reverse-engineering the compressed output than I would have parsing a full explanation. This is the opposite of efficiency.

Nuanced decision-making: Any task where the “why” matters more than the “what” suffers. Architecture decisions, design rationale, trade-off discussions—these need the full context that Caveman Mode strips away.

My heuristic: Caveman Mode excels at structured, predictable tasks—code generation, formula writing, data transformation. It fails at creative, explanatory, or analytical tasks where the reasoning process itself provides value.

The Deeper Question: Is Human-Like AI a Liability?

Here’s what keeps me up at night about this.

Caveman Mode saves 65% of tokens because it makes AI less human. Less natural language, fewer explanations, minimal context. The AI operates in a mode that’s efficient but feels… mechanical.

And that raises a uncomfortable question: what have we been optimizing for?

The entire trajectory of AI development has chased human-like outputs. More conversational responses. More comprehensive explanations. More context and nuance. We celebrate AI that sounds like us. But every step toward humanity is a step toward higher token consumption, longer context windows, greater compute costs.

The irony is stark: we built AI to sound human, then discovered that sounding less human is dramatically more efficient.

This isn’t a bug—it’s a fundamental characteristic. Human communication is redundant by design. We repeat ourselves for emphasis, add context for clarity, layer emotion into tone. All of that is valuable when you’re talking to another human. When you’re interfacing with a machine that needs precise instructions, all that redundancy is noise.

A Prediction: The Layered AI Communication Era

I think we’re heading toward a fundamental split in how AI systems communicate.

AI-to-AI communication will converge on compressed, efficient protocols. Think of it like machine code versus natural language—humans can read machine code, but it’s wildly inefficient for us to write. Future AI systems will likely develop implicit protocols that maximize information density per token, trading readability for efficiency. We already see this in token-saving techniques like semantic compression and structured output formats.

AI-to-human communication will retain the natural language layer—the explanations, the context, the warmth. Humans need this. Not because the AI requires it, but because we require it. The value isn’t in the information transfer; it’s in the trust and comprehension it builds.

In practice, this means AI systems will increasingly operate in two modes: a compressed internal mode for processing and computation, and an expanded external mode for human-facing output. The Caveman Mode phenomenon is an early signal of this bifurcation—the industry is discovering that one-size-fits-all communication is inefficient.

Key Takeaways

• Caveman Mode reduces token consumption by 50-87% on structured coding tasks by removing explanatory overhead
• It’s not a universal solution—creative, analytical, and explanatory tasks suffer under aggressive compression
• The efficiency comes with a cost: each step toward human-like AI output increases token consumption and compute cost
• The future is likely layered: AI-to-AI communication will use compressed protocols humans can’t easily read, while AI-to-human communication retains natural language
• The fundamental insight: AI is not inherently more valuable when it sounds more human. Sometimes, the machine-like version is exactly what you need

The Real Lesson

Caveman Mode isn’t really about compressing prompts. It’s about matching communication style to the actual requirements of the task.

When you need efficient computation, strip the human veneer. When you need explainability and trust, keep it. The mistake isn’t using either mode—the mistake is applying them blindly.

We’re in an early phase of understanding human-AI interaction efficiency. Caveman Mode is a crude first attempt at a nuanced problem. But the pattern it reveals—that human-like AI has a real cost—will shape how we build AI systems for the next decade.

Questions for you: Where have you found AI outputs to be wastefully verbose? And does the idea of AI-to-AI compressed communication feel natural to you, or does it feel like a step backward? I’d genuinely like to hear your perspective—drop it in the comments.

If you found this useful, consider subscribing. I write about AI engineering, cost optimization, and the practical realities of building with large language models.

A security researcher recently scanned 900 publicly accessible configuration files on GitHub. Within minutes, they found 41 valid, active cloud service API keys — keys that granted immediate, unauthenticated access to production servers. No brute force, no social engineering. Just a simple git grep across misconfigured repos.

This is not a hypothetical vulnerability. This is happening right now, at scale, across thousands of organizations.

Every one of those 41 keys could be used to:

• Spin up GPU instances on someone else’s bill
• Exfiltrate internal databases through API access
• Impersonate the application to end users

And here’s the uncomfortable truth: if your team uses LLM APIs — OpenAI, Anthropic, DeepSeek, or any of the dozens of providers — you almost certainly have the same problem. The only difference is you haven’t been scanned yet.

The Problem: Credential Sprawl

Modern AI-powered applications touch multiple LLM providers. A typical setup might look like this:

# .env — lives on every developer's machine
OPENAI_API_KEY=sk-...
ANTHROPIC_API_KEY=sk-ant-...
DEEPSEEK_API_KEY=sk-...
REPLICATE_API_KEY=r8-...

Each of these keys is a skeleton key to your cloud bill. But here’s how they actually get managed in practice:

• Hardcoded in source code — AI coding assistants generate boilerplate fast, and secrets end up in committed files
• Scattered across .env files — every developer, every staging server, every CI runner has a copy
• Shared team-wide — one key for everyone, impossible to revoke without breaking everything
• Stored in plaintext configs — config.json, docker-compose.yml, even README.md examples

The worst part? Most teams don’t discover the leak until the bill arrives.

A startup I spoke with discovered their OpenAI key had been exposed for six months. The attacker had been quietly running inference workloads, racking up $47,000 in charges. The breach was only noticed when the monthly bill tripled. By then, the key had already been rotated five times — and each rotation only temporarily stopped the bleeding because the key was still embedded in deployed containers.

Why This Is the Router Problem All Over Again

Twenty years ago, every device in a home needed a public IP address to access the internet. This was a nightmare: finite IPv4 addresses, security nightmares, impossible management. Then someone invented the home router.

The router solved three things:

1. Centralized access — one public IP for the whole house
2. Isolation — internal devices stay invisible from outside
3. Management — add/remove devices without rewiring the street

Every home has a router today. Not because everyone understands networking — because the problem was universal and the solution was simple.

LLM API key management is the same story. Today, every application, every microservice, every developer tool holds its own API key directly. This is the pre-router era of AI infrastructure. What you need is an LLM gateway — a centralized proxy that sits between your applications and every LLM provider.

┌─────────────┐     ┌──────────────┐     ┌─────────────┐
│ Application  │     │              │     │   OpenAI    │
│    A         │────▶│              │────▶│─────────────│
├─────────────┤     │  LLM Gateway │     │  Anthropic  │
│ Application  │     │  (proxy)     │────▶│─────────────│
│    B         │────▶│              │     │  DeepSeek   │
├─────────────┤     │  Key Mgmt    │     ├─────────────┤
│ Application  │     │  Cost Logs   │     │  Replicate  │
│    C         │────▶│  Rate Limit  │     └─────────────┘
└─────────────┘     └──────────────┘

Applications never hold provider keys. They only know the gateway.

What an LLM Gateway Actually Does

1. Key Centralization

All provider API keys live in one place — the gateway server. Applications authenticate to the gateway with short-lived, application-specific virtual keys. If a key is compromised, you revoke one virtual key without touching the underlying provider keys or affecting other applications.

2. Provider Abstraction

Your application sends OpenAI-format requests to the gateway. The gateway translates and routes to any provider. Switch from GPT-4 to Claude to DeepSeek with a config change — no code changes needed.

# Before: hardcoded provider in every service
response = openai.ChatCompletion.create(
    model="gpt-4",
    api_key=os.environ["OPENAI_API_KEY"],  # exposed everywhere
    messages=[...]
)

# After: gateway handles routing
response = requests.post("http://gateway:4000/v1/chat/completions", {
    "model": "gpt-4",          # or "claude-3-opus", "deepseek-chat"
    "messages": [...]
}, headers={
    "Authorization": "Bearer vk-xxxx"  # virtual key, one per app
})

3. Cost Visibility

Every request gets logged with model, token count, latency, and cost. Teams get a dashboard showing:

{
  "app": "customer-support-bot",
  "model": "gpt-4o",
  "input_tokens": 12500,
  "output_tokens": 340,
  "cost": 0.042,
  "latency_ms": 1200,
  "timestamp": "2026-05-27T10:30:00Z"
}

No more surprise bills. You can set per-application budgets and get alerts before costs spiral.

4. Intelligent Routing

• Cost optimization: route transcription to cheap models, complex reasoning to premium ones
• Load balancing: distribute requests across multiple provider accounts to avoid rate limits
• Failover: if one provider is down, automatically retry on another
• Rate limiting: prevent any single application from consuming the entire budget

Open Source Solution: LiteLLM

The most mature open source LLM gateway is LiteLLM — 48,000+ stars on GitHub, used by Stripe, Netflix, and Google.

Key capabilities:

• 100+ model providers unified under a single OpenAI-compatible API
• Virtual keys — generate per-application keys with spend limits, rate limits, and expiration
• Request logging — full audit trail of every LLM call
• Budget controls — set spend limits per key, per user, per project
• Model fallback — automatic retry with different models on failure
• Docker deployment — one container, zero dependencies

Deploying it takes five minutes:

docker run -d \
  --name litellm-proxy \
  -p 4000:4000 \
  -e OPENAI_API_KEY=sk-... \
  -e ANTHROPIC_API_KEY=sk-ant-... \
  ghcr.io/berriai/litellm:main-latest \
  --config /app/config.yaml

Then generate virtual keys for each application:

curl -X POST http://localhost:4000/key/generate \
  -H "Authorization: Bearer sk-admin-key" \
  -H "Content-Type: application/json" \
  -d '{
    "max_budget": 50.0,
    "metadata": {"app": "customer-support-bot"},
    "models": ["gpt-4o", "claude-3-opus"]
  }'

Response:

{
  "key": "vk-xxxxxxxxxxxxxxxxxxxxx",
  "expires": "2026-06-27T00:00:00Z",
  "max_budget": 50.0
}

How Enterprises Should Roll This Out

You don’t need to do this all at once. The pragmatic rollout:

Phase 1: Centralize (Week 1)

Deploy the gateway. Migrate all provider keys into the gateway config. Point existing applications to the gateway without changing application code — the gateway is OpenAI-compatible, so most SDKs work with just a base URL swap.

Phase 2: Virtualize (Week 2)

Generate one virtual key per application. Remove direct provider keys from all .env files, CI/CD secrets, and deployment configs. If a key leaks now, you revoke one application — not your entire infrastructure.

Phase 3: Observe (Ongoing)

Enable request logging. Build a dashboard showing per-application spend, latency, and error rates. Identify which applications use expensive models where cheaper alternatives would work.

Phase 4: Optimize (Ongoing)

Set up cost-based routing. Route bulk embedding tasks to the cheapest model, production chat to the most reliable, experimental workloads to the newest. Configure automatic failover between providers.

Phase 5: Govern (When ready)

Set per-application budgets, alerting thresholds, and automatic rate limiting. Implement approval workflows for expensive model access.

Individual Developer Self-Check

Even without a gateway, here’s what every developer should do today:

1. Scan your repos — search for patterns like sk-, api_key, secret in your codebase. Use tools like git-secrets or trufflehog to scan git history.
2. Never commit .env files — add them to .gitignore immediately. Use .env.example with placeholder values instead.
3. Rotate exposed keys — if you find keys in git history, assume they’re compromised. Rotate them now, not later.
4. Audit cloud console — check your provider dashboard for active keys. Revoke any you don’t recognize.
5. Use separate keys per service — stop sharing one key across your entire stack. The inconvenience of managing multiple keys is trivial compared to a single point of failure.

The Bottom Line

API key leakage is not a matter of if, but when. The technical debt of scattered credentials compounds daily, and the explosion of LLM usage has turned a manageable problem into a systemic risk.

The solution isn’t more discipline or better training — it’s architecture. An LLM gateway transforms credential management from a people problem into an infrastructure problem with a well-understood solution pattern.

Every enterprise needs an LLM gateway today, just like every home needed a router twenty years ago. The analogy isn’t perfect, but it’s close enough to be actionable.

Start this week. Not next quarter, not after the audit. Before your keys show up in someone else’s scan.

Have you deployed an LLM gateway in production? What’s your experience with LiteLLM or other solutions? I’d love to hear your stories and lessons learned.

DeepSeek’s disk-based automatic context caching is famous for near 90% input token savings: cached prefix tokens cost just a tiny fraction of standard input pricing, with zero manual configuration required. Thousands of developers switch to DeepSeek chasing this aggressive discount for long system prompts, code rules, and repeated tool definitions.

But a costly reality hits teams running ClaudeCode / OpenCode (code agent runtimes built for Anthropic-style cache_control) against the DeepSeek API:

Even with DeepSeek caching enabled globally, your cache hit rate collapses to near-zero, and you never see the promised ultra-low cached token billing.

This is not a bug, nor misconfiguration. It is a fundamental architectural incompatibility between two entirely different caching paradigms: Anthropic’s manual segmented block caching, and DeepSeek’s rigid full-sequence prefix-only matching.

In this post, we break down the mechanics, agent workflow pain points, and why mixing these stacks wastes your biggest cost-saving feature.

1. Core Background: How Each Caching System Works

1.1 DeepSeek Automatic Prefix Cache (Strict Rule Set)

DeepSeek enables caching for all API keys by default, with one non-negotiable matching rule:

✅ A cache hit only triggers when the full token sequence starts identical from index 0 (the very first token).

• The entire messages[] array must be an exact prefix extension: new content can only be appended to the END of the list.
• Any insertion, deletion, or content change anywhere before the final position breaks the full prefix hash → full cache miss.
• No manual tagging, no custom breakpoints, no separate cache segments; the entire message chain is treated as one single prefix unit.
• Pricing benefit: Miss = full standard input cost; Hit = ultra-low discounted rate for matched prefix tokens.

1.2 Anthropic cache_control Segmented Block Caching (What OpenCode / ClaudeCode Relies On)

Anthropic designed cache_control explicitly for dynamic agent workflows:

• Developers add a cache_control tag inside individual content blocks (system prompts, tool definitions, static rule chunks) to create independent cache segments.
• Up to four isolated cache breakpoints per request; each segment has its own TTL (ephemeral / long_lived) and independent storage.
• Critical advantage: Modifications to later blocks do NOT invalidate earlier cached segments. If you insert tool_use / tool_result messages between marked static blocks, the pre-tagged system/tool definitions remain cached at discounted pricing.

OpenCode / ClaudeCode are hardcoded to inject these cache_control markers automatically for long system rules, code guidelines, and tool schemas — this is their core cost-optimization logic for multi-turn code agents.

The First Hard Block: DeepSeek Ignores cache_control Entirely

When OpenCode sends requests with embedded cache_control fields:

1. DeepSeek’s API silently drops the unknown field and does not parse any manual segment tags.
2. No independent blocks are created; the entire messages list is still evaluated as one single full prefix.
3. LiteLLM / proxy gateways also strip the field before forwarding to avoid invalid parameter errors.

Your agent’s intelligent segmented caching logic becomes completely invisible to DeepSeek.

2. Why Code Agent Workflows Destroy DeepSeek’s Prefix Match

Standard code agents (ClaudeCode / OpenCode) run a repeating loop that guarantees middle-position message insertion — the exact scenario that breaks full-prefix caching.

Step-by-Step Agent Loop Breakdown

1. Initial request: [SystemPrompt (code rules) → User task] DeepSeek caches this full 2-block prefix after first call.

2. Model returns assistant with tool_calls (file read, shell run, code edit).

3. Critical breaking step: Your agent appends a standalone role: tool message between the last assistant and the next user message, not only at the list tail.

   New full sequence: [System → User → Assistant(tool_call) → ToolResult]

What happens on DeepSeek side:

• The original cached prefix was length 3 items; new request has 4 items total.
• Even though the first three messages are textually identical, the overall sequence length and array structure differ from the stored prefix hash.
• Result: 100% cache miss; you pay full price for the entire long system prompt every round.

Additional Failure Modes in Multi-Agent Setups

Most code platforms run multiple specialized agents, each with its own unique system prompt:

• Agent A: Code writer system rules
• Agent B: Linter & reviewer system rules
• Agent C: Shell executor rules

With DeepSeek prefix caching:

• Every unique system prompt creates a separate cache entry.
• No cross-agent sharing of common content (shared tool definitions, global coding constraints), because the starting system block differs per agent.
• Cache storage fills rapidly with fragmented entries; LRU eviction purges frequently used static prompts, worsening miss rates further.

Dynamic Variables Inside System Prompts Kill Consistency

OpenCode commonly injects real-time variables into system prompts:

• Current date (resets daily)
• Project working directory / file paths (switches per workspace)

Even minor text changes at the start of the system block rewrite the full prefix hash. DeepSeek cannot isolate the fixed rule portion; the entire thousands-of-token prompt misses cache overnight or on workspace switch.

With Anthropic segmented caching: only the small dynamic date/path segment re-runs the write premium; massive static code rules stay cached daily.

3. The Cost Gap: Real-World Comparison

Scenario: 15-turn code agent run | 12k-token static system prompt

A) Native Claude + cache_control

• 1x write premium for system/tool blocks
• Next 14 rounds: static segments hit 10% discounted read pricing
• Total input cost: ~2.15 × base price

B) OpenCode + DeepSeek (default deployment)

• Every tool insertion = full cache miss on all turns
• You pay full standard input cost for the 12k system prompt 15 times consecutively
• Total input cost: 15 × base price → ~7x more expensive than expected DeepSeek discount

C) Pure DeepSeek simple chat (only tail-appended messages)

• Stable full prefix hit every turn
• Total input cost: ~1.8 × base price (maxed DeepSeek discount)

The agent workflow eliminates all DeepSeek economic benefits entirely.

4. Can We Fix This With Workarounds?

Workaround 1: Remove all cache_control injection

Disabling automatic tagging makes requests valid for DeepSeek, but does not solve the core prefix-break issue during tool calls. Hit rates remain extremely low.

Workaround 2: Force all dynamic content to the very end of messages[]

Move dates, paths, and variable data strictly after all static system rules and history. This slightly improves hit rates for simple chats, but cannot fix middle tool message insertion in agent loops.

Workaround 3: Pre-warm fixed prefixes

Pre-send requests for all agent system templates to populate cache ahead of traffic. This helps static one-off calls but fails for tool loops, as insertion still invalidates matches.

Hard Truth

There is no reliable workaround to make Anthropic-style segmented agents work with DeepSeek full-prefix caching. The two systems have opposing design constraints.

5. Two Valid Deployment Options

Option A: Keep OpenCode / ClaudeCode → Use Anthropic / MiniMax natively

These models natively support cache_control block segmentation. Tool insertions only affect variable segments; static system/tool definitions retain discounted reads. This matches your agent runtime’s built-in optimization logic perfectly.

Option B: Keep DeepSeek → Rewrite agent logic for strict full-prefix workflow

Mandate these rules for your agent:

1. Never insert tool messages anywhere except the absolute end of the message array.
2. Freeze the full system prompt structure; avoid dynamic dates/paths inside the leading system block.
3. Disable all cache_control injection in OpenCode.

This enables DeepSeek’s automatic caching, but sacrifices flexible multi-agent & complex tool workflows.

Never Recommend: Hybrid OpenCode + DeepSeek

It combines the overhead of an agent built for segmented caching with a model that cannot honor that logic — you pay double engineering cost with zero discount gains.

Conclusion

DeepSeek’s automatic prefix caching delivers industry-leading savings only for simple sequential conversations where new messages are exclusively appended to the end.

Runtimes like ClaudeCode / OpenCode are engineered around Anthropic’s flexible cache_control block tagging, designed for dynamic agent loops with mid-sequence tool message insertion. When paired with DeepSeek:

1. cache_control tags are ignored; no segmented caching occurs.
2. Tool result insertion breaks the mandatory full-start prefix match every turn.
3. Cache hit rates plummet, and you never receive DeepSeek’s advertised ultra-low cached token pricing.

Choose your stack based on caching architecture, not just per-token sticker price:

• Complex code agents with frequent tool calls → Anthropic / MiniMax (cache_control native support)
• Simple long-chat workloads with fixed trailing history → DeepSeek automatic prefix cache

Author Note

If you audit your API usage dashboards and see prompt_cache_hit_tokens near zero despite enabling DeepSeek caching, this architecture mismatch is almost certainly the root cause.