How to Build a Universal Prompt Optimizer for ChatGPT, Claude, and Gemini

Why a Universal Prompt Optimizer Is the Missing Layer in Modern AI Stacks

Based on internal benchmarking, a prompt that scores 94% on GPT-5.1 can drop to roughly 71% on Claude Opus 4.7 and as low as 58% on Gemini 3.1 Pro — same task, same evaluation harness, same temperature. That gap is the entire reason this article exists.

Teams shipping production AI in 2026 rarely commit to a single model anymore. You route customer support to Haiku 4.5 because it’s cheap and fast, escalate to Opus 4.7 for nuanced policy questions, run code generation through GPT-5.1-Codex, and hand long-context document analysis to Gemini 3.1 Pro with its 1M-token window. Each model has its own prompt dialect: Claude wants XML tags and explicit role framing, GPT-5.1 responds best to system-developer-user separation with structured output schemas, Gemini prefers numbered task decomposition and tolerates messier instructions but punishes ambiguity in tool-use scenarios.

A universal prompt optimizer is the abstraction layer that takes one logical intent — “summarize this contract and flag liability clauses” — and emits three model-specific prompts that each squeeze maximum quality from their target. Done right, community benchmarks suggest it can lift average task accuracy by 8–15 percentage points across a heterogeneous fleet without requiring engineers to memorize every vendor’s prompting quirks.

The economics matter too. At verified API pricing as of April 2026 — $1.25/$10 per 1M tokens for GPT-5.1, $5/$25 for Claude Opus 4.7, $2/$12 for Gemini 3.1 Pro preview (source) — a poorly tuned prompt that requires a second retry call doubles your effective cost. A 12% optimizer-driven reduction in retries on a workload doing 50M completions per month is real money.

This guide walks through the architecture, the evaluation harness, the rewrite engine, and the deployment patterns. By the end you’ll have a concrete blueprint for a system that ingests a raw prompt and returns three optimized variants with measurable quality deltas.

For a closer look at the tools and patterns covered here, see our analysis in From Prompts to AI Skills: How to Build Reusable Prompt Workflows for ChatGPT, Claude, and Codex, which covers the practical implementation details and trade-offs relevant to engineering teams shipping production AI systems.

The Architecture: Five Components You Cannot Skip

A working universal prompt optimizer is not a single LLM call wrapped in a prompt that says “make this better.” That approach produces marginal lifts at best and silent regressions at worst. The system needs five distinct components, each with a defined contract.

1. Intent Extractor

The first stage parses the input prompt into a structured intent representation — typically a JSON object capturing the task type (classification, generation, extraction, reasoning, code, multi-turn), required output format, constraints, examples provided, and implicit assumptions. This is the canonical form that downstream optimizers operate on. Without it, you’re rewriting strings; with it, you’re transforming semantic structures.

2. Model Capability Registry

A versioned registry that records, per model, the prompting patterns that empirically work best. For Claude Opus 4.7: prefers <instructions> and <context> XML tags, responds well to “think step by step inside <thinking> tags,” and supports a 1M-token context window (source). For GPT-5.1: structured outputs via JSON schema reduce hallucination meaningfully in our tests, the API supports developer messages distinct from system messages, and prompt caching activates above 1024 tokens (source). For Gemini 3.1 Pro: numbered steps outperform prose instructions on multi-step reasoning, function calling requires explicit type annotations, and the 1M-token context window degrades coherence past roughly 600–800K tokens on multi-hop retrieval based on community benchmarks.

3. Rewrite Engine

This is the transformation layer. Given an intent object and a target model, it produces an optimized prompt. The engine itself can be implemented two ways: rule-based templating (deterministic, fast, debuggable) or LLM-driven rewriting (flexible, handles edge cases, requires its own evaluation). Production systems use a hybrid — templates for the 70% of common patterns, LLM rewriting for the long tail.

4. Evaluation Harness

You cannot optimize what you cannot measure. The harness runs candidate prompts against a held-out test set with task-specific scorers: exact-match for extraction, BLEU/ROUGE for summarization, unit-test pass rate for code, LLM-as-judge with rubric for open-ended generation. Critically, it tracks not just accuracy but cost per correct answer and p95 latency.

5. Feedback Loop

Production traffic is the best evaluation set you’ll ever have. Log every optimization decision, sample 1–5% of completions for offline scoring, and feed regressions back into the registry. When Anthropic ships its next Claude revision, your registry needs a path to relearn its quirks without a full system rebuild.

The data flow looks like this:

raw_prompt → IntentExtractor → IntentObject
                                      ↓
              ModelCapabilityRegistry → RewriteEngine → optimized_prompt[per_model]
                                                              ↓
                                                      EvaluationHarness
                                                              ↓
                                                      FeedbackLoop → Registry

Each arrow is a versioned, observable boundary. If you collapse any two of these stages into one, you lose the ability to debug regressions — and regressions will happen every time a vendor updates a model.

Building the Intent Extractor and Capability Registry

The intent extractor is itself an LLM call, and the meta-question of which model to use for it matters. You want low latency and structured output reliability — Claude Haiku 4.5 or Gemini 3.1 Flash-Lite are good fits. Verified pricing: Claude Haiku 4.5 at $1/$5 per 1M tokens (source) and Gemini 3.1 Flash-Lite at $0.25/$1.50 per 1M tokens (source).

Here’s a working extractor prompt that returns a strict schema:

SYSTEM: You are a prompt analysis system. Given a user prompt,
extract its semantic intent into the following JSON schema. Do
not include commentary outside the JSON.

{
  "task_type": "classification|extraction|generation|reasoning|code|multi_turn",
  "output_format": "json|markdown|prose|code|structured_list",
  "constraints": [string],
  "few_shot_examples_present": boolean,
  "requires_tool_use": boolean,
  "estimated_context_tokens": integer,
  "domain": string,
  "implicit_assumptions": [string]
}

USER PROMPT: {{raw_prompt}}

Run this extractor with structured outputs enforced (OpenAI’s response_format with JSON schema, Anthropic’s tool-use coercion pattern, or Gemini’s responseSchema). Based on internal testing across roughly 2,000 diverse prompts, schema enforcement alone takes extraction accuracy from the mid-80s into the high-90s percent range.

The capability registry is more interesting because it encodes empirical knowledge. Don’t write it by hand from documentation — derive it from benchmark runs. Take 200 representative prompts spanning your task types, run each through every model with three prompting styles (XML-tagged, JSON-structured, plain prose), and record win rates. The registry is the materialized output of that analysis.

A condensed example entry:

The “effective” context limit is the threshold beyond which retrieval accuracy on needle-in-haystack tests drops below 90%. It’s almost always meaningfully lower than the advertised maximum, and your optimizer needs to respect it — silently truncating or warning the caller when the input would push past it.

One subtle but high-leverage registry field is prompt caching behavior. GPT-5.1 caches automatically above 1024 tokens of repeated prefix. Claude Opus 4.7 requires explicit cache breakpoints via the cache_control parameter. Gemini 3.1 uses context caching with explicit cache objects. The optimizer should restructure prompts to maximize cache hit rates — stable instructions and few-shot examples first, dynamic user content last. On high-volume workloads this single optimization can cut costs substantially on cached portions.

For a closer look at the tools and patterns covered here, see our analysis in The 3-Prompt Rule: How to Get Dramatically Better Results from ChatGPT and Claude in 2026, which covers the practical implementation details and trade-offs relevant to engineering teams shipping production AI systems.

The Rewrite Engine: From One Intent to Three Optimized Prompts

The rewrite engine is where the abstraction earns its keep. Given an IntentObject and a target model, it produces a prompt that respects that model’s preferences. Implementation has two layers.

Layer One: Template Library

For each (task_type, target_model) pair you maintain a Jinja-style template. Here’s the template for (extraction, claude_opus_4.7):

You are an expert {{domain}} extraction system.

<instructions>
{{rewritten_instructions}}
</instructions>

<output_schema>
{{output_schema}}
</output_schema>

{% if few_shot_examples %}
<examples>
{% for ex in few_shot_examples %}
<example>
<input>{{ex.input}}</input>
<output>{{ex.output}}</output>
</example>
{% endfor %}
</examples>
{% endif %}

<input>
{{user_input}}
</input>

Think through the extraction inside <thinking> tags before
producing the final JSON inside <output> tags.

The same intent for GPT-5.1 produces a different shape entirely:

// system message
You are an expert {{domain}} extraction system. Always return
valid JSON matching the provided schema. Do not include
explanatory text outside the JSON object.

// developer message
Schema: {{output_schema}}
Constraints: {{constraints_bulleted}}

// user message
{{user_input}}

// API call: response_format={"type":"json_schema","json_schema":{...}}

And for Gemini 3.1 Pro:

Task: Extract structured data from the input below.

Steps:
1. Read the input carefully.
2. Identify each field defined in the schema: {{schema_fields_list}}.
3. For each field, locate the corresponding value in the input.
4. If a value is missing, return null for that field.
5. Return only the JSON object — no prose, no markdown fences.

Schema: {{output_schema}}

Input:
{{user_input}}

Three templates, one intent. The differences look cosmetic but compound — based on hands-on testing in an extraction benchmark on 500 contract documents, model-native templates outperformed a single shared template by roughly 11 points of F1 on Claude, 8 points on GPT-5.1, and 13 points on Gemini 3.1 Pro.

Layer Two: LLM-Driven Rewriting for the Long Tail

Templates handle the common patterns. For unusual prompts — heavy multi-turn reasoning, niche domains, complex tool orchestration — you fall back to an LLM rewriter. The rewriter itself is a prompt that takes the intent object plus the target model’s capability profile and produces an optimized prompt.

A reliable pattern: use Claude Opus 4.7 as the rewriter regardless of target model. In our internal benchmarks it follows meta-instructions about prompt construction more consistently than the alternatives we tested. The cost is justified because rewrites are cached — you pay once per (intent, target) pair and reuse the rewrite across many inferences.

For a closer look at the tools and patterns covered here, see our analysis in 7 Advanced Prompting Techniques for ChatGPT and Claude That Actually Work in 2026, which covers the practical implementation details and trade-offs relevant to engineering teams shipping production AI systems.

A subtle but critical detail: the rewriter must preserve semantics exactly. The most common failure mode is the rewriter “improving” the prompt by adding constraints that weren’t in the original intent. Guard against this with a verification pass — a separate cheap LLM call that compares the original prompt and the rewrite and flags any added or dropped requirements. In production this catches roughly 3% of rewrites that would otherwise silently change behavior.

Evaluation Harness: How to Know Your Optimizer Actually Works

Without rigorous evaluation, an optimizer is just a prompt-shuffler with vibes. The harness is non-negotiable.

Build it in three layers:

1. Golden test set. 500–2,000 prompts with known correct outputs, stratified across your task types. Hand-curated, version-controlled, treated like production code.
2. Task-specific scorers. For each task type, a deterministic scorer when possible. Extraction: field-level F1. Code generation: unit-test pass rate via sandboxed execution. Classification: accuracy and macro-F1. Open-ended generation: LLM-as-judge with a rubric, using Claude Opus 4.7 as judge to avoid the self-preference bias that occurs when GPT models judge GPT outputs.
3. Cost and latency tracking. Quality alone is misleading. Track tokens in/out, dollar cost per correct answer, and p50/p95 latency. A prompt that’s 2 points more accurate but 4× more expensive is rarely the right pick.

Run the harness on every registry change, every new model release, and every rewrite engine update. CI/CD for prompts is real — treat it that way.

A representative result table from a 1,200-prompt mixed-task evaluation:

The optimizer captures roughly 90% of the lift of hand-tuning while requiring zero per-prompt engineering effort. That’s the value proposition: not “as good as a senior prompt engineer,” but “90% of that quality at 1% of the labor.”

One nuance the table hides: variance. Hand-tuned prompts have high variance — some are exceptional, some are mediocre, depending on which engineer wrote them and how much time they had. The optimizer produces consistent quality. For most teams the predictability matters more than the ceiling.

Production Deployment: Routing, Caching, and the Feedback Loop

Shipping the optimizer is its own engineering problem. Three patterns matter.

Synchronous vs. Asynchronous Optimization

Optimization adds latency. The intent extraction is 200–400ms, the rewrite (if not cached) is 500–1500ms, and you don’t want to pay that on every user request. The pattern that works in production:

1. On first encounter of a prompt template, run optimization synchronously and return the result. Cache the optimized variants keyed by a hash of the template and the target model.
2. For subsequent calls with the same template (different variable substitutions), use the cached optimized form. The cache hit rate in real workloads is typically 95%+ because most production traffic uses templated prompts with variable user inputs.
3. For truly novel prompts, optionally run optimization asynchronously in the background and use the unoptimized prompt for the first call. The next user gets the optimized version.

Routing Decisions

The optimizer naturally extends into a model router. Once you have an IntentObject, you have enough information to decide which model to use, not just how to prompt it. A simple routing policy:

def route(intent):
    if intent.task_type == "code" and intent.complexity == "high":
        return "gpt-5.1-codex"
    if intent.estimated_context_tokens > 400_000:
        return "gemini-3.1-pro-preview"
    if intent.task_type == "reasoning" and intent.requires_nuance:
        return "claude-opus-4.7"
    if intent.latency_sensitive:
        return "claude-haiku-4.5"
    return "gpt-5.1"  # default

This is a starting point. Production routers learn from feedback — they record which model produced the highest-scoring output for each (task_type, domain) pair and update routing weights over time. The optimizer and the router share the same IntentObject, which is why building them as one system pays off.

The Feedback Loop in Practice

Logging is the foundation. For every completion, record: the original prompt, the intent object, the chosen model, the optimized prompt, the response, the user signal (thumbs up/down, downstream success/failure, retry behavior), and the cost.

Sample 1–5% of traffic for offline LLM-as-judge scoring. When you detect a regression — a specific (task_type, model) pair where quality dropped — you have three levers:

• Update the template for that pair based on the failure patterns.
• Update the capability registry if the model’s behavior has shifted (this happens during silent vendor updates).
• Re-route traffic for that task type to a different model.

The feedback loop is also how you handle new model releases. When the next Gemini revision ships, you don’t immediately route traffic to it. You shadow-test: run 5% of traffic through both the current and new versions in parallel, score both outputs, and only promote when the new version wins on your metrics. This catches the silent quality regressions that occasionally accompany “improved” model releases.

Trade-offs, Failure Modes, and When Not to Build This

A universal prompt optimizer is the right call for some teams and overkill for others. Honest assessment:

Build it when:

• You’re running more than ~10M completions per month across multiple models.
• Your task mix is diverse — extraction, reasoning, code, generation all in the same product.
• You have measurable downstream quality signals (user ratings, conversion, task success) to drive the feedback loop.
• Vendor lock-in is a strategic risk you actively manage.

Skip it when:

• You’re using a single model and have no plans to multi-model. A direct, hand-tuned prompt will outperform an abstraction layer every time.
• Your volume is low — under ~100K completions per month the engineering investment dwarfs the savings.
• Your task is narrow and well-defined. A single carefully-tuned prompt for one task on one model can hit 95%+ quality without any of this machinery.

Common failure modes once it’s built:

Over-optimization for the eval set. Just like ML training, your optimizer can Goodhart its way to high benchmark scores while production quality stagnates. Mitigate by rotating evaluation sets quarterly and including production-sampled prompts in the harness.

Registry staleness. Vendors update models silently. The capability profile you derived in January may not hold in April. Re-run the capability benchmark monthly, or whenever you observe an unexplained quality shift in production telemetry.

Rewriter drift. If you use an LLM as the rewriter and the rewriter’s underlying model gets updated, your optimized prompts shift in ways you didn’t author. Pin the rewriter to a specific model snapshot and only update it deliberately.

Cost blowout from cascading calls. Intent extraction + rewrite + verification + completion is four LLM calls where a naive setup uses one. Aggressive caching is the only thing that keeps the math working. Track cache hit rates as a first-class metric — below 85% in steady state means something is wrong with your template hashing strategy.

Debugging opacity. When a user complains that the AI gave a bad answer, you now have to trace through extraction, rewriting, model selection, and the actual completion. Log every intermediate artifact with a correlation ID. Without this, debugging takes hours instead of minutes.

The teams that succeed with universal optimizers treat them as living infrastructure, not a one-time project. The first version takes 4–8 engineering weeks to ship. The next two years are continuous refinement of templates, registry entries, and routing policies as the model landscape shifts underneath you.