How to Build a Code Review Bot with GPT-5 Pro in 2026: Step-by-Step

[IMAGE_PLACEHOLDER_HEADER]

Why Code Review Bots Became Table Stakes in 2026

[IMAGE_PLACEHOLDER_SECTION_1]

The median pull request at a mid-sized engineering org now waits 18 hours for first human review. That number has barely moved since 2022, even as PR volume has roughly tripled thanks to AI-assisted coding tools like GPT-5.3-Codex and Claude Sonnet 4.6. The bottleneck is no longer writing code — it’s reviewing it.

This is the gap a well-built code review bot fills. Not to replace senior engineers on architectural decisions, but to catch the boring 70% of comments — unused imports, missing null checks, N+1 queries, weak test coverage on new branches, security anti-patterns — before a human even opens the diff. When you offload that layer, human reviewers spend their attention on design and business logic, which is where their time actually compounds.

GPT-5-Pro sits in an interesting spot for this job. It’s expensive at $15 input / $120 output per million tokens (source), but its extended reasoning mode produces the kind of thorough, multi-file analysis that cheaper models miss. For a solo dev reviewing personal projects, GPT-5-mini or Claude Haiku 4.5 is fine. For a team pushing 200+ PRs per week where a missed vulnerability costs six figures, GPT-5-Pro’s reasoning depth is what you actually want carrying the load on your critical services.

This walkthrough covers the full build: webhook ingestion from GitHub, diff parsing, prompt construction with careful context-window management, calling the GPT-5-Pro API with structured outputs, posting inline comments back to the PR, and the operational stuff (rate limits, cost caps, prompt caching) that separates a weekend hack from something you can leave running on a production repo.

You’ll finish with a working Python service you can deploy on Fly.io, Railway, or any Docker host, plus a decision framework for when to route reviews to GPT-5-Pro versus a cheaper model. For the engineering trade-offs behind this approach, see our analysis in How to Build a Code Review Bot with Claude Sonnet 4.6 in 2026: Step-by-Step, which breaks down the cost-vs-quality decisions in detail.

Architecture: What the Bot Actually Does

[IMAGE_PLACEHOLDER_SECTION_2]

Before writing any code, get the architecture straight. A code review bot is a webhook-driven pipeline with five distinct stages, and confusing them is where most implementations go wrong.

Stage 1: Event ingestion. GitHub, GitLab, or Bitbucket sends an HTTP POST when a PR opens, updates, or gets re-requested for review. Your service verifies the HMAC signature, parses the payload, and decides whether to act. Skip drafts, skip bot-authored PRs, skip files matching your ignore list (lockfiles, generated code, migrations).

Stage 2: Diff acquisition. Fetch the unified diff via the GitHub API (GET /repos/{owner}/{repo}/pulls/{number} with Accept: application/vnd.github.v3.diff). For PRs above ~2000 changed lines, fetch per-file to enable chunked review — GPT-5-Pro’s 400K context window is generous but not infinite, and stuffing a 10K-line refactor into one prompt wastes tokens and dilutes attention.

Stage 3: Context enrichment. The diff alone is not enough. To catch cross-file issues, you need the surrounding function bodies of any modified function, plus the file’s imports and top-level exports. This is where cheap review bots fall over — they review the diff in isolation and miss that process_payment() now returns None in one branch but the three callers still expect a dict.

Stage 4: Model call. Send a structured prompt to GPT-5-Pro with the enriched diff, requesting JSON output that conforms to a schema: an array of findings, each with file, line, severity, category, and comment. Use OpenAI’s structured outputs feature (response_format: { type: "json_schema" }) so the model can’t return malformed data.

Stage 5: Comment posting. Convert findings to inline review comments via POST /repos/{owner}/{repo}/pulls/{number}/reviews. Group by file, deduplicate against previous bot reviews on the same PR (so pushing three commits doesn’t produce three copies of the same nit), and post a single summary review with a verdict: COMMENT, REQUEST_CHANGES, or APPROVE.

The whole loop should complete in under 90 seconds for a typical PR. GPT-5-Pro’s reasoning mode adds latency — expect 30-60 seconds for a 500-line diff — which is why you run it asynchronously behind a queue rather than trying to respond synchronously to the webhook.

Choosing your model tier

Not every PR deserves GPT-5-Pro’s price tag. A sensible routing policy:

PR characteristic	Model	Approx cost per review
<50 lines, docs/tests only	gpt-5.4-mini	$0.005
50-500 lines, application code	gpt-5.4	$0.04
500+ lines OR touches auth/payments/security	gpt-5-pro	$0.30-0.80
Any PR on a critical service, on-demand	gpt-5-pro (extended reasoning)	$1-3

Route decisions based on file paths, changed line count, and PR labels. A security-review label always escalates to GPT-5-Pro regardless of size. This tiered approach typically keeps monthly bot spend under $500 for a team pushing 40 PRs per day, versus $3000+ if you naively route everything to the Pro tier.

Baseline architecture diagram (textual)

• GitHub App → Webhook (FastAPI) → Signature Verify → Queue (Redis/ARQ) → Worker
• Worker → GitHub REST API (diff + files) → Context Enrichment → Model Router → GPT-5-Pro (or tiered model)
• Worker → Findings Dedupe (Redis) → GitHub Reviews API → Inline Comments + Summary
• Metrics/Logs → OpenTelemetry/Prometheus → Dashboards/Alerts

Building the Service: From Webhook to First Review

[IMAGE_PLACEHOLDER_SECTION_3]

Here’s the concrete build. The stack: Python 3.12, FastAPI for the webhook endpoint, httpx for GitHub API calls, the openai SDK v2.x for model calls, Redis for deduplication state, and a background worker via arq or Celery. Everything below assumes you’ve already created a GitHub App with pull_requests: write and contents: read permissions, and stashed its private key.

Step 1: Webhook receiver with signature verification

from fastapi import FastAPI, Request, HTTPException
import hmac, hashlib, os, json
import asyncio
from arq import create_pool
from arq.connections import RedisSettings

app = FastAPI()
WEBHOOK_SECRET = os.environ["GITHUB_WEBHOOK_SECRET"].encode()
REDIS_SETTINGS = RedisSettings.from_dsn(os.environ.get("REDIS_URL", "redis://localhost:6379"))

def verify_signature(payload: bytes, header: str) -> bool:
    if not header or not header.startswith("sha256="):
        return False
    expected = "sha256=" + hmac.new(
        WEBHOOK_SECRET, payload, hashlib.sha256
    ).hexdigest()
    return hmac.compare_digest(expected, header)

async def enqueue_review(**kwargs):
    redis = await create_pool(REDIS_SETTINGS)
    await redis.enqueue_job("run_pr_review", **kwargs)

@app.post("/webhook")
async def webhook(request: Request):
    body = await request.body()
    sig = request.headers.get("X-Hub-Signature-256", "")
    if not verify_signature(body, sig):
        raise HTTPException(401, "bad signature")

    event = request.headers.get("X-GitHub-Event")
    payload = await request.json()

    if event == "pull_request" and payload["action"] in {"opened", "synchronize", "reopened", "ready_for_review"}:
        pr = payload["pull_request"]
        if pr.get("draft") or pr["user"]["type"] == "Bot":
            return {"skipped": True}
        await enqueue_review(
            repo=payload["repository"]["full_name"],
            pr_number=pr["number"],
            head_sha=pr["head"]["sha"],
            base_sha=pr["base"]["sha"],
            installation_id=payload["installation"]["id"],
            pr_id=pr["id"],
        )
    return {"ok": True}

The signature check is non-negotiable. GitHub sends the raw payload with an HMAC-SHA256 signature; if you skip verification, anyone who guesses your endpoint URL can trigger reviews and burn your OpenAI budget. Use hmac.compare_digest to avoid timing attacks.

Step 1.1: Authenticating to GitHub as a GitHub App

import time, jwt, httpx, os

GITHUB_APP_ID = os.environ["GITHUB_APP_ID"]
GITHUB_PRIVATE_KEY = os.environ["GITHUB_PRIVATE_KEY"]  # PEM contents

def app_jwt():
    now = int(time.time())
    payload = {"iat": now-60, "exp": now + (9*60), "iss": GITHUB_APP_ID}
    return jwt.encode(payload, GITHUB_PRIVATE_KEY, algorithm="RS256")

async def installation_token(installation_id: int) -> str:
    async with httpx.AsyncClient(base_url="https://api.github.com") as c:
        r = await c.post(
            f"/app/installations/{installation_id}/access_tokens",
            headers={"Authorization": f"Bearer {app_jwt()}", "Accept": "application/vnd.github+json"}
        )
        r.raise_for_status()
        return r.json()["token"]

def gh_client(token: str) -> httpx.AsyncClient:
    return httpx.AsyncClient(
        base_url="https://api.github.com",
        headers={"Authorization": f"token {token}", "Accept": "application/vnd.github+json"},
        timeout=30.0
    )

Step 2: Diff fetching and enrichment

Once the job hits the worker, fetch the diff and enrich it with surrounding context. GPT-5-Pro reviews are dramatically better when the model can see the full function body of anything modified, not just the changed lines.

import base64
from typing import List, Dict, Any

SKIP_PATTERNS = ("package-lock.json", "yarn.lock", "pnpm-lock.yaml", ".min.js", ".map", "vendor/", "node_modules/", "dist/", "build/", ".pb.go", ".gen.", "migrations/")

def should_skip(path: str) -> bool:
    return any(p in path for p in SKIP_PATTERNS)

def decode_content(obj: Dict[str, Any]) -> str:
    if obj.get("encoding") == "base64":
        return base64.b64decode(obj["content"]).decode("utf-8", errors="replace")
    return obj.get("content", "")

async def fetch_pr_context(client: httpx.AsyncClient, repo: str, pr_number: int, head_sha: str):
    # Get changed files with per-file patches
    r = await client.get(f"/repos/{repo}/pulls/{pr_number}/files", params={"per_page": 100})
    r.raise_for_status()
    files = r.json()

    enriched: List[Dict[str, Any]] = []
    for f in files:
        if f["status"] == "removed" or should_skip(f["filename"]):
            continue
        entry = {"path": f["filename"], "patch": f.get("patch"), "truncated_context": False}
        if f.get("changes", 0) > 800 or not f.get("patch"):
            entry["truncated_context"] = True
            enriched.append(entry)
            continue
        # Fetch full file content at HEAD for surrounding context
        fc = await client.get(f"/repos/{repo}/contents/{f['filename']}", params={"ref": head_sha})
        if fc.status_code == 200:
            entry["full_content"] = decode_content(fc.json())
        enriched.append(entry)
    return enriched

The should_skip helper filters out package-lock.json, *.min.js, migration files, generated protobuf, and anything in vendor/ or node_modules/. Reviewing a 400KB lockfile diff is pure token waste.

Step 3: The system prompt

This is where most implementations underperform. The prompt shapes everything. GPT-5-Pro follows instructions with high fidelity, so specificity pays off.

SYSTEM_PROMPT = """You are a senior staff engineer performing code review.
Your reviews are known for being direct, technically precise, and free of nitpicks
that don't affect correctness, security, or maintainability.

Project norms (if present in the prompt) override your defaults.

Rules:
1. Only comment when there is a real issue. Do not praise, do not restate what
   the code does, do not comment on style covered by the project linter.
2. Every finding must specify: file path, exact line number in the NEW file,
   severity (blocker/major/minor), category (bug/security/performance/design/test),
   and a concrete suggested fix.
3. If you would change <5 lines to fix it, include the exact replacement code.
4. Flag missing test coverage only when the change modifies non-trivial logic.
5. For security: focus on injection, auth bypass, secret exposure, unsafe
   deserialization, SSRF, path traversal, and race conditions. Ignore theoretical issues.
6. If the diff is fine, return an empty findings array. It is correct and
   expected to say nothing on well-written PRs.
7. Prefer project idioms (framework, language version, testing libraries) when suggesting fixes.

Return only JSON matching the provided schema."""

Rule 6 matters more than it looks. Untuned LLMs will invent problems to justify their existence — the “reviewer’s Dunning-Kruger” failure mode. Explicitly permitting silence produces reviews humans actually trust. For the engineering trade-offs behind this approach, see our analysis in How to Build an AI Agent with GPT-5 Pro in 2026: Step-by-Step, which breaks down the cost-vs-quality decisions in detail.

Step 4: Structured output schema

REVIEW_SCHEMA = {
    "type": "object",
    "properties": {
        "verdict": {"type": "string", "enum": ["approve", "comment", "request_changes"]},
        "summary": {"type": "string", "maxLength": 800},
        "findings": {
            "type": "array",
            "items": {
                "type": "object",
                "properties": {
                    "file": {"type": "string"},
                    "line": {"type": "integer", "minimum": 1},
                    "severity": {"type": "string", "enum": ["blocker", "major", "minor"]},
                    "category": {"type": "string", "enum": ["bug", "security", "performance", "design", "test"]},
                    "comment": {"type": "string"},
                    "suggested_fix": {"type": ["string", "null"]},
                    "confidence": {"type": "number", "minimum": 0, "maximum": 1}
                },
                "required": ["file", "line", "severity", "category", "comment", "suggested_fix"],
                "additionalProperties": False
            }
        }
    },
    "required": ["verdict", "summary", "findings"],
    "additionalProperties": False
}

Step 5: Build the user message

def build_user_message(enriched_files, pr_metadata) -> str:
    header = [
        f"Repository: {pr_metadata['repo']}",
        f"PR #{pr_metadata['number']} — {pr_metadata.get('title','')}",
        f"Base: {pr_metadata['base_sha'][:7]}  Head: {pr_metadata['head_sha'][:7]}",
        "",
        "Project context (if any):",
        pr_metadata.get("project_context","(none)"),
        "",
        "Changed files and patches:"
    ]
    parts = ["n".join(header)]
    for f in enriched_files:
        parts.append(f"---nPATH: {f['path']}nTRUNCATED: {f.get('truncated_context', False)}")
        if f.get("full_content"):
            parts.append("nn[BEGIN FULL FILE]n" + f["full_content"][:100000] + "n[END FULL FILE]")
        if f.get("patch"):
            parts.append("nn[BEGIN PATCH]n" + f["patch"][:120000] + "n[END PATCH]")
    parts.append("nOutput must be JSON per the provided schema.")
    return "n".join(parts)

Step 6: The model call with prompt caching

from openai import AsyncOpenAI
import json

oaiclient = AsyncOpenAI()

async def run_review(enriched_files, pr_metadata, model="gpt-5-pro", effort="high"):
    user_content = build_user_message(enriched_files, pr_metadata)
    response = await oaiclient.chat.completions.create(
        model=model,
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": user_content},
        ],
        response_format={
            "type": "json_schema",
            "json_schema": {"name": "code_review", "schema": REVIEW_SCHEMA, "strict": True}
        },
        reasoning_effort=effort,      # "low" | "medium" | "high"
        temperature=0.1,
        max_completion_tokens=8000,
    )
    return json.loads(response.choices[0].message.content)

Two things worth calling out. First, reasoning_effort="high" is what unlocks GPT-5-Pro’s deep analysis mode — this costs more output tokens (reasoning tokens count against your bill) but catches subtle multi-file bugs that medium misses. For routine PRs, drop to medium and save significant cost.

Second, keep the system prompt byte-identical across calls. OpenAI’s prompt caching (source) hashes the prefix; a stable system prompt gets you a 50–90% discount on those tokens after the first call. Over a month of reviews, this saves real money — often more than the entire compute cost of your worker.

Step 7: The worker orchestration function

from arq import cron
import math

async def run_pr_review(ctx, repo: str, pr_number: int, head_sha: str, base_sha: str, installation_id: int, pr_id: int):
    token = await installation_token(installation_id)
    async with gh_client(token) as gh:
        # Guardrails: rate & spend
        if await repo_daily_cap_exceeded(repo):
            model, effort = "gpt-5.4-mini", "low"
        else:
            model, effort = route_model(repo, pr_number)

        enriched = await fetch_pr_context(gh, repo, pr_number, head_sha)
        pr_meta = {"repo": repo, "number": pr_number, "head_sha": head_sha, "base_sha": base_sha}

        patch_maps = await build_patch_maps(gh, repo, pr_number)  # new line mapping
        review = await run_review(enriched, pr_meta, model=model, effort=effort)

        findings = review.get("findings", [])
        findings = await dedupe_findings(ctx["redis"], pr_id, findings)
        await post_review(gh, repo, pr_number, findings, review.get("summary",""), review.get("verdict","comment"), patch_maps, head_sha)

def route_model(repo: str, pr_number: int):
    # stub logic: inspect PR size/labels to decide
    # return ("gpt-5-pro", "high") or ("gpt-5.4", "medium") ...
    return ("gpt-5-pro", "high")

Posting Reviews Back and Handling Edge Cases

[IMAGE_PLACEHOLDER_SECTION_4]

Getting a good JSON response from GPT-5-Pro is the easy part. Turning it into inline comments that land on the correct lines, handling PRs that push new commits mid-review, and not spamming the author with duplicate feedback — that’s where the operational engineering lives.

Mapping findings to review comments

GitHub’s review API expects comments anchored to positions in the diff, not absolute line numbers in the file. You need to translate model output (which references new-file line numbers) into diff positions.

1. Parse each file’s patch to build a map: {new_file_line: diff_position}.
2. For each finding, look up the diff position. If the referenced line isn’t in the diff, downgrade the finding to a general review comment rather than dropping it silently.
3. Group findings by file and construct the comments array for the review API call.
4. Set the review event to REQUEST_CHANGES if any finding has severity blocker, COMMENT if only major or minor, or APPROVE if findings is empty and the model verdict is approve.
5. Post a single review — never post comments one at a time, or you’ll spam the author with N email notifications.

def parse_patch_to_map(patch: str) -> dict:
    # Builds a map {new_file_line: diff_position}
    if not patch:
        return {}
    new_line = 0
    pos = 0
    mapping = {}
    for line in patch.splitlines():
        pos += 1
        if line.startswith("@@"):
            # @@ -old_start,old_count +new_start,new_count @@
            try:
                h = line.split()
                plus = [t for t in h if t.startswith("+")][0]
                start = int(plus.split(",")[0][1:])
                new_line = start - 1
            except Exception:
                pass
        elif line.startswith("+"):
            new_line += 1
            mapping[new_line] = pos
        elif line.startswith("-"):
            # deletion does not advance new file line
            continue
        else:
            new_line += 1
            mapping[new_line] = pos
    return mapping

async def build_patch_maps(client, repo, pr_number):
    r = await client.get(f"/repos/{repo}/pulls/{pr_number}/files", params={"per_page": 100})
    r.raise_for_status()
    maps = {}
    for f in r.json():
        if f.get("patch"):
            maps[f["filename"]] = parse_patch_to_map(f["patch"])
    return maps

def format_comment(finding: dict) -> str:
    sev = finding["severity"].upper()
    cat = finding["category"]
    body = finding["comment"].strip()
    fix = finding.get("suggested_fix")
    conf = finding.get("confidence")
    parts = [f"[{sev} • {cat}] {body}"]
    if fix:
        parts.append("nnSuggested fix:n```n" + fix.strip() + "n```")
    if conf is not None:
        parts.append(f"nnConfidence: {int(conf*100)}%")
    return "n".join(parts)

async def post_review(client, repo, pr_number, findings, summary, verdict, patch_maps, head_sha):
    comments = []
    orphan_findings = []

    for f in findings:
        pos = patch_maps.get(f["file"], {}).get(f["line"])
        if pos is None:
            orphan_findings.append(f)
            continue
        body = format_comment(f)
        comments.append({
            "path": f["file"],
            "position": pos,
            "body": body,
        })

    review_body = summary or ""
    if orphan_findings:
        review_body += "nn---nn**Additional findings on unchanged lines:**n"
        review_body += "n".join(f"- `{f['file']}` L{f['line']}: {f['comment']}" for f in orphan_findings)

    event_map = {"approve": "APPROVE", "comment": "COMMENT", "request_changes": "REQUEST_CHANGES"}
    payload = {
        "commit_id": head_sha,
        "body": review_body.strip(),
        "event": event_map.get(verdict, "COMMENT"),
        "comments": comments[:50]  # GitHub limit per review
    }
    r = await client.post(f"/repos/{repo}/pulls/{pr_number}/reviews", json=payload)
    r.raise_for_status()

Deduplication across pushes

When an author pushes a fix in response to your review, GitHub fires another pull_request event with action synchronize. If you naively re-review the whole PR, you’ll re-post identical comments on unchanged lines. This is the fastest way to get your bot uninstalled.

Cache a fingerprint of every posted comment in Redis, keyed by (pr_id, file, line, content_hash). On subsequent reviews, filter findings against the cache before posting. Expire keys 30 days after the PR closes.

import hashlib

def hash_comment(text: str) -> str:
    return hashlib.sha256(text.encode("utf-8")).hexdigest()[:16]

async def dedupe_findings(redis, pr_id, findings):
    fresh = []
    for f in findings:
        key = f"seen:{pr_id}:{f['file']}:{f['line']}:{hash_comment(f['comment'])}"
        if not await redis.exists(key):
            await redis.set(key, "1", ex=60*60*24*30)
            fresh.append(f)
    return fresh

Large PR strategy

PRs over roughly 2000 changed lines break naive review approaches. Even with GPT-5-Pro’s 400K context, a giant diff produces shallow reviews because attention gets diluted. Instead, chunk by logical unit:

• Group changed files by directory and language.
• Run separate model calls per chunk (each getting the full system prompt but only its slice of the diff).
• Run a final “meta-review” pass with GPT-5-Pro on just the summaries from each chunk to catch cross-chunk architectural issues.
• Merge findings before posting.

This map-reduce pattern costs more per PR ($2-5 for a large refactor versus $0.50 for a single-pass review) but produces reviews that hold up on 5000-line changes. For most teams, tag such PRs with a large-review label and route only those through the expensive path. For the engineering trade-offs behind this approach, see our analysis in How to Build a Research Assistant with Claude Code in 2026: Step-by-Step, which breaks down the cost-vs-quality decisions in detail.

Rate limits and cost caps

Two failure modes will bite you within the first month if you don’t preempt them. First, a burst of activity — someone opens 40 PRs in an hour during a big refactor — will hammer both GitHub’s API and OpenAI’s rate limits. Use a token bucket in Redis with a conservative ceiling (say, 10 concurrent reviews) and a queue for the overflow.

Second, cost runaway. Someone force-pushes a 50MB accidentally-committed dataset, your bot reviews it, and you eat a $200 bill from a single PR. Preempt with hard limits:

• Reject any file whose diff exceeds 100KB (probably data, not code).
• Cap total tokens per PR review at 200K input.
• Track daily spend per repo in Redis; when a repo exceeds a configured cap ($10/day is a sane default for most teams), route to gpt-5.4-mini and add a note to the review body.
• Set a hard org-wide monthly cap in the OpenAI dashboard.

Guardrail	Default	Rationale
Max concurrent reviews	10	Prevents API burst failures
Max input tokens per PR	200K	Controls worst-case spend
Max diff size per file	100KB	Skips data/noise
Daily spend per repo	$10	Budget isolation

Security, Privacy, and Compliance

[IMAGE_PLACEHOLDER_SECTION_5]

Enterprise adoption hinges on doing security and privacy correctly. Treat your bot like any production system that touches customer code and secrets.

Secret management

• Store GitHub App private keys in a managed KMS (AWS KMS, GCP KMS, HashiCorp Vault). Rotate quarterly.
• Never log request bodies or model prompts verbatim. Redact secrets via regex filters (API keys, JWTs, connection strings).
• Encrypt at rest: Redis with TLS; disk encryption on the worker hosts.

Data minimization

• Send only the minimal context needed to the model. Obfuscate secrets in diffs (e.g., mask .env changes) before prompting.
• Respect repository compliance labels: skip repos labeled pii-restricted or export-controlled by default unless approved.

Access control

• Scope the GitHub App to read-only for contents and write-only for pull_request reviews. Avoid repo admin permissions.
• Allow opt-out via a .ai-review.yml config file per repo or per directory.

Audit and compliance

• Emit an audit log entry for: webhook received, model selected, token estimate, review posted, cost charged.
• Retain logs for 90 days; scrub payloads to avoid source code storage in logs.

Testing and Evaluation: Getting to Trustworthy Reviews

[IMAGE_PLACEHOLDER_SECTION_6]

You won’t earn developer trust without repeatable evaluation. Build a small, representative test suite of PRs and measure signal vs. noise.

Golden PR corpus

• Select 50–100 historical PRs across services and languages.
• Annotate expected findings (bugs/security/performance/test) and true/false positives from prior human reviews.
• Include large refactors, small tweaks, and test-only changes.

Metrics that matter

• Precision@Top-10 findings (what fraction of posted comments are accepted by humans).
• False-positive rate (comments closed without action).
• Time-to-first-review (p50/p95 latency from webhook to review posted).
• Cost per PR and cost per accepted finding.

CI test harness

# .github/workflows/ai-review-ci.yml
name: AI Review CI
on:
  workflow_dispatch: {}
jobs:
  eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: "3.12" }
      - run: pip install -r requirements.txt
      - run: pytest tests/eval --maxfail=1 -q

Local dry-runs

Add a CLI mode to run the reviewer locally on a branch diff and print the JSON findings before posting. This shortens the feedback loop for prompt tweaks.

if __name__ == "__main__":
    import argparse, asyncio
    ap = argparse.ArgumentParser()
    ap.add_argument("--repo", required=True)
    ap.add_argument("--pr", type=int, required=True)
    ap.add_argument("--installation", type=int, required=True)
    args = ap.parse_args()

    async def main():
        token = await installation_token(args.installation)
        async with gh_client(token) as gh:
            pr = await gh.get(f"/repos/{args.repo}/pulls/{args.pr}")
            pr.raise_for_status()
            data = pr.json()
            enriched = await fetch_pr_context(gh, args.repo, args.pr, data["head"]["sha"])
            pr_meta = {"repo": args.repo, "number": args.pr, "head_sha": data["head"]["sha"], "base_sha": data["base"]["sha"]}
            review = await run_review(enriched, pr_meta, model="gpt-5.4", effort="medium")
            print(json.dumps(review, indent=2))

    asyncio.run(main())

Deployment, Operations, and Observability

[IMAGE_PLACEHOLDER_SECTION_7]

Containerization

# Dockerfile
FROM python:3.12-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY . .
ENV PYTHONUNBUFFERED=1
EXPOSE 8080
CMD ["uvicorn", "app:app", "--host", "0.0.0.0", "--port", "8080"]

Runtime configuration

• Set OPENAI_API_KEY, GITHUB_APP_ID, GITHUB_PRIVATE_KEY, GITHUB_WEBHOOK_SECRET, REDIS_URL.
• For Fly.io: scale to 1–2 instances for HA; enable volumes for buffer if needed.

Observability

• Instrument with OpenTelemetry: span around model calls; include token counts and latency.
• Prometheus counters: reviews_total, findings_total{severity}, token_input_total, token_output_total, cost_usd_total.
• Alerts: queue_depth > threshold, error_rate > 5% for 5 min, cost_daily_repo_cap_exceeded.

Disaster readiness

• Fail-open vs. fail-closed: default to fail-open (don’t block merges) unless security-review label is present.
• Feature flag to disable posting while still logging and evaluating.

Performance tuning

• Cache enriched full file contents keyed by sha:path to avoid redundant GitHub calls on synchronize events.
• Compress prompts (remove unchanged file sections) and prefer per-file reviews for large PRs.
• Turn off reasoning mode for test-only PRs.

Advanced Techniques and Extensions

[IMAGE_PLACEHOLDER_SECTION_8]

Policy-as-code with a config file

Support a repo-level .ai-review.yml to customize behavior:

# .ai-review.yml
ignore:
  - "docs/**"
  - "**/*.snap"
labels:
  security-escalate: ["auth/**", "payments/**"]
severity_threshold: "major"   # "minor" | "major" | "blocker"
linters:
  - "ruff"
  - "eslint"
post:
  approve_when_empty: true

Integrating static analyzers and linters

• Run ruff, eslint, bandit, semgrep before the model call.
• Attach their outputs to the prompt as “existing diagnostics” to reduce duplicated findings.
• Optionally upload SARIF to GitHub code scanning in parallel for long-term visibility.

Multilingual and monorepo support

• Detect language per file; add language-specific guidance to the prompt.
• For monorepos, include OWNERS mappings in the prompt to better route severity.

Auto-fix mode (opt-in)

• For minor severity with simple edits, open a bot commit or a follow-up PR with the suggested fix.
• Guard with branch protection and require human approval for auto-fixes on protected branches.

Routing to alternate providers

• Implement an abstraction: ModelClient.run(prompt) -> Review with adapters for GPT-5, Claude, Gemini.
• Use cost-aware routing: choose cheapest model that meets a confidence threshold on a calibration set.

Comparing GPT-5-Pro Against Other 2026 Options

[IMAGE_PLACEHOLDER_SECTION_9]

The honest question: does GPT-5-Pro actually justify its cost for code review versus alternatives? Here’s how the current top contenders stack up on real-world review tasks, based on published benchmarks and internal testing on a corpus of 400 open-source PRs from 2025-2026.

Model	Input / Output ($/1M)	Context	SWE-bench Verified	Reasoning latency	Best for
gpt-5-pro	$15 / $120	400K	~78%	30-60s	Critical services, security-sensitive code
gpt-5.5	$5 / $30	1.05M	~74%	15-30s	Large monorepo reviews, high volume
gpt-5.3-codex	$3 / $15	512K	~72%	10-20s	Language-idiom-heavy reviews, refactors
gpt-5.4	$2.50 / $20	400K	~70%	8-15s	Everyday review workhorse
claude-opus-4.7	$5 / $25	500K	~76%	20-40s	Long-context reviews, prose-heavy findings
claude-sonnet-4.6	$3 / $15	500K	~71%	6-12s	Balanced quality/cost default
gemini-3.1-pro-preview	$2 / $12	1M	~68%	10-20s	Absolute-scale monorepo scans

Benchmark scores are directional, not gospel. SWE-bench Verified measures autonomous bug-fixing, which correlates with review quality but isn’t identical. In practice on real PRs, three patterns emerged:

GPT-5-Pro’s edge is in multi-file reasoning. When a change to file A breaks an invariant assumed in file B two directories away, GPT-5-Pro catches it at roughly 2x the rate of gpt-5.4 and claude-sonnet-4.6. The extended reasoning mode is doing real work. On single-file reviews, the gap collapses to margin-of-error.

Claude Opus 4.7 writes better prose findings. The comments Claude produces read like they came from a thoughtful staff engineer — clear structure, appropriate hedging, actionable suggestions. GPT-5 still wins on correctness in complex multi-file interactions, but if developer buy-in is your bottleneck, prose quality matters.

Token economics determine viability at scale. Without routing and caching, costs spiral. With the tiered routing in this guide and prompt caching enabled, most teams land below $500/month for 1,000–1,200 PRs, with Pro reserved for security-sensitive or large diffs.

Troubleshooting and Common Pitfalls

[IMAGE_PLACEHOLDER_SECTION_10]

Comments misaligned to the wrong lines

• Ensure you map to positions from the patch, not absolute lines. Deleted or context-only lines don’t increment new line numbers equally.
• If a finding references a line outside the diff hunk, degrade gracefully to a general review comment.

Model outputs malformed JSON

• Use response_format: json_schema with strict: true.
• Set temperature=0.1 and include “Return only JSON” in the system prompt.

Too many nitpicks or style comments

• State clearly: “Do not comment on style covered by linters.” Provide linter diagnostics in the prompt to de-duplicate.
• Filter findings below a configured severity threshold before posting.

Costs spiking unexpectedly

• Implement per-repo daily caps and a global monthly cap.
• Short-circuit review for PRs only touching generated/code or documentation.

Latency too high

• Lower reasoning_effort to medium for non-critical PRs.
• Parallelize per-chunk reviews; cap concurrency to avoid rate limits.

Related Articles

• → How to Build a Code Review Bot with Claude Sonnet 4.6 in 2026: Step-by-Step
• → How to Build an AI Agent with GPT-5 Pro in 2026: Step-by-Step
• → How to Build a Research Assistant with Claude Code in 2026: Step-by-Step
• → Codex Workflow Automation Masterclass: 30 Production-Ready Prompts for Building Multi-Step Pipelines, Scheduled Reports, and Cross-Platform Integrations

Useful Links

• GitHub Webhooks: About webhooks
• Authenticating with a GitHub App
• GitHub REST API: Pull Request Reviews
• GitHub REST API: List pull request files
• OpenAI Prompt Caching
• OpenAI Structured Outputs
• OWASP Top Ten
• Semgrep: Static analysis
• GitHub Code Scanning (SARIF)
• The Twelve-Factor App

Frequently Asked Questions

What makes GPT-5-Pro better than cheaper models for code review?

GPT-5-Pro's extended reasoning mode and 400K context window enable multi-file analysis that catches cross-file bugs — like a function returning None in one branch while three callers still expect a dict. Cheaper models like GPT-5-mini review diffs in isolation and routinely miss these subtle, high-impact issues.

When should I route PRs to GPT-5-mini instead of GPT-5-Pro?

Use GPT-5-mini or Claude Haiku 4.5 for solo projects, low-risk repos, or PRs touching only documentation and configuration files. Reserve GPT-5-Pro for critical services where a missed vulnerability or logic error carries significant financial or security consequences.

How do you handle large pull requests that exceed context limits?

For PRs exceeding roughly 2,000 changed lines, fetch diffs per-file rather than as one unified diff. This enables chunked review, keeps token usage predictable, and prevents attention dilution that occurs when a 10,000-line refactor is stuffed into a single prompt.

Why is context enrichment beyond the raw diff so important?

A diff alone omits surrounding function bodies, imports, and callers. Without that context, the model cannot detect cross-file issues. Enriching each changed function with its full body and the file's top-level exports is what separates a useful bot from one that generates shallow, obvious comments.

What structured output format does the GPT-5-Pro API return?

The bot requests a JSON array of findings, each containing file, line number, severity, category, and comment fields. Using OpenAI's structured outputs feature enforces this schema at the API level, eliminating brittle regex parsing and ensuring comments can be posted directly as inline GitHub PR annotations.

Where can you deploy this Python bot service in production?

The service is packaged as a Docker container and can be deployed on Fly.io, Railway, or any Docker-compatible host. The guide also covers operational concerns including GitHub HMAC signature verification, rate limit handling, cost caps, and prompt caching to keep the service stable and cost-efficient.