Table of Contents

What Is WarpGrep?

WarpGrep is a hybrid search engine for source code that blends traditional text and structural querying with semantic understanding powered by embeddings and machine learning. Unlike a conventional grep that scans bytes, WarpGrep:

• Builds a multi-modal index: lexical tokens, AST features, symbol tables, references, and vector embeddings.
• Executes queries using a cost-based planner that chooses the fastest path—literal, regex, structural, or semantic—often combining them.
• Reranks results using code-aware signals: symbol centrality, test coverage proximity, code ownership, and recent changes.
• Provides a streamlined developer experience with a fast CLI, editor integrations, HTTP API, and CI plugins.

In short, WarpGrep turns AI code search from a novelty into an everyday superpower. It helps you find the right code, faster—whether you know the keyword, the intent, or just a vague description of behavior.

Why 15x Faster AI Code Search

“15x faster” is not a marketing flourish; it’s a target derived from real-world constraints: how long developers wait for answers determines how often they ask better questions. WarpGrep achieves speedups by combining:

• Incremental indexing that updates in milliseconds as files change, avoiding cold starts.
• Parallel I/O and SIMD-accelerated tokenization for raw throughput.
• Structure-aware pruning that narrows scanning to relevant subtrees and symbol scopes.
• Vector pre-filtering to shortlist candidates before expensive semantic reranking.
• Plan fusion that composes multiple filters (e.g., “only in Python, changed last 30 days, implements interface X”).

In head-to-head tasks like “find every function performing OAuth token validation across a monorepo” or “locate call sites vulnerable to SQL injection,” WarpGrep consistently returns high-quality results in seconds instead of minutes. This is especially true when queries are ambiguous or natural-language heavy, where semantic hints dramatically cut search space.

Benchmarks vary by repository size, hardware, and cache warmth. See Benchmarking Methodology for reproducible steps. Also see Benchmarks.

Architecture Overview

WarpGrep’s internal architecture comprises five subsystems that work together to support fast, relevant AI code search:

1. Ingestion: Watches your workspace, reads Git history, and normalizes inputs (file content, metadata, ownership).
2. Parsing: Uses language frontends (e.g., Tree-sitter) to build ASTs and symbol graphs with errors tolerated.
3. Indexing: Stores multi-modal artifacts—postings lists, n-grams, AST features, dependency graphs, and embeddings—into tiered stores.
4. Query Planning: Translates queries into a DAG of operators (regex, structural, semantic, metadata) and picks a cost-optimal plan.
5. Ranking: Combines lexical BM25, vector similarity, cross-encoder signals, and code heuristics to produce final results.

Key Design Principles

• Local-first: Works offline or on air-gapped networks. Cloud optional.
• Deterministic fallbacks: If a semantic model is unavailable, WarpGrep still returns lexical/structural results fast.
• Composable: Every feature is an operator you can combine declaratively.
• Observability: Queries and indexing are traceable; performance regressions are diagnosable.

Install and Quickstart

Get WarpGrep running in minutes. The following examples assume a Unix-like environment, but Windows is supported via native binaries and WSL.

Prerequisites

• 64-bit CPU with SSE4.2/AVX2 support (optional GPU for embedding acceleration).
• Git installed and accessible in PATH.
• ~2–4 GB free disk space for indexes on a medium repository (scales with code size).
• Optional: Docker for containerized deployment.

Install Options

# macOS (arm64)
curl -L https://example.org/warpgrep/releases/latest/warpgrep-darwin-arm64 -o /usr/local/bin/warpgrep
chmod +x /usr/local/bin/warpgrep

# Linux (x86_64)
curl -L https://example.org/warpgrep/releases/latest/warpgrep-linux-x86_64 -o /usr/local/bin/warpgrep
chmod +x /usr/local/bin/warpgrep

# Verify
warpgrep --version
      

# Homebrew
brew tap warpgrep/tap
brew install warpgrep

# Debian/Ubuntu
curl -fsSL https://example.org/warpgrep/key.gpg | sudo gpg --dearmor -o /usr/share/keyrings/warpgrep.gpg
echo "deb [signed-by=/usr/share/keyrings/warpgrep.gpg] https://example.org/warpgrep/apt stable main" | sudo tee /etc/apt/sources.list.d/warpgrep.list
sudo apt update && sudo apt install warpgrep
      

Quickstart

# 1) Initialize a project index
cd ~/code/my-repo
warpgrep init

# 2) Build the initial index (lexical + structural)
warpgrep index --languages=go,ts,py

# 3) Enable semantic embeddings (optional, recommended)
warpgrep embeddings build --model=all-MiniLM-L6-v2

# 4) Run your first query (literal)
warpgrep search "AuthService" --path=services/

# 5) Try a semantic query (natural language)
warpgrep search --semantic "validate oauth bearer token and refresh flow"

# 6) Open TUI to explore results interactively
warpgrep tui
  

Documentation quick links: Install WarpGrep, Getting Started, Semantic Mode

Indexing Your Codebase

Effective AI code search depends on building a rich, accurate index. WarpGrep’s index is multi-layered so each query operator has a fast path.

Index Layers

• Lexical: Case-folded tokens, n-grams, postings lists with positional info.
• Structural: AST-derived nodes (function defs, imports), symbol tables, reference edges.
• Metadata: File paths, languages, ownership, code coverage, churn, timestamps.
• Semantic: Text and code embeddings for fragments (e.g., functions, classes, docs).

Configuring Indexing

Place a warpgrep.yaml at the repository root or in your home directory for global defaults:

# warpgrep.yaml
project:
  name: my-repo
  root: "."
  ignore:
    - "node_modules/**"
    - "**/.git/**"
    - "**/dist/**"
languages:
  enable: ["go", "python", "typescript", "sql", "markdown"]
index:
  granularity: "function"       # or "file", "class", "block"
  min_token_length: 2
  follow_symlinks: false
  respect_gitignore: true
  parallelism: auto
  snapshot_git: true            # store commit ids for provenance
embeddings:
  provider: "local"             # or "remote"
  model: "all-MiniLM-L6-v2"
  batch_size: 256
  normalize: true
  storage: "faiss"              # or "qdrant", "milvus"
metadata:
  ownership:
    provider: "codeowners"      # map paths to teams
  coverage:
    report_glob: "coverage/**/lcov.info"
security:
  pii_redaction: true
  secrets_detector: true
observability:
  logging: "info"
  tracing: "otlp"
  metrics: "prometheus"
  

Incremental Indexing

WarpGrep watches the filesystem and Git changes. On each save or pull:

1. Dirty files are tokenized and parsed in-memory.
2. Updated fragments are re-embedded (if semantic mode on).
3. Postings and vectors are updated transactionally.
4. Caches are warmed for recently touched symbols.

# Start a watcher to keep the index hot
warpgrep watch --debounce=75ms --semantic
  

Chunking Strategy

The semantic layer works best with coherent chunks. Recommended defaults:

• Functions/methods as primary chunks.
• Classes as secondary chunks for object-oriented code.
• Docstrings and README sections as separate text chunks.
• Limit chunks to < 512 tokens for efficient embedding.

Language Support

WarpGrep supports common languages out of the box and can be extended:

• First-class: Java, Kotlin, Go, Rust, Python, TypeScript/JavaScript, C/C++, C#, Ruby, PHP, Swift.
• Data/config: SQL, GraphQL, JSON, YAML, TOML, Dockerfile.
• Docs: Markdown, reStructuredText, AsciiDoc.
• Extend via plugins with Tree-sitter grammars and symbol visitors.

See also: Indexing, Chunking, Language Plugins

Querying: From Regex to Semantics

WarpGrep exposes a query language that scales from simple literal searches to structured AST patterns and natural language. Its strength is composability.

Basic Queries

# Literal substring
warpgrep search "AuthService"

# Regex with flags
warpgrep search --regex "(?i)bearer\\s+token"

# File/path filters
warpgrep search "logger" --path="**/src/**" --exclude="**/vendor/**" --lang=go
  

Structural Queries

Use structure to find constructs independent of formatting:

# Find all Python functions named 'validate' with at least two params
warpgrep ast "def name:validate params:>=2" --lang=python

# Go: Interfaces implementing io.Reader
warpgrep ast "interface method:Read signature:([]byte) (int, error)" --lang=go

# TypeScript: Calls to fetch with { credentials: 'include' }
warpgrep ast "call:fetch arg:object(credentials:include)" --lang=typescript
  

Semantic Queries

Express intent in natural language. WarpGrep uses embeddings to shortlist candidates:

# High-level intent
warpgrep search --semantic "verify JWT signature and expiration"

# Security-focused
warpgrep search --semantic "places where user input builds SQL without parameterization"
  

Hybrid Queries

Combine operators for precision and performance:

# Natural language + filters
warpgrep search --semantic "oauth refresh token rotation"
  --lang=go --path="services/auth/**" --changed-since="30d"

# Structural + lexical constraints
warpgrep ast "func name:*Validate* returns:error?" --lang=go \
  | warpgrep search --stdin --regex "expired|revoked"

# Ownership-aware
warpgrep search "featureFlag" --owner="team-platform"
  

Interactive TUI

The TUI offers fuzzy navigation, previews, and live filtering:

warpgrep tui
# Keybindings: / search, : filters, o open file, n/p next/prev result, <tab> add filter
  

Explore: WarpGrep Query Language, Structural Search, Semantic Search

Semantic Search and Embeddings

Semantic AI code search augments keywords with meaning. WarpGrep transforms code fragments and documentation into vectors, letting it match by intent even when terms differ.

Model Choices

• General-purpose sentence encoders (e.g., MiniLM variants) for mixed code/text.
• Code-focused encoders trained on repositories for tighter alignment across languages.
• Cross-encoders for reranking to improve precision@k on the top candidates.

Workflow

1. Chunk code into stable units (functions, classes, blocks).
2. Compute embeddings offline or on-demand; store in FAISS/Qdrant/Milvus.
3. On query, embed input text, run ANN search to get k candidates.
4. Optionally rerank with a cross-encoder that compares query-candidate pairs.
5. Fuse with lexical/structural signals to form the final ranking.

Configuration Examples

# Switch to a code-specialized model and enable GPU
warpgrep embeddings configure \
  --model=code-bert-multilang \
  --device=cuda:0 \
  --store=qdrant --store-url=http://localhost:6333

# Set hybrid ranking weights
warpgrep rank configure \
  --bm25-weight=0.5 \
  --vector-weight=0.35 \
  --cross-encoder-weight=0.15
  

Domain Adapters

For specialized domains (e.g., embedded systems or quant finance), lightweight adapters can shift embeddings toward in-domain semantics without retraining from scratch. Provide a small set of Q/A pairs or code-doc pairs for contrastive fine-tuning, then:

warpgrep embeddings adapt --dataset=domain_pairs.jsonl --epochs=2 --lr=1e-5
  

Speed vs. Quality

You can dial speed up or down:

• Turn off cross-encoder reranking for minimal latency.
• Lower k (candidate count) while tightening lexical filters.
• Use smaller models for interactive, larger ones for batch jobs.

Ranking, Reranking, and Relevance

Good search isn’t only about speed; it’s about surfacing the most useful result first. WarpGrep fuses multiple signals to compute a final score for each candidate.

Signals

• Lexical: BM25F with field boosts (symbols, comments, strings).
• Semantic: Cosine similarity between query and fragment embeddings.
• Cross-Encoder: Pairwise relevance model on the top-k.
• Structure: AST depth, symbol role (definition vs. reference), API centrality.
• Freshness/Churn: Recent commits and file stability.
• Context: Co-occurrence with known tech stack or dependencies.
• Ownership: Team-relevant surfaces for triage or review.

Custom Scoring

# weights.yaml
weights:
  lexical: 0.45
  semantic: 0.30
  cross_encoder: 0.10
  structure: 0.10
  freshness: 0.05

# apply
warpgrep rank apply --config=weights.yaml
  

Evaluation

Track NDCG@k, MRR, and Success@1 on a set of known tasks. WarpGrep can run offline evals:

# eval.jsonl contains queries and gold targets
warpgrep eval --dataset=eval.jsonl --metrics=ndcg,mrr,success@1 --k=10
  

Tie evaluation to CI to detect regressions as indexes or configs change. See Relevance Tuning.

Benchmarking Methodology and Reproducibility

Transparent benchmarking is vital. This section documents how to reproduce a 15x speedup claim on representative tasks.

Hardware and Environment

• CPU: 8-core modern x86_64 with AVX2; 32 GB RAM.
• Disk: NVMe SSD with at least 1 GB/s read.
• GPU: Optional RTX 3060 for embeddings (not required for search).
• OS: Linux 5.x or macOS 13+; file system with case sensitivity recommended.

Datasets

• Monorepo A: ~5M LOC, Go + TS + Python, microservices + frontends.
• Monorepo B: ~12M LOC, Java/Kotlin backend + React/TS.
• Polyglot C: ~3M LOC, Rust + C++ + Python data pipelines.

Tasks

1. Security: “Find unparameterized SQL query construction paths.”
2. Refactor: “Locate all deprecated API call sites and nearest suggested replacements.”
3. Auth: “Identify refresh token rotation logic and tests.”
4. Infra: “Where are S3 buckets created with public ACLs?”

Procedure

1. Warm cache: run a baseline lexical query set.
2. Index with WarpGrep: lexical + structural; enable semantic vectors.
3. Compare against baseline tools: pure grep/ripgrep and a non-hybrid semantic tool.
4. Measure: end-to-end latency to first relevant result and to top-10 results.
5. Record: success@1 and NDCG@10 from a gold corpus.

Results Summary

In our measured runs, WarpGrep achieved:

• Latency to first relevant result: median 0.9s vs. 13.8s (15.3x faster) on Monorepo B Task 1.
• Latency to top-10: median 2.4s vs. 22.7s (9.5x faster) on Monorepo A Task 3.
• Success@1 improved 12–28% with semantic reranking enabled.

Actual numbers vary. To reproduce, clone the benchmark harness and datasets indicated in Benchmarks and run:

warpgrep bench run --suite=monorepo-bench --export=results.json
warpgrep bench compare --results=results.json --against=rg,baseline-semantic
  

Monorepos, Microservices, and Polyglot Code

WarpGrep is built for scale. Whether your code lives in a monorepo or a constellation of services, a few practices keep search fast and useful.

Repository Layout Tips

• Organize by domain or service; name directories consistently.
• Maintain CODEOWNERS and tags for ownership filters.
• Keep generated code out of the index or in a separate tier.

Selective Indexing

# Only index source and docs; exclude vendored code
index:
  include:
    - "src/**"
    - "services/**"
    - "packages/**"
    - "docs/**"
  exclude:
    - "**/vendor/**"
    - "**/build/**"
    - "**/generated/**"
  

Cross-Repo Search

Large organizations often need to search many repos at once. Use a workspace:

# workspace.yaml
workspace:
  name: core-platform
  repos:
    - name: auth-service
      path: "~/code/auth-service"
    - name: payments
      path: "~/code/payments"
    - name: frontends
      path: "~/code/frontends"
  embeddings:
    shared_store: "qdrant://localhost:6333/platform"

# Search across the workspace
warpgrep ws search --semantic "graceful retry with exponential backoff"
  

Language-Specific Hints

• Go: surface interfaces and implementations; module-aware imports.
• TypeScript: type-aware searches for union/intersection types.
• Python: support virtualenv/poetry metadata for dependency context.
• Java/Kotlin: symbol indexes across Gradle modules; handle Lombok-generated code gracefully.

Explore Workspaces, Multi-Repo Search, Language Guides

Editor, CLI, and CI/CD Integration

WarpGrep meets you where you work—terminal, editor, or pipeline.

CLI Snippets

# Show only definitions
warpgrep search "ConnectionFactory" --kind=definition

# Search by commit range
warpgrep search --semantic "remove deprecated API" --rev-range="v1.9..HEAD"

# Export results as JSON for automation
warpgrep search "jwt" --json --limit=50 > jwt.json
  

Editor Extensions

• VS Code: inline previews, jump-to-symbol, and natural-language command palette.
• JetBrains: gutter actions to search references across modules.
• Vim/Neovim: Telescope integration via LSP-like provider.

HTTP API

POST /v1/search
{
  "query": "sanitize user input before db write",
  "semantic": true,
  "filters": { "lang": ["python", "go"], "path": ["services/**"] },
  "limit": 25
}

# curl
curl -s http://localhost:7777/v1/search -H "Content-Type: application/json" -d @req.json
  

CI/CD

Make search a quality gate. Example: block merges if deprecated APIs remain:

# .github/workflows/search.yml
name: WarpGrep Check
on: [pull_request]
jobs:
  deprecated-apis:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: warpgrep/setup-action@v1
      - name: Build index
        run: warpgrep index
      - name: Scan for deprecated API usage
        run: |
          warpgrep ast "call:OldApi.*" --changed-since="refs/pull/${{ github.event.pull_request.number }}/merge" --json > out.json
          jq -e '.results | length == 0' out.json
  

Learn more: API Reference, Editor Integration, CI/CD

Scaling WarpGrep in Teams and the Cloud

For small teams, local-first is enough. At organizational scale, centralizing embeddings and caches reduces duplication and speeds cold starts.

Shared Services

• Central Vector Store: Qdrant or Milvus with per-repo collections and ACLs.
• Index Cache: Object storage for prebuilt indexes per commit/tag.
• API Gateway: AuthN/Z, rate limits, and observability.

Docker Compose

version: "3.8"
services:
  warpgrep:
    image: ghcr.io/warpgrep/server:latest
    ports: ["7777:7777"]
    environment:
      WG_STORE_URL: "qdrant://qdrant:6333"
      WG_OBSERVABILITY: "prometheus"
    volumes:
      - ./repos:/data/repos
  qdrant:
    image: qdrant/qdrant:latest
    ports: ["6333:6333"]
    volumes:
      - qdrant:/qdrant/storage
volumes:
  qdrant: {}
  

Kubernetes

# helm values excerpt
server:
  replicas: 3
  resources:
    requests:
      cpu: "1"
      memory: "2Gi"
    limits:
      cpu: "4"
      memory: "8Gi"
  autoscaling:
    enabled: true
    minReplicas: 3
    maxReplicas: 12
    targetCPUUtilizationPercentage: 70
vectorStore:
  type: qdrant
  url: http://qdrant:6333
  

Caching Strategies

• Pin embeddings at release tags to avoid churn.
• Precompute for hot paths (e.g., auth, payments) each night.
• Serve warmed indexes to CI runners to avoid repeated cold builds.

More: Deployment, Scaling, Caching

Security, Privacy, and Compliance

AI code search touches sensitive IP. WarpGrep includes safeguards to keep data private.

Data Residency

• Local-first indexing with no outbound calls by default.
• Optional remote embeddings use customer-managed endpoints.

Access Control

• Results filtered by repository permissions and path-level ACLs.
• Ownership-aware queries limited to authorized teams.

PII and Secret Hygiene

• PII redaction for logs and previews.
• Built-in detectors for secrets and tokens; exclude from semantic stores by default.

Auditability

• Query logs with hashed identifiers and purpose flags (dev, triage, audit).
• Exportable trails for compliance reviews.

See Security, Privacy, Compliance

Troubleshooting and Performance Tuning

If WarpGrep feels slower than expected, or results look off, use this checklist.

Index Health

• Run warpgrep diag for a summary of missing layers and warnings.
• Ensure generated code is excluded to reduce noise.
• Rebuild embeddings after major refactors (warpgrep embeddings rebuild).

Latency Tips

• Prefer hybrid queries with filters to narrow scope.
• Keep k small for ANN (e.g., 100–300) and enable quantization for large stores.
• Warm caches with a nightly job on hot subsystems.

Quality Tips

• Use a code-specialized embedding model for better alignment.
• Enable cross-encoder reranking for ambiguous natural language tasks.
• Tune weights and evaluate routinely with a gold set.

Common Errors

# Missing parser for language
Error: no frontend for 'elixir'
Fix: warpgrep lang install elixir

# Store unavailable
Error: connect ECONNREFUSED qdrant:6333
Fix: Check WG_STORE_URL and service health; fall back to local FAISS.

# High memory usage
Cause: unbounded chunk size or parallelism too high
Fix: index.granularity=function, parallelism=auto
  

More help: Troubleshooting, Performance, Diagnostics

FAQ

Is WarpGrep a drop-in replacement for ripgrep?

No. For raw text search, ripgrep is extremely fast. WarpGrep shines when you need structural or semantic understanding, cross-language intent, or consistent filters and ranking across a large codebase. Many teams use both.

What does “15x” depend on?

Speedups are largest on ambiguous, intent-heavy queries where semantics prune massive search spaces. On simple literal queries, expect parity with optimized text search.

Do I need a GPU?

No. Embeddings can be precomputed on CPU or a separate machine. Search itself is CPU-bound with fast vector stores and quantization. A GPU helps for high-throughput batch embedding.

How big is the index?

Depends on layers. Typical overhead ranges from 1–3x source size for lexical/structural data plus 0.5–1.5x for semantic vectors (with 384–768 dims and quantization).

What if my code contains secrets?

Enable secret detection and exclusion. You can also turn on PII redaction for logs and previews.

Can WarpGrep integrate with my LLM assistant?

When implementing these advanced strategies, it is often helpful to reference our deeper analysis on RAG Patterns, which explores the technical nuances of 25 ChatGPT-5.5 Prompts for Software Engineers: Code Review, Architecture Design, Debugging, and Technical Documentation in the context of modern AI workflows.

Roadmap and Best Practices

Near-Term Enhancements

• Deeper IDE integrations with inline semantic tooltips.
• AST diff-aware embeddings to update vectors even faster.
• SDAM-style adaptive scoring that learns from team feedback.

Best Practices Checklist

• Keep CODEOWNERS updated for ownership filters.
• Exclude generated and vendored code from semantic layers.
• Choose model dimensions that balance quality and latency (384–768 often best).
• Use hybrid filters (lang, path, freshness) to accelerate queries.
• Evaluate relevance regularly with a gold task set.
• Warm caches nightly and pin embeddings at release tags.
• Adopt structural queries for refactors and large-scale changes.

See Best Practices, Roadmap, Feedback

Conclusion and Next Steps

WarpGrep elevates search from a blunt instrument to a code-intelligent assistant. By unifying lexical, structural, and semantic signals, it delivers the practical promise of AI code search—reliable, explainable, and fast enough to use all day. With the right indexing strategy, hybrid queries, and relevance tuning, many teams see double-digit speedups in daily discovery and triage tasks, and up to 15x faster time-to-first-answer on complex, intent-driven searches.

Ready to try it? Start with the quickstart, enable semantic embeddings, and wire up the TUI or your editor of choice. Then measure impact using the benchmarking harness and tune weights to your domain. The payoff is compounding: faster answers lead to better questions, tighter feedback loops, and more maintainable codebases.

• Install WarpGrep
• Getting Started
• Benchmarks
• WarpGrep Query Language
• Editor Integration