Why Anvesha?
Most "search + AI" projects are built for one dataset and one use case. When the dataset changes, the project gets rewritten.
Anvesha inverts that:
- It is not a legal search engine — legal documents are just the first dataset.
- It is not an AI chatbot — answering questions is one capability among many.
- It is search infrastructure — a set of generic, replaceable components that turn any document collection into a world-class search engine.
The test of success: pointing Anvesha at a second, completely different domain should require only configuration and a small plugin — zero changes to the core.
Design principles
Everything in Anvesha must be generic · modular · extensible · replaceable · scalable · observable.
The core is a small set of interfaces; every concrete technology is a plugin behind one of them. Swap BM25 for neural retrieval — nothing else changes.
| Interface | Responsibility | Example implementations |
|---|---|---|
| Connector | Bring documents in | Web crawler, sitemap, RSS, API, bulk upload |
| Processor | One transform step | HTML→text, cleaning, dedup, chunking, metadata extraction |
| BlobStore | Raw originals | Filesystem, S3-compatible object store |
| DocumentStore | Canonical processed docs + metadata | Postgres (the single system of record) |
| SearchIndex | Make text searchable | Built from scratch: inverted index + BM25 |
| VectorStore | Embeddings + kNN | pgvector, dedicated vector DB |
| Retriever | Query → candidates | BM25, vector, hybrid |
| Ranker | Reorder candidates | Score fusion, reranker models, citation authority |
| AnswerEngine | Question → cited answer | RAG over retrieved passages |
Collections & domain packs
A Collection is a named dataset with its own configuration: which connectors feed it, which processing pipeline it runs, and which retrieval/ranking stack serves it.
A domain pack is a folder of plugins + config that a Collection references (e.g. a legal pack that extracts courts, judges, acts and citations). Domains are data — the core never changes.
Architecture at a glance
┌────────────────────────────────────────────┐
│ REST API │
│ /collections /ingest /search /ask │
└──────┬──────────────────────────┬──────────┘
│ enqueues jobs │ reads
▼ ▼
┌────────────────────────────┐ ┌────────────────────────────────┐
│ Job queue + workers │ │ Query pipeline │
│ ingest → process → index │ │ parse → retrieve → fuse → rank │
│ (each step: plugin chain) │ │ → answer (cited) │
└──────────────┬─────────────┘ └────────────────────────────────┘
▼
┌──────────────────────────────────────────────────────────────┐
│ Storage: BlobStore │ DocumentStore │ SearchIndex │ VectorStore │
└──────────────────────────────────────────────────────────────┘A document has one canonical form (in the DocumentStore); the search index, vectors and graphs are rebuildable projections of it. Ingestion is asynchronous (queue + workers) from day one; querying is synchronous. The system starts as a well-factored monolith and grows into distributed components along pre-designed seams.
Repository layout
anvesha/
core/ Document models, plugin interfaces, registry, Collection config
connectors/ Source plugins: bulk, sitemap, RSS, crawler
processors/ Pipeline stages: parse, clean, dedup, chunk, extract
stores/ BlobStore + DocumentStore (system of record)
index/ The from-scratch inverted index + BM25 ← the heart
jobs/ Postgres-backed job queue + workers
api/ FastAPI app: /collections /ingest /search /ask
packs/legal/ First domain pack: Indian legal documents
tests/ Unit tests + relevance evaluation harness
docs/adr/ Architecture Decision RecordsEach folder has its own README explaining what belongs there and when it gets built.
First dataset
Public Indian legal documents. Chosen because they are freely available, well-structured and rich in relationships — citations between judgments, courts, judges, acts and dates — which makes them a perfect stress test for metadata extraction, faceted search and citation-graph ranking.
Roadmap
Roughly three months, weekly milestones. Each week ends with something that runs. Details are tracked in issues and milestones.
| Phase | Scope |
|---|---|
| Phase 0 — Skeleton | Repo layout, core interfaces, Collection config, first connector, raw storage |
| Phase 1 — Search | Processing pipeline, tokenizer, inverted index, BM25 ranking, boolean/phrase queries, faceted search API, relevance evaluation harness |
| Phase 2 — Understanding | Embeddings, chunking, vector + hybrid retrieval, reranking |
| Phase 3 — Answers | RAG with paragraph-level citations, citation graph + authority ranking |
| Phase 4 — Genericity | A second domain pack with zero core changes |
| Phase 5 — Production | Deployment, monitoring, metrics, docs, v1.0 |
Learning in public
Anvesha doubles as a curriculum. Building it means building: web crawlers (politeness, frontiers, dedup), text processing (boilerplate removal, near-duplicate detection), a real inverted index (postings lists, compression, BM25), vector search, ranking fusion, RAG with citations, graph algorithms (PageRank-style authority on legal citations), job queues, evaluation (nDCG, MRR), observability and deployment.
Major technical decisions are recorded as Architecture Decision Records in
docs/adr/ — the reasoning is part of the project.
Contributing
The project is in its design phase, so right now the most valuable contributions are:
- Design discussions — challenge the architecture, propose alternatives.
- IR & search expertise — pointers, papers, war stories.
- Data sources — well-structured public document collections, legal or otherwise.
- Later — relevance judgments for the evaluation set, then code.
Open an issue or a discussion — everything is on the table at this stage.