dsearch — Full-disk semantic search

▶ Quick Start

dsearch "pension fund certificate"                       # fast terminal results
dsearch -w "invoice Q3"                                  # sortable HTML table in browser
dsearch -CLexpand-CLrank-CLanswer "why was X delayed"    # full AI answer with citations
dsearch -CLexpand-CLrank-CLanswer -deep "query"          # deep multi-hop retrieval
dsearch2 "ασφαλιστικό ταμείο"                           # v2 hybrid retriever (RRF + TF)

Explore

⬡

Interactive Architecture

Drag-and-drop pipeline diagram. Click any node to explore components, data flow, and design decisions.

Interactive · click nodes

⚙

Setup Guide

Full installation walkthrough — venvs, ChromaDB indexing, Cohere API, Recoll BM25, and config file setup.

Step-by-step guide

▶

Script Flow Diagram

Annotated code-level flow of the answer pipeline. Click each node to jump to the corresponding source line.

Clickable · links to code

📄

Project Report

Benchmark results, model comparison, design decisions, and experiment log. MRR, Hit@1, Hit@5 across 5 models.

Full report

How It Works

Step 1 — Claude

Query understanding: expands the user query into core terms, variants, entities, and a HyDE hypothetical answer to bridge vocabulary gaps.

Step 2 — Cohere + ChromaDB

Embeds all query variants with embed-multilingual-v3.0, queries ~1M chunks, applies MMR diversity. -deep loops up to 3×.

Step 3 — Extraction

Reads full text from top 100 files: PDF (pdftotext), Word (antiword), Office XML (zipfile), and OCR sidecars for scanned images.

Step 4 — 10 Parallel Agents

10 Claude agents rank 10 files each (0–10 score). Run in parallel via ThreadPoolExecutor. Ranking only — no answers yet.

Step 5 — Merge

Aggregate agent scores, stem-dedup (removes email thread duplicates), select top 15 files for synthesis.

Step 6 — Final Answer

Claude reads the full original text of top 15 files and synthesises a cited answer with [filename] inline references. Subscription-based — no per-token billing.

dsearch2 — Hybrid Retriever (v2)

dsearch2 "pension fund certificate"           # vector + TF + RRF hybrid
dsearch2 --json "insurance claim" | jq .      # structured JSON output
dsearch2 --debug "ασφαλιστικό"               # show alias expansion + fetch stats
dsearch2-answer "why was X delayed"           # passage-first RAG answer

RRF Fusion

Combines vector rank + TF lexical rank via Reciprocal Rank Fusion (k=60). No calibration needed between similarity scales.

Alias Expansion

Built-in bilingual synonym map (Greek + English). Sliding-window longest-match phrase expansion — no Claude call needed.

Source Tiers A–D

Each chunk tagged with credibility tier: A = official/government, B = professional, C = personal notes, D = ambiguous. Used for RRF tiebreaking and answer citations.

HNSW Candidate Floor

max(n_results × 3, 1000) minimum fetch pool — prevents HNSW recall degradation on 950K+ chunk collections.

Benchmark Results

Model	MRR	Hit@1	Hit@5
BM25 + Cohere hybrid	0.94	0.85	0.99
Cohere embed-v3	0.91	0.82	0.97
E5-large	0.87	0.76	0.95
E5-base	0.84	0.73	0.93
GTE	0.81	0.70	0.91
BGE-M3	0.78	0.66	0.89
BM25 only	0.71	0.58	0.85