All projects
Multi-Agent Market Research Analyst
A three-agent LangGraph pipeline that turns market questions into citation-backed theme maps, and an eval harness to prove it doesn't make things up.
Spring 2026 · CMPE 258 Deep Learning, SJSU2-person team · I led retrieval, the agents, the LLM client, evals and the API
View source on GitHubLangGraph
FAISS
MiniLM-L6-v2
FastAPI
Langfuse
Pydantic
Python
48×higher entity F1 than a zero-shot baseline (0.337 vs. 0.007)
8.0% → 5.3%corpus-level hallucination with retrieval + Critic
4 LLMsbenchmarked on 55 hand-labeled queries, cost and latency included
The problem
Analysts ask questions like "map the key players in agentic AI platforms" and need answers backed by evidence. General chatbots produce plausible market maps but invent companies; pure retrieval returns snippets without synthesizing themes or enforcing citations.
The goal was an evaluation-first system: every stage instrumented, and every claim traceable to a document in a controlled 195-document corpus (startups, YC pages, tech press, newsletters).
How it works
- 1ScoutEmbeds the query with MiniLM-L6-v2 and pulls the top-8 documents from a local FAISS index. Model-agnostic, runs on CPU, no API cost.
- 2MapperLLM clusters companies from the retrieved snippets into named themes with explicit citation titles, as strict JSON (one retry on malformed output).
- 3CriticRe-reads the same documents and removes any company or theme without textual support, enforcing the same schema.
What I built
- Built the corpus scraper and FAISS ingest pipeline.
- Led the Scout, Mapper and Critic agents and wrote the unified LLM client: one invoke() surface with a per-model pricing table, token accounting, JSON retry and Langfuse spans.
- Led the evaluation framework (entity precision/recall/F1, hallucination rate, theme coverage and purity, cost, latency) and the report generator.
- Led the FastAPI backend and web UI, including a parallel multi-model comparison endpoint.
Results
| Model | Entity F1 | Recall | Hallucination | Latency | Eval cost |
|---|---|---|---|---|---|
| GPT-5.4 | 0.302 | 0.544 | 3.9% | 14.7 s | $1.57 |
| Gemini 2.0 Flash | 0.292 | 0.539 | 6.6% | 30.1 s | $0.00 |
| Nemotron | 0.263 | 0.522 | 1.6% | 50.0 s | $0.00 |
| Llama 3.3 70B | 0.009 | 0.015 | 98.2% | 0.1 s | $0.003 |
Ablation (Nemotron, 20 queries): removing retrieved context drops entity F1 from 0.337 to 0.007 and raises hallucination from 5.3% to 8.0%.
What I learned
- Retrieval is doing the heavy lifting. Without it, the model names well-known brands from memory that aren't in the corpus: on one query, 12 of 13 zero-shot companies were hallucinated, versus 0 of 9 for the full pipeline.
- Free models are competitive. Gemini matched GPT-5.4's recall at zero cost, and Nemotron had the lowest hallucination rate, trading latency for grounding.
- Open-weight models are prompt-sensitive. Llama 3.3's near-zero F1 came from unreliable structured JSON output, not retrieval, which is a prompt/format problem rather than a model-quality verdict.
- Per-agent Langfuse traces gave real cost and latency numbers for the tables and made malformed-JSON and context-overflow failures easy to debug.
What I'd do next
- Alias resolution for entity matching (substring matching inflates recall).
- Semantic theme matching; lexical matching under-counts coverage.
- Few-shot prompting or a fine-tuned extraction layer for open-weight models.
- A larger corpus and response caching for cheaper eval sweeps.