Richy Koshy

MAX AIGrounded answers from the company’s own documents, with citations.

PRMI’s mortgage staff and its branch partners’ loan officers ask questions of the company’s guidelines, training and marketing material. MAX AI answers from those documents alone, cites the page, and says so when they don’t hold the answer. I owned the Python backend.

Role
Full technical ownership, including the Python backend across all three services
Team
A front-end developer built the React console
Impact
PRMI’s primary knowledge retrieval tool
Stack
  • Python
  • FastAPI
  • Azure OpenAI
  • Azure AI Search
  • Durable Functions
  • LangChain
  • tiktoken
  • SQL Server
  • Docker
  • Azure Container Registry
Problem

A regulated lender can’t act on an answer it can’t trace. A model’s own knowledge goes stale the day a guideline changes, can’t point to a source, and invents fluent answers when it doesn’t know.

And staff and partners may each see only the documents meant for them.

Approach

Retrieval instead of fine-tuning: documents live in a search index, one per project, and every answer is grounded in retrieved passages that carry their file and page.

Search is hybrid, keyword and vector together, then reranked; when a question’s first results come back weak, it is retried with HyDE.

The models run on Azure OpenAI inside PRMI’s own Azure tenant, and Entra ID groups decide which projects each role can query.

I built the loan-matrix dataset with PRMI’s domain experts and tuned retrieval and model settings with Azure’s RAG evaluation tooling.

Outcome
Shipped as PRMI’s primary knowledge retrieval tool: grounded, cited answers over the company’s own documents, for internal staff and external partners.
Next
Parallel ingestion, one task per file with its own retries and change detection by content; many-to-many links between projects and indexes; and an automated evaluation harness that runs on every change.
Write path: batch, Durable Functions Read path: interactive, FastAPI starts calls log vectors, in batches question top-k query chunks HyDE retry if recall is low tokens Blob container pdf, docx, pptx, mp4 HTTP starter validates, returns 202 Orchestrator replayed generator Activity all real work, no timeout Progress log checkpoint per file Per-file pipeline skip if indexed ✓ → read pages → embed → upsert Azure AI Search, one index per project BM25 keywords + HNSW vectors (1536-d, cosine) semantic reranker on top ← the only shared contract React console reads the stream live Chat API validate, pick model Hybrid retrieve rerank, top-k Azure OpenAI, GPT-4o grounded prompt, temp 0, stream NDJSON stream: one JSON object per line frame 0: citations + metadata → token frames → final frame: follow-up questions Write path: batch, Durable Functions starts calls log vectors, in batches Blob container pdf, docx, pptx, mp4 HTTP starter validates, returns 202 Orchestrator replayed generator Activity all real work, no timeout Per-file pipeline skip if indexed ✓ → read pages → embed → upsert Progress log checkpoint per file Azure AI Search one index per project BM25 + HNSW vectors (1536-d, cosine) semantic reranker on top the only shared contract Read path: interactive, FastAPI question top-k tokens React console reads the stream live Chat API validate, pick model Hybrid retrieve HyDE retry if recall is low rerank, top-k Azure OpenAI, GPT-4o grounded prompt, temp 0, stream NDJSON stream frame 0: citations + metadata → token frames → final frame: follow-up questions
Write path on top, read path on the bottom. The search index in the middle is the only thing the two halves share.

Why retrieval, not a bigger prompt or a fine-tune

A language model’s knowledge is frozen in its weights: a mortgage guideline revised last Tuesday cannot be in them. It has no provenance: an answer is a blend of billions of gradient updates with no pointer back to a source.

And when it does not know, it fabricates fluently. For a regulated lender, each of those three is disqualifying on its own. Answers must be current, cited, and able to refuse.

Fine-tuning solves none of this economically. Re-indexing one changed PDF takes seconds; a fine-tune takes hours and a redeploy.

Weights cannot point a compliance officer at a source line. Retrieval can be filtered per user at query time; weights cannot be un-learned per user.

So the design moves facts out of the model and into a search index, and converts one open-ended generation problem into two smaller, separately debuggable ones:

  • retrieval (did the right chunk surface?)
  • reading comprehension (did the model use it faithfully?).

When an answer is wrong, you can tell which half failed. That separability drives everything below.

How a PDF becomes searchable chunks with page-level citations

The durable trio. A single HTTP function cannot index a corpus: the work outlives any HTTP timeout, and at scale a crash mid-run is a certainty, not a possibility. Durable Functions splits accept, coordinate, and work.

An async HTTP starter validates the request and returns 202 Accepted with a status-polling URL in milliseconds, the asynchronous request-reply pattern.

The orchestrator is a replayed Python generator that describes the workflow and nothing else. The activity does all the real work, with the function timeout removed entirely.

  • The per-file pipeline:

    • download and detect encoding
    • dispatch by file type to the right loader
    • read page by page, keeping each passage’s file and page for citations
    • embed with ada-002 into 1,536-dimension vectors
    • upsert in batches of 1,000, the API limit.

  • Idempotent by construction: every file writes a success or failure row to a progress log, and each run starts by reading the successes back and skipping them.

    The progress log is the checkpoint; a crashed run is resumed by re-invoking the endpoint.

  • Per-file failure isolation: the unit of failure is one document, not the batch. One bad PDF cannot abort a five-thousand-file run, and the exception text is persisted so triage means a query, not grepping logs.

  • The corpus is not just text: every PDF page is rasterized to a PNG so the UI can show the exact cited page.

    Training videos are frame-sampled with OpenCV, then near-duplicates are dropped by perceptual hashing, so a 40-minute recording yields a handful of distinct slides instead of 72,000 frames.

    Audio runs through the Azure Speech batch API behind a polling loop.

Retrieve, ground, stream

Retrieval is hybrid, in three layers

BM25 over a lemmatizing English analyzer catches exact identifiers that embeddings blur: program names, form numbers. An HNSW vector index (1,536-d, cosine) catches paraphrase that keywords miss entirely.

A semantic reranker then re-scores the top candidates with a cross-encoder: too expensive for the whole index, affordable for fifty. Recall is stage one’s job; precision is stage two’s.

HyDE closes the gap between short questions and long documents

Queries and documents sit in different regions of embedding space, not because they differ in meaning but because they differ in register and length: eight words of telegraphic shorthand versus a thousand characters of formal prose.

So instead of embedding the question, an LLM drafts a plausible answer and that gets embedded.

The draft is wrong on facts but right on form, so its vector lands in the document neighborhood and retrieves the real chunks the fake one imitates.

The fabricated text is thrown away; only its position in space is used. It costs one extra completion, so it runs only as a fallback: when the first hybrid pass comes back weak, the question is retried with HyDE, and easy questions never pay for it.

Grounding is a data-formatting decision as much as a prompting one

Each retrieved chunk is prefixed with its own file and page label inside the context window, which is the only reason the model can emit citations like [guide.pdf#page=12].

The system prompt constrains answers to the provided sources and mandates refusal when they do not contain the answer.

Generation runs at temperature 0 with a hard token ceiling: for a compliance-facing tool, determinism and a bounded cost per call beat variety.

Streaming is NDJSON

One complete JSON object per line over ordinary chunked HTTP.

A single JSON array cannot be parsed until its closing bracket arrives; SSE imposes framing and reconnection semantics this app does not need; WebSockets add a stateful channel for what is a one-shot exchange.

NDJSON gives incremental parseability with zero protocol overhead. Frame 0 carries the citations and model metadata before the first token, so the UI paints the sources panel while generation is still in flight.

Token frames follow, and the final frame carries suggested follow-up questions, extracted from the stream by a small state machine so its delimiters never leak into the visible answer.

Three more choices, each with a reason

A container because of OCR.

Local PDF partitioning and table extraction use the unstructured library, which needs two system packages: poppler to rasterize pages and Tesseract to read them.

Installed over SSH, they don’t last: the host keeps only /home across restarts, so they vanish on every redeploy or scale-out.

Baking them into a Docker image served from Azure Container Registry makes every instance identical.

One search index per project.
Multi-tenancy by physical isolation, with a project-to-index map in SQL. Simple to reason about, and nothing to filter at query time.
Caching built, then left off.

I built two cache layers and measured their hit rate, then kept caching off in production.

A plausible but wrong cached answer to a mortgage-guideline question costs more than the tokens it saves.

Two halves, opposite performance profiles, one shared contract

Ingestion is throughput-bound: CPU-heavy, tolerant of hours of latency, and it must survive restarts. Chat is latency-bound: almost pure network I/O, stateless, and judged on time to first token.

Coupling them into one process would force one set of trade-offs onto both, so they ship as separate deployables: a Durable Functions app with unlimited timeout for the write path, and an always-warm FastAPI service for the read path.

A third Function App carries the utility surface: auth, projects, users, session history, feedback.

Provenance, stated plainly: the platform grew out of Microsoft’s open-source RAG reference architecture.

The bespoke engineering on top is:

  • the HyDE retrieval path
  • the per-project prompt and index registry
  • the incremental re-index driven by the progress log
  • the multimodal loaders
  • the NDJSON streaming protocol with in-band follow-up questions.

Shipped as PRMI’s primary knowledge retrieval tool

Grounded, cited answers over the organization’s own corpus.

What I’d do next: parallel ingestion, shared indexes, automated evaluation

  • Fan-out/fan-in ingestion: list the files in the orchestrator and schedule one activity per file, each retried on its own, with changes detected by content rather than by file name, turning a serial hours-long run into parallel work.
  • Many-to-many links between projects and indexes, so one document set can serve several teams without being indexed twice.
  • An automated retrieval evaluation harness that runs on every change: a labelled question set with recall@k tracking, so top-k and index parameters are tuned against measurement, not anecdote.
  • Structured outputs for the follow-up channel: replace in-band text delimiters with a tool call, so the transport stops depending on the model’s punctuation.