Portrait of Vartika Singh

Product Manager · AI products

I turn messy workflows into AI products people actually use.

Gurugram / NCR · Open to remote

Product manager at Policybazaar, owning advisor-facing products on its B2B2C platform, from AI-assisted selling and learning to investments, renewals and onboarding. On the side, I build RAG, agentic and voice AI prototypes spec-first, to understand how AI products really behave.

Projects

Three AI projects built spec-first, and one quick prototype. I write the requirements, make the scope and guardrail calls, and design the evals. AI coding tools write most of the code. Each project links to its live demo, code and specs.

RAG · Spec-driven

The Mahabharata Guide

Ask a question about the Mahabharata and get an answer written only from the real text, with every citation traced back to the passages it came from. It is built on Kisari Mohan Ganguli's English translation, not a model reciting the epic from memory.

Why it's interesting: citations are built in code from the passage numbers the model reports, so an answer can never cite a passage that wasn't retrieved. If an answer has no valid citation, the app says "the text doesn't cover this" instead of showing it.

To try it: the app is bring-your-own-key. Paste an OpenAI, Anthropic or Gemini key when you ask your first question. The key stays in your browser tab and is never stored on the server. The first request may be slow while the free-tier backend wakes up.

  • React + TypeScript
  • FastAPI
  • BM25 + pgvector
  • Supabase
  • Reciprocal Rank Fusion
  • Kiro specs
How it works
  • Classify. An LLM labels each question factual, philosophical or ambiguous, with a confidence score. Low confidence is downgraded to ambiguous, and ambiguous questions get two short readings side by side, each cited.
  • Retrieve. Hybrid search over 5,578 passages from all 18 parvas: BM25 keyword search plus vector similarity (pgvector), merged with Reciprocal Rank Fusion because the two scores aren't on the same scale. A failed search retries with period vocabulary ("siblings" becomes "brothers and sisters").
  • Answer. The model writes only from the retrieved passages. Guardrails handle off-topic questions and prompt-injection attempts.
Product decisions I made
  • Cut scope to Q&A. A story-telling mode was specced, then removed to keep v1 focused on one thing done well.
  • Changed the architecture after measuring. Self-hosted embedding and re-ranking models needed about 703MB of memory against a 512MB free-tier limit. I moved to an external embeddings API plus pgvector instead of cutting hybrid search or paying for a bigger host. Re-ranking is deferred to v2.
  • Chose embeddings on cost and limits. Gemini's free tier would have taken about 6 days to embed the corpus. OpenAI's embeddings cost about $0.07 for all of it.
  • Asked for the API key late. The key prompt appears at the first question, not up front, so visitors see value before being asked for a paid key.
  • Tested the classifier against real questions. A hand-labeled eval set, and I relabeled rows after live-testing when the model's reading was the more defensible one.
What's next

Add a re-ranker once there's a hosting decision for it, run the labeled eval set across all three providers, and tackle modern-dilemma questions (for example, "should I buy more plants?") that need their own validated classifier.

Agentic AI · MCP · Evals

MoodMeals

A "kya khaun, batao" agent that turns a vague mood like "thaka hua hoon, kuch halka" into one concrete plan: cook at home, grab a ready-to-eat item from Instamart, or order in from Swiggy Food. It gives the reason and the price, asks at most three questions, and stops for approval.

Why agentic: the number of steps isn't known up front. If every restaurant is closed it switches to Instamart, and if your budget drops mid-run it replans. A fixed script only handles the happy path.

What the live demo is: the hosted app runs on made-up restaurants and groceries (pick "Scripted demo", no key needed), because Swiggy's sign-in only allows localhost redirects. The real-data run, in read-only dry-run mode, is in the demo video. The app contains no order, checkout or payment tool. Not affiliated with Swiggy. The first load may take a few seconds while the app wakes up.

  • Python
  • Streamlit
  • Swiggy MCP (Food + Instamart)
  • Hand-written agent loop
  • 24-scenario eval suite
How the agent is designed
  • A hand-written loop, no framework. Each turn the model picks one of four actions: call a tool, ask a question, propose a plan, or stop searching.
  • The model decides, code enforces. Code owns the three-question limit, run limits, a plan validator that checks every item and price against real tool results, and a write gate that needs approval for each action.
  • Privacy by design. The model never sees addresses or phone numbers.
Eval results, with the caveats

24 scenarios covering happy paths, missing information, contradictions, infeasible requests, tool failures, mid-run changes and safety. Scoring uses deterministic checks, not another model. Runs passing every check (gpt-5-mini):

StrategyPassing
Agent71 of 71
Agent, validator switched off67 of 71
Fixed code-only workflow16 of 24
  • The validator caught three plans that broke a rule. No strategy invented an item or wrote without approval.
  • Read it with care. It's one model with three runs per scenario. The first full run scored 65 of 71. I then fixed the prompt and code for three scenarios, so 71 of 71 is tuned, not a held-out score. The whole eval programme cost about ₹161.
Product decisions I made
  • Dropped the "go out" (Dineout) path after finding open tool defects and empty results from every search, and documented why in the requirements.
  • Moved a rule from prompt to code. The agent once gave up after two restaurant searches without checking Instamart. Code now refuses to stop searching until both Food and Instamart have been tried.
  • Published failures. A scoring bug, a validator rule that never fired and rate-limit noise are all written up in the case study.
Voice AI · Spec-driven · Evals

Pia: a Hinglish voice shopping assistant

Say what you want in English or Hinglish, and Pia finds the lowest-priced matches for fashion and beauty across Myntra, Amazon and Nykaa. She answers in your language, takes spoken follow-ups ("aur 3 dikhao", "Myntra par hi dikhao") and says so when nothing matches. It is built for shoppers in smaller cities who know exactly what they want but have to search store by store.

Why it's interesting: the language model turned out to be the reliable part. The weak link is the speech recogniser, which mishears Indian-pronounced brand names ("Kay Beauty" came back as "Ke koti"). So the project is mostly about measuring that: a scored test set, a taxonomy of error types, and fixes that sit in code in front of the model, such as asking "Kay Beauty or Korean beauty?" before the model can guess.

To try it: the app is bring-your-own-key. Paste a Sarvam AI key when you start; it stays in your browser tab and is sent only to Sarvam. I pay for the product search, so it has a daily cap of 60 live searches and says so when the cap is hit. The first load may take up to a minute while the free-tier server wakes up. Prices come from a third-party search and may differ in the store. Not affiliated with Myntra, Amazon or Nykaa.

  • Sarvam AI (speech, Sarvam-105B, voice)
  • FastAPI
  • Plain JavaScript
  • Render
  • Speech evaluation harness
  • 21-conversation eval
  • Specs first
How it works
  • Speak. Press to talk. Sarvam's speech model turns the audio into Roman-letter text, which is what the stores' search understands.
  • Understand. A language model decides whether to search, ask a question, offer choices or politely refuse anything that isn't fashion or beauty shopping.
  • Search. A small backend queries the stores, groups the same product across them, drops poorly rated items and ranks by lowest listed price.
  • Reply. A short generated answer in the chosen language is spoken, while three product cards appear. Code checks every reply: the language, no rude forms of address, and every price spoken must exist in the search results.
Eval results, with the caveats

Speech recognition on 56 clips I recorded in English and Hinglish, scored against the brands, products and prices in each clip. Agreed spelling variants count as correct:

Sarvam setupBrand names heardProduct and colour words heard
Roman-letter output (used in the app)65% (20 of 31)92%
Newer default model42%39%
Mixed-script output55%73%
  • The last two score low partly because they return Devanagari for Hinglish, while search needs Roman letters.
  • The language model and page logic, on what was meant (21 scripted conversations, three runs): 78 of 78 turns correct. On what the recogniser actually heard: 21 of 26. All five misses were misheard brands (Asics, Van Heusen, Schwarzkopf, Fabindia).
  • Read it with care. One speaker (me): 41 clips with a fan running and 15 in a quiet room. With 31 brand mentions, a few points either way is noise. My own floor for brand accuracy was 70% and I have not met it. Dropped words showed up three times, too rarely to estimate a rate. The 78 of 78 comes after two prompt fixes the test exposed, so it is tuned, not held-out, and the pipeline test uses simulated search and voice.
Product decisions I made
  • Dropped a feature to protect the learning goal. Coupons and offers from a deals site needed per-user sign-in and had an unresolved terms question. I cut them so the time went into evaluating the voice pipeline.
  • Kept replies generated, not scripted. Scripted sentences would have hidden the Hinglish quality problem. I would rather measure it and fix it.
  • Defined what counts as correct for a brand. An agreed spelling variant is fine, a name that could mean two things must trigger a question, and anything else is a miss.
  • Made recovery cheap. A type-instead box and a pencil on each spoken message, because brand names were the main failure.
  • Bring-your-own-key for the voice service, and a daily cap on the search I pay for.
  • Tested it hard myself and wrote up real bugs: a platform tab that spoke and lost the brand, an endless brand question and a wrong-language reply.
What's next

Record other speakers, test with real Hinglish shoppers (voice against typing, timed), compare against the stores' own voice search, and add a demo video.

Vibe-coded prototype

Nourish

A mobile-first conversational app that guides users toward healthier food choices in real time. I built it quickly with AI coding tools, without a spec, to test an idea. It sits here as a contrast to the two spec-driven projects above.

  • Chatbot UI
  • Mobile-first
  • Vercel

Experience

Product Manager (title: Assistant Manager II), Policybazaar – PB Partners, Jul 2024 to present. I work on the B2B2C platform used by insurance advisors.

  • AI sales copilot (0→1 and scale-up)A conversational assistant with intent routing, customer profiling with memory, retrieval-based plan recommendations under compliance rules, and objection handling from call transcripts. 10K+ leads from 17K weekly active advisors.
  • Micro-learning platform (0→1)Replaced a low-engagement legacy LMS with a personalized platform that turns training content into videos and progressive quiz modules through an AI content pipeline. Engagement went from 3% to 30% of MAU, and cost per video fell 98%, from ₹10–20K to ₹220.
  • Investment sales journeyLed a five-stage journey from discovery to transaction, built from advisor and sales-team insights, supporting growth to ₹500Cr+ in Annualized Premium Equivalent.
  • RenewalsRedesigned renewals as a guided pipeline with urgency indicators and behavioural nudges. Engagement went from 15% to 60% of MAU, with visibility into ₹40Cr+ of renewal premium.
  • Advisor onboardingIterative product changes to a mobile-first flow brought activation turnaround to 3.5 days overall, with 80% of advisors activated within 24 hours (P80).

Before product: software engineer at Amadeus, then sales, enterprise consulting and political strategy roles. That mix is why I'm comfortable with engineers, sales leaders and clients in the same room. The full history is in my resume.

Resume

If the preview doesn't load, use the buttons above.