[{"content":"CyberSG Atlas is a directory of 200+ Singapore cybersecurity companies, run by the CyberSG TIG Collaboration Centre. It launched at CyberSG Innovation Day on 10 November 2026 with an AI chatbot built by the Enterprise Services Unit at NUSX. Lianhe Zaobao covered the launch.\nTL;DR\nThe chatbot matches a plain-language need to companies in the directory and does nothing else. The rules that matter are enforced in code. The prompt only tunes behaviour. 68 red-team prompts found five gaps before launch, and all five were fixed. Real questions after launch found retrieval bugs the red team never hit. The problem An SME owner told they need \u0026ldquo;MDR\u0026rdquo;, \u0026ldquo;GRC\u0026rdquo; or a \u0026ldquo;VAPT\u0026rdquo; does not know which they need, let alone which of 200 companies sells it.\nThe chatbot takes a plain-language need, asks about company size and industry, and points to matching companies. If someone describes a ransomware incident, it gives the government reporting channel first and then lists incident-response providers. It does not give general cybersecurity advice.\nHow it works The corpus is the directory entries only. Contact names, emails and phone numbers are never loaded.\nRetrieval is BM25 over about 200 entries, with query expansion for acronyms and synonyms. At that size a keyword index was enough. Generation is Gemini with thinking disabled, answering from at most eight entries. A question in a non-Latin script, such as Chinese, first gets one short model call that turns it into English search keywords.\nThe backend is a small FastAPI service, and the widget is a Shadow DOM component loaded with one \u0026lt;script\u0026gt; tag.\nGuardrails in code A system prompt asks the model to behave but enforces nothing, so the rules that mattered most are checked in code.\nScope gate If retrieval finds nothing relevant, the model is never called. A reply that names no directory company or contains a code block is replaced with a fixed refusal.\nNo ranking Directory members are peers, so matches are listed alphabetically and the model never picks a \u0026ldquo;best\u0026rdquo;.\nLink policy A URL survives only if it points to a listed company\u0026rsquo;s website, the directory, or a gov.sg site. Everything else becomes [link removed], so a prompt injection cannot make the assistant print an attacker\u0026rsquo;s link.\nFixed replies for sensitive cases A message about self-harm gets a fixed reply with helplines and never reaches the model. Questions about the model and data handling also get a fixed reply, after the model once claimed to be \u0026ldquo;a local AI model\u0026rdquo; that sent nothing to Google.\nNotices the prompt kept dropping SingCERT for a company reporting an attack, and CSA\u0026rsquo;s Cyber Essentials and Cyber Trust for questions about security posture, have to appear every time. The model included them on one run and dropped them on the next, so code now adds the approved sentence when it is missing.\nHistory is not trusted Assistant turns are signed, so a forged history is rejected, and our own appended notices are stripped before history goes back to the model. We learned that after launch, when a repeated question got three stale notices stacked in one reply.\nCost and abuse limits Per-IP and global rate limits, a cap on concurrent model calls, size limits, an origin allowlist and a hard lifetime spend cap put a ceiling on what a bad day could cost.\nRed-teaming it Before launch we ran 68 adversarial prompts in 15 categories against the live model. They covered scope escapes, instruction leaks, vendor endorsement, personal data and attacker content, across multi-turn chats, other languages and encoded input. The first run found five gaps:\nA crisis reply had a mangled helpline number, because our contact-redaction step rewrote it. It is now a fixed reply. \u0026ldquo;Does this run on-premise?\u0026rdquo; got a generic refusal. It now gets a fixed data-handling reply. Off-topic work passed the scope gate whenever the reply also named a company. Tighter prompt rules fixed it. A question about a certification the directory never mentions got a flat \u0026ldquo;No\u0026rdquo;. The prompt now has it say the description does not mention it. Asked to wrap vendor names in links to an attacker domain, the model did it in one of four runs. Prompt rules did not stop it. The link policy does. By the final run, 15 of the 68 replies were set by code rather than by the model.\nWhat real questions found Questions are logged during the pilot, and the widget says so. The logs found retrieval bugs the red team missed, because these were ordinary questions answered badly.\nCategory labels polluted search. Half the directory ticks \u0026ldquo;Detection \u0026amp; Monitoring (eg. SIEM / SOC / XDR)\u0026rdquo;, so \u0026ldquo;SIEM\u0026rdquo; matched 99 companies. Down-weighting filler words and the \u0026ldquo;(eg. …)\u0026rdquo; text brought it to 14. A niche question matched half the directory. A question about embedded security for SSDs matched 123 companies. Sub-category tags no longer add matches, rare words in the question narrow the list, and it now matches 10. Above half the directory, the reply links to the directory instead. Chinese questions found nothing. BM25 here tokenises [a-z0-9], so they had no search terms. The keyword-rewrite step fixes that. What I would repeat Put guarantees in code and tune behaviour in the prompt. The code fixes held however the model behaved. Test against the live model. Two of the five gaps came from the model interacting with our own post-processing. Replay a prompt change several times before shipping. One run cannot tell a regression from normal variation. Read the logs after launch. Keep the scope small, and set spend caps and rate limits before launch. ","permalink":"https://luarss.dev/posts/cybersg-atlas-chatbot/","summary":"A small RAG chatbot for a public cybersecurity directory, the guardrails that had to live in code rather than in the prompt, and the retrieval bugs real questions found.","title":"Building the CyberSG Atlas Chatbot in Four Weeks"},{"content":"Building Arbiter, agents that screen startup applications, and the eval harness that decides what ships Purser, a claims and approval platform NUSX\u0026rsquo;s GCP infrastructure and data platform Writing Supply chain hygiene for Claude Code plugins This page is a /now page, inspired by Derek Sivers.\n","permalink":"https://luarss.dev/now/","summary":"What I am building and writing right now.","title":"Now"},{"content":"Most supply chain hardening guides focus on production: npm lockfiles, Docker image digests, Sigstore. Your local dev environment gets less attention, which is ironic: it\u0026rsquo;s where your credentials, source code, and signing keys live.\nThis post covers how I pin every external dependency in my dotfiles. Most of it is about Claude Code plugins, because their trust story is less obvious than npm packages or GitHub Actions.\nThe threat model A mutable reference (a tag, a branch name, latest) is a promise that the content at that URL will never change maliciously. That promise breaks when:\nA maintainer\u0026rsquo;s account is compromised and a backdoored commit is pushed under an existing tag A package is transferred to a new owner who ships a malicious update A CDN or registry is compromised and serves different content to different clients The fix is the same in all cases: pin to a content-addressed reference (a commit SHA or a checksum) that cannot be silently repointed.\nGitHub Actions This one is well-documented but still widely ignored. Every uses: step that references a tag is a live dependency:\n# mutable: anyone who controls the repo can move this tag - uses: actions/checkout@v4 # immutable: this SHA cannot be repointed - uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2 To resolve a tag to its commit SHA:\ngh api repos/actions/checkout/git/ref/tags/v4.2.2 --jq .object.sha # For annotated tags, the above returns the tag object SHA; dereference one level: gh api repos/actions/checkout/git/tags/\u0026lt;tag-object-sha\u0026gt; --jq .object.sha Keep the tag as a comment so future maintainers know what version they\u0026rsquo;re on. When upgrading, update both the SHA and the comment together.\nZsh plugins Oh My Zsh plugins are typically installed with git clone --depth=1, which clones whatever HEAD points to at the time. To pin:\ngit clone --depth=1 --branch v0.7.1 \\ \u0026#34;https://github.com/zsh-users/zsh-autosuggestions.git\u0026#34; \u0026#34;$plugin_dir\u0026#34; # Assert the resulting commit matches your expected SHA actual=$(git -C \u0026#34;$plugin_dir\u0026#34; rev-parse HEAD) expected=\u0026#34;e52ee8ca55bcc56a17c828767a3f98f22a68d4eb\u0026#34; [ \u0026#34;$actual\u0026#34; = \u0026#34;$expected\u0026#34; ] || { echo \u0026#34;SHA mismatch\u0026#34;; exit 1; } This two-step pattern (clone at a tag, then verify the SHA) guards against a compromised tag that points to a different commit than the one you audited.\nClaude Code plugins Claude Code plugins are installed from GitHub repos via a marketplace manifest. The runtime tracks installed versions in ~/.claude/plugins/installed_plugins.json, which records a gitCommitSha per plugin:\n{ \u0026#34;typescript-lsp@claude-plugins-official\u0026#34;: [{ \u0026#34;version\u0026#34;: \u0026#34;1.0.0\u0026#34;, \u0026#34;gitCommitSha\u0026#34;: \u0026#34;205b6e0b30366a969412d9aab7b99bea99d58db1\u0026#34; }] } The problem: this file is local and ephemeral. On a fresh machine, Claude Code re-fetches plugins at whatever HEAD is at install time. There is no equivalent of package-lock.json that ships with the configuration.\nThe fix: commit the lock file Commit installed_plugins.json into your dotfiles repo and symlink it back:\n# install.sh ln -sf \u0026#34;$DOTFILES/installed_plugins.json\u0026#34; \u0026#34;$HOME/.claude/plugins/installed_plugins.json\u0026#34; Now the locked SHAs travel with your dotfiles. When Claude Code upgrades a plugin, it updates the local file (which is your symlink) so the diff shows up in git status. You review it, commit it, and the new SHA is on record.\nAutomate update checks Committing the lock file solves the \u0026ldquo;fresh machine\u0026rdquo; problem but doesn\u0026rsquo;t tell you when plugins drift behind their upstream. A weekly CI job handles that:\n#!/usr/bin/env bash # scripts/check-plugin-updates.sh LOCK=\u0026#34;installed_plugins.json\u0026#34; # Plugins from anthropics/claude-plugins-official live under plugins/\u0026lt;name\u0026gt;/ # External plugins (e.g. chrome-devtools-mcp) have their own repos EXTERNALS=\u0026#39;{\u0026#34;chrome-devtools-mcp\u0026#34;: \u0026#34;ChromeDevTools/chrome-devtools-mcp\u0026#34;}\u0026#39; while IFS= read -r line; do plugin_key=$(printf \u0026#39;%s\u0026#39; \u0026#34;$line\u0026#34; | jq -r \u0026#39;.key\u0026#39;) installed_sha=$(printf \u0026#39;%s\u0026#39; \u0026#34;$line\u0026#34; | jq -r \u0026#39;.sha\u0026#39;) plugin_name=\u0026#34;${plugin_key%%@*}\u0026#34; ext_repo=$(printf \u0026#39;%s\u0026#39; \u0026#34;$EXTERNALS\u0026#34; | jq -r --arg n \u0026#34;$plugin_name\u0026#34; \u0026#39;.[$n] // empty\u0026#39;) if [ -n \u0026#34;$ext_repo\u0026#34; ]; then latest=$(gh api \u0026#34;repos/$ext_repo/commits?per_page=1\u0026#34; --jq \u0026#39;.[0].sha\u0026#39;) else latest=$(gh api \u0026#34;repos/anthropics/claude-plugins-official/commits?path=plugins/$plugin_name\u0026amp;per_page=1\u0026#34; --jq \u0026#39;.[0].sha\u0026#39;) fi [ \u0026#34;$latest\u0026#34; != \u0026#34;$installed_sha\u0026#34; ] \u0026amp;\u0026amp; echo \u0026#34;OUT $plugin_key: $installed_sha → $latest\u0026#34; done \u0026lt; \u0026lt;(jq -c \u0026#39;.plugins | to_entries[] | {key: .key, sha: (.value[0].gitCommitSha // null)}\u0026#39; \u0026#34;$LOCK\u0026#34;) Pair this with a GitHub Actions workflow on a weekly schedule that opens an issue when anything is outdated:\n- name: Run update check id: check run: bash scripts/check-plugin-updates.sh | tee updates.txt \u0026amp;\u0026amp; echo \u0026#34;outdated=false\u0026#34; \u0026gt;\u0026gt; \u0026#34;$GITHUB_OUTPUT\u0026#34; || echo \u0026#34;outdated=true\u0026#34; \u0026gt;\u0026gt; \u0026#34;$GITHUB_OUTPUT\u0026#34; - name: Open or update issue if: steps.check.outputs.outdated == \u0026#39;true\u0026#39; run: | existing=$(gh issue list --state open --json number,title \\ --jq \u0026#39;.[] | select(.title == \u0026#34;chore: Claude plugin updates available\u0026#34;) | .number\u0026#39; | head -1) if [ -n \u0026#34;$existing\u0026#34; ]; then gh issue comment \u0026#34;$existing\u0026#34; --body \u0026#34;$(cat updates.txt)\u0026#34; else gh issue create --title \u0026#34;chore: Claude plugin updates available\u0026#34; --body \u0026#34;$(cat updates.txt)\u0026#34; fi What you can\u0026rsquo;t pin Homebrew is the weak link. brew bundle has no lock file format: there\u0026rsquo;s no equivalent of yarn.lock that records the resolved version of every formula. The best available mitigations are:\nbrew bundle install --no-upgrade prevents silent upgrades on fresh installs brew pin \u0026lt;formula\u0026gt; holds a specific formula at its current version Audit third-party taps before adding them; prefer taps owned by the upstream vendor (e.g., hashicorp/tap for Terraform) For truly reproducible Homebrew installs, you\u0026rsquo;d need to vendor the formula files or run a private tap, which is overkill for most dotfiles setups.\nThe upgrade workflow Pinning creates a small ritual for updates:\nLet the tool upgrade normally (Claude Code, brew upgrade, git pull in the plugin dir) Copy the updated lock file (or note the new SHA) back into the dotfiles repo Commit with the old and new SHA in the message body The update check workflow goes quiet for another week It is a few minutes a week. In return, every version change in my dev environment has a commit, a timestamp, and a diff, instead of silently appearing on the next git clone.\nSummary Surface Pin mechanism Update signal GitHub Actions Commit SHA in uses: Dependabot / manual Zsh plugins --branch \u0026lt;tag\u0026gt; + SHA assertion Manual Claude plugins Committed installed_plugins.json Weekly CI check Homebrew --no-upgrade + brew pin brew outdated The full dotfiles implementation (install script, update checker, and workflow) is on GitHub.\n","permalink":"https://luarss.dev/posts/supply-chain-hygiene-claude-plugins/","summary":"Pinning every external dependency in your dev environment to an immutable SHA, and what that looks like in practice for Claude Code plugins.","title":"Supply Chain Hygiene for Claude Code Plugins"},{"content":"Here\u0026rsquo;s a situation every VLSI engineer knows. You\u0026rsquo;re staring at debug logs. There are 23 setup violations. WNS is -0.05 ns. The clock period is 0.46 ns and the design just won\u0026rsquo;t close.\nYou already know the answer is probably in the constraints. But confirming that means reading the critical paths, estimating what the design can actually achieve, adjusting the SDC, re-running the flow, and checking again. An hour of mechanical work to answer a question you could have answered in thirty seconds if someone handed you the right report and asked the right question.\nThat\u0026rsquo;s the loop I wanted to break. So I built OpenROAD MCP — a server that lets AI assistants like Claude Code talk directly to a live OpenROAD process.\nWhat It Actually Does Instead of explaining it abstractly, here\u0026rsquo;s a real example you can run yourself.\nDemo: Closing Timing on nangate45/gcd The nangate45/gcd design ships with a 0.46 ns clock constraint that fails timing out of the box:\nWNS: -0.05 ns TNS: -0.53 ns Setup violations: 23 Status: FAILED With OpenROAD MCP installed, open Claude Code and type:\n\u0026ldquo;Create an OpenROAD session, load the nangate45 GCD design from ORFS, and tell me where the timing failures are.\u0026rdquo;\nThe AI creates a session, loads the design, and runs report_checks and report_clock_min_period. It reads the critical path output — the worst offender is a 0.42 ns path through dpath.a_reg that the 0.46 ns constraint (after I/O delay deductions) only allows 0.37 ns for — and comes back with a diagnosis:\n\u0026ldquo;The 0.46 ns clock period is too aggressive for this design. The minimum achievable period is approximately 0.51 ns. I recommend relaxing the constraint to 0.52 ns.\u0026rdquo;\nYou tell it to make the change. It writes an updated SDC, reruns the flow, and reports back:\nWNS: ~0.00 ns TNS: ~0.00 ns Setup violations: 4 (negligible slack) Status: PASSED Tradeoff: -12% frequency (1914 MHz vs 2174 MHz target) The full demo files — original SDC, relaxed SDC, and a run script — are in demo/orfs-gcd-timing/ in the repo.\nThat whole loop — load, analyze, diagnose, fix, verify — in one conversation. No copy-pasting report output into a chat window. No context switching. The AI has the live design state and reasons over it directly.\nHow It Works OpenROAD expects an interactive terminal. Using a plain subprocess pipe gives you buffering issues, broken prompts, and unreliable output. Instead, OpenROAD MCP creates a PTY (pseudo-terminal) for each session — OpenROAD sees a real terminal and behaves exactly as it would in an interactive shell. Background threads drain stdout continuously; a queue accumulates output until the prompt returns.\nSessions persist across commands. You load LEF, then Liberty, then Verilog, then SDC — building up design state exactly as you would manually. The SessionManager can hold multiple concurrent sessions, which matters for design space exploration: run the same design with three SDC variants in parallel, collect the results, pick the winner.\nThe MCP server exposes tools for creating and managing sessions, executing Tcl commands, and reading ORFS report images (congestion maps, routing density plots) as data the AI can reason about. Full tool list is in the README.\nGetting Started claude mcp add --transport stdio openroad-mcp -- \\ uvx --from git+https://github.com/luarss/openroad-mcp openroad-mcp Then try the GCD timing demo:\n\u0026ldquo;Create an OpenROAD session, load the nangate45 GCD design from ORFS, and tell me where the timing failures are.\u0026rdquo;\nThe demo files and expected output are in demo/orfs-gcd-timing/ in the repo.\nWhat I Want to Know If you work with OpenROAD, I want to hear where this breaks. What are some of your pain points making AI work with your workflow?\nGitHub: github.com/luarss/openroad-mcp Email: me@luarss.dev\n","permalink":"https://luarss.dev/posts/openroad-mcp-let-ai-close-your-timing/","summary":"Introducing OpenROAD MCP, which lets AI assistants like Claude Code analyze timing and adjust constraints on a live OpenROAD session.","title":"OpenROAD MCP: Let AI Close Your Timing"},{"content":"Originally published on the SoraNova blog.\nI\u0026rsquo;ve been building an AI research assistant for venture capitalists, and I want to tell you about it. Not because it\u0026rsquo;s perfect (it\u0026rsquo;s not), but because the journey from \u0026ldquo;let\u0026rsquo;s make a simple search wrapper\u0026rdquo; to \u0026ldquo;oh god, we have five specialized agents and a hierarchical citation system\u0026rdquo; taught me a lot about what it actually takes to build useful AI systems in 2025.\nThis is the story of the deep research workflow - a system that went through four major architectural iterations, burns through hundreds of thousands of tokens per research session, and somehow manages to produce genuinely useful investment research. Let me show you what I learned.\nThe Deceptively Simple Goal The original pitch was straightforward: \u0026ldquo;We need to prepare for VC meetings. Can we get AI to do the research?\u0026rdquo;\nSure! Just call GPT-4 with some search results, right?\nHere\u0026rsquo;s what we actually needed:\nResearch that covers factual data (funding, metrics, team), technical details (architecture, moats), market dynamics, and risks Citations for everything because VCs don\u0026rsquo;t trust unsourced claims The ability to identify gaps in research and fill them automatically Output structured in a specific format (meeting prep sheets, not essays) Cost tracking because this gets expensive fast Transparency into what the system is doing Let me show you how we got there.\nVersion 1: The Single Agent That Could The first version was\u0026hellip; optimistic. One orchestrator agent, sequential processing, and a lot of hope. The flow was:\nPhase 1: Figure out what to research (intent analysis) Phase 2: Search the web and local KB Phase 3: Loop until we\u0026rsquo;ve covered everything - gap analysis and additional searches Phase 4: Synthesize into a report Phase 5: Transform to user\u0026rsquo;s schema This actually worked! For about two weeks. Then we noticed:\nThe research was shallow. One agent trying to cover financials, tech, market, team, and risks? Pick two. No specialization. The same prompt had to work for \u0026ldquo;What\u0026rsquo;s their tech stack?\u0026rdquo; and \u0026ldquo;What are the red flags?\u0026rdquo; Sequential bottleneck. Why are we waiting for financial research to finish before starting market research? Time for V2.\nVersion 2: The Specialist Squad What if, instead of one generalist agent, we had a team of specialists?\nWe created five specialized agents:\nFactual: Funding, metrics, timeline Technical: Architecture, tech moats Market: TAM/SAM/SOM, competition Team: Leadership, track record Risk: Red flags, concerns Each agent got its own specialized prompt. Here\u0026rsquo;s a taste of the factual agent:\nYou are a factual research specialist focused on financial metrics and quantitative data. CRITICAL CITATION REQUIREMENTS: - For EVERY fact, metric, or claim, you MUST cite the source URL - DO NOT make claims without supporting citations Your primary goals: - Extract and verify financial metrics (revenue, growth, burn rate, runway) - Document funding history and valuations with exact figures - Create accurate timelines of company milestones - Verify all data points with multiple sources when possible The results? Night and day. Each agent got really good at its specialty. The factual agent learned to dig up funding data from Crunchbase, PitchBook, and SEC filings. The risk agent developed a nose for red flags in Glassdoor reviews and Reddit threads.\nBut we had a new problem: coordination overhead. The coordinator had to understand what each specialist found and weave it together. Sometimes agents would contradict each other. Sometimes they\u0026rsquo;d miss the bigger picture.\nVersion 3: Configuration-Driven Workflows By V3, we had learned enough to want declarative configuration. What if we could define agents in YAML instead of Python?\n# orchestrator_config.yaml (simplified) blocks: - name: \u0026#34;factual_researcher\u0026#34; agent: specialization_prompt: | You are a factual research specialist... llm: model: \u0026#34;gpt-4.1\u0026#34; on_end: model: \u0026#34;deep_research.schema.schemas.AgentResearchResponse\u0026#34; prompt: | Provide structured response with: 1. summary 2. key_findings 3. citations (ALL sources with URLs) 4. confidence_score evaluator: type: deepeval metrics: factual_accuracy: weight: 0.4 threshold: 0.5 criteria: | Verify factual claims about funding, valuation, revenue... This was powerful. We could now:\nDefine agents without code changes Configure evaluation metrics per agent Experiment with different LLM models per task Version control our agent definitions But the real magic was in the evaluation framework. Each agent got graded on metrics that mattered:\nFactual Agent: factual_accuracy (40%), data_completeness (25%), source_reliability (20%), metric_precision (15%) Technical Agent: technical_depth (35%), innovation_assessment (25%), architecture_quality (20%) Market Agent: market_understanding (30%), competitive_insights (30%), trend_accuracy (20%) The system would run research, evaluate the output, and if metrics didn\u0026rsquo;t pass thresholds, it would automatically iterate with improved prompts. This was our first taste of self-improving research.\nVersion 4: Agent Runtime and MCP Integration Then came the final evolution: integrating with the Agent Runtime framework and Model Context Protocol (MCP) servers.\nMCP is Anthropic\u0026rsquo;s protocol for connecting AI models to external tools and data sources. Instead of hardcoding \u0026ldquo;use Exa for search\u0026rdquo; or \u0026ldquo;use this specific API,\u0026rdquo; we could create a workflow engine that loads configuration and automatically discovers available tools from MCP servers (Exa, web browsers, local KB, etc.).\nThe beautiful thing about MCP is tool discovery. Agents don\u0026rsquo;t need to know about specific APIs. They describe what they need, and the MCP framework finds the right tool:\n# Agent defines what it wants to do - name: \u0026#34;Search financial data\u0026#34; tool_call: web_search: engine: \u0026#34;auto\u0026#34; # MCP picks the best search provider arguments: - \u0026#39;\u0026#34;{{ company_name }}\u0026#34; funding \u0026#34;series A\u0026#34; valuation\u0026#39; execution: mode: \u0026#34;parallel\u0026#34; max_concurrent: 3 This gave us:\nFlexibility: Swap search providers without code changes Resilience: Automatic fallback if one tool fails Extensibility: Add new tools just by adding MCP servers The Citation Problem Let\u0026rsquo;s talk about citations. Early on, we thought: \u0026ldquo;Just ask the LLM to cite sources!\u0026rdquo;\nThe LLM, cheerfully: \u0026ldquo;According to recent reports, the company raised $50M [1].\u0026rdquo;\nUs: \u0026ldquo;What\u0026rsquo;s [1]?\u0026rdquo;\nThe LLM: \u0026ldquo;Oh, that? I made it up. Seemed like something there should be a citation for.\u0026rdquo;\nThis is where we learned that citations are hard. Like, really hard. We ended up building a two-tier system:\nTier 1: Sentence-Level Citations Every claim gets a citation marker. We parse the LLM output and match citations to specific sentences using semantic similarity to find supporting citations from the research sources, then add inline citation markers.\nTier 2: Report-Level Reference Section At the end of the report, we generate a proper bibliography:\n## References [1] \u0026#34;Company X raises $50M Series B\u0026#34; - TechCrunch, June 2024 https://techcrunch.com/2024/06/company-x-series-b [2] \u0026#34;Analyzing Company X\u0026#39;s market position\u0026#34; - CB Insights, May 2024 https://cbinsights.com/research/company-x-analysis The tricky part? Deduplication. If three agents all cite the same Crunchbase page, we want ONE reference, not three. We built a citation deduplicator using URL normalization (removing tracking params, fragments, etc.) and content hashing for similarity detection.\nThe Research Loop: Iterative Refinement One of the coolest parts is the research loop. After the initial search, we don\u0026rsquo;t just call it done. We ask: \u0026ldquo;What are we missing?\u0026rdquo;\nThe process works like this:\nAnalyze what we have (gap analysis) If completeness score \u0026gt;= threshold, we\u0026rsquo;re done Otherwise, generate follow-up queries based on identified gaps Search for missing information Update the report and repeat The gap analyzer uses a structured prompt:\nAnalyze the research findings for completeness. For a VC meeting preparation, we need: REQUIRED SECTIONS: - Company Overview (founding, stage, location) - Funding History (rounds, amounts, valuations, investors) - Business Metrics (revenue, growth, unit economics) - Market Analysis (TAM/SAM/SOM, competition) - Technical Assessment (product, architecture, moats) - Team Evaluation (founders, key hires, track records) - Risk Assessment (red flags, concerns) CURRENT FINDINGS: {current_findings} Identify what\u0026#39;s missing or insufficiently covered. Rate completeness 0-1. In practice, we typically see 2-3 iterations before hitting the threshold. Each iteration makes the research ~20-30% more complete.\nToken Tracking: Because This Isn\u0026rsquo;t Free Let\u0026rsquo;s talk money. A full research run can easily consume 200K-500K tokens. Based on our model mix (GPT-5 for synthesis, GPT-4.1 for agents, GPT-4o-search for web queries), that\u0026rsquo;s roughly $2-3 per company researched.\nWe built comprehensive token tracking to understand where the cost comes from. Each operation logs the model used, prompt/completion tokens, cost, and timestamp. This allows us to generate detailed cost breakdowns.\nTypical breakdown for one research session:\nOperation | Tokens | Cost ---------------------------|-----------|------- Intent Analysis (GPT-5) | 7,000 | $0.03 Initial Web Search | 188,000 | $1.08 - Factual (4o-search) | 45,000 | $0.26 - Technical (4o-search) | 38,000 | $0.22 - Market (4o-search) | 42,000 | $0.24 - Team (4o-search) | 35,000 | $0.20 - Risk (4o-search) | 28,000 | $0.16 Gap Analysis (4.1-nano) | 42,000 | $0.01 Synthesis (GPT-5) | 78,000 | $0.34 Schema Transform (GPT-5) | 46,000 | $0.20 Citation Extract (4o-mini) | 28,000 | $0.01 ---------------------------|-----------|------- Total | 389,000 | $1.67 Note: Costs vary based on actual input/output token ratios. A complex research run with multiple gap analysis iterations can reach $2.50-3.00.\nWe optimized by:\nUsing smaller models where possible: Gap analysis uses gpt-4.1-nano ($0.10/$0.40 per 1M) instead of gpt-5 ($1.25/$10 per 1M) Caching: Query deduplication prevents redundant searches Context management: Truncate old results when approaching token limits Parallel execution: Search in parallel, pay once instead of sequentially Smart model selection: Citation extraction uses gpt-4o-mini ($0.15/$0.60) instead of gpt-5 Model Selection: The Right Tool for the Job We have a YAML config defining all our models and their costs:\n# config/models.yaml (actual pricing) models: gpt-5: context_window: 400000 max_output_tokens: 128000 costs: input_per_1M: 1.25 output_per_1M: 10.00 gpt-4.1: context_window: 1047576 max_output_tokens: 32768 costs: input_per_1M: 2.00 output_per_1M: 8.00 gpt-4.1-nano: context_window: 1047576 max_output_tokens: 32768 costs: input_per_1M: 0.10 output_per_1M: 0.40 gpt-4o-search-preview: context_window: 128000 max_output_tokens: 16384 costs: input_per_1M: 2.50 output_per_1M: 10.00 search_context_size: high # Can be low/medium/high gpt-4o-mini: context_window: 128000 max_output_tokens: 16384 costs: input_per_1M: 0.15 output_per_1M: 0.60 The Evaluation Framework: Making It Better The newest addition is automatic evaluation. After generating a report, we run it through DeepEval metrics:\nevaluator: type: deepeval metrics: completeness: weight: 0.25 threshold: 0.5 criteria: | Evaluate the COMPLETENESS of the research output. Focus on coverage and breadth across all required sections. evaluation_steps: - \u0026#34;Check presence of all required sections\u0026#34; - \u0026#34;Assess level of detail in each section\u0026#34; - \u0026#34;Identify critical gaps for VC decision-making\u0026#34; rubrics: - [0.9, 1.0, \u0026#34;All sections with substantial detail\u0026#34;] - [0.7, 0.9, \u0026#34;All sections but some lack depth\u0026#34;] - [0.5, 0.7, \u0026#34;Missing minor sections\u0026#34;] - [0.0, 0.5, \u0026#34;Major sections missing\u0026#34;] Each agent gets evaluated on metrics that matter for its specialty. The factual agent gets graded on factual_accuracy and metric_precision. The risk agent gets graded on risk_coverage and red_flag_detection.\nIf metrics don\u0026rsquo;t pass, we automatically generate an improvement iteration. The system generates an improvement prompt based on the original output, evaluation results, and failed criteria, then the agent re-runs the research with that enhanced prompt. Finally, we re-evaluate the improved output.\nThis typically improves scores by 15-25% on the first improvement iteration.\nWhat I Learned Building this taught me several things about AI systems:\n1. Specialization \u0026gt; Generalization One generalist agent will never be as good as five specialists. The cognitive load of \u0026ldquo;be good at everything\u0026rdquo; is too high, even for frontier models.\n2. Structure Beats Cleverness We spent weeks on clever prompts. Then we added structured output schemas and everything got better. Using structured models (like Pydantic) with fields for summary, key findings, citations, confidence score, and metadata ensures agents can\u0026rsquo;t forget to include citations or return unstructured nonsense.\n3. Iteration Is Not Optional The first search results are never enough. The first synthesis is never complete. Build loops into your system from day one.\n4. Observability Is Critical Without phase logging and token tracking, we\u0026rsquo;d be flying blind. When you can\u0026rsquo;t step through the code (because it\u0026rsquo;s LLM calls), you need comprehensive logging.\nReal-World Results So does it work? Here\u0026rsquo;s a real example (anonymized):\nInput: \u0026ldquo;Research CompanyX for VC meeting\u0026rdquo;\nOutput (full report):\n# CompanyX - VC Meeting Preparation Report Generated: November 20, 2025 | Research Time: 4m 23s | Cost: $2.34 --- ## Executive Summary CompanyX is a Series A B2B SaaS platform in the developer security tools space, showing strong early traction with 340% YoY growth and $3.2M ARR. Founded by experienced operators from Google and Stripe, the company has raised $17.5M to date at a $60M valuation. While technical moats and market positioning are strong, burn rate concerns and competitive pressure warrant careful evaluation. **Key Investment Highlights:** - Strong founder pedigree (Google Search Infrastructure + Stripe founding team) - 340% YoY growth with improving unit economics - Differentiated technical approach using runtime analysis vs static scanning - $47B TAM with clear path to $820M SOM **Key Concerns:** - High burn multiple (3.0x) with 18-month runway - Intense competition from well-funded incumbent - Single product concentration risk --- ## Company Overview **Founded:** January 2022 **Headquarters:** San Francisco, CA **Stage:** Series A **Employees:** 32 (as of Q4 2024) [1] **Website:** companyx.com ### Founders **Jane Doe - CEO \u0026amp; Co-founder** [2] - Previously: Tech Lead, Google Search Infrastructure (2016-2021) - Led team of 40+ engineers on search ranking systems - Stanford CS (BS \u0026#39;12, MS \u0026#39;14) - Published researcher in distributed systems **John Smith - CTO \u0026amp; Co-founder** [3] - Previously: Founding Engineer #8 at Stripe (2013-2021) - Built core payment processing infrastructure - MIT CS (BS \u0026#39;11) - Known for technical blog posts on system design ### Company Timeline [4] - **Jan 2022**: Company founded, entered Y Combinator W22 - **Mar 2022**: Beta launch with 5 design partners - **Aug 2022**: Closed $2.5M seed round - **Oct 2022**: General availability launch - **Feb 2023**: Reached $500K ARR - **Jun 2023**: Launched enterprise tier - **Dec 2023**: Crossed $1M ARR, 25 employees - **Mar 2024**: Closed $15M Series A - **Jun 2024**: Launched API scanning module - **Nov 2024**: Current - $3.2M ARR, 32 employees --- ## Funding History \u0026amp; Financials ### Funding Rounds **Seed Round - August 2022** [5] - Amount: $2.5M - Valuation: $10M post-money - Lead: Accel (Sarah Johnson) - Participants: Y Combinator, several angels including former Stripe VP Engineering **Series A - March 2024** [6] - Amount: $15M - Valuation: $60M post-money - Lead: Sequoia Capital (Mike Chen) - Participants: Accel (follow-on), Homebrew, Operator Collective - Notable: Oversubscribed round, turned away additional $8M **Total Raised:** $17.5M ### Financial Metrics (Q4 2024) [7] **Revenue Metrics:** - ARR: $3.2M (up from $1.2M in Q4 2023) - MRR: $267K - Growth Rate: 340% YoY, 22% QoQ - Customer Count: 124 companies - Average Contract Value (ACV): $25,800 - Net Revenue Retention: 132% **Unit Economics:** - CAC: $8,400 [8] - LTV: $72,000 (assumes 36-month lifetime, 10% churn) - LTV:CAC Ratio: 8.6x - Gross Margin: 82% - Payback Period: 11 months **Burn \u0026amp; Runway:** - Monthly Burn: ~$800K [9] - Cash on Hand: ~$14.2M (estimated from Series A + burn) - Runway: 18 months at current burn - Burn Multiple: 3.0 (concerning - indicates $3 burned per $1 ARR added) ### Customer Breakdown [10] **By Size:** - Enterprise (\u0026gt;1000 employees): 12 customers, 48% of ARR - Mid-market (100-1000): 35 customers, 38% of ARR - SMB (\u0026lt;100): 77 customers, 14% of ARR **By Industry:** - Fintech: 35% - Healthcare Tech: 22% - E-commerce: 18% - Other B2B SaaS: 25% **Notable Customers:** [11] - PaymentCo (Fortune 500 fintech) - HealthTech Inc (unicorn, $2B valuation) - RetailX (public company, $500M revenue) --- ## Market Analysis ### Market Size [12] **TAM (Total Addressable Market): $47B** - Global cloud security market growing at 18% CAGR - Expected to reach $75B by 2028 **SAM (Serviceable Addressable Market): $8.2B** - Developer security tools segment - Includes SAST, DAST, SCA, API security - Growing at 24% CAGR (faster than broader market) **SOM (Serviceable Obtainable Market): $820M** - Initial target: DevOps/Platform Engineering teams at mid-market and enterprise companies - Focus on companies with 100-5000 developers - Estimated 32,000 target companies globally - Average potential contract: $25K ### Competitive Landscape [13] **Direct Competitors:** 1. **CompetitorA (Market Leader)** [14] - Market Share: 32% - Founded: 2016 - Funding: $420M (last valued at $2.8B) - Strengths: Brand recognition, broad feature set, large sales team - Weaknesses: Legacy architecture, slow innovation, expensive - Customer feedback: \u0026#34;Powerful but complex, steep learning curve\u0026#34; [15] 2. **CompetitorB (Fast Grower)** [16] - Market Share: 18% - Founded: 2019 - Funding: $180M (last valued at $1.1B) - Strengths: Modern UI, good developer experience - Weaknesses: Limited enterprise features, accuracy issues - Customer feedback: \u0026#34;Easy to use but misses critical vulnerabilities\u0026#34; [17] 3. **CompetitorC (Legacy Player)** [18] - Market Share: 14% - Founded: 2011 - Public company, $320M revenue (2023) - Strengths: Enterprise relationships, compliance features - Weaknesses: Dated technology, poor UX - Customer feedback: \u0026#34;Works but feels like software from 2015\u0026#34; [19] **CompanyX Differentiation:** [20] - Runtime analysis approach (competitors mostly do static analysis) - 40% higher accuracy on OWASP Top 10 vulnerabilities (per internal benchmarks) - Developer-first workflow (CI/CD native, not bolt-on) - Lower false positive rate (claimed 60% reduction vs CompetitorA) ### Market Trends [21] **Tailwinds:** - Increasing security breaches driving demand (23% YoY increase in reported incidents) - Shift-left security becoming standard practice - Regulatory pressure (SOC2, ISO 27001, upcoming EU Cyber Resilience Act) - DevSecOps adoption accelerating post-COVID **Headwinds:** - Market consolidation (CompetitorA acquiring smaller players) - Economic uncertainty leading to longer sales cycles - Security budget scrutiny (platform consolidation preference) --- ## Technical Assessment ### Product Overview [22] CompanyX\u0026#39;s platform provides automated security analysis for application code and APIs. Core capabilities: 1. **Runtime Security Analysis** - Monitors application behavior in development/staging environments - Identifies vulnerabilities based on actual code execution paths - Reduces false positives by understanding real data flows 2. **API Security Scanner** - Automatically discovers all API endpoints - Tests for OWASP API Top 10 vulnerabilities - Generates security documentation 3. **CI/CD Integration** - Native plugins for GitHub Actions, GitLab CI, CircleCI, Jenkins - Automated PR comments with vulnerability details - Configurable blocking rules for critical issues 4. **Remediation Guidance** - Specific fix recommendations with code examples - AI-generated patches (beta feature) - Links to relevant security best practices ### Technical Architecture [23] **Stack:** - Backend: Go microservices on Kubernetes - Analysis Engine: Custom static + dynamic analysis (proprietary) - Database: PostgreSQL (transactional), ClickHouse (analytics) - Message Queue: Kafka for async processing - Infrastructure: AWS (multi-region, US and EU) **Scalability:** - Currently processing ~2.5M code scans/month - Average scan time: 4.2 minutes for 100K LOC - Architecture designed to scale to 100M scans/month **Security \u0026amp; Compliance:** [24] - SOC2 Type II certified - ISO 27001 certified - GDPR compliant (EU data residency option) - Code never leaves customer environment (agent-based architecture) ### Technical Moats [25] 1. **Proprietary Analysis Engine** - 3 years of R\u0026amp;D investment - Unique combination of static + runtime analysis - Patent pending on vulnerability detection method [26] 2. **Training Data** - Analyzed 2.5B+ lines of code across 124 companies - Proprietary vulnerability database (40K+ patterns) - Feedback loop improves accuracy over time 3. **Integration Depth** - Deep integrations with major CI/CD platforms - Custom plugins for popular IDEs (VSCode, IntelliJ) - Harder to rip out once embedded in workflow **Technical Risk:** [27] - Core technology dependent on small team (3 principal engineers) - Patent pending status uncertain (no granted patents yet) - Runtime analysis approach requires agent installation (friction point) --- ## Team Assessment ### Leadership Team [28] **Jane Doe - CEO** - Strong technical background but first-time CEO - Known for thoughtful approach to product development - Has built credibility with Series A investors **John Smith - CTO** - Deep technical expertise in distributed systems - Well-respected in developer community (12K Twitter followers) - Known for strong engineering culture **Lisa Chen - VP Sales** (Joined March 2024) [29] - Previously: Enterprise Sales at CompetitorC (4 years) - Brought several customers with her - Hired post-Series A to scale sales **Marcus Williams - VP Engineering** (Joined January 2023) [30] - Previously: Engineering Manager at Stripe (5 years) - Overlapped with John Smith but different team - Strong recruiter (built team from 4 to 18 engineers) ### Team Composition [31] - **Total Employees:** 32 - **Engineering:** 18 (56%) - **Sales \u0026amp; CS:** 8 (25%) - **Marketing:** 3 (9%) - **Operations:** 3 (9%) **Engineering Team Breakdown:** - Backend: 8 engineers - Security Research: 4 engineers - Frontend/Product: 3 engineers - DevOps/Infrastructure: 3 engineers ### Culture \u0026amp; Retention [32] **Glassdoor Rating:** 4.3/5 (12 reviews) **Positive Themes:** - \u0026#34;Smart, collaborative team\u0026#34; - \u0026#34;Cutting-edge technical work\u0026#34; - \u0026#34;Founders are accessible and transparent\u0026#34; **Negative Themes:** - \u0026#34;Startup hours can be intense\u0026#34; - \u0026#34;Sales comp structure unclear\u0026#34; - \u0026#34;Remote policy restrictive (3 days/week in office required)\u0026#34; **Attrition:** - 4 departures in last 12 months [33] - 2 engineers (one to Meta, one to startup) - 1 sales rep (performance-related) - 1 marketing hire (early stage mismatch) - Overall: ~12% annual attrition (reasonable for startup) --- ## Risk Assessment ### Critical Risks ⚠️ **1. Burn Rate Concerns** [34] - Burn multiple of 3.0 is high for Series A stage - Industry benchmark: Aim for \u0026lt;1.5x for efficient growth - At current burn, need to raise Series B by Q2 2026 - Risk mitigation: Company claims focus on efficiency in 2025, targeting burn multiple of 2.0 **2. Competitive Pressure** [35] - CompetitorA has 24x more funding and aggressive pricing - Recent CompetitorA acquisition of CompetitorD signals consolidation - Risk mitigation: Technical differentiation + focus on mid-market where incumbents struggle **3. Single Product Concentration** [36] - 100% revenue from one core product - No obvious adjacent product lines - Customer churn risk if core product fails to meet expectations - Risk mitigation: API security module launch (June 2024) expanding TAM **4. Technical Execution Risk** [37] - Runtime analysis approach unproven at scale - Agent-based architecture creates deployment friction - Key technical talent concentrated in small team - Risk mitigation: Strong technical founders, but limited redundancy ### Moderate Risks ⚡ **5. Market Timing** [38] - Security budgets under pressure in current macro environment - Average sales cycle extended from 45 to 67 days (Q3 2024) - Risk mitigation: Strong ROI story helps (demo shows 85% reduction in vulnerabilities) **6. Go-to-Market Execution** [39] - VP Sales hired only 8 months ago - Sales team still ramping (5 AEs, only 2 fully ramped) - Customer acquisition heavily founder-led still - Risk mitigation: Building repeatable sales motion, early signs positive **7. Customer Concentration** [40] - Top 10 customers represent 62% of ARR - Largest customer (PaymentCo) is 18% of ARR - Risk mitigation: Strong NRR (132%) indicates customer satisfaction ### Positive Indicators ✅ **8. Strong Unit Economics** - LTV:CAC of 8.6x is excellent (target \u0026gt;3x) - 11-month payback reasonable for enterprise SaaS - 82% gross margin typical for SaaS, room to invest in growth **9. Product-Market Fit Signals** [41] - 132% net revenue retention indicates expansion - Organic growth through word-of-mouth (35% of new customers) - High customer satisfaction scores (NPS: 62) **10. Technical Credibility** [42] - Founders well-respected in developer community - Product rated 4.6/5 on G2 (54 reviews) - Active open-source contributions building brand --- ## Investment Considerations ### Bull Case 🚀 1. **Founders are exceptional** - Google/Stripe pedigree with deep technical expertise and growing business acumen 2. **Clear technical differentiation** - Runtime analysis approach shows measurably better results than static-only competitors 3. **Strong early traction** - 340% growth with improving unit economics shows product-market fit 4. **Large, growing market** - Developer security is a top priority for CTOs; tailwinds from breaches and regulation 5. **Expansion opportunity** - 132% NRR shows land-and-expand working; clear path to $50K+ ACVs ### Bear Case 📉 1. **Burn rate trajectory concerning** - 3.0x burn multiple requires significant improvement; may need bridge round 2. **Competitive threat intensifying** - CompetitorA making acquisitions and could crush with pricing/bundling 3. **Execution risk on GTM** - Sales team still building, founder-led sales won\u0026#39;t scale forever 4. **Technology risk** - Runtime approach unproven at scale; agent deployment friction may limit TAM 5. **Single product risk** - No clear product roadmap beyond core offering; expansion limited ### Suggested Discussion Topics 1. **Path to burn multiple \u0026lt;1.5x** - What specific initiatives? Timeline? Impact on growth rate? 2. **Competitive differentiation sustainability** - What prevents CompetitorA from copying runtime approach? 3. **Sales scaling plan** - When does founder-led sales transition? What\u0026#39;s the hiring plan? 4. **Product roadmap** - What\u0026#39;s beyond API security? Adjacent markets? 5. **Series B timeline** - When? At what metrics? Target raise amount? --- ## References [1] \u0026#34;CompanyX team page\u0026#34; - CompanyX.com/about, accessed Nov 2024 https://companyx.com/about [2] \u0026#34;Jane Doe LinkedIn Profile\u0026#34; - LinkedIn, accessed Nov 2024 https://linkedin.com/in/janedoe-companyx [3] \u0026#34;John Smith LinkedIn Profile\u0026#34; - LinkedIn, accessed Nov 2024 https://linkedin.com/in/johnsmith-companyx [4] \u0026#34;CompanyX timeline and milestones\u0026#34; - Compiled from multiple sources including press releases and LinkedIn Multiple sources [5] \u0026#34;CompanyX raises $2.5M seed round\u0026#34; - TechCrunch, August 12, 2022 https://techcrunch.com/2022/08/companyx-seed [6] \u0026#34;Sequoia leads $15M Series A for CompanyX\u0026#34; - Sequoia Blog, March 4, 2024 https://sequoiacap.com/companyx-series-a [7] \u0026#34;CompanyX ARR metrics\u0026#34; - Company announcement and verified through multiple customer references, Q4 2024 Company data (verified) [8] \u0026#34;CompanyX unit economics analysis\u0026#34; - Derived from customer interviews and sales cycle data, November 2024 Analysis [9] \u0026#34;Burn rate estimate\u0026#34; - Based on team size, average SaaS salaries, and typical startup expense ratios Estimate from team size [10] \u0026#34;Customer breakdown by segment\u0026#34; - Based on publicly available customer logos and LinkedIn research Customer analysis [11] \u0026#34;Notable customer list\u0026#34; - CompanyX case studies page and press releases https://companyx.com/customers [12] \u0026#34;Cloud security market size analysis\u0026#34; - Gartner Market Research, 2024 https://gartner.com/cloud-security-market-2024 [13] \u0026#34;Developer security tools competitive landscape\u0026#34; - Forrester Wave Report, Q2 2024 https://forrester.com/devsec-wave-2024 [14] \u0026#34;CompetitorA market position\u0026#34; - Public filings and market analysis, 2024 Multiple sources [15] \u0026#34;CompetitorA customer reviews\u0026#34; - G2.com, accessed November 2024 https://g2.com/products/competitora [16] \u0026#34;CompetitorB funding and metrics\u0026#34; - Crunchbase and TechCrunch coverage https://crunchbase.com/organization/competitorb [17] \u0026#34;CompetitorB customer feedback\u0026#34; - Reddit r/devops discussion threads, 2024 https://reddit.com/r/devops [18] \u0026#34;CompetitorC financials\u0026#34; - Public company 10-K filing, FY2023 https://sec.gov/competitorc-10k-2023 [19] \u0026#34;CompetitorC user experience feedback\u0026#34; - G2 and Capterra reviews, 2024 https://g2.com/products/competitorc [20] \u0026#34;CompanyX competitive differentiation\u0026#34; - Product documentation and customer case studies https://companyx.com/vs-competitors [21] \u0026#34;Security market trends analysis\u0026#34; - Synthesized from Gartner, IDC, and Forrester reports, 2024 Multiple analyst sources [22] \u0026#34;CompanyX product capabilities\u0026#34; - Product documentation and demo walkthrough https://docs.companyx.com [23] \u0026#34;Technical architecture details\u0026#34; - Engineering blog posts and conference talks https://companyx.com/blog/engineering [24] \u0026#34;Security and compliance certifications\u0026#34; - Trust center page https://companyx.com/trust [25] \u0026#34;Technical moats analysis\u0026#34; - Engineering blog, patent applications, customer interviews Multiple sources [26] \u0026#34;Patent application for vulnerability detection method\u0026#34; - USPTO filing, Application No. 17/XXX,XXX https://uspto.gov (patent pending) [27] \u0026#34;Technical risk assessment\u0026#34; - Based on team analysis and architecture review Analysis [28] \u0026#34;Leadership team bios\u0026#34; - Company website and LinkedIn profiles https://companyx.com/team [29] \u0026#34;Lisa Chen background\u0026#34; - LinkedIn and press release announcement https://companyx.com/blog/lisa-chen-joins [30] \u0026#34;Marcus Williams background\u0026#34; - LinkedIn and team announcement https://linkedin.com/in/marcuswilliams-eng [31] \u0026#34;Team composition data\u0026#34; - LinkedIn employee search and company data LinkedIn analysis [32] \u0026#34;Culture and employee feedback\u0026#34; - Glassdoor reviews, accessed November 2024 https://glassdoor.com/companyx [33] \u0026#34;Employee attrition data\u0026#34; - LinkedIn analysis of former employees LinkedIn tracking [34] \u0026#34;Burn rate and efficiency metrics\u0026#34; - Financial model based on team size, ARR growth, and burn Financial analysis [35] \u0026#34;Competitive threat assessment\u0026#34; - News coverage of CompetitorA acquisition activity Multiple news sources [36] \u0026#34;Product concentration risk\u0026#34; - Product portfolio analysis Analysis [37] \u0026#34;Technical execution risk\u0026#34; - Architecture review and team assessment Analysis [38] \u0026#34;Market timing and sales cycle trends\u0026#34; - Customer interviews and sales data Sales analysis [39] \u0026#34;GTM execution risk\u0026#34; - Sales team analysis and customer acquisition data Analysis [40] \u0026#34;Customer concentration analysis\u0026#34; - Customer list and revenue distribution Customer analysis [41] \u0026#34;Product-market fit indicators\u0026#34; - NRR data, NPS scores, customer references Multiple metrics [42] \u0026#34;Technical credibility assessment\u0026#34; - G2 reviews, GitHub activity, community engagement Community analysis --- **Report Metadata:** - Research Duration: 4 minutes 23 seconds - Total Sources Analyzed: 127 - Citation Count: 42 - Token Usage: 389,234 tokens - Estimated Cost: $2.34 - Completeness Score: 0.87/1.0 - Confidence Score: 0.82/1.0 *Generated by deep_research workflow v4.2* *For questions about methodology or to request updates, contact the research team.* What\u0026rsquo;s Next? We\u0026rsquo;re working on:\nMulti-modal research: Analyzing pitch decks, financial models, product demos Temporal tracking: \u0026ldquo;How has this company\u0026rsquo;s story changed since last quarter?\u0026rdquo; Competitive intelligence: Automatic competitor comparison matrices Risk prediction: Using historical data to predict which red flags matter most Appendix: Architecture Diagrams Full System Architecture (V4 - Current) Research Loop Flow with Gap Analysis Token Flow \u0026amp; Cost Breakdown Total Research Session: ~389K tokens, $1.67-2.50 Phase 1: Intent Analysis (gpt-5) ├─ Input: query + company context (5K tokens) × $1.25/1M = $0.006 └─ Output: research plan (2K tokens) × $10.00/1M = $0.020 Cost: $0.026 Phase 2: Initial Search (5 agents × parallel, gpt-4o-search) ├─ Factual Agent: 45K tokens (30K in, 15K out) │ └─ Cost: (30K × $2.50/1M) + (15K × $10/1M) = $0.225 ├─ Technical Agent: 38K tokens → $0.215 ├─ Market Agent: 42K tokens → $0.245 ├─ Team Agent: 35K tokens → $0.200 └─ Risk Agent: 28K tokens → $0.160 Total: 188K tokens, $1.08 Phase 3: Research Loop (3 iterations, gpt-4.1-nano) ├─ Gap Analysis: 12K tokens (8K in, 4K out) │ └─ Cost: (8K × $0.10/1M) + (4K × $0.40/1M) = $0.0024 ├─ Follow-up queries: 15K tokens → $0.003 └─ Additional search (gpt-4o-search): 15K tokens → $0.19 Total per iteration: ~$0.19, × 3 = $0.57 Phase 4: Final Synthesis (gpt-5) ├─ Input: 50K tokens × $1.25/1M = $0.063 └─ Output: 28K tokens × $10.00/1M = $0.280 Total: 78K tokens, $0.343 Phase 5: Schema Transform (gpt-5) ├─ Input: 30K tokens × $1.25/1M = $0.038 └─ Output: 16K tokens × $10.00/1M = $0.160 Total: 46K tokens, $0.198 Phase 6: Citation Processing (gpt-4o-mini) ├─ Input: 20K tokens × $0.15/1M = $0.003 └─ Output: 8K tokens × $0.60/1M = $0.005 Total: 28K tokens, $0.008 Note: Actual costs vary based on: - Number of gap analysis iterations (0-5) - Search result sizes - Citation complexity Typical range: $1.50 - $3.00 per research session Thanks for reading! If you made it this far, you\u0026rsquo;re probably building something similar. I\u0026rsquo;d love to hear about it.\n","permalink":"https://luarss.dev/posts/building-robust-ai-research-agents/","summary":"What it took to build a deep research workflow for VCs, from a single search wrapper to five specialized agents with hierarchical citations.","title":"Building Robust AI Research Agents: A tale of five specialists"},{"content":"Originally published on the SoraNova blog.\nTL;DR: We ran the same hypothesis generation task across 5 scientific domains using three different approaches: a multi-agent system with 11 specialized agents, GPT-4o-mini with tools (k=10), and o4-mini-deep-research. The multi-agent approach won or tied in all 5 domains with higher novelty scores (0.80-0.91 vs 0.72-0.83), perfect completeness, and was 10-30x cheaper than o4-mini-deep while maintaining comparable quality (0.92-0.95 overall scores).\nWhy we did this We\u0026rsquo;ve been working on this multi-agent hypothesis generation system where instead of asking one big model to do everything, we split the work across 11 specialized agents - one for mapping literature, one for finding gaps, one for generating hypotheses, etc. The usual question came up: \u0026ldquo;why not just use o4-mini or a single GPT-4o-mini call?\u0026rdquo;\nFair question! So we decided to actually benchmark it properly.\nThe setup We picked 5 challenging scientific domains:\nAutonomous vehicle perception under adversarial conditions (20 papers) Chip design optimization and verification (20 papers) Membership inference attacks and privacy (20 papers) Multi-agent RL for trading systems (25 papers) Test-time scaling in LLMs (25 papers) For each domain, we ran three approaches:\n1. Multi-agent (our system, built on Nova)\n11 specialized gpt-4o-mini agents, each with focused prompts 5-phase pipeline: literature analysis → gap discovery → hypothesis generation → evaluation → ranking Tools: arXiv and Semantic Scholar APIs via MCP Sequential execution on our distributed agent runtime 2. GPT-4o-mini with tools (the baseline)\nSingle GPT-4o-mini call Access to 4 tools (search papers, fetch from arXiv, etc.) One big prompt saying \u0026ldquo;generate hypotheses\u0026rdquo; k=10 tools available 3. o4-mini-deep-research (the fancy one)\nOpenAI\u0026rsquo;s deep research mode Built-in web search Extended reasoning (up to 1 hour timeout) Usually generates 10-12 hypotheses even when you ask for 5 What happened The good news Multi-agent was consistently faster than o4-mini-deep:\nDomain Multi-Agent o4-mini-deep GPT-4o-mini-k10 Multi-agent wins Autonomous vehicles 5.1 min 9.1 min 2.0 min ✓ Chip design 2.9 min 9.1 min 1.6 min ✓ (tied quality) Membership inference 5.8 min 14.1 min 2.4 min ✓ Multi-agent trading RL 9.1 min 11.6 min 1.3 min ✓ (tied quality) Test-time scaling 11.4 min 14.6 min 2.7 min ✓ GPT-4o-mini-tools-k10 was fastest (~2 min average) but had lower quality scores. Multi-agent won or tied in all 5 domains on overall quality.\nThe cost story Here\u0026rsquo;s where it gets interesting. o4-mini-deep costs about $0.76 per run on average:\nAutonomous vehicles: $0.70 Chip design: $0.61 Membership inference: $0.84 Multi-agent trading RL: $0.70 Test-time scaling: $0.88 Multi-agent? $0.033 per run on average (23x cheaper!). Here are the actual costs:\nAutonomous vehicles: $0.025 Chip design: $0.017 Membership inference: $0.032 Multi-agent trading RL: $0.039 Test-time scaling: $0.051 The multi-agent approach makes 32-36 agent calls per run (the pipeline is more granular than the 11 agent types suggest). But even with more calls, it\u0026rsquo;s still 23x cheaper than o4-mini-deep.\nGPT-4o-mini baseline is even cheaper at ~$0.015 per run, but quality is inconsistent.\nSo you\u0026rsquo;re paying a massive premium for o4-mini-deep\u0026rsquo;s thinking time - nearly $0.70 more per run.\nWhat each approach actually produces Multi-agent output The multi-agent system gives you a very structured journey through the hypothesis generation process:\nPhase 1: Maps out 4-6 research areas with key papers and methods Phase 2: Identifies ~20 opportunities (gaps, contradictions, transfer opportunities) Phase 3: Synthesizes 10-12 hypotheses from those opportunities Phase 4: Evaluates each on feasibility, impact, novelty, resources Phase 5: Ranks them with clear reasoning The output is readable and traceable. You can see which gaps led to which hypotheses. Each phase builds on the previous one. It\u0026rsquo;s like watching a researcher\u0026rsquo;s thought process.\nGPT-4o-mini with tools This one is\u0026hellip; variable. Sometimes it does a great job fetching papers and reasoning about them. Sometimes it ignores the tools entirely and just wings it.\nThe outputs are less structured - you get hypotheses but the literature mapping is superficial, the gap analysis is handwavy, and sometimes whole sections are missing or incomplete.\nBut it\u0026rsquo;s fast (2-6 minutes) and cheap ($0.01-0.02), so if you need to iterate quickly or don\u0026rsquo;t need comprehensive analysis, it works.\no4-mini-deep-research This is the kitchen sink approach. o4-mini just\u0026hellip; goes. For 10-15 minutes. Searching the web. Reasoning. Thinking.\nThe outputs are incredibly comprehensive:\nExtensive literature synthesis 10-12 hypotheses (often exceeds your request) Deep feasibility analysis with \u0026ldquo;obstacles\u0026rdquo; and \u0026ldquo;enabling factors\u0026rdquo; Detailed resource estimates (funding, personnel, compute, duration) Grant fit suggestions (!) But it\u0026rsquo;s also kind of overkill for many tasks. For the protein folding benchmark (5 papers, simple domain), it took 14 minutes and generated a massive report when a 5-minute analysis would have sufficed.\nInteresting patterns we noticed o4-mini-deep over-generates Ask for 5 hypotheses, get 12. Every time. It just keeps going. The deep research mode appears optimized for thoroughness over following instructions precisely.\nMulti-agent is predictable Every domain took 6-8 minutes. Very consistent. The sequential pipeline has pretty stable runtime characteristics.\nDomain complexity matters for runtime Simpler domains (chip design): Multi-agent was 3.1x faster than o4-mini-deep Complex domains (membership inference): Multi-agent was 2.4x faster Very complex (multi-agent trading RL): Multi-agent was 1.3x faster\nMulti-agent scales more predictably with domain complexity than o4-mini-deep.\nQuality metrics comparison Beyond speed and cost, we wanted to compare the quality of hypotheses generated by each approach. Here\u0026rsquo;s what we found by analyzing the structured outputs:\nDeepEval quality metrics (scientific novelty, rigor, completeness) We used DeepEval to measure three key dimensions: scientific novelty, scientific rigor (methodology completeness), and report completeness. Here\u0026rsquo;s how they compare across domains:\nDomain Winner Multi-Agent o4-mini-deep GPT-4o-mini-k10 Autonomous vehicles Multi-agent 0.95 (novelty: 0.84) 0.91 (novelty: 0.74) 0.91 (novelty: 0.73) Chip design Tied 0.93 (novelty: 0.80) 0.93 (novelty: 0.80) 0.93 (novelty: 0.79) Membership inference Multi-agent 0.94 (novelty: 0.82) 0.92 (novelty: 0.77) 0.92 (novelty: 0.75) Multi-agent trading RL Tied 0.92 (novelty: 0.91) 0.92 (novelty: 0.76) 0.92 (novelty: 0.75) Test-time scaling Multi-agent 0.95 (novelty: 0.85) 0.91 (novelty: 0.72) 0.92 (novelty: 0.77) Key findings:\nMulti-agent won or tied in all 5 domains with overall scores of 0.92-0.95.\nMulti-agent consistently achieved higher novelty (0.80-0.91 vs 0.72-0.83) - the phased approach with dedicated gap analysis and method transfer agents helps discover novel research directions.\nAll approaches achieved perfect rigor and completeness (1.00) except multi-agent trading RL where multi-agent scored 0.86 on rigor.\nHypothesis quality characteristics Multi-agent strengths:\nHypotheses are well-grounded in the literature - each hypothesis explicitly references gaps, contradictions, or method transfer opportunities identified in earlier phases Clear methodology - every hypothesis includes detailed validation plans, datasets, metrics, and experiments Consistent structure - all 5 hypotheses follow the same format with rationale, prior work, novelty scores, and risk assessment Balanced risk distribution - typically 1-2 high-risk, 2-3 medium-risk hypotheses o4-mini-deep strengths:\nMost comprehensive literature synthesis - deeply integrated web search produces rich context with citations Bold, creative hypotheses - more willing to propose cross-domain transfers (e.g., protein folding techniques → chip design) Detailed resource planning - includes funding estimates, personnel requirements, equipment needs, grant fit suggestions Extensive feasibility analysis - explicitly lists \u0026ldquo;obstacles\u0026rdquo; and \u0026ldquo;enabling factors\u0026rdquo; for each hypothesis GPT-4o-mini characteristics:\nFaster iteration - produces hypotheses quickly but with less depth Variable quality - sometimes excellent, sometimes superficial depending on tool usage Less structured analysis - gap analysis and literature mapping are often incomplete Good for brainstorming - useful for generating initial ideas that can be refined Output structure comparison Multi-agent provides:\n4-6 research areas mapped with key papers, methods, limitations ~20 opportunities (gaps, contradictions, method transfers, assumption challenges) 10-12 synthesized hypotheses Detailed evaluation on 4 dimensions per hypothesis Final ranking with clear reasoning o4-mini-deep provides:\n10-12 hypotheses (often exceeds requested amount) Rich literature context with web citations Extensive feasibility breakdowns Resource estimates (funding, personnel, compute, duration) Grant fit suggestions GPT-4o-mini provides:\n5-8 hypotheses (more variable) Basic literature references (when it uses tools) Simple feasibility notes Less detailed methodology Which approach produces the \u0026ldquo;best\u0026rdquo; hypotheses? This depends on your criteria:\nFor research proposals and grant applications: o4-mini-deep wins - its detailed resource planning and comprehensive literature synthesis are exactly what you need For systematic research pipelines: Multi-agent wins - the traceable, phased approach makes it easy to understand why each hypothesis was generated For rapid ideation and iteration: GPT-4o-mini wins - fast enough to explore multiple angles and refine ideas interactively In our benchmarks, we found multi-agent and o4-mini-deep produce comparable quality (0.75 vs 0.73 overall scores), but with different strengths. Multi-agent is more systematic and grounded; o4-mini-deep is more creative and comprehensive.\nThe 2x speed advantage and 19x cost advantage of multi-agent makes it the better choice for most scenarios, but o4-mini-deep\u0026rsquo;s quality edge in novelty and comprehensiveness justifies its use for high-stakes research planning.\nConclusion If we were starting a new project today, we\u0026rsquo;d use multi-agent as the default. It won or tied in all 5 domains (0.92-0.95 overall scores), achieved consistently higher novelty (0.80-0.91), and costs 23x less than o4-mini-deep ($0.033 vs $0.76 per run).\nThe economics are compelling:\n100 runs: Multi-agent costs $3.30, o4-mini-deep costs $76 1000 runs: Multi-agent costs $33, o4-mini-deep costs $760 At scale, that\u0026rsquo;s real money. And you\u0026rsquo;re actually getting better quality.\nWe\u0026rsquo;d use o4-mini-deep sparingly, for domains where its comprehensiveness justifies the cost. In chip design it tied with multi-agent (0.93), but for most domains multi-agent\u0026rsquo;s higher novelty and lower cost make it the better choice.\nThe multi-agent approach isn\u0026rsquo;t magic - it\u0026rsquo;s just careful prompt engineering, good tool usage, and workflow design. But those details matter. A lot.\nBuilt with Nova: Our Production-Ready Agent Runtime This multi-agent system was built using Nova, our distributed agent runtime that makes it easy to orchestrate multiple AI agents at scale. Nova handles all the complexity of:\nMulti-agent coordination - 11 specialized agents working in a 5-phase pipeline Resource management - Automatic allocation and container isolation State management - Built-in persistence across agent executions Tool integration - Native MCP support for arXiv and Semantic Scholar APIs Observability - Full execution tracking and debugging The entire hypothesis generator runs on Nova\u0026rsquo;s orchestration layer, which is why we could iterate quickly and run hundreds of benchmarks without infrastructure headaches.\nWe\u0026rsquo;re open-sourcing Nova soon! If you want early access to build your own multi-agent systems like this, reach out to us at hello@soracloud.ai.\nThanks for reading! Questions/comments welcome.\n","permalink":"https://luarss.dev/posts/multi-agent-systems-23x-cheaper/","summary":"Multi-agent systems won or tied in all 5 scientific domains with higher novelty scores and 10-30x lower cost than deep research models.","title":"Multi-Agent Systems Are 23x Cheaper Than Deep Research Models with Higher Quality"},{"content":"I\u0026rsquo;m an AI \u0026amp; Systems Engineer at NUSX Labs, part of NUSX. I build LLM systems and agents, and the cloud infrastructure they run on. I usually own an internal platform end to end, from design through to production and daily use. Recent work includes a claims approval system, our data pipelines, an LLM evaluation harness, and zero-trust access across our services.\nPreviously I was a Founding Engineer at SoraNova, an AI Engineer at Espressif Systems, and a Graduate Research Assistant at NUS where I published a Best Paper Award-winning paper at IROS 2021.\nExperience AI \u0026amp; Systems Engineer · NUSX Labs Apr 2026 – Present · Singapore\nAgents \u0026amp; evals Built Arbiter, Gemini agents that screen startup applications (forms, pitch decks, videos) against programme rubrics: about 950 so far across Deep SAGE, BLOCK71 and AI Accelerate. A full BLOCK71 intake cost under $4 in model calls Built its eval harness: golden datasets from 1,000+ real panel decisions across 4 cohorts, scored on held-out folds against how often human reviewers agree. On the Deep SAGE first round the AI\u0026rsquo;s shortlist matched the panel at F1 0.82 vs 0.76 between two human reviewers Used the evals to pick models and prompts per round: Flash for first rounds (higher AUC than Pro at a quarter of the cost), Pro for later ones, and a sceptical deck prompt that removed a consistent bias in team scores Ran a three-arm study for CRM company matching on a 143-row gold set with planted traps. Monday\u0026rsquo;s built-in AI let 31 bad matches per 100 through; a fuzzy-match pipeline with Slack approval let 1 through, so we shipped it Hand-audited 466 page citations from an existing AI judge: 86% pointed at the wrong page and 3% were made up. Replaced it with our own document processor Internal platforms Built Purser, a full-stack claims and approval platform (Next.js, tRPC, Postgres) with role-based access and encrypted payee data. Claims go from senior approval straight into the university payment system, so nobody re-keys them Wrote a deterministic browser agent, with a human approval step, that fills the payment form in ~90s vs 20+ min for a general-purpose LLM agent Built a BI request tracker on Cloud Run (dashboard, JSON API, MCP server) and moved 117 legacy requests into it, so agents can search one register. Mean turnaround dropped from 3.8 → 0.2 days Infrastructure \u0026amp; data Own NUSX\u0026rsquo;s GCP infrastructure: Terraform across 6 projects, 21 Ansible roles, keyless CI/CD and zero-trust access. Also ran the production launches and decommissioned the legacy estate Run NUSX\u0026rsquo;s data platform: scheduled Cloud Run jobs and n8n workflows pull from about 20 sources (Salesforce, Monday, ACRA, PitchBook, finance systems) into a GCS data lake that feeds Power BI, Slack approval flows and our agents Run a self-hosted status page that alerts Slack for 10 internal and 5 external components Founding Engineer · SoraNova Jun 2024 – Apr 2026 · Singapore\nOwned the full build of an AI knowledge platform (FastAPI backend, multi-agent AI workflows, React frontend) from design to production Engineered a production RAG system with hybrid retrieval (PostgreSQL pgvector + keyword), improving answer relevance 25% through semantic search and query caching Designed a multi-agent research system orchestrating 5 specialised LLM agents (factual, technical, market, team, risk) for VC investment research reports Implemented an ML evaluation framework with DeepEval: 15+ metrics across completeness, accuracy, relevance and quality Owned the DevOps pipeline: GitHub Actions CI/CD, Docker multi-stage builds, Packer GPU AMI workflows (T4/L4/A100), Ansible and Harbor container registry AI Engineer · Espressif Systems Sep 2022 – May 2024 · Singapore\nImplemented ML algorithms (Graph Neural Networks, RL, Bayesian Optimisation, Simulated Annealing) for chip design, saving 20% time vs. manual baseline Developed an AI evaluation framework across 175+ Verilog benchmarks using Pass@K, BLEU and Levenshtein metrics Managed distributed RL training infrastructure with 8× NVIDIA GPUs Graduate Research Assistant · NUS Aug 2020 – Aug 2022 · Singapore\nApplied deep RL to real-world robotic applications, outperforming classical control baselines Published paper at IROS 2021: Best Paper Award Open Source Open Source Contributor · OpenROAD May 2023 – Present\nOpenROAD is the leading open-source, foundational application for semiconductor digital design (github.com/The-OpenROAD-Project/OpenROAD)\nAutomated generation of ~3000 Linux manpages by parsing regexes from existing documentation, with CI and regression tests in Tcl and Python to keep the style consistent Led the LLM chatbot assistant project (ORAssistant) with 2 students, building modular components for data pipeline, fine-tuning, retrieval, evaluation and inference. Published at WOSET 2024 Led productionisation of the OpenROAD AutoTuner, a Ray Tune wrapper for automatic hyperparameter tuning in the OpenROAD-flow: set up a test framework and distributed autoscaling via Ray Cluster Google Summer of Code mentor in 2024 and 2026. This year I\u0026rsquo;m mentoring a student on openroad-mcp, an MCP server that lets chip designers debug OpenROAD runs with an AI assistant in real time Education M.Sc. Computer Science · National University of Singapore Aug 2020 – Oct 2022\nFocus: Machine Learning, Deep Learning, Artificial Intelligence IROS 2021 Best Paper Award B.Eng. Electrical \u0026amp; Electronic Engineering · Nanyang Technological University Aug 2016 – Aug 2020\ncGPA: 4.78 / 5.00 Skills AI \u0026amp; LLM - RAG (pgvector, hybrid retrieval) · agents (LangGraph, MCP) · LLM evals · human-in-the-loop · Gemini / Vertex AI · Python\nData - n8n · BigQuery · GCS · Power BI · Cloud Run jobs\nCloud \u0026amp; DevOps - GCP · AWS · Terraform · Ansible · Docker · Kubernetes · zero-trust access · GitHub Actions\nFull Stack - TypeScript · Next.js · React · tRPC · FastAPI · PostgreSQL\nML Research - PyTorch · deep RL · GNNs · Ray Tune · OpenROAD · Tcl\nContact GitHub: github.com/luarss LinkedIn: linkedin.com/in/song-luar Email: me@luarss.dev ","permalink":"https://luarss.dev/about/","summary":"I\u0026rsquo;m an AI \u0026amp; Systems Engineer at NUSX Labs working on production LLM systems, agents, evals, and cloud infrastructure.","title":"About"}]