Studio XID · Design & development (solo)
Decision Graph
- LLM Pipeline
- Eval
- Cost Optimization
Problem
Decisions are made all over Slack and Notion, but later it was hard to find who decided what and why, along with the evidence — and the project's overall flow of decisions was hard to see. Extraction would be automated with an LLM, but it had to be trustworthy — no invented decisions or missing conditions — and cheap and fast enough to run every day.
Key decisions
- 01Decisions as a graph, not a list — the system links prerequisite, replacement and stop relations between decisions and references the source text, so the project's decision flow is quick to grasp
- 02Only decisions from what was said are raised as candidates, and nothing goes into the record until a person confirms it
- 03Evidence is cited by source sentence number and checked twice — by code and by a separate verification stage
- 04Checked quality on every change against a golden set with human-marked answers and 24 test cases built from common failure cases (e.g. missing conditions)
- 05Two-tier inference — a lightweight model filters out small talk first, and only decision extraction and verification go to the high-capability model (switching everything to the lightweight model was rejected after it passed only 17 of 24 test cases)
- 06Content analyzed once is stored, and only conversations that changed are analyzed again
Outcome
Compared against a golden set where people manually marked the correct decisions (and evidence) on one real project from the past year, and confirmed it correctly extracted 80%+ of the human-labeled decisions. Extracted decisions are drawn automatically as a relation map and timeline. While keeping quality — all 24 test cases passed, with no decisions wrongly excluded — cut calls by 74% (1,289 → 336) and run cost by 25% ($10.2 → $7.6) on the same data. Re-running only the conversations that changed costs $0.7.
As of the internal alpha stage.
From scattered conversations
to decisions with evidence
Decision relation map drawn automatically by the system — prerequisite, replacement and stop relations and statuses between extracted decisions (decision text blurred: internal data)
Decision timeline drawn automatically by the system — decision level (company, project, feature, task), timing and milestones
Details
- Project name
- Decision Graph (LLM pipeline for extracting decisions from Slack and Notion)
- Client · Domain
- Studio XID / New product PoC (internal alpha)
- Period
- 2026.08 - 2026.10
- Scope of work
- Hands-onTurned PO requirements into a technical spec and validated technical feasibility with a PoC — new product exploration
- Hands-onDesigned and developed, solo, an LLM pipeline that extracts who decided what and why from Slack and Notion, with quoted source evidence
- Hands-onVisualized extracted decisions automatically as a relation map and timeline — prerequisite, replacement and stop relations linked by the system, with each decision linking straight to its source text
- Hands-onValidated the PoC against a golden set where people manually marked the correct decisions (and evidence) in the conversations of one real project from the past year — extracted 80%+ of the human-labeled decisions
- Hands-onVerified and reviewed the technical risks raised during the PoC myself (e.g. judging quality against human-made ground truth, one-way recording from source → decision → document, preventing duplicate entries on re-runs)
- Hands-onDesigned a multi-stage pipeline: split into analysis units → first-pass classification (lightweight model) → decision extraction → evidence verification (code + model) → duplicate and relationship analysis
- Hands-onTwo-tier inference — compared models on the same data and the same tests, then chose a lightweight model for first-pass classification and a high-capability model for extraction and verification
- Hands-onLogged latency, queue time, retries, cache hits, tokens and cost per request to find bottlenecks and optimize the call structure
- Hands-onImplemented a content-hash cache, incremental runs (re-processing only what changed), batching by conversation and thread, sentence-number citations, and a scheduler that overlaps stages
- Hands-onUsed 24 test cases with known correct answers to check that quality didn't drop with every prompt or model change (with a cap on evaluation cost)
- Role
Design & development (solo)
- Tech stack
- Node.js
- React
- TypeScript
- OpenAI API
- LLM Pipeline
- Eval
- Telemetry