Real-time voice AI learning product
ZeroPoint
A browser-based voice AI dance tutor for K-pop choreography.
MVP tested before a product pivot · Co-built with a co-founder
Stack
Overview
ZeroPoint was a browser-based voice AI dance tutor. It taught K-pop choreography using pre-recorded instructional clips and a real-time voice AI coach.
It was an MVP built to test product-market fit. About 100 testing sessions were completed before the product pivoted. Two co-founders built it, and the source repository is private, so this page focuses on architecture and decisions.
Product and problem
Our working hypothesis was that K-pop fans who do not dance get stuck because online tutorials move too fast, classes feel intimidating, and it is unclear where to start. The app teaches one routine part by part with a voice-driven coach that can slow down and repeat sections on request. This was a hypothesis we set out to test, not a validated finding.
The engineering problem was coordination. A voice conversation, video playback and the on-screen UI are all asynchronous and independent, yet the learner experiences them as one lesson. If a language model can trigger actions in that system directly, the lesson can drift out of sync or jump to a state the product never intended.
My role
I co-built ZeroPoint with my co-founder. My focus was the state architecture that keeps voice, video and UI coordinated, the LLM tool layer, and the AWS telemetry pipeline.
What I worked on
- SessionController architecture and the transition state machine.
- LLM tool handlers and the deterministic validation around them.
- ElevenLabs agent and tool configuration, including prompt and tool setup.
- Frontend and product flows in React and TypeScript.
- A serverless telemetry pipeline (API Gateway, Lambda and S3) for remote session debugging, including a Slack alert when a new session starts.
- Contributed to CI/CD deploying development and production builds to S3 and CloudFront using GitHub Actions and AWS OIDC.
- Technical specifications and planning.
Shared or owned by my co-founder
- Shared with my co-founder: prompts and plan generation, remote log capture, and the lesson phase registry.
- Mostly my co-founder's: the lesson engine and voice-agent modules, volume control and voice configuration. The application's store architecture was also my co-founder's redesign, and my state modules build on it.
Key engineering challenges
Keeping voice, video and UI in step
- Problem
- Speech, clip playback and screen transitions each run on their own timing. Left to coordinate themselves, they race: the coach can talk about a step the video has not reached, or the UI can advance while the coach is mid-sentence.
- Approach
- A SessionController owns application state and coordinates the three. Transitions are modelled as explicit state machines, and the React UI reflects that state.
Letting an LLM act without letting it control state
- Problem
- A voice agent needs to do things, such as move the lesson forward or start a clip, but a language model can call the wrong action or call it at the wrong moment.
- Approach
- The model selects actions through tools. Deterministic TypeScript handlers validate every call against the current phase before anything changes: calls that are not valid now are refused or deferred, and structured outputs are validated. The model is treated as an untrusted controller.
Staying reliable in a real-time voice loop
- Problem
- Real-time voice depends on network calls and third-party services that can be slow or fail, and the agent needs to know what is happening in the lesson to respond sensibly.
- Approach
- Operations use timeouts, retries and fallbacks. The agent's prompt context is updated from the application's current state rather than assumed to be static.
Seeing what happens in test sessions
- Problem
- MVP testing needs visibility into what happens during sessions that run in someone else's browser, where problems cannot be reproduced locally.
- Approach
- I built a serverless telemetry pipeline on API Gateway, Lambda and S3 for remote session debugging, alongside Sentry and PostHog. It also posts a Slack alert when a new session starts, skipping internal sessions.
Architecture
The core of the system is a control loop in which the model proposes and deterministic code decides. In this MVP the browser connected directly to the voice provider's agents, so the loop ran entirely client-side. The diagrams are simplified and sanitized.
- LearnerVoiceBrowser · WebRTC
- LLM agentElevenLabs voice agentGemini LLM · selects actions via tool calls
- Deterministic TypeScriptTool handlersValidate against current phase · refuse or deferSessionControllerOwns state · transition state machines
- ExperienceVideo playbackReact UI
Application state also flows back to the agent as updated prompt context.
- ClientReact app in the browser
- AWS entryAPI Gateway
- IngestionLambda
- OutputsS3Session logsSlackNew-session alert
Selected technical decisions
Treat the LLM as an untrusted controller
- Decision
- The model can only request actions through tools; TypeScript owns and validates application state.
- Trade-off
- More code: every tool needs a handler with validation, and the conversation has to cope with refused or deferred actions. In return, an unexpected model response cannot push the lesson into an invalid state.
Gate actions by phase, and defer when needed
- Decision
- Tool calls are checked against the current phase; actions that arrive too early can be deferred until the right transition rather than dropped or forced.
- Trade-off
- The controller carries more state and more transitions to test. The benefit is that timing mistakes by the model become handled cases instead of visible glitches.
One controller instead of components coordinating themselves
- Decision
- A single SessionController is the place where voice, video and UI ordering is decided.
- Trade-off
- It becomes a central module that must stay well structured. It also gives one place to reason about, and one place to test, how a lesson progresses.
Authenticate CI to AWS with OIDC
- Decision
- The GitHub Actions pipeline that deploys development and production builds gets AWS access through OIDC rather than stored access keys.
- Trade-off
- The IAM trust setup takes more effort than pasting keys into a secret, but no long-lived AWS credential exists to leak or rotate.
Production and delivery
- A static browser app served through AWS S3 and CloudFront. The voice tutor and the lesson-plan generation connected from the browser straight to the voice provider's agents, so the product needed no application backend of its own. The one Lambda behind API Gateway received session logs and posted a Slack alert when a new session started.
- CI/CD in GitHub Actions deploys development and production builds to S3 and CloudFront, authenticating to AWS with OIDC.
- Sentry and PostHog provided error monitoring and product analytics during testing.
- The product was an MVP, not a scaled production service, so no uptime or load figures are claimed.
- No automated test suite was configured for this MVP.
- Development used AI coding assistance (Claude Code).
Results
- testing sessions completedSessions, not unique users
- ~100
- Testing ran before the product pivot. The number is an approximate count of sessions supplied by Lucas; it is not a user count.
What I would improve
- Move the lesson transition logic into pure, unit-testable functions and add automated tests around its timing behavior before adding more lesson modes.
- Improve telemetry delivery reliability with buffering and retry.
- Separate speech policy and state from tool wiring so the prompt layer depends on neither.
Links
The source repository for ZeroPoint is private. GitHub shows public code only.