Case 01 · Multimodal agent
Hermes Live Companion
A human-governed multimodal companion that connects voice, vision, memory, tools, and interruption control in one local operating surface.
Project snapshot
A local companion with explicit human control.
Delivery Loop trail
The case moved through four evidence gates.
The loop did not remove human judgment. It made each transition—from question to product to release—visible and reversible.
Understand
- Future-device question
- Real laptop capabilities
Design
- Product contract
- Safety and non-goals
Deliver
- Text-to-voice slice
- Vision and interruption
Verify
- Runtime evidence
- Public delivery
01 · Operational problem
Can today's devices deliver part of the AI-hardware future?
The project began during a conversation about future AI hardware. A laptop already had a microphone, speakers, camera, screen, network, and compute. Hermes already had sessions, memory, tools, and vision.
The gap was not another model. It was a governed real-time interaction layer that could connect those capabilities without taking over the existing system.
Can the devices and agents already on hand do it first?
02 · Product hypothesis
One session could connect six governed capabilities.
The MVP would be credible only if inputs, reasoning, outputs, status, and human controls remained visible together.
Voice loop
Microphone input, recognition, streaming response, and TTS.
Vision
User-approved camera captures and scene analysis.
Memory and sessions
Multi-turn context with explicit reset and stop controls.
Tools and approvals
Existing Hermes tools with visible progress and boundaries.
Interruption control
Keyboard, button, and acoustic barge-in keep the human in charge.
Local-first safety
Loopback services, temporary tokens, bounded files, and cleanup rules.
03 · Technical path
A visible local runtime, not a black box.
Voice and vision enter through controlled channels. A dedicated Hermes worker coordinates memory and tools, then streams text and speech back to the user.
Bounded locally
Browser and services remain on 127.0.0.1.
Isolated worker
The companion does not take over the regular Hermes desktop environment.
Temporary inputs
Camera and microphone data are treated as bounded runtime inputs.
Visible controls
Progress, approvals, stop, reset, and interruption remain inspectable.
04 · AI delivery
Speed came from governance, not one-click generation.
A detailed product plan reduced guessing. Narrow phases and explicit checks let Codex execute quickly while the human retained direction, risk approval, and release judgment.
Plan-driven
Define experience, boundaries, non-goals, safety, and acceptance first.
Phase-based
Prove the smallest complete chain before adding new modalities.
Human checkpoints
Direction, priority, risk, and readiness remain human decisions.
Rollback-first
Isolation, pinned dependencies, and Git checkpoints keep changes reversible.
05 · Real constraints
The build happened in fragments, then had to survive public delivery.
The 8h 46m figure is active development time across two days—not total elapsed time. The local prototype also needed cleaning, scanning, documentation, packaging, and a public evidence trail.
- 01
Question and product concept
- 02
Plan and safety boundaries
- 03
Text and session foundation
- 04
Voice and interruption
- 05
Vision and approvals
- 06
Packaging and security review
- 07
Demo and Build Week submission
06 · Verification
The interface makes agent activity inspectable.
Conversation, camera analysis, microphone status, runtime settings, and interruption controls remain visible in one operating surface.
- Voice, vision, and memory in one session
- Visible runtime and model status
- Human-accessible controls throughout
07 · Reusable capability
Shipping became part of the product method.
The durable output was more than the companion. It was a repeatable way to govern agent development from definition through public evidence.
- 01
Clean and isolate the public package
- 02
Test, scan, document, and rebuild
- 03
Prepare the demo and public evidence
- 04
Submit through Devpost
08 · Limits and public boundary
The evidence is public. The private system is not.
The demo and Devpost page verify the public result. The source repository, credentials, local configuration, machine paths, and unpublished development records remain private.
Demo video
Verifies voice, vision, memory, visible runtime state, and interruption controls.
Watch ↗Devpost
Verifies the public OpenAI Build Week submission and delivery context.
Open project ↗Related work
The method changes shape across domains.
Explore how the same emphasis on constraints and evidence appears in visual reconstruction and offline family operations.
The larger lesson
AI accelerates the build. Human judgment makes it trustworthy.
Human value moves upstream: finding the problem, defining the system, controlling risk, and deciding when the result is ready for the real world.
A similar operational constraint?