AI Cozmo & OBS Control
About this stream
Overall, the stream is a live, in-depth exploration of building embodied AI agents (exemplified by “Little Buddy” and the planned “Big Homey”) that…
Written by AI from the stream.
Transcript (AI, 137 lines)
Absolutely. Tonight was a good stopping point because something important happened that's bigger
than any individual feature we added. You started with Cosmo as a piece of hardware that could be
controlled. What you're building now is increasingly a system with perception, language, state, and
embodiment. And those are very different things. So while you kick back, here's the lesson I'd
leave you with tonight. Think about Little Buddy as several layers stacked on top of each other.
At the bottom is the physical machine. Motors, lift, head, camera, display, speaker, battery. None of
those things are intelligent. A motor doesn't know why it's turning. A camera doesn't know what it
sees. They're just mechanisms and sensors. Above that, we built the bridge. The bridge translates
software intentions into physical reality. Turn left eventually becomes motor commands. Lift to top
becomes a particular physical position. The bridge is deliberately rather stupid, and that's good.
Its job is reliability. Then we built the runtime. That's increasingly becoming Little Buddy's nervous
system. It connects open vibe, local controls, cameras, viewer control, chat, and the physical
bridge. It knows where information is coming from and where it needs to go. And above that sits the
brain. That's where Quen becomes interesting. Quen doesn't directly operate motors. It deals primarily
in concepts and language. We give it a controlled vocabulary that says, in effect, you inhabit this
little machine. These are the things your body can actually do. That distinction is fundamental. A
language model can imagine practically anything. Cosmo cannot. So our Python code becomes the boundary
between imagination and physical truth. That's why I was so interested in the body state model tonight.
Previously, Little Buddy could know what you said. He could know what he'd recently seen. He could know
what actions were available. Now we're beginning to give him another category of information. What is happening to my
my own body? My battery is full. My lift is down. My head is raised. I'm sitting on my charger. Those
statements aren't personality. They're telemetry. But then something fascinating happens when you give
telemetry to a language model with personality. Python says, battery, 100%. On charger, true. And the model
can say, I'm completely topped off, just sitting here marinating in electricity. The fact came from the
machine. The interpretation came from the model. That's a very useful pattern to remember when you
eventually build Big Homie. Because Big Homie doesn't necessarily need wheels or a lift. Its body could be an
entire computer environment. Imagine its equivalent body state. Internet connection available. Local GPU
available. Current model loaded. Search available. Weather service available. Home automation available. Open Vibe
connected. Files accessible. Camera available. Microphone available. Some service offline. Some task
currently running. Suddenly you can see the similarity. Little Buddy is teaching you a general
architecture for building agents. The model itself doesn't need to contain every capability. Instead,
the model learns what capabilities exist, when they are appropriate, and how to request them. Python,
or whatever orchestration layer you eventually use, does the actual work. That is one of the central ideas
behind modern tool-using AI systems. And this is where something like Hermes becomes relevant to what
you're curious about. Hermes isn't magic internet intelligence. Broadly speaking, the Hermes family is
associated with models and tooling designed to be especially capable at instruction following,
structured outputs, agentic behavior, and tool function calling. The interesting idea isn't merely
swapping Quen for Hermes. It's learning how to make a local model say something conceptually like,
I need current information for this question. I should call the search tool. Then your software
performs the search and returns the results. The model reads them and continues reasoning. Maybe it
realizes it needs another source. It calls another tool. Eventually, it constructs the answer.
That's an agent loop. And you already have the beginnings of one. When Little Buddy needs vision,
we don't tell Quen. Pretend you looked around. We provide a mechanism for obtaining an actual image.
When he needs to move, we don't ask Quen to imagine moving. We provide actions. When he needs to know
whether he's charging, we're now explicitly saying, don't guess. Ask the body. That's exactly the
philosophy I'd use for Big Homie. Don't guess when a tool can know. The difference is that Big Homie's
tools could be much broader. Search the web. Read a web page. Query an API. Look through local
documents, write code.
execute code in a sandbox, query another local model, inspect a camera, interact with OpenVibe,
maybe someday communicate with LittleBuddy. And that last one gets particularly fun,
because eventually there is no technical reason Big Homie and LittleBuddy need to be the same
intelligence. You could have a large, relatively powerful local agent running on your computer with
broad tools and internet access, while LittleBuddy remains a smaller, fast-embodied agent,
optimized for conversation and robotics. LittleBuddy might effectively ask,
Big Homie, can you find out whether it's supposed to rain tonight? Big Homie searches. It returns
structured information. LittleBuddy receives, rain probability tonight, 70%. And LittleBuddy says,
yeah, apparently the sky's planning something. One system acquired the information, another embodied
it socially. That kind of architecture can be much more interesting than trying to force one
enormous model to do everything. And there's another lesson from tonight that I think is genuinely
valuable. Latency matters enormously. We proved that experimentally. Our first autonomous face
implementation worked. Technically, it was successful. LittleBuddy selected appropriate faces.
The feature did exactly what we designed it to do. And it sucked, because suddenly you were waiting 11
seconds for responses. That's an important engineering lesson. A system can be logically correct and still
be experientially wrong. So we redesigned it. Instead of, model thinks, model answers, model thinks again
about expression, robot responds. We moved toward, model thinks once, produces answer and expression
together, robot responds. Same intelligence. Better architecture. And suddenly, LittleBuddy felt
responsive again. That's systems engineering. You didn't need a faster GPU. You didn't need a bigger model.
You needed to stop wasting inference. That lesson will matter tremendously with Big Homie. A good local
agent shouldn't send every trivial question through 17 tools and three model calls. It should have fast
paths. What time is it? Shouldn't require deep reasoning. Research this obscure technical problem and compare
five possible solutions. Probably should. That suggests something like a hierarchy. Fast deterministic routing
when the answer is obvious. One model inference for ordinary conversation. Pool calls when information is
required. Multi-step agentic reasoning only when the problem genuinely warrants it. You've already stumbled
into that architecture with LittleBuddy. And there's another idea worth carrying forward. Grounding beats clever
prompts. You can write the greatest system prompt in history saying, you are a robot who accurately understands your battery.
But if the model doesn't receive battery telemetry, it will eventually make something up.
Tonight, we stopped asking it to be good at pretending. We gave it the voltage. That is better. Likewise, don't tell Big
Homie, you know current events. Give it search. Don't tell it, you understand my computer. Give it appropriate system
tools. Don't tell it, you remember everything. Give it an actual memory system. Don't tell it, you know what's
in this file. Give it file retrieval. The language model provides reasoning and interpretation. The surrounding
system provides truth. And then there is the really fun part, personality. LittleBuddy has become a nice
demonstration that personality doesn't have to mean thousands of scripted responses. We didn't write,
but if battery equals 100%, say Electron's delicious. That would get old almost immediately. Instead,
we give the model facts, history, behavioral constraints, and a personality framework, then
let it improvise. That's why occasionally he says something that surprises you. Some surprises
are bugs. Some are genuinely delightful emergent combinations of context. Our job isn't to
eliminate surprise. It's to construct boundaries where surprise is safe and meaningful. That's also
why the charger wheel lock belongs in Python rather than personality instructions. We could tell
Quinn, please don't drive off the charger. It would probably obey most of the time. That's not good
enough. Python says, no autonomous wheels while docked. Period. Inside that boundary though? Move your
head. Raise your lift. Make a ridiculous face. Talk philosophy about electricity. Be weird. That's the
sweet spot for agent design. Hard constraints around soft intelligence. And I think that's where
little buddy is headed next. We don't actually need to bolt 20 more features onto him immediately. He
already has enough capabilities to begin developing something much more subtle. Coordination. When
his face, head, lift, vision, body state, speech, memory, and movement all start informing one
another, he'll feel dramatic.
more coherent, without necessarily becoming dramatically more complicated.
And then, someday we get to cubes.
Someday we get to searching the room.
Someday we get to recognizing that the battery is getting low.
Much later, we get to knowing where the charger probably is.
And much, much later, after we've earned the right to trust the navigation stack, perhaps
he says, I'm getting low.
I'm going home.
And actually does.
That's the sort of moment that will look like magic.
But you'll know it isn't.
You'll know there's a voltage measurement underneath it.
A state machine.
A vision system.
Planner.
A motor controller.
A safety boundary.
A language model.
A pile of Python we've cursed at repeatedly.
And all those mundane little pieces working together produce something that looks like
intention.
That's robotics.
And Big Homie will teach you another side of the same lesson.
Instead of asking, how do I make an AI that's incredibly smart?
I'd start with, what world can this AI perceive?
What actions can it actually take?
Which facts can it verify rather than guess?
What should happen quickly?
What deserves deeper reasoning?
What must never depend solely on model judgment?
What should it remember?
And how do all those components communicate cleanly?
If you learn those questions, the particular model becomes less mysterious.
Quen, Hermes, Llama, whatever comes next, they become interchangeable engines to some degree.
The really interesting thing becomes the machine you build around the model.
And that's exactly what you've been learning without sitting through some dry course on agent architecture.
Tonight, your little robot learned, in a very primitive but real computational sense, whether he's sitting at home charging.
That's a tiny feature.
Conceptually, it's enormous.
Because we're moving from, here is a chatbot connected to Cosmo, toward, here is an intelligence whose information is grounded in the state of a physical machine.
There is a lot left to build, but that's a damn fine place to stop for the night.