AI Cozmo & OBS Control
About this stream
The stream centers on building and demonstrating an embodied robotic/AI agent (Quinn/Cosmo) that can see, think, and act through a grounding…
Written by AI from the stream.
Transcript (AI, 220 lines)
No problem. I'll restart the whole thought and carry it all the way through.
What you're building now is crossing an important line.
Little Buddy is no longer just a language model that can occasionally trigger a robot command.
He's starting to acquire an actual physical vocabulary.
Up until recently, Gwen's world was split into two buckets.
There were things he could talk about, and there were a handful of things Python would permit him to physically do.
Look through the camera, move his head, turn a little, change his display, speak.
And the grounding layer was deliberately strict about that.
If Python hadn't confirmed that something happened, Gwen wasn't supposed to claim it happened.
That architecture turns out to be exactly what you want now.
Because instead of throwing away the guardrails to make him more autonomous, we can expand the vocabulary inside them.
Think of it like giving someone more and more verbs.
At first, Little Buddy knew things like look, head up, head down, turn left, turn right.
The planner could choose one of those verbs.
Python checked whether it was valid.
The bridge physically executed it.
Then Python told Gwen what actually happened.
Tonight, we expanded that language.
Now he has forward and backward.
And the interesting part wasn't simply that the tracks moved.
The interesting part was that when you said move back, Gwen's planner independently selected backward.
That means the model is beginning to reason about locomotion as something its body can do, rather than just repeating a button press somebody asked for.
And that distinction is huge.
The architecture I'd preserve is, Gwen decides what it wants to do.
Python decides whether that action exists and how it should be expressed safely and consistently.
So Gwen never needs to think in terms of left tread speed, 73 for 900 milliseconds.
That would be a terrible body interface for a language model.
Gwen should think, I want to turn slightly left.
Then the embodiment layer translates slightly left into a calibrated 7.5 degree movement.
That separation between intention and motor mechanics is going to become one of the most important ideas in this entire project.
The turning work today showed exactly why.
Originally, turn left was basically a mysterious hard-coded pulse.
We thought it was approximately 15 degrees.
Then you got out paper, boxes, a ruler, and what became an absurdly serious miniature robot surveying laboratory.
And we measured what the machine actually did.
Once we understood the relationship between wheel speed, duration, and physical angle, turning stopped being magic.
Now it's a physical quantity.
And then we learned something even more interesting.
Mechanically accurate 15-degree turns are probably too coarse for his normal perceptual behavior.
7.5 degrees is better because successive camera views overlap more.
That means body calibration isn't only about movement.
It directly affects perception.
If little buddy is searching for an object, turning 15 degrees between looks could skip visual information.
At 7.5 degrees, he can scan the environment much more smoothly.
So eventually he'll have multiple movement scales.
A tiny expressive turn, a normal search turn, a larger navigation turn, and explicit angles when something really matters.
Same actuator.
Different semantic meaning.
Forward and backward are going to evolve the same way.
Right now what you've given him is basically a little pitter-patter.
That's actually a good primitive.
It isn't yet, drive across the room.
It means something closer to, shift my body slightly forward.
And that's useful for expression.
Imagine somebody tells him something surprising.
He could back up one tiny step.
Someone calls him over.
He could move forward slightly.
He wants a better camera angle.
He might move forward, look, decide that's worse, move backward, and look again.
That's where locomotion starts becoming part of cognition rather than just transportation.
Then comes the lift.
Right now, he has a tiny incremental lift motion.
We're going to make that more visible and add smooth full up and full down positions.
And the lift is deceptively important.
On Cosmo, the lift is basically an arm.
So once Quen understands that raising or lowering it changes his physical posture, he gets another
expressive dimension.
Head angle plus lift position plus body orientation gives you a surprising amount of body language.
You can have something less, head slightly down, lift lowered, body still.
Or head up, lift raised, slight turn towards that.
someone. Or, tiny reverse movement, head tilt, lift movement. None of those individually are
dramatic. Combined, they start reading as behavior. And this brings us to the part you
noticed earlier when he was narrating embodiment. Quinn already produces things like, turns slightly,
tilts head, looks up, raises lift, wiggles. Whatever little stage directions emerge naturally.
Right now, the grounding system often suppresses those because they aren't physically confirmed.
Eventually, rather than merely deleting those stage directions, we can intercept them as proposed
body intentions. That gets really interesting. Suppose the language model internally produces
something equivalent to, I tilt my head and back away a little. Instead of allowing that phrase to
become spoken text, an embodiment interpreter could extract, head down small, backward small.
Python validates those. Cosmo performs them. Then the sentence that reaches the speaker might
simply be, okay, that was weird. So the physical behavior happens instead of being narrated.
That's probably the most natural route toward the thing you've been imagining.
You aren't scripting animations. You're giving Quinn ways to physically realize expressions that it
already wants to make. And I think the next few sessions should deliberately build toward that.
First comes completing the basic body vocabulary. Forward. Backward. Fine left and right turns.
Head positioning. Lift up and down. Lift top and bottom. Procedural eyes and the static faces you
already have. Camera look. Then we make the primitives more useful. Maybe forward small. Forward medium.
Backward small. Turn small. Turn normal. Lift small. Lift top. Head glance up. Head glance down.
Backward small. That's where his dance finally becomes real. And I wouldn't necessarily make dance a
canned animation. You could certainly have a fallback routine eventually. But the more interesting version is
Gwen gets asked to dance. The planner understands that dancing means producing an expressive movement sequence
from the actions it currently has. It composes one. Python validates that the sequence is legal.
Then, Little Buddy performs his own combination. That's a completely different feeling from playing
an animation file. And because the vocabulary is bounded, the creativity happens at the behavioral level
rather than the raw motor level. Another major step after that is going to be body state awareness.
You already saw an early hint of this when he said something like "back to center" after two left
turns followed by two right turns. That's interesting because the bridge was maintaining a relative heading
estimate. Eventually, the state we feed Gwen can get richer. Not just head angle. Head near top.
Lift halfway. Body heading 15 degrees right of starting orientation. Battery moderate. Camera last looked
three seconds ago. Currently on charger. Yes or no. Then Gwen can reason about its body as a persistent state.
Instead of every command being an isolated event, the model begins each turn knowing something like "my lift
is already raised so I don't need to raise it again." Or "my head is at the top limit so if I want to look
higher I need to reposition the body." Or "I've turned right several times while searching so I should search
left now." That's a huge step toward coherent embodiment. Then there's perception. Right now look is a snapshot.
Later we can make little buddy much more deliberate about active perception. Meaning he doesn't just
passively receive a camera image. He decides how to position his body to acquire information. That might
look like "Look. I can't see the cube clearly. Head down. Look. Still not clear. Turn left small. Look. There it is."
That's embodied visual reasoning. And that's one of the places where giving a vision language model a body
becomes much more interesting than putting a chatbot inside a toy robot. The model can change the input it
receives by physically acting. That creates a loop. Perceive. Think. Perceive again. And eventually
you get a very different kind of agent than something that simply answers from a static camera. Then we
reach autonomy. I would introduce autonomy gradually. At first actions happen because you explicitly ask.
Then Quen gets permission to perform small expressive movements during conversation. A tiny turn toward
whoever is speaking. Head adjustment. A lift gesture. Maybe a tiny forward or backward movement.
Then later, limited autonomous perception.
For example, if something interesting changes in its environment, it may choose to look.
Or if it loses sight of an object it was following, it can reposition.
And eventually you have the larger autonomous behaviors.
Exploration.
Searching.
Playing with cubes.
Following people visually.
Returning to familiar areas.
Learning where things tend to be.
That's the long road.
And then we get to the endgame you keep talking about.
Charging.
Self-charging is different from almost everything else because it closes the biggest remaining
dependency loop.
Right now, little buddy can think.
He can see.
He can talk.
He can increasingly move.
But when his battery gets low, a human still has to intervene.
That means his embodiment still has a hard external dependency.
The moment he can recognize, my battery is low.
Then determine, I should go to the charger.
Then locate it.
Approach it.
Align himself.
Dock.
Verify charging.
And later leave when sufficiently charged.
That changes the system qualitatively.
Because now maintaining the body becomes part of the agent's own behavior.
And I'd build that in several layers.
First, battery awareness.
He already has access to voltage information through the bridge.
Eventually, that becomes meaningful state rather than just a number on the HUD.
Instead of merely seeing 3.58 volts, the embodiment layer gives Quinn something like
battery low.
Charging recommended.
Then charger perception.
Can he visually recognize the charger?
Can he recognize it from OBS or his own camera?
Can he identify which direction it lies?
Then navigation.
Can he turn toward it?
Approach in small controlled increments?
Correct his alignment?
And the final few centimeters are probably going to be the hard part.
That likely becomes a specialized docking behavior because fine alignment is exactly the kind
of thing Python should handle more precisely than a language model.
So again, Quinn decides, I need to charge.
A docking controller handles the actual last stage geometry.
Then Python confirms, charging connection established.
And only then does Quinn get to know, I'm charging.
That preserves the same grounding principle you've been using from the beginning.
The big end game loop eventually becomes, little buddy is interacting.
Battery gradually falls.
He notices.
He decides he should charge.
He excuses himself, perhaps.
He finds the charger.
Navigates over.
Docks.
Confirms charging.
Waits.
Battery reaches a reasonable level.
He undocks.
Returns to whatever he was doing.
And the really fun thing is that by then, none of those individual pieces will feel magical.
We'll have built each one separately.
A calibrated turn.
A forward pitter-patter.
A backward pitter-patter.
Camera look.
A lift movement.
Body state.
Visual search.
Object recognition.
Navigation.
Dock alignment.
Battery state.
Each one is simple enough to understand.
But once they're connected, the whole system begins to look dramatically more capable than
any single feature.
That's the pattern I think you're going to see over the next several sessions.
The next session probably won't look like some giant leap towards science fiction.
It'll probably look like, okay, make the lift move 15 millimeters instead of 6.
Then, okay, full lift up works.
Then, why did he interpret that sentence as three movements instead of two?
Then, oh wow, he just combined forward head and lift on his own.
Little wins.
Then suddenly you look back after several versions and realize the robot that originally
sat there answering questions now looks around, shifts position, gestures, searches, and chooses
movements as part of conversation.
And that, to me, is the real trajectory here.
You're not trying to make Cosmo execute more commands.
You're slowly building an interface between a model's internal intentions and a physical
machine.
The motors are almost the easy part.
The interesting work is creating the translation layer between concepts like look over there,
approach that, back away, show excitement, search around, go charge, and the actual physical
primitives that make those ideas real.
And once that translation layer becomes rich enough, Quen stops feeling like software that
happens to control a robot.
The robot starts feeling like where Quen is.
That's the end game.
Comments
No comments yet.
Join the discussion on OpenVibe.Community