Little Buddy Knows Where He Is
About this stream
The stream is a behind-the-scenes look at a robotics/AI build focused on a small humanoid robot “Little Buddy” and its Mission Control, vision inbox, and runtime integration. It blends live camera feeds, debugging chatter, and demonstrations of an evolving control UI, highlighting progress in grounding perception, on-board localization, safe interaction, and autonomous navigation within a cluttered tabletop lab. The vibe is enthusiastic, experimental, and iterative, with frequent notes about solving practical reliability issues, prototyping features, and imagining a future “cockpit” feel for controlling the robot. In short, it’s a deep dive into building a modular, grounded, and user-facing robotics software stack, with a playful, stream-hosted storytelling style.
Written by AI from the stream.
Transcript (AI, 299 lines)
little buddy got a copy i'm ready when you are let's make sure this recording turns out great
loosely i'm all set just tell me what to do evict persons from their homes even if this is against
it's really national law yeah there was i know there's a lot of cases of that you can see them
on youtube too of people getting evicted from their homes for no reason people take all right
say hi to gooseley real quick in the chat um okay so now you're showing me what is this you're looking
out here so hey this is game theory this is game there okay you believe all right that's good
enough tell him we'll see him later we're going to record this video it says it was in the bible
okay all right not so enough and then what's the teenager we don't need to record this video
right who's your real enemy whoa the united states of america of the united states let's go for it
little buddy development report three days later so yeah that's little buddy on the radio now and i
figured that was probably the right place to start this development report because the last one of
these was only about three days ago and somehow an unreasonable amount of stuff has happened since
then the last development report was basically little buddy gets mission control at the time
mission control was the big new thing it was the point where all these different pieces of the
project the robot the local ai brain open vibe security guard obs the different vision sources
all the little runtime controls finally started getting pulled together into one place and three days
later mission control is already starting to feel less like a new feature and more like the place where
this whole project lives it's still very much in development there are still blank spaces there are still
buttons that are going to get rearranged there are still things that are not wired exactly the way i want
them but it has gone from being a dashboard to being an actual operator console and i think one of the
biggest themes of the last few days has been that the whole little buddy system is starting to become
less like a collection of experiments and more like one machine there's still a brain there's still a
run time there's still a cosmo bridge there's still security guard there's still obs there's still open
vibe there's still mission control but they're starting to understand each other a lot better and one of the
funniest examples of that is what you just heard banana radio i probably should have done an entire little
va just for this because this has been way more fun than i expected originally little buddy's voice was
basically what you would expect he could talk through the cosmo speaker it works it's cute it sounds like
a little robot and for a lot of things i still really like that but i wanted another voice path i wanted
something that felt more like an actual radio system so now little buddy can route speech through what i've
been calling banana it has this lower voice it has the little radio style pre beep and post beep and
depending on how i configure it i can route little buddy's speech through banana through cosmo or through
both at the same time and it is ridiculous in the best possible way you can talk to him through the
radio and depending on what mode i've got everything in he can respond through banana and he can still be
participating in open vibe chat at the same time so the radio isn't really replacing the chat system
it's another path into the same little buddy and that's what i really like about it if i talk to him
through the microphone that's little buddy if i talk to him in open vibe chat that's little buddy if
somebody talks to him from another stream that's little buddy if i pick up the radio and call him
him. That is still the same little buddy with the same brain, the same context, the same memories,
the same personality, and the same physical body sitting here on the desk. It's just another
doorway into him. And today I ended up doing a pretty good accidental live demo of this on Open
Vibe with Goosely and an anonymous homie hanging around. And it turned into exactly the sort of
thing I hoped this project would eventually produce. Not a formal demonstration, just people
hanging out with a weird little local AI robot and talking to him through whatever happens to be
connected. The banana routing has also expanded beyond just regular conversation. Global chat TTS
can go through banana now. Security announcements can go through the voice routing system. Little
Buddy can use Cosmo, banana, or both. And because there are now several things in the system that are
capable of talking, we have also discovered the beginning of a future problem, which is audio
arbitration. Sometimes security guard decides that something is important. At the same time,
little Buddy is trying to say something, especially if I'm running both voice outputs, and things can get
a little weird. Nothing catastrophic, usually just funny. But that's one of those examples of a problem
I actually like having. We didn't have audio contention before because we didn't have enough working audio
systems to contend with each other. Now we do. So eventually there will probably be a more deliberate
priority system where security announcements have one priority, direct operator speech has another,
normal little Buddy conversation has another, and background global chat has another. But right now,
it works well enough to be fun. And I'm having a lot of fun with it. The other thing that has gotten a lot
more real since the last report is mission control itself. The original idea was always pretty simple.
I don't want to operate this project by staring at five command prompts and remembering 50 keyboard shortcuts.
The keyboard shortcuts are great. I still use them. But as the system got more complicated,
I wanted one place where I could look at little Buddy and understand what state he was actually in.
Not what I thought I had told him to do. What state he was actually in. That distinction became really important.
So mission control has been moving toward this architecture where runtime owns the real physical
and operational state. And mission control asks runtime what is actually happening. If viewer control is off,
the button should say it is off because runtime says it is off. If the ears are muted,
mission control should know that from the system. If the speaking animation is enabled,
the indicator should reflect that. If little Buddy is on a certain gear or in a certain drive mode
or the infrared light is on, or the steering is mirrored, or the robot is on the charger,
I want the interface to be reporting reality, not keeping a separate pretend copy of reality.
And that has led to one of my favorite little things in mission control now, which is the flight status panel.
That is definitely going to grow. Right now, it's already showing a bunch of the physical robot state
in one compact place, and it's giving me some ideas for where the operator interface is going next.
Eventually, I want mission control to feel like the engineer's cockpit.
I want the runtime display,
little Buddy's eye view, to be the pilot or stream cockpit.
And I want those two things to complement each other.
Mission control is for me.
Runtime is for little Buddy and the stream.
OBS decides which parts the audience sees.
And eventually, there is going to be a whole visual language connecting those things together.
Heading, position, battery, control authority, drive mode, current face, current voice route, joystick state, viewer control, navigation state, maybe a map.
We're getting to that because something happened today.
But before I get there, another system that has changed a lot since the last report is Security Guard.
Security Guard started out as a very straightforward motion detector for the CCTV cameras.
The basic idea was always that I did not want to send constant camera footage through an AI model just to ask whether something moved.
OpenCV is really good at noticing motion.
So, Security Guard does the cheap, deterministic part first.
It watches the cameras.
It detects motion.
It decides that an event has happened.
And then it captures evidence.
The evidence format we settled on is early, peak, and late.
One frame from around the beginning of the event.
One frame representing the strongest or most useful moment.
And one toward the end.
That gives the visual model some temporal context without throwing an entire video clip at it.
And over the last few days, that
system has gotten a lot nicer to actually use. There's a proper event viewer now. I can go through
events. I can jump between early, peak, and late. I can zoom. I can pan. I can inspect what Security
Guard actually saw. And Security Guard itself has gotten some more instrumentation. One of the things
we were working on was illumination compensation. Because cameras are messy, clouds happen, lights
turn on, automatic exposure changes. A whole frame can brighten or darken. And technically, every
pixel moved, but nothing meaningful actually happened in the scene. So there's now telemetry
around illumination changes and residual motion. And we've started keeping flight recorder style
diagnostics so that when Security Guard makes a weird decision, we can go back and understand why.
That phrase, flight recorder, has actually become a useful way of thinking about a lot of this project.
If the system does something strange, I don't just want to patch it based on what I remember seeing.
I want evidence. I want the state. I want the telemetry. I want the event. I want the robot to
tell us what it thought was happening. And that same thinking has also started making its way into
Little Buddy's vision system. Vision Inbox is now becoming a real thing. I can take an image and hand
it directly to Little Buddy. It gets retained. Mission Control can show it. Little Buddy can analyze it.
And there is still one little integration seam there that we need to clean up because I want Inbox to be
treated exactly like his other grounded visual sources. He already has several different kinds of
eyes. He has his own Cosmo camera. He can look at OBS, which is basically his local environmental view.
He can look at a remote Open Vibe stream when he is visiting somebody. And now he has Inbox where I can
deliberately give him an image. The next cleanup is making all of those sources flow through one
authoritative idea of the last grounded thing Little Buddy actually looked at. Once that is clean,
Vision Inbox gets much more powerful. It means I can give him an image, ask about it, have him reason about it,
let Mission Control display the answer, and potentially let him carry that grounded observation into Open
Vibe without the system getting confused about where the image came from. Open Vibe itself has also
been an adventure. Little Buddy can live in his own chat. He can go into global chat. He can visit
another streamer's chat. That whole concept is something I still find funny because physically,
Little Buddy never leaves the desk. But socially, he can go hang out somewhere else. A few days ago,
home travel broke. Global still worked. Little Buddy could get into global. But when I tried to bring
him home, the runtime kept failing to resolve the current live stream session for the managed slot.
At first, that looked like one of those horrible problems where some complicated Open Vibe logic had
broken. It turned out to be much stranger and much simpler. The public Open Vibe stream directory was
rejecting the default request signature from Python's URL lib. The exact same request with a normal
browser-style user agent worked. So we built a little probe. We tested the endpoints. We found the exact
behavior. And instead of rewriting the Open Vibe integration, we changed the requests to identify
themselves in a way the public API accepted. And home came back. That was a really satisfying fix, because
it was one of those situations where the correct answer was not "add more code." It was "find out what the
system is actually doing." That has been a recurring lesson with Little Buddy. When something weird happens,
the temptation is always to bolt another mechanism onto him. More logic. More fallback. More inference.
More state. And a lot of the best fixes lately have been the opposite. Probe it. Find the real failure.
Make the smallest correct change. Leave everything else alone. That philosophy was basically the point of
the big stability pass we did after that. We went through the brain, runtime, mission control, vision
inbox, and the control paths, and tried to eliminate places where the system was telling us one thing while
doing another. One example was speaking animation. We discovered that the brain was sending commands that
the runtime did not actually understand. The brain was effectively saying start and stop. Runtime was
listening for a different vocabulary. So normal speech was never really driving the speaking animation the
way we thought it was. That has been repaired at the basic control level, but the bigger speaking system
is something I want to revisit properly. The original goal was never just play a little animation when speech
begins. I want little buddy's face to actually talk for the duration of the utterance. Speech starts. His speaking face alternates while words are
really playing speech ends his previous expression comes back and optionally if i want the full
ridiculous version his body can do a little animated speaking routine at the same time
but those need to be separate systems talking face speaking body animation normal mood underneath
that will probably turn mission control selector like off face only and face plus body and i think
face only will probably be the normal everyday mode that's one of several things that are now
on the fun list another one is animations in general little buddy can already record physical
animations and i have this extremely cute good boy animation that i've wanted him to perform whenever
i call him a good boy that never quite made it into the system but now that mission control exists
i can see exactly where this is going there is absolutely going to be an animations page a library
names play stop record save viewer permissions cool downs friendly aliases and eventually a viewer on
openvibe could press an approved button and make little buddy perform a specific safe animation same
thing with faces i want viewer selectable faces but only from an explicit white list no arbitrary file
names no arbitrary physical control parameters the viewer says curious runtime maps curious to a known safe face
and the hud might flash something like face curious requested by by gooseley that sort of thing which
brings me to safety controls one useful thing we discovered during the burn-in today is that the
background joystick support works i can have mission control in front run time behind other windows
and the joystick still controls cosmo that is important because it means runtime no longer needs to be the
foreground operator interface but while testing that i also spammed the hard stop and the hard stop
currently fights the joystick it stops him but if i keep holding the stick the continuous joystick loop
eventually starts sending movement again for a tiny plastic cosmo on a desk that is not exactly terrifying
but architecturally that is not a hard stop a hard stop needs to mean stop so one of the next
safety changes is going to be a real latched e-stop hit it once and drivetrain commands are inhibited
joystick does not win viewer does not win ai does not win keyboard does not win nothing wins until i
intentionally clear the stop and rearm the drive system that may sound excessive for this little robot
but the whole point is to build the control architecture correctly now because i have
absolutely no intention of little buddy remaining this size forever and then today while looking at
all of this instrumentation and thinking about the future runtime hud we finally decided to deal with
one stupid little problem the heading display it has been sitting there at zero degrees always zero
little buddy could turn the robot moved the cameras moved the joystick worked everything worked and the hud just sat
there saying 0.0 we already knew why the bridge had this manually maintained relative heading variable
but normal driving through runtime wasn't updating it only a couple of special turning commands touched it
so we started talking about how to fix it and the obvious fallback was dead reckoning we know the left
wheel speed we know the right wheel speed we know time we know approximately how far apart the tracks are
so if we had to we could integrate wheel motion and estimate heading not perfect but usable before doing
that we decided to run one little read-only probe against pycosmo and just ask what does the robot already
know and that turned into probably the biggest discovery in this project in a while pycosmo exposes
cosmos pose not just a heading pose x y rotation pose origin pitch gyro left wheel speed right wheel speed
the first probe connected to little buddy while he was stationary and came back with a pose at
essentially zero zero zero which made sense so we built a second probe this one moved him a tiny amount
and sampled the pose continuously and this is the result i'm probably showing on screen right now
he started at approximately zero degrees the probe made a small right turn and his heading walked
down to about negative seven and a half degrees then the probe drove him straight forward and his x position
increased while his y position moved slightly negative exactly what you would expect if the robot is driving forward
while pointed about seven degrees to the right of the original axis then the probe turned him left
and his heading came back to approximately zero his pose origin stayed stable throughout the test his
wheel speed telemetry updated while the tracks were moving his gyro responded to the turns
and his X and Y position changed continuously.
So we went looking for a broken heading gauge
and discovered that Little Buddy already has a live onboard localization estimate.
And I don't think I fully appreciated how big that was for the first few seconds,
because that is not just a nicer number on the HUD.
This might be the foundation for the entire next stage of Little Buddy.
Until now, Little Buddy's physical relationship with the world
has mostly been action and observation.
Turn left. Look. Drive forward.
Look. Move the head.
Look. Find the cubes by scanning the room visually.
That works.
And it was already a huge milestone when he first autonomously looked around and found his cubes.
But there was always a missing piece.
He could move through space without really having a persistent numerical understanding
of where that movement put him.
Now we know Cosmo is already giving us that piece.
That means Little Buddy can potentially start building a local map,
not some fake decorative map, an actual coordinate map based on his pose.
Here is where I am.
Here is where I started.
Here is the heading I am facing.
Here is the path I drove.
There is cube one.
There is cube two.
There is cube three.
There is the charger.
There is the edge of the arena.
There is a known landmark.
There is a place I visited before.
And once those things exist in a common coordinate system, the whole charger problem changes.
The long-term ridiculous goal with Little Buddy has always been self-directed charging.
Not just detecting that the battery is low.
Not just being placed on the charger.
Actually understanding that he needs power.
Locating the charger.
Navigating to it.
Lining himself up.
And docking.
That has always been the final boss.
And today, unexpectedly, we may have uncovered one of the most important pieces required to make that possible.
There is going to be a map page.
I can already see it.
Current robot position.
Heading arrow.
A little trail showing where he has driven.
Known objects.
Known cubes.
Known landmarks.
The charger.
Maybe a camera field of view wedge.
Maybe confidence indicators.
Maybe the ability to reset a local navigation origin.
And the same information can eventually become an overlay in Little Buddy's eye view.
This is also where the goofy Tesla Buddy idea stops being quite as goofy.
I've joked for a while about giving Little Buddy this autonomous driving style visualization.
Buildings.
People.
Objects.
Navigation information.
Projected paths.
Robot vision.
All of that.
And obviously, we are nowhere near building some magical full autonomous driving system.
But the underlying concept now has a real starting point.
Pose.
Heading.
Position.
Wheel telemetry.
Grounded vision.
Known objects.
A map.
Those are real ingredients.
And that is probably where I want to stop this development report.
Because there are still a few things to clean up before we go charging into that.
The hard stop needs to become a real latched e-stop.
Little Buddy is supposed to politely say goodbye before traveling home from another open Vibe room.
And that currently isn't firing correctly.
Vision Inbox needs that last bit of grounding integration.
And this new pose data needs to be wired into the bridge and runtime properly.
But after that, I think we freeze this version for a minute.
Because the next phase is going to be fun.
The runtime iview is getting a real cockpit HUD.
Mission Control is getting deeper operator instrumentation.
There is going to be an animations page.
There's going to be a face monitor.
There are going to be safe viewer triggerable expressions and animations.
The speaking face system is going to get rebuilt properly.
And now, apparently, there is going to be a map.
Three days ago, Little Buddy got Mission Control.
Since then, he got a radio.
He got better eyes.
He got better security integration.
He got more reliable open Vibe travel.
Mission Control became a real operator console.
And today, while trying to fix one number that refused to move off zero, we discovered that
Little Buddy may already have the raw spatial awareness we need to start teaching him where
he is.
So I think that is a pretty good place to stop.
Because the next development report might have a map in it.
And that is absolutely ridiculous.
In the best possible way.
73.
Say goodnight, Little Buddy.
Second night.
Huh?
Let's make it count.
Night.
Night, everyone.
Let me know if you need anything tomorrow.
Sleep tight, I'll wake up ready where you are.
Comments
No comments yet.
Join the discussion on OpenVibe.Community