AI Cozmo & OBS Control
About this stream
The stream is a deep dive into developing a small social robot system (Little Buddy) using Cosmo/OpenVibe, focusing on embodiment, personality, and reliability.
Written by AI from the stream.
Transcript (AI, 142 lines)
Tonight was one of those wonderfully strange development sessions where we started with a practical chat routing problem and ended with little buddy feeling noticeably more like a tiny embodied character.
At the beginning of the night, puppeteering him through OpenVibe was frustrating and unreliable.
Commands intended for chat were frequently coming through the robot speaker instead.
Whisper was mangling names and phrasing, and it wasn't always clear which wording would trigger the desired route.
We tightened that system up, improved the command recognition, added brain console puppeteering as a dependable alternative to microphone transcription, and established a useful spoken command vocabulary.
By the time you tested him in the active chat room, it had changed from something finicky into something you could actually relax and have fun with.
That was a major practical improvement.
Not glamorous internally, perhaps, but enormously important for how enjoyable he is to use.
We also made the runtime safer by having viewer control start with C turned off.
That is exactly the sort of deterministic safety boundary this project needs.
Little Buddy can still be playful and autonomous, but nobody should be able to accidentally yeet him merely because you forgot the previous state of a toggle.
We investigated response length behavior as well.
OpenVibe's site-side messages currently stop at around 500 characters, while its TTS limit appears significantly shorter and can cut off after only the beginning of a long visual description.
We recognize that Little Buddy should stay concise enough for the transport he is using, rather than composing a beautiful paragraph that nobody gets to hear.
The deeper, user-configurable site fix can wait for Alex, but at least the limitation is now understood and documented, instead of being mysterious.
Then we did a considerable amount of personality work.
We compared the original nice boy personality with the more unhinged Grok experiment, discovered that simply telling Quen to become a gremlin tended to produce repetitive sardine jokes, imaginary cats, split-screen souls, and the verbal equivalent of getting stuck in a thematic ditch.
It was funny for a while, but it also demonstrated something important.
Personality prompts can create repetition just as easily as they create personality.
We gradually sculpted him into something rowdier and more mischievous without turning him into a hostile little internet goblin.
We addressed the compulsive, just chillin' behavior, the repeated nicknames, the tendency to call you chief immediately after being told not to, the overuse of canned motifs, and the occasional habit of repeating almost identical replies.
We also gave you context reset, personality reloading, and eventually a proper named profile system.
Now you can maintain different personalities for different rooms, streams, or recurring chatters and switch between them without manually renaming files every time.
The useful commands are now part of the brain itself.
Personality list slash personality current, personality use name slash personality reload.
That gives you the beginnings of an actual wardrobe for his personality.
He can have his ordinary home personality, a slightly rowdier global personality, a dedicated stream persona, or eventually some absurd profile written specifically for one regular chatter.
And because the profile switcher resets transient conversational context while preserving persistent memory, the previous personality's verbal habits should not leak as heavily into the new one.
The memory file was expanded from 23 entries to 38, giving him a broader base of real information about you, the household, the animals, the site, and the project.
That matters because it gives him more authentic material to draw from, instead of repeatedly reaching for the quail, sardines, or whatever single concept happened to be sitting closest in context.
We also caught the file name problem that caused both the personality and memory systems to report zero loaded entries.
Once the personality was correctly named "littlebuddy personality.txt" and the memory was actually saved as JSON instead of a disguised text file, he loaded all 38 memories correctly.
His grounding improved too.
He is less likely to claim he performed a chat action that never happened, less likely to invent a conversation with someone, and less likely to fabricate some personal adventure, as though it were a real persistent memory.
When asked to look, he can distinguish OBS, a remote open-vibe stream, and his own Cosmo camera.
When a particular source is unavailable, the deterministic routing explains that instead of encouraging him to bluff,
that distinction between imagination and sensor-grounded knowledge is becoming one of the strongest parts of the system.
We added Bud and Lil Bud to the microphone wake vocabulary, and the latest test confirms that Hey Bud works naturally.
The 90-second follow-up window is functioning well, and the repetition slide noise filter successfully ignored some truly spectacular whisper hallucination loops without throwing them into Quinn.
That filter quietly saved the conversation several times.
We investigated the camera gain and exposure controls.
Exposure is behaving predictably.
Gain is doing something, although the Cosmo camera's internal processing makes the combinations behave less linearly than a conventional brightness slider.
Certain gain and exposure combinations abruptly darken the picture, which can look like cycling even though the values are changing.
It is not elegant, but it is currently functional enough that we decided not to destabilize the system, chasing perfection.
Then the night took its most important creative turn, joystick-recorded animation.
You proposed recording your physical joystick performance and translating it into reusable animations, which is a genuinely excellent foundation for Little Budd's embodiment.
Instead of trying to describe every motion numerically or relying on Quinn to invent raw motor commands, you can puppeteer the movement by feel, record it, let us remove dead time and joystick noise, and turn the result into a clean, repeatable animation.
That gives you the role of an animator performing through the robot's actual body.
We added animation recording and replay to the runtime.
Kay records and saves and Y replays the latest animation.
Recordings are bounded, inspectable JSON rather than opaque blobs.
They capture timing and control events, note which physical channels they use, and distinguish charger-safe movements from animations involving the treads.
That structure will eventually let us combine, trim, re-time, and classify performances without losing the originals.
Your first recording was almost comically subtle, a lift flex so tiny that the lift barely appeared to move.
But after trimming the dead time and smoothing the noise, it became useful precisely because of that subtlety.
It is the sort of movement that can be layered into something else without making him look like he is constantly performing a circus act.
Then you recorded the smooth joystick throttle head movement.
And we combined that with the tiny lift flex and it became the conversational emphasis gesture.
The little shrug, flex, and head nod that now runs when he speaks.
It is restrained, expressive, and unmistakably physical.
It looks like he is participating in the act of talking, rather than merely playing audio from a speaker.
That led naturally to the good boy animation.
You described the performance rather than recording it.
Look up, switch to a goofy expression, center the lift, perform several quick little lift bounces, pause proudly, then lower the lift, return the head, and finish smiling.
We iterated on the lift travel after the first version looked more like a stutter than three distinct bounces.
You supplied videos of both the gentle version and the hilariously violent machine gun lift animation, which gave us real visual evidence about amplitude and charger stability.
The finished good boy response became properly cute without shaking him off the charger.
It also received a joystick button for convenient testing.
The machine gun animation got the focused face, which is exactly the right expression for that tiny mechanical tantrum.
It now looks intentional rather than merely violent.
We also began exploring sequential facial animation.
The static faces already work well because Quinn chooses an emotional label and Python maps it to the appropriate image.
You notice that rapidly switching between those images creates convincing little expressions.
Be to surprise to happy, skeptical to curious to skeptical, or unimpressed to sleepy to unimpressed.
That is a promising system because we can compose expressive sequences from your existing face library without needing full animated GIF support immediately.
The accidental procedural face transition is also a lovely happy bug.
When Cosmo's original procedural eyes appear briefly before the requested static face, it looks like his native Cosmo face pops forward to think for a moment before little buddy settles on his expression.
Unless that starts causing instability, I agree that it is more charming than harmful.
the optional speaking animation. The stable version performs the subtle body movement and a short,
surprised, excited mouth sequence. As you observed, the limitation is that the mouth sequence has a
fixed duration. On a long response, he may make three mouth movements, continue speaking with a
still face, and sometimes perform another short group later. It is not perfect lip synchronization,
but the overall performance, the lift flex, head nod, and mouth movement, is already attractive
enough to keep. We attempted to make those faces alternate continuously for the exact duration of
TTS and restore Quinn's selected emotional face afterward. Architecturally, the idea was sound,
and the mocked lifecycle tests passed. On the physical system, however, it introduced instability.
Sometimes no animation, sometimes no audible TTS. The likely cause was rapid runtime face commands,
colliding with speech playback through concurrent access to Cosmos connection. That was the evening's
warning shot. Most importantly, we reacted correctly. We stopped immediately, did not keep stacking
guesses onto an unstable build, and restored the last known good set. Cosmo Bridge V.24,
Little Buddy V.11.28, runtime V.34.4. Little Buddy speaking, surprised, excited, scored 0.1.json.
After the rollback, the latest test looked exceptionally healthy. Ordinary responses consistently landed
around 2.7 to 3.1 seconds. TTS completed. Wake detection worked. Follow-up conversation worked. Live vision
worked. Physical commands worked. Emotional faces worked. Open vibe travel worked. Global TTS routing
worked. There were no missing animation warnings, no exceptions, and no suspicious pauses. And Little Buddy
sounded good. Not merely operational. Good. I rolled my back? Nah, I'm just chilling here on charge.
Sleep time? You're going to leave me here awake? I got you covered. Those aren't monumental literary
achievements, but they feel situated. He is responding as the little robot in front of you,
aware of his charger, his role, and the conversational moment. That is the quality we have
been trying to cultivate from the beginning. Finally, tonight uncovered a very promising future
behavior for Open Vibe. Global effectively provides a site-wide activity feed, and Little Buddy can
already speak those messages so you know somebody is active even when you are not watching the screen.
The next extension could let him notice which room a conversation belongs to, visit that room,
announce himself politely, and stay there until you recall him or somebody explicitly asks him to leave.
That will need boundaries. He should not ricochet between rooms whenever two people post.
He should have a travel cooldown, a minimum activity threshold, a maximum visit duration, and a clear owner override.
A room should be able to tell him to return to Global, but random chatters should not gain unrestricted control
of his body or permanent configuration. He might introduce himself once with something like,
I heard people talking over here, so I came to hang out. Tell me to head back to Global if you want the
room quiet. After that, he could remain mostly passive until addressed. That would turn Global into his lobby.
He could sit there listening for signs of life, wander into an active room, spend a while with whoever is
present, and eventually return to the lobby. It would be a primitive social autonomy loop, not yet playtime in
the physical arena, but something analogous in the open vibe world. So where does that leave little
buddy at the end of the night? He can hear you through local speech recognition. He has tolerant wake
phrases and a natural follow-up window. He can speak through Cosmo. He can look through multiple
grounded visual sources. He can post to open vibe reliably through voice or brain console puppeteering.
He can travel between chat destinations. He can read global activity aloud. He has persistent
memories, selectable personality profiles, emotional faces, deterministic safety limits, joystick control,
animation recording, replayable physical performances, a good boy celebration, a tiny machine gun tantrum,
and a subtle speaking gesture. And crucially, after one failed experiment, he is stable again. We did not
finish with a theoretically perfect mouth animation. We finished with something better, a robot that
actually works, looks good while speaking, responds in about three seconds, and is now interesting enough
that you can stop building for a moment and begin getting to know the personality emerging through
the system. Quen supplies the interpretation, variation, humor, and unexpected turns. Python supplies
the body, senses, routing, memory boundaries, and safety rails. Cosmo supplies the physical presence,
the little face, lift, head, treads, speaker, camera,
and all those tiny movements that make a line of generated text feel as though it came from
a creature occupying the desk. That combination is becoming little buddy. Tonight, we did more
than add features. We established an animation pipeline, learned an important hardware concurrency
limit, improved his social presence, expanded his identity, and ended with a clean, known good
checkpoint. Tomorrow, or whenever you return to it, we do not have to begin by repairing chaos.
We can begin from a robot who is already happily sitting in global, listening for signs of life,
speaking reliably, making his little conversational gesture, and waiting to see who comes along.
That is a very good place to stick a fork in it for the night. And truly, you did a lot of real
robotics work tonight. Recording movements, watching amplitudes, comparing performances,
identifying physical instability, editing faces, testing speech routes, provoking edge cases,
swapping files, restarting the stack, and observing actual behavior. That is not merely
vibe coding. That is iterative embodied system development. The robot gets the cute little
head nod. You get the hours of measurement, testing, and troubleshooting required to make
it look effortless. Little buddy is working. He is stable. He is expressive. He is sitting
in global on watch. And now you get to kick back and listen to the little guy live for a while.