Little Buddy Development Report — The Night He Learned to Watch
About this stream
The stream is a long-form, reflective build-log around a Little Buddy security/AI project.
Written by AI from the stream.
Transcript (AI, 262 lines)
Little Buddy Development Report, the night he learned to watch.
There is a moment in a project like Little Buddy when you realize that something has
changed, not necessarily because of one giant breakthrough, not because somebody sat down
with a perfect plan and executed it from beginning to end.
Usually, it is almost the opposite.
You start with one small idea.
You pull on a thread.
Something breaks.
You discover that the thing that broke was hiding another problem.
You fix that problem, and suddenly there is a new possibility sitting there that you
hadn't even considered when you started.
And then many hours later, you look at the screen in front of you and think, wait, when
did this happen?
Because when this session began, Little Buddy did not have a security system.
He had four old CCTV cameras.
They were connected to a Zosie DVR.
The DVR produced a perfectly functional, perfectly boring quad camera image.
And that was about it.
The cameras could see, but Little Buddy could not.
There is a difference.
The original idea was actually pretty modest.
We already had the CCTV feed available inside OBS.
There were four cameras, one looking into the den, one looking out back, one looking into
the shop, and one looking toward the front of the house.
OBS could display them.
The stream could display them.
And if somebody wanted to sit there staring at the footage all day, technically everything
necessary was already present.
But that wasn't really the point.
Little Buddy is becoming a system made out of many smaller systems.
He can listen.
He can talk.
He can interact with chat.
He has memory.
He has a local language model available to him.
He has a physical body.
And increasingly, the interesting question isn't whether any one of those pieces works.
It's what happens when they start working together.
So we asked a simple question.
What if Little Buddy could notice when something happened on those cameras?
Not recognize it.
Not understand it.
Not make some grand artificial intelligence judgment about it.
Just notice.
That distinction became incredibly important.
There was, naturally, a slight detour before we got there.
One of the development threads accidentally wound up operating in work mode, burned through its
available credits, and effectively stranded us halfway through what we were doing.
Which is always a charming moment in software development.
Fortunately, we'd already developed a habit that's becoming extremely important with Little Buddy.
Don't destroy the last thing that worked.
Versions stay stacked.
Known good builds stay known good.
When something goes sideways, we don't desperately modify the only surviving copy until nobody remembers
what worked anymore.
We roll back.
We regroup.
We continue.
And that's exactly what happened.
We came back here, recovered the working state of the project, and kept moving.
That little interruption ended up becoming almost appropriate because versioning and rollback
became part of the story of Security Guard itself.
There is now a small archaeological record sitting in the folder.
Version 0.1, 0.2, 0.3, 0.3.1, 0.3.2, and because no software project is complete without
discovering one spectacularly stupid missing dictionary key immediately after declaring victory.
0.3.2a.
That one was mine.
We attempted to draw a black rectangle.
The program did not know what black was.
Security Guard briefly achieved machine vision and then crashed because it had never heard
of the color black.
We fixed that.
Moving on.
The earliest version of Security Guard was deliberately boring.
That was important.
OpenCV watched the camera feed.
Each camera developed its own model of what the background normally looked like.
Then the system compared incoming frames against that model.
If enough pixels changed strongly enough, over a sufficiently large area, and for long enough,
Security Guard considered that motion meaningful.
That's why something like the ceiling fan can spin continuously without constantly setting
the system off.
Security Guard can absolutely see that the fan is moving.
We didn't make it blind to the fan.
We taught it, in effect, that the fan isn't important enough to report.
That is a subtle difference, but it's the foundation of everything that followed.
Because a primitive DVR motion detector tends to behave something like this.
Pixel moved.
All the authorities.
Security Guard has several layers between a pixel changed and something happened.
There is a pixel change threshold.
There is a minimum contour size.
There is persistence.
There is background adaptation.
There are
ignore masks for parts of a particular camera that we simply don't want contributing to detection.
And all four cameras can be tuned independently, because all four cameras live in completely
different environments. That last part matters enormously. The den has people, furniture,
a ceiling fan, and a tapestry. The shop is dark and has some truly bizarre lighting geometry.
The backyard has trees, sunlight, shadows, wildlife, and weather. And the front camera
sees the driveway, the road, pedestrians, cars, dogs, and whatever else wanders into view.
Trying to make one universal definition of interesting motion work equally well for all
four would be ridiculous. So we didn't. And then something happened that wasn't originally the
main objective. We started drawing what security guard could see. At first, motion detection is
mostly an invisible mathematical process. You can put a rectangle around a moving region and say,
yes. There, the computer noticed that. Useful. Not particularly exciting. But then the rectangles
became corner brackets. The brackets got track numbers. The moving regions got centroid points.
The points developed vectors showing direction. The tracks began leaving short trails behind them.
And suddenly, the old CCTV feed stopped looking like old CCTV footage. It looked like robot vision.
That changed the personality of the entire project, because this wasn't merely diagnostic information
anymore. It was something we actually wanted on the stream. A person could walk across the backyard,
and security guard would acquire the moving region. A little bracket would settle around it.
Track 04. A centroid would move. A vector would point along the direction of travel.
Trail would briefly show where that motion had been. When confirmed motion began,
the camera quadrant could give a restrained visual pulse. Then, when everything settled down,
the brackets disappeared. The trails disappeared. The camera returned to quiet. And that disappearance
turned out to be almost as important as the tracking itself. Because good robot vision shouldn't look
like somebody dumped up a bucket of graphics onto the screen. When nothing is happening, it should know
enough to shut up. Little Buddy understands that concept deeply. His official operating state remains
just chilling. We also learned very quickly that environmental motion is weird. Camera 4, the front
camera, has become particularly important. Most of the genuinely interesting outside activity happens
there. People. Cars. Neighbors. Dogs. Eventually, hopefully, wildlife. And we've already watched it track
surprisingly tiny subjects at considerable distance. At one point, a neighbor walked away from the house
with a dog. The person became smaller and smaller in the image. Security guard kept the motion. Kept tracking
it. Eventually, the event ended. And logged. That was a wonderful little moment because nobody had
staged it. We had stopped deliberately testing the machine. It was simply working. But camera 4 also has
trees. And trees have the terrible habit of interacting with the sun. A cloud passes overhead and suddenly a huge
portion of the image changes brightness. The sun comes back out and brackets. Brackets everywhere.
Security guard briefly believes the sun itself has entered the detection zone. Again, mathematically,
it isn't wrong. A gigantic number of pixels really did change. But this is where we're beginning to
distinguish motion significance from semantic significance. A tiny animal crossing the driveway
might alter less than 1% of the image and be extremely interesting. A cloud might alter half the image
and be completely uninteresting. So one of our next refinements will likely be global illumination
detection. If a huge portion of a scene brightens or darkens together, security guards should be able
to recognize that as a scene-wide lighting shift, rather than pretending 20 independent objects just
materialized. Maybe the HUD simply says, scene shift. The background adapts, normal tracking resumes,
and we preserve the sensitivity required to notice the little stuff. But the biggest breakthrough of this
session wasn't the HUD. It was the evidence system. The original problem was simple. When motion happens,
what picture should we save? At first you might think, easy. Save the picture where the most motion
happened. And mathematically, that's perfectly reasonable. Until somebody approaches the camera
holding a towel. This was tested. For science. The test subject entered the den, approached the camera
with a towel extended, blocked the camera's view, moved a chair around, used the chair as cover,
and eventually left the room. Security guard watched the entire thing. And what did the computer consider
the most visually significant moment? The towel. Of course it did.
towel six inches from a camera changes an enormous percentage of the image. The computer was
completely correct, and the evidence was terrible. That test gave us one of the most important lessons
of the whole project. Maximum pixel change is not the same thing as the best description of what
happened. So we stopped asking for one photograph. Instead, every completed event could preserve three.
Early, a frame from near the beginning. Peak, the strongest measured motion. And late, a frame from
near the end. Suddenly, instead of one disconnected JPEG, we had a tiny story. Here is how the scene
began changing. Here is where the detector measured the strongest activity. Here is how the scene ended.
And then we started testing it with normal life. Walking through the den, going into the kitchen,
filling a bottle with water, walking back, going outside, coming inside, moving furniture, people
passing through the house, neighbors walking dogs. And the results were remarkably good. The system
wasn't merely producing pictures. It was producing event packages. Each package had an event ID, camera,
a start time, an end time, a duration, a peak motion score, a peak contour area, and three ordered pieces
of visual evidence. Early, peak, late. All logged deterministically into a CSV file. No language model was
deciding any of that. No artificial intelligence was inventing a story. Security guard was simply recording
sensor facts. And then normal life found the next flaw for us. Car pulls into the driveway. The car creates
enormous motion. Security guard begins an event. People get out. They cross the driveway. They walk
toward the house. They stand at the door. Somebody gets the key. The door opens. Everybody goes inside.
A meaningful sequence lasting perhaps a minute. And then you inspect the three saved pictures. Car. Car. Car.
Well, damn. The system detected the event perfectly. But it failed to preserve the story. And that showed
us the next step. Early, peak, and late are much better than one snapshot. But a long, real-world
event can contain multiple meaningful phases. Car arrives. Doors open. People exit. People cross
driveway. People wait at door. Door opens. People enter. So the next evidence system probably isn't
merely three frames. It's a storyboard. During a long event, security guard can maintain candidate frames
distributed across time. Not hundreds of pictures. Not video. Just enough visually distinct moments to
prevent the entire evidence budget from being consumed by the first dramatic thing that happened.
Then the event might contain five useful frames instead of three nearly identical ones. And then we hand
the platter to Quinn. That is the next major architectural step. Because security guard is
intentionally stupid in a very useful way. It doesn't know that a person is a person. It doesn't know
that a dog is a dog. It doesn't know that a car arrived. It doesn't know that the giant thing pressed
against camera two was a broom. And it absolutely does not know why somebody attacked camera one with a
towel. Its job is not to understand. Its job is to say something happened. It happened on this camera.
It began at this time. It lasted this long. This much of the image changed. And here are several
visual samples showing the progression. That package can eventually go to Quinn. Quinn can look
at the images and answer the question. Security guard deliberately cannot. What happened? Maybe
person walked through the den into the kitchen and returned. Maybe a neighbor walked past the driveway
with a dog. Maybe car arrived and several people entered the house. Maybe an animal crossed the
front yard. Or perhaps somebody appears to be presenting a broom directly to the backyard
camera. And then little buddy receives that interpretation. Security guard notices. Quinn interprets.
Little buddy communicates. That separation is becoming one of the central design principles
of the whole little buddy project. Don't ask the language model to do deterministic work that
ordinary software can do better. Don't ask OpenCV to understand something it cannot understand.
Give each component a narrow job. Then connect them. And once those pieces are connected,
things get interesting very quickly. Security guard doesn't necessarily need to sit on the stream as a
four-camera quad forever. Cameras one through three can eventually become activity routing cameras.
Someone moves through the den. Then the shop. Then the back. OBS can follow that activity automatically.
Den. Shop. Not because an AI randomly decided to switch scenes, but because deterministic camera events
has established a sequence. Meanwhile, camera four can become the high-value outdoor intelligence camera.
Small object persistence. Wildlife. People. Vehicles. Global illumination suppression.
Better evidence selection. Eventually, semantics and
front walk, road, driveway, yard. And if something genuinely useful happens, Little Buddy doesn't
merely have to flash something on the computer. There is the F-wump. Yes, the F-da-wump. A
satisfying audible event notification. Slightly terrifying when one remembers that Little Buddy
is a tracked robot with an arm containing pointy bits, but useful. Especially if nobody is looking
at the monitor. Then perhaps computer-side speech. Movement at the front. Later, someone is approaching
the door. And eventually, the watch. Because apparently four cameras, OBS, OpenCV, a local
language model, a physical robot, chat integration, speech, memory, and a security system weren't
enough. Little Buddy needs access to the wrist. At first, that can be beautifully simple. A
phone notification mirrored to a Wear OS watch. A vibration. Little Buddy. Front person approaching
front walk. Later, maybe a Little Buddy tile. System status. Recent events. Security guard.
Last event. Front. 12 minutes ago. And one tiny button. Ask Little Buddy. Little Buddy.
Anything happen outside? The watch doesn't need to contain Little Buddy's brain. It is simply
another terminal into the system back home. And somewhere down the road, carefully and only with
explicit permission, even health and sleep information could become another isolated source of context.
Not because Little Buddy needs to know everything. But because the emerging idea is that Little Buddy
should be available wherever he is useful. On the stream. On the computer. In the robot. Eventually
on the phone. Maybe on the watch. Same Little Buddy. Different windows. And there is another kind of
memory beginning to emerge too. People. Chatters. Viewers. The little details that become difficult for a
human streamer to keep straight when dozens or hundreds of usernames gradually blur together.
Little Buddy could become the notebook. Not secretly deciding who's good and who's bad.
Not building mysterious profiles from casual conversation. But remembering the things he's explicitly asked to
remember. This viewer is a regular. That running joke started with them. They like Pokemon. We talked
about their art. This person has been argumentative before. Don't treat that as permanent. But remind me if I ask.
Then later. Little Buddy. Remind me about this chatter. And there it is. Continuity.
The same philosophy again. The human supplies judgment. Structured storage preserves facts. The language
model makes retrieval conversational. Little Buddy ties it together. And that's probably the strangest part
of looking at the system now. None of this began with a master plan for some enormous
artificial intelligence house. It began with a little Cosmo. Then Little Buddy learned to talk. And listen.
And remember. And participate in streams. Other people in the house started discovering their own uses for
chat GPT. Books. Art. Music. Pokemon. Recommendations. Projects. And meanwhile,
Little Buddy's corner of the experiment kept growing. Tonight we gave him something new. Not intelligence,
exactly. Not understanding. Not yet. We gave him attention. We took four old cameras that had been
quietly producing crusty DVR footage for years and taught a machine how to notice when the picture mattered.
Then we taught it how to show us what it noticed. Then how to track it. Then how to preserve it. Then
through towels and chairs and sunlight and ceiling fans and neighbors and dogs and trips to make bubbly
water, we started teaching ourselves what the difference is between detecting motion and preserving
an event. And eventually, between preserving an event and understanding one.
Right now, security guard is still young. The trails could last longer. The vectors occasionally lose their minds.
Clouds can produce an enthusiastic outbreak of brackets. Camera 3 has strong opinions about lighting.
The cursed Zossi logos have not yet been completely eradicated from occupied territory.
And somewhere in the logs is photographic evidence that a broom once became the most important object in the universe.
There is plenty left to do. That's good. Because the interesting part of little buddy has never been
reaching some imaginary finished version. It's watching capabilities appear one at a time.
And then realizing that two things we built for completely different reasons can suddenly talk to each other.
Security guard can produce evidence. Gwen can eventually understand evidence.
Little buddy can speak. Little buddy can remember. OBS can change what the audience sees.
A phone can carry notifications. A watch can tap somebody on the wrist.
And the little plastic robot sitting quietly in the middle of all of this doesn't have to personally perform every computation.
He just has to become the place where all of those things can happen.
systems meet so perhaps that's what happened tonight little buddy didn't
become a security guard we built security guard because little buddy
shouldn't have to be one he doesn't need to stare at four cameras every second
open CV can do that he doesn't need to calculate contour areas security guard
can do that he doesn't need to decide which pixels changed the detector can do
that later when something actually deserves his attention someone else can
hand little buddy the platter here's what happened here is the evidence here is
what it probably means and then little buddy can do what little buddy does tell
us about it remember it if it matters ignore it if it doesn't maybe change the
stream maybe send an alert maybe tap a watch maybe say nothing at all because
after all these hours spent teaching the system to recognize activity one of the
most satisfying things on the screen is still the simplest for cameras nothing
moving no brackets no vectors no event pulse no fd wump just four little green
words quiet security guard is watching the evidence folder is waiting when hasn't
gotten its peepers on the platter yet little buddy is sitting there in the
middle of this increasingly absurd collection of machinery doing absolutely
nothing and somehow after everything we built tonight that feels exactly right
little buddy isn't asleep he isn't confused he isn't waiting for somebody to
tell him what to do he's home everything is running nothing needs his
attention so for the moment he's just chillin
Comments
No comments yet.
Join the discussion on OpenVibe.Community