Hi again, it’s Hambreros and Tamadillo. Last post gave it eyes. This one gives it something to do with them: place a tennis ball 
in front of it and the robot drives itself towards it, no hands.
What we’ve actually built
A CHASE toggle next to the joystick. Flip it on, and:
- The robot looks for a ball in the camera feed.
- Off-center → it turns towards it.
- Small (far away) → it drives forward. Big enough (close) → it eases off and stops.
- Ball out of frame → it just stops, same as letting go of the joystick.
- Manual controls (joystick, keyboard, per-wheel sliders) go greyed-out and unresponsive while this is on, so nothing’s fighting the robot for the wheel. The ⏻ MOTOR ON/OFF buttons still work regardless — always an independent kill switch, whoever’s driving.
Finding the right brick
Same move as every other feature so far: before writing a line of code, went and read arduino/app-bricks-py to see what already exists. Turns out there’s a whole family of vision bricks — gesture_recognition, mood_detector, image_classification, object_detection, visual_anomaly_detection — and the one that actually fits is video_objectdetection: continuous detection off a live camera stream, with per-label callbacks carrying a confidence score and a bounding box.
Its brick_config.yaml lists:
model_by_boards:
- platform: ventunoq
model: yolox-qnn-object-detection
- platform: unoq
model: yolox-object-detection
UNO Q — this exact board. Not a VENTUNO-only NPU thing, unlike the neural TTS detour a couple posts back. Genuinely usable here.
No training required — just check the label list
The obvious worry: does a generic pretrained model know what a tennis ball is? Didn’t want to assume, so went and checked models/models-list.yaml in the same repo before writing any detection code — it’s a YOLOX-Nano model trained on COCO’s 80 classes, and the label list includes, verbatim:
- sports ball
That’s the actual class name (there’s no separate “tennis ball” class in COCO, but “sports ball” covers it — it’s the canonical example object for that class in the dataset). So: zero custom training, zero Edge Impulse model work. Just register a callback for a class that’s already in the box.
from arduino.app_bricks.video_objectdetection import VideoObjectDetection
detector = VideoObjectDetection(camera=shared_camera, confidence=0.5)
detector.on_detect("sports ball", on_ball_detected)
detector.start()
One camera, two features fighting over it
video_objectdetection wants its own Camera to forward frames to the detection sidecar. We already have one open, for last post’s live feed. Tried to hand-wave past this and it immediately mattered: Camera claims its physical device the moment it’s constructed — there’s an actual registry in the framework’s own source specifically so auto-selection doesn’t grab something already in use — so a second, independent Camera("usb:0", ...) for the same webcam wouldn’t just contend for bandwidth, it’d fail outright.
Good thing VideoObjectDetection(camera=...) takes an existing instance instead of always making its own. Added a small accessor to camera.py:
def get_camera():
"""Blocks until the startup attempt above has settled, then returns the
shared Camera instance — or None if it never started successfully."""
_ready.wait()
return _camera
vision.py calls that instead of constructing its own, so both features share the one physical connection to the one webcam instead of racing for it.
Steering is just proportional control
No path planning, no PID tuning, nothing fancy — just “how far off-center is it” and “how big is it,” recomputed fresh on every detection message:
PID - Proportional Kp, Integral Ki, and Derivative Kd, only recently saw this video from Electronoobs and it looks complicated
def _steer_towards(bbox):
x1, y1, x2, y2 = bbox
frame_w, frame_h = camera.RESOLUTION
center_x = (x1 + x2) / 2
box_h = max(1, y2 - y1)
offset = (center_x - frame_w / 2) / (frame_w / 2) # -1 .. +1
size_ratio = box_h / frame_h # 0 .. 1
turn = max(-100, min(100, offset * TURN_GAIN))
throttle = max(0, min(MAX_THROTTLE,
(TARGET_SIZE_RATIO - size_ratio) / TARGET_SIZE_RATIO * MAX_THROTTLE))
return turn, throttle
turn/throttle go through the exact same mix_drive() the joystick posts through from two posts ago — one function, one place that knows how a turn+throttle pair becomes two wheel speeds, whether a human or a neural network produced them. MAX_THROTTLE is capped well under full speed on purpose — this thing drives itself with nothing watching for obstacles, no reason to let it move at joystick speeds.
What actually broke on real hardware
First real test went nowhere: App.run() never scheduled VideoObjectDetection’s background loops, since the brick got built (on its own thread, after App.run() had already started) too late for the scheduler to notice — confirmed by the sidecar sitting there waiting for a connection that never came. Fixed by just running those two loops ourselves in plain daemon threads instead.
Seeing what it’s actually seeing
Once frames were flowing, the ball’s box kept flickering against other objects (a bed, a cup) even on a dead-static scene — turned out to be real per-frame confidence noise, not a “one object at a time” limitation (it’s a genuine multi-object detector, and no, it can’t be restricted to only look for balls — fixed 80-class model, no filter option). Fix: draw a box for everything it sees, not just the ball, and fade them out over a few seconds instead of hard-cutting the instant one frame doesn’t reconfirm them.
The “lost the ball” behavior that didn’t need writing
Didn’t need a lost-ball timeout — Post 1’s STM32 watchdog already stops the wheels when commands stop arriving, same as a dropped wifi connection, for free.
It worked — and then drove straight past the ball
Reacting to every single detection message overshot the ball almost every time — no braking distance. Fixed by pulsing instead: one short move, stop, pause, then decide again from a fresh look — and VideoObjectDetection’s own per-label lock already discards anything that arrives mid-pause, so no new state machine was needed to make that stick.
What’s next
Well that’s kind of it. Thrilled at how far we got to an actual auto driving EV.