The ghost on the mound
For a few days the backlog has carried one line: on the bumpy test lawn, with 7 cm grass, the robot sometimes stops for nothing. We'd guessed it was grass on a bump reading too tall. It wasn't. The robot was stopping for a person that didn't exist.
Catching it
The first job was to make it happen on demand. Two repeat runs passed. A third, with the grass laid out differently, failed:
STOP: something that moves within 2 m (3d lidar)
Nothing moves in that world. The run log showed the safety gate saw a "moving object" three times: 4.3 m away, then 4.2 m, then 1.7 m, which stops the robot. Running the same grass layout again, it passed. So the bug was intermittent, which also explains why it took this long to pin down.
The gate doesn't record its lidar scans: at about 10 a second they're far too big for the run log. So we reran the same layout with a recorder that saves every scan along with the robot's pose, and replayed them offline through the gate's mover tracker. That rerun also had a ghost, a short pause for a "person", just not close enough to stop.
Not a bank, not something upright
The ghosts kept turning up in the same places, 3.5 to 6 m east of the robot whenever it was on the east side of the test lawn's 25 cm mound. The raw points there were strange. As the robot drove, the "object" was hit by one lidar ring, then the next ring up, then the next, at heights of 0.2, 0.4, 0.5, 0.7 and 0.8 m.
The first theory was the sloped test patch next door, its 20° bank swept by beams as the robot rocked. That fell apart when we checked the world file: the bumpy test world doesn't even load the slope patch. Out there is flat lawn.
The second theory was a filter: people and dogs are upright, and ground isn't. We measured it before writing any code. The ghost points were exactly as upright as the real walker and dog in earlier recordings (median 86°). Of course they were: they were grass blades. That idea went in the bin.
The real cause: a tilted ruler
To decide what's an obstacle, the gate measures how high each lidar return is above the ground. Near the robot, about 1.4 m around it, it does this carefully: it fits the ground and follows each bump cell by cell. Beyond that, it measures from a flat plane fitted around the robot and stretched outward.
On the side of the mound, that plane tilts with the robot. Stretched out 4 m to the east it keeps going down, but the real lawn stops going down at the mound's foot and levels off. So the flat lawn out there measured 20 to 40 cm tall. That alone wouldn't make a mover. But the robot rocks as it drives over the bumps, so its near-level beams would pass over that "tall" lawn on one scan and catch its grass on the next. To the tracker, that's exactly what a person stepping into empty space looks like.
The fix
The careful cell-by-cell ground following now reaches out to the tracker's full 6 m range, in every direction. Each patch of ground is measured from its own lowest return, and is allowed to rise at most 4 cm above the patch nearer the robot. A person or a dog keeps their full height, because their own patch of ground includes their feet. This only feeds the mover tracker. Obstacle stops are unchanged.
Before running anything in the sim, we replayed every recording we have:
| Recording | Ghost movers before → after | Real mover first caught |
|---|---|---|
| Bumpy lawn (this bug) | 9 → 0 | none there |
| Person walking across (3 recordings) | 0 → 0 | same scan |
| Dog running at the robot | 0 → 0 | one scan later (0.1 s) |
| Back yard | 2 → 3, all more than 5 m away | none there |
Then in the sim:
| Run | Result |
|---|---|
| Bumpy lawn, grass layouts 3, 3, 4 and 5 | PASS, 0 ghost warnings. Slowed 3 s instead of 378 s |
| A dog crossing on the bumpy lawn | seen at 4.2 m, stopped 1.8 m away, blade off throughout |
| Person walks across | PASS, seen at 2.65 m |
| Clover runs at the robot | PASS, seen at 3.78 m, stopped |
| Child sits down ahead | PASS, seen at 4.22 m |
One thing it doesn't fix: the back-yard recording has a ghost on a fence line, 5.5 m away. It's beyond the 5 m ring, so it never slows the robot. It's the same "wall seen edge-on" kind as the two already there, and stays on the list.
A watchdog for a hung simulator
Chris asked for one more thing: the simulator sometimes hangs at start-up. The robot appears in the world, and then nothing runs until the run's time limit, often 25 minutes, ends it with no data. That happened in 15 of our last 255 mowing runs, including one of today's.
A healthy run starts its mission within 48 s (4.5 s typically). Now, if no mission has started after 120 s, the run is cleared and tried again, up to three times. Testing the watchdog found one more bug of its own. The polite "please stop" signal (SIGINT) never reaches a program started in the background of a script, because such programs inherit an instruction to ignore it. A hang would only have been cleared by a hard kill 20 seconds later. It uses a different signal (SIGTERM) now, which does arrive.
Not covered yet
- Both dogs at once.
- The back yard's docking failures from the mover runs.
- Real yard photos for the camera's detector, whenever Chris has the time.