Teaching a toy drone to scan rooms on its own

The setup

The hackathon gave us a few DJI RoboMaster TT drones, the “Tello Talent” edition of DJI’s classroom drone. Old, but gold. You talk to it over Wi-Fi with plain text commands over UDP (takeoff, forward 50, cw 90, land) and it streams 960×720 video back. That is about all it gives you:

• No position. The drone does not know where it is in the room. It only knows what it was told to do.
• No depth. One forward camera. The distance sensor on the extension board answers sometimes.
• No memory. Ask it to search a room and it will happily circle the same corner forever, because nothing tells it it has been there.

Our goal was a drone you could give a mission like “find the coherency sign, explore the corners”, and that would go and find it. The hard part was not the language model. It was giving the drone a memory of the place, so the model had something to reason about.

Pivot 1: from photos to a live map

What did not work

We tried three ways to give the drone a sense of the room before one stuck.

1. Plan from one photo. Someone takes a photo of the area from a high spot, we draw a grid on it, Grok plans a route. It works on paper, but it plans blind: the photo is out of date the moment someone moves, and nobody wants to climb on a chair before every mission.
2. Map from a spin. The drone turns 360° in one spot and photographs 8 directions. A depth model turns each photo into distances, and since the drone did not move, the photos can be placed by heading alone. Much better, but only for what you can see from one spot.
3. Proper 3D reconstruction. Match features between frames, recover how the camera moved, triangulate the shared points into a coloured point cloud, the way photogrammetry tools do it. This was way too slow for a drone that is waiting for its next move, and drone footage is blurry and low on texture, so frames often do not share enough points to match at all.

What worked: depth on every frame

Newer monocular depth models changed the plan. Depth Anything V2 (the small, indoor, metric version) looks at a single image and estimates how far away every pixel is, in metres. On a laptop GPU it does that in 0.16 s per frame (measured on my MacBook), so the drone can map the room live while it flies.

Each frame, plus where the drone thinks it is, becomes a map in three steps:

1. Pixels to points. Every pixel’s depth, the camera’s field of view and the drone’s pose put that pixel somewhere in the room in 3D.
2. Points to voxels. Points land in 10 cm cubes that keep a hit count and an average colour. This is the 3D map in the dashboard.
3. Voxels to a floor grid. The room is cut into 50 cm floor cells: free, obstacle or unknown. A cell is an obstacle when something between 30 cm and 2 m off the floor was seen in it more often than the camera saw through it. Blurry frames are skipped, and only the newest frame is processed, so the map never falls behind.

That grid is the drone’s memory, and it is what Grok reads: a picture of it, the same grid as text, and a list of every cell already visited and what was seen there.

Pivot 2: from 47 seconds per decision to under 3

The first time the full loop ran (in the simulator, inside a 360° photo of a hotel room), Grok took 47 seconds to make one decision. And the decision was to give up:

The reachable room was fully scanned in the 360 spin with no television visible, and the only frontiers lie beyond obstacles, so nothing reachable remains to search.

Both halves of that were a problem. A drone cannot hover for 47 seconds waiting to be told where to go, and a search agent that gives up after one look is not a search agent. Two changes fixed it:

• A fast, non-reasoning model. Switching to grok-4.20-0309-non-reasoning brought decisions down to a few seconds. The thinking that matters here is spatial, and the map already does most of it; the model needs to read the map and pick, not deliberate.
• No way out. The give_up action came out of the schema, and the prompt now says “There is no giving up”: if every reachable cell was visited, go to the edge of the unknown and look. give_up was chosen 3 times in the first two runs and never again.

The loop: observe, think, check, act, verify

Every move the drone makes goes through the same five steps. Grok is in exactly two of them.

1. Observe. The live camera frame, the views from the last spin, the map as a picture and as text, and the memory of every visited cell.
2. Think. Grok answers with one action in a fixed JSON format: goto <cell>, look <heading>, spin or found, plus one sentence on why.
3. Check. Code turns a goto into a flight path over known-free cells only. Unknown space is never flown into. The path is flown in hops of at most 1 m, with a fresh depth check before each hop. Short hops, and cells already visited twice, are refused so Grok cannot pace back and forth.
4. Act. The drone flies the checked path, the map updates, and the loop repeats.
5. Verify. When Grok says found, a second, narrower Grok call looks only at the live camera and answers a yes/no question with a bounding box. Then a human confirms or rejects it. A rejection goes back into the memory and the search continues.

This is what Grok actually received on the decision that found the sign, as text next to three images (the map, the camera and the spin views):

Map as text (. free, # obstacle, ? unknown, v visited, D drone):
    A B C D E F G H I J K L M N O P Q
  1 . . D . . . . . . . . . . . . ? ?
  2 . . . v . . . . . . . . . . . . ?
  3 . . . . v v . . . # . . . . . . .
  4 . . . . . v v . . # # # . . . . .
  5 . . # # # # v . . # # # . . . . .
  ...
Drone: cell C1, heading 270 (west), height 3.1 m. This is decision 2.
Frontier (unknown cells next to free ones, where exploring pays off): P1, Q2, A9, A10, B11, ...
Next scan corner: Q13

And what it answered, in 2.7 seconds:

{
  "seen": "white walls, windows, shelves with old computers and a neon 'coherency' sign",
  "thought": "The coherency sign is clearly visible in the current camera view (image 2), right in the middle on a shelf.",
  "action": "found",
  "cell": "C1",
  "heading": 270,
  "evidence": "neon 'coherency' sign on shelf, visible in the middle-left of the image on a white shelving unit",
  "found_in": "camera"
}

The check step is what makes it safe to let a language model fly. Here is a harder mission, where it earned its keep: Grok proposes, code refuses or cuts moves short, and Grok adapts.

Did it work?

Sometimes, and when it did it looked like magic. This is the best mission: the sign was across the room, out of sight from takeoff.

Here is everything the drone reported as found on real flights, and what happened to it:

Confirmed

Confirmed

Confirmed

Confirmed

Confirmed

Rejected

Rejected

Rejected

The honest numbers, from the logs of every real flight:

Most missions did not end in a find; they ended when we pressed Stop to fix something, change the prompt or swap a battery. That is what a hackathon test log looks like, and it is why there is no success rate here. We never ran the planned comparison against a fixed lawnmower sweep, so I cannot tell you how much Grok’s choices beat flying in rows.

The failures are more interesting than the count. The second look checks whether the image matches the words, and the words were the weak point: the event’s own QR screens literally said “Coherency”, so a “coherency sign” check passed. The human confirmation step is not a formality. It is the part of the loop that knows what you meant.

What I’d do next

• Measure it. Found-rate and time-to-find against a fixed sweep, on a fixed set of placed targets. The evaluation script exists; it never ran on real cases.
• Better pose. The drone only knows where it was told to go, and every hop drifts a little. The depth model already sees the walls, so it could correct the drone’s position instead of only mapping.
• One goal, many devices. The rover and the drone each ran their own agent with their own goal. The next step is Grok splitting one mission across both and assigning areas.
• Ask better questions. Verification should check the distinctive part of the description (“neon”, “on a shelf”) and reject look-alikes on purpose, instead of matching the label.

Credits

Built in two days at the Grok Hackathon at the Museum (Ctrl+Alt Museum, Pavia, 4-5 November 2026) with Muhammad Abbasi, who connected the drones and built the rover bridge and its room mapping; Kamran Hasanli, who built the Skytrace website and demo dashboard; Fahad, who wrote the drone camera scripts; and Ayoud, who built the first dashboard. I built the agent loop, the live mapper, the safety checks and the operator app.

Code: github.com/eddii1/seek. Depth model: Depth Anything V2 (Apache-2.0). Decisions, verification and voice: xAI’s Grok API.