Computer vision in space

Vision solves problems in space that no other sensor solves, and it does so under constraints that would be considered unreasonable anywhere else: a processor three orders of magnitude slower than a phone, no opportunity to retry, and no human in the loop.

Terrain relative navigation

The problem is that a lander knows its orbit well and its position relative to the ground poorly. Navigation errors accumulate through entry, and the difference between a safe site and a boulder field is metres.

Terrain relative navigation solves this by matching what the descent camera sees against a stored map built from orbital reconnaissance. Recognising landmarks gives an absolute position fix, correcting the accumulated error.

The improvement is what makes scientifically interesting sites accessible. Curiosity’s landing ellipse had to fit entirely within safe terrain, which excluded most interesting geology. Perseverance’s precision allowed targeting a delta deposit in Jezero crater surrounded by hazards that would have been disqualifying under earlier accuracy.

Hazard detection and avoidance is the companion capability: identify rocks, slopes and craters in the descent imagery in real time and divert to a safe spot. This must run in seconds during powered descent, on flight hardware, with no possibility of a second attempt.

Both were flown on Perseverance’s Lander Vision System, and the same class of capability is needed for lunar landings into polar terrain, where the illumination conditions described in Lunar surface systems make it considerably harder.

Relative pose for rendezvous

Approaching another spacecraft requires knowing its position and orientation relative to yours, continuously, at closing rates measured in centimetres per second.

The third case is the one that matters for active debris removal and for servicing the existing fleet, and it is unsolved operationally. The spacecraft must build a model of an object it has never seen, estimate its rotation about a possibly changing axis, and plan a capture, all while the lighting changes completely every orbit.

Lighting is the underappreciated difficulty. In orbit there is no atmospheric scattering, so illumination is a point source against black. Shadows are absolute, contrast is extreme, and the target’s appearance changes entirely between orbital positions. Vision algorithms developed on terrestrial imagery frequently fail on this.

Visual odometry

A rover wheel odometer measures wheel rotation, not motion. On loose regolith wheels slip, and the error accumulates without bound.

Visual odometry tracks features between successive stereo image pairs and computes the actual motion. The Mars Exploration Rovers demonstrated it, and it proved essential: on some slopes Opportunity’s wheels turned while the rover slid backwards, which only visual odometry detected.

The cost is computation. Early implementations took minutes per step on flight processors, so it was used selectively rather than continuously. It is a clear case where the limiting factor is processing capability rather than algorithm quality. See Avionics and onboard computing.

Star identification

The most mature vision application in spaceflight, flown for decades on almost every three-axis stabilised spacecraft.

A star tracker images a patch of sky, centroids the stars, and matches the observed pattern against an onboard catalogue. The output is absolute attitude to arcsecond precision from a single frame, with no prior knowledge required, which is why it is the primary attitude reference. See Guidance, navigation and control.

The technical core is the lost-in-space problem: identify the field with no prior attitude estimate. Standard solutions use geometric invariants such as angular distances between star triangles, indexed for fast lookup. Solving it reliably in bounded time on a small processor is what distinguishes a good star tracker from a poor one.

Running vision on flight hardware

The division at the bottom is the operative one. Learned models are widely accepted for payload data processing, where a wrong answer costs a downlink slot. They are not accepted for direct control authority during a landing, where a wrong answer destroys the vehicle.

For safety-critical vision, the flown approaches remain classical: correlation, feature matching and geometric estimation, which are deterministic, analysable and testable across their input domain. Perseverance’s Lander Vision System used this class of method, not a neural network.

The likely path forward is the runtime assurance pattern: a learned perception front end producing a hypothesis, and a simple verifiable check confirming it is geometrically consistent before any action is taken.

Constraints that shape the algorithms

The no-retry constraint is the one that most distinguishes space vision from terrestrial practice. A terrestrial system that fails can try again on the next frame. A landing system has one descent, and an algorithm that occasionally takes ten times its nominal runtime has failed even if it eventually produces the right answer.

References

• Johnson, A. E. et al. “The Mars 2020 Lander Vision System Field Test”, AAS 17-036.
• Cheng, Y., Maimone, M. and Matthies, L. “Visual Odometry on the Mars Exploration Rovers”, IEEE Robotics and Automation Magazine, 13:54, 2006.
• Liebe, C. C. “Accuracy performance of star trackers: a tutorial”, IEEE Transactions on Aerospace and Electronic Systems, 38:587, 2002.
• Opromolla, R. et al. “A review of cooperative and uncooperative spacecraft pose determination techniques for close-proximity operations”, Progress in Aerospace Sciences, 93:53, 2017.
• Kothari, V. et al. “The Final Frontier: Deep Learning in Space”, HotMobile 2020.