When it comes to making robots more efficient, I think it’s fair to say that when one obstacle is overcome, another obstacle springs up. And the more advanced these systems need to be, the more engineered the tech needs to be. Imagine a robot working away in a warehouse, doing what it’s programmed to do, yet the conveyor belt is speeding components along the line, and the lighting overhead is a jittery fluorescent tube from the 1970s. Or a self-driving car, loaded with passengers, coming out of a dark tunnel and then being hit with the blinding sunlight. Both of these pieces of technology rely on a camera that gathers light for a fixed sliver of time, then reads out whatever it’s caught. If there is too little light, the image drowns in noise. If there is too much motion, it smears. The trouble is, the technology surrounding cameras hasn’t changed that much in a long time. Every camera on every robot today has to weigh up the pros and cons and work with whatever trade-off they’ve decided to lean into. It’s one of the reasons machines struggle to “see” as reliably as we’d like them to.
It is this problem that Sebastian Bauer, CEO and Co-Founder of startup Ubicept, has spent his career circling. And he is most certainly on to something. Last month, TSMC and Sony announced a $4.7 billion partnership for a new image-sensor venture. The vision is to develop next-generation sensors in Kumamoto, Japan, at their Advanced Vision Semiconductor Manufacturing Corp., with production starting in 2029. The bigger picture here is to sell these sensors into “Physical AI” – the eyes of robots and autonomous machines, not just phones. This is what Bauer has also been building towards – using a sensor called the SPAD.
What is a SPAD?
Having not heard of a SPAD before, I asked Bauer to explain it to me without jargon. He likened it to rain. “A conventional camera is a bit like putting a bucket outside for a fixed amount of time and then measuring how much water [it] collected. You know how much rain fell during that interval, but you’ve lost the information about when each drop arrived. And, most importantly, your bucket can overflow,” he says. A SPAD (a single-photon avalanche diode) removes the need for a bucket entirely. “[It’s] more like detecting the individual raindrops as they hit.” It is sensitive enough that a single particle of light triggers a signal the sensor can register on its own and in real time.
But what does that actually mean, and what does it fix? Bauer explains: “If you use a short exposure to freeze motion, you collect fewer photons and the image gets noisy. If you expose for longer, you get more signal, but moving objects blur.” SPADs work by timing light instead of just collecting it, borrowing extra photons only where and when they’re actually needed. The payoff, at least in theory, is that a camera will work just as well in a dark alley as it would in direct sunlight, without having to swap lenses or adding a spotlight.
However, there is a catch. A sensor that can register every single photon is going to produce a huge amount of data – more data than an onboard robot could process in real time. “The key is to deal with that data close to the sensor,” he says, rather than shipping raw photon counts downstream to be sorted out later. It is this data processing that Bauer says is trickier to navigate than the sensor itself. “A sensor can capture incredible information, but that doesn’t help much if you need enormous bandwidth and a huge processor to use it.”
When asked if the TSMC and Sony partnership is a breakthrough for SPAD technology, he likens it to proof of a pattern reshaping the whole of the chip industry, where sensor designers lean on manufacturers instead of building everything themselves, rather than a specific milestone for SPADs themselves. But he believes this partnership could “make it much faster to bring new architectures to market.”
But where will this new technology show up first? Not in your phone. Bauer thinks it is likely to be in places where cameras already cause a lot of headaches, such as fast-moving factory lines, warehouse robots, or self-driving cars. He points out that scientific imaging is already there, noting ZEISS’s recent acquisition of SPAD maker Pi Imaging Technology as an early signal.
Looking to the next five years, Bauer isn’t thinking about picture quality at all; he is picturing what stops being a constraint. “Today, we often design around the limitations of imaging – by adding illumination, limiting speed, controlling the environment, or adding other sensors,” he says. Take that constraint away, and the question stops being what the camera can see and starts being what we actually want the machine to do.
