By ATN Editorial Team
An image signal processor (ISP) is the processing stage that converts raw image sensor data into a usable image, applying steps such as demosaicing, denoising, colour correction, tone mapping and lens correction in real time. Depending on the architecture, the ISP may be integrated into the image sensor, sit inside the camera module, or be implemented in a downstream processing SoC. Its settings directly influence how reliably an ADAS perception model detects pedestrians, vehicles and traffic signals. This explainer covers what an automotive ISP does stage by stage, why tuning for machine vision differs from tuning for the human eye, where the ISP sits in a vehicle camera system, and why its parameters are increasingly set in simulation.
That last point is why the ISP has moved to the centre of camera engineering. The image a perception model consumes is not the scene: it is the ISP’s rendering of the scene, so image processing belongs alongside the sensor and the optics as a design variable rather than downstream of them. In August 2026, simulation specialist rFpro integrated Solectrix’s SXIVE software ISP into its AV elevate platform so camera pipelines can be modelled and tuned before hardware exists. One caveat at the outset: there is no single standard automotive ISP architecture. The term traditionally describes a dedicated processing block, but modern vehicles distribute image-processing functions across the sensor, the camera module and central compute in varying combinations.
What Is An Image Signal Processor (ISP)?
An image sensor does not output pictures. It outputs a stream of raw values, one per photosite, sampled through a colour filter array, typically a Bayer pattern in which half the pixels sample green light and a quarter each sample red and blue. Raw Bayer data is not directly suitable for conventional displays and requires further processing before many perception pipelines can use it: it is colour-incomplete, noisy, vignetted by the lens and encoded with sensor-specific characteristics.
The ISP is the processing chain that reconstructs a coherent image from that stream at video rate. An 8.3-megapixel front camera running at 60 frames per second produces roughly half a billion pixel values every second, and that processing has to happen at low latency, inside a system that must survive years mounted behind a windscreen in direct sunlight.
The reason the ISP matters to perception engineers is simple: a neural network never sees the road. It sees the ISP’s rendering of the road. Change the tone curve, the denoising strength or the colour correction matrix, and the input distribution to the perception model changes with it.
What Does An ISP Do? Inside The Processing Pipeline
A production automotive ISP runs a broadly standard sequence of stages, each with tunable parameters. Sensor corrections come first: black level subtraction, defective pixel correction and lens shading correction, which compensates for brightness and colour fall-off towards the edge of the lens. Demosaicing (debayering) then interpolates the two missing colour values at every pixel, and white balance and the colour correction matrix map the sensor’s response into a consistent colour space. Alongside the image path, the ISP also runs the control loops, auto exposure and auto white balance, that set sensor gain and integration times frame by frame.
The automotive-specific stages sit in the middle. A camera exiting a tunnel must resolve a dark interior and sunlit sky in the same frame, which is why sensor makers now specify extreme dynamic range: OmniVision’s OX03C10 viewing sensor combines 140 dB HDR with LED flicker mitigation at 1920 x 1280 pixels and 60 frames per second, and onsemi’s AR0823AT Hyperlux reaches up to 150 dB. There is no single route to those figures. Many architectures capture several exposure levels, commonly long, short and very short, which the pipeline combines into one high dynamic range (HDR) image. Others extend range within a single capture using multiple conversion gains, split-pixel designs or high-capacity pixels: onsemi attributes the AR0823AT’s 150 dB to a 2.1 µm super-exposure pixel that does not require auto-exposure adjustment. The distinction matters because multi-exposure fusion introduces motion artefacts on fast-moving subjects, and single-capture techniques exist largely to avoid them.
Whichever route is taken, LED flicker mitigation is a separate requirement. LED headlamps, traffic signals and variable message signs can be modulated rapidly, often by pulse-width modulation, at frequencies human vision does not perceive. A short exposure can sample an LED during an off period, making a live traffic signal appear dark or flashing across successive frames, a direct hazard for automated driving logic.
Tone mapping then compresses the high dynamic range signal into the lower range and bit depth available downstream, while preserving the contrast that matters. Noise reduction, sharpening and geometric correction of lens distortion complete the pipeline. A full pipeline exposes hundreds of parameters, and they interact: aggressive noise reduction removes the fine texture detection networks rely on, while heavy sharpening creates halo artefacts that can register as phantom edges. The combined setting of these parameters is what engineers mean by ISP tuning.
Machine Vision vs Human Vision: Why ISP Tuning Differs
ISPs have historically been tuned heavily around perceptual image quality, using subjective evaluation refined over months. Researchers at Valeo noted in a 2019 review that ISPs are generally tuned for human viewing rather than for computer vision, and that tuning for detection performance is a materially different objective. For a perception stack, aesthetics are irrelevant: what matters is whether the features a network relies on survive processing, and a tuning that pleases a human eye can suppress exactly the low-contrast texture a pedestrian detector needs at night.
The strongest evidence is quantitative. A CVPR 2020 paper by Mosleh et al. applied hardware-in-the-loop optimisation to a hardware ISP’s hyperparameters directly against an end-to-end detection objective, using an Arm Mali-C71 ISP processing captures from a Sony sensor. For automotive 2D object detection the method outperformed manual expert tuning by 30% mean average precision, and outperformed earlier methods based on differentiable ISP approximations by 18% mAP. The authors also report the finding that should interest any camera team: the optimised images had lower perceptual quality than the expert-tuned ones, yet more cars were correctly detected. Image quality and detection performance are not the same objective, and can pull in opposite directions.
“Advantages can be gained by enhancing the images for machine rather than human vision and ultimately improving the perception model performance and overall safety of the system,” said Matt Daley, technical director at rFpro, announcing the SXIVE integration.
The production expression of this split is the dual-output camera. Sony’s ISX038, announced in October 2024 as what Sony describes as the industry’s first automotive sensor to process and output RAW and YUV images simultaneously, carries its own ISP on a stacked logic chip. The RAW stream is intended for ADAS and automated driving detection, with announced compatibility with Mobileye’s EyeQ6 SoC, while the processed YUV stream is intended for human-facing functions such as the drive recorder. Sony specifies 106 dB of dynamic range with LED flicker mitigation active, or 130 dB in a dynamic range priority mode. The trade-off is real: no single tuning is optimal for both consumers, and serving both means either compromise or two processing paths.
Where Does The ISP Sit In A Vehicle Camera System?
Three common architectural locations for an automotive ISP are the sensor itself, the camera module and the central compute. Production vehicles use all three, and hybrid arrangements that split functions between them.
In-sensor. Sony’s ISX038 stacks a logic chip carrying the ISP directly beneath the pixel array, so a single camera module outputs finished images with no external processing. Sony positions this as saving space, cost and power by letting one camera serve applications that would otherwise need two. The processing budget is constrained by the sensor package’s thermal limits.
In-module. A discrete ISP sits beside the sensor inside the camera housing, a common architecture for standalone viewing cameras such as digital mirrors. It keeps the camera self-contained, at the cost of size, heat and silicon in every camera on the vehicle.
Centralised. The camera head contains the sensor and a serialiser; raw or minimally processed sensor data travels over automotive SerDes links such as GMSL or FPD-Link to an ISP integrated into the central compute SoC. Arm’s Mali-C78AE, an ISP licensed for integration into automotive SoCs, processes up to four real-time or 16 virtual camera streams. Centralising can consolidate silicon, keep camera heads small and cool, and simplify retuning after the vehicle has shipped. It also moves the burden elsewhere, onto link bandwidth and onto the thermal and compute budget of a shared processing domain.
Where an ISP sits within the safety path of an ADAS function, its functional safety characteristics become part of the overall system safety case, and suppliers increasingly design for that. Arm says the Mali-C78AE provides safety mechanisms targeting ASIL B/SIL 2 diagnostic requirements, and ASIL D/SIL 3 for the avoidance of systematic failure, including more than 380 fault detection circuits, continuous built-in self-test and detection of faults in the connected camera, with the aim of catching a fault within a single frame. The reasoning behind that level of diagnostic coverage is straightforward: a corrupted frame processed and passed to a perception model without detection is a silent failure, and the perception model has no way of knowing.
Hardware ISP vs Software ISP: What Is The Difference?
A hardware ISP implements much of the pipeline in dedicated silicon, offering deterministic latency and high throughput per watt. That does not mean its behaviour is frozen. The hardware architecture is fixed after manufacture, but many processing functions remain configurable through firmware and parameters, and hardware blocks are routinely combined with programmable DSP, GPU or AI compute. A software ISP implements the same functions predominantly as software running on programmable compute such as CPUs, GPUs or DSPs. The trade is flexibility against compute and power efficiency, which makes software implementations particularly valuable during development, prototyping and simulation, though with sufficient compute they can also be deployed in production.
Solectrix’s SXIVE is a software-based ISP ecosystem combining processing software, hardware accelerators and plugins. Within rFpro’s AV elevate, the two companies say the integration lets engineers model the ISP processing stage of a simulated camera, modify image processing modules or swap sensor versions in real time, and evaluate the effect of those changes on downstream perception.
The research frontier replaces hand-designed blocks with learned ones: a 2023 study in Sensors demonstrated combined deep learning tone mapping and demosaicing for automotive vision systems. The unresolved challenge is validation. Introducing learned processing into the image formation path of a safety-relevant camera makes verification, explainability, failure-mode analysis and safety-case development materially more complex than for a conventional deterministic processing chain, and that work is still in progress across the industry.
Why Are ISP Parameters Now Tuned In Simulation?
Tuning an ISP against real-world driving is slow, expensive and unrepeatable. The conditions that stress a camera pipeline hardest, low sun on a wet road, tunnel exits, oncoming LED headlights, a pedestrian in rain at night, are rare, dangerous to seek out and impossible to hold constant while parameters are varied.
Simulation inverts that. The rFpro and Solectrix integration lets ISP and camera parameters be tuned against repeatable edge cases and, critically, varied across synthetic training data production. “It enables critical image quality ISP parameters to be varied through the dataset production, this simply isn’t feasible when relying solely on physical hardware image collection,” said Dr Roman Tzschoppe, R&D manager at Solectrix. rFpro cites a concrete use: assessing how colour-balancing decisions influence pedestrian detection in a dark, wet urban environment versus a sunny day.
There is a commercial driver too. “OEMs now increasingly expect Tier 1 suppliers to deliver a highly accurate digital twin of their sensor system alongside a new product, except they want this long before the hardware is available,” said Daley. For a digital twin intended to reproduce the images presented to downstream perception, omitting the ISP leaves out a significant part of the image formation chain.
The practical consequence for camera and perception teams: ISP configuration is moving from a late-stage calibration exercise performed on hardware to an early-stage design variable explored in simulation alongside the sensor, the lens and the perception model. The underlying shift is from thinking of the chain as sensor, then image, then perception, to treating the ISP as part of what the camera fundamentally sees. Teams that specify the ISP alongside their training data will find edge-case failures earlier than teams that treat it as a fixed property of a purchased camera.
Frequently Asked Questions
An automotive ISP is an image signal processor designed for vehicle camera systems. It converts raw image sensor data into usable images for driver displays and ADAS perception, and must meet automotive requirements including real-time operation, wide temperature ranges and, where it sits in the safety path, functional safety mechanisms aligned to ISO 26262. It may be integrated into the sensor, the camera module or a downstream compute SoC.
In a camera, ISP stands for image signal processor. It is unrelated to the networking term internet service provider. The ISP performs demosaicing, denoising, high dynamic range processing, tone mapping, colour correction and lens corrections on raw sensor data.
A hardware ISP implements much of the pipeline in dedicated silicon, offering deterministic latency and high throughput per watt. Its hardware architecture is fixed after manufacture, though many processing functions remain configurable through firmware and parameters. A software ISP implements the same functions predominantly as software on programmable compute such as CPUs, GPUs or DSPs, offering greater flexibility at the cost of compute and power efficiency.
LED headlamps, traffic signals and road signs can be modulated rapidly, often using pulse-width modulation, at frequencies humans do not perceive. A camera using short exposures can sample an LED during an off period, making a live traffic signal appear dark or flashing across frames. LED flicker mitigation coordinates sensor exposure and processing so LED sources are rendered consistently.
Partly. Some perception platforms consume RAW or minimally processed data and apply their own processing inside the SoC; Sony’s ISX038 is designed to output a RAW stream intended for ADAS detection alongside a processed YUV stream for human viewing. The processing does not disappear in these architectures, it moves, and human-facing functions generally require processing that converts sensor data into a displayable representation.
Instead of optimising for perceptual image quality, machine-oriented tuning optimises ISP parameters against perception performance, such as object detection accuracy. Research has demonstrated hardware-in-the-loop optimisation directly against detection loss, and simulation platforms now allow parameters to be varied across synthetic edge cases and training datasets before camera hardware exists.
For more sensor technology news, click here.
Further Reading From The ATN Camera And Simulation Library
- rFpro Integrates Solectrix SXIVE ISP For Camera Simulation
- Sony Showcases Next-Generation Camera Technology Using AV elevate
- What Is A DIL Simulator? A Driver-in-the-Loop Engineer’s Guide
- KTM Adopts rFpro To Develop Headlight Systems In Simulation
- Testing AV Systems In Simulation
- UK-Funded Project Accelerates In-Cabin Sensor Simulation For Driver Monitoring Systems
- Lynred Demos Smallest Thermal Imaging Sensor For AEB Systems
