By ATN Editorial Team
Summary: rFpro has integrated Solectrix’s SXIVE image signal processing technology into its AV elevate platform, creating a camera simulation environment in which engineers can design, model and optimise the entire camera pipeline virtually, before physical prototypes are available.
Key engineering takeaway: The software-based ISP brings key processing steps into camera simulation, including debayering, colour correction, tone mapping, denoising, lens distortion and lens shading correction, and allows ISP parameters to be varied across synthetic training datasets and repeatable edge cases.
Why it matters: OEMs increasingly expect Tier 1 suppliers to deliver an accurate digital twin of a camera system long before hardware exists, so camera simulation with an ISP in the loop moves sensor optimisation, perception training and validation earlier in the development cycle and reduces cost.
- rFpro integrates Solectrix’s SXIVE technology into AV elevate, enabling full virtual camera pipeline development.
- Helps Tier 1s meet growing OEM demand for accurate digital-twins of camera systems before hardware exists.
- Supports earlier, faster and more cost-effective ADAS and automated driving system development.
UK software specialist, rFpro, has added Solectrix’s SXIVE (Simplified eXtensive Image and Video Engine) image-processing technology to its AV elevate™ platform. It will allow engineers to design, model and optimise the entire camera pipeline in simulation. By enabling development before physical prototypes are available, it allows sensor optimisation and training dataset production to shift earlier in the development cycle, significantly reducing costs.
The integration will support the tuning of Image Signal Processing (ISP) and camera parameters against edge cases, the training of perception models with SXIVE-processed synthetic data and testing complete vision pipelines under highly variable conditions.
Solectrix ISP Processing Steps In The rFpro Camera Model
The update brings Solectrix’s software-based ISP into rFpro’s camera model, enabling engineers to optimise key processing steps directly in simulation, such as debayering, colour correction, tone mapping, denoising, lens distortion and lens shading correction.
Tuning Cameras For Machine Vision, Not The Human Eye
“With camera-based perception being central to most ADAS and autonomous vehicle performance, ISP optimisation is a key competitive differentiator,” said Matt Daley, Technical Director at rFpro. “Advantages can be gained by enhancing the images for machine rather than human vision and ultimately improving the perception model performance and overall safety of the system. Our combined simulation environment and ISP model create a far faster and safer way for engineers to experience tuning set options in dynamic driving conditions, rather than heading out onto public roads and finding these edge cases conditions.”
“Providing sufficient amounts of high-quality video data is a real challenge for the development of any modern AD or ADAS system,” said Dr. Roman Tzschoppe, R&D Manager at Solectrix. “Our collaboration with rFpro enables engineers to design and execute hundreds of thousands of test scenarios and generate a wide range of highly-specific training data for camera-based perception models. It enables critical image quality ISP parameters to be varied through the dataset production, this simply isn’t feasible when relying solely on physical hardware image collection.”
Outside of ADAS and automated driving there are also a wide range of camera-based human vision applications with the rapid uptake of digital mirrors and surround view systems in many passenger vehicles, buses and heavy goods vehicles. Critical light conditions from low sun, highly reflective roads, bright vehicle headlights or covered and underground parking areas all create difficult ISP optimisation challenges when tuning these systems.
A Complete Environment To Tune, Train And Test
AV elevate is a fully integrated simulation solution that accelerates the development of ADAS and automated driving systems. It enables sensor tuning, perception and control system training, and testing of the full AV technology stack in a single environment. The platform supports both closed-loop perception testing and the generation of engineering-grade synthetic training data.
SXIVE is a comprehensive image processing ecosystem consisting of the actual image processing software, hardware accelerators, and a variety of apps and plugins. It enables image processing professionals to practice rapid prototyping as well as real-time processing and analysis of images and video streams. With its flexible architecture, SXIVE can be tailored to the requirements of any imaging project, such as ADAS/ AD camera systems, driver monitoring systems or even the video feed for remote driver operating.
The integration of AV elevate and SXIVE gives users flexibility on both the input and output side of camera development. Driving scenarios, the weather and location can be easily varied to continually create new test data and the image processing modules or apps can be modified, or sensor versions swapped out in real-time with a click of a button. For example, this allows engineers to assess how colour-balancing decisions influence pedestrian detection in a dark, wet urban environment versus a sunny day.
“OEMs now increasingly expect Tier 1 suppliers to deliver a highly accurate digital twin of their sensor system alongside a new product, except they want this long before the hardware is available,” said Matt Daley, Technical Director at rFpro. “Developing a high-quality camera model isn’t just a benefit for development but it is now becoming a commercial necessity. By integrating SXIVE into AV elevate we are giving the industry a complete environment to tune, train and test the entire camera system.”
Camera Simulation: Frequently Asked Questions
What is an image signal processor (ISP) in a vehicle camera?
An ISP converts the raw output of an image sensor into a usable image by applying processing steps such as debayering, colour correction, tone mapping, denoising and lens corrections. In camera-based ADAS, these settings directly influence how well perception models detect objects.
What does the rFpro and Solectrix integration do?
It brings Solectrix’s SXIVE software-based ISP into rFpro’s AV elevate camera model, so the entire camera pipeline can be designed, modelled and optimised in simulation, including the generation of SXIVE-processed synthetic training data for perception models.
Why tune a camera for machine vision rather than the human eye?
According to rFpro, advantages can be gained by enhancing images for machine rather than human vision, improving perception model performance and overall system safety. Simulation lets engineers assess tuning options in dynamic driving conditions instead of searching for edge cases on public roads.
Which ISP parameters can be optimised in camera simulation?
The integration supports optimisation of debayering, colour correction, tone mapping, denoising, lens distortion and lens shading correction, with image processing modules modified or sensor versions swapped in real time.
Why do OEMs want digital twins of camera systems before hardware exists?
OEMs increasingly expect Tier 1 suppliers to deliver a highly accurate digital twin of a sensor system alongside a new product, long before the hardware is available, which makes a high-quality camera model a commercial necessity as well as an engineering benefit.
Further Reading
- What Is A DIL Simulator? A Driver-in-the-Loop Engineer’s Guide
- Sony Showcases Next-Generation Camera Technology Using AV elevate
- KTM Adopts rFpro To Develop Headlight Systems In Simulation
For more simulation news, click here.
