By ATN Editorial Team
rFpro has launched its latest simulation rendering technology which has transformed the realism of sensor simulation. It is enabling the development of autonomous vehicles and ADAS to be rapidly accelerated by creating synthetic training data rather than relying purely on data collected from the real world. We talk to Matt Daley, rFpro Technical Director to find out more.
What is ray tracing and how is it different to what rFpro already does?
Ray-tracing is a highly accurate way of rendering a simulation. The new rendering engine technology will sit alongside our established rasterization-based rendering engine. Whereas rasterization simulates light taking single bounces through a scene, ray tracing uses multiple light rays through the scene to more accurately capture all the nuances of the real world. The technology has been driven by the growth of vehicle sensors, such as camera, lidar and radar, for ADAS and autonomous vehicles. These sensors have a safety critical function, so it’s essential that we are creating the highest fidelity virtual representations of them. Electronic sensors see the world differently to the human eye and electronic perception systems are harder to fool than a human driving a simulator. As a result, to create meaningful high-value training data for the development of these systems the world must be simulated to a far higher level of accuracy than ever before.
Will the rasterization engine still be used?
Absolutely. The choice of rendering engine will be dictated by what you are looking to develop. Our rasterization engine has been designed for real-time simulation and powers rFpro’s industry-leading driver-in-the-loop solution that is used across the automotive and professional motorsports industries. So, rasterization will be the engine technology of choice for real-time simulations where a driver has input. However, if you are developing sensor hardware or software and aren’t constrained by real-time then the ray tracing engine offers a much superior fidelity image, which is essential for training and testing the perception systems of autonomous vehicles.
How will it help Tier 1 suppliers and vehicle manufacturers to develop autonomous vehicles and ADAS technologies?
Simulation is now widely accepted across the automotive industry as a key tool in accelerating development. The power of simulation offers huge value in the ADAS and autonomous vehicle sector. It is the only way to safely and thoroughly subject AVs and autonomy systems to a substantial number of edge cases to train AI and prove they are safe. However, traditionally the fidelity of simulation hasn’t been high enough to replace real-world data. Our ray tracing technology is a physically modelled simulation solution that has been specifically developed for sensor systems to accurately replicate the way they ‘see’ the world. So now, for the first time, manufacturers can generate useable training data for the development of autonomous systems. The availability of this high-fidelity data generated from simulation also enables sensors to be developed before they physically exist using sensor models. This removes the need to wait for a real sensor before collecting data and starting development. It will significantly accelerate the advancement of AVs and sophisticated ADAS technologies and reduce the requirement to drive so many developmental vehicles on public roads.
How do sensor systems ‘see’ the world differently to humans and why is it important to replicate this in simulation?
Modern HDR (High Dynamic Range) cameras used in the automotive industry capture multiple exposures of varying lengths of time. For example, a short, medium and long exposure per frame. The longer the exposure period, the larger the blurring effect of moving objects. We don’t see this as humans but cameras do – and the perception systems that ingest these images have to learn to deal with it.
Imagine a fast-moving vehicle on a highway, a long exposure image generated by a camera will ‘see’ blurring in the rotation of its wheels. The same blurring is seen as vehicles cross in front of you at junctions. There is also the added challenge that some sensors have a rolling shutter rather than global shutter. A global shutter will expose all of the pixels on the sensor at exactly the same time, whereas a rolling shutter sensor actually has a small time-offset for the exposure of each individual line of pixels on the sensor. This means that essentially the top and the bottom of the image are not actually taken at exactly the same time. The rolling shutter effect of the camera will also distort objects when the ego vehicle or the objects themselves are moving at speed. For example, a traffic cone on the side of the road could appear stretched or slanted due to the time difference of the exposure at the top and bottom of the image.
It is critical to accurately simulate these nuances as this is what the camera will see in the real world and the perception systems then have to cope with them, otherwise, the data used to train AI systems can be too simplistic and misleading. By creating this highly-realistic, engineering-grade simulation it significantly reduces the industry’s dependence on generating real-world data that has these nuances in every image. To achieve this rFpro has introduced its multi-exposure camera API that can now make individual ray-traced images for each of the exposure time periods of a sensor. This ensures that the simulated images contain accurate blurring, caused by fast vehicle motions or road vibrations, alongside physically modelled rolling shutter effects.
What are the environments where ray tracing is most effective?
As a multi-path technique, ray tracing more reliably simulates the huge number of reflections that happen around a sensor – so every scene is simulated to a higher level of accuracy regardless of the time of day or surrounding environment. The means that even the most challenging low-light scenarios or environments where there are multiple light sources are now as accurately simulated as the bright sunlight daytime scenes. There is a huge increase in the fidelity of the reflections and shadows. The most striking examples include multi-storey car parks and illuminated tunnels with bright ambient daylight at their exits, or urban night driving under multiple street lights. It is these types of environments that AVs typically find most challenging. So being able to rigorously and accurately test them in simulation here, rather than on the public road, has massive advantages.
Are there any real-time restrictions of this technology?
Unlike some alternative technologies, our ray tracing rendering is applied to every element in a simulated scene, which has been physically modelled to include accurate material properties to create the highest-fidelity images. As this is computationally demanding it can be decoupled from real-time, slowing it down or speeding it up as necessary to suit the level of detail required. This enables high-fidelity rendering to be carried out and then used as training data for deep learning systems or played back in subsequent real-time runs to test out developed perception systems. This overcomes the usual trade-off between rendering quality and running speed when you are forced to lock the simulation to real-time. What’s really nice about the way that rFpro has implemented the new technology is that it is so easy to switch between the real-time rasterization engine and the new ray-tracing engine. End users can quickly and efficiently set up their tests and check specific scenarios running in real-time, then simply run the test again with the ray-tracing engine enabled to get the higher fidelity results output. The end user doesn’t have to change anything about their prepared environment or test scenarios.
When will this new technology be available to customers?
Our new ray tracing capability is available now to complement existing desktop options and it will also be available soon within High Performance Computing (HPC) solutions.
For more news about rFpro, click here.
