Critical scenario identification for testing of autonomous driving systems
Recording from AI Lund lunch seminar 18 May 2022
Title: Critical scenario identification for testing of autonomous driving systems
Speaker: Qunying Song, WASP PhD Student, Department of Computer Science Lund University
When: 18 May at 12.00-13.15
Spoken language: English
Testing of autonomous driving systems is a prerequisite to verify the safety and reliability of such systems. Current approaches for testing autonomous driving systems that rely on substantial real-world testing, or collecting real traffic data at scale, are considered both inefficient and ineffective as it is expensive, time-consuming, and may still not cover the rare-occurring traffic situations.
During the seminar, I will introduce an approach for testing autonomous driving systems based on critical scenario identification. Specifically, I will go through some tools and a workflow for generating critical test scenarios, and I will also demonstrate the effectiveness of the said approach using two real autonomous driving systems from industry.
- Ulbrich, S., Menzel, T., Reschka, A., Schuldt, F., Maurer, M.: Defining and substantiating the terms scene, situation, and scenario for automated driving. In: IEEE 18th International Conference on Intelligent Transportation Systems, pp. 982–988 (2015). IEEE
- Menzel, T., Bagschik, G., Maurer, M.: Scenarios for development, test and validation of automated vehicles. In: IEEE Intelligent Vehicles Symposium (IV), pp. 1821–1827 (2018). IEEE
- Bagschik, G., Menzel, T., Maurer, M.: Ontology based scene creation for the development of automated vehicles. In: IEEE Intelligent Vehicles Symposium (IV), pp. 1813–1820 (2018). IEEE
- Scholtes, Maike, Lukas Westhofen, Lara Ruth Turner, Katrin Lotto, Michael Schuldes, Hendrik Weber, Nicolas Wagener et al. "6-layer model for a structured description and categorization of urban traffic and environment." IEEE Access 9 (2021).
- Kluck, F., Zimmermann, M., Wotawa, F., Nica, M.: Genetic algorithm-based test parameter optimization for ADAS system testing. In: IEEE 19th International Conference on Software Quality, Reliability and Security (QRS), pp. 418–425 (2019). IEEE
- ISO, ISO/PAS 21448:2019 Road vehicles — Safety of the intended functionality, 2019.
- Simens, White paper: Scenario-based verification and validation of self-driving vehicles: relevant safety metrics, 2022.
- Ponn, T., Breitfuß, M., Yu, X., Diermeyer, F.: Identification of challenging highway-scenarios for the safety validation of automated vehicles based on real driving data. In: 15th International Conference on Ecological Vehicles and Renewable Energies (EVER), pp. 1–10 (2020). IEEE
- Gambi, A., Huynh, T., Fraser, G.: Generating effective test cases for selfdriving cars from police reports. In: Proceedings of the 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, pp. 257–267 (2019)
- Ding, W., Chen, B., Xu, M., Zhao, D.: Learning to collide: An adaptive safety-critical scenarios generating method. In: IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 2243–2250 (2020). IEEE
- Klischat, M., Althoff, M.: Generating critical test scenarios for automated vehicles with evolutionary algorithms. In: IEEE Intelligent Vehicles Symposium (IV), pp. 2352–2358 (2019). IEEE
- Gambi, A., Mueller, M., Fraser, G.: Automatically testing self-driving cars with search-based procedural content generation. In: Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis, pp. 318–328 (2019)
- Song, Qunying, Kaige Tan, Per Runeson, and Stefan Persson. "An Industrial Workbench for Test Scenario Identification for Autonomous Driving Software." In 2021 IEEE International Conference on Artificial Intelligence Testing (AITest), pp. 81-82. IEEE, 2021.
- Song, Qunying, Kaige Tan, Per Runeson, and Stefan Persson. "Critical Scenario Identification for Realistic Testing of Autonomous Driving Systems." (2022).