Generative AI for smarter CCAM: enhancing perception, decision-making, and validation (CCAM Partnership)

This Destination includes activities addressing safe and smart mobility services for passengers and goods.

This Destination contributes directly to the Strategic Plan’s Key Strategic Orientations ‘Green transition’, ‘Digital transition’ and ‘A more resilient, competitive, inclusive and democratic Europe’.

In line with the Strategic Plan, the overall expected impact of this Destination is to contribute to the ‘Multimodal systems and services for climate-neutral, smart and safe mobility’.

The main impacts to be generated by topics under this Destination are:

Connected, Cooperative and Automated Mobility (CCAM)

1. Improved mobility for people and goods in all weather conditions, ensuring safe, shared, inclusive, affordable, attractive, and accessible door-to-door mobility, for private and public transport in mixed traffic and confined areas, as well as open roads.
2. Seamless integration of CCAM solutions into existing transport ecosystems to ensure interoperability, promote multimodality, enhance traffic safety, catering to diverse user needs and behaviours.
3. Resilient, climate-neutral, and sustainable mobility solutions with reduced carbon footprints, resulting in greener, less congested, cost-effective, and demand-responsive transport systems.
4. Increased competitiveness of the transport system using secure and hyper-advanced technologies such as real-time perception, situational awareness, and decision-making systems, based on trustworthy Artificial Intelligence (including Edge and Generative AI), satellite navigation, smart traffic management, and tools for software development for CCAM applications.

Multimodal and sustainable transport systems for passengers and goods

1. Enhanced resilience of transport networks through improved operational efficiency for both passenger and intermodal freight transport, future-proofed mobility systems supporting EU competitiveness while ensuring affordable and accessible transport for all passengers.

Safety and resilience

1. Drastic reduction in road fatalities for all types of users, especially on rural areas
2. Improved resilience of the public transport system via the use of AI
3. Advanced technologies and methods for improved reliability in complex environments for aviation

Legal entities established in China are not eligible to participate in both Research and Innovation Actions (RIAs) and Innovation Actions (IAs) falling under this destination. For additional information please see “Restrictions on the participation of legal entities established in China” found in General Annex B of the General Annexes.

Expected Outcome:

Project results are expected to contribute to all the following expected outcomes:

• Availability and integration of advanced, trustworthy, energy-efficient perception systems, exploiting technological advancements of Generative AI (GenAI) to enhance situational awareness and support safe decision-making;
• Enhanced Vulnerable Road User (VRU) safety, based on elevated, more temper-proof perception and understanding of their behaviour and intention predictions;
• Enhanced robustness of CCAM systems - both on-board and on the infrastructure side - in critical situations due to their training, virtual testing and validation in scenarios generated by GenAI, complementing existing scenario databases for the testing and validation of CCAM systems;
• Enhanced understanding of the relevance and limitations of using GenAI for CCAM;
• Tools and harmonised approaches for the use of GenAI in mobility technology development, training and validation, as well as for systemic applications such as traffic management and remote control, integrating them into existing approaches.

Scope:

Pilots and demonstrations using Level 3 and 4 vehicle services face major challenges in perception and decision making, highlighting the necessity for low-latency solutions that enhance responsiveness and situational awareness in real-time operating conditions. This is especially relevant for driving in more complex environments like urban areas, where environmental variance is higher and where new scenarios can be regularly encountered. Furthermore, there is the need to limit the latency, bandwidth and energy use for on-board calculations, as well as the need to enhance the security, privacy and reliability (e.g. scene understanding and prediction of near-future scenario development). For rapid decision-making in interactions with VRUs, this is essential for implementing CCAM-enabled solutions and ensuring scalability.

At the same time, developments of sector-agnostic technologies show advancements -such as GenAI- that can be beneficial for CCAM. First exploratory steps can be expected from a project funded under HORIZON-CL5-2023-D6-01-02^[1] regarding the potential in the virtual generation of edge cases, which could be used for the development, training, virtual testing and validation of CCAM systems.

Further advancements in GenAI applications specifically for the CCAM domain need to be developed, trained and validated^[2]. Thus, proposed actions shall include approaches to exploit further technological advancements for CCAM. Major steps are needed to advance to highly advanced, ultra-safe, trustworthy and energy efficient real-time perception and decision-making systems for automated vehicles, specifically focusing on scalable solutions and the exploitation of GenAI. These advancements should leverage low latency systems or distributed computing resources to facilitate real-time processing, thereby improving system responsiveness and safety. This topic will thus contribute to the AI Continent Action Plan^[3] by fostering AI development and adoption in the automotive sector.

Proposed actions are expected to address all the following aspects:

• Development of tools and approaches for robust environment perception and decision making (at the edge, on-board, at infrastructure or back-office). These approaches shall aim at accelerating and advancing the reasoning of decision making, increasing the level of efficiency, (cyber)-security and reliability of the applications, with path planning as initial use case. This is to support amongst others the perception of VRUs, the prediction of their behaviour and their intentions, and includes data sharing approaches for CCAM solutions to create a larger time window for actions in near accident scenarios. The use of advanced GenAI, including Large Language Models (LLMs), Vision Language Models (VLMs) or Vision Language Action (VLAs) can significantly enhance these capabilities by leveraging their advanced contextual reasoning and pattern recognition. Furthermore, GenAI can complement existing perception systems by improving sensory input interpretation and providing enriched environmental contexts, which enhance decision-making and adaptability.
• Scenario generation of interactions of CCAM enabled vehicles with other road users, which is essential for advances in validation and testing, extending existing datasets and scenarios as GenAI can, based on existing data, deliver variations of scenarios (e.g. cultural differences of road users and infrastructure variability.)
• Integration of GenAI technologies into existing approaches (development, training and validation) for their further enrichment. Understanding the limits of using GenAI technologies as well as the benefits and develop guidelines for valid approaches for this integration (including consideration of gender biases and fairness to ensure AI systems are transparent and accountable) and providing an outlook on the uptake of the tools and approaches developed can be done for a variety of CCAM components and technologies, as well as for systemic applications such as traffic management and remote control.
• Encouraging collaboration with the European Software-defined Vehicle (SDV) initiative by adopting existing interfaces and building blocks, and proposing new ones developed within the project for potential inclusion in the SDV framework.

Proposed actions should include measures to ensure close coordination with the European Connected and Autonomous Vehicle Alliance (ECAVA) announced in the European Automotive Action Plan.

This topic implements the co-programmed European Partnership on ‘Connected, Cooperative and Automated Mobility’ (CCAM). As such, projects resulting from this topic will be expected to report on results to the European Partnership ‘Connected, Cooperative and Automated Mobility’ (CCAM) in support of the monitoring of its KPIs.

Projects funded under this topic are expected to liaise with the ADRA Partnership^[4] in order to explore and leverage complementarities between their respective activities and findings.

Projects resulting from this topic are expected to apply the European Common Evaluation Methodology (EU-CEM) for CCAM^[5].

null

[1] https://cordis.europa.eu/project/id/101146542

[2] This topic also encourages alignment with national projects and initiatives to allow building up of existing efforts and solutions.

[3] https://digital-strategy.ec.europa.eu/en/library/ai-continent-action-plan

[4] AI, Data and Robotics Association: https://adr-association.eu/

[5] See the evaluation methodology here.