With various countries/regions around the world formulating regulations related to artificial intelligence, engineers designing AI based systems must meet the requirements of these new regulations and standards. On October 30, 2023, the White House also issued an executive order on artificial intelligence regulations, emphasizing the importance of robust verification and validation (V&V) processes for AI based systems. This directive requires AI companies to report and test specific models to ensure that AI systems operate as expected and meet specified requirements.

Artificial intelligence regulations and V&V processes will have a significant impact on safety critical systems. Artificial intelligence is increasingly being used in system design, including safety critical applications in fields such as automotive and aerospace industries.

Verification and confirmation in artificial intelligence based systems

Verification aims to determine whether the artificial intelligence model is designed and developed according to specified requirements, while confirmation is to check whether the product meets customer requirements and expectations. By adopting the V&V method, engineers can ensure that the output of artificial intelligence models meets specifications, thereby achieving early bug detection and mitigating the risk of data bias.

One advantage of using artificial intelligence in safety critical systems is that AI models can simulate physical systems and validate designs. Engineers can simulate the entire artificial intelligence based system and use data to test the system in different scenarios, including outlier events. If V&V is executed in safety critical scenarios, it can ensure that AI based safety critical systems can maintain their performance levels in various situations.

Most industries that develop artificial intelligence enhancement products require engineers to follow relevant standards before the product is launched. These certification processes ensure that specific elements are incorporated into such products. Engineers can perform V&V to test the functionality of these elements, making certification easier.

In the automotive industry, ISO/CD PAS 8800 is a proposed standard aimed at explaining the safety related attributes and risk factors of road vehicles. Certification is a mandatory requirement in the aerospace and defense fields. Current standards such as software considerations (DO178C) in onboard system and equipment certification may not directly help address the unique challenges posed by artificial intelligence. Therefore, a new ARP6983 process standard is being developed to provide specifications for the development and certification of aviation safety related products that implement artificial intelligence.

The Deep Learning Toolbox Verification Library and MATLAB Test can help engineers develop software that helps comply with industry standards and simplifies the validation and testing of artificial intelligence models in large systems, thereby keeping them at the forefront of V&V in the aviation and automotive fields.

The aerospace engineering team uses model-based design to manage and coordinate complex requirements, automatically generate code, and rigorously test models and systems.

V&V Artificial Intelligence Processes in Safety Critical Systems

When executing V&V, the goal of engineers is to ensure that artificial intelligence components can meet specified requirements while demonstrating reliability and safety in various operating conditions, so they can be deployed at any time. The V&V process related to artificial intelligence involves executing software assurance activities, including a combination of static and dynamic analysis, testing, formal methods, and real-world operational monitoring.

The V&V process may vary slightly across different industries, but the main steps of the V&V process include:

Analyze the decision-making process to solve black box problems;

Test the model based on representative datasets;

Perform artificial intelligence system simulation;

Ensure that the model runs within an acceptable range.

The following steps in the V&V process are iterative steps. As engineers collect new data, gain in-depth information, and integrate operational feedback, artificial intelligence systems can be continuously improved and perfected.

Analyze the decision-making process to solve black box problems

When using artificial intelligence models to add automation features to a system, engineers face black box issues. Understanding how AI based systems make decisions is crucial for providing transparency, as it enables engineers and scientists to build trust in model predictions and understand decisions.

The feature importance analysis method can help engineers determine which input variables have the greatest impact on model predictions. The working method of this analysis method varies depending on the model (such as tree based models and linear models), but the general process assigns a feature importance score to each input variable. The higher the importance score, the greater the impact of this feature on model decision-making. For safety critical systems in the automotive industry, variables may include environmental factors such as precipitation or the presence and behavior of other vehicles.

Explainability methods help to gain a deeper understanding of model behavior. This method is particularly important when the black box nature of the model makes it impossible for us to use other methods. Taking images as an example, these methods can be used to identify the regions in the image that contribute the most to the final prediction. In this way, engineers can understand the main focus of the model when making predictions.

Test the model based on representative datasets

Typically, engineers evaluate the performance of artificial intelligence models in real-world scenarios to ensure that safety critical systems can operate robustly in these scenarios. Their goal is to identify various limitations to improve the accuracy and reliability of the model. Engineers first collect a large number of representative real datasets and make them suitable for testing by cleaning the data. Then, they will design test cases to evaluate various aspects of the model, such as accuracy and reproducibility. Finally, the engineer will apply the model to the dataset, record the results, and compare them with the expected output. The model design will be improved based on the results of data testing.

Perform artificial intelligence system simulation

With artificial intelligence based system simulation, engineers can evaluate and evaluate the performance of the system in a controlled environment. During the simulation, engineers will create a virtual environment to simulate real systems under various conditions. Firstly, they will define the inputs and parameters required for the simulation system, such as initial conditions and environmental factors. Then, they use software such as Simulink to perform simulations, which will output the system's response to the suggested scenario. Like data testing, simulation results are compared with expected or known results to facilitate gradual improvement of the model.

In order for artificial intelligence models to operate safely and reliably, it is necessary to establish boundaries and monitor the behavior of the model to ensure that it operates within these boundaries. If the model has been trained on a limited dataset and encounters unprecedented data during runtime, one of the most common boundary problems will occur. Similarly, the model may not be robust enough and may lead to unpredictable behavior.

Engineers use methods to alleviate data bias and enhance data to ensure that artificial intelligence models operate within an acceptable range.

One way to alleviate data bias is to make the data used to train artificial intelligence models variable, which helps reduce the model's dependence on repetitive patterns that limit its learning. By utilizing data augmentation methods, it is possible to ensure that data representing different categories and populations are processed fairly and equally. In the autonomous vehicle scene, data enhancement may involve the use of pedestrian photos from different angles to help the model detect pedestrians, regardless of their posture. The data balancing method is usually combined with data augmentation, which includes similar samples from each data class. Taking pedestrians as an example, balanced data means that for each different pedestrian scene, such as different body types, clothing styles, lighting conditions, and backgrounds, the dataset must contain a corresponding number of images. This method can minimize bias and improve the model's generalization ability in various real-world situations.

When deploying neural networks in security critical scenarios, robustness is the primary consideration. Minor and imperceptible changes can bring significant risks, leading to misclassification in neural networks. These interferences may lead to incorrect or dangerous output from neural networks. This situation is particularly concerning in systems where errors can lead to disasters. One solution is to incorporate formal methods into the development and validation process. Formal methods are the use of rigorous mathematical models to establish and prove the correctness attributes of neural networks. By applying these methods, engineers can enhance the network's ability to resist certain types of interference, thereby ensuring that security critical applications have higher robustness and reliability.

The W-shaped development process is a non-linear V&V workflow aimed at ensuring the accuracy and reliability of artificial intelligence models.

Conclusion

In the era of safety critical systems based on artificial intelligence, the V&V process will become crucial for obtaining industry certification and complying with legal requirements. To build and maintain trustworthy systems, engineers need to adopt validation methods to provide interpretability and transparency for the artificial intelligence models running these systems. As engineers utilize artificial intelligence to assist in executing V&V processes, they must explore various testing methods to address the increasingly complex challenges brought about by artificial intelligence technology. In safety critical systems, these tasks ensure that artificial intelligence is used in a responsible and transparent manner.

Author: Dr. Lucas Garcia, Chief Product Manager for Deep Learning at MathWorks