Building AI System Security Defenses with Endogenous Security Thinking

[baoshi.rao]

Currently, many AI system models and algorithms struggle to ensure data quality and cleanliness during training, with significant issues in model design security and training stability. Therefore, for the ubiquitous and widespread AI application systems, various endogenous security problems and dangers are bound to emerge endlessly.

AI application systems also face common endogenous security issues in their hardware environments, such as vulnerabilities and backdoors, which pose significant challenges in cyberspace. Vulnerabilities are difficult to completely avoid, backdoors cannot be eradicated, and current technical capabilities cannot thoroughly investigate these vulnerabilities—these are the common problems.

At the recently held 11th Internet Security Conference (ISC 2023) in Beijing, Academician Wu Jiangxing of the Chinese Academy of Engineering delivered a speech titled Dynamic Heterogeneous Redundancy (DHR) Architecture Empowers Endogenous Security Experiments in AI Application Systems. Wu Jiangxing pointed out that the AI era is a "double-edged sword," bringing tremendous advancements to human society while also posing potential massive security threats, even catastrophic risks. It is crucial to prioritize security in the AI era and address shortcomings using endogenous security methods.

Currently, AI applications centered on deep learning are experiencing a new wave of rapid development. Breakthroughs in artificial intelligence technology, including generative AI, are providing informatization, digitalization, and intelligent solutions across various fields, triggering significant transformations in economic structures and driving overall leaps in social productivity.

AI security issues include both endogenous and non-endogenous security problems, with endogenous security further divided into specific and common issues. AI application systems consist of foundational hardware and software, environmental or data systems, and AI algorithms and models. However, deep learning AI models face three potential problems:

1. The black-box nature of neural networks leads to AI's inexplicability.
2. Deep learning's over-reliance on training samples results in indeterminable learning outcomes.
3. The irreversible forward propagation of neural networks leads to non-deducible results. These are the "three indeterminables" of AI.

Regarding specific issues, from data collection and model design to training, pattern discovery, and task execution based on optimization, there are four characteristics: algorithmic opacity, data dependency, model security, and input sensitivity.

Beyond specific issues, AI application systems also face common endogenous security problems in hardware environments, such as vulnerabilities and backdoors, which are major challenges in cyberspace. Vulnerabilities are hard to completely avoid, backdoors cannot be eliminated, and current technical capabilities cannot fully investigate these vulnerabilities—these are the common problems.

In AI application systems, specific and common endogenous security issues often intertwine and overlap, including problems caused by vulnerabilities and backdoors as well as those arising from the black-box effect. This makes AI application systems more complex in terms of security, presenting unprecedented challenges for safe usage and maintenance.

Wu Jiangxing proposed using the DHR architecture from endogenous security theory to empower AI application systems with endogenous security. This approach can effectively block and control common endogenous security issues, preventing them from escalating into security incidents. Recent theoretical research and engineering practices demonstrate that endogenous security can fundamentally address common endogenous security problems in AI system hardware and software environments. It provides a "three-high" (high reliability, high trustworthiness, high availability) endogenous security foundation for the entire AI application process—from data collection and model training to algorithm application—without relying on or rejecting any additional defensive measures.

On this basis, Wu Jiangxing conducted a theoretical analysis of personalization issues and arrived at seven conclusions:

1. High-quality construction of training datasets, particularly using data with high probability density distributions, is more critical for enhancing a model's resistance to attacks.

2. Heterogeneous interpretability or differentiated training model methods can yield diverse AI models, as endogenous security requires a varied environment for relative judgment.

3. Based on the lottery ticket hypothesis, it's proven that deep learning networks with different structures or varying degrees of sparsity within the same structure are unlikely to produce common modes.

4. Using neural architecture search technology can rapidly construct multiple deep learning AI models with specificity.

5. The transferability of adversarial samples is strongly correlated with the structure of deep neural networks, making redundant deployment of heterogeneous models an essential security mechanism.

6. Theoretically, it can be proven that random dynamic deployment methods for multiple heterogeneous models can significantly enhance the security of AI application systems.

7. AI model uncertainty estimation methods can provide optimal theoretical support for DHR architecture to execute dynamic scheduling algorithms.

Wu Jiangxing believes these theoretical studies demonstrate that even AI personalization issues can be addressed using endogenous security methods. Whether dealing with common or personalized problems, they provide engineered solutions based on AI model construction and offer theoretical support at the foundational level.