Machine learning principles. Secure operation

Content

Understand and mitigate the risks of using continual learning (CL)
Appropriately sanitize inputs to your model in use
Develop incident and vulnerability management processes

Here we are at the conclusion of the series of articles on the principles of the National Cyber Security Centre that help make informed decisions about the design, development, deployment, and operation of machine learning systems. Today, we will discuss the final section, which focuses on recommendations that apply to ML components once they are in operation.

Understand and mitigate the risks of using continual learning (CL)

Using CL is essential for ensuring the safety and performance of the system. It helps address issues such as model drift and erroneous predictions by creating a "moving target" for attackers. However, the process of updating the model based on user data can open new avenues for attacks, as the model's behavior may change under the influence of unreliable sources. Retraining on updated data can lead to the loss of old but correct behaviors of the model. Therefore, after retraining, the model should be treated as new.

What can help implement this principle?

Develop effective MLOps so performance targets are achieved before an updated model goes into production

The creation of up-to-date CL models should be an automated process that incorporates best practices in MLOps and appropriate security monitoring. It is important to track data throughout the CL process to identify sources of errors or attacks, as well as to manage model drift when its predictions begin to change due to new data.

Various tools are available for effective MLOps, such as Continuous Machine Learning (CML), Data Version Control (DVC), and MLflow.

It is recommended to use metadata and model parameters as an alert system for significant deviations, which may indicate abnormal or malicious behavior. The frequency of manual testing of model updates will depend on your system and level of risk.

Consider using a pipeline architecture with checkpoints for testing

It is advisable to run several models in parallel so that updates and testing occur before deployment. For this, sandboxed environments can be used to compare updated models or organize experimental deployments targeted at small user groups to interact with new models.

Capture updates to datasets and models in their associated metadata

During the collection of new data in CL, it is important to consider the same security issues as in the development phase. The reliability of data sources must be assessed: are they trusted users or is access open to all? Follow the recommendations for creating metadata for assets that we discussed in the previous article.

Consider processing data and training locally

If you do not plan to update the central model, CL can be conducted locally on the user's device without leaving it. Techniques such as federated learning can be used to make local changes to the central model.

Appropriately sanitize inputs to your model in use

Proper sanitization or preprocessing of data can protect the model from attacks using specially crafted inputs. For example, evasion attacks, where minor changes to an image can deceive the model. In this case, simple JPEG compression may help.

If the model is trained on sensitive data, proper sanitization or anonymization can reduce the risk of data extraction and lower the likelihood of attacks. However, CL creates additional challenges for privacy protection, as it requires collecting data directly from users.

What can help implement this principle?

Implement tracking and filtering of data

Filters in data collection processes and MLOps help cleanse input data, protecting the model from unwanted or malicious information. It is recommended to use standard transformation sets to unify input data during model retraining.

The necessary filters depend on the type of system and the reliability of data sources. Filtered data will likely require regular human oversight, and the filters themselves will need periodic updates to guard against potential bypasses by attackers.

Implement out of distribution detection on model inputs

It is essential to select the best method for out-of-distribution (OoD) detection. Consider using maximum softmax probability and temperature scaling.

For complex models with large feature spaces, it is worth exploring whether you can factor the OoD distance into the model’s confidence scores, which may help in detecting attacks.

Use appropriate techniques to anonymise user data

Methods that ensure privacy and reduce security risks include:

Statistical methods that provide confidentiality without distorting overall statistics.
Techniques that render data unreadable to humans (e.g., encryption, hashing).
Data generalization to anonymize individual contributions while preserving overall trends.
Data swapping between entries while maintaining the underlying statistics of the dataset.
Removal of attributes to prevent data from being linked to a specific record without additional information.

Develop incident and vulnerability management processes

ML systems, like any other software, are susceptible to vulnerabilities, and having vulnerability management processes helps minimize risks. The rapid development of ML and its integration into critical systems make information sharing about vulnerabilities especially important. Receiving feedback from users about potential vulnerabilities in your products will help address security issues and ensure compliance with legal requirements for safe data handling.

What can help implement this principle?

Develop a vulnerability disclosure process for your system and organization

Vulnerabilities are discovered constantly, and security professionals strive to report them directly to the responsible organizations. Such reports help improve system security, as they enable organizations to respond promptly to vulnerabilities, reducing the risk of compromise and reputational damage from public disclosure.

Develop a process for responsibly sharing relevant threat intelligence

Share knowledge about cyber threats on specialized platforms. In a previous article, we discussed the benefits and risks of publishing details about your systems. Nonetheless, strive to find ways to share information that will help others in developing secure ML while avoiding the disclosure of critical data.

We have finally covered all the NCSC principles regarding interaction with machine learning. We hope that the information has been helpful and has provided you with a better understanding of the key aspects of ensuring the security and reliability of ML systems.

You can find previous reviews of other sections of the article below ⬇️

Secure design

Secure development

Secure deployment