Protecting ML models running on edge devices and mobile apps

1. Security challenges for Machine Learning models
2. System architecture for delivering and executing ML models on edge device
3. Risks and threats of running ML models in mobile apps
4. Security defences for ML models
   • ML model encryption
   • Cloud and API protection
   • Mobile application anti-tampering controls
   • Proactive anti-fraud security measures
5. Conclusion

ML models are unique combinations of data and algorithms that have been trained on massive volumes of data to provide answers, classify incoming data, and transform it. Companies value these models as strategic assets able to extract patterns from large datasets, improve efficiency of operations, quality control, predict maintenance, reduce human efforts and unlock competitive advantage.

Machine Learning model security involves protecting the model and the data it operates on. Depending on risk model, ML model security could consist of several overlapping layers: encryption, obfuscation, secure execution environment, reverse engineering protection, watermarking, and so on. In this guide, we will introduce you to the unique security challenges of ML models while focusing on real-world risks and practical ML model security measures.

Security challenges for Machine Learning models #

ML-based solutions covered in this article consist of the data used for training and testing, the processes of training, testing, and deploying machine learning models, the infrastructure supporting all of the processes, and the ML model itself.

Effective attacks against machine learning occur throughout the lifecycle, including design, implementation, training, testing, and production deployment. The nature and power of these attacks vary: They can exploit both ML model vulnerabilities and gaps in the infrastructure that support AI systems.

The MITRE ATLAS Matrix below is a knowledge base of adversary tactics and techniques based on observations of actual attacks and realistic demonstrations from AI red teams and security groups:

This article focuses on protecting ML models deployed on edge devices, such as mobile phones and IoT modules, from leakage and misuse. It excludes threats and attacks on model training and testing, like poisoning attacks, evasion attacks, etc. If you are interested in getting a better understanding of the entire landscape of ML attacks, we recommend the NIST AI 100-2 document Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations.

Quite often, Machine Learning models are run on cloud infrastructure, leveraging MLaaS to handle extensive data processing and computations within a controlled environment that provides isolation and secure execution.

Deploying machine learning models to edge devices is a business need. It reflects the trend towards decentralised computing, the desire to process data closer to its source or the user, and the decision to offload computational load from backend servers.

This approach boosts real-time data processing and optimises network bandwidth usage. By processing data on the client side, this strategy shifts responsibility for user data security—transmission, processing, and storage—away from service providers, hence improving compliance with privacy and data protection regulations.

The edge devices can include a variety of hardware such as mobile apps on mobile phones, IoT devices like smart home appliances, and specialised Industrial Internet of Things (IIoT) devices used in manufacturing and industrial settings.

However, executing machine learning models on edge devices creates additional security risks of models being leaked or stolen that must be addressed. Imagine spending millions of dollars training a cutting-edge machine learning model to have it exposed in a mobile app or compromised in a specialised device.

On-device ML models needs a comprehensive approach to security, with an emphasis on protecting the models and data they process at the edge while maintaining their integrity and confidentiality, and having enough information to detect breaches and tampering.

System architecture for delivering and executing ML models on edge device #

Let’s discuss system architecture and how ML models reach the edge device. In this case, a mobile phone will serve as our edge device, with a mobile app receiving personalised ML models customised per user behaviour. ML models, in turn, are trained by cloud workers using TensorFlow, then stored on cloud storage systems such as Google Cloud Platform or AWS S3, processed and sent to the mobile apps via API, and finally stored locally and run on the mobile app.

The architecture involves the following key components:

1. Mobile applications: Native iOS and Android apps allow users to interact with the data, processed by the ML models.
2. API: Serves as a communication layer between mobile apps and cloud services with the responsibility of customising models for each user.
3. Google Cloud Engine (GCE) Worker with TensorFlow (TF): Responsible for generating individualised machine learning models (IMLs) to be sent to the exact user on their phone.
4. Google Cloud Platform (GCP) Storage: Stores the IMLs before sending them to the mobile apps.
5. Training servers: The primary ML infrastructure where models are developed and trained using large datasets.

This dataflow demonstrates the ML model’s omnipresence: It exists in the cloud, in transit, and on the mobile phone. Let’s discuss the security layer that goes through all these platforms.

Risks and threats of running ML models on edge devices #

With the provided architecture, it’s easier to identify the risks and threats associated with deploying ML models on the client-side devices such as mobile phones, consumer IoT, and industrial IoT systems.

Note: While the architecture above serves as an example, it’s important to acknowledge that each system can present its own set of security challenges.

According to the abovementioned NIST AI 100-2, research on security vulnerabilities in ML highlights the need for a well-rounded security approach that includes software, data, and model supply chains, as well as network and storage systems.

When the model runs on the backend, it is mainly vulnerable to black-box attacks (the attackers don’t know ML model architecture). However, when the model is transferred to the user’s side, it becomes susceptible to white-box attacks, where attackers have full control over the model and its parameters, which are significantly more effective.

When using ML models on the mobile applications and specialised edge devices, the following threats must be considered:

• Unauthorised access: Leakage of data collected and processed by ML model, or the ML model itself.
• API vulnerabilities: Leakage of sensitive information (users data and credentials), as well as the misuse of ML model generation services.
• Cloud storage breach: Access to cloud-stored ML models through data collection, backup recovery, or log access.
• Model interception and tampering: The model could be intercepted and tampered with during data transfer via passive or active man-in-the-middle (MitM) attacks.
• Reverse engineering and firmware vulnerabilities: Extraction of ML models through reverse engineering methods, firmware and software specific vulnerabilities, vulnerabilities in supply chain components, and third-party libraries.
• Scalability and interoperability issues: Integrating ML models with existing edge devices can be challenging. Incompatibility between different ML model versions, devices and platforms can lead to security gaps and operational inefficiencies.
• Side channel attacks and resource constraints: Edge devices often have limited processing power and memory, which can restrict the effectiveness of ML models protection. Side channel attacks exploit physical or logical characteristics of edge devices, such as power consumption, electromagnetic leaks, or timing information, to gain insights into the ML model or the data it processes.
• Limited network connectivity: Many edge devices operate in environments with limited or no internet access, making it difficult to regularly update ML models. The connectivity limitation also complicates the implementation of DRM-like solutions for ML security that depend on internet access.

Investing in security for Machine Learning models is a wise choice, especially considering the potential risks and threats. The cost of implementing efficient security measures should be less than the potential losses from breaches or intellectual property theft.

Security defences for ML models #

One of the main sensitive assets is the ML model itself—an output of model training system/process, containing structured data and procedures (weights).

Protecting the ML model’s integrity and confidentiality requires a complex approach with several lines of defence, ranging from model encryption to application security controls, anti-reverse engineering checks, anti-fraud measures, and execution restrictions. Let’s discuss them one by one.

Read about protecting unique IP (ML models) against leakage and misuse.

ML & TensorFlow

AI/ML

Cryptographic IP protection for AI/ML product

Protecting unique IP (ML models) against leakage and misuse using multi-layered encryption on ephemeral keys.

Read story

ML model encryption #

Encryption protects ML models in storage and during transmission, ensuring even if the system is breached, ML models remain encrypted. Also, encryption strengthens model integrity by preventing unnoticed modifications or malicious injections. Using user-specific encryption keys isolate and bind encrypted models to their authorised users.

For secure deployment, the model encryption layer should be fast, seamless, and adaptable across cloud and edge environments.

ML model encryption flow for edge devices

As an illustration, let’s discuss a simple ML model encryption scheme for the particular user (e.g. tied to specific user’s key). Client-side can be anything—a mobile app, an IoT sensor, etc. Depending on real-world context, this scheme could be extended with key generation, secret sharing schemes, fully end-to-end encryption flows, watermark embedding, model weight obfuscation, and much more.

1. After generation, the backend places unique ML models in the S3 buckets/other cloud storage. Each model is encrypted with a unique symmetric key: “Model Encryption Key”. This key is stored on the backend in the key vault.
2. Before downloading a model, the mobile application generates a pair of ephemeral keys using elliptic curve cryptography (ECC). The keys are ephemeral: indicating that they are temporary and won’t be stored or reused for more than one transaction. Cryptographic options for edge devices are more complicated; not all of them support ECC. NIST Lightweight Cryptography was specifically designed to operate efficiently on resource-constrained devices.
3. The mobile app or edge device initiates a model request via an API, sending its authentication token, the public key of the keypair, and some device-specific information for additional context and audit logs, like device ID or fingerprint.
4. In response to the model request, the backend uses public key encryption (ECIES or HPKE) to encrypt the Model Encryption Key specifically for this device. This encrypted key and a pre-signed link for downloading the model from AWS S3 bucket is then sent back to the client device.
5. The client app downloads the model from the S3 bucket, decrypts the Model Encryption Key using the private key of the ephemeral key pair, and the backend’s public key. The keypair is destroyed after the process.
6. The device decrypts the model using the decrypted Model Encryption Key. 7. After receiving the model from the backend, the device re-encrypts the model symmetrically. We recommend using AEAD (such as AES-GCM) or lightweight cryptography ciphers (such as ASCON) with a randomly generated key (stored in key storage where available). The model is stored encrypted and gets decrypted only before usage.

While the cryptographic details may involve additional steps, such as using an KDF for key derivation, the overall approach ensures reliable encryption and decryption.

Depending on the hardware support of the edge device, it could be efficient to use Hybrid Public Key Encryption. HPKE is a cryptographic primitive, defined in RFC 9180, a combination of symmetric and asymmetric cryptography.

In our case, a symmetric cipher with authentication encryption (AES-GCM) is used to encrypt the Machine Learning model. Public key encryption is used to encrypt the Model Encryption Key for the mobile app. As described in the RFC, HPKE outlines various formats and methodologies for combining different ciphers and hash functions. It is also possible to build a lightweight version of HPKE that balances security with hardware restrictions.

Implementation advice for the encryption layer on edge device:

• Minimise plain text exposure: Encrypt models as close to their generation point and store/transmit them encrypted.
• Proper key management: Use unique encryption key per model and ephemeral key pair.
• Simple & efficient cryptography: Stick to industry-proven ciphers (AES-GCM, HPKE, ASCON).
• Cross-platform compatibility: Use cryptographic libraries that have compatible API across mobile firmware/hardware of selected edge devices, and backend platforms. Finding correct encryption parameters might be tricky and could result in unpredictable errors.

Cloud and API protection #

Cryptography alone cannot protect the whole system with Machine Learning models. Integration with application security controls, API security, cloud infrastructure security and fraud prevention is important.

API protection and cloud security measures:

• Strong authentication and authorisation: Ensure that only authorised users and applications can access and download the ML model. We recommend referring to the OWASP ASVS chapters V2 Authentication, V3 Session management, and V4 Access control to determine appropriate security requirements.
• API and transport security: Proper TLS settings, rate limiting, firewalls, HTTP security headers, parameters validation for API requests, DoS protection, and so on. Other OWASP ASVS chapters are waiting for your attention :)
• Monitoring: Use SIEM, monitor logs and security events for suspicious activity and unauthorised access attempts. For example, you can use AWS CloudTrail to monitor all activities related to the S3 bucket.
• Secure distribution: The ML model should be accessible only by a presigned URL. Prevent direct access by configuring corresponding policies. A presigned URL is generated by a Cloud Storage (GCS or S3) for the user who has limited time access to the object. The expiration time and other parameters of the presigned URL can be specified during the URL generation process. Set a strict expiration time on each URL, limiting their validity to a short time frame—around 5 minutes—to minimise the risk of unauthorised access.
• Secure cloud infrastructure: ML models should be processed in secure cloud environments, using features offered by cloud providers. Think about network segmentation for isolating different components of the ML workflow (e.g., training data, model server) to minimise the attack surface. Configure IAM roles to limit who can generate pre-signed URLs. Grant only the necessary permissions and avoid giving full access to the storage. . Refer to the cloud security guidelines, like Google Cloud Platform storage guidelines and AWS S3 bucket configuration.

By implementing the measures mentioned above, you contribute to creating a secure environment for ML model generation, storage, processing, and transfer to the client device side.

Mobile application anti-tampering controls #

As an illustration of an edge component, let’s discuss how local on-device protection work in a mobile app.

In the context of mobile application security, particularly when deploying Machine Learning models inside the mobile application, a sophisticated approach to reverse engineering controls is required. Refer to OWASP MASVS V8: Resilience Requirements to learn more about resilience controls.

1. Root and jailbreak detection: Essential for preventing downloading ML model to compromised devices and executing on it. These measures detect and respond to indicators of root access.
2. Runtime integrity checks: Critical for verifying that the model is run in the application that operates in its authentic, unmodified state.
3. Anti-debugging techniques: Deters analysis through debugging by detecting debugging environments or behaviours.
4. Resource integrity verification: Prevents tampering with application assets by using hash validation and digital signatures to ensure all resources remain untampered.
5. TLS certificate pinning: Protects against man-in-the-middle attacks by ensuring the app communicates exclusively with the intended server.
6. Code obfuscation: Serves as one of the barriers against reverse engineering by making code difficult to read and interpret.

A single security measure cannot guarantee absolute protection. This is especially true when attackers possess physical access to the device, time, and advanced technical skills. Security through obscurity should be considered an additional layer of defence, not the foundation of a strong security strategy.

As OWASP MASTG highlights, “The lack of any of these measures does not cause a vulnerability—instead, they are meant to increase the app’s resilience against reverse engineering and specific client-side attacks.”

Focusing on a single check for attack identification is unreliable. For a publicly available app, a persistent attacker can bypass anti-debugging controls by patching the binary or dynamically modifying the app’s behaviour at runtime with tools such as Frida. For a stronger defence, these checks should complement other security measures outlined in this article.

Security guidance for protecting ML models locally on IoT devices differs in detail, but the high-level approach remains largely the same.

Proactive anti-fraud security measures #

Another line of defence is a proactive anti-fraud system which monitors user and application behaviour. It determines whether the user’s actions deviate from their typical patterns, indicating that they might be misusing or abusing ML model usage.

What indicators should be taken into account?

The following user behaviour patterns increase risk scores: Switching device ID often under the same user account, multiple identical device IDs simultaneously online, downloading ML models many times during the session, having multiple user sessions at the same time, and repeated app reinstalls.

The following stop factors might indicate that the user is actively trying to break the system: detected device tampering (for mobile: rooted or jailbroken device), invalid application signature, failed remote device attestation check, or triggered honeypot APIs.

Based on the calculated risk score and stop factors, the system dynamically reacts, limiting the usage of ML models:

• Low risk scores result in no limits in ML model usage, contributing to a positive user experience.
• When risk scores exceed a certain value, the application might trigger additional verification steps, repeated authentication, limit API usage, introduce delays, etc.
• High risk scores or stop factors mean that the app should delete downloaded ML models, and the backend should restrict the user account.

Device integrity and app validation mechanisms for mobile apps

Features like AppAttest for iOS and Play Integrity for Android allow to verify application authenticity and device integrity. This verification should be done before downloading the ML model to the mobile app. Both AppAttest API and PlayIntegrity API ensure that the app was downloaded from a legitimate store (App Store or PlayMarket) and check for operating system errors.

Device Integrity and validation of IoT and IIoT devices Trusted Platform Modules (TPMs) provide a hardware-based root of trust allowing for the secure storing of cryptographic keys and integrity monitoring of device firmware and software. Services like Google Cloud IoT Core Device Attestation use TPMs for remote validation of device identity and integrity. Before deploying ML models, ensure that the devices pass these checks to prevent unauthorised access or tampering.

Honeypots for the win

Integrating Honeypot API calls or Honeypot source code files into the software that works with ML model also helps detect unusual user behaviour. Decoy endpoints emulate real API services, trapping attackers who attempt to reverse-engineer the app’s logic. Honeypots aren’t limited to API calls or source code – they can also include fake encryption keys, model files, API tokens, and more.

We recommend strategically placing honeypot APIs that are programmatically invoked from seemingly legitimate, yet unused, code sections. This design makes the traps look real. The application’s backend continuously monitors these endpoints, logging any incoming calls. Detected interactions trigger the blocking of suspicious device IDs or IP addresses.

The goal is to allow only real and verified applications to use the ML models, reducing the likelihood of theft or misuse.

Conclusion #

Protecting ML models, particularly those deployed on mobile apps, IoT, IIoT and other edge devices, requires a multi-layered approach and proactive security measures. By employing encryption, adhering to security best practices, and implementing advanced anti-fraud systems, you may improve the overall security of assets such as ML models and supporting infrastructure and mitigate the risks associated with reverse engineering and other security threats. Remember, it may be more worthwhile to invest in security protections of intellectual property than to face the costs of its leakage.