DataTerminology.comNEW
Also known as: Model Extraction
An adversary reconstructs a target model’s functionality by querying it and training a substitute, often via a prediction API.
DESCRIPTION
Model stealing, also called model extraction, replicates a model by observing its inputs and outputs. Attackers use systematic queries to train a substitute approximating the original system’s decision boundary. This threatens intellectual property, safety, and safety filters. Defenses include watermarking/fingerprinting, query monitoring, output perturbation, rate limiting, and legal protections. Differential privacy and confidence-limited outputs can reduce leakage. Continuous monitoring and red teaming help surface extraction attempts.
ALTERNATIVE DEFINITIONS
arXiv
We show that a simple and widely available attack can steal the functionality of machine learning models via prediction APIs.
Introduces the core concept of stealing model functionality through APIs.
Microsoft Learn
Model extraction (stealing) attacks attempt to recreate a model by observing its outputs to carefully crafted inputs.
Highlights careful input crafting and output observation as the attack mechanism.
RELATED TERMS
Membership Inference Adversarial Machine Learning Prompt Injection API Security Confidential Computing Rate Limiting Threat Modeling
COMPETING TERMS
Watermarking Model Fingerprinting Differential Privacy
KNOWLEDGE AREAS
Data Security AI Governance AI MLOps
TARGET AUDIENCE
AI Engineer MLOps Engineer DevOps Engineer Data Governance Manager Chief Technology Officer Security Architect
EXTERNAL SOURCES
- Wikipedia - Adversarial machine learning#Model extraction attacks
- Tramèr et al. - Seminal model extraction paper
- Microsoft - Security threats and mitigations
- arXiv - Survey on stealing and defenses
TOP QUESTIONS
Why is Model Stealing a serious threat?
It compromises IP, safety features, and revenue by enabling clones. Attackers avoid training costs and can remove guardrails.
Category: Risk
How does Model Stealing typically work?
Attackers query a target model, collect input-output pairs, and train a surrogate model to mimic its behavior.
Category: Explanation
How can I detect Model Stealing attempts?
Monitor for unusual query patterns, high-volume sampling, boundary probing, and distribution shifts in inputs.
Category: Diagnostics
What defenses exist against Model Stealing?
Apply watermarking/fingerprinting, rate limiting, output obfuscation, differential privacy, and legal terms enforcement.
Category: Techniques
Does confidence truncation help against Model Stealing?
Yes, returning top-1 labels or rounded probabilities reduces information leakage and hinders precise boundary recovery.
Category: Best practices
Can adversaries still steal models under rate limits?
Yes, but limits raise cost and time. Combined defenses improve resilience.
Category: Risk
What’s a common mistake defending Model Stealing?
Relying on a single defense. Layered controls and monitoring achieve better protection.
Category: Mistakes
Is Model Stealing illegal?
It may violate terms of service, IP, or anti-circumvention laws, varying by jurisdiction. Consult legal counsel.
Category: Compliance
How does Model Stealing relate to fine-tuning?
Attackers may steal a base model then fine-tune privately. Fingerprinting can detect such derivatives.
Category: Relationship