Models & Evaluation

Adversarial Attacks (AI)

Adversarial attacks are inputs deliberately perturbed to make a machine learning model misclassify or misbehave, often with changes too small for a human to notice. They expose a basic fragility in how neural networks map inputs to outputs.

What an adversarial attack is

An adversarial attack feeds a model an input that has been deliberately changed to force a wrong or harmful output. The change is usually tiny, calibrated against the model's own internals so that it shifts the prediction while staying nearly invisible to a person. A photo that any human reads as a stop sign can be nudged, pixel by pixel, into something a vision model labels as a speed-limit sign, with the two images looking identical side by side.

The phenomenon was first documented in computer vision but it is a general property of high-dimensional learned models, not a quirk of one architecture. The same idea now shows up in audio systems, malware classifiers, recommendation engines, and large language models. Wherever a model has learned a decision boundary from data, there tends to be a nearby input that lands on the wrong side of that boundary.

The defining trait is intent: the perturbation is searched for, not random noise.
Perturbations are often constrained to be imperceptible, but they do not have to be.
An attack can target a specific wrong label (targeted) or any wrong label (untargeted).

Where it started: Szegedy 2013 and FGSM

The seminal result came from Szegedy and colleagues in the 2013 paper "Intriguing properties of neural networks" (arXiv 1312.6199). They showed that a deep network's learned input-output mapping is surprisingly discontinuous: an imperceptible perturbation, found by maximizing the network's prediction error, could flip a confident classification. These crafted inputs became known as adversarial examples.

In 2014, Goodfellow, Shlens, and Szegedy published "Explaining and Harnessing Adversarial Examples" (arXiv 1412.6572), which argued that the vulnerability stems largely from the near-linear behavior of neural networks in high dimensions rather than from overfitting. That view produced a fast recipe for building adversarial examples, the Fast Gradient Sign Method (FGSM). The paper's canonical demonstration takes an image that GoogLeNet labels "panda" at 57.7 percent confidence, adds a perturbation scaled by ε = 0.007, and gets "gibbon" at 99.3 percent confidence, with the two images indistinguishable to a human.

x_adv = x + ε · sign(∇ₓ J(θ, x, y))

Fast Gradient Sign Method (FGSM). The clean input x is pushed by a fixed step ε in the direction that most increases the loss J for the model with parameters θ on true label y. Taking the sign of the gradient ∇ₓJ makes every pixel move by the same magnitude, bounding the change so it stays nearly imperceptible.

python

import torch

def fgsm(model, x, y, epsilon, loss_fn=torch.nn.functional.cross_entropy):
    x = x.clone().detach().requires_grad_(True)
    logits = model(x)
    loss = loss_fn(logits, y)
    loss.backward()                      # gradient of loss wrt input
    x_adv = x + epsilon * x.grad.sign()  # one FGSM step
    return x_adv.clamp(0, 1).detach()    # keep a valid image

A minimal FGSM implementation in PyTorch. The gradient is taken with respect to the input, not the weights, and the perturbation is the signed gradient scaled by epsilon.

Szegedy et al. 2013 (1312.6199): adversarial examples exist and transfer across models.
Goodfellow et al. 2014 (1412.6572): introduced FGSM and the linearity explanation.
The panda-to-gibbon image is the textbook illustration of an imperceptible attack.

Threat models: white-box vs black-box

How much the attacker knows changes everything. In a white-box setting the attacker has full access to the model, including its architecture, weights, and gradients, so methods like FGSM that read the gradient directly apply. This is the strongest assumption and the standard one for measuring how a model holds up in the worst case.

In a black-box setting the attacker can only query the model and observe outputs, with no access to internals. Attacks then estimate gradients from query responses or, more cheaply, exploit transferability: an adversarial example crafted against one model often fools a different model trained on similar data. That transfer property, noted in the original 2013 work, is what makes black-box attacks against deployed systems practical.

White-box: full access to weights and gradients; the worst-case benchmark.
Black-box: query-only access; relies on gradient estimation or transferability.
Transferability lets an attacker craft examples on a local surrogate and fire them at a hidden target.

A taxonomy of attacks

Adversarial examples that fool a deployed model at prediction time are only one family. The NIST taxonomy of adversarial machine learning (NIST AI 100-2) organizes the broader space by where in the lifecycle the attack lands and what the attacker wants. For predictive AI it defines three top-level categories, evasion, poisoning, and privacy, the last of which groups attacks aimed at the model or its training data rather than at its accuracy.

Evasion attacks operate at inference time and are the classic adversarial-example case: perturb a test input so the model gives the wrong answer. Poisoning attacks operate at training time by tampering with the training data or process so the model learns a flaw or a backdoor. Privacy attacks target the model or its data: model extraction queries a model to steal a functional copy of it or its parameters, and membership inference probes a model to learn whether a specific record was in its training set. NIST nests both extraction and membership inference under privacy.

Evasion: inference-time perturbation to cause misclassification (FGSM and successors).
Poisoning: corrupt the training data or process, sometimes planting a backdoor.
Privacy (model extraction): reconstruct a model's behavior or weights through queries.
Privacy (membership inference): determine whether a given example was in the training data.

From pixels to prompts: the LLM analog

Large language models inherit the same fragility, just expressed in tokens instead of pixels. Researchers have found adversarial suffixes, short strings of seemingly nonsensical characters appended to a request, that reliably push aligned models into producing content their safety training was meant to refuse. These suffixes are often discovered with gradient-based search, the text-domain cousin of FGSM, and they transfer across models much as image attacks do.

The broader practice of crafting inputs that defeat a model's guardrails is jailbreaking, and prompt injection is the analog that targets applications wired around an LLM: malicious instructions hidden in retrieved documents, web pages, or tool output hijack the model's behavior. All three are adversarial attacks in the original sense, an input engineered to make the model misbehave, which is why defending an LLM application borrows heavily from the adversarial-defense playbook built for vision.

Adversarial suffixes are gradient-found token strings that bypass alignment and transfer across models.
Jailbreaking is the LLM equivalent of an evasion attack on a safety classifier.
Prompt injection extends the threat to the application layer, via untrusted content the model reads.

Defenses and the arms race

The most effective general defense is adversarial training: generate adversarial examples during training and teach the model to classify them correctly, which hardens the decision boundary at real computational cost. Input preprocessing defenses try to scrub perturbations before they reach the model through steps like denoising, quantization, or feature squeezing, but many of these were later shown to fail against attacks that account for the preprocessing.

Certified defenses aim for guarantees rather than empirical hope. Randomized smoothing, for example, turns a base classifier into a smoothed one by averaging predictions over noisy copies of the input, and yields a provable radius within which the prediction cannot change. Even so, no defense is a silver bullet: stronger defenses invite stronger attacks, certified radii stay small relative to real threats, and hardening usually trades off against clean accuracy. Adversarial defense is best treated as an ongoing arms race to be managed, not a problem to be closed.

Adversarial training: the strongest empirical defense, and the most expensive.
Input preprocessing: cheap but frequently broken by adaptive attacks.
Certified / randomized smoothing: provable but limited guarantees.
No single defense holds; expect a continuing attack-defense cycle.

Key takeaways

An adversarial attack is a deliberately perturbed input that makes a model misclassify, often invisibly to humans.
FGSM, x_adv = x + ε·sign(∇ₓJ), is the fast white-box method that turned a panda into a 99.3 percent gibbon.
White-box attacks read the model's gradients; black-box attacks rely on queries or on transferability.
Evasion, poisoning, model extraction, and membership inference are distinct attack categories with different goals.
Jailbreaks, adversarial suffixes, and prompt injection are the language-model versions of the same problem.
Adversarial training and certified smoothing help, but no defense is final; it is an ongoing arms race.

Frequently asked questions

It is an input that has been deliberately altered to make a model produce a wrong or harmful output. The alteration is usually small and calibrated against the model so it stays nearly invisible to a human while still flipping the prediction. The attacks were first found in image classifiers and now affect audio models, malware detectors, and large language models.

The Fast Gradient Sign Method computes x_adv = x + ε·sign(∇ₓJ(θ, x, y)). It takes the gradient of the loss with respect to the input, keeps only its sign so every feature moves by a fixed amount, and scales that by a small step ε. The result is a bounded, nearly imperceptible perturbation that pushes the input across the decision boundary in one step.

In a white-box attack the adversary has full access to the model's weights and gradients, so gradient-based methods like FGSM apply directly. In a black-box attack the adversary can only send queries and observe outputs, so it estimates gradients from responses or exploits transferability, where an example crafted on one model also fools another. White-box is the worst-case benchmark; black-box is closer to attacking a deployed service.

It is the canonical demonstration from Goodfellow et al. 2014. The GoogLeNet model labels an image as a panda at 57.7 percent confidence, then a tiny perturbation scaled by ε = 0.007 is added. The model now labels it a gibbon at 99.3 percent confidence, even though the two images look identical to a person, which shows how confidently a model can be wrong.

Yes. A jailbreak is an input crafted to defeat a language model's safety training, the text-domain version of an evasion attack, and adversarial suffixes are gradient-found strings that do this and transfer across models. Prompt injection extends the same idea to the application layer by hiding malicious instructions in content the model reads. All are inputs engineered to make a model misbehave, the original definition of an adversarial attack.

No defense is complete today. Adversarial training is the strongest empirical defense but raises training cost and tends to lower clean accuracy, and many input-preprocessing defenses have been broken by adaptive attacks. Certified methods like randomized smoothing give provable guarantees, but only within small radii. Defense is best treated as an arms race to manage rather than a solved problem.

Put the idea into practice

MemX is an AI memory agent built on these ideas: store anything, skip the folders, and find it again by asking in plain English.

Try MemX Free