Training & Alignment

Transfer Learning

By Aditya Kumar Jha, Engineer

Transfer learning reuses knowledge a model gained on a source task or domain to improve learning on a related target task, usually by pretraining on large data and then adapting the pretrained weights to a smaller target dataset.

What Transfer Learning Is

Transfer learning is the practice of taking a model trained on one problem, the source task, and reusing part of what it learned to do better on a different but related problem, the target task. Instead of starting from random weights, the target model starts from weights that already encode useful structure. This is most valuable when the target task has limited labeled data, because the pretrained model supplies general features that would otherwise require a large dataset to learn from scratch.

The dominant recipe is pretrain-then-adapt. A model is first pretrained on a large, often general dataset, then adapted to the target task with comparatively little data and compute. The 2010 survey by Sinno Jialin Pan and Qiang Yang formalized the field, defining a domain as a feature space plus a marginal distribution and a task as a label space plus a predictive function, and framing transfer learning as improving the target predictive function using knowledge from a source domain and task.

Transfer learning matters because data and compute are expensive. Reusing a pretrained backbone often reaches higher accuracy, trains faster, and generalizes better than training a fresh model, especially in the low-data regime where from-scratch training tends to overfit.

  • Source task supplies reusable knowledge; target task is what you actually want to solve.
  • Pretrain on large general data, then adapt to a smaller target dataset.
  • Pan and Yang (2010) gave the standard domain-and-task formalization.
  • Most useful when target labels are scarce or expensive to obtain.

Feature Extraction vs Fine-Tuning

There are two main ways to adapt a pretrained model. In feature extraction, the pretrained network, the backbone, is frozen and used as a fixed feature encoder. Only a new task-specific head, often a single linear classifier, is trained on top of those features. In fine-tuning, some or all of the pretrained weights are also updated during target training, letting the representation itself shift toward the target task.

Feature extraction is cheaper and resists overfitting because few parameters change, which suits very small target datasets or cases where the source and target are closely related. Full fine-tuning is more flexible and usually more accurate when the target dataset is larger or differs more from the source distribution, at the cost of more compute and a higher risk of overfitting. A common middle path is to freeze the lower layers and fine-tune only the upper ones, or to use a smaller learning rate on pretrained layers than on the new head.

Fine-tuning can be written as a standard supervised objective whose only special feature is its starting point: the weights are initialized from the pretrained parameters rather than from random values, which biases optimization toward solutions near the pretrained representation.

θ* = argmin_θ (1/N) Σᵢ L( f_θ(xᵢ), yᵢ ), with θ initialized at θ₀ (pretrained)
Fine-tuning is ordinary empirical risk minimization over target pairs (xᵢ, yᵢ); the transfer comes from starting optimization at pretrained weights θ₀ instead of random initialization.
Feature extraction: freeze g_φ, train only h_ψ; predict ŷ = h_ψ( g_φ(x) ), optimize ψ only
In feature extraction the backbone g_φ stays fixed and only the new head h_ψ is trained.
  • Feature extraction: freeze the backbone, train only a new head.
  • Fine-tuning: update some or all pretrained weights for the target task.
  • Smaller or noisier target data favors feature extraction or partial fine-tuning.
  • Lower learning rates on pretrained layers protect learned features.

Why It Works

Transfer learning works because deep networks tend to learn features that grow from general to specific as you move from input to output. In convolutional neural networks trained on images, early layers learn generic primitives such as edges, color blobs, and simple textures, while later layers learn task-specific combinations like object parts and whole objects. The early, general features are useful far beyond the original task, so they transfer well.

Language models show an analogous pattern. A model pretrained to predict tokens over a large corpus acquires broad knowledge of vocabulary, syntax, and many factual and semantic regularities. These general capabilities carry over to downstream tasks like classification, question answering, or extraction, where the pretrained representation already encodes much of what the target task needs.

Because the lower layers are reusable, transfer concentrates target-data demand on the smaller set of task-specific parameters. This is why a model pretrained on millions of examples can be adapted with thousands, or even fewer, and still generalize well.

  • Early layers learn general features; later layers learn task-specific ones.
  • CNN early layers learn edges and textures that reuse across vision tasks.
  • Pretrained language models capture syntax and broad knowledge for reuse.
  • Transfer shifts most of the data burden onto a few task-specific parameters.

From ImageNet to BERT and GPT

Transfer learning became standard practice in two waves. In computer vision, the ImageNet dataset, introduced by Jia Deng and colleagues in 2009 with millions of labeled images organized by the WordNet hierarchy, enabled large supervised pretraining. Networks pretrained on ImageNet classification became reusable backbones: practitioners would take such a network, replace its classification head, and fine-tune on a new vision task with far less data.

In natural language processing, the analogous shift came with self-supervised pretraining on raw text. BERT, introduced by Jacob Devlin and colleagues in 2018, is pretrained with a masked language modeling objective and then, in the authors' words, can be fine-tuned with just one additional output layer to reach strong results on many tasks without substantial task-specific architecture changes. Generative pretrained transformers in the GPT line followed the same pretrain-then-adapt logic with a next-token objective.

These two eras established the now-default workflow: train one large model once on broad data, then cheaply adapt it many times to specific tasks. That workflow is the direct ancestor of today's foundation models.

  • ImageNet (Deng et al., 2009) enabled reusable pretrained vision backbones.
  • Swap the classification head and fine-tune for a new vision task.
  • BERT (Devlin et al., 2018) made text pretraining plus fine-tuning standard in NLP.
  • GPT-style next-token pretraining follows the same pretrain-then-adapt pattern.

Domain Adaptation, Negative Transfer, and Foundation Models

Domain adaptation is the case where the source and target share the same task but differ in data distribution, for example a sentiment classifier trained on product reviews applied to tweets. Methods here try to align the source and target feature distributions so source-trained knowledge holds in the target domain. When source and target are too dissimilar, transfer can hurt rather than help, a failure mode called negative transfer, where reusing source knowledge yields worse performance than training on target data alone.

Modern foundation models generalize this picture. A large model is pretrained once, then adapted in several ways that are all forms of transfer: full fine-tuning updates every weight, while parameter-efficient fine-tuning (PEFT) methods update only a small set. LoRA, introduced by Edward Hu and colleagues in 2021, freezes the pretrained weights and trains small low-rank update matrices instead, reducing trainable parameters and memory while keeping the frozen backbone intact. PEFT is the umbrella term for such methods.

Across all of these, the core idea is unchanged: knowledge captured during pretraining is reused to make target learning cheaper and stronger, whether by updating all weights, a few weights, or only a lightweight adapter.

python
import torch, torchvision

# Feature extraction: load a pretrained backbone and freeze it
model = torchvision.models.resnet50(weights="IMAGENET1K_V2")
for param in model.parameters():
    param.requires_grad = False  # freeze the backbone

# Replace the final layer with a fresh head for the target task
num_target_classes = 10
model.fc = torch.nn.Linear(model.fc.in_features, num_target_classes)

# Only the new head's parameters are trainable
trainable = [p for p in model.parameters() if p.requires_grad]
optimizer = torch.optim.Adam(trainable, lr=1e-3)
Feature extraction in PyTorch: load an ImageNet-pretrained ResNet, freeze it, attach a new classification head, and train only that head. For full fine-tuning, leave requires_grad True and use a smaller learning rate.
  • Domain adaptation: same task, different data distribution between source and target.
  • Negative transfer: dissimilar source knowledge can make the target worse.
  • Foundation models pretrain once, then adapt via fine-tuning or PEFT.
  • LoRA freezes weights and trains low-rank updates, a parameter-efficient transfer.

Transfer Learning vs In-Context Learning

Transfer learning is often confused with few-shot and in-context learning, but they differ in a key way. Transfer learning either updates the model's weights through fine-tuning or reuses a pretrained backbone whose weights were learned earlier. In-context learning, by contrast, changes no weights at all: a large language model is given a few examples or instructions inside its prompt at inference time and adapts its behavior for that single request only, with nothing persisted afterward.

Few-shot learning describes the goal of learning from very few labeled examples, and it can be approached either way, through gradient-based adaptation of a pretrained model (a transfer method) or through in-context examples (not a transfer method in the weight-update sense). The practical distinction is durability and cost: transfer produces a modified or specialized model artifact, while in-context conditioning is ephemeral and repeats its prompt cost on every call.

Both rely on a capable pretrained model, so they are complementary. Pretraining builds the general capability; fine-tuning or PEFT bakes target behavior into weights, while in-context learning steers behavior per request without retraining.

  • Transfer learning updates weights or reuses a pretrained backbone.
  • In-context learning changes no weights; it conditions on the prompt at inference.
  • Few-shot is a goal that either approach can pursue.
  • Transfer yields a persistent model; in-context adaptation is per-request and ephemeral.

Key takeaways

  • Transfer learning reuses source-task knowledge to learn a related target task with less data.
  • Pretrain-then-adapt is the standard recipe; Pan and Yang (2010) formalized the field.
  • Feature extraction freezes the backbone and trains a head; fine-tuning updates pretrained weights.
  • It works because lower layers learn general features (edges in CNNs, syntax in language models).
  • ImageNet pretraining in vision and BERT/GPT pretraining in NLP made transfer standard practice.
  • Foundation-model adaptation, full fine-tuning and PEFT methods like LoRA, are all forms of transfer.

Frequently asked questions

It is reusing a model that already learned something useful on one task to do better on a related task. Instead of training from scratch, you start from pretrained weights and adapt them, which needs far less data and compute. It is most helpful when your target task has little labeled data.
Feature extraction freezes the pretrained backbone and trains only a new task-specific head on top of its fixed features. Fine-tuning instead updates some or all of the pretrained weights during target training. Feature extraction is cheaper and resists overfitting, while fine-tuning is more flexible and often more accurate on larger or more different target datasets.
Deep networks learn features that go from general to specific across layers. Early layers capture broadly useful structure, such as edges in image models or syntax in language models, which transfers well to new tasks. Reusing those general layers means the target task only has to learn a small amount of task-specific structure.
Transfer learning updates the model's weights or reuses a pretrained backbone, producing a persistent specialized model. In-context learning changes no weights at all; a language model adapts to examples placed in its prompt for one request and forgets them afterward. Transfer is durable and a one-time cost, while in-context conditioning repeats on every call.
Negative transfer is when reusing knowledge from a source task makes the target task worse than training on the target data alone. It happens when the source and target are too dissimilar, so the transferred features mislead the target model. Choosing a closely related source and adapting carefully helps avoid it.
Yes. LoRA, introduced by Hu et al. in 2021, is a parameter-efficient fine-tuning method that freezes the pretrained weights and trains small low-rank update matrices instead. It adapts a pretrained model to a target task, which is the defining goal of transfer learning, while changing very few parameters.

Put the idea into practice

MemX is an AI memory agent built on these ideas: store anything, skip the folders, and find it again by asking in plain English.

Try MemX Free