Loss Function

What is a loss function?

A loss function is a function that takes a model's prediction and the true target and returns a single number measuring how wrong the prediction is. A larger number means a worse prediction; a perfect prediction usually gives a loss of zero or near zero. That single number is the quantity training tries to make small.

The loss function is the objective of training. It converts a vague goal (the model should be accurate) into a concrete mathematical surface that an optimizer can traverse. Without a loss, there is no notion of better or worse weights, and no signal telling the network which direction to move. The loss is what defines the direction.

In supervised learning, the loss compares a predicted output to a known label. Averaged over a batch of examples, it produces the training loss the optimizer minimizes. The shape of that loss surface, set by the choice of loss function, determines how learning behaves: which mistakes are penalized hardest, how gradients flow, and how stable training is.

• A loss function maps (prediction, true target) to a scalar that measures error.
• Lower loss means a better prediction; training minimizes the average loss over examples.
• The choice of loss defines what counts as a mistake and how hard it is penalized.

Loss, cost, and objective: the terminology

Three terms get used loosely and worth pinning down. Loss usually refers to the error on a single example. Cost (or cost function) usually refers to the loss averaged over the whole training set or a batch. Objective is the most general term: the full quantity the optimizer minimizes, which is often the cost plus a regularization penalty.

In practice the words are interchangeable in most conversations, and many frameworks call everything a loss. The distinction that matters is between the per-example loss and the aggregate the optimizer actually steps on. Gradient descent works on the averaged value, because averaging keeps the gradient scale stable as batch size changes.

Regularization terms such as L2 weight decay are added to the objective to discourage large weights and reduce overfitting. The model then minimizes prediction error and weight magnitude at the same time, trading a small increase in training loss for better generalization.

• Loss: error on one example. Cost: loss averaged over a batch or dataset.
• Objective: the full thing minimized, often cost plus a regularization penalty.
• Regularization (such as L2 weight decay) is added to the loss to curb overfitting.

Mean squared error: the standard regression loss

Mean squared error (MSE) is the default loss for regression, where the model predicts a continuous number such as a price or a temperature. It averages the squared difference between each prediction and its target. Squaring makes all errors positive and penalizes large errors much more than small ones, since the penalty grows with the square of the gap.

That quadratic penalty has consequences. MSE is smooth and has a clean, well-behaved gradient, which makes optimization easy. But because it squares the error, it is sensitive to outliers: a single far-off prediction can dominate the loss. When outliers are a concern, mean absolute error (which penalizes errors linearly) or the Huber loss (quadratic for small errors, linear for large ones) are common alternatives.

Minimizing MSE has a statistical meaning. Under the assumption that targets are corrupted by Gaussian noise, minimizing squared error is equivalent to maximum likelihood estimation. This is why MSE shows up so naturally in regression: it is not an arbitrary choice but the loss implied by a Gaussian noise model.

• MSE averages squared errors; large errors are penalized far more than small ones.
• Smooth and easy to optimize, but sensitive to outliers.
• Equivalent to maximum likelihood under a Gaussian noise assumption; MAE and Huber are outlier-resistant alternatives.

Cross-entropy: the standard classification loss

Cross-entropy loss, also called log loss, is the default for classification, where the model outputs a probability distribution over classes. It measures the gap between the predicted distribution and the true distribution (which puts all its mass on the correct class). The loss is the negative log probability the model assigned to the right answer, so confident correct predictions incur almost no loss and confident wrong predictions incur a large one.

For binary classification with a sigmoid output, the loss is binary cross-entropy: −y log(ŷ) − (1−y) log(1−ŷ), where y is the true label (0 or 1) and ŷ is the predicted probability. For multi-class problems with a softmax output, categorical cross-entropy generalizes this to −Σ yᵢ log(ŷᵢ) across classes. Because the true distribution is one-hot, only the log probability of the correct class actually contributes.

Cross-entropy pairs naturally with softmax and sigmoid because the combination produces a clean, well-scaled gradient: the gradient at the output reduces to (predicted probability − true label). That simplicity is one reason the softmax-plus-cross-entropy pairing is the standard output stage for classifiers and language models, almost always fused into one numerically stable operation.

Cross-entropy is not the only classification loss. Hinge loss, defined as max(0, 1 − t·s) for a true label t of +1 or −1 and a raw score s, is the loss behind support vector machines. It penalizes predictions that are correct but not confident by at least a margin, and ignores examples already classified correctly with room to spare. Cross-entropy dominates in deep learning because it gives a smooth probabilistic gradient everywhere, while hinge loss has a flat zero region that stops pushing once the margin is met. Most classification losses, cross-entropy and hinge included, are convex in the model's raw scores, which is part of why they optimize reliably.

• Cross-entropy (log loss) is the negative log probability assigned to the correct class.
• Binary cross-entropy uses sigmoid; categorical cross-entropy uses softmax over classes.
• Confident-but-wrong predictions are penalized heavily; the softmax-plus-cross-entropy gradient simplifies to (prediction − label).
• Hinge loss, max(0, 1 − t·s), is the margin-based alternative used by support vector machines.

How the loss drives gradient descent and backpropagation

The loss function is the source of the training signal. After a forward pass produces a prediction, the loss scores it against the target. Backpropagation then computes the gradient of that loss with respect to every weight in the network, using the chain rule to propagate the error backward layer by layer. Gradient descent takes those gradients and nudges each weight in the direction that reduces the loss.

This loop is the whole of training: forward pass to get a prediction, loss to score it, backpropagation to compute gradients of the loss, optimizer step to update weights, repeat. Every weight update traces back to the loss. If the loss is poorly chosen, the gradients point in unhelpful directions and the model learns the wrong thing efficiently.

For backpropagation to work, the loss must be differentiable (or differentiable almost everywhere). That requirement is why training losses are smooth surrogates for the metric we actually care about. Classification accuracy, for example, is a step function with zero gradient almost everywhere, so it cannot train a network directly. Cross-entropy is the differentiable stand-in that produces usable gradients while still pushing accuracy up.

import torch
import torch.nn.functional as F

# logits: raw model outputs, shape (batch, num_classes)
# targets: integer class labels, shape (batch,)
logits = model(inputs)
loss = F.cross_entropy(logits, targets)  # softmax + log loss, fused

loss.backward()        # backprop: gradients of loss w.r.t. all weights
optimizer.step()       # gradient-descent update
optimizer.zero_grad()  # reset for the next batch

• Backpropagation computes the gradient of the loss with respect to each weight via the chain rule.
• Gradient descent steps weights in the direction that lowers the loss; every update traces back to the loss.
• The loss must be differentiable, which is why smooth losses like cross-entropy stand in for non-differentiable metrics like accuracy.

Choosing a loss function by task

The right loss follows from the task and the output structure, not from training-speed preferences. Regression problems use MSE by default, or mean absolute error and Huber loss when outliers need to be down-weighted. Binary classification uses binary cross-entropy with a single sigmoid output. Multi-class, single-label classification uses categorical cross-entropy with softmax.

Multi-label problems, where several classes can be true at once, use independent binary cross-entropy per class rather than softmax, because softmax forces the outputs to compete for a fixed total probability. Ranking and retrieval tasks use contrastive or triplet losses that compare relative distances rather than absolute targets. Object detection and segmentation often combine several losses, such as a classification term plus a localization term.

A useful rule: match the output activation to the loss. Linear output with MSE for regression, sigmoid with binary cross-entropy for binary or multi-label, softmax with categorical cross-entropy for single-label multi-class. Mismatching them (for example softmax with MSE) trains far more slowly because the gradients are poorly scaled.

• Regression: MSE by default; MAE or Huber when you need resistance to outliers.
• Binary or multi-label classification: per-class binary cross-entropy with sigmoid.
• Single-label multi-class: categorical cross-entropy with softmax.
• Ranking and retrieval: contrastive or triplet losses; detection often sums multiple losses.

Reading the loss curve: convergence, overfitting, and instability

The loss curve over training is the primary diagnostic for whether a model is learning. A training loss that falls smoothly and then plateaus signals healthy convergence. A loss that spikes, oscillates, or diverges usually points to a learning rate that is too high, exploding gradients, or numerical problems such as taking the log of zero in cross-entropy.

Tracking the validation loss alongside the training loss is how overfitting is caught. When training loss keeps dropping but validation loss starts rising, the model is memorizing the training data rather than generalizing. The gap between the two curves is the standard signal to stop early, add regularization, or gather more data.

A few practical guards keep the loss well-behaved. Cross-entropy implementations clamp probabilities away from exactly 0 and 1 to avoid infinite loss, and fuse softmax with the log to stay numerically stable. Gradient clipping caps the gradient norm to prevent a single large step from wrecking the weights. These are not optional polish; without them, a loss can silently become NaN and halt all learning.

• A smoothly decreasing then plateauing training loss indicates healthy convergence.
• Rising validation loss while training loss keeps falling is the classic overfitting signal.
• Spiking or NaN loss usually means too-high learning rate, exploding gradients, or log-of-zero; clip gradients and stabilize the log.

Loss function versus evaluation metric

A loss function and an evaluation metric serve different purposes and are often not the same thing. The loss is optimized during training and must be differentiable so backpropagation can use it. The metric is what you report and care about in the end, and it does not need to be differentiable at all. Accuracy, F1 score, BLEU, and AUC are metrics, not training losses.

Cross-entropy is the loss a classifier trains on; accuracy is the metric you read off afterward. They move together but are not identical, and a model can improve cross-entropy without improving accuracy, or vice versa, especially near a decision boundary. This is why both are tracked: the loss tells the optimizer where to go, the metric tells you whether the result is actually good.

When a metric is non-differentiable, training uses a differentiable surrogate loss that correlates with it. Reinforcement learning takes a different route, optimizing an expected reward rather than a fixed per-example loss, which is how models are tuned toward objectives that cannot be written as a simple labeled-example loss, including human preference in RLHF.

• Loss is the differentiable thing you optimize; the metric is the (often non-differentiable) thing you report.
• Cross-entropy is the loss; accuracy, F1, BLEU, and AUC are metrics.
• Non-differentiable goals use surrogate losses, or are optimized as expected reward as in reinforcement learning and RLHF.