Loss Function

Definition: A function that quantifies how wrong a model’s prediction is compared to the true value — the number that training tries to minimize.

How It Works

Regression tasks: mean squared error (MSE), mean absolute error (MAE)
Classification tasks: cross-entropy loss
The chosen loss shapes what “good” means for the model and directly drives gradient descent’s updates
Computed per example, then typically averaged (or summed) across a batch to produce a single scalar that Backpropagation can differentiate
Must be differentiable (or at least sub-differentiable) with respect to the model’s outputs, since Gradient Descent relies on that gradient to know which direction to update parameters
Distinct from an evaluation metric — the loss is what’s optimized during training, while a metric (accuracy, F1, AUC) is often what’s actually reported to stakeholders, and the two don’t have to move in perfect lockstep
Often composed of multiple terms in practice — a primary task loss plus a regularization penalty (see Regularization (L1, L2, Dropout)) added in to discourage overly complex solutions

Visualizing the Computation

Every loss function sits at the same junction in the training loop — predictions and true labels flow in, a task-specific formula decides how “wrong” they are, and a single scalar flows out for Backpropagation to differentiate. The branch taken depends entirely on the task:

Whichever branch is taken, the output shape is identical — one number — which is precisely what makes loss functions swappable: a training loop doesn’t need to know or care whether that scalar came from MSE or cross-entropy, only that it’s differentiable with respect to the model’s outputs.

Under the Hood

Mean Squared Error (MSE): L = (1/n) * Σ(y_pred - y_true)² — squares the error, so large errors are penalized disproportionately more than small ones, and the gradient is proportional to the error itself, giving smooth, well-behaved updates
Mean Absolute Error (MAE): L = (1/n) * Σ|y_pred - y_true| — penalizes all errors linearly, making it more robust to outliers than MSE, but its gradient has a constant magnitude regardless of error size, which can make fine convergence near zero error less smooth
Binary Cross-Entropy: L = -(1/n) * Σ[y*log(p) + (1-y)*log(1-p)] — measures the distance between two probability distributions (true label and predicted probability p), with loss approaching infinity as a confident wrong prediction gets more confident
Categorical Cross-Entropy: the multi-class generalization, L = -Σ y_i * log(p_i) summed over classes, typically paired with a softmax output layer so predictions form a valid probability distribution
The loss landscape’s shape (convex vs. non-convex, smooth vs. jagged) is determined jointly by the loss function and the model architecture — a convex loss on a linear model gives a bowl-shaped surface with one global minimum; the same loss on a deep neural network produces a highly non-convex surface with many local minima and saddle points

A loss function never runs in isolation — it’s one step in a repeating cycle with the model and the optimizer. Tracing a single training step end-to-end:

The loss function is the only component in this loop that knows what “correct” means for the task — the model just produces numbers, and the optimizer just follows gradients; swapping MSE for cross-entropy changes what the whole system is trying to achieve without touching a single line of the model or optimizer code.

Variants

Mean Squared Error (MSE) / L2 loss — standard for regression; sensitive to outliers because errors are squared
Mean Absolute Error (MAE) / L1 loss — regression loss more robust to outliers than MSE, at the cost of a non-smooth gradient at zero error
Huber Loss — quadratic for small errors, linear for large ones, combining MSE’s smooth gradient near zero with MAE’s robustness to outliers; controlled by a threshold parameter delta
Binary Cross-Entropy — standard for binary classification, paired with a sigmoid output
Categorical Cross-Entropy — standard for multi-class classification, paired with a softmax output
Hinge Loss — used by support vector machines (and some neural net classifiers); penalizes predictions that are correct but not confidently past a margin, not just wrong predictions: L = max(0, 1 - y*f(x))
KL Divergence — measures how one probability distribution diverges from a reference distribution; used in variational autoencoders, knowledge distillation, and anywhere a model’s output distribution needs to match a target distribution rather than a single label
Focal Loss — a modified cross-entropy that down-weights well-classified examples and focuses training on hard, misclassified ones; designed for extreme class imbalance (e.g., object detection where background pixels vastly outnumber object pixels)
Contrastive / Triplet Loss — used in embedding learning (face recognition, similarity search): pulls representations of similar examples together and pushes dissimilar ones apart, rather than predicting a label directly
Dice Loss / IoU Loss — common in image segmentation, where they directly optimize the overlap between predicted and true regions rather than treating each pixel as an independent classification, which handles severe foreground/background imbalance better than plain cross-entropy
Wasserstein Loss — used in Wasserstein GANs (Arjovsky, Chintala, and Bottou, 2017) in place of the original GAN’s cross-entropy-based discriminator loss; approximates the Earth Mover’s distance between real and generated data distributions, giving a smoother, more informative training signal that reduces mode collapse and the vanishing gradients the original GAN loss was prone to
CTC Loss (Connectionist Temporal Classification) — used for sequence tasks where input and output lengths differ with no pre-aligned mapping between them (e.g., speech-to-text, handwriting recognition); sums over all possible alignments between input frames and output labels so the model can train without frame-level labels

Why It Matters

Picking the wrong loss function optimizes the model for the wrong thing, even if training “works” smoothly
Different losses penalize errors differently (e.g., MSE punishes large errors much more than small ones)
The loss function encodes the actual objective — accuracy alone can’t be optimized directly since it’s not differentiable, so the loss is often a proxy for what you actually care about, and a mismatch between the two is a common source of models that train well but perform poorly on the real objective
Loss choice also affects training dynamics independent of the “correctness” of the objective — e.g., cross-entropy’s gradient stays large even when a prediction is very wrong, which speeds up learning compared to a loss whose gradient vanishes for confident wrong predictions
The differentiability requirement is why so much of ML engineering revolves around finding smooth proxies for objectives that are naturally discrete or rank-based (accuracy, top-k retrieval, BLEU score) — the proxy’s shape, not just its “correctness,” determines how well gradient descent can actually use it
Loss choice also shapes model calibration — how well predicted probabilities match real-world frequencies — which matters enormously in applications like medical risk scoring or weather forecasting, even when it has little effect on a task’s raw accuracy number

Common Pitfalls

Using accuracy as if it were a loss function — it’s not differentiable and can’t directly drive gradient descent
Using MSE for classification tasks instead of cross-entropy, leading to slow or unstable training
Reporting only the final loss value without the metric it’s meant to serve as a proxy for, making it hard for anyone downstream to judge whether the model is actually fit for purpose
Assuming a lower loss on one run versus another is a fair comparison when the two runs used different batch sizes, different data splits, or different loss reduction settings (mean vs. sum)
Forgetting that a framework’s cross-entropy loss expects log-probabilities or raw logits (depending on the API), not already-softmaxed probabilities — double-applying softmax silently degrades training
Using an unweighted loss on heavily imbalanced classification data, letting the model achieve low loss by just predicting the majority class
Not checking whether the loss on the validation set is actually decreasing in the metric that matters (e.g., low loss but poor precision/recall on a rare class) — see Precision, Recall, and F1 Score
Applying MAE where its non-smooth gradient near zero causes optimizer instability with certain optimizers/learning rates, when Huber loss would be a better middle ground
Comparing loss values across different loss functions as if they were on the same scale — an MSE of 0.5 and a cross-entropy of 0.5 mean completely different things and aren’t interchangeable as a measure of “how good” a model is
Leaving label smoothing or class weighting out of the loss when the labels are noisy or imbalanced, then being surprised the model is overconfident or biased toward the majority class
Choosing a loss based on what’s default in a tutorial or framework rather than what actually matches the label distribution and task structure
Ignoring numerical stability — computing cross-entropy as -log(softmax(x)) in two separate steps rather than using a fused, numerically stable implementation, which can produce NaN losses on confident, correct predictions due to floating-point underflow
Treating a loss function designed for one output range as if it were scale-invariant — e.g., comparing MSE across targets on wildly different scales (predicting prices in dollars vs. predicting prices in thousands of dollars) without normalizing, making the loss value meaningless across experiments

Best Practices

Match the loss to both the task type (regression vs. classification) and the output distribution (Gaussian errors → MSE, Bernoulli/categorical labels → cross-entropy)
Use class weighting or focal loss when classes are imbalanced, rather than leaving the raw loss to implicitly favor the majority class
Watch both the training loss and a task-relevant metric (accuracy, F1, RMSE in original units) — loss is what’s optimized, but it’s rarely the number stakeholders care about directly
Consider Huber loss over plain MSE when the regression target has outliers or noisy labels
When combining multiple objectives (e.g., a reconstruction loss plus a regularization term), weight them deliberately and track each component separately, not just the summed total
Apply label smoothing (replacing hard 0/1 targets with, e.g., 0.9/0.1) when training large classifiers, which tends to improve calibration and generalization at negligible cost
Sanity-check the loss value at initialization — for an untrained classifier with C classes and cross-entropy loss, the initial loss should be close to ln(C) (random guessing); a wildly different number usually signals a bug
Report both the loss and a task-relevant metric side by side in any experiment log, since a reviewer (or future you) will want to judge results by the metric, not the raw loss value
Be explicit about the loss reduction mode (mean vs. sum over a batch) when comparing runs or reproducing a paper’s reported numbers — the two produce different absolute values for the same underlying error
Normalize or standardize regression targets before applying MSE/MAE so the loss magnitude is comparable across features and experiments, and so a single learning rate performs sensibly across all output dimensions
Use a fused, numerically stable loss implementation (e.g., nn.CrossEntropyLoss operating on raw logits) rather than manually chaining a softmax and a log — the fused version avoids the underflow a naive two-step implementation is prone to
When designing a custom loss, unit-test it directly — feed in known predictions and labels and assert the returned value matches a hand-computed expectation, the same way you’d test any other pure function

Worked Example

Take four small predictions against a true value of 1, reinterpreted two ways — as regression outputs and as classifier confidences — to see side by side how differently each loss scores identical-looking errors:

Prediction	True value	Squared error (MSE term)	Absolute error (MAE term)	Cross-Entropy term
0.90	1.0	0.0100	0.100	0.105
0.60	1.0	0.1600	0.400	0.511
0.40	1.0	0.3600	0.600	0.916
0.05	1.0	0.9025	0.950	2.996

Averaging each column gives MSE ≈ 0.358, MAE ≈ 0.5125, and Cross-Entropy ≈ 1.132. Notice the pattern in the rightmost columns: going from a prediction of 0.40 to 0.05 increases the squared error by about 2.5x (0.36 → 0.90) but more than triples the cross-entropy term (0.916 → 2.996) — cross-entropy’s -log(p) grows unboundedly as p approaches zero, while MSE’s (p-1)² is capped at 1.0 no matter how wrong the prediction gets. That unbounded growth is exactly why cross-entropy keeps producing a strong corrective gradient on badly miscalibrated predictions long after MSE’s gradient has started to flatten out.

Code Example

import torch
import torch.nn as nn

# Regression
mse_loss = nn.MSELoss()
huber_loss = nn.SmoothL1Loss()  # PyTorch's Huber-style loss

# Classification
bce_loss = nn.BCEWithLogitsLoss()       # binary, expects raw logits
ce_loss = nn.CrossEntropyLoss()         # multi-class, expects raw logits + class indices

preds = model(x_batch)          # raw logits, shape [batch, num_classes]
loss = ce_loss(preds, y_batch)  # y_batch: integer class labels, shape [batch]
loss.backward()

Cross-entropy implemented from scratch for a single example, to make the formula concrete:

import numpy as np

def binary_cross_entropy(y_true, p_pred, eps=1e-12):
    p_pred = np.clip(p_pred, eps, 1 - eps)  # avoid log(0)
    return -(y_true * np.log(p_pred) + (1 - y_true) * np.log(1 - p_pred))

print(binary_cross_entropy(1, 0.05))  # ~3.00 -- confident wrong prediction, high loss
print(binary_cross_entropy(1, 0.90))  # ~0.105 -- confident correct prediction, low loss

The same contrast, made interactive — run this to compute both losses over a batch and watch the gap widen as predictions get more confidently wrong:

Comparison

Loss	Task	Sensitive to outliers	Gradient behavior	Differentiable everywhere?	Typical use
MSE (L2)	Regression	High	Smooth, proportional to error	Yes	Default regression loss
MAE (L1)	Regression	Low	Constant magnitude	No — kink at zero error	Outlier-heavy regression targets
Huber	Regression	Medium	Smooth near zero, linear far away	Yes	Robust regression
Binary Cross-Entropy	Binary classification	N/A	Large for confident wrong predictions	Yes (within open interval 0 to 1)	Default binary classifier loss
Categorical Cross-Entropy	Multi-class classification	N/A	Large for confident wrong predictions	Yes (within valid probability range)	Default multi-class loss
Hinge	Classification (margin-based)	N/A	Zero once past margin	No — kink at the margin	SVMs, margin classifiers
Focal Loss	Imbalanced classification	N/A	Down-weights easy examples	Yes	Object detection, rare-class problems
KL Divergence	Distribution matching	N/A	Large when distributions disagree sharply	Yes, where both distributions are nonzero	VAEs, knowledge distillation
Wasserstein	Generative modeling	N/A	Smooth even when distributions barely overlap	Yes, under the Kantorovich-Rubinstein form	GAN training (WGAN)
Contrastive / Triplet	Embedding / similarity learning	N/A	Zero once margin between pairs is satisfied	No — kink at the margin	Face recognition, similarity search
Dice / IoU	Image segmentation	N/A	Reflects region overlap, not per-pixel error	Yes, in the smoothed variant	Medical imaging, semantic segmentation
CTC	Sequence alignment	N/A	Summed over all valid alignments	Yes	Speech-to-text, handwriting recognition

History

Least-squares fitting — minimizing squared error, the same objective MSE generalizes — dates to Gauss and Legendre in the early 1800s, developed independently for astronomical orbit calculations. Cross-entropy’s statistical grounding comes from Claude Shannon’s 1948 information theory and Ronald Fisher’s earlier work on maximum likelihood estimation: minimizing cross-entropy loss is mathematically equivalent to maximizing the likelihood of the data under the model’s predicted distribution, which is why cross-entropy is the default for classification rather than an arbitrary design choice. Focal loss was introduced by Lin et al. in the 2017 RetinaNet paper specifically to address the extreme foreground/background imbalance in object detection, where standard cross-entropy let the overwhelming number of easy negative examples dominate the loss and swamp the learning signal from hard, informative ones.

Selected milestones not covered above:

1951 — Solomon Kullback and Richard Leibler publish “On Information and Sufficiency” (Annals of Mathematical Statistics), introducing the divergence measure behind KL Divergence loss decades before it was used to train neural networks
1964 — Peter Huber’s “Robust Estimation of a Location Parameter” (Annals of Mathematical Statistics) introduces what becomes Huber loss, originally as a robust statistical estimator rather than a neural network training tool
2016 — Christian Szegedy and coauthors introduce label smoothing in “Rethinking the Inception Architecture for Computer Vision” (CVPR 2016); the same year, Fausto Milletari and coauthors introduce Dice loss in the V-Net paper for volumetric medical image segmentation
2017 — Martin Arjovsky, Soumith Chintala, and Léon Bottou publish the Wasserstein GAN loss, replacing the original GAN’s cross-entropy-based discriminator objective with one based on the Earth Mover’s distance

Real-World Example

A fraud detection model trained on a dataset where fraudulent transactions make up 0.5% of all examples will, under plain cross-entropy loss, achieve very low loss by predicting “not fraud” almost every time — the loss contribution from the rare positive class is drowned out by the overwhelming volume of easy negatives. A practical fix combines class-weighted cross-entropy (multiplying the fraud class’s loss contribution by roughly the inverse of its frequency) with monitoring precision and recall on the fraud class specifically, since the loss curve alone would look deceptively good throughout training even as the model fails at the one thing it’s meant to catch.

Two more concrete, named cases of loss-function choice driving outcomes:

RetinaNet (Facebook AI Research, 2017) — before focal loss, one-stage object detectors trailed two-stage detectors in accuracy because standard cross-entropy let the overwhelming number of easy background regions dominate the gradient; switching to focal loss alone closed most of that accuracy gap while keeping the speed advantage of a one-stage architecture
Inception-v3 and label smoothing (Szegedy et al., 2016) — replacing hard 0/1 classification targets with softened targets during training measurably improved both top-1 accuracy and model calibration on ImageNet, and label smoothing has since become a near-default choice for training large classifiers

FAQ

Why can’t I just use accuracy as the loss? Accuracy is a step function of the model’s output (right or wrong), so its gradient is zero almost everywhere and undefined at the decision boundary — gradient descent has nothing to follow. Cross-entropy or hinge loss are smooth, differentiable proxies that push predictions in the right direction even when the final classification hasn’t flipped yet.

Is a lower loss always a better model? Not necessarily. A model can achieve a low training loss by overfitting (see Overfitting vs Underfitting), and even a low validation loss doesn’t guarantee good performance on the metric that actually matters for the application (e.g., recall on a rare but important class).

What’s the difference between a loss function and a metric? The loss function must be differentiable, since it drives gradient-based training. A metric (accuracy, F1, AUC) doesn’t need to be differentiable and is often what’s actually reported, but it’s frequently unsuitable as a direct optimization target.

Why is cross-entropy preferred over MSE for classification? MSE on softmax probabilities produces small, easily-saturated gradients when a prediction is very wrong (the sigmoid/softmax curve flattens out near 0 and 1), which slows learning exactly when the model most needs to correct itself. Cross-entropy’s gradient stays proportional to the error, so confidently wrong predictions produce strong corrective gradients throughout training, not just near the decision boundary.

Can a model have multiple loss functions at once? Yes — multi-task models commonly sum (or weight-sum) several loss terms, such as a classification loss plus a regularization penalty, or separate losses for multiple prediction heads. The combined scalar is what gradient descent actually minimizes, but tracking each term separately is essential for debugging which objective is driving (or fighting) the others.

Does the choice of loss function affect which optimizer or learning rate works best? Yes, indirectly — different losses produce gradients of different typical magnitude and shape, which interacts with the learning rate and with adaptive optimizers’ internal scaling. Switching loss functions without re-checking the learning rate is a common source of a training run that “used to work” suddenly diverging or stalling.

Is it possible to design a custom loss function for a specific problem? Yes, and it’s common in specialized domains — as long as the function is differentiable with respect to the model’s outputs, it can be plugged into the same Gradient Descent loop. Custom losses show up in ranking problems, generative models, and anywhere the standard MSE/cross-entropy pair doesn’t capture what “good” actually means for the task.

Why do some loss functions expect raw logits while others expect probabilities? Framework implementations of cross-entropy (e.g., PyTorch’s CrossEntropyLoss or BCEWithLogitsLoss) often fuse the softmax/sigmoid and the log into one numerically stable operation, which is why they expect raw, un-normalized logits rather than already-softmaxed probabilities. Passing already-normalized probabilities into a logits-expecting loss silently double-applies the activation function and quietly corrupts training — one of the more common framework-specific bugs in the Common Pitfalls above.

What does it mean for a loss function to be a “proper scoring rule”? A proper scoring rule is one where the expected loss is minimized precisely when the predicted probability matches the true underlying probability — in other words, the model has no incentive to report anything other than its honest best estimate. Cross-entropy is a proper scoring rule, which is part of why it produces well-calibrated probabilities in addition to correct classifications; not every loss function that “works” for classification has this property.

Common Interview Questions

Why is accuracy not a valid loss function? (It’s a non-differentiable step function — its gradient is zero almost everywhere, giving gradient descent nothing to follow.)
When would you choose Huber loss over MSE? (When the regression targets contain outliers or label noise you don’t want to dominate the gradient.)
What does it mean for cross-entropy loss to be connected to maximum likelihood estimation? (Minimizing cross-entropy is mathematically equivalent to maximizing the likelihood of the observed labels under the model’s predicted probability distribution.)
How would you handle a severely imbalanced classification problem at the loss-function level? (Class-weighted cross-entropy, focal loss, or a metric-aligned loss like Dice loss for segmentation — any approach that stops the majority class from dominating the gradient signal.)
What happens to training if the loss function isn’t differentiable everywhere? (Optimization stalls or becomes unreliable at the non-differentiable points; in practice, sub-gradients are used where a true gradient doesn’t exist, as with MAE’s kink at zero error.)
What’s the relationship between minimizing cross-entropy and maximizing likelihood, in one sentence? (They’re the same optimization — minimizing cross-entropy loss is mathematically identical to maximum likelihood estimation of the model’s parameters under the predicted distribution.)
If you added an L2 regularization term to a model’s loss, would you expect the raw loss value to go up or down compared to training without it? (Up, usually — the regularization term adds a penalty on top of the task loss, so the total reported loss is higher even when the model generalizes better; this is exactly why tracking task loss and regularization loss separately matters.)

Example

Cross-entropy loss heavily penalizes a model that predicts “5% cat” when the true label is “cat,” pushing it to be both correct and confident. Numerically: if the true label is 1 (cat) and the model predicts a probability of 0.05, the binary cross-entropy contribution is -log(0.05) ≈ 3.0 — a large loss value. If instead the model predicts 0.9, the loss drops to -log(0.9) ≈ 0.105. This steep penalty for confident-but-wrong predictions is exactly what drives the model to adjust its weights aggressively when it’s badly miscalibrated, and to make only small adjustments once its predictions are already close to correct — a behavior MSE wouldn’t produce nearly as strongly on probability outputs.

Referenced by

Dig deeper

Spot an error in this note?