Training & Alignment

Hyperparameter Tuning

By Aditya Kumar Jha, Engineer

Hyperparameter tuning is the search for the configuration values set before training, such as learning rate, batch size, or regularization strength, that a model cannot learn on its own. The goal is to find settings that minimize validation error rather than fit the training data.

Hyperparameters vs Parameters

A machine learning model has two distinct kinds of numerical settings. Parameters are the values the training algorithm learns directly from data, such as the weights and biases of a neural network or the split thresholds inside a decision tree. Hyperparameters are the settings chosen before training begins, which control how learning happens but are not themselves adjusted by gradient descent or any other fitting procedure. Because the optimizer never touches them, they must be selected by a separate, outer search.

Common hyperparameters include the learning rate and batch size used by gradient descent, the number of layers and the width of each layer in a network, the strength of L1 or L2 regularization, and the dropout rate. Tree-based models add the number of trees and the maximum depth. For large language models adapted with low-rank techniques, the LoRA rank and the scaling factor are hyperparameters, and at inference the decoding temperature that controls output randomness is one as well.

The boundary matters because the two kinds of value are optimized by different mechanisms. Parameters are fit by minimizing a loss function on the training set. Hyperparameters are chosen by measuring performance on held-out data, since a setting that lowers training loss can still raise the error on unseen examples. Treating a hyperparameter as if the model could learn it, or tuning it on data later used to report results, produces misleading numbers.

  • Parameters are learned from data by the training algorithm; hyperparameters are fixed before training and control how learning proceeds.
  • Optimization-related hyperparameters include learning rate, batch size, and number of training epochs.
  • Architecture and capacity hyperparameters include number of layers, layer width, regularization strength, dropout rate, and for trees the count and maximum depth.
  • For LLM adaptation, LoRA rank and the decoding temperature used at inference are hyperparameters, not learned weights.

Grid search defines a discrete set of candidate values for each hyperparameter and evaluates every combination of them. It is exhaustive and reproducible, but its cost grows multiplicatively with the number of hyperparameters, a pattern often called the curse of dimensionality. With ten candidate values for each of four hyperparameters, a full grid contains ten thousand evaluations, which becomes infeasible when each evaluation means training a model.

Random search instead samples configurations independently from distributions placed over each hyperparameter. James Bergstra and Yoshua Bengio showed in their 2012 paper "Random Search for Hyper-Parameter Optimization," published in the Journal of Machine Learning Research, that random search finds models as good or better than grid search within a small fraction of the computation. The intuition is that for most problems only a few hyperparameters actually matter, and which ones matter varies by dataset.

When effective dimensionality is low, grid search wastes its budget by repeating the same values of the important hyperparameter while it sweeps the unimportant ones. Random search, by contrast, gives a distinct value to every hyperparameter on every trial, so a fixed number of trials covers far more distinct settings of whichever hyperparameter dominates. This is why practitioners reach for random search before grid search whenever the budget is limited and the important hyperparameters are not known in advance.

trials_grid = ∏ⱼ kⱼ vs. N trials sampled independently per hyperparameter
A grid over m hyperparameters with kⱼ values each requires the product of the kⱼ evaluations, growing multiplicatively. Random search runs a chosen number N of trials regardless of dimensionality, and each trial gives a unique value to every hyperparameter.
  • Grid search evaluates every combination on a predefined mesh; its cost grows multiplicatively with the number of hyperparameters.
  • Random search samples configurations from distributions and reaches comparable accuracy with far fewer trials.
  • Random search wins when only a few hyperparameters matter, because each trial tests a fresh value of the important one.
  • Both methods are embarrassingly parallel, since trials are independent and can run on separate machines.

Bayesian Optimization and Bandit Methods

Grid and random search are uninformed: each trial ignores what earlier trials revealed. Bayesian optimization closes that loop by building a probabilistic surrogate model of the function that maps hyperparameters to validation score, then using an acquisition function to choose the next configuration that best balances exploring uncertain regions against exploiting promising ones. Jasper Snoek, Hugo Larochelle, and Ryan P. Adams modeled this surrogate with a Gaussian process in their 2012 paper "Practical Bayesian Optimization of Machine Learning Algorithms." The Tree-structured Parzen Estimator is a popular alternative surrogate used by several modern tuning libraries.

Bandit methods attack a different inefficiency: the cost of training each candidate to completion. Successive halving, analyzed by Kevin Jamieson and Ameet Talwalkar in their 2016 paper "Non-stochastic Best Arm Identification and Hyperparameter Optimization," trains many configurations with a small budget, discards the worst-performing fraction, and reallocates the saved budget to the survivors, repeating until one remains. Hyperband, introduced by Lisha Li and colleagues and published in the Journal of Machine Learning Research in 2018, runs successive halving at several budget-versus-breadth tradeoffs to avoid committing to one early-stopping aggressiveness.

Population-based training, introduced by Max Jaderberg and colleagues at DeepMind in 2017, trains a population of models in parallel and periodically copies the weights of strong performers into weak ones while perturbing their hyperparameters. Unlike the other methods, it discovers a schedule of hyperparameter values that changes during training rather than a single fixed configuration, which suits settings such as reinforcement learning where the best learning rate shifts over time.

Each rung: keep top n/η configs, multiply per-config budget by η, repeat
Successive halving with a reduction factor η (often 3 or 4) starts with many configurations on a tiny budget. At each rung it keeps the best fraction 1/η and gives the survivors η times more budget, concentrating compute on configurations that already look strong.
  • Bayesian optimization fits a surrogate, often a Gaussian process or Tree-structured Parzen Estimator, and an acquisition function picks the next trial.
  • Successive halving allocates a small budget to many configurations, then keeps reallocating budget to the best survivors.
  • Hyperband wraps successive halving across several budget allocations to hedge how early it stops weak runs.
  • Population-based training evolves a population and produces a time-varying hyperparameter schedule rather than one fixed setting.

Evaluating Configurations Without Leaking

A hyperparameter setting is only as trustworthy as the data used to score it. The standard tool is k-fold cross-validation: the training data is split into k equal folds, the model is trained on k minus one folds and validated on the held-out fold, and this repeats k times so every example serves once as validation. Averaging the k validation scores gives a lower-variance estimate of how a configuration generalizes than a single split would provide.

The danger is data leakage. If the test set is consulted while choosing hyperparameters, the reported test score is optimistically biased, because the search has effectively been fitted to that data. The same problem appears more subtly when preprocessing steps such as feature scaling or feature selection are computed on the full dataset before splitting, leaking information from validation folds into training. Correct practice fits every preprocessing step inside the cross-validation loop and reserves a final test set that no tuning decision ever sees.

Even with clean splits, running many configurations introduces a multiple-comparisons effect: the best validation score across hundreds of trials is itself an optimistic estimate of true performance. This is why a held-out test set, untouched during the search, is needed to report an honest final number, and why nested cross-validation is used when both model selection and performance estimation must be unbiased on small datasets.

CV(λ) = (1/k) Σᵢ₌₁ᵏ L( f_{-i}(λ), Dᵢ )
The k-fold cross-validation estimate of a configuration λ averages the loss L on each held-out fold Dᵢ, where f_{-i}(λ) is the model trained on all folds except the i-th. Lower CV(λ) indicates a configuration expected to generalize better.
  • k-fold cross-validation rotates the validation fold k times and averages, lowering the variance of the score estimate.
  • Tuning on the test set, directly or through full-dataset preprocessing, biases the reported result and must be avoided.
  • The best score across many trials is optimistic; a final untouched test set gives an honest estimate.
  • Nested cross-validation separates the inner tuning loop from the outer evaluation loop on small datasets.

Tools and a Worked Example

Several open-source libraries implement these strategies. Scikit-learn provides GridSearchCV and RandomizedSearchCV, which wrap any estimator and run grid or random search with built-in cross-validation, exposing the chosen configuration through a best_params_ attribute. Optuna is a define-by-run framework whose default sampler for single-objective problems is the Tree-structured Parzen Estimator, and it supports pruning of unpromising trials. Ray Tune scales tuning across clusters and ships implementations of Hyperband, population-based training, and integrations with other optimizers. Weights and Biases sweeps coordinate grid, random, and Bayesian searches while logging every run.

The example below uses scikit-learn's RandomizedSearchCV to sample twenty configurations of a random forest with five-fold cross-validation. The param_distributions dictionary defines the search space, n_iter sets the number of sampled configurations, and cv sets the number of folds. After fitting, best_params_ holds the configuration with the highest mean cross-validation score, and best_score_ holds that score.

For larger search spaces or expensive models, an informed optimizer such as Optuna usually reaches a good configuration in fewer trials than random search, because its surrogate concentrates sampling where past trials performed well. The choice among tools depends on scale: scikit-learn suits single-machine experiments, while Ray Tune and Optuna's distributed backends suit many parallel workers and early stopping.

python
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import RandomizedSearchCV
from scipy.stats import randint

param_distributions = {
    "n_estimators": randint(100, 500),
    "max_depth": randint(3, 20),
    "min_samples_leaf": randint(1, 10),
}

search = RandomizedSearchCV(
    estimator=RandomForestClassifier(random_state=0),
    param_distributions=param_distributions,
    n_iter=20,        # sample 20 configurations
    cv=5,             # 5-fold cross-validation
    scoring="accuracy",
    random_state=0,
)

search.fit(X_train, y_train)
print(search.best_params_)
print(search.best_score_)  # mean CV accuracy of the best config
Random search over a random forest's hyperparameters with five-fold cross-validation in scikit-learn. The final test set is kept separate and is never passed to the search.
  • scikit-learn's GridSearchCV and RandomizedSearchCV wrap any estimator with built-in cross-validation and expose best_params_.
  • Optuna defaults to a Tree-structured Parzen Estimator sampler for single-objective search and can prune weak trials early.
  • Ray Tune provides distributed Hyperband and population-based training; Weights and Biases sweeps log and coordinate runs.
  • Informed optimizers reach good configurations in fewer trials, which matters most when each evaluation is expensive.

Key takeaways

  • Hyperparameters are set before training and control how learning happens; parameters are the weights the algorithm learns from data.
  • Grid search cost grows multiplicatively with dimensions, while random search reaches comparable accuracy in far fewer trials.
  • Random search beats grid search when only a few hyperparameters matter, a result from Bergstra and Bengio's 2012 JMLR paper.
  • Bayesian optimization and bandit methods such as Hyperband use past trials and partial training to spend the budget more efficiently.
  • Cross-validation gives a lower-variance score for each configuration, but the test set must never inform tuning decisions.
  • Optuna, Ray Tune, scikit-learn's GridSearchCV and RandomizedSearchCV, and Weights and Biases sweeps are common tuning tools.

Frequently asked questions

Parameters are the values a model learns from data during training, such as neural network weights or decision tree split thresholds. Hyperparameters are settings chosen before training that the algorithm does not adjust, such as the learning rate, batch size, or regularization strength. Because the optimizer never touches hyperparameters, they must be selected by a separate search that scores them on held-out data.
On most problems only a few hyperparameters strongly affect the outcome, and grid search wastes its budget by repeating identical values of the important one while sweeping the unimportant ones. Random search gives a fresh value to every hyperparameter on each trial, so a fixed number of trials explores many more distinct settings of whichever one matters. Bergstra and Bengio demonstrated this advantage empirically in their 2012 JMLR paper.
Split the data into training, validation, and test sets, and use only the training and validation data while searching for hyperparameters. Fit every preprocessing step inside the cross-validation loop so no information leaks from validation folds, and reserve the test set for a single final evaluation after tuning is complete. Reporting the best validation score across many trials as if it were test performance is a common source of optimistic bias.
Bayesian optimization builds a probabilistic surrogate model that predicts validation score from a hyperparameter configuration, then uses an acquisition function to pick the next configuration that balances exploring uncertain regions against exploiting promising ones. Common surrogates are Gaussian processes, used in Snoek, Larochelle, and Adams's 2012 paper, and the Tree-structured Parzen Estimator. It typically needs fewer trials than random search because it learns from past evaluations.
Successive halving trains many configurations on a small budget, discards the worst fraction, and reallocates budget to the survivors, repeating until one remains. Hyperband, published in JMLR in 2018 by Lisha Li and colleagues, runs successive halving at several budget-versus-breadth tradeoffs so it does not have to guess how aggressively to stop weak runs. Both methods speed up tuning by spending little compute on configurations that already look poor.
scikit-learn provides GridSearchCV and RandomizedSearchCV for single-machine searches with built-in cross-validation. Optuna offers define-by-run search with a Tree-structured Parzen Estimator default sampler and trial pruning, while Ray Tune scales Hyperband and population-based training across clusters. Weights and Biases sweeps coordinate grid, random, and Bayesian searches and log every run.

Put the idea into practice

MemX is an AI memory agent built on these ideas: store anything, skip the folders, and find it again by asking in plain English.

Try MemX Free