Learn the exact definition of a hyperparameter in machine learning. We explain why settings like the learning rate control your model’s training process and overall accuracy.
Question
Table of Contents
Which of the following is an example of a hyperparameter?
A. Model output labels
B. Feature values in the dataset
C. Predicted probability for a class
D. Learning rate in a gradient boosting model
Answer
D. Learning rate in a gradient boosting model
Explanation
To understand why this is the right choice, you have to look at how machine learning algorithms actually process information. When you build and train an AI model, you are managing two distinctly different types of variables: parameters and hyperparameters.
Here is exactly how they differ and why the learning rate falls into the latter category.
Parameters vs. Hyperparameters
The core difference lies in who—or what—sets the value.
Parameters are internal variables that the model learns completely on its own during the training phase. You do not set these manually. As the algorithm analyzes patterns in your dataset, it automatically adjusts these internal numbers. The weights and biases in a neural network are standard examples of parameters.
Hyperparameters are the external configuration settings that you, the engineer, must define before the training process ever begins. They act as the operational constraints for the algorithm, dictating exactly how the model should go about learning from the data.
Why the Learning Rate is a Hyperparameter
In algorithms like gradient boosting, the learning rate dictates the step size the model takes when updating its internal parameters to minimize errors.
Because you have to manually configure this value prior to training to guide the optimization process, it is a hyperparameter. Getting this setting right is a balancing act:
- Setting it too high: The model takes massive steps and can easily overshoot the optimal solution, leading to highly unstable training and poor accuracy.
- Setting it too low: The model takes tiny, overly cautious steps. The training process becomes incredibly slow, burns through computational resources, and might get stuck before ever finding the best solution.
Engineers spend a significant amount of time running structured experiments—such as Grid Search or Random Search—specifically to find the perfect learning rate for their specific dataset.