Different ways to find a hyperparameters

Finding appropriate hyperparameters for deep neural networks (DNN) or gradient boosting models like XGBoost is a critical step in the process of building machine learning models. Hyperparameters are settings that can be tuned to control the behavior of a machine learning algorithm. The process of finding the optimal hyperparameters is known as hyperparameter tuning or hyperparameter optimization. Here are some strategies and approaches you can use:

1. Manual Tuning

This is the simplest approach, where the practitioner manually adjusts the hyperparameters based on intuition and experience, and observes the change in model performance. While it can be useful for learning and for small-scale problems, it's not efficient or practical for larger, more complex models or datasets.

2. Grid Search

Grid search is a traditional hyperparameter tuning method where you define a set of values for each hyperparameter you want to tune. It then trains a model for every combination of these hyperparameters and selects the best one. This method can be computationally expensive as the number of models to train grows exponentially with the number of hyperparameters and their potential values.

3. Random Search

Random search is another method where you define a range for each hyperparameter. The algorithm then randomly selects values within these ranges and trains a model. This process is repeated for a fixed number of iterations. This method is less exhaustive and can be faster than grid search, but there's no guarantee of finding the optimal solution.

4. Bayesian Optimization

Bayesian optimization is a probabilistic model-based approach for finding the minimum of any function that returns a real-value metric. It builds a probabilistic model of the objective function and uses it to select the most promising hyperparameters to evaluate in the true objective function. It's particularly suited for optimization over continuous domains of less than 20 dimensions and is often more efficient than grid search or random search.

5. Automated Hyperparameter Tuning

Automated Hyperparameter Tuning methods such as AutoML or services like Google Cloud's HyperTune, use machine learning to optimize model hyperparameters. These methods can be more efficient than manual tuning, grid search, or random search.