Chapter 80
6.8 XGBoost parameter tuning
6.8 XGBoost parameter tuning
Notes
XGBoost has various tunable parameters but the three most important ones are:
eta(default=0.3)- It is also called
learning_rateand is used to prevent overfitting by regularizing the weights of new features in each boosting step. range: [0, 1]
- It is also called
max_depth(default=6)- Maximum depth of a tree. Increasing this value will make the model more complex and more likely to overfit. range: [0, inf]
min_child_weight(default=1)- Minimum number of samples in leaf node. range: [0, inf]
For XGBoost models, there are other ways of finding the best parameters as well but the one we implement in the notebook follows the sequence of:
- First find the best value for
eta - Second, find the best value for
max_depth - Third, find the best value for
min_child_weight
Other useful parameter are:
subsample(default=1)- Subsample ratio of the training instances. Setting it to 0.5 means that model would randomly sample half of the training data prior to growing trees. range: (0, 1]
colsample_bytree(default=1)- This is similar to random forest, where each tree is made with the subset of randomly choosen features.
lambda(default=1)- Also called
reg_lambda. L2 regularization term on weights. Increasing this value will make model more conservative.
- Also called
alpha(default=0)- Also called
reg_alpha. L1 regularization term on weights. Increasing this value will make model more conservative.
- Also called
Alternative: Tuning XGBoost using a loop
Instead of repeating the training process manually for each eta value, you can use a loop to automate it (and other parameters to be tuned):
python
# Train XGBoost models for each eta and store AUC results
scores = {} # dictionary to store results for each eta
etas = [0.01, 0.05, 0.1, 0.3, 1.0] # list of parameter values. in this case it is 'eta'.
for eta in etas:
evals_result = {}
xgb_params = {
'eta': eta,
'max_depth': 6,
'min_child_weight': 1,
'objective': 'binary:logistic',
'eval_metric': 'auc',
'nthread': 8,
'seed': 1,
'verbosity': 1
}
model = xgb.train(
xgb_params,
dtrain,
evals=watchlist,
verbose_eval=0,
num_boost_round=200,
evals_result=evals_result
)
columns = ['iter', 'train_auc', 'val_auc']
train_aucs = list(evals_result['train'].values())[0]
val_aucs = list(evals_result['val'].values())[0]
df_results = pd.DataFrame(
list(zip(range(1, len(train_aucs) + 1), train_aucs, val_aucs)),
columns=columns
)
key = f'eta={eta}'
scores[key] = df_resultsAdd notes from the video (PRs are welcome)