Chapter 62

6. Decision Trees and Ensemble Learning

NotebookPython 3 (ipykernel)119 cells

6. Decision Trees and Ensemble Learning

This week, we'll talk about decision trees and tree-based ensemble algorithms

6.1 Credit risk scoring project

Dataset: https://github.com/gastonstat/CreditScoring

In [9]python · cell 3

python

import pandas as pd
import numpy as np

import seaborn as sns
from matplotlib import pyplot as plt
%matplotlib inline

6.2 Data cleaning and preparation

Downloading the dataset
Re-encoding the categorical variables
Doing the train/validation/test split

In [5]python · cell 6

python

data = 'https://raw.githubusercontent.com/alexeygrigorev/mlbookcamp-code/master/chapter-06-trees/CreditScoring.csv'

In [6]python · cell 7

python

!wget $data

Output

--2021-10-10 07:19:46--  https://raw.githubusercontent.com/alexeygrigorev/mlbookcamp-code/master/chapter-06-trees/CreditScoring.csv
Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.108.133, 185.199.109.133, 185.199.110.133, ...
Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.108.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 182489 (178K) [text/plain]
Saving to: ‘CreditScoring.csv’

CreditScoring.csv   100%[===================>] 178.21K   396KB/s    in 0.5s    

2021-10-10 07:19:47 (396 KB/s) - ‘CreditScoring.csv’ saved [182489/182489]

In [7]python · cell 8

python

!head CreditScoring.csv

Output

"Status","Seniority","Home","Time","Age","Marital","Records","Job","Expenses","Income","Assets","Debt","Amount","Price"
1,9,1,60,30,2,1,3,73,129,0,0,800,846
1,17,1,60,58,3,1,1,48,131,0,0,1000,1658
2,10,2,36,46,2,2,3,90,200,3000,0,2000,2985
1,0,1,60,24,1,1,1,63,182,2500,0,900,1325
1,0,1,36,26,1,1,1,46,107,0,0,310,910
1,1,2,60,36,2,1,1,75,214,3500,0,650,1645
1,29,2,60,44,2,1,1,75,125,10000,0,1600,1800
1,9,5,12,27,1,1,1,35,80,0,0,200,1093
1,0,2,60,32,2,1,3,90,107,15000,0,1200,1957

In [10]python · cell 9

python

df = pd.read_csv(data)

In [13]python · cell 10

python

df.columns = df.columns.str.lower()

In [16]python · cell 11

python

df.status.value_counts()

Output

1    3200
2    1254
0       1
Name: status, dtype: int64

In [18]python · cell 12

python

status_values = {
    1: 'ok',
    2: 'default',
    0: 'unk'
}

df.status = df.status.map(status_values)

In [20]python · cell 13

python

home_values = {
    1: 'rent',
    2: 'owner',
    3: 'private',
    4: 'ignore',
    5: 'parents',
    6: 'other',
    0: 'unk'
}

df.home = df.home.map(home_values)

marital_values = {
    1: 'single',
    2: 'married',
    3: 'widow',
    4: 'separated',
    5: 'divorced',
    0: 'unk'
}

df.marital = df.marital.map(marital_values)

records_values = {
    1: 'no',
    2: 'yes',
    0: 'unk'
}

df.records = df.records.map(records_values)

job_values = {
    1: 'fixed',
    2: 'partime',
    3: 'freelance',
    4: 'others',
    0: 'unk'
}

df.job = df.job.map(job_values)

In [21]python · cell 14

python

df.head()

Output

    status  seniority   home  time  age  marital records        job  expenses  \
0       ok          9   rent    60   30  married      no  freelance        73   
1       ok         17   rent    60   58    widow      no      fixed        48   
2  default         10  owner    36   46  married     yes  freelance        90   
3       ok          0   rent    60   24   single      no      fixed        63   
4       ok          0   rent    36   26   single      no      fixed        46   

   income  assets  debt  amount  price  
0     129       0     0     800    846  
1     131       0     0    1000   1658  
2     200    3000     0    2000   2985  
3     182    2500     0     900   1325  
4     107       0     0     310    910

.dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; }

	status	seniority	home	time	age	marital	records	job	expenses	income	assets	amount	price
0	ok	9	rent	60	30	married	no	freelance	73	129	0	800	846
1	ok	17	rent	60	58	widow	no	fixed	48	131	0	1000	1658
2	default	10	owner	36	46	married	yes	freelance	90	200	3000	2000	2985
3	ok	0	rent	60	24	single	no	fixed	63	182	2500	900	1325
4	ok	0	rent	36	26	single	no	fixed	46	107	0	310	910

In [23]python · cell 15

python

df.describe().round()

Output

       seniority    time     age  expenses      income      assets  \
count     4455.0  4455.0  4455.0    4455.0      4455.0      4455.0   
mean         8.0    46.0    37.0      56.0    763317.0   1060341.0   
std          8.0    15.0    11.0      20.0   8703625.0  10217569.0   
min          0.0     6.0    18.0      35.0         0.0         0.0   
25%          2.0    36.0    28.0      35.0        80.0         0.0   
50%          5.0    48.0    36.0      51.0       120.0      3500.0   
75%         12.0    60.0    45.0      72.0       166.0      6000.0   
max         48.0    72.0    68.0     180.0  99999999.0  99999999.0   

             debt  amount    price  
count      4455.0  4455.0   4455.0  
mean     404382.0  1039.0   1463.0  
std     6344253.0   475.0    628.0  
min           0.0   100.0    105.0  
25%           0.0   700.0   1118.0  
50%           0.0  1000.0   1400.0  
75%           0.0  1300.0   1692.0  
max    99999999.0  5000.0  11140.0

.dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; }

	seniority	time	age	expenses	income	assets	debt	amount	price
count	4455.0	4455.0	4455.0	4455.0	4455.0	4455.0	4455.0	4455.0	4455.0
mean	8.0	46.0	37.0	56.0	763317.0	1060341.0	404382.0	1039.0	1463.0
std	8.0	15.0	11.0	20.0	8703625.0	10217569.0	6344253.0	475.0	628.0
min	0.0	6.0	18.0	35.0	0.0	0.0	0.0	100.0	105.0
25%	2.0	36.0	28.0	35.0	80.0	0.0	0.0	700.0	1118.0
50%	5.0	48.0	36.0	51.0	120.0	3500.0	0.0	1000.0	1400.0
75%	12.0	60.0	45.0	72.0	166.0	6000.0	0.0	1300.0	1692.0
max	48.0	72.0	68.0	180.0	99999999.0	99999999.0	99999999.0	5000.0	11140.0

In [28]python · cell 16

python

for c in ['income', 'assets', 'debt']:
    df[c] = df[c].replace(to_replace=99999999, value=np.nan)

In [29]python · cell 17

python

df.describe().round()

Output

       seniority    time     age  expenses  income    assets     debt  amount  \
count     4455.0  4455.0  4455.0    4455.0  4421.0    4408.0   4437.0  4455.0   
mean         8.0    46.0    37.0      56.0   131.0    5403.0    343.0  1039.0   
std          8.0    15.0    11.0      20.0    86.0   11573.0   1246.0   475.0   
min          0.0     6.0    18.0      35.0     0.0       0.0      0.0   100.0   
25%          2.0    36.0    28.0      35.0    80.0       0.0      0.0   700.0   
50%          5.0    48.0    36.0      51.0   120.0    3000.0      0.0  1000.0   
75%         12.0    60.0    45.0      72.0   165.0    6000.0      0.0  1300.0   
max         48.0    72.0    68.0     180.0   959.0  300000.0  30000.0  5000.0   

         price  
count   4455.0  
mean    1463.0  
std      628.0  
min      105.0  
25%     1118.0  
50%     1400.0  
75%     1692.0  
max    11140.0

.dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; }

	seniority	time	age	expenses	income	assets	debt	amount	price
count	4455.0	4455.0	4455.0	4455.0	4421.0	4408.0	4437.0	4455.0	4455.0
mean	8.0	46.0	37.0	56.0	131.0	5403.0	343.0	1039.0	1463.0
std	8.0	15.0	11.0	20.0	86.0	11573.0	1246.0	475.0	628.0
min	0.0	6.0	18.0	35.0	0.0	0.0	0.0	100.0	105.0
25%	2.0	36.0	28.0	35.0	80.0	0.0	0.0	700.0	1118.0
50%	5.0	48.0	36.0	51.0	120.0	3000.0	0.0	1000.0	1400.0
75%	12.0	60.0	45.0	72.0	165.0	6000.0	0.0	1300.0	1692.0
max	48.0	72.0	68.0	180.0	959.0	300000.0	30000.0	5000.0	11140.0

In [33]python · cell 18

python

df = df[df.status != 'unk'].reset_index(drop=True)

In [34]python · cell 19

python

from sklearn.model_selection import train_test_split

df_full_train, df_test = train_test_split(df, test_size=0.2, random_state=11)
df_train, df_val = train_test_split(df_full_train, test_size=0.25, random_state=11)

In [36]python · cell 20

python

df_train = df_train.reset_index(drop=True)
df_val = df_val.reset_index(drop=True)
df_test = df_test.reset_index(drop=True)

In [41]python · cell 21

python

y_train = (df_train.status == 'default').astype('int').values
y_val = (df_val.status == 'default').astype('int').values
y_test = (df_test.status == 'default').astype('int').values

In [42]python · cell 22

python

del df_train['status']
del df_val['status']
del df_test['status']

In [43]python · cell 23

python

df_train

Output

      seniority     home  time  age  marital records        job  expenses  \
0            10    owner    36   36  married      no  freelance        75   
1             6  parents    48   32   single     yes      fixed        35   
2             1  parents    48   40  married      no      fixed        75   
3             1  parents    48   23   single      no    partime        35   
4             5    owner    36   46  married      no  freelance        60   
...         ...      ...   ...  ...      ...     ...        ...       ...   
2667         18  private    36   45  married      no      fixed        45   
2668          7  private    60   29  married      no      fixed        60   
2669          1  parents    24   19   single      no      fixed        35   
2670         15    owner    48   43  married      no  freelance        60   
2671         12    owner    48   27  married     yes      fixed        45   

      income   assets    debt  amount  price  
0        0.0  10000.0     0.0    1000   1400  
1       85.0      0.0     0.0    1100   1330  
2      121.0      0.0     0.0    1320   1600  
3       72.0      0.0     0.0    1078   1079  
4      100.0   4000.0     0.0    1100   1897  
...      ...      ...     ...     ...    ...  
2667   220.0  20000.0     0.0     800   1600  
2668    51.0   3500.0   500.0    1000   1290  
2669    28.0      0.0     0.0     400    600  
2670   100.0  18000.0     0.0    2500   2976  
2671   110.0   5000.0  1300.0     450   1636  

[2672 rows x 13 columns]

.dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; }

	seniority	home	time	age	marital	records	job	expenses	income	assets	debt	amount	price
0	10	owner	36	36	married	no	freelance	75	0.0	10000.0	0.0	1000	1400
1	6	parents	48	32	single	yes	fixed	35	85.0	0.0	0.0	1100	1330
2	1	parents	48	40	married	no	fixed	75	121.0	0.0	0.0	1320	1600
3	1	parents	48	23	single	no	partime	35	72.0	0.0	0.0	1078	1079
4	5	owner	36	46	married	no	freelance	60	100.0	4000.0	0.0	1100	1897
...	...	...	...	...	...	...	...	...	...	...	...	...	...
2667	18	private	36	45	married	no	fixed	45	220.0	20000.0	0.0	800	1600
2668	7	private	60	29	married	no	fixed	60	51.0	3500.0	500.0	1000	1290
2669	1	parents	24	19	single	no	fixed	35	28.0	0.0	0.0	400	600
2670	15	owner	48	43	married	no	freelance	60	100.0	18000.0	0.0	2500	2976
2671	12	owner	48	27	married	yes	fixed	45	110.0	5000.0	1300.0	450	1636

2672 rows × 13 columns

6.3 Decision trees

How a decision tree looks like
Training a decision tree
Overfitting
Controlling the size of a tree

In [44]python · cell 25

python

def assess_risk(client):
    if client['records'] == 'yes':
        if client['job'] == 'parttime':
            return 'default'
        else:
            return 'ok'
    else:
        if client['assets'] > 6000:
            return 'ok'
        else:
            return 'default'

In [47]python · cell 26

python

xi = df_train.iloc[0].to_dict()

In [48]python · cell 27

python

assess_risk(xi)

Output

'ok'

In [65]python · cell 28

python

from sklearn.tree import DecisionTreeClassifier
from sklearn.feature_extraction import DictVectorizer
from sklearn.metrics import roc_auc_score
from sklearn.tree import export_text

In [60]python · cell 29

python

train_dicts = df_train.fillna(0).to_dict(orient='records')

In [61]python · cell 30

python

dv = DictVectorizer(sparse=False)
X_train = dv.fit_transform(train_dicts)

In [62]python · cell 31

python

dt = DecisionTreeClassifier()
dt.fit(X_train, y_train)

Output

DecisionTreeClassifier()

In [63]python · cell 32

python

val_dicts = df_val.fillna(0).to_dict(orient='records')
X_val = dv.transform(val_dicts)

In [69]python · cell 33

python

y_pred = dt.predict_proba(X_val)[:, 1]
roc_auc_score(y_val, y_pred)

Output

0.6548400377806302

In [70]python · cell 34

python

y_pred = dt.predict_proba(X_train)[:, 1]
roc_auc_score(y_train, y_pred)

Output

1.0

In [81]python · cell 35

python

dt = DecisionTreeClassifier(max_depth=2)
dt.fit(X_train, y_train)

Output

DecisionTreeClassifier(max_depth=2)

In [82]python · cell 36

python

y_pred = dt.predict_proba(X_train)[:, 1]
auc = roc_auc_score(y_train, y_pred)
print('train:', auc)

y_pred = dt.predict_proba(X_val)[:, 1]
auc = roc_auc_score(y_val, y_pred)
print('val:', auc)

Output

train: 0.7054989859726213
val: 0.6685264343319367

In [83]python · cell 37

python

print(export_text(dt, feature_names=list(dv.get_feature_names_out())))

Output

|--- records=no <= 0.50
|   |--- seniority <= 6.50
|   |   |--- class: 1
|   |--- seniority >  6.50
|   |   |--- class: 0
|--- records=no >  0.50
|   |--- job=partime <= 0.50
|   |   |--- class: 0
|   |--- job=partime >  0.50
|   |   |--- class: 1

6.4 Decision tree learning algorithm

Finding the best split for one column
Finding the best split for the entire dataset
Stopping criteria
Decision tree learning algorithm

In [85]python · cell 39

python

data = [
    [8000, 'default'],
    [2000, 'default'],
    [   0, 'default'],
    [5000, 'ok'],
    [5000, 'ok'],
    [4000, 'ok'],
    [9000, 'ok'],
    [3000, 'default'],
]

df_example = pd.DataFrame(data, columns=['assets', 'status'])
df_example

Output

   assets   status
0    8000  default
1    2000  default
2       0  default
3    5000       ok
4    5000       ok
5    4000       ok
6    9000       ok
7    3000  default

.dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; }

	assets	status
0	8000	default
1	2000	default
2	0	default
3	5000	ok
4	5000	ok
5	4000	ok
6	9000	ok
7	3000	default

In [86]python · cell 40

python

df_example.sort_values('assets')

Output

   assets   status
2       0  default
1    2000  default
7    3000  default
5    4000       ok
3    5000       ok
4    5000       ok
0    8000  default
6    9000       ok

.dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; }

	assets	status
2	0	default
1	2000	default
7	3000	default
5	4000	ok
3	5000	ok
4	5000	ok
0	8000	default
6	9000	ok

In [91]python · cell 41

python

Ts = [0, 2000, 3000, 4000, 5000, 8000]

In [96]python · cell 42

python

T = 4000
df_left = df_example[df_example.assets <= T]
df_right = df_example[df_example.assets > T]

display(df_left)
print(df_left.status.value_counts(normalize=True))
display(df_right)
print(df_left.status.value_counts(normalize=True))

Output

   assets   status
1    2000  default
2       0  default
5    4000       ok
7    3000  default

.dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; }

	assets	status
1	2000	default
2	0	default
5	4000	ok
7	3000	default

default    0.75
ok         0.25
Name: status, dtype: float64

   assets   status
0    8000  default
3    5000       ok
4    5000       ok
6    9000       ok

.dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; }

	assets	status
0	8000	default
3	5000	ok
4	5000	ok
6	9000	ok

default    0.75
ok         0.25
Name: status, dtype: float64

In [92]python · cell 43

python

from IPython.display import display

In [98]python · cell 44

python

for T in Ts:
    print(T)
    df_left = df_example[df_example.assets <= T]
    df_right = df_example[df_example.assets > T]
    
    display(df_left)
    print(df_left.status.value_counts(normalize=True))
    display(df_right)
    print(df_right.status.value_counts(normalize=True))

    print()

Output

   assets   status
2       0  default

.dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; }

	assets	status
2	0	default

default    1.0
Name: status, dtype: float64

   assets   status
0    8000  default
1    2000  default
3    5000       ok
4    5000       ok
5    4000       ok
6    9000       ok
7    3000  default

.dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; }

	assets	status
0	8000	default
1	2000	default
3	5000	ok
4	5000	ok
5	4000	ok
6	9000	ok
7	3000	default

ok         0.571429
default    0.428571
Name: status, dtype: float64

2000

   assets   status
1    2000  default
2       0  default

.dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; }

	assets	status
1	2000	default
2	0	default

default    1.0
Name: status, dtype: float64

   assets   status
0    8000  default
3    5000       ok
4    5000       ok
5    4000       ok
6    9000       ok
7    3000  default

.dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; }

	assets	status
0	8000	default
3	5000	ok
4	5000	ok
5	4000	ok
6	9000	ok
7	3000	default

ok         0.666667
default    0.333333
Name: status, dtype: float64

3000

   assets   status
1    2000  default
2       0  default
7    3000  default

.dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; }

	assets	status
1	2000	default
2	0	default
7	3000	default

default    1.0
Name: status, dtype: float64

   assets   status
0    8000  default
3    5000       ok
4    5000       ok
5    4000       ok
6    9000       ok

.dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; }

	assets	status
0	8000	default
3	5000	ok
4	5000	ok
5	4000	ok
6	9000	ok

ok         0.8
default    0.2
Name: status, dtype: float64

4000

   assets   status
1    2000  default
2       0  default
5    4000       ok
7    3000  default

.dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; }

	assets	status
1	2000	default
2	0	default
5	4000	ok
7	3000	default

default    0.75
ok         0.25
Name: status, dtype: float64

   assets   status
0    8000  default
3    5000       ok
4    5000       ok
6    9000       ok

.dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; }

	assets	status
0	8000	default
3	5000	ok
4	5000	ok
6	9000	ok

ok         0.75
default    0.25
Name: status, dtype: float64

5000

   assets   status
1    2000  default
2       0  default
3    5000       ok
4    5000       ok
5    4000       ok
7    3000  default

.dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; }

	assets	status
1	2000	default
2	0	default
3	5000	ok
4	5000	ok
5	4000	ok
7	3000	default

default    0.5
ok         0.5
Name: status, dtype: float64

   assets   status
0    8000  default
6    9000       ok

.dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; }

	assets	status
0	8000	default
6	9000	ok

default    0.5
ok         0.5
Name: status, dtype: float64

8000

   assets   status
0    8000  default
1    2000  default
2       0  default
3    5000       ok
4    5000       ok
5    4000       ok
7    3000  default

.dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; }

	assets	status
0	8000	default
1	2000	default
2	0	default
3	5000	ok
4	5000	ok
5	4000	ok
7	3000	default

default    0.571429
ok         0.428571
Name: status, dtype: float64

   assets status
6    9000     ok

.dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; }

	assets	status
6	9000	ok

ok    1.0
Name: status, dtype: float64

In [99]python · cell 45

python

data = [
    [8000, 3000, 'default'],
    [2000, 1000, 'default'],
    [   0, 1000, 'default'],
    [5000, 1000, 'ok'],
    [5000, 1000, 'ok'],
    [4000, 1000, 'ok'],
    [9000,  500, 'ok'],
    [3000, 2000, 'default'],
]

df_example = pd.DataFrame(data, columns=['assets', 'debt', 'status'])
df_example

Output

   assets  debt   status
0    8000  3000  default
1    2000  1000  default
2       0  1000  default
3    5000  1000       ok
4    5000  1000       ok
5    4000  1000       ok
6    9000   500       ok
7    3000  2000  default

.dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; }

	assets	debt	status
0	8000	3000	default
1	2000	1000	default
2	0	1000	default
3	5000	1000	ok
4	5000	1000	ok
5	4000	1000	ok
6	9000	500	ok
7	3000	2000	default

In [100]python · cell 46

python

df_example.sort_values('debt')

Output

   assets  debt   status
6    9000   500       ok
1    2000  1000  default
2       0  1000  default
3    5000  1000       ok
4    5000  1000       ok
5    4000  1000       ok
7    3000  2000  default
0    8000  3000  default

.dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; }

	assets	debt	status
6	9000	500	ok
1	2000	1000	default
2	0	1000	default
3	5000	1000	ok
4	5000	1000	ok
5	4000	1000	ok
7	3000	2000	default
0	8000	3000	default

In [101]python · cell 47

python

thresholds = {
    'assets': [0, 2000, 3000, 4000, 5000, 8000],
    'debt': [500, 1000, 2000]
}

In [102]python · cell 48

python

for feature, Ts in thresholds.items():
    print('#####################')
    print(feature)
    for T in Ts:
        print(T)
        df_left = df_example[df_example[feature] <= T]
        df_right = df_example[df_example[feature] > T]

        display(df_left)
        print(df_left.status.value_counts(normalize=True))
        display(df_right)
        print(df_right.status.value_counts(normalize=True))

        print()
    print('#####################')

Output

#####################
assets
0

   assets  debt   status
2       0  1000  default

.dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; }

	assets	debt	status
2	0	1000	default

default    1.0
Name: status, dtype: float64

   assets  debt   status
0    8000  3000  default
1    2000  1000  default
3    5000  1000       ok
4    5000  1000       ok
5    4000  1000       ok
6    9000   500       ok
7    3000  2000  default

.dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; }

	assets	debt	status
0	8000	3000	default
1	2000	1000	default
3	5000	1000	ok
4	5000	1000	ok
5	4000	1000	ok
6	9000	500	ok
7	3000	2000	default

ok         0.571429
default    0.428571
Name: status, dtype: float64

2000

   assets  debt   status
1    2000  1000  default
2       0  1000  default

.dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; }

	assets	debt	status
1	2000	1000	default
2	0	1000	default

default    1.0
Name: status, dtype: float64

   assets  debt   status
0    8000  3000  default
3    5000  1000       ok
4    5000  1000       ok
5    4000  1000       ok
6    9000   500       ok
7    3000  2000  default

.dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; }

	assets	debt	status
0	8000	3000	default
3	5000	1000	ok
4	5000	1000	ok
5	4000	1000	ok
6	9000	500	ok
7	3000	2000	default

ok         0.666667
default    0.333333
Name: status, dtype: float64

3000

   assets  debt   status
1    2000  1000  default
2       0  1000  default
7    3000  2000  default

.dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; }

	assets	debt	status
1	2000	1000	default
2	0	1000	default
7	3000	2000	default

default    1.0
Name: status, dtype: float64

   assets  debt   status
0    8000  3000  default
3    5000  1000       ok
4    5000  1000       ok
5    4000  1000       ok
6    9000   500       ok

.dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; }

	assets	debt	status
0	8000	3000	default
3	5000	1000	ok
4	5000	1000	ok
5	4000	1000	ok
6	9000	500	ok

ok         0.8
default    0.2
Name: status, dtype: float64

4000

   assets  debt   status
1    2000  1000  default
2       0  1000  default
5    4000  1000       ok
7    3000  2000  default

.dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; }

	assets	debt	status
1	2000	1000	default
2	0	1000	default
5	4000	1000	ok
7	3000	2000	default

default    0.75
ok         0.25
Name: status, dtype: float64

   assets  debt   status
0    8000  3000  default
3    5000  1000       ok
4    5000  1000       ok
6    9000   500       ok

.dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; }

	assets	debt	status
0	8000	3000	default
3	5000	1000	ok
4	5000	1000	ok
6	9000	500	ok

ok         0.75
default    0.25
Name: status, dtype: float64

5000

   assets  debt   status
1    2000  1000  default
2       0  1000  default
3    5000  1000       ok
4    5000  1000       ok
5    4000  1000       ok
7    3000  2000  default

.dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; }

	assets	debt	status
1	2000	1000	default
2	0	1000	default
3	5000	1000	ok
4	5000	1000	ok
5	4000	1000	ok
7	3000	2000	default

default    0.5
ok         0.5
Name: status, dtype: float64

   assets  debt   status
0    8000  3000  default
6    9000   500       ok

.dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; }

	assets	debt	status
0	8000	3000	default
6	9000	500	ok

default    0.5
ok         0.5
Name: status, dtype: float64

8000

   assets  debt   status
0    8000  3000  default
1    2000  1000  default
2       0  1000  default
3    5000  1000       ok
4    5000  1000       ok
5    4000  1000       ok
7    3000  2000  default

.dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; }

	assets	debt	status
0	8000	3000	default
1	2000	1000	default
2	0	1000	default
3	5000	1000	ok
4	5000	1000	ok
5	4000	1000	ok
7	3000	2000	default

default    0.571429
ok         0.428571
Name: status, dtype: float64

   assets  debt status
6    9000   500     ok

.dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; }

	assets	debt	status
6	9000	500	ok

ok    1.0
Name: status, dtype: float64

#####################
#####################
debt
500

   assets  debt status
6    9000   500     ok

.dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; }

	assets	debt	status
6	9000	500	ok

ok    1.0
Name: status, dtype: float64

   assets  debt   status
0    8000  3000  default
1    2000  1000  default
2       0  1000  default
3    5000  1000       ok
4    5000  1000       ok
5    4000  1000       ok
7    3000  2000  default

.dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; }

	assets	debt	status
0	8000	3000	default
1	2000	1000	default
2	0	1000	default
3	5000	1000	ok
4	5000	1000	ok
5	4000	1000	ok
7	3000	2000	default

default    0.571429
ok         0.428571
Name: status, dtype: float64

1000

   assets  debt   status
1    2000  1000  default
2       0  1000  default
3    5000  1000       ok
4    5000  1000       ok
5    4000  1000       ok
6    9000   500       ok

.dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; }

	assets	debt	status
1	2000	1000	default
2	0	1000	default
3	5000	1000	ok
4	5000	1000	ok
5	4000	1000	ok
6	9000	500	ok

ok         0.666667
default    0.333333
Name: status, dtype: float64

   assets  debt   status
0    8000  3000  default
7    3000  2000  default

.dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; }

	assets	debt	status
0	8000	3000	default
7	3000	2000	default

default    1.0
Name: status, dtype: float64

2000

   assets  debt   status
1    2000  1000  default
2       0  1000  default
3    5000  1000       ok
4    5000  1000       ok
5    4000  1000       ok
6    9000   500       ok
7    3000  2000  default

.dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; }

	assets	debt	status
1	2000	1000	default
2	0	1000	default
3	5000	1000	ok
4	5000	1000	ok
5	4000	1000	ok
6	9000	500	ok
7	3000	2000	default

ok         0.571429
default    0.428571
Name: status, dtype: float64

   assets  debt   status
0    8000  3000  default

.dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; }

	assets	debt	status
0	8000	3000	default

default    1.0
Name: status, dtype: float64

#####################

6.5 Decision trees parameter tuning

selecting max_depth
selecting min_samples_leaf

In [27]python · cell 50

python

depths = [1, 2, 3, 4, 5, 6, 10, 15, 20, None]

for depth in depths: 
    dt = DecisionTreeClassifier(max_depth=depth)
    dt.fit(X_train, y_train)
    
    y_pred = dt.predict_proba(X_val)[:, 1]
    auc = roc_auc_score(y_val, y_pred)
    
    print('%4s -> %.3f' % (depth, auc))

Output

   1 -> 0.606
   2 -> 0.669
   3 -> 0.739
   4 -> 0.761
   5 -> 0.767
   6 -> 0.744
  10 -> 0.683
  15 -> 0.654
  20 -> 0.654
None -> 0.662

In [28]python · cell 51

python

scores = []

for depth in [4, 5, 6]:
    for s in [1, 5, 10, 15, 20, 500, 100, 200]:
        dt = DecisionTreeClassifier(max_depth=depth, min_samples_leaf=s)
        dt.fit(X_train, y_train)

        y_pred = dt.predict_proba(X_val)[:, 1]
        auc = roc_auc_score(y_val, y_pred)
        
        scores.append((depth, s, auc))

In [29]python · cell 52

python

columns = ['max_depth', 'min_samples_leaf', 'auc']
df_scores = pd.DataFrame(scores, columns=columns)

In [30]python · cell 53

python

df_scores_pivot = df_scores.pivot(index='min_samples_leaf', columns=['max_depth'], values=['auc'])
df_scores_pivot.round(3)

Output

                    auc              
max_depth             4      5      6
min_samples_leaf                     
1                 0.761  0.767  0.759
5                 0.761  0.768  0.759
10                0.761  0.762  0.778
15                0.764  0.772  0.785
20                0.761  0.774  0.774
100               0.756  0.763  0.776
200               0.747  0.759  0.768
500               0.680  0.680  0.680

.dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead tr th { text-align: left; } .dataframe thead tr:last-of-type th { text-align: right; }

	auc
max_depth	4	5	6
min_samples_leaf
1	0.761	0.767	0.759
5	0.761	0.768	0.759
10	0.761	0.762	0.778
15	0.764	0.772	0.785
20	0.761	0.774	0.774
100	0.756	0.763	0.776
200	0.747	0.759	0.768
500	0.680	0.680	0.680

In [31]python · cell 54

python

sns.heatmap(df_scores_pivot, annot=True, fmt=".3f")

Output

<AxesSubplot:xlabel='None-max_depth', ylabel='min_samples_leaf'>

<Figure size 432x288 with 2 Axes>

In [32]python · cell 55

python

dt = DecisionTreeClassifier(max_depth=6, min_samples_leaf=15)
dt.fit(X_train, y_train)

Output

DecisionTreeClassifier(max_depth=6, min_samples_leaf=15)

In [33]python · cell 56

python

print(export_text(dt, feature_names=list(dv.get_feature_names_out())))

Output

|--- records=no <= 0.50
|   |--- seniority <= 6.50
|   |   |--- amount <= 862.50
|   |   |   |--- price <= 925.00
|   |   |   |   |--- amount <= 525.00
|   |   |   |   |   |--- class: 1
|   |   |   |   |--- amount >  525.00
|   |   |   |   |   |--- class: 1
|   |   |   |--- price >  925.00
|   |   |   |   |--- price <= 1382.00
|   |   |   |   |   |--- class: 0
|   |   |   |   |--- price >  1382.00
|   |   |   |   |   |--- class: 0
|   |   |--- amount >  862.50
|   |   |   |--- assets <= 8250.00
|   |   |   |   |--- job=fixed <= 0.50
|   |   |   |   |   |--- assets <= 3425.00
|   |   |   |   |   |   |--- class: 1
|   |   |   |   |   |--- assets >  3425.00
|   |   |   |   |   |   |--- class: 1
|   |   |   |   |--- job=fixed >  0.50
|   |   |   |   |   |--- age <= 31.50
|   |   |   |   |   |   |--- class: 1
|   |   |   |   |   |--- age >  31.50
|   |   |   |   |   |   |--- class: 1
|   |   |   |--- assets >  8250.00
|   |   |   |   |--- income <= 132.50
|   |   |   |   |   |--- class: 1
|   |   |   |   |--- income >  132.50
|   |   |   |   |   |--- class: 0
|   |--- seniority >  6.50
|   |   |--- income <= 103.50
|   |   |   |--- assets <= 4500.00
|   |   |   |   |--- seniority <= 12.50
|   |   |   |   |   |--- class: 1
|   |   |   |   |--- seniority >  12.50
|   |   |   |   |   |--- class: 1
|   |   |   |--- assets >  4500.00
|   |   |   |   |--- class: 0
|   |   |--- income >  103.50
|   |   |   |--- time <= 33.00
|   |   |   |   |--- class: 0
|   |   |   |--- time >  33.00
|   |   |   |   |--- seniority <= 11.50
|   |   |   |   |   |--- age <= 37.50
|   |   |   |   |   |   |--- class: 1
|   |   |   |   |   |--- age >  37.50
|   |   |   |   |   |   |--- class: 0
|   |   |   |   |--- seniority >  11.50
|   |   |   |   |   |--- assets <= 7250.00
|   |   |   |   |   |   |--- class: 0
|   |   |   |   |   |--- assets >  7250.00
|   |   |   |   |   |   |--- class: 0
|--- records=no >  0.50
|   |--- job=partime <= 0.50
|   |   |--- income <= 74.50
|   |   |   |--- assets <= 4250.00
|   |   |   |   |--- income <= 20.00
|   |   |   |   |   |--- seniority <= 1.50
|   |   |   |   |   |   |--- class: 1
|   |   |   |   |   |--- seniority >  1.50
|   |   |   |   |   |   |--- class: 1
|   |   |   |   |--- income >  20.00
|   |   |   |   |   |--- expenses <= 71.00
|   |   |   |   |   |   |--- class: 0
|   |   |   |   |   |--- expenses >  71.00
|   |   |   |   |   |   |--- class: 1
|   |   |   |--- assets >  4250.00
|   |   |   |   |--- debt <= 1600.00
|   |   |   |   |   |--- seniority <= 2.50
|   |   |   |   |   |   |--- class: 0
|   |   |   |   |   |--- seniority >  2.50
|   |   |   |   |   |   |--- class: 0
|   |   |   |   |--- debt >  1600.00
|   |   |   |   |   |--- class: 1
|   |   |--- income >  74.50
|   |   |   |--- seniority <= 5.50
|   |   |   |   |--- amount <= 1330.00
|   |   |   |   |   |--- assets <= 3326.00
|   |   |   |   |   |   |--- class: 0
|   |   |   |   |   |--- assets >  3326.00
|   |   |   |   |   |   |--- class: 0
|   |   |   |   |--- amount >  1330.00
|   |   |   |   |   |--- assets <= 3750.00
|   |   |   |   |   |   |--- class: 1
|   |   |   |   |   |--- assets >  3750.00
|   |   |   |   |   |   |--- class: 0
|   |   |   |--- seniority >  5.50
|   |   |   |   |--- income <= 114.50
|   |   |   |   |   |--- expenses <= 84.00
|   |   |   |   |   |   |--- class: 0
|   |   |   |   |   |--- expenses >  84.00
|   |   |   |   |   |   |--- class: 0
|   |   |   |   |--- income >  114.50
|   |   |   |   |   |--- amount <= 1188.50
|   |   |   |   |   |   |--- class: 0
|   |   |   |   |   |--- amount >  1188.50
|   |   |   |   |   |   |--- class: 0
|   |--- job=partime >  0.50
|   |   |--- assets <= 7250.00
|   |   |   |--- time <= 27.00
|   |   |   |   |--- price <= 909.50
|   |   |   |   |   |--- class: 1
|   |   |   |   |--- price >  909.50
|   |   |   |   |   |--- class: 0
|   |   |   |--- time >  27.00
|   |   |   |   |--- age <= 34.50
|   |   |   |   |   |--- price <= 1049.00
|   |   |   |   |   |   |--- class: 1
|   |   |   |   |   |--- price >  1049.00
|   |   |   |   |   |   |--- class: 1
|   |   |   |   |--- age >  34.50
|   |   |   |   |   |--- assets <= 2750.00
|   |   |   |   |   |   |--- class: 1
|   |   |   |   |   |--- assets >  2750.00
|   |   |   |   |   |   |--- class: 1
|   |   |--- assets >  7250.00
|   |   |   |--- class: 0

6.6 Ensembles and random forest

Board of experts
Ensembling models
Random forest - ensembling decision trees
Tuning random forest

In [34]python · cell 58

python

from sklearn.ensemble import RandomForestClassifier

In [35]python · cell 59

python

scores = []

for n in range(10, 201, 10):
    rf = RandomForestClassifier(n_estimators=n, random_state=1)
    rf.fit(X_train, y_train)

    y_pred = rf.predict_proba(X_val)[:, 1]
    auc = roc_auc_score(y_val, y_pred)
    
    scores.append((n, auc))

In [36]python · cell 60

python

df_scores = pd.DataFrame(scores, columns=['n_estimators', 'auc'])

In [37]python · cell 61

python

plt.plot(df_scores.n_estimators, df_scores.auc)

Output

[<matplotlib.lines.Line2D at 0xff7f36d618e0>]

<Figure size 432x288 with 1 Axes>

In [41]python · cell 62

python

scores = []

for d in [5, 10, 15]:
    for n in range(10, 201, 10):
        rf = RandomForestClassifier(n_estimators=n,
                                    max_depth=d,
                                    random_state=1)
        rf.fit(X_train, y_train)

        y_pred = rf.predict_proba(X_val)[:, 1]
        auc = roc_auc_score(y_val, y_pred)

        scores.append((d, n, auc))

In [42]python · cell 63

python

columns = ['max_depth', 'n_estimators', 'auc']
df_scores = pd.DataFrame(scores, columns=columns)

In [43]python · cell 64

python

for d in [5, 10, 15]:
    df_subset = df_scores[df_scores.max_depth == d]
    
    plt.plot(df_subset.n_estimators, df_subset.auc,
             label='max_depth=%d' % d)

plt.legend()

Output

<matplotlib.legend.Legend at 0xff7f375fe490>

<Figure size 432x288 with 1 Axes>

In [44]python · cell 65

python

max_depth = 10

In [45]python · cell 66

python

scores = []

for s in [1, 3, 5, 10, 50]:
    for n in range(10, 201, 10):
        rf = RandomForestClassifier(n_estimators=n,
                                    max_depth=max_depth,
                                    min_samples_leaf=s,
                                    random_state=1)
        rf.fit(X_train, y_train)

        y_pred = rf.predict_proba(X_val)[:, 1]
        auc = roc_auc_score(y_val, y_pred)

        scores.append((s, n, auc))

In [47]python · cell 67

python

columns = ['min_samples_leaf', 'n_estimators', 'auc']
df_scores = pd.DataFrame(scores, columns=columns)

In [48]python · cell 68

python

colors = ['black', 'blue', 'orange', 'red', 'grey']
values = [1, 3, 5, 10, 50]

for s, col in zip(values, colors):
    df_subset = df_scores[df_scores.min_samples_leaf == s]
    
    plt.plot(df_subset.n_estimators, df_subset.auc,
             color=col,
             label='min_samples_leaf=%d' % s)

plt.legend()

Output

<matplotlib.legend.Legend at 0xff7f363cd310>

<Figure size 432x288 with 1 Axes>

In [49]python · cell 69

python

min_samples_leaf = 3

In [50]python · cell 70

python

rf = RandomForestClassifier(n_estimators=200,
                            max_depth=max_depth,
                            min_samples_leaf=min_samples_leaf,
                            random_state=1)
rf.fit(X_train, y_train)

Output

RandomForestClassifier(max_depth=10, min_samples_leaf=3, n_estimators=200,
                       random_state=1)

Other useful parametes:

max_features
bootstrap

https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html

6.7 Gradient boosting and XGBoost

Gradient boosting vs random forest
Installing XGBoost
Training the first model
Performance monitoring
Parsing xgboost's monitoring output

In [52]python · cell 73

python

!pip install xgboost

Output

Requirement already satisfied: xgboost in /home/alexey/.pyenv/versions/3.8.11/lib/python3.8/site-packages (1.4.2)
Requirement already satisfied: numpy in /home/alexey/.pyenv/versions/3.8.11/lib/python3.8/site-packages (from xgboost) (1.21.1)
Requirement already satisfied: scipy in /home/alexey/.pyenv/versions/3.8.11/lib/python3.8/site-packages (from xgboost) (1.7.0)

In [53]python · cell 74

python

import xgboost as xgb

In [56]python · cell 75

python

features = list(dv.get_feature_names_out())
dtrain = xgb.DMatrix(X_train, label=y_train, feature_names=features)
dval = xgb.DMatrix(X_val, label=y_val, feature_names=features)

In [65]python · cell 76

python

xgb_params = {
    'eta': 0.3, 
    'max_depth': 6,
    'min_child_weight': 1,
    
    'objective': 'binary:logistic',
    'nthread': 8,
    
    'seed': 1,
    'verbosity': 1,
}

model = xgb.train(xgb_params, dtrain, num_boost_round=10)

Output

[22:13:34] WARNING: ../src/learner.cc:1095: Starting in XGBoost 1.3.0, the default evaluation metric used with the objective 'binary:logistic' was changed from 'error' to 'logloss'. Explicitly set eval_metric if you'd like to restore the old behavior.

In [66]python · cell 77

python

y_pred = model.predict(dval)

In [67]python · cell 78

python

roc_auc_score(y_val, y_pred)

Output

0.8152745150274878

In [68]python · cell 79

python

watchlist = [(dtrain, 'train'), (dval, 'val')]

In [72]python · cell 80

python

%%capture output

xgb_params = {
    'eta': 0.3, 
    'max_depth': 6,
    'min_child_weight': 1,
    
    'objective': 'binary:logistic',
    'eval_metric': 'auc',

    'nthread': 8,
    'seed': 1,
    'verbosity': 1,
}

model = xgb.train(xgb_params, dtrain, num_boost_round=200,
                  verbose_eval=5,
                  evals=watchlist)

In [76]python · cell 81

python

s = output.stdout

In [97]python · cell 82

python

print(s[:200])

Output

[0]	train-auc:0.86300	val-auc:0.76818
[5]	train-auc:0.92863	val-auc:0.80606
[10]	train-auc:0.95002	val-auc:0.81558
[15]	train-auc:0.96558	val-auc:0.81680
[20]	train-auc:0.97316	val-auc:0.81775
[25]	tr

In [91]python · cell 83

python

def parse_xgb_output(output):
    results = []

    for line in output.stdout.strip().split('\n'):
        it_line, train_line, val_line = line.split('\t')

        it = int(it_line.strip('[]'))
        train = float(train_line.split(':')[1])
        val = float(val_line.split(':')[1])

        results.append((it, train, val))
    
    columns = ['num_iter', 'train_auc', 'val_auc']
    df_results = pd.DataFrame(results, columns=columns)
    return df_results

In [93]python · cell 84

python

df_score = parse_xgb_output(output)

In [98]python · cell 85

python

plt.plot(df_score.num_iter, df_score.train_auc, label='train')
plt.plot(df_score.num_iter, df_score.val_auc, label='val')
plt.legend()

Output

<matplotlib.legend.Legend at 0xff7f789d6c70>

<Figure size 432x288 with 1 Axes>

In [95]python · cell 86

python

plt.plot(df_score.num_iter, df_score.val_auc, label='val')
plt.legend()

Output

<matplotlib.legend.Legend at 0xff7f78c5fd90>

<Figure size 432x288 with 1 Axes>

6.8 XGBoost parameter tuning

Tuning the following parameters:

eta
max_depth
min_child_weight

In [100]python · cell 88

python

scores = {}

In [114]python · cell 89

python

%%capture output

xgb_params = {
    'eta': 0.01, 
    'max_depth': 6,
    'min_child_weight': 1,
    
    'objective': 'binary:logistic',
    'eval_metric': 'auc',

    'nthread': 8,
    'seed': 1,
    'verbosity': 1,
}

model = xgb.train(xgb_params, dtrain, num_boost_round=200,
                  verbose_eval=5,
                  evals=watchlist)

In [100]python · cell 90

python

scores = {}

In [115]python · cell 91

python

key = 'eta=%s' % (xgb_params['eta'])
scores[key] = parse_xgb_output(output)
key

Output

'eta=0.01'

In [121]python · cell 92

python

scores = {}

In [128]python · cell 93

python

%%capture output

xgb_params = {
    'eta': 0.1, 
    'max_depth': 10,
    'min_child_weight': 1,
    
    'objective': 'binary:logistic',
    'eval_metric': 'auc',

    'nthread': 8,
    'seed': 1,
    'verbosity': 1,
}

model = xgb.train(xgb_params, dtrain, num_boost_round=200,
                  verbose_eval=5,
                  evals=watchlist)

In [129]python · cell 94

python

key = 'max_depth=%s' % (xgb_params['max_depth'])
scores[key] = parse_xgb_output(output)
key

Output

'max_depth=10'

In [131]python · cell 95

python

del scores['max_depth=10']

In [133]python · cell 96

python

for max_depth, df_score in scores.items():
    plt.plot(df_score.num_iter, df_score.val_auc, label=max_depth)

plt.ylim(0.8, 0.84)
plt.legend()

Output

<matplotlib.legend.Legend at 0xff7f787909d0>

<Figure size 432x288 with 1 Axes>

In [121]python · cell 97

python

scores = {}

In [138]python · cell 98

python

%%capture output

xgb_params = {
    'eta': 0.1, 
    'max_depth': 3,
    'min_child_weight': 30,
    
    'objective': 'binary:logistic',
    'eval_metric': 'auc',

    'nthread': 8,
    'seed': 1,
    'verbosity': 1,
}

model = xgb.train(xgb_params, dtrain, num_boost_round=200,
                  verbose_eval=5,
                  evals=watchlist)

In [139]python · cell 99

python

key = 'min_child_weight=%s' % (xgb_params['min_child_weight'])
scores[key] = parse_xgb_output(output)
key

Output

'min_child_weight=30'

In [144]python · cell 100

python

for min_child_weight, df_score in scores.items():
    plt.plot(df_score.num_iter, df_score.val_auc, label=min_child_weight)

plt.ylim(0.82, 0.84)
plt.legend()

Output

<matplotlib.legend.Legend at 0xff7f784f6b50>

<Figure size 432x288 with 1 Axes>

In [145]python · cell 101

python

xgb_params = {
    'eta': 0.1, 
    'max_depth': 3,
    'min_child_weight': 1,

    'objective': 'binary:logistic',
    'eval_metric': 'auc',

    'nthread': 8,
    'seed': 1,
    'verbosity': 1,
}

model = xgb.train(xgb_params, dtrain, num_boost_round=175)

Other parameters: https://xgboost.readthedocs.io/en/latest/parameter.html

Useful ones:

subsample and colsample_bytree
lambda and alpha

6.9 Selecting the final model

Choosing between xgboost, random forest and decision tree
Training the final model
Saving the model

In [146]python · cell 104

python

dt = DecisionTreeClassifier(max_depth=6, min_samples_leaf=15)
dt.fit(X_train, y_train)

Output

DecisionTreeClassifier(max_depth=6, min_samples_leaf=15)

In [150]python · cell 105

python

y_pred = dt.predict_proba(X_val)[:, 1]
roc_auc_score(y_val, y_pred)

Output

0.7850802838390931

In [148]python · cell 106

python

rf = RandomForestClassifier(n_estimators=200,
                            max_depth=10,
                            min_samples_leaf=3,
                            random_state=1)
rf.fit(X_train, y_train)

Output

RandomForestClassifier(max_depth=10, min_samples_leaf=3, n_estimators=200,
                       random_state=1)

In [151]python · cell 107

python

y_pred = rf.predict_proba(X_val)[:, 1]
roc_auc_score(y_val, y_pred)

Output

0.8249709379767989

In [149]python · cell 108

python

xgb_params = {
    'eta': 0.1, 
    'max_depth': 3,
    'min_child_weight': 1,

    'objective': 'binary:logistic',
    'eval_metric': 'auc',

    'nthread': 8,
    'seed': 1,
    'verbosity': 1,
}

model = xgb.train(xgb_params, dtrain, num_boost_round=175)

In [152]python · cell 109

python

y_pred = model.predict(dval)
roc_auc_score(y_val, y_pred)

Output

0.8360387251459157

In [155]python · cell 110

python

df_full_train = df_full_train.reset_index(drop=True)

In [157]python · cell 111

python

y_full_train = (df_full_train.status == 'default').astype(int).values

In [159]python · cell 112

python

del df_full_train['status']

In [161]python · cell 113

python

dicts_full_train = df_full_train.to_dict(orient='records')

dv = DictVectorizer(sparse=False)
X_full_train = dv.fit_transform(dicts_full_train)

dicts_test = df_test.to_dict(orient='records')
X_test = dv.transform(dicts_test)

In [164]python · cell 114

python

dfulltrain = xgb.DMatrix(X_full_train, label=y_full_train,
                    feature_names=dv.get_feature_names_out())

dtest = xgb.DMatrix(X_test, feature_names=dv.get_feature_names_out())

In [165]python · cell 115

python

xgb_params = {
    'eta': 0.1, 
    'max_depth': 3,
    'min_child_weight': 1,

    'objective': 'binary:logistic',
    'eval_metric': 'auc',

    'nthread': 8,
    'seed': 1,
    'verbosity': 1,
}

model = xgb.train(xgb_params, dfulltrain, num_boost_round=175)

In [166]python · cell 116

python

y_pred = model.predict(dtest)

In [169]python · cell 117

python

roc_auc_score(y_test, y_pred)

Output

0.8322662626460096

6.10 Summary

Decision trees learn if-then-else rules from data.
Finding the best split: select the least impure split. This algorithm can overfit, that's why we control it by limiting the max depth and the size of the group.
Random forest is a way of combininig multiple decision trees. It should have a diverse set of models to make good predictions.
Gradient boosting trains model sequentially: each model tries to fix errors of the previous model. XGBoost is an implementation of gradient boosting.

6.11 Explore more

For this dataset we didn't do EDA or feature engineering. You can do it to get more insights into the problem.
For random forest, there are more parameters that we can tune. Check max_features and bootstrap.
There's a variation of random forest caled "extremely randomized trees", or "extra trees". Instead of selecting the best split among all possible thresholds, it selects a few thresholds randomly and picks the best one among them. Because of that extra trees never overfit. In Scikit-Learn, they are implemented in ExtraTreesClassifier. Try it for this project.
XGBoost can deal with NAs - we don't have to do fillna for it. Check if not filling NA's help improve performance.
Experiment with other XGBoost parameters: subsample and colsample_bytree.
When selecting the best split, decision trees find the most useful features. This information can be used for understanding which features are more important than otheres. See example here for random forest (it's the same for plain decision trees) and for xgboost
Trees can also be used for solving the regression problems: check DecisionTreeRegressor, RandomForestRegressor and the objective=reg:squarederror parameter for XGBoost.

Chapter 62

6. Decision Trees and Ensemble Learning

NotebookPython 3 (ipykernel)119 cells

6. Decision Trees and Ensemble Learning

This week, we'll talk about decision trees and tree-based ensemble algorithms

6.1 Credit risk scoring project

Dataset: https://github.com/gastonstat/CreditScoring

In [9]python · cell 3

python

import pandas as pd
import numpy as np

import seaborn as sns
from matplotlib import pyplot as plt
%matplotlib inline

6.2 Data cleaning and preparation

Downloading the dataset
Re-encoding the categorical variables
Doing the train/validation/test split

In [5]python · cell 6

python

data = 'https://raw.githubusercontent.com/alexeygrigorev/mlbookcamp-code/master/chapter-06-trees/CreditScoring.csv'

In [6]python · cell 7

python

!wget $data

Output

--2021-10-10 07:19:46--  https://raw.githubusercontent.com/alexeygrigorev/mlbookcamp-code/master/chapter-06-trees/CreditScoring.csv
Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.108.133, 185.199.109.133, 185.199.110.133, ...
Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.108.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 182489 (178K) [text/plain]
Saving to: ‘CreditScoring.csv’

CreditScoring.csv   100%[===================>] 178.21K   396KB/s    in 0.5s    

2021-10-10 07:19:47 (396 KB/s) - ‘CreditScoring.csv’ saved [182489/182489]

In [7]python · cell 8

python

!head CreditScoring.csv

Output

"Status","Seniority","Home","Time","Age","Marital","Records","Job","Expenses","Income","Assets","Debt","Amount","Price"
1,9,1,60,30,2,1,3,73,129,0,0,800,846
1,17,1,60,58,3,1,1,48,131,0,0,1000,1658
2,10,2,36,46,2,2,3,90,200,3000,0,2000,2985
1,0,1,60,24,1,1,1,63,182,2500,0,900,1325
1,0,1,36,26,1,1,1,46,107,0,0,310,910
1,1,2,60,36,2,1,1,75,214,3500,0,650,1645
1,29,2,60,44,2,1,1,75,125,10000,0,1600,1800
1,9,5,12,27,1,1,1,35,80,0,0,200,1093
1,0,2,60,32,2,1,3,90,107,15000,0,1200,1957

In [10]python · cell 9

python

df = pd.read_csv(data)

In [13]python · cell 10

python

df.columns = df.columns.str.lower()

In [16]python · cell 11

python

df.status.value_counts()

Output

1    3200
2    1254
0       1
Name: status, dtype: int64

In [18]python · cell 12

python

status_values = {
    1: 'ok',
    2: 'default',
    0: 'unk'
}

df.status = df.status.map(status_values)

In [20]python · cell 13

python

home_values = {
    1: 'rent',
    2: 'owner',
    3: 'private',
    4: 'ignore',
    5: 'parents',
    6: 'other',
    0: 'unk'
}

df.home = df.home.map(home_values)

marital_values = {
    1: 'single',
    2: 'married',
    3: 'widow',
    4: 'separated',
    5: 'divorced',
    0: 'unk'
}

df.marital = df.marital.map(marital_values)

records_values = {
    1: 'no',
    2: 'yes',
    0: 'unk'
}

df.records = df.records.map(records_values)

job_values = {
    1: 'fixed',
    2: 'partime',
    3: 'freelance',
    4: 'others',
    0: 'unk'
}

df.job = df.job.map(job_values)

In [21]python · cell 14

python

df.head()

Output

    status  seniority   home  time  age  marital records        job  expenses  \
0       ok          9   rent    60   30  married      no  freelance        73   
1       ok         17   rent    60   58    widow      no      fixed        48   
2  default         10  owner    36   46  married     yes  freelance        90   
3       ok          0   rent    60   24   single      no      fixed        63   
4       ok          0   rent    36   26   single      no      fixed        46   

   income  assets  debt  amount  price  
0     129       0     0     800    846  
1     131       0     0    1000   1658  
2     200    3000     0    2000   2985  
3     182    2500     0     900   1325  
4     107       0     0     310    910

.dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; }

	status	seniority	home	time	age	marital	records	job	expenses	income	assets	amount	price
0	ok	9	rent	60	30	married	no	freelance	73	129	0	800	846
1	ok	17	rent	60	58	widow	no	fixed	48	131	0	1000	1658
2	default	10	owner	36	46	married	yes	freelance	90	200	3000	2000	2985
3	ok	0	rent	60	24	single	no	fixed	63	182	2500	900	1325
4	ok	0	rent	36	26	single	no	fixed	46	107	0	310	910

In [23]python · cell 15

python

df.describe().round()

Output

       seniority    time     age  expenses      income      assets  \
count     4455.0  4455.0  4455.0    4455.0      4455.0      4455.0   
mean         8.0    46.0    37.0      56.0    763317.0   1060341.0   
std          8.0    15.0    11.0      20.0   8703625.0  10217569.0   
min          0.0     6.0    18.0      35.0         0.0         0.0   
25%          2.0    36.0    28.0      35.0        80.0         0.0   
50%          5.0    48.0    36.0      51.0       120.0      3500.0   
75%         12.0    60.0    45.0      72.0       166.0      6000.0   
max         48.0    72.0    68.0     180.0  99999999.0  99999999.0   

             debt  amount    price  
count      4455.0  4455.0   4455.0  
mean     404382.0  1039.0   1463.0  
std     6344253.0   475.0    628.0  
min           0.0   100.0    105.0  
25%           0.0   700.0   1118.0  
50%           0.0  1000.0   1400.0  
75%           0.0  1300.0   1692.0  
max    99999999.0  5000.0  11140.0

.dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; }

	seniority	time	age	expenses	income	assets	debt	amount	price
count	4455.0	4455.0	4455.0	4455.0	4455.0	4455.0	4455.0	4455.0	4455.0
mean	8.0	46.0	37.0	56.0	763317.0	1060341.0	404382.0	1039.0	1463.0
std	8.0	15.0	11.0	20.0	8703625.0	10217569.0	6344253.0	475.0	628.0
min	0.0	6.0	18.0	35.0	0.0	0.0	0.0	100.0	105.0
25%	2.0	36.0	28.0	35.0	80.0	0.0	0.0	700.0	1118.0
50%	5.0	48.0	36.0	51.0	120.0	3500.0	0.0	1000.0	1400.0
75%	12.0	60.0	45.0	72.0	166.0	6000.0	0.0	1300.0	1692.0
max	48.0	72.0	68.0	180.0	99999999.0	99999999.0	99999999.0	5000.0	11140.0

In [28]python · cell 16

python

for c in ['income', 'assets', 'debt']:
    df[c] = df[c].replace(to_replace=99999999, value=np.nan)

In [29]python · cell 17

python

df.describe().round()

Output

       seniority    time     age  expenses  income    assets     debt  amount  \
count     4455.0  4455.0  4455.0    4455.0  4421.0    4408.0   4437.0  4455.0   
mean         8.0    46.0    37.0      56.0   131.0    5403.0    343.0  1039.0   
std          8.0    15.0    11.0      20.0    86.0   11573.0   1246.0   475.0   
min          0.0     6.0    18.0      35.0     0.0       0.0      0.0   100.0   
25%          2.0    36.0    28.0      35.0    80.0       0.0      0.0   700.0   
50%          5.0    48.0    36.0      51.0   120.0    3000.0      0.0  1000.0   
75%         12.0    60.0    45.0      72.0   165.0    6000.0      0.0  1300.0   
max         48.0    72.0    68.0     180.0   959.0  300000.0  30000.0  5000.0   

         price  
count   4455.0  
mean    1463.0  
std      628.0  
min      105.0  
25%     1118.0  
50%     1400.0  
75%     1692.0  
max    11140.0

.dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; }

	seniority	time	age	expenses	income	assets	debt	amount	price
count	4455.0	4455.0	4455.0	4455.0	4421.0	4408.0	4437.0	4455.0	4455.0
mean	8.0	46.0	37.0	56.0	131.0	5403.0	343.0	1039.0	1463.0
std	8.0	15.0	11.0	20.0	86.0	11573.0	1246.0	475.0	628.0
min	0.0	6.0	18.0	35.0	0.0	0.0	0.0	100.0	105.0
25%	2.0	36.0	28.0	35.0	80.0	0.0	0.0	700.0	1118.0
50%	5.0	48.0	36.0	51.0	120.0	3000.0	0.0	1000.0	1400.0
75%	12.0	60.0	45.0	72.0	165.0	6000.0	0.0	1300.0	1692.0
max	48.0	72.0	68.0	180.0	959.0	300000.0	30000.0	5000.0	11140.0

In [33]python · cell 18

python

df = df[df.status != 'unk'].reset_index(drop=True)

In [34]python · cell 19

python

from sklearn.model_selection import train_test_split

df_full_train, df_test = train_test_split(df, test_size=0.2, random_state=11)
df_train, df_val = train_test_split(df_full_train, test_size=0.25, random_state=11)

In [36]python · cell 20

python

df_train = df_train.reset_index(drop=True)
df_val = df_val.reset_index(drop=True)
df_test = df_test.reset_index(drop=True)

In [41]python · cell 21

python

y_train = (df_train.status == 'default').astype('int').values
y_val = (df_val.status == 'default').astype('int').values
y_test = (df_test.status == 'default').astype('int').values

In [42]python · cell 22

python

del df_train['status']
del df_val['status']
del df_test['status']

In [43]python · cell 23

python

df_train

Output

      seniority     home  time  age  marital records        job  expenses  \
0            10    owner    36   36  married      no  freelance        75   
1             6  parents    48   32   single     yes      fixed        35   
2             1  parents    48   40  married      no      fixed        75   
3             1  parents    48   23   single      no    partime        35   
4             5    owner    36   46  married      no  freelance        60   
...         ...      ...   ...  ...      ...     ...        ...       ...   
2667         18  private    36   45  married      no      fixed        45   
2668          7  private    60   29  married      no      fixed        60   
2669          1  parents    24   19   single      no      fixed        35   
2670         15    owner    48   43  married      no  freelance        60   
2671         12    owner    48   27  married     yes      fixed        45   

      income   assets    debt  amount  price  
0        0.0  10000.0     0.0    1000   1400  
1       85.0      0.0     0.0    1100   1330  
2      121.0      0.0     0.0    1320   1600  
3       72.0      0.0     0.0    1078   1079  
4      100.0   4000.0     0.0    1100   1897  
...      ...      ...     ...     ...    ...  
2667   220.0  20000.0     0.0     800   1600  
2668    51.0   3500.0   500.0    1000   1290  
2669    28.0      0.0     0.0     400    600  
2670   100.0  18000.0     0.0    2500   2976  
2671   110.0   5000.0  1300.0     450   1636  

[2672 rows x 13 columns]

.dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; }

	seniority	home	time	age	marital	records	job	expenses	income	assets	debt	amount	price
0	10	owner	36	36	married	no	freelance	75	0.0	10000.0	0.0	1000	1400
1	6	parents	48	32	single	yes	fixed	35	85.0	0.0	0.0	1100	1330
2	1	parents	48	40	married	no	fixed	75	121.0	0.0	0.0	1320	1600
3	1	parents	48	23	single	no	partime	35	72.0	0.0	0.0	1078	1079
4	5	owner	36	46	married	no	freelance	60	100.0	4000.0	0.0	1100	1897
...	...	...	...	...	...	...	...	...	...	...	...	...	...
2667	18	private	36	45	married	no	fixed	45	220.0	20000.0	0.0	800	1600
2668	7	private	60	29	married	no	fixed	60	51.0	3500.0	500.0	1000	1290
2669	1	parents	24	19	single	no	fixed	35	28.0	0.0	0.0	400	600
2670	15	owner	48	43	married	no	freelance	60	100.0	18000.0	0.0	2500	2976
2671	12	owner	48	27	married	yes	fixed	45	110.0	5000.0	1300.0	450	1636

2672 rows × 13 columns

6.3 Decision trees

How a decision tree looks like
Training a decision tree
Overfitting
Controlling the size of a tree

In [44]python · cell 25

python

def assess_risk(client):
    if client['records'] == 'yes':
        if client['job'] == 'parttime':
            return 'default'
        else:
            return 'ok'
    else:
        if client['assets'] > 6000:
            return 'ok'
        else:
            return 'default'

In [47]python · cell 26

python

xi = df_train.iloc[0].to_dict()

In [48]python · cell 27

python

assess_risk(xi)

Output

'ok'

In [65]python · cell 28

python

from sklearn.tree import DecisionTreeClassifier
from sklearn.feature_extraction import DictVectorizer
from sklearn.metrics import roc_auc_score
from sklearn.tree import export_text

In [60]python · cell 29

python

train_dicts = df_train.fillna(0).to_dict(orient='records')

In [61]python · cell 30

python

dv = DictVectorizer(sparse=False)
X_train = dv.fit_transform(train_dicts)

In [62]python · cell 31

python

dt = DecisionTreeClassifier()
dt.fit(X_train, y_train)

Output

DecisionTreeClassifier()

In [63]python · cell 32

python

val_dicts = df_val.fillna(0).to_dict(orient='records')
X_val = dv.transform(val_dicts)

In [69]python · cell 33

python

y_pred = dt.predict_proba(X_val)[:, 1]
roc_auc_score(y_val, y_pred)

Output

0.6548400377806302

In [70]python · cell 34

python

y_pred = dt.predict_proba(X_train)[:, 1]
roc_auc_score(y_train, y_pred)

Output

1.0

In [81]python · cell 35

python

dt = DecisionTreeClassifier(max_depth=2)
dt.fit(X_train, y_train)

Output

DecisionTreeClassifier(max_depth=2)

In [82]python · cell 36

python

y_pred = dt.predict_proba(X_train)[:, 1]
auc = roc_auc_score(y_train, y_pred)
print('train:', auc)

y_pred = dt.predict_proba(X_val)[:, 1]
auc = roc_auc_score(y_val, y_pred)
print('val:', auc)

Output

train: 0.7054989859726213
val: 0.6685264343319367

In [83]python · cell 37

python

print(export_text(dt, feature_names=list(dv.get_feature_names_out())))

Output

|--- records=no <= 0.50
|   |--- seniority <= 6.50
|   |   |--- class: 1
|   |--- seniority >  6.50
|   |   |--- class: 0
|--- records=no >  0.50
|   |--- job=partime <= 0.50
|   |   |--- class: 0
|   |--- job=partime >  0.50
|   |   |--- class: 1

6.4 Decision tree learning algorithm

Finding the best split for one column
Finding the best split for the entire dataset
Stopping criteria
Decision tree learning algorithm

In [85]python · cell 39

python

data = [
    [8000, 'default'],
    [2000, 'default'],
    [   0, 'default'],
    [5000, 'ok'],
    [5000, 'ok'],
    [4000, 'ok'],
    [9000, 'ok'],
    [3000, 'default'],
]

df_example = pd.DataFrame(data, columns=['assets', 'status'])
df_example

Output

   assets   status
0    8000  default
1    2000  default
2       0  default
3    5000       ok
4    5000       ok
5    4000       ok
6    9000       ok
7    3000  default

.dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; }

	assets	status
0	8000	default
1	2000	default
2	0	default
3	5000	ok
4	5000	ok
5	4000	ok
6	9000	ok
7	3000	default

In [86]python · cell 40

python

df_example.sort_values('assets')

Output

   assets   status
2       0  default
1    2000  default
7    3000  default
5    4000       ok
3    5000       ok
4    5000       ok
0    8000  default
6    9000       ok

.dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; }

	assets	status
2	0	default
1	2000	default
7	3000	default
5	4000	ok
3	5000	ok
4	5000	ok
0	8000	default
6	9000	ok

In [91]python · cell 41

python

Ts = [0, 2000, 3000, 4000, 5000, 8000]

In [96]python · cell 42

python

T = 4000
df_left = df_example[df_example.assets <= T]
df_right = df_example[df_example.assets > T]

display(df_left)
print(df_left.status.value_counts(normalize=True))
display(df_right)
print(df_left.status.value_counts(normalize=True))

Output

   assets   status
1    2000  default
2       0  default
5    4000       ok
7    3000  default

.dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; }

	assets	status
1	2000	default
2	0	default
5	4000	ok
7	3000	default

default    0.75
ok         0.25
Name: status, dtype: float64

   assets   status
0    8000  default
3    5000       ok
4    5000       ok
6    9000       ok

.dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; }

	assets	status
0	8000	default
3	5000	ok
4	5000	ok
6	9000	ok

default    0.75
ok         0.25
Name: status, dtype: float64

In [92]python · cell 43

python

from IPython.display import display

In [98]python · cell 44

python

for T in Ts:
    print(T)
    df_left = df_example[df_example.assets <= T]
    df_right = df_example[df_example.assets > T]
    
    display(df_left)
    print(df_left.status.value_counts(normalize=True))
    display(df_right)
    print(df_right.status.value_counts(normalize=True))

    print()

Output

   assets   status
2       0  default

.dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; }

	assets	status
2	0	default

default    1.0
Name: status, dtype: float64

   assets   status
0    8000  default
1    2000  default
3    5000       ok
4    5000       ok
5    4000       ok
6    9000       ok
7    3000  default

.dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; }

	assets	status
0	8000	default
1	2000	default
3	5000	ok
4	5000	ok
5	4000	ok
6	9000	ok
7	3000	default

ok         0.571429
default    0.428571
Name: status, dtype: float64

2000

   assets   status
1    2000  default
2       0  default

.dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; }

	assets	status
1	2000	default
2	0	default

default    1.0
Name: status, dtype: float64

   assets   status
0    8000  default
3    5000       ok
4    5000       ok
5    4000       ok
6    9000       ok
7    3000  default

.dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; }

	assets	status
0	8000	default
3	5000	ok
4	5000	ok
5	4000	ok
6	9000	ok
7	3000	default

ok         0.666667
default    0.333333
Name: status, dtype: float64

3000

   assets   status
1    2000  default
2       0  default
7    3000  default

.dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; }

	assets	status
1	2000	default
2	0	default
7	3000	default

default    1.0
Name: status, dtype: float64

   assets   status
0    8000  default
3    5000       ok
4    5000       ok
5    4000       ok
6    9000       ok

.dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; }

	assets	status
0	8000	default
3	5000	ok
4	5000	ok
5	4000	ok
6	9000	ok

ok         0.8
default    0.2
Name: status, dtype: float64

4000

   assets   status
1    2000  default
2       0  default
5    4000       ok
7    3000  default

.dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; }

	assets	status
1	2000	default
2	0	default
5	4000	ok
7	3000	default

default    0.75
ok         0.25
Name: status, dtype: float64

   assets   status
0    8000  default
3    5000       ok
4    5000       ok
6    9000       ok

.dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; }

	assets	status
0	8000	default
3	5000	ok
4	5000	ok
6	9000	ok

ok         0.75
default    0.25
Name: status, dtype: float64

5000

   assets   status
1    2000  default
2       0  default
3    5000       ok
4    5000       ok
5    4000       ok
7    3000  default

.dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; }

	assets	status
1	2000	default
2	0	default
3	5000	ok
4	5000	ok
5	4000	ok
7	3000	default

default    0.5
ok         0.5
Name: status, dtype: float64

   assets   status
0    8000  default
6    9000       ok

.dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; }

	assets	status
0	8000	default
6	9000	ok

default    0.5
ok         0.5
Name: status, dtype: float64

8000

   assets   status
0    8000  default
1    2000  default
2       0  default
3    5000       ok
4    5000       ok
5    4000       ok
7    3000  default

.dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; }

	assets	status
0	8000	default
1	2000	default
2	0	default
3	5000	ok
4	5000	ok
5	4000	ok
7	3000	default

default    0.571429
ok         0.428571
Name: status, dtype: float64

   assets status
6    9000     ok

.dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; }

	assets	status
6	9000	ok

ok    1.0
Name: status, dtype: float64

In [99]python · cell 45

python

data = [
    [8000, 3000, 'default'],
    [2000, 1000, 'default'],
    [   0, 1000, 'default'],
    [5000, 1000, 'ok'],
    [5000, 1000, 'ok'],
    [4000, 1000, 'ok'],
    [9000,  500, 'ok'],
    [3000, 2000, 'default'],
]

df_example = pd.DataFrame(data, columns=['assets', 'debt', 'status'])
df_example

Output

   assets  debt   status
0    8000  3000  default
1    2000  1000  default
2       0  1000  default
3    5000  1000       ok
4    5000  1000       ok
5    4000  1000       ok
6    9000   500       ok
7    3000  2000  default

.dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; }

	assets	debt	status
0	8000	3000	default
1	2000	1000	default
2	0	1000	default
3	5000	1000	ok
4	5000	1000	ok
5	4000	1000	ok
6	9000	500	ok
7	3000	2000	default

In [100]python · cell 46

python

df_example.sort_values('debt')

Output

   assets  debt   status
6    9000   500       ok
1    2000  1000  default
2       0  1000  default
3    5000  1000       ok
4    5000  1000       ok
5    4000  1000       ok
7    3000  2000  default
0    8000  3000  default

.dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; }

	assets	debt	status
6	9000	500	ok
1	2000	1000	default
2	0	1000	default
3	5000	1000	ok
4	5000	1000	ok
5	4000	1000	ok
7	3000	2000	default
0	8000	3000	default

In [101]python · cell 47

python

thresholds = {
    'assets': [0, 2000, 3000, 4000, 5000, 8000],
    'debt': [500, 1000, 2000]
}

In [102]python · cell 48

python

for feature, Ts in thresholds.items():
    print('#####################')
    print(feature)
    for T in Ts:
        print(T)
        df_left = df_example[df_example[feature] <= T]
        df_right = df_example[df_example[feature] > T]

        display(df_left)
        print(df_left.status.value_counts(normalize=True))
        display(df_right)
        print(df_right.status.value_counts(normalize=True))

        print()
    print('#####################')

Output

#####################
assets
0

   assets  debt   status
2       0  1000  default

.dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; }

	assets	debt	status
2	0	1000	default

default    1.0
Name: status, dtype: float64

   assets  debt   status
0    8000  3000  default
1    2000  1000  default
3    5000  1000       ok
4    5000  1000       ok
5    4000  1000       ok
6    9000   500       ok
7    3000  2000  default

.dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; }

	assets	debt	status
0	8000	3000	default
1	2000	1000	default
3	5000	1000	ok
4	5000	1000	ok
5	4000	1000	ok
6	9000	500	ok
7	3000	2000	default

ok         0.571429
default    0.428571
Name: status, dtype: float64

2000

   assets  debt   status
1    2000  1000  default
2       0  1000  default

.dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; }

	assets	debt	status
1	2000	1000	default
2	0	1000	default

default    1.0
Name: status, dtype: float64

   assets  debt   status
0    8000  3000  default
3    5000  1000       ok
4    5000  1000       ok
5    4000  1000       ok
6    9000   500       ok
7    3000  2000  default

.dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; }

	assets	debt	status
0	8000	3000	default
3	5000	1000	ok
4	5000	1000	ok
5	4000	1000	ok
6	9000	500	ok
7	3000	2000	default

ok         0.666667
default    0.333333
Name: status, dtype: float64

3000

   assets  debt   status
1    2000  1000  default
2       0  1000  default
7    3000  2000  default

.dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; }

	assets	debt	status
1	2000	1000	default
2	0	1000	default
7	3000	2000	default

default    1.0
Name: status, dtype: float64

   assets  debt   status
0    8000  3000  default
3    5000  1000       ok
4    5000  1000       ok
5    4000  1000       ok
6    9000   500       ok

.dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; }

	assets	debt	status
0	8000	3000	default
3	5000	1000	ok
4	5000	1000	ok
5	4000	1000	ok
6	9000	500	ok

ok         0.8
default    0.2
Name: status, dtype: float64

4000

   assets  debt   status
1    2000  1000  default
2       0  1000  default
5    4000  1000       ok
7    3000  2000  default

.dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; }

	assets	debt	status
1	2000	1000	default
2	0	1000	default
5	4000	1000	ok
7	3000	2000	default

default    0.75
ok         0.25
Name: status, dtype: float64

   assets  debt   status
0    8000  3000  default
3    5000  1000       ok
4    5000  1000       ok
6    9000   500       ok

.dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; }

	assets	debt	status
0	8000	3000	default
3	5000	1000	ok
4	5000	1000	ok
6	9000	500	ok

ok         0.75
default    0.25
Name: status, dtype: float64

5000

   assets  debt   status
1    2000  1000  default
2       0  1000  default
3    5000  1000       ok
4    5000  1000       ok
5    4000  1000       ok
7    3000  2000  default

.dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; }

	assets	debt	status
1	2000	1000	default
2	0	1000	default
3	5000	1000	ok
4	5000	1000	ok
5	4000	1000	ok
7	3000	2000	default

default    0.5
ok         0.5
Name: status, dtype: float64

   assets  debt   status
0    8000  3000  default
6    9000   500       ok

.dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; }

	assets	debt	status
0	8000	3000	default
6	9000	500	ok

default    0.5
ok         0.5
Name: status, dtype: float64

8000

   assets  debt   status
0    8000  3000  default
1    2000  1000  default
2       0  1000  default
3    5000  1000       ok
4    5000  1000       ok
5    4000  1000       ok
7    3000  2000  default

.dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; }

	assets	debt	status
0	8000	3000	default
1	2000	1000	default
2	0	1000	default
3	5000	1000	ok
4	5000	1000	ok
5	4000	1000	ok
7	3000	2000	default

default    0.571429
ok         0.428571
Name: status, dtype: float64

   assets  debt status
6    9000   500     ok

.dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; }

	assets	debt	status
6	9000	500	ok

ok    1.0
Name: status, dtype: float64

#####################
#####################
debt
500

   assets  debt status
6    9000   500     ok

.dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; }

	assets	debt	status
6	9000	500	ok

ok    1.0
Name: status, dtype: float64

   assets  debt   status
0    8000  3000  default
1    2000  1000  default
2       0  1000  default
3    5000  1000       ok
4    5000  1000       ok
5    4000  1000       ok
7    3000  2000  default

.dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; }

	assets	debt	status
0	8000	3000	default
1	2000	1000	default
2	0	1000	default
3	5000	1000	ok
4	5000	1000	ok
5	4000	1000	ok
7	3000	2000	default

default    0.571429
ok         0.428571
Name: status, dtype: float64

1000

   assets  debt   status
1    2000  1000  default
2       0  1000  default
3    5000  1000       ok
4    5000  1000       ok
5    4000  1000       ok
6    9000   500       ok

.dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; }

	assets	debt	status
1	2000	1000	default
2	0	1000	default
3	5000	1000	ok
4	5000	1000	ok
5	4000	1000	ok
6	9000	500	ok

ok         0.666667
default    0.333333
Name: status, dtype: float64

   assets  debt   status
0    8000  3000  default
7    3000  2000  default

.dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; }

	assets	debt	status
0	8000	3000	default
7	3000	2000	default

default    1.0
Name: status, dtype: float64

2000

   assets  debt   status
1    2000  1000  default
2       0  1000  default
3    5000  1000       ok
4    5000  1000       ok
5    4000  1000       ok
6    9000   500       ok
7    3000  2000  default

.dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; }

	assets	debt	status
1	2000	1000	default
2	0	1000	default
3	5000	1000	ok
4	5000	1000	ok
5	4000	1000	ok
6	9000	500	ok
7	3000	2000	default

ok         0.571429
default    0.428571
Name: status, dtype: float64

   assets  debt   status
0    8000  3000  default

.dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; }

	assets	debt	status
0	8000	3000	default

default    1.0
Name: status, dtype: float64

#####################

6.5 Decision trees parameter tuning

selecting max_depth
selecting min_samples_leaf

In [27]python · cell 50

python

depths = [1, 2, 3, 4, 5, 6, 10, 15, 20, None]

for depth in depths: 
    dt = DecisionTreeClassifier(max_depth=depth)
    dt.fit(X_train, y_train)
    
    y_pred = dt.predict_proba(X_val)[:, 1]
    auc = roc_auc_score(y_val, y_pred)
    
    print('%4s -> %.3f' % (depth, auc))

Output

   1 -> 0.606
   2 -> 0.669
   3 -> 0.739
   4 -> 0.761
   5 -> 0.767
   6 -> 0.744
  10 -> 0.683
  15 -> 0.654
  20 -> 0.654
None -> 0.662

In [28]python · cell 51

python

scores = []

for depth in [4, 5, 6]:
    for s in [1, 5, 10, 15, 20, 500, 100, 200]:
        dt = DecisionTreeClassifier(max_depth=depth, min_samples_leaf=s)
        dt.fit(X_train, y_train)

        y_pred = dt.predict_proba(X_val)[:, 1]
        auc = roc_auc_score(y_val, y_pred)
        
        scores.append((depth, s, auc))

In [29]python · cell 52

python

columns = ['max_depth', 'min_samples_leaf', 'auc']
df_scores = pd.DataFrame(scores, columns=columns)

In [30]python · cell 53

python

df_scores_pivot = df_scores.pivot(index='min_samples_leaf', columns=['max_depth'], values=['auc'])
df_scores_pivot.round(3)

Output

                    auc              
max_depth             4      5      6
min_samples_leaf                     
1                 0.761  0.767  0.759
5                 0.761  0.768  0.759
10                0.761  0.762  0.778
15                0.764  0.772  0.785
20                0.761  0.774  0.774
100               0.756  0.763  0.776
200               0.747  0.759  0.768
500               0.680  0.680  0.680

	auc
max_depth	4	5	6
min_samples_leaf
1	0.761	0.767	0.759
5	0.761	0.768	0.759
10	0.761	0.762	0.778
15	0.764	0.772	0.785
20	0.761	0.774	0.774
100	0.756	0.763	0.776
200	0.747	0.759	0.768
500	0.680	0.680	0.680

In [31]python · cell 54

python

sns.heatmap(df_scores_pivot, annot=True, fmt=".3f")

Output

<AxesSubplot:xlabel='None-max_depth', ylabel='min_samples_leaf'>

<Figure size 432x288 with 2 Axes>

In [32]python · cell 55

python

dt = DecisionTreeClassifier(max_depth=6, min_samples_leaf=15)
dt.fit(X_train, y_train)

Output

DecisionTreeClassifier(max_depth=6, min_samples_leaf=15)

In [33]python · cell 56

python

print(export_text(dt, feature_names=list(dv.get_feature_names_out())))

Output

|--- records=no <= 0.50
|   |--- seniority <= 6.50
|   |   |--- amount <= 862.50
|   |   |   |--- price <= 925.00
|   |   |   |   |--- amount <= 525.00
|   |   |   |   |   |--- class: 1
|   |   |   |   |--- amount >  525.00
|   |   |   |   |   |--- class: 1
|   |   |   |--- price >  925.00
|   |   |   |   |--- price <= 1382.00
|   |   |   |   |   |--- class: 0
|   |   |   |   |--- price >  1382.00
|   |   |   |   |   |--- class: 0
|   |   |--- amount >  862.50
|   |   |   |--- assets <= 8250.00
|   |   |   |   |--- job=fixed <= 0.50
|   |   |   |   |   |--- assets <= 3425.00
|   |   |   |   |   |   |--- class: 1
|   |   |   |   |   |--- assets >  3425.00
|   |   |   |   |   |   |--- class: 1
|   |   |   |   |--- job=fixed >  0.50
|   |   |   |   |   |--- age <= 31.50
|   |   |   |   |   |   |--- class: 1
|   |   |   |   |   |--- age >  31.50
|   |   |   |   |   |   |--- class: 1
|   |   |   |--- assets >  8250.00
|   |   |   |   |--- income <= 132.50
|   |   |   |   |   |--- class: 1
|   |   |   |   |--- income >  132.50
|   |   |   |   |   |--- class: 0
|   |--- seniority >  6.50
|   |   |--- income <= 103.50
|   |   |   |--- assets <= 4500.00
|   |   |   |   |--- seniority <= 12.50
|   |   |   |   |   |--- class: 1
|   |   |   |   |--- seniority >  12.50
|   |   |   |   |   |--- class: 1
|   |   |   |--- assets >  4500.00
|   |   |   |   |--- class: 0
|   |   |--- income >  103.50
|   |   |   |--- time <= 33.00
|   |   |   |   |--- class: 0
|   |   |   |--- time >  33.00
|   |   |   |   |--- seniority <= 11.50
|   |   |   |   |   |--- age <= 37.50
|   |   |   |   |   |   |--- class: 1
|   |   |   |   |   |--- age >  37.50
|   |   |   |   |   |   |--- class: 0
|   |   |   |   |--- seniority >  11.50
|   |   |   |   |   |--- assets <= 7250.00
|   |   |   |   |   |   |--- class: 0
|   |   |   |   |   |--- assets >  7250.00
|   |   |   |   |   |   |--- class: 0
|--- records=no >  0.50
|   |--- job=partime <= 0.50
|   |   |--- income <= 74.50
|   |   |   |--- assets <= 4250.00
|   |   |   |   |--- income <= 20.00
|   |   |   |   |   |--- seniority <= 1.50
|   |   |   |   |   |   |--- class: 1
|   |   |   |   |   |--- seniority >  1.50
|   |   |   |   |   |   |--- class: 1
|   |   |   |   |--- income >  20.00
|   |   |   |   |   |--- expenses <= 71.00
|   |   |   |   |   |   |--- class: 0
|   |   |   |   |   |--- expenses >  71.00
|   |   |   |   |   |   |--- class: 1
|   |   |   |--- assets >  4250.00
|   |   |   |   |--- debt <= 1600.00
|   |   |   |   |   |--- seniority <= 2.50
|   |   |   |   |   |   |--- class: 0
|   |   |   |   |   |--- seniority >  2.50
|   |   |   |   |   |   |--- class: 0
|   |   |   |   |--- debt >  1600.00
|   |   |   |   |   |--- class: 1
|   |   |--- income >  74.50
|   |   |   |--- seniority <= 5.50
|   |   |   |   |--- amount <= 1330.00
|   |   |   |   |   |--- assets <= 3326.00
|   |   |   |   |   |   |--- class: 0
|   |   |   |   |   |--- assets >  3326.00
|   |   |   |   |   |   |--- class: 0
|   |   |   |   |--- amount >  1330.00
|   |   |   |   |   |--- assets <= 3750.00
|   |   |   |   |   |   |--- class: 1
|   |   |   |   |   |--- assets >  3750.00
|   |   |   |   |   |   |--- class: 0
|   |   |   |--- seniority >  5.50
|   |   |   |   |--- income <= 114.50
|   |   |   |   |   |--- expenses <= 84.00
|   |   |   |   |   |   |--- class: 0
|   |   |   |   |   |--- expenses >  84.00
|   |   |   |   |   |   |--- class: 0
|   |   |   |   |--- income >  114.50
|   |   |   |   |   |--- amount <= 1188.50
|   |   |   |   |   |   |--- class: 0
|   |   |   |   |   |--- amount >  1188.50
|   |   |   |   |   |   |--- class: 0
|   |--- job=partime >  0.50
|   |   |--- assets <= 7250.00
|   |   |   |--- time <= 27.00
|   |   |   |   |--- price <= 909.50
|   |   |   |   |   |--- class: 1
|   |   |   |   |--- price >  909.50
|   |   |   |   |   |--- class: 0
|   |   |   |--- time >  27.00
|   |   |   |   |--- age <= 34.50
|   |   |   |   |   |--- price <= 1049.00
|   |   |   |   |   |   |--- class: 1
|   |   |   |   |   |--- price >  1049.00
|   |   |   |   |   |   |--- class: 1
|   |   |   |   |--- age >  34.50
|   |   |   |   |   |--- assets <= 2750.00
|   |   |   |   |   |   |--- class: 1
|   |   |   |   |   |--- assets >  2750.00
|   |   |   |   |   |   |--- class: 1
|   |   |--- assets >  7250.00
|   |   |   |--- class: 0

6.6 Ensembles and random forest

Board of experts
Ensembling models
Random forest - ensembling decision trees
Tuning random forest

In [34]python · cell 58

python

from sklearn.ensemble import RandomForestClassifier

In [35]python · cell 59

python

scores = []

for n in range(10, 201, 10):
    rf = RandomForestClassifier(n_estimators=n, random_state=1)
    rf.fit(X_train, y_train)

    y_pred = rf.predict_proba(X_val)[:, 1]
    auc = roc_auc_score(y_val, y_pred)
    
    scores.append((n, auc))

In [36]python · cell 60

python

df_scores = pd.DataFrame(scores, columns=['n_estimators', 'auc'])

In [37]python · cell 61

python

plt.plot(df_scores.n_estimators, df_scores.auc)

Output

[<matplotlib.lines.Line2D at 0xff7f36d618e0>]

<Figure size 432x288 with 1 Axes>

In [41]python · cell 62

python

scores = []

for d in [5, 10, 15]:
    for n in range(10, 201, 10):
        rf = RandomForestClassifier(n_estimators=n,
                                    max_depth=d,
                                    random_state=1)
        rf.fit(X_train, y_train)

        y_pred = rf.predict_proba(X_val)[:, 1]
        auc = roc_auc_score(y_val, y_pred)

        scores.append((d, n, auc))

In [42]python · cell 63

python

columns = ['max_depth', 'n_estimators', 'auc']
df_scores = pd.DataFrame(scores, columns=columns)

In [43]python · cell 64

python

for d in [5, 10, 15]:
    df_subset = df_scores[df_scores.max_depth == d]
    
    plt.plot(df_subset.n_estimators, df_subset.auc,
             label='max_depth=%d' % d)

plt.legend()

Output

<matplotlib.legend.Legend at 0xff7f375fe490>

<Figure size 432x288 with 1 Axes>

In [44]python · cell 65

python

max_depth = 10

In [45]python · cell 66

python

scores = []

for s in [1, 3, 5, 10, 50]:
    for n in range(10, 201, 10):
        rf = RandomForestClassifier(n_estimators=n,
                                    max_depth=max_depth,
                                    min_samples_leaf=s,
                                    random_state=1)
        rf.fit(X_train, y_train)

        y_pred = rf.predict_proba(X_val)[:, 1]
        auc = roc_auc_score(y_val, y_pred)

        scores.append((s, n, auc))

In [47]python · cell 67

python

columns = ['min_samples_leaf', 'n_estimators', 'auc']
df_scores = pd.DataFrame(scores, columns=columns)

In [48]python · cell 68

python

colors = ['black', 'blue', 'orange', 'red', 'grey']
values = [1, 3, 5, 10, 50]

for s, col in zip(values, colors):
    df_subset = df_scores[df_scores.min_samples_leaf == s]
    
    plt.plot(df_subset.n_estimators, df_subset.auc,
             color=col,
             label='min_samples_leaf=%d' % s)

plt.legend()

Output

<matplotlib.legend.Legend at 0xff7f363cd310>

<Figure size 432x288 with 1 Axes>

In [49]python · cell 69

python

min_samples_leaf = 3

In [50]python · cell 70

python

rf = RandomForestClassifier(n_estimators=200,
                            max_depth=max_depth,
                            min_samples_leaf=min_samples_leaf,
                            random_state=1)
rf.fit(X_train, y_train)

Output

RandomForestClassifier(max_depth=10, min_samples_leaf=3, n_estimators=200,
                       random_state=1)

Other useful parametes:

max_features
bootstrap

https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html

6.7 Gradient boosting and XGBoost

Gradient boosting vs random forest
Installing XGBoost
Training the first model
Performance monitoring
Parsing xgboost's monitoring output

In [52]python · cell 73

python

!pip install xgboost

Output

Requirement already satisfied: xgboost in /home/alexey/.pyenv/versions/3.8.11/lib/python3.8/site-packages (1.4.2)
Requirement already satisfied: numpy in /home/alexey/.pyenv/versions/3.8.11/lib/python3.8/site-packages (from xgboost) (1.21.1)
Requirement already satisfied: scipy in /home/alexey/.pyenv/versions/3.8.11/lib/python3.8/site-packages (from xgboost) (1.7.0)

In [53]python · cell 74

python

import xgboost as xgb

In [56]python · cell 75

python

features = list(dv.get_feature_names_out())
dtrain = xgb.DMatrix(X_train, label=y_train, feature_names=features)
dval = xgb.DMatrix(X_val, label=y_val, feature_names=features)

In [65]python · cell 76

python

xgb_params = {
    'eta': 0.3, 
    'max_depth': 6,
    'min_child_weight': 1,
    
    'objective': 'binary:logistic',
    'nthread': 8,
    
    'seed': 1,
    'verbosity': 1,
}

model = xgb.train(xgb_params, dtrain, num_boost_round=10)

Output

[22:13:34] WARNING: ../src/learner.cc:1095: Starting in XGBoost 1.3.0, the default evaluation metric used with the objective 'binary:logistic' was changed from 'error' to 'logloss'. Explicitly set eval_metric if you'd like to restore the old behavior.

In [66]python · cell 77

python

y_pred = model.predict(dval)

In [67]python · cell 78

python

roc_auc_score(y_val, y_pred)

Output

0.8152745150274878

In [68]python · cell 79

python

watchlist = [(dtrain, 'train'), (dval, 'val')]

In [72]python · cell 80

python

%%capture output

xgb_params = {
    'eta': 0.3, 
    'max_depth': 6,
    'min_child_weight': 1,
    
    'objective': 'binary:logistic',
    'eval_metric': 'auc',

    'nthread': 8,
    'seed': 1,
    'verbosity': 1,
}

model = xgb.train(xgb_params, dtrain, num_boost_round=200,
                  verbose_eval=5,
                  evals=watchlist)

In [76]python · cell 81

python

s = output.stdout

In [97]python · cell 82

python

print(s[:200])

Output

[0]	train-auc:0.86300	val-auc:0.76818
[5]	train-auc:0.92863	val-auc:0.80606
[10]	train-auc:0.95002	val-auc:0.81558
[15]	train-auc:0.96558	val-auc:0.81680
[20]	train-auc:0.97316	val-auc:0.81775
[25]	tr

In [91]python · cell 83

python

def parse_xgb_output(output):
    results = []

    for line in output.stdout.strip().split('\n'):
        it_line, train_line, val_line = line.split('\t')

        it = int(it_line.strip('[]'))
        train = float(train_line.split(':')[1])
        val = float(val_line.split(':')[1])

        results.append((it, train, val))
    
    columns = ['num_iter', 'train_auc', 'val_auc']
    df_results = pd.DataFrame(results, columns=columns)
    return df_results

In [93]python · cell 84

python

df_score = parse_xgb_output(output)

In [98]python · cell 85

python

plt.plot(df_score.num_iter, df_score.train_auc, label='train')
plt.plot(df_score.num_iter, df_score.val_auc, label='val')
plt.legend()

Output

<matplotlib.legend.Legend at 0xff7f789d6c70>

<Figure size 432x288 with 1 Axes>

In [95]python · cell 86

python

plt.plot(df_score.num_iter, df_score.val_auc, label='val')
plt.legend()

Output

<matplotlib.legend.Legend at 0xff7f78c5fd90>

<Figure size 432x288 with 1 Axes>

6.8 XGBoost parameter tuning

Tuning the following parameters:

eta
max_depth
min_child_weight

In [100]python · cell 88

python

scores = {}

In [114]python · cell 89

python

%%capture output

xgb_params = {
    'eta': 0.01, 
    'max_depth': 6,
    'min_child_weight': 1,
    
    'objective': 'binary:logistic',
    'eval_metric': 'auc',

    'nthread': 8,
    'seed': 1,
    'verbosity': 1,
}

model = xgb.train(xgb_params, dtrain, num_boost_round=200,
                  verbose_eval=5,
                  evals=watchlist)

In [100]python · cell 90

python

scores = {}

In [115]python · cell 91

python

key = 'eta=%s' % (xgb_params['eta'])
scores[key] = parse_xgb_output(output)
key

Output

'eta=0.01'

In [121]python · cell 92

python

scores = {}

In [128]python · cell 93

python

%%capture output

xgb_params = {
    'eta': 0.1, 
    'max_depth': 10,
    'min_child_weight': 1,
    
    'objective': 'binary:logistic',
    'eval_metric': 'auc',

    'nthread': 8,
    'seed': 1,
    'verbosity': 1,
}

model = xgb.train(xgb_params, dtrain, num_boost_round=200,
                  verbose_eval=5,
                  evals=watchlist)

In [129]python · cell 94

python

key = 'max_depth=%s' % (xgb_params['max_depth'])
scores[key] = parse_xgb_output(output)
key

Output

'max_depth=10'

In [131]python · cell 95

python

del scores['max_depth=10']

In [133]python · cell 96

python

for max_depth, df_score in scores.items():
    plt.plot(df_score.num_iter, df_score.val_auc, label=max_depth)

plt.ylim(0.8, 0.84)
plt.legend()

Output

<matplotlib.legend.Legend at 0xff7f787909d0>

<Figure size 432x288 with 1 Axes>

In [121]python · cell 97

python

scores = {}

In [138]python · cell 98

python

%%capture output

xgb_params = {
    'eta': 0.1, 
    'max_depth': 3,
    'min_child_weight': 30,
    
    'objective': 'binary:logistic',
    'eval_metric': 'auc',

    'nthread': 8,
    'seed': 1,
    'verbosity': 1,
}

model = xgb.train(xgb_params, dtrain, num_boost_round=200,
                  verbose_eval=5,
                  evals=watchlist)

In [139]python · cell 99

python

key = 'min_child_weight=%s' % (xgb_params['min_child_weight'])
scores[key] = parse_xgb_output(output)
key

Output

'min_child_weight=30'

In [144]python · cell 100

python

for min_child_weight, df_score in scores.items():
    plt.plot(df_score.num_iter, df_score.val_auc, label=min_child_weight)

plt.ylim(0.82, 0.84)
plt.legend()

Output

<matplotlib.legend.Legend at 0xff7f784f6b50>

<Figure size 432x288 with 1 Axes>

In [145]python · cell 101

python

xgb_params = {
    'eta': 0.1, 
    'max_depth': 3,
    'min_child_weight': 1,

    'objective': 'binary:logistic',
    'eval_metric': 'auc',

    'nthread': 8,
    'seed': 1,
    'verbosity': 1,
}

model = xgb.train(xgb_params, dtrain, num_boost_round=175)

Other parameters: https://xgboost.readthedocs.io/en/latest/parameter.html

Useful ones:

subsample and colsample_bytree
lambda and alpha

6.9 Selecting the final model

Choosing between xgboost, random forest and decision tree
Training the final model
Saving the model

In [146]python · cell 104

python

dt = DecisionTreeClassifier(max_depth=6, min_samples_leaf=15)
dt.fit(X_train, y_train)

Output

DecisionTreeClassifier(max_depth=6, min_samples_leaf=15)

In [150]python · cell 105

python

y_pred = dt.predict_proba(X_val)[:, 1]
roc_auc_score(y_val, y_pred)

Output

0.7850802838390931

In [148]python · cell 106

python

rf = RandomForestClassifier(n_estimators=200,
                            max_depth=10,
                            min_samples_leaf=3,
                            random_state=1)
rf.fit(X_train, y_train)

Output

RandomForestClassifier(max_depth=10, min_samples_leaf=3, n_estimators=200,
                       random_state=1)

In [151]python · cell 107

python

y_pred = rf.predict_proba(X_val)[:, 1]
roc_auc_score(y_val, y_pred)

Output

0.8249709379767989

In [149]python · cell 108

python

xgb_params = {
    'eta': 0.1, 
    'max_depth': 3,
    'min_child_weight': 1,

    'objective': 'binary:logistic',
    'eval_metric': 'auc',

    'nthread': 8,
    'seed': 1,
    'verbosity': 1,
}

model = xgb.train(xgb_params, dtrain, num_boost_round=175)

In [152]python · cell 109

python

y_pred = model.predict(dval)
roc_auc_score(y_val, y_pred)

Output

0.8360387251459157

In [155]python · cell 110

python

df_full_train = df_full_train.reset_index(drop=True)

In [157]python · cell 111

python

y_full_train = (df_full_train.status == 'default').astype(int).values

In [159]python · cell 112

python

del df_full_train['status']

In [161]python · cell 113

python

dicts_full_train = df_full_train.to_dict(orient='records')

dv = DictVectorizer(sparse=False)
X_full_train = dv.fit_transform(dicts_full_train)

dicts_test = df_test.to_dict(orient='records')
X_test = dv.transform(dicts_test)

In [164]python · cell 114

python

dfulltrain = xgb.DMatrix(X_full_train, label=y_full_train,
                    feature_names=dv.get_feature_names_out())

dtest = xgb.DMatrix(X_test, feature_names=dv.get_feature_names_out())

In [165]python · cell 115

python

xgb_params = {
    'eta': 0.1, 
    'max_depth': 3,
    'min_child_weight': 1,

    'objective': 'binary:logistic',
    'eval_metric': 'auc',

    'nthread': 8,
    'seed': 1,
    'verbosity': 1,
}

model = xgb.train(xgb_params, dfulltrain, num_boost_round=175)

In [166]python · cell 116

python

y_pred = model.predict(dtest)

In [169]python · cell 117

python

roc_auc_score(y_test, y_pred)

Output

0.8322662626460096

6.10 Summary

Decision trees learn if-then-else rules from data.
Finding the best split: select the least impure split. This algorithm can overfit, that's why we control it by limiting the max depth and the size of the group.
Random forest is a way of combininig multiple decision trees. It should have a diverse set of models to make good predictions.
Gradient boosting trains model sequentially: each model tries to fix errors of the previous model. XGBoost is an implementation of gradient boosting.

6.11 Explore more

For this dataset we didn't do EDA or feature engineering. You can do it to get more insights into the problem.
For random forest, there are more parameters that we can tune. Check max_features and bootstrap.
There's a variation of random forest caled "extremely randomized trees", or "extra trees". Instead of selecting the best split among all possible thresholds, it selects a few thresholds randomly and picks the best one among them. Because of that extra trees never overfit. In Scikit-Learn, they are implemented in ExtraTreesClassifier. Try it for this project.
XGBoost can deal with NAs - we don't have to do fillna for it. Check if not filling NA's help improve performance.
Experiment with other XGBoost parameters: subsample and colsample_bytree.
When selecting the best split, decision trees find the most useful features. This information can be used for understanding which features are more important than otheres. See example here for random forest (it's the same for plain decision trees) and for xgboost
Trees can also be used for solving the regression problems: check DecisionTreeRegressor, RandomForestRegressor and the objective=reg:squarederror parameter for XGBoost.