Chapter 47
4. Evaluation Metrics for Classification
NotebookPython 3 (ipykernel)81 cells
In [1]python · cell 1
python
%autosave 0Output
Autosave disabled
4. Evaluation Metrics for Classification
In the previous session we trained a model for predicting churn. How do we know if it's good?
4.1 Evaluation metrics: session overview
- Dataset: https://www.kaggle.com/blastchar/telco-customer-churn
- https://raw.githubusercontent.com/alexeygrigorev/mlbookcamp-code/master/chapter-03-churn-prediction/WA_Fn-UseC_-Telco-Customer-Churn.csv
Metric - function that compares the predictions with the actual values and outputs a single number that tells how good the predictions are
In [3]python · cell 3
python
import pandas as pd
import numpy as np
import matplotlib.pyplot as pltIn [4]python · cell 4
python
from sklearn.model_selection import train_test_split
from sklearn.feature_extraction import DictVectorizer
from sklearn.linear_model import LogisticRegressionIn [5]python · cell 5
python
df = pd.read_csv('data-week-3.csv')
df.columns = df.columns.str.lower().str.replace(' ', '_')
categorical_columns = list(df.dtypes[df.dtypes == 'object'].index)
for c in categorical_columns:
df[c] = df[c].str.lower().str.replace(' ', '_')
df.totalcharges = pd.to_numeric(df.totalcharges, errors='coerce')
df.totalcharges = df.totalcharges.fillna(0)
df.churn = (df.churn == 'yes').astype(int)In [6]python · cell 6
python
df_full_train, df_test = train_test_split(df, test_size=0.2, random_state=1)
df_train, df_val = train_test_split(df_full_train, test_size=0.25, random_state=1)
df_train = df_train.reset_index(drop=True)
df_val = df_val.reset_index(drop=True)
df_test = df_test.reset_index(drop=True)
y_train = df_train.churn.values
y_val = df_val.churn.values
y_test = df_test.churn.values
del df_train['churn']
del df_val['churn']
del df_test['churn']In [7]python · cell 7
python
numerical = ['tenure', 'monthlycharges', 'totalcharges']
categorical = [
'gender',
'seniorcitizen',
'partner',
'dependents',
'phoneservice',
'multiplelines',
'internetservice',
'onlinesecurity',
'onlinebackup',
'deviceprotection',
'techsupport',
'streamingtv',
'streamingmovies',
'contract',
'paperlessbilling',
'paymentmethod',
]In [8]python · cell 8
python
dv = DictVectorizer(sparse=False)
train_dict = df_train[categorical + numerical].to_dict(orient='records')
X_train = dv.fit_transform(train_dict)
model = LogisticRegression()
model.fit(X_train, y_train)Output
LogisticRegression()
In [9]python · cell 9
python
val_dict = df_val[categorical + numerical].to_dict(orient='records')
X_val = dv.transform(val_dict)
y_pred = model.predict_proba(X_val)[:, 1]
churn_decision = (y_pred >= 0.5)
(y_val == churn_decision).mean()Output
0.8034066713981547
4.2 Accuracy and dummy model
- Evaluate the model on different thresholds
- Check the accuracy of dummy baselines
In [16]python · cell 11
python
len(y_val)Output
1409
In [19]python · cell 12
python
(y_val == churn_decision).mean()Output
0.8034066713981547
In [18]python · cell 13
python
1132/ 1409Output
0.8034066713981547
In [24]python · cell 14
python
from sklearn.metrics import accuracy_scoreIn [29]python · cell 15
python
accuracy_score(y_val, y_pred >= 0.5)Output
0.8034066713981547
In [30]python · cell 16
python
thresholds = np.linspace(0, 1, 21)
scores = []
for t in thresholds:
score = accuracy_score(y_val, y_pred >= t)
print('%.2f %.3f' % (t, score))
scores.append(score)Output
0.00 0.274 0.05 0.509 0.10 0.591 0.15 0.666 0.20 0.710 0.25 0.739 0.30 0.760 0.35 0.772 0.40 0.785 0.45 0.793 0.50 0.803 0.55 0.801 0.60 0.795 0.65 0.786 0.70 0.766 0.75 0.744 0.80 0.735 0.85 0.726 0.90 0.726 0.95 0.726 1.00 0.726
In [24]python · cell 17
python
plt.plot(thresholds, scores)Output
[<matplotlib.lines.Line2D at 0xff7b24eb48e0>]
<Figure size 432x288 with 1 Axes>
In [34]python · cell 18
python
from collections import CounterIn [35]python · cell 19
python
Counter(y_pred >= 1.0)Output
Counter({False: 1409})In [39]python · cell 20
python
1 - y_val.mean()Output
0.7260468417317246
4.3 Confusion table
- Different types of errors and correct decisions
- Arranging them in a table
In [25]python · cell 22
python
actual_positive = (y_val == 1)
actual_negative = (y_val == 0)In [26]python · cell 23
python
t = 0.5
predict_positive = (y_pred >= t)
predict_negative = (y_pred < t)In [27]python · cell 24
python
tp = (predict_positive & actual_positive).sum()
tn = (predict_negative & actual_negative).sum()
fp = (predict_positive & actual_negative).sum()
fn = (predict_negative & actual_positive).sum()In [28]python · cell 25
python
confusion_matrix = np.array([
[tn, fp],
[fn, tp]
])
confusion_matrixOutput
array([[922, 101],
[176, 210]])In [29]python · cell 26
python
(confusion_matrix / confusion_matrix.sum()).round(2)Output
array([[0.65, 0.07],
[0.12, 0.15]])4.4 Precision and Recall
In [30]python · cell 28
python
p = tp / (tp + fp)
pOutput
0.6752411575562701
In [31]python · cell 29
python
r = tp / (tp + fn)
rOutput
0.5440414507772021
4.5 ROC Curves
TPR and FRP
In [32]python · cell 31
python
tpr = tp / (tp + fn)
tprOutput
0.5440414507772021
In [33]python · cell 32
python
fpr = fp / (fp + tn)
fprOutput
0.09872922776148582
In [34]python · cell 33
python
scores = []
thresholds = np.linspace(0, 1, 101)
for t in thresholds:
actual_positive = (y_val == 1)
actual_negative = (y_val == 0)
predict_positive = (y_pred >= t)
predict_negative = (y_pred < t)
tp = (predict_positive & actual_positive).sum()
tn = (predict_negative & actual_negative).sum()
fp = (predict_positive & actual_negative).sum()
fn = (predict_negative & actual_positive).sum()
scores.append((t, tp, fp, fn, tn))In [35]python · cell 34
python
columns = ['threshold', 'tp', 'fp', 'fn', 'tn']
df_scores = pd.DataFrame(scores, columns=columns)
df_scores['tpr'] = df_scores.tp / (df_scores.tp + df_scores.fn)
df_scores['fpr'] = df_scores.fp / (df_scores.fp + df_scores.tn)In [36]python · cell 35
python
plt.plot(df_scores.threshold, df_scores['tpr'], label='TPR')
plt.plot(df_scores.threshold, df_scores['fpr'], label='FPR')
plt.legend()Output
<matplotlib.legend.Legend at 0xffaca7c6f9a0>
<Figure size 432x288 with 1 Axes>
Random model
In [37]python · cell 37
python
np.random.seed(1)
y_rand = np.random.uniform(0, 1, size=len(y_val))In [38]python · cell 38
python
((y_rand >= 0.5) == y_val).mean()Output
0.5017743080198722
In [47]python · cell 39
python
def tpr_fpr_dataframe(y_val, y_pred):
scores = []
thresholds = np.linspace(0, 1, 101)
for t in thresholds:
actual_positive = (y_val == 1)
actual_negative = (y_val == 0)
predict_positive = (y_pred >= t)
predict_negative = (y_pred < t)
tp = (predict_positive & actual_positive).sum()
tn = (predict_negative & actual_negative).sum()
fp = (predict_positive & actual_negative).sum()
fn = (predict_negative & actual_positive).sum()
scores.append((t, tp, fp, fn, tn))
columns = ['threshold', 'tp', 'fp', 'fn', 'tn']
df_scores = pd.DataFrame(scores, columns=columns)
df_scores['tpr'] = df_scores.tp / (df_scores.tp + df_scores.fn)
df_scores['fpr'] = df_scores.fp / (df_scores.fp + df_scores.tn)
return df_scoresIn [40]python · cell 40
python
df_rand = tpr_fpr_dataframe(y_val, y_rand)In [41]python · cell 41
python
plt.plot(df_rand.threshold, df_rand['tpr'], label='TPR')
plt.plot(df_rand.threshold, df_rand['fpr'], label='FPR')
plt.legend()Output
<matplotlib.legend.Legend at 0xffaca7bec8b0>
<Figure size 432x288 with 1 Axes>
Ideal model
In [42]python · cell 43
python
num_neg = (y_val == 0).sum()
num_pos = (y_val == 1).sum()
num_neg, num_posOutput
(1023, 386)
In [43]python · cell 44
python
y_ideal = np.repeat([0, 1], [num_neg, num_pos])
y_ideal
y_ideal_pred = np.linspace(0, 1, len(y_val))In [44]python · cell 45
python
1 - y_val.mean()Output
0.7260468417317246
In [45]python · cell 46
python
accuracy_score(y_ideal, y_ideal_pred >= 0.726)Output
1.0
In [51]python · cell 47
python
df_ideal = tpr_fpr_dataframe(y_ideal, y_ideal_pred)
df_ideal[::10]Output
threshold tp fp fn tn tpr fpr 0 0.0 386 1023 0 0 1.000000 1.000000 10 0.1 386 882 0 141 1.000000 0.862170 20 0.2 386 741 0 282 1.000000 0.724340 30 0.3 386 600 0 423 1.000000 0.586510 40 0.4 386 459 0 564 1.000000 0.448680 50 0.5 386 319 0 704 1.000000 0.311828 60 0.6 386 178 0 845 1.000000 0.173998 70 0.7 386 37 0 986 1.000000 0.036168 80 0.8 282 0 104 1023 0.730570 0.000000 90 0.9 141 0 245 1023 0.365285 0.000000 100 1.0 1 0 385 1023 0.002591 0.000000
.dataframe tbody tr th:only-of-type {
vertical-align: middle;
}
.dataframe tbody tr th {
vertical-align: top;
}
.dataframe thead th {
text-align: right;
}
| threshold | tp | fp | fn | tn | tpr | fpr | |
|---|---|---|---|---|---|---|---|
| 0 | 0.0 | 386 | 1023 | 0 | 0 | 1.000000 | 1.000000 |
| 10 | 0.1 | 386 | 882 | 0 | 141 | 1.000000 | 0.862170 |
| 20 | 0.2 | 386 | 741 | 0 | 282 | 1.000000 | 0.724340 |
| 30 | 0.3 | 386 | 600 | 0 | 423 | 1.000000 | 0.586510 |
| 40 | 0.4 | 386 | 459 | 0 | 564 | 1.000000 | 0.448680 |
| 50 | 0.5 | 386 | 319 | 0 | 704 | 1.000000 | 0.311828 |
| 60 | 0.6 | 386 | 178 | 0 | 845 | 1.000000 | 0.173998 |
| 70 | 0.7 | 386 | 37 | 0 | 986 | 1.000000 | 0.036168 |
| 80 | 0.8 | 282 | 0 | 104 | 1023 | 0.730570 | 0.000000 |
| 90 | 0.9 | 141 | 0 | 245 | 1023 | 0.365285 | 0.000000 |
| 100 | 1.0 | 1 | 0 | 385 | 1023 | 0.002591 | 0.000000 |
In [52]python · cell 48
python
plt.plot(df_ideal.threshold, df_ideal['tpr'], label='TPR')
plt.plot(df_ideal.threshold, df_ideal['fpr'], label='FPR')
plt.legend()Output
<matplotlib.legend.Legend at 0xffaca7af4a90>
<Figure size 432x288 with 1 Axes>
Putting everything together
In [53]python · cell 50
python
plt.plot(df_scores.threshold, df_scores['tpr'], label='TPR', color='black')
plt.plot(df_scores.threshold, df_scores['fpr'], label='FPR', color='blue')
plt.plot(df_ideal.threshold, df_ideal['tpr'], label='TPR ideal')
plt.plot(df_ideal.threshold, df_ideal['fpr'], label='FPR ideal')
# plt.plot(df_rand.threshold, df_rand['tpr'], label='TPR random', color='grey')
# plt.plot(df_rand.threshold, df_rand['fpr'], label='FPR random', color='grey')
plt.legend()Output
<matplotlib.legend.Legend at 0xffaca7a695b0>
<Figure size 432x288 with 1 Axes>
In [54]python · cell 51
python
plt.figure(figsize=(5, 5))
plt.plot(df_scores.fpr, df_scores.tpr, label='Model')
plt.plot([0, 1], [0, 1], label='Random', linestyle='--')
plt.xlabel('FPR')
plt.ylabel('TPR')
plt.legend()Output
<matplotlib.legend.Legend at 0xffaca6b72e50>
<Figure size 360x360 with 1 Axes>
In [55]python · cell 52
python
from sklearn.metrics import roc_curveIn [57]python · cell 53
python
fpr, tpr, thresholds = roc_curve(y_val, y_pred)In [58]python · cell 54
python
plt.figure(figsize=(5, 5))
plt.plot(fpr, tpr, label='Model')
plt.plot([0, 1], [0, 1], label='Random', linestyle='--')
plt.xlabel('FPR')
plt.ylabel('TPR')
plt.legend()Output
<matplotlib.legend.Legend at 0xffaca5d3eaf0>
<Figure size 360x360 with 1 Axes>
4.6 ROC AUC
- Area under the ROC curve - useful metric
- Interpretation of AUC
In [60]python · cell 56
python
from sklearn.metrics import aucIn [61]python · cell 57
python
auc(fpr, tpr)Output
0.843850505725819
In [62]python · cell 58
python
auc(df_scores.fpr, df_scores.tpr)Output
0.8438796286447967
In [63]python · cell 59
python
auc(df_ideal.fpr, df_ideal.tpr)Output
0.9999430203759136
In [68]python · cell 60
python
fpr, tpr, thresholds = roc_curve(y_val, y_pred)
auc(fpr, tpr)Output
0.843850505725819
In [65]python · cell 61
python
from sklearn.metrics import roc_auc_scoreIn [66]python · cell 62
python
roc_auc_score(y_val, y_pred)Output
0.843850505725819
In [70]python · cell 63
python
neg = y_pred[y_val == 0]
pos = y_pred[y_val == 1]In [73]python · cell 64
python
import randomIn [82]python · cell 65
python
n = 100000
success = 0
for i in range(n):
pos_ind = random.randint(0, len(pos) - 1)
neg_ind = random.randint(0, len(neg) - 1)
if pos[pos_ind] > neg[neg_ind]:
success = success + 1
success / nOutput
0.8434
In [90]python · cell 66
python
n = 50000
np.random.seed(1)
pos_ind = np.random.randint(0, len(pos), size=n)
neg_ind = np.random.randint(0, len(neg), size=n)
(pos[pos_ind] > neg[neg_ind]).mean()Output
0.84646
4.7 Cross-Validation
- Evaluating the same model on different subsets of data
- Getting the average prediction and the spread within predictions
In [121]python · cell 68
python
def train(df_train, y_train, C=1.0):
dicts = df_train[categorical + numerical].to_dict(orient='records')
dv = DictVectorizer(sparse=False)
X_train = dv.fit_transform(dicts)
model = LogisticRegression(C=C, max_iter=1000)
model.fit(X_train, y_train)
return dv, modelIn [123]python · cell 69
python
dv, model = train(df_train, y_train, C=0.001)In [110]python · cell 70
python
def predict(df, dv, model):
dicts = df[categorical + numerical].to_dict(orient='records')
X = dv.transform(dicts)
y_pred = model.predict_proba(X)[:, 1]
return y_predIn [98]python · cell 71
python
y_pred = predict(df_val, dv, model)In [99]python · cell 72
python
from sklearn.model_selection import KFoldIn [100]python · cell 73
python
In [112]python · cell 74
python
!pip install tqdmOutput
Requirement already satisfied: tqdm in /home/alexey/.pyenv/versions/3.8.11/lib/python3.8/site-packages (4.61.2) [33mWARNING: You are using pip version 21.2.2; however, version 21.2.4 is available. You should consider upgrading via the '/home/alexey/.pyenv/versions/3.8.11/bin/python3.8 -m pip install --upgrade pip' command.[0m
In [113]python · cell 75
python
from tqdm.auto import tqdmIn [129]python · cell 76
python
n_splits = 5
for C in tqdm([0.001, 0.01, 0.1, 0.5, 1, 5, 10]):
kfold = KFold(n_splits=n_splits, shuffle=True, random_state=1)
scores = []
for train_idx, val_idx in kfold.split(df_full_train):
df_train = df_full_train.iloc[train_idx]
df_val = df_full_train.iloc[val_idx]
y_train = df_train.churn.values
y_val = df_val.churn.values
dv, model = train(df_train, y_train, C=C)
y_pred = predict(df_val, dv, model)
auc = roc_auc_score(y_val, y_pred)
scores.append(auc)
print('C=%s %.3f +- %.3f' % (C, np.mean(scores), np.std(scores)))Output
0%| | 0/7 [00:00<?, ?it/s]
C=0.001 0.825 +- 0.009 C=0.01 0.840 +- 0.009 C=0.1 0.841 +- 0.008 C=0.5 0.840 +- 0.007 C=1 0.841 +- 0.008 C=5 0.841 +- 0.008 C=10 0.841 +- 0.008
In [133]python · cell 77
python
scoresOutput
[0.8419433083969826, 0.8458047775129122, 0.8325145494681918, 0.8325466042079682, 0.8525462018763139]
In [131]python · cell 78
python
dv, model = train(df_full_train, df_full_train.churn.values, C=1.0)
y_pred = predict(df_test, dv, model)
auc = roc_auc_score(y_test, y_pred)
aucOutput
0.8572386167896259
4.8 Summary
- Metric - a single number that describes the performance of a model
- Accuracy - fraction of correct answers; sometimes misleading
- Precision and recall are less misleading when we have class inbalance
- ROC Curve - a way to evaluate the performance at all thresholds; okay to use with imbalance
- K-Fold CV - more reliable estimate for performance (mean + std)
4.9 Explore more
- Check the precision and recall of the dummy classifier that always predict "FALSE"
- F1 score = 2 * P * R / (P + R)
- Evaluate precision and recall at different thresholds, plot P vs R - this way you'll get the precision/recall curve (similar to ROC curve)
- Area under the PR curve is also a useful metric
Other projects:
- Calculate the metrics for datasets from the previous week
In [ ]python · cell 81
python