Chapter 31
Tabular Modeling Deep Dive
NotebookPython 3 (ipykernel)146 cells
In [ ]python · cell 1
python
#hide
! [ -e /content ] && pip install -Uqq fastbook kaggle waterfallcharts treeinterpreter dtreeviz==1.4.1
import fastbook
fastbook.setup_book()In [ ]python · cell 2
python
#hide
from fastbook import *
from pandas.api.types import is_string_dtype, is_numeric_dtype, is_categorical_dtype
from fastai.tabular.all import *
from sklearn.ensemble import RandomForestRegressor
from sklearn.tree import DecisionTreeRegressor
from dtreeviz.trees import *
from IPython.display import Image, display_svg, SVG
pd.options.display.max_rows = 20
pd.options.display.max_columns = 8Tabular Modeling Deep Dive
Categorical Embeddings
Beyond Deep Learning
The Dataset
Kaggle Competitions
In [ ]python · cell 8
python
creds = ''In [ ]python · cell 9
python
cred_path = Path('~/.kaggle/kaggle.json').expanduser()
if not cred_path.exists():
cred_path.parent.mkdir(exist_ok=True)
cred_path.write_text(creds)
cred_path.chmod(0o600)In [ ]python · cell 10
python
comp = 'bluebook-for-bulldozers'
path = URLs.path(comp)
pathIn [ ]python · cell 11
python
#hide
Path.BASE_PATH = pathIn [ ]python · cell 12
python
from kaggle import api
if not path.exists():
path.mkdir(parents=true)
api.competition_download_cli(comp, path=path)
shutil.unpack_archive(str(path/f'{comp}.zip'), str(path))
path.ls(file_type='text')Look at the Data
In [ ]python · cell 14
python
df = pd.read_csv(path/'TrainAndValid.csv', low_memory=False)In [ ]python · cell 15
python
df.columnsIn [ ]python · cell 16
python
df['ProductSize'].unique()In [ ]python · cell 17
python
sizes = 'Large','Large / Medium','Medium','Small','Mini','Compact'In [ ]python · cell 18
python
df['ProductSize'] = df['ProductSize'].astype('category')
df['ProductSize'].cat.set_categories(sizes, ordered=True, inplace=True)In [ ]python · cell 19
python
dep_var = 'SalePrice'In [ ]python · cell 20
python
df[dep_var] = np.log(df[dep_var])Decision Trees
Handling Dates
In [ ]python · cell 23
python
df = add_datepart(df, 'saledate')In [ ]python · cell 24
python
df_test = pd.read_csv(path/'Test.csv', low_memory=False)
df_test = add_datepart(df_test, 'saledate')In [ ]python · cell 25
python
' '.join(o for o in df.columns if o.startswith('sale'))Using TabularPandas and TabularProc
In [ ]python · cell 27
python
procs = [Categorify, FillMissing]In [ ]python · cell 28
python
cond = (df.saleYear<2011) | (df.saleMonth<10)
train_idx = np.where( cond)[0]
valid_idx = np.where(~cond)[0]
splits = (list(train_idx),list(valid_idx))In [ ]python · cell 29
python
cont,cat = cont_cat_split(df, 1, dep_var=dep_var)In [ ]python · cell 30
python
to = TabularPandas(df, procs, cat, cont, y_names=dep_var, splits=splits)In [ ]python · cell 31
python
len(to.train),len(to.valid)In [ ]python · cell 32
python
to.show(3)In [ ]python · cell 33
python
to1 = TabularPandas(df, procs, ['state', 'ProductGroup', 'Drive_System', 'Enclosure'], [], y_names=dep_var, splits=splits)
to1.show(3)In [ ]python · cell 34
python
to.items.head(3)In [ ]python · cell 35
python
to1.items[['state', 'ProductGroup', 'Drive_System', 'Enclosure']].head(3)In [ ]python · cell 36
python
to.classes['ProductSize']In [ ]python · cell 37
python
save_pickle(path/'to.pkl',to)Creating the Decision Tree
In [ ]python · cell 39
python
#hide
to = load_pickle(path/'to.pkl')In [ ]python · cell 40
python
xs,y = to.train.xs,to.train.y
valid_xs,valid_y = to.valid.xs,to.valid.yIn [ ]python · cell 41
python
m = DecisionTreeRegressor(max_leaf_nodes=4)
m.fit(xs, y);In [ ]python · cell 42
python
draw_tree(m, xs, size=10, leaves_parallel=True, precision=2)In [ ]python · cell 43
python
samp_idx = np.random.permutation(len(y))[:500]
dtreeviz(m, xs.iloc[samp_idx], y.iloc[samp_idx], xs.columns, dep_var,
fontname='DejaVu Sans', scale=1.6, label_fontsize=10,
orientation='LR')In [ ]python · cell 44
python
xs.loc[xs['YearMade']<1900, 'YearMade'] = 1950
valid_xs.loc[valid_xs['YearMade']<1900, 'YearMade'] = 1950In [ ]python · cell 45
python
m = DecisionTreeRegressor(max_leaf_nodes=4).fit(xs, y)
dtreeviz(m, xs.iloc[samp_idx], y.iloc[samp_idx], xs.columns, dep_var,
fontname='DejaVu Sans', scale=1.6, label_fontsize=10,
orientation='LR')In [ ]python · cell 46
python
m = DecisionTreeRegressor()
m.fit(xs, y);In [ ]python · cell 47
python
def r_mse(pred,y): return round(math.sqrt(((pred-y)**2).mean()), 6)
def m_rmse(m, xs, y): return r_mse(m.predict(xs), y)In [ ]python · cell 48
python
m_rmse(m, xs, y)In [ ]python · cell 49
python
m_rmse(m, valid_xs, valid_y)In [ ]python · cell 50
python
m.get_n_leaves(), len(xs)In [ ]python · cell 51
python
m = DecisionTreeRegressor(min_samples_leaf=25)
m.fit(to.train.xs, to.train.y)
m_rmse(m, xs, y), m_rmse(m, valid_xs, valid_y)In [ ]python · cell 52
python
m.get_n_leaves()Categorical Variables
Random Forests
In [ ]python · cell 55
python
#hide
# pip install —pre -f https://sklearn-nightly.scdn8.secure.raxcdn.com scikit-learn —UCreating a Random Forest
In [ ]python · cell 57
python
def rf(xs, y, n_estimators=40, max_samples=200_000,
max_features=0.5, min_samples_leaf=5, **kwargs):
return RandomForestRegressor(n_jobs=-1, n_estimators=n_estimators,
max_samples=max_samples, max_features=max_features,
min_samples_leaf=min_samples_leaf, oob_score=True).fit(xs, y)In [ ]python · cell 58
python
m = rf(xs, y);In [ ]python · cell 59
python
m_rmse(m, xs, y), m_rmse(m, valid_xs, valid_y)In [ ]python · cell 60
python
preds = np.stack([t.predict(valid_xs) for t in m.estimators_])In [ ]python · cell 61
python
r_mse(preds.mean(0), valid_y)In [ ]python · cell 62
python
plt.plot([r_mse(preds[:i+1].mean(0), valid_y) for i in range(40)]);Out-of-Bag Error
In [ ]python · cell 64
python
r_mse(m.oob_prediction_, y)Model Interpretation
Tree Variance for Prediction Confidence
In [ ]python · cell 67
python
preds = np.stack([t.predict(valid_xs) for t in m.estimators_])In [ ]python · cell 68
python
preds.shapeIn [ ]python · cell 69
python
preds_std = preds.std(0)In [ ]python · cell 70
python
preds_std[:5]Feature Importance
In [ ]python · cell 72
python
def rf_feat_importance(m, df):
return pd.DataFrame({'cols':df.columns, 'imp':m.feature_importances_}
).sort_values('imp', ascending=False)In [ ]python · cell 73
python
fi = rf_feat_importance(m, xs)
fi[:10]In [ ]python · cell 74
python
def plot_fi(fi):
return fi.plot('cols', 'imp', 'barh', figsize=(12,7), legend=False)
plot_fi(fi[:30]);Removing Low-Importance Variables
In [ ]python · cell 76
python
to_keep = fi[fi.imp>0.005].cols
len(to_keep)In [ ]python · cell 77
python
xs_imp = xs[to_keep]
valid_xs_imp = valid_xs[to_keep]In [ ]python · cell 78
python
m = rf(xs_imp, y)In [ ]python · cell 79
python
m_rmse(m, xs_imp, y), m_rmse(m, valid_xs_imp, valid_y)In [ ]python · cell 80
python
len(xs.columns), len(xs_imp.columns)In [ ]python · cell 81
python
plot_fi(rf_feat_importance(m, xs_imp));Removing Redundant Features
In [ ]python · cell 83
python
cluster_columns(xs_imp)In [ ]python · cell 84
python
def get_oob(df):
m = RandomForestRegressor(n_estimators=40, min_samples_leaf=15,
max_samples=50000, max_features=0.5, n_jobs=-1, oob_score=True)
m.fit(df, y)
return m.oob_score_In [ ]python · cell 85
python
get_oob(xs_imp)In [ ]python · cell 86
python
{c:get_oob(xs_imp.drop(c, axis=1)) for c in (
'saleYear', 'saleElapsed', 'ProductGroupDesc','ProductGroup',
'fiModelDesc', 'fiBaseModel',
'Hydraulics_Flow','Grouser_Tracks', 'Coupler_System')}In [ ]python · cell 87
python
to_drop = ['saleYear', 'ProductGroupDesc', 'fiBaseModel', 'Grouser_Tracks']
get_oob(xs_imp.drop(to_drop, axis=1))In [ ]python · cell 88
python
xs_final = xs_imp.drop(to_drop, axis=1)
valid_xs_final = valid_xs_imp.drop(to_drop, axis=1)In [ ]python · cell 89
python
save_pickle(path/'xs_final.pkl', xs_final)
save_pickle(path/'valid_xs_final.pkl', valid_xs_final)In [ ]python · cell 90
python
xs_final = load_pickle(path/'xs_final.pkl')
valid_xs_final = load_pickle(path/'valid_xs_final.pkl')In [ ]python · cell 91
python
m = rf(xs_final, y)
m_rmse(m, xs_final, y), m_rmse(m, valid_xs_final, valid_y)Partial Dependence
In [ ]python · cell 93
python
p = valid_xs_final['ProductSize'].value_counts(sort=False).plot.barh()
c = to.classes['ProductSize']
plt.yticks(range(len(c)), c);In [ ]python · cell 94
python
ax = valid_xs_final['YearMade'].hist()In [ ]python · cell 95
python
from sklearn.inspection import plot_partial_dependence
fig,ax = plt.subplots(figsize=(12, 4))
plot_partial_dependence(m, valid_xs_final, ['YearMade','ProductSize'],
grid_resolution=20, ax=ax);Data Leakage
Tree Interpreter
In [ ]python · cell 98
python
#hide
import warnings
warnings.simplefilter('ignore', FutureWarning)
from treeinterpreter import treeinterpreter
from waterfall_chart import plot as waterfallIn [ ]python · cell 99
python
row = valid_xs_final.iloc[:5]In [ ]python · cell 100
python
prediction,bias,contributions = treeinterpreter.predict(m, row.values)In [ ]python · cell 101
python
prediction[0], bias[0], contributions[0].sum()In [ ]python · cell 102
python
waterfall(valid_xs_final.columns, contributions[0], threshold=0.08,
rotation_value=45,formatting='{:,.3f}');Extrapolation and Neural Networks
The Extrapolation Problem
In [ ]python · cell 105
python
#hide
np.random.seed(42)In [ ]python · cell 106
python
x_lin = torch.linspace(0,20, steps=40)
y_lin = x_lin + torch.randn_like(x_lin)
plt.scatter(x_lin, y_lin);In [ ]python · cell 107
python
xs_lin = x_lin.unsqueeze(1)
x_lin.shape,xs_lin.shapeIn [ ]python · cell 108
python
x_lin[:,None].shapeIn [ ]python · cell 109
python
m_lin = RandomForestRegressor().fit(xs_lin[:30],y_lin[:30])In [ ]python · cell 110
python
plt.scatter(x_lin, y_lin, 20)
plt.scatter(x_lin, m_lin.predict(xs_lin), color='red', alpha=0.5);Finding Out-of-Domain Data
In [ ]python · cell 112
python
df_dom = pd.concat([xs_final, valid_xs_final])
is_valid = np.array([0]*len(xs_final) + [1]*len(valid_xs_final))
m = rf(df_dom, is_valid)
rf_feat_importance(m, df_dom)[:6]In [ ]python · cell 113
python
m = rf(xs_final, y)
print('orig', m_rmse(m, valid_xs_final, valid_y))
for c in ('SalesID','saleElapsed','MachineID'):
m = rf(xs_final.drop(c,axis=1), y)
print(c, m_rmse(m, valid_xs_final.drop(c,axis=1), valid_y))In [ ]python · cell 114
python
time_vars = ['SalesID','MachineID']
xs_final_time = xs_final.drop(time_vars, axis=1)
valid_xs_time = valid_xs_final.drop(time_vars, axis=1)
m = rf(xs_final_time, y)
m_rmse(m, valid_xs_time, valid_y)In [ ]python · cell 115
python
xs['saleYear'].hist();In [ ]python · cell 116
python
filt = xs['saleYear']>2004
xs_filt = xs_final_time[filt]
y_filt = y[filt]In [ ]python · cell 117
python
m = rf(xs_filt, y_filt)
m_rmse(m, xs_filt, y_filt), m_rmse(m, valid_xs_time, valid_y)Using a Neural Network
In [ ]python · cell 119
python
df_nn = pd.read_csv(path/'TrainAndValid.csv', low_memory=False)
df_nn['ProductSize'] = df_nn['ProductSize'].astype('category')
df_nn['ProductSize'].cat.set_categories(sizes, ordered=True, inplace=True)
df_nn[dep_var] = np.log(df_nn[dep_var])
df_nn = add_datepart(df_nn, 'saledate')In [ ]python · cell 120
python
df_nn_final = df_nn[list(xs_final_time.columns) + [dep_var]]In [ ]python · cell 121
python
cont_nn,cat_nn = cont_cat_split(df_nn_final, max_card=9000, dep_var=dep_var)In [ ]python · cell 122
python
cont_nnIn [ ]python · cell 123
python
df_nn_final[cat_nn].nunique()In [ ]python · cell 124
python
xs_filt2 = xs_filt.drop('fiModelDescriptor', axis=1)
valid_xs_time2 = valid_xs_time.drop('fiModelDescriptor', axis=1)
m2 = rf(xs_filt2, y_filt)
m_rmse(m2, xs_filt2, y_filt), m_rmse(m2, valid_xs_time2, valid_y)In [ ]python · cell 125
python
cat_nn.remove('fiModelDescriptor')In [ ]python · cell 126
python
procs_nn = [Categorify, FillMissing, Normalize]
to_nn = TabularPandas(df_nn_final, procs_nn, cat_nn, cont_nn,
splits=splits, y_names=dep_var)In [ ]python · cell 127
python
dls = to_nn.dataloaders(1024)In [ ]python · cell 128
python
y = to_nn.train.y
y.min(),y.max()In [ ]python · cell 129
python
learn = tabular_learner(dls, y_range=(8,12), layers=[500,250],
n_out=1, loss_func=F.mse_loss)In [ ]python · cell 130
python
learn.lr_find()In [ ]python · cell 131
python
learn.fit_one_cycle(5, 1e-2)In [ ]python · cell 132
python
preds,targs = learn.get_preds()
r_mse(preds,targs)In [ ]python · cell 133
python
learn.save('nn')Sidebar: fastai's Tabular Classes
End sidebar
Ensembling
In [ ]python · cell 137
python
rf_preds = m.predict(valid_xs_time)
ens_preds = (to_np(preds.squeeze()) + rf_preds) /2In [ ]python · cell 138
python
r_mse(ens_preds,valid_y)Boosting
Combining Embeddings with Other Methods
Conclusion: Our Advice for Tabular Modeling
Questionnaire
- What is a continuous variable?
- What is a categorical variable?
- Provide two of the words that are used for the possible values of a categorical variable.
- What is a "dense layer"?
- How do entity embeddings reduce memory usage and speed up neural networks?
- What kinds of datasets are entity embeddings especially useful for?
- What are the two main families of machine learning algorithms?
- Why do some categorical columns need a special ordering in their classes? How do you do this in Pandas?
- Summarize what a decision tree algorithm does.
- Why is a date different from a regular categorical or continuous variable, and how can you preprocess it to allow it to be used in a model?
- Should you pick a random validation set in the bulldozer competition? If no, what kind of validation set should you pick?
- What is pickle and what is it useful for?
- How are
mse,samples, andvaluescalculated in the decision tree drawn in this chapter? - How do we deal with outliers, before building a decision tree?
- How do we handle categorical variables in a decision tree?
- What is bagging?
- What is the difference between
max_samplesandmax_featureswhen creating a random forest? - If you increase
n_estimatorsto a very high value, can that lead to overfitting? Why or why not? - In the section "Creating a Random Forest", just after <<max_features>>, why did
preds.mean(0)give the same result as our random forest? - What is "out-of-bag-error"?
- Make a list of reasons why a model's validation set error might be worse than the OOB error. How could you test your hypotheses?
- Explain why random forests are well suited to answering each of the following question:
- How confident are we in our predictions using a particular row of data?
- For predicting with a particular row of data, what were the most important factors, and how did they influence that prediction?
- Which columns are the strongest predictors?
- How do predictions vary as we vary these columns?
- What's the purpose of removing unimportant variables?
- What's a good type of plot for showing tree interpreter results?
- What is the "extrapolation problem"?
- How can you tell if your test or validation set is distributed in a different way than your training set?
- Why do we ensure
saleElapsedis a continuous variable, even although it has less than 9,000 distinct values? - What is "boosting"?
- How could we use embeddings with a random forest? Would we expect this to help?
- Why might we not always use a neural net for tabular modeling?
Further Research
- Pick a competition on Kaggle with tabular data (current or past) and try to adapt the techniques seen in this chapter to get the best possible results. Compare your results to the private leaderboard.
- Implement the decision tree algorithm in this chapter from scratch yourself, and try it on the dataset you used in the first exercise.
- Use the embeddings from the neural net in this chapter in a random forest, and see if you can improve on the random forest results we saw.
- Explain what each line of the source of
TabularModeldoes (with the exception of theBatchNorm1dandDropoutlayers).
In [ ]python · cell 146
python