Chapter 30

2. Machine Learning for Regression

NotebookPython 3130 cells

2. Machine Learning for Regression

In [1]python · cell 2

python

import pandas as pd
import numpy as np

2.2 Data preparation

In [2]python · cell 4

python

data = 'https://raw.githubusercontent.com/alexeygrigorev/mlbookcamp-code/master/chapter-02-car-price/data.csv'

In [5]python · cell 5

python

!wget $data

Output

--2021-09-18 22:31:04--  https://raw.githubusercontent.com/alexeygrigorev/mlbookcamp-code/master/chapter-02-car-price/data.csv
Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.111.133, 185.199.110.133, 185.199.109.133, ...
Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.111.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1475504 (1,4M) [text/plain]
Saving to: ‘data.csv’

data.csv            100%[===================>]   1,41M  9,27MB/s    in 0,2s    

2021-09-18 22:31:04 (9,27 MB/s) - ‘data.csv’ saved [1475504/1475504]

In [5]python · cell 6

python

df = pd.read_csv('data.csv')

In [6]python · cell 7

python

df.columns = df.columns.str.lower().str.replace(' ', '_')

In [7]python · cell 8

python

df['make'].str.lower().str.replace(' ', '_')

Output

0            bmw
1            bmw
2            bmw
3            bmw
4            bmw
          ...   
11909      acura
11910      acura
11911      acura
11912      acura
11913    lincoln
Name: make, Length: 11914, dtype: object

In [8]python · cell 9

python

strings = list(df.dtypes[df.dtypes == 'object'].index)
strings

Output

['make',
 'model',
 'engine_fuel_type',
 'transmission_type',
 'driven_wheels',
 'market_category',
 'vehicle_size',
 'vehicle_style']

In [9]python · cell 10

python

for col in strings:
    df[col] = df[col].str.lower().str.replace(' ', '_')

In [10]python · cell 11

python

df.dtypes

Output

make                  object
model                 object
year                   int64
engine_fuel_type      object
engine_hp            float64
engine_cylinders     float64
transmission_type     object
driven_wheels         object
number_of_doors      float64
market_category       object
vehicle_size          object
vehicle_style         object
highway_mpg            int64
city_mpg               int64
popularity             int64
msrp                   int64
dtype: object

2.3 Exploratory data analysis

In [11]python · cell 13

python

for col in df.columns:
    print(col)
    print(df[col].unique()[:5])
    print(df[col].nunique())
    print()

Output

make
['bmw' 'audi' 'fiat' 'mercedes-benz' 'chrysler']
48

model
['1_series_m' '1_series' '100' '124_spider' '190-class']
914

year
[2011 2012 2013 1992 1993]
28

engine_fuel_type
['premium_unleaded_(required)' 'regular_unleaded'
 'premium_unleaded_(recommended)' 'flex-fuel_(unleaded/e85)' 'diesel']
10

engine_hp
[335. 300. 230. 320. 172.]
356

engine_cylinders
[ 6.  4.  5.  8. 12.]
9

transmission_type
['manual' 'automatic' 'automated_manual' 'direct_drive' 'unknown']
5

driven_wheels
['rear_wheel_drive' 'front_wheel_drive' 'all_wheel_drive'
 'four_wheel_drive']
4

number_of_doors
[ 2.  4.  3. nan]
3

market_category
['factory_tuner,luxury,high-performance' 'luxury,performance'
 'luxury,high-performance' 'luxury' 'performance']
71

vehicle_size
['compact' 'midsize' 'large']
3

vehicle_style
['coupe' 'convertible' 'sedan' 'wagon' '4dr_hatchback']
16

highway_mpg
[26 28 27 25 24]
59

city_mpg
[19 20 18 17 16]
69

popularity
[3916 3105  819  617 1013]
48

msrp
[46135 40650 36350 29450 34500]
6049

In [12]python · cell 14

python

df

Output

          make       model  year                engine_fuel_type  engine_hp  \
0          bmw  1_series_m  2011     premium_unleaded_(required)      335.0   
1          bmw    1_series  2011     premium_unleaded_(required)      300.0   
2          bmw    1_series  2011     premium_unleaded_(required)      300.0   
3          bmw    1_series  2011     premium_unleaded_(required)      230.0   
4          bmw    1_series  2011     premium_unleaded_(required)      230.0   
...        ...         ...   ...                             ...        ...   
11909    acura         zdx  2012     premium_unleaded_(required)      300.0   
11910    acura         zdx  2012     premium_unleaded_(required)      300.0   
11911    acura         zdx  2012     premium_unleaded_(required)      300.0   
11912    acura         zdx  2013  premium_unleaded_(recommended)      300.0   
11913  lincoln      zephyr  2006                regular_unleaded      221.0   

       engine_cylinders transmission_type      driven_wheels  number_of_doors  \
0                   6.0            manual   rear_wheel_drive              2.0   
1                   6.0            manual   rear_wheel_drive              2.0   
2                   6.0            manual   rear_wheel_drive              2.0   
3                   6.0            manual   rear_wheel_drive              2.0   
4                   6.0            manual   rear_wheel_drive              2.0   
...                 ...               ...                ...              ...   
11909               6.0         automatic    all_wheel_drive              4.0   
11910               6.0         automatic    all_wheel_drive              4.0   
11911               6.0         automatic    all_wheel_drive              4.0   
11912               6.0         automatic    all_wheel_drive              4.0   
11913               6.0         automatic  front_wheel_drive              4.0   

                             market_category vehicle_size  vehicle_style  \
0      factory_tuner,luxury,high-performance      compact          coupe   
1                         luxury,performance      compact    convertible   
2                    luxury,high-performance      compact          coupe   
3                         luxury,performance      compact          coupe   
4                                     luxury      compact    convertible   
...                                      ...          ...            ...   
11909             crossover,hatchback,luxury      midsize  4dr_hatchback   
11910             crossover,hatchback,luxury      midsize  4dr_hatchback   
11911             crossover,hatchback,luxury      midsize  4dr_hatchback   
11912             crossover,hatchback,luxury      midsize  4dr_hatchback   
11913                                 luxury      midsize          sedan   

       highway_mpg  city_mpg  popularity   msrp  
0               26        19        3916  46135  
1               28        19        3916  40650  
2               28        20        3916  36350  
3               28        18        3916  29450  
4               28        18        3916  34500  
...            ...       ...         ...    ...  
11909           23        16         204  46120  
11910           23        16         204  56670  
11911           23        16         204  50620  
11912           23        16         204  50920  
11913           26        17          61  28995  

[11914 rows x 16 columns]

.dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; }

	make	model	year	engine_fuel_type	engine_hp	engine_cylinders	transmission_type	driven_wheels	number_of_doors	market_category	vehicle_size	vehicle_style	highway_mpg	city_mpg	popularity	msrp
0	bmw	1_series_m	2011	premium_unleaded_(required)	335.0	6.0	manual	rear_wheel_drive	2.0	factory_tuner,luxury,high-performance	compact	coupe	26	19	3916	46135
1	bmw	1_series	2011	premium_unleaded_(required)	300.0	6.0	manual	rear_wheel_drive	2.0	luxury,performance	compact	convertible	28	19	3916	40650
2	bmw	1_series	2011	premium_unleaded_(required)	300.0	6.0	manual	rear_wheel_drive	2.0	luxury,high-performance	compact	coupe	28	20	3916	36350
3	bmw	1_series	2011	premium_unleaded_(required)	230.0	6.0	manual	rear_wheel_drive	2.0	luxury,performance	compact	coupe	28	18	3916	29450
4	bmw	1_series	2011	premium_unleaded_(required)	230.0	6.0	manual	rear_wheel_drive	2.0	luxury	compact	convertible	28	18	3916	34500
...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...
11909	acura	zdx	2012	premium_unleaded_(required)	300.0	6.0	automatic	all_wheel_drive	4.0	crossover,hatchback,luxury	midsize	4dr_hatchback	23	16	204	46120
11910	acura	zdx	2012	premium_unleaded_(required)	300.0	6.0	automatic	all_wheel_drive	4.0	crossover,hatchback,luxury	midsize	4dr_hatchback	23	16	204	56670
11911	acura	zdx	2012	premium_unleaded_(required)	300.0	6.0	automatic	all_wheel_drive	4.0	crossover,hatchback,luxury	midsize	4dr_hatchback	23	16	204	50620
11912	acura	zdx	2013	premium_unleaded_(recommended)	300.0	6.0	automatic	all_wheel_drive	4.0	crossover,hatchback,luxury	midsize	4dr_hatchback	23	16	204	50920
11913	lincoln	zephyr	2006	regular_unleaded	221.0	6.0	automatic	front_wheel_drive	4.0	luxury	midsize	sedan	26	17	61	28995

11914 rows × 16 columns

Distribution of price

In [13]python · cell 16

python

import matplotlib.pyplot as plt
import seaborn as sns

%matplotlib inline

In [14]python · cell 17

python

sns.histplot(df.msrp, bins=50)

Output

<AxesSubplot:xlabel='msrp', ylabel='Count'>

<Figure size 432x288 with 1 Axes>

In [15]python · cell 18

python

sns.histplot(df.msrp[df.msrp < 100000], bins=50)

Output

<AxesSubplot:xlabel='msrp', ylabel='Count'>

<Figure size 432x288 with 1 Axes>

In [16]python · cell 19

python

np.log1p([0, 1, 10, 1000, 100000])

Output

array([ 0.        ,  0.69314718,  2.39789527,  6.90875478, 11.51293546])

In [17]python · cell 20

python

np.log([0 + 1, 1+ 1, 10 + 1, 1000 + 1, 100000])

Output

array([ 0.        ,  0.69314718,  2.39789527,  6.90875478, 11.51292546])

In [18]python · cell 21

python

price_logs = np.log1p(df.msrp)

In [19]python · cell 22

python


sns.histplot(price_logs, bins=50)

Output

<AxesSubplot:xlabel='msrp', ylabel='Count'>

<Figure size 432x288 with 1 Axes>

Missing values

In [20]python · cell 24

python

df.isnull().sum()

Output

make                    0
model                   0
year                    0
engine_fuel_type        3
engine_hp              69
engine_cylinders       30
transmission_type       0
driven_wheels           0
number_of_doors         6
market_category      3742
vehicle_size            0
vehicle_style           0
highway_mpg             0
city_mpg                0
popularity              0
msrp                    0
dtype: int64

2.4 Setting up the validation framework

Let's draw it

In [21]python · cell 27

python

n = len(df)

n_val = int(n * 0.2)
n_test = int(n * 0.2)
n_train = n - n_val - n_test

In [22]python · cell 28

python

Output

In [23]python · cell 29

python

n_val, n_test, n_train

Output

(2382, 2382, 7150)

In [24]python · cell 30

python

df.iloc[[10, 0, 3, 5]]

Output

   make       model  year             engine_fuel_type  engine_hp  \
10  bmw    1_series  2013  premium_unleaded_(required)      300.0   
0   bmw  1_series_m  2011  premium_unleaded_(required)      335.0   
3   bmw    1_series  2011  premium_unleaded_(required)      230.0   
5   bmw    1_series  2012  premium_unleaded_(required)      230.0   

    engine_cylinders transmission_type     driven_wheels  number_of_doors  \
10               6.0            manual  rear_wheel_drive              2.0   
0                6.0            manual  rear_wheel_drive              2.0   
3                6.0            manual  rear_wheel_drive              2.0   
5                6.0            manual  rear_wheel_drive              2.0   

                          market_category vehicle_size vehicle_style  \
10                luxury,high-performance      compact         coupe   
0   factory_tuner,luxury,high-performance      compact         coupe   
3                      luxury,performance      compact         coupe   
5                      luxury,performance      compact         coupe   

    highway_mpg  city_mpg  popularity   msrp  
10           28        20        3916  39600  
0            26        19        3916  46135  
3            28        18        3916  29450  
5            28        18        3916  31200

.dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; }

	make	model	year	engine_fuel_type	engine_hp	engine_cylinders	transmission_type	driven_wheels	number_of_doors	market_category	vehicle_size	vehicle_style	highway_mpg	city_mpg	popularity	msrp
10	bmw	1_series	2013	premium_unleaded_(required)	300.0	6.0	manual	rear_wheel_drive	2.0	luxury,high-performance	compact	coupe	28	20	3916	39600
0	bmw	1_series_m	2011	premium_unleaded_(required)	335.0	6.0	manual	rear_wheel_drive	2.0	factory_tuner,luxury,high-performance	compact	coupe	26	19	3916	46135
3	bmw	1_series	2011	premium_unleaded_(required)	230.0	6.0	manual	rear_wheel_drive	2.0	luxury,performance	compact	coupe	28	18	3916	29450
5	bmw	1_series	2012	premium_unleaded_(required)	230.0	6.0	manual	rear_wheel_drive	2.0	luxury,performance	compact	coupe	28	18	3916	31200

In [25]python · cell 31

python

df_train = df.iloc[:n_train]
df_val = df.iloc[n_train:n_train+n_val]
df_test = df.iloc[n_train+n_val:]

In [26]python · cell 32

python

idx = np.arange(n)

In [27]python · cell 33

python

np.random.seed(2)
np.random.shuffle(idx)

In [28]python · cell 34

python

df_train = df.iloc[idx[:n_train]]
df_val = df.iloc[idx[n_train:n_train+n_val]]
df_test = df.iloc[idx[n_train+n_val:]]

In [29]python · cell 35

python

df_train.head()

Output

             make    model  year          engine_fuel_type  engine_hp  \
2735    chevrolet   cobalt  2008          regular_unleaded      148.0   
6720       toyota   matrix  2012          regular_unleaded      132.0   
5878       subaru  impreza  2016          regular_unleaded      148.0   
11190  volkswagen  vanagon  1991          regular_unleaded       90.0   
4554         ford    f-150  2017  flex-fuel_(unleaded/e85)      385.0   

       engine_cylinders transmission_type      driven_wheels  number_of_doors  \
2735                4.0            manual  front_wheel_drive              2.0   
6720                4.0         automatic  front_wheel_drive              4.0   
5878                4.0         automatic    all_wheel_drive              4.0   
11190               4.0            manual   rear_wheel_drive              3.0   
4554                8.0         automatic   four_wheel_drive              4.0   

      market_category vehicle_size      vehicle_style  highway_mpg  city_mpg  \
2735              NaN      compact              coupe           33        24   
6720        hatchback      compact      4dr_hatchback           32        25   
5878        hatchback      compact      4dr_hatchback           37        28   
11190             NaN        large  passenger_minivan           18        16   
4554        flex_fuel        large    crew_cab_pickup           21        15   

       popularity   msrp  
2735         1385  14410  
6720         2031  19685  
5878          640  19795  
11190         873   2000  
4554         5657  56260

.dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; }

	make	model	year	engine_fuel_type	engine_hp	engine_cylinders	transmission_type	driven_wheels	number_of_doors	market_category	vehicle_size	vehicle_style	highway_mpg	city_mpg	popularity	msrp
2735	chevrolet	cobalt	2008	regular_unleaded	148.0	4.0	manual	front_wheel_drive	2.0	NaN	compact	coupe	33	24	1385	14410
6720	toyota	matrix	2012	regular_unleaded	132.0	4.0	automatic	front_wheel_drive	4.0	hatchback	compact	4dr_hatchback	32	25	2031	19685
5878	subaru	impreza	2016	regular_unleaded	148.0	4.0	automatic	all_wheel_drive	4.0	hatchback	compact	4dr_hatchback	37	28	640	19795
11190	volkswagen	vanagon	1991	regular_unleaded	90.0	4.0	manual	rear_wheel_drive	3.0	NaN	large	passenger_minivan	18	16	873	2000
4554	ford	f-150	2017	flex-fuel_(unleaded/e85)	385.0	8.0	automatic	four_wheel_drive	4.0	flex_fuel	large	crew_cab_pickup	21	15	5657	56260

In [30]python · cell 36

python

len(df_train), len(df_val), len(df_test)

Output

(7150, 2382, 2382)

In [31]python · cell 37

python

df_train = df_train.reset_index(drop=True)
df_val = df_val.reset_index(drop=True)
df_test = df_test.reset_index(drop=True)

In [32]python · cell 38

python

y_train = np.log1p(df_train.msrp.values)
y_val = np.log1p(df_val.msrp.values)
y_test = np.log1p(df_test.msrp.values)

In [33]python · cell 39

python

del df_train['msrp']
del df_val['msrp']
del df_test['msrp']

In [34]python · cell 40

python

len(y_train)

Output

2.5 Linear regression

draw

In [35]python · cell 43

python

df_train.iloc[10]

Output

make                                 rolls-royce
model                     phantom_drophead_coupe
year                                        2015
engine_fuel_type     premium_unleaded_(required)
engine_hp                                  453.0
engine_cylinders                            12.0
transmission_type                      automatic
driven_wheels                   rear_wheel_drive
number_of_doors                              2.0
market_category        exotic,luxury,performance
vehicle_size                               large
vehicle_style                        convertible
highway_mpg                                   19
city_mpg                                      11
popularity                                    86
Name: 10, dtype: object

In [ ]python · cell 44

python

In [36]python · cell 45

python

xi = [453, 11, 86]
w0 = 7.17
w = [0.01, 0.04, 0.002]

In [37]python · cell 46

python

def linear_regression(xi):
    n = len(xi)

    pred = w0

    for j in range(n):
        pred = pred + w[j] * xi[j]

    return pred

In [38]python · cell 47

python

xi = [453, 11, 86]
w0 = 7.17
w = [0.01, 0.04, 0.002]

In [39]python · cell 48

python

linear_regression(xi)

Output

12.312

In [40]python · cell 49

python

np.expm1(12.312)

Output

222347.2221101062

In [41]python · cell 50

python

np.log1p(222347.2221101062)

Output

12.312

2.6 Linear regression vector form

In [42]python · cell 52

python

def dot(xi, w):
    n = len(xi)
    
    res = 0.0
    
    for j in range(n):
        res = res + xi[j] * w[j]
    
    return res

In [43]python · cell 53

python

def linear_regression(xi):
    return w0 + dot(xi, w)

In [44]python · cell 54

python

w_new = [w0] + w

In [45]python · cell 55

python

w_new

Output

[7.17, 0.01, 0.04, 0.002]

In [46]python · cell 56

python

def linear_regression(xi):
    xi = [1] + xi
    return dot(xi, w_new)

In [47]python · cell 57

python

linear_regression(xi)

Output

12.312

In [48]python · cell 58

python

w0 = 7.17
w = [0.01, 0.04, 0.002]
w_new = [w0] + w

In [49]python · cell 59

python

x1  = [1, 148, 24, 1385]
x2  = [1, 132, 25, 2031]
x10 = [1, 453, 11, 86]

X = [x1, x2, x10]
X = np.array(X)
X

Output

array([[   1,  148,   24, 1385],
       [   1,  132,   25, 2031],
       [   1,  453,   11,   86]])

In [50]python · cell 60

python

def linear_regression(X):
    return X.dot(w_new)

In [51]python · cell 61

python

linear_regression(X)

Output

array([12.38 , 13.552, 12.312])

2.7 Training a linear regression model

In [52]python · cell 63

python

def train_linear_regression(X, y):
    pass

In [53]python · cell 64

python

X = [
    [148, 24, 1385],
    [132, 25, 2031],
    [453, 11, 86],
    [158, 24, 185],
    [172, 25, 201],
    [413, 11, 86],
    [38,  54, 185],
    [142, 25, 431],
    [453, 31, 86],
]

X = np.array(X)
X

Output

array([[ 148,   24, 1385],
       [ 132,   25, 2031],
       [ 453,   11,   86],
       [ 158,   24,  185],
       [ 172,   25,  201],
       [ 413,   11,   86],
       [  38,   54,  185],
       [ 142,   25,  431],
       [ 453,   31,   86]])

In [54]python · cell 65

python

ones = np.ones(X.shape[0])
ones

Output

array([1., 1., 1., 1., 1., 1., 1., 1., 1.])

In [55]python · cell 66

python

X = np.column_stack([ones, X])

In [56]python · cell 67

python

y = [10000, 20000, 15000, 20050, 10000, 20000, 15000, 25000, 12000]

In [57]python · cell 68

python

XTX = X.T.dot(X)
XTX_inv = np.linalg.inv(XTX)
w_full = XTX_inv.dot(X.T).dot(y)

In [ ]python · cell 69

python

In [ ]python · cell 70

python

In [58]python · cell 71

python

w0 = w_full[0]
w = w_full[1:]

In [59]python · cell 72

python

w0, w

Output

(25844.75405576683, array([ -16.08906468, -199.47254894,   -1.22802883]))

In [60]python · cell 73

python

def train_linear_regression(X, y):
    ones = np.ones(X.shape[0])
    X = np.column_stack([ones, X])

    XTX = X.T.dot(X)
    XTX_inv = np.linalg.inv(XTX)
    w_full = XTX_inv.dot(X.T).dot(y)
    
    return w_full[0], w_full[1:]

In [61]python · cell 74

python

train_linear_regression(X, y)

Output

(2.0460143454602068e+20,
 array([1.26525728e+20, 8.71025484e+01, 2.59709297e+02, 0.00000000e+00]))

2.8 Car price baseline model

In [62]python · cell 76

python

df_train.columns

Output

Index(['make', 'model', 'year', 'engine_fuel_type', 'engine_hp',
       'engine_cylinders', 'transmission_type', 'driven_wheels',
       'number_of_doors', 'market_category', 'vehicle_size', 'vehicle_style',
       'highway_mpg', 'city_mpg', 'popularity'],
      dtype='object')

In [63]python · cell 77

python

base = ['engine_hp', 'engine_cylinders', 'highway_mpg',
        'city_mpg', 'popularity']

X_train = df_train[base].fillna(0).values

w0, w = train_linear_regression(X_train, y_train)

y_pred = w0 + X_train.dot(w)

In [64]python · cell 78

python

w0

Output

7.9272573880700605

In [65]python · cell 79

python

Output

array([ 9.70589522e-03, -1.59103494e-01,  1.43792133e-02,  1.49441072e-02,
       -9.06908672e-06])

In [ ]python · cell 80

python

In [66]python · cell 81

python

sns.histplot(y_pred, color='red', alpha=0.5, bins=50)
sns.histplot(y_train, color='blue', alpha=0.5, bins=50)

Output

<AxesSubplot:ylabel='Count'>

<Figure size 432x288 with 1 Axes>

2.9 RMSE

In [67]python · cell 83

python

def rmse(y, y_pred):
    se = (y - y_pred) ** 2
    mse = se.mean()
    return np.sqrt(mse)

In [68]python · cell 84

python

rmse(y_train, y_pred)

Output

0.7554192603920132

2.10 Validating the model

In [69]python · cell 86

python

def prepare_X(df):
    df_num = df[base]
    df_num = df_num.fillna(0)
    X = df_num.values
    return X

In [70]python · cell 87

python

X_train = prepare_X(df_train)
w0, w = train_linear_regression(X_train, y_train)

X_val = prepare_X(df_val)
y_pred = w0 + X_val.dot(w)
rmse(y_val, y_pred)

Output

0.7616530991301577

2.11 Simple feature engineering

In [71]python · cell 89

python

def prepare_X(df):
    df = df.copy()
    
    df['age'] = 2017 - df['year']
    features = base + ['age']
    
    df_num = df[features]
    df_num = df_num.fillna(0)
    X = df_num.values

    return X

In [72]python · cell 90

python

X_train = prepare_X(df_train)
w0, w = train_linear_regression(X_train, y_train)

X_val = prepare_X(df_val)
y_pred = w0 + X_val.dot(w)
rmse(y_val, y_pred)

Output

0.5172055461058291

In [73]python · cell 91

python

sns.histplot(y_pred, label='prediction', color='red', alpha=0.5, bins=50)
sns.histplot(y_val, label='target', color='blue',  alpha=0.5, bins=50)
plt.legend()

Output

<matplotlib.legend.Legend at 0xff7e413fe3a0>

<Figure size 432x288 with 1 Axes>

2.12 Categorical variables

In [74]python · cell 93

python

categorical_columns = [
    'make', 'model', 'engine_fuel_type', 'driven_wheels', 'market_category',
    'vehicle_size', 'vehicle_style']

categorical = {}

for c in categorical_columns:
    categorical[c] = list(df_train[c].value_counts().head().index)

In [75]python · cell 94

python

def prepare_X(df):
    df = df.copy()
    
    df['age'] = 2017 - df['year']
    features = base + ['age']

    for v in [2, 3, 4]:
        df['num_doors_%d' % v] = (df.number_of_doors == v).astype(int)
        features.append('num_doors_%d' % v)

    for name, values in categorical.items():
        for value in values:
            df['%s_%s' % (name, value)] = (df[name] == value).astype(int)
            features.append('%s_%s' % (name, value))

    df_num = df[features]
    df_num = df_num.fillna(0)
    X = df_num.values

    return X

In [76]python · cell 95

python

X_train = prepare_X(df_train)
w0, w = train_linear_regression(X_train, y_train)

X_val = prepare_X(df_val)
y_pred = w0 + X_val.dot(w)
rmse(y_val, y_pred)

Output

266.5994263627081

In [77]python · cell 96

python

w0, w

Output

(3066759258862621.0,
 array([-7.90863204e-01,  1.80621909e+02,  1.97903973e+01,  1.46968041e+01,
        -3.39713455e-04,  7.74924546e+00,  9.76297867e+03,  9.83480038e+03,
         9.77003629e+03, -2.36936868e+01, -6.98582110e+00,  7.93435108e+01,
        -7.33167695e+01, -6.60897799e+00, -1.00310530e+01,  7.79383982e+01,
        -4.19367134e+01, -6.14213575e+01,  2.12412304e+02,  1.02166851e+03,
         9.08719092e+02,  9.84633740e+02,  1.06771237e+03,  7.00547494e+02,
        -3.06675926e+15, -3.06675926e+15, -3.06675926e+15, -3.06675926e+15,
         2.52356626e+00,  3.40712703e+00, -1.27416249e+01,  1.22844560e+01,
        -1.51620939e+01,  3.09315910e+02,  2.88692636e+02,  2.78158171e+02,
        -1.44115660e-01, -2.62579827e-02,  1.75913981e-01,  3.65037816e-01,
        -2.90235596e-01]))

2.13 Regularization

In [78]python · cell 98

python

X = [
    [4, 4, 4],
    [3, 5, 5],
    [5, 1, 1],
    [5, 4, 4],
    [7, 5, 5],
    [4, 5, 5.00000001],
]

X = np.array(X)
X

Output

array([[4.        , 4.        , 4.        ],
       [3.        , 5.        , 5.        ],
       [5.        , 1.        , 1.        ],
       [5.        , 4.        , 4.        ],
       [7.        , 5.        , 5.        ],
       [4.        , 5.        , 5.00000001]])

In [79]python · cell 99

python

y= [1, 2, 3, 1, 2, 3]

In [80]python · cell 100

python

XTX = X.T.dot(X)
XTX

Output

array([[140.        , 111.        , 111.00000004],
       [111.        , 108.        , 108.00000005],
       [111.00000004, 108.00000005, 108.0000001 ]])

In [81]python · cell 101

python

XTX_inv = np.linalg.inv(XTX)

In [82]python · cell 102

python

XTX_inv

Output

array([[ 3.85321698e-02,  1.20696663e+05, -1.20696696e+05],
       [ 1.20696640e+05, -2.74658839e+14,  2.74658839e+14],
       [-1.20696680e+05,  2.74658839e+14, -2.74658839e+14]])

In [83]python · cell 103

python

XTX_inv.dot(X.T).dot(y)

Output

array([ 6.24269892e-01,  3.44329390e+06, -3.44329299e+06])

In [84]python · cell 104

python

XTX = [
    [1, 2, 2],
    [2, 1, 1.0000001],
    [2, 1.0000001, 1]
]

XTX = np.array(XTX)

In [85]python · cell 105

python

np.linalg.inv(XTX)

Output

array([[-3.33333356e-01,  3.33333339e-01,  3.33333339e-01],
       [ 3.33333339e-01, -5.00000008e+06,  4.99999991e+06],
       [ 3.33333339e-01,  4.99999991e+06, -5.00000008e+06]])

In [86]python · cell 106

python

XTX = XTX + 0.01 * np.eye(3)

In [87]python · cell 107

python

np.linalg.inv(XTX)

Output

array([[ -0.33668908,   0.33501399,   0.33501399],
       [  0.33501399,  49.91590897, -50.08509104],
       [  0.33501399, -50.08509104,  49.91590897]])

In [88]python · cell 108

python

def train_linear_regression_reg(X, y, r=0.001):
    ones = np.ones(X.shape[0])
    X = np.column_stack([ones, X])

    XTX = X.T.dot(X)
    XTX = XTX + r * np.eye(XTX.shape[0])

    XTX_inv = np.linalg.inv(XTX)
    w_full = XTX_inv.dot(X.T).dot(y)
    
    return w_full[0], w_full[1:]

In [89]python · cell 109

python

X_train = prepare_X(df_train)
w0, w = train_linear_regression_reg(X_train, y_train, r=0.01)

X_val = prepare_X(df_val)
y_pred = w0 + X_val.dot(w)
rmse(y_val, y_pred)

Output

0.4608208286209523

2.14 Tuning the model

In [90]python · cell 111

python

for r in [0.0, 0.00001, 0.0001, 0.001, 0.1, 1, 10]:
    X_train = prepare_X(df_train)
    w0, w = train_linear_regression_reg(X_train, y_train, r=r)

    X_val = prepare_X(df_val)
    y_pred = w0 + X_val.dot(w)
    score = rmse(y_val, y_pred)
    
    print(r, w0, score)

Output

0.0 3066759258862621.0 266.5994263627081
1e-05 6.6398149018651305 0.46081532123609276
0.0001 7.12965014007955 0.4608153659783837
0.001 7.13085491393288 0.46081585838957173
0.1 7.0002324128629025 0.4608736549134577
1 6.250747846974607 0.46158128382779856
10 4.7295125856710305 0.472609877266903

In [91]python · cell 112

python

r = 0.001
X_train = prepare_X(df_train)
w0, w = train_linear_regression_reg(X_train, y_train, r=r)

X_val = prepare_X(df_val)
y_pred = w0 + X_val.dot(w)
score = rmse(y_val, y_pred)
score

Output

0.46081585838957173

2.15 Using the model

In [92]python · cell 114

python

df_full_train = pd.concat([df_train, df_val])

In [93]python · cell 115

python

df_full_train = df_full_train.reset_index(drop=True)

In [94]python · cell 116

python

X_full_train = prepare_X(df_full_train)

In [95]python · cell 117

python

X_full_train

Output

array([[148.,   4.,  33., ...,   1.,   0.,   0.],
       [132.,   4.,  32., ...,   0.,   0.,   1.],
       [148.,   4.,  37., ...,   0.,   0.,   1.],
       ...,
       [332.,   8.,  23., ...,   0.,   0.,   0.],
       [148.,   4.,  34., ...,   0.,   0.,   0.],
       [290.,   6.,  25., ...,   0.,   0.,   0.]])

In [96]python · cell 118

python

y_full_train = np.concatenate([y_train, y_val])

In [97]python · cell 119

python

w0, w = train_linear_regression_reg(X_full_train, y_full_train, r=0.001)

In [98]python · cell 120

python

X_test = prepare_X(df_test)
y_pred = w0 + X_test.dot(w)
score = rmse(y_test, y_pred)
score

Output

0.4600753970266562

In [99]python · cell 121

python

car = df_test.iloc[20].to_dict()
car

Output

{'make': 'toyota',
 'model': 'sienna',
 'year': 2015,
 'engine_fuel_type': 'regular_unleaded',
 'engine_hp': 266.0,
 'engine_cylinders': 6.0,
 'transmission_type': 'automatic',
 'driven_wheels': 'front_wheel_drive',
 'number_of_doors': 4.0,
 'market_category': nan,
 'vehicle_size': 'large',
 'vehicle_style': 'passenger_minivan',
 'highway_mpg': 25,
 'city_mpg': 18,
 'popularity': 2031}

In [100]python · cell 122

python

df_small = pd.DataFrame([car])
df_small

Output

     make   model  year  engine_fuel_type  engine_hp  engine_cylinders  \
0  toyota  sienna  2015  regular_unleaded      266.0               6.0   

  transmission_type      driven_wheels  number_of_doors  market_category  \
0         automatic  front_wheel_drive              4.0              NaN   

  vehicle_size      vehicle_style  highway_mpg  city_mpg  popularity  
0        large  passenger_minivan           25        18        2031

.dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; }

	make	model	year	engine_fuel_type	engine_hp	engine_cylinders	transmission_type	driven_wheels	number_of_doors	market_category	vehicle_size	vehicle_style	highway_mpg	city_mpg	popularity
0	toyota	sienna	2015	regular_unleaded	266.0	6.0	automatic	front_wheel_drive	4.0	NaN	large	passenger_minivan	25	18	2031

In [101]python · cell 123

python

X_small = prepare_X(df_small)

In [102]python · cell 124

python

y_pred = w0 + X_small.dot(w)
y_pred = y_pred[0]
y_pred

Output

10.63249250912739

In [103]python · cell 125

python

np.expm1(y_pred)

Output

41459.336786653585

In [104]python · cell 126

python

np.expm1(y_test[20])

Output

35000.00000000001

2.16 Next steps

We included only 5 top features. What happens if we include 10?

Other projects

Predict the price of a house - e.g. boston dataset
https://archive.ics.uci.edu/ml/datasets.php?task=reg
https://archive.ics.uci.edu/ml/datasets/Student+Performance

2.17 Summary

EDA - looking at data, finding missing values
Target variable distribution - long tail => bell shaped curve
Validation framework: train/val/test split (helped us detect problems)
Normal equation - not magic, but math
Implemented it with numpy
RMSE to validate our model
Feature engineering: age, categorical features
Regularization to fight numerical instability

In [ ]python · cell 130

python

Chapter 30

2. Machine Learning for Regression

NotebookPython 3130 cells