Chapter 36
Data preparation
NotebookPython 3 (ipykernel)32 cells
In [2]python · cell 1
python
%matplotlib inline
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as snsIn [3]python · cell 2
python
!wget https://raw.githubusercontent.com/alexeygrigorev/datasets/master/AB_NYC_2019.csv -O AB_NYC_2019.csvOutput
--2021-09-27 17:03:29-- https://raw.githubusercontent.com/alexeygrigorev/datasets/master/AB_NYC_2019.csv Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.108.133, 185.199.109.133, 185.199.110.133, ... Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.108.133|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 7077973 (6.8M) [text/plain] Saving to: ‘AB_NYC_2019.csv’ AB_NYC_2019.csv 100%[===================>] 6.75M 7.92MB/s in 0.9s 2021-09-27 17:03:30 (7.92 MB/s) - ‘AB_NYC_2019.csv’ saved [7077973/7077973]
Data preparation
In [3]python · cell 4
python
usecols = [
'room_type', 'neighbourhood_group',
'latitude', 'longitude', 'price','minimum_nights',
'number_of_reviews', 'reviews_per_month',
'calculated_host_listings_count', 'availability_365'
]
df = pd.read_csv('AB_NYC_2019.csv', usecols=usecols)In [4]python · cell 5
python
df['reviews_per_month'] = df.reviews_per_month.fillna(0)In [5]python · cell 6
python
df['price'] = df['price'] >= 152In [8]python · cell 7
python
from sklearn.model_selection import train_test_splitIn [9]python · cell 8
python
df_train_full, df_test = train_test_split(df, test_size=0.2, random_state=42)
df_train, df_val = train_test_split(df_train_full, test_size=0.25, random_state=42)
df_train = df_train.reset_index(drop=True)
df_val = df_val.reset_index(drop=True)
df_test = df_test.reset_index(drop=True)
y_train = df_train.price.values
y_val = df_val.price.values
y_test = df_test.price.values
del df_train['price']
del df_val['price']
del df_test['price']In [11]python · cell 9
python
from sklearn.feature_extraction import DictVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_scoreIn [12]python · cell 10
python
cat = ['neighbourhood_group', 'room_type']
num = [
'latitude', 'longitude', 'minimum_nights', 'number_of_reviews',
'reviews_per_month', 'calculated_host_listings_count',
'availability_365'
]Training the model
You get a convergence warning:
In [13]python · cell 12
python
train_dict = df_train[cat + num].to_dict(orient='records')
dv = DictVectorizer(sparse=False)
dv.fit(train_dict)
X_train = dv.transform(train_dict)
model = LogisticRegression(solver='lbfgs', C=1.0)
model.fit(X_train, y_train)Output
/home/alexey/.pyenv/versions/3.8.11/lib/python3.8/site-packages/sklearn/linear_model/_logistic.py:763: ConvergenceWarning: lbfgs failed to converge (status=1):
STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.
Increase the number of iterations (max_iter) or scale the data as shown in:
https://scikit-learn.org/stable/modules/preprocessing.html
Please also refer to the documentation for alternative solver options:
https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression
n_iter_i = _check_optimize_result(
LogisticRegression()
We can fix this model by using a scaler. You can read more about scalers here.
Also, we'll show you how to use OneHotEncoding instead of DictVectorizer
Feature scaling + OHE
In [14]python · cell 15
python
from sklearn.preprocessing import StandardScaler
from sklearn.preprocessing import MinMaxScaler
from sklearn.preprocessing import OneHotEncoderFirst, we prepare the numerical variables. We'll use the scaler for that
and write the results to X_train_num:
In [21]python · cell 17
python
X_train_num = df_train[num].values
scaler = StandardScaler()
#scaler = MinMaxScaler()
X_train_num = scaler.fit_transform(X_train_num)The scaler scales the numerical features. Compare the un-scaled version of latitude with the scaled one:
In [22]python · cell 19
python
df_train.latitude.valuesOutput
array([40.7276 , 40.70847, 40.83149, ..., 40.79994, 40.69585, 40.64438])
In [23]python · cell 20
python
X_train_num[:, 0]Output
array([-0.02524398, -0.37616878, 1.88053632, ..., 1.3017764 ,
-0.60767275, -1.5518494 ])Now let's process categorical features using OneHotEncoding.
We'll write the results to X_train_cat:
In [24]python · cell 22
python
ohe = OneHotEncoder(sparse=False, handle_unknown='ignore')In [25]python · cell 23
python
X_train_cat = ohe.fit_transform(df_train[cat].values)In [26]python · cell 24
python
ohe.get_feature_names()Output
array(['x0_Bronx', 'x0_Brooklyn', 'x0_Manhattan', 'x0_Queens',
'x0_Staten Island', 'x1_Entire home/apt', 'x1_Private room',
'x1_Shared room'], dtype=object)Now we need to combine two matrices into one - X_train:
In [27]python · cell 26
python
X_train = np.column_stack([X_train_num, X_train_cat])And now let's train the model:
In [28]python · cell 28
python
model = LogisticRegression(solver='lbfgs', C=1.0, random_state=42)
model.fit(X_train, y_train)Output
LogisticRegression(random_state=42)
We can check it's accuracy:
In [29]python · cell 30
python
X_val_num = df_val[num].values
X_val_num = scaler.transform(X_val_num)
X_val_cat = ohe.transform(df_val[cat].values)
X_val = np.column_stack([X_val_num, X_val_cat])In [30]python · cell 31
python
y_pred = model.predict_proba(X_val)[:, 1]
accuracy_score(y_val, y_pred >= 0.5)Output
0.7976275692811126
It's a little bit better than the version without scaled features.