Chapter 29
homework
NotebookPython 3 (ipykernel)30 cells
In [1]python · cell 1
python
import pandas as pd
from sklearn.feature_extraction import DictVectorizer
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error
import seaborn as sns
import matplotlib.pyplot as plt
import warnings
warnings.filterwarnings('ignore')Output
C:\Users\alexe\anaconda3\lib\site-packages\scipy\__init__.py:155: UserWarning: A NumPy version >=1.18.5 and <1.25.0 is required for this version of SciPy (detected version 1.26.2
warnings.warn(f"A NumPy version >={np_minversion} and <{np_maxversion}"
Q1. Downloading the data
In [4]python · cell 3
python
df = pd.read_parquet('https://d37ci6vzurychx.cloudfront.net/trip-data/yellow_tripdata_2023-01.parquet')In [5]python · cell 4
python
df.head()Output
VendorID tpep_pickup_datetime tpep_dropoff_datetime passenger_count \ 0 2 2023-01-01 00:32:10 2023-01-01 00:40:36 1.0 1 2 2023-01-01 00:55:08 2023-01-01 01:01:27 1.0 2 2 2023-01-01 00:25:04 2023-01-01 00:37:49 1.0 3 1 2023-01-01 00:03:48 2023-01-01 00:13:25 0.0 4 2 2023-01-01 00:10:29 2023-01-01 00:21:19 1.0 trip_distance RatecodeID store_and_fwd_flag PULocationID DOLocationID \ 0 0.97 1.0 N 161 141 1 1.10 1.0 N 43 237 2 2.51 1.0 N 48 238 3 1.90 1.0 N 138 7 4 1.43 1.0 N 107 79 payment_type fare_amount extra mta_tax tip_amount tolls_amount \ 0 2 9.3 1.00 0.5 0.00 0.0 1 1 7.9 1.00 0.5 4.00 0.0 2 1 14.9 1.00 0.5 15.00 0.0 3 1 12.1 7.25 0.5 0.00 0.0 4 1 11.4 1.00 0.5 3.28 0.0 improvement_surcharge total_amount congestion_surcharge airport_fee 0 1.0 14.30 2.5 0.00 1 1.0 16.90 2.5 0.00 2 1.0 34.90 2.5 0.00 3 1.0 20.85 0.0 1.25 4 1.0 19.68 2.5 0.00
.dataframe tbody tr th:only-of-type {
vertical-align: middle;
}
.dataframe tbody tr th {
vertical-align: top;
}
.dataframe thead th {
text-align: right;
}
| VendorID | tpep_pickup_datetime | tpep_dropoff_datetime | passenger_count | trip_distance | RatecodeID | store_and_fwd_flag | PULocationID | DOLocationID | payment_type | fare_amount | extra | mta_tax | tip_amount | tolls_amount | improvement_surcharge | total_amount | congestion_surcharge | airport_fee | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 2 | 2023-01-01 00:32:10 | 2023-01-01 00:40:36 | 1.0 | 0.97 | 1.0 | N | 161 | 141 | 2 | 9.3 | 1.00 | 0.5 | 0.00 | 0.0 | 1.0 | 14.30 | 2.5 | 0.00 |
| 1 | 2 | 2023-01-01 00:55:08 | 2023-01-01 01:01:27 | 1.0 | 1.10 | 1.0 | N | 43 | 237 | 1 | 7.9 | 1.00 | 0.5 | 4.00 | 0.0 | 1.0 | 16.90 | 2.5 | 0.00 |
| 2 | 2 | 2023-01-01 00:25:04 | 2023-01-01 00:37:49 | 1.0 | 2.51 | 1.0 | N | 48 | 238 | 1 | 14.9 | 1.00 | 0.5 | 15.00 | 0.0 | 1.0 | 34.90 | 2.5 | 0.00 |
| 3 | 1 | 2023-01-01 00:03:48 | 2023-01-01 00:13:25 | 0.0 | 1.90 | 1.0 | N | 138 | 7 | 1 | 12.1 | 7.25 | 0.5 | 0.00 | 0.0 | 1.0 | 20.85 | 0.0 | 1.25 |
| 4 | 2 | 2023-01-01 00:10:29 | 2023-01-01 00:21:19 | 1.0 | 1.43 | 1.0 | N | 107 | 79 | 1 | 11.4 | 1.00 | 0.5 | 3.28 | 0.0 | 1.0 | 19.68 | 2.5 | 0.00 |
In [6]python · cell 5
python
len(df.columns)Output
19
Q2. Computing duration
In [7]python · cell 7
python
df['duration'] = df.tpep_dropoff_datetime - df.tpep_pickup_datetime
df['duration'] = df.duration.dt.total_seconds() / 60In [8]python · cell 8
python
df.duration.mean()Output
15.668995167332046
In [10]python · cell 9
python
df.duration.std()Output
42.594351241920904
Q3. Dropping outliers
In [11]python · cell 11
python
len(df[(df.duration >= 1) & (df.duration <= 60)]) / len(df) * 100Output
98.1220282212598
In [12]python · cell 12
python
df = df[(df.duration >= 1) & (df.duration <= 60)].copy()Q4. One-hot encoding
In [13]python · cell 14
python
categorical = ['PULocationID', 'DOLocationID']In [14]python · cell 15
python
df[categorical] = df[categorical].astype(str)In [15]python · cell 16
python
train_dicts = df[categorical].to_dict(orient='records')In [16]python · cell 17
python
dv = DictVectorizer()
X_train = dv.fit_transform(train_dicts)In [17]python · cell 18
python
print(f'Feature matrix size: {X_train.shape}')Output
Feature matrix size: (3009173, 515)
Q5. Training a model
In [18]python · cell 20
python
target = 'duration'
y_train = df[target].valuesIn [19]python · cell 21
python
lr = LinearRegression()
lr.fit(X_train, y_train)
y_pred = lr.predict(X_train)
print(f'Train RMSE: {mean_squared_error(y_train, y_pred, squared=False)}')Output
Train RMSE: 7.64926195987998
In [20]python · cell 22
python
sns.histplot(y_pred, label='prediction')
sns.histplot(y_train, label='actual')
plt.legend();Output
<Figure size 640x480 with 1 Axes>
Q6. Evaluating the model
In [21]python · cell 24
python
categorical = ['PULocationID', 'DOLocationID']
def read_data(filename):
df = pd.read_parquet(filename)
df['duration'] = df.tpep_dropoff_datetime - df.tpep_pickup_datetime
df.duration = df.duration.dt.total_seconds() / 60
df = df[(df.duration >= 1) & (df.duration <= 60)].copy()
df[categorical] = df[categorical].astype('str')
return dfIn [23]python · cell 25
python
df_val = read_data('https://d37ci6vzurychx.cloudfront.net/trip-data/yellow_tripdata_2023-02.parquet')In [24]python · cell 26
python
val_dicts = df_val[categorical].to_dict(orient='records')In [25]python · cell 27
python
X_val = dv.transform(val_dicts)
y_val = df_val.duration.valuesIn [26]python · cell 28
python
y_pred = lr.predict(X_val)In [27]python · cell 29
python
print(f'Val RMSE: {mean_squared_error(y_val, y_pred, squared=False)}')Output
Val RMSE: 7.81181633377777
In [ ]python · cell 30
python