Chapter 03
Machine Learning Zoomcamp
NotebookPython 3 (ipykernel)52 cells
Machine Learning Zoomcamp
1.9 Introduction to Pandas
Plan:
- Data Frames
- Series
- Index
- Accessing elements
- Element-wise operations
- Filtering
- String operations
- Summarizing operations
- Missing values
- Grouping
- Getting the NumPy arrays
In [4]python · cell 2
python
import numpy as np
import pandas as pdDataFrames
In [5]python · cell 4
python
data = [
['Nissan', 'Stanza', 1991, 138, 4, 'MANUAL', 'sedan', 2000],
['Hyundai', 'Sonata', 2017, None, 4, 'AUTOMATIC', 'Sedan', 27150],
['Lotus', 'Elise', 2010, 218, 4, 'MANUAL', 'convertible', 54990],
['GMC', 'Acadia', 2017, 194, 4, 'AUTOMATIC', '4dr SUV', 34450],
['Nissan', 'Frontier', 2017, 261, 6, 'MANUAL', 'Pickup', 32340],
]
columns = [
'Make', 'Model', 'Year', 'Engine HP', 'Engine Cylinders',
'Transmission Type', 'Vehicle_Style', 'MSRP'
]In [8]python · cell 5
python
df = pd.DataFrame(data, columns=columns)In [9]python · cell 6
python
dfOutput
Make Model Year Engine HP Engine Cylinders Transmission Type \ 0 Nissan Stanza 1991 138.0 4 MANUAL 1 Hyundai Sonata 2017 NaN 4 AUTOMATIC 2 Lotus Elise 2010 218.0 4 MANUAL 3 GMC Acadia 2017 194.0 4 AUTOMATIC 4 Nissan Frontier 2017 261.0 6 MANUAL Vehicle_Style MSRP 0 sedan 2000 1 Sedan 27150 2 convertible 54990 3 4dr SUV 34450 4 Pickup 32340
.dataframe tbody tr th:only-of-type {
vertical-align: middle;
}
.dataframe tbody tr th {
vertical-align: top;
}
.dataframe thead th {
text-align: right;
}
| Make | Model | Year | Engine HP | Engine Cylinders | Transmission Type | Vehicle_Style | MSRP | |
|---|---|---|---|---|---|---|---|---|
| 0 | Nissan | Stanza | 1991 | 138.0 | 4 | MANUAL | sedan | 2000 |
| 1 | Hyundai | Sonata | 2017 | NaN | 4 | AUTOMATIC | Sedan | 27150 |
| 2 | Lotus | Elise | 2010 | 218.0 | 4 | MANUAL | convertible | 54990 |
| 3 | GMC | Acadia | 2017 | 194.0 | 4 | AUTOMATIC | 4dr SUV | 34450 |
| 4 | Nissan | Frontier | 2017 | 261.0 | 6 | MANUAL | Pickup | 32340 |
In [10]python · cell 7
python
data = [
{
"Make": "Nissan",
"Model": "Stanza",
"Year": 1991,
"Engine HP": 138.0,
"Engine Cylinders": 4,
"Transmission Type": "MANUAL",
"Vehicle_Style": "sedan",
"MSRP": 2000
},
{
"Make": "Hyundai",
"Model": "Sonata",
"Year": 2017,
"Engine HP": None,
"Engine Cylinders": 4,
"Transmission Type": "AUTOMATIC",
"Vehicle_Style": "Sedan",
"MSRP": 27150
},
{
"Make": "Lotus",
"Model": "Elise",
"Year": 2010,
"Engine HP": 218.0,
"Engine Cylinders": 4,
"Transmission Type": "MANUAL",
"Vehicle_Style": "convertible",
"MSRP": 54990
},
{
"Make": "GMC",
"Model": "Acadia",
"Year": 2017,
"Engine HP": 194.0,
"Engine Cylinders": 4,
"Transmission Type": "AUTOMATIC",
"Vehicle_Style": "4dr SUV",
"MSRP": 34450
},
{
"Make": "Nissan",
"Model": "Frontier",
"Year": 2017,
"Engine HP": 261.0,
"Engine Cylinders": 6,
"Transmission Type": "MANUAL",
"Vehicle_Style": "Pickup",
"MSRP": 32340
}
]In [12]python · cell 8
python
df = pd.DataFrame(data)
dfOutput
Make Model Year Engine HP Engine Cylinders Transmission Type \ 0 Nissan Stanza 1991 138.0 4 MANUAL 1 Hyundai Sonata 2017 NaN 4 AUTOMATIC 2 Lotus Elise 2010 218.0 4 MANUAL 3 GMC Acadia 2017 194.0 4 AUTOMATIC 4 Nissan Frontier 2017 261.0 6 MANUAL Vehicle_Style MSRP 0 sedan 2000 1 Sedan 27150 2 convertible 54990 3 4dr SUV 34450 4 Pickup 32340
.dataframe tbody tr th:only-of-type {
vertical-align: middle;
}
.dataframe tbody tr th {
vertical-align: top;
}
.dataframe thead th {
text-align: right;
}
| Make | Model | Year | Engine HP | Engine Cylinders | Transmission Type | Vehicle_Style | MSRP | |
|---|---|---|---|---|---|---|---|---|
| 0 | Nissan | Stanza | 1991 | 138.0 | 4 | MANUAL | sedan | 2000 |
| 1 | Hyundai | Sonata | 2017 | NaN | 4 | AUTOMATIC | Sedan | 27150 |
| 2 | Lotus | Elise | 2010 | 218.0 | 4 | MANUAL | convertible | 54990 |
| 3 | GMC | Acadia | 2017 | 194.0 | 4 | AUTOMATIC | 4dr SUV | 34450 |
| 4 | Nissan | Frontier | 2017 | 261.0 | 6 | MANUAL | Pickup | 32340 |
In [14]python · cell 9
python
df.head(n=2)Output
Make Model Year Engine HP Engine Cylinders Transmission Type \ 0 Nissan Stanza 1991 138.0 4 MANUAL 1 Hyundai Sonata 2017 NaN 4 AUTOMATIC Vehicle_Style MSRP 0 sedan 2000 1 Sedan 27150
.dataframe tbody tr th:only-of-type {
vertical-align: middle;
}
.dataframe tbody tr th {
vertical-align: top;
}
.dataframe thead th {
text-align: right;
}
| Make | Model | Year | Engine HP | Engine Cylinders | Transmission Type | Vehicle_Style | MSRP | |
|---|---|---|---|---|---|---|---|---|
| 0 | Nissan | Stanza | 1991 | 138.0 | 4 | MANUAL | sedan | 2000 |
| 1 | Hyundai | Sonata | 2017 | NaN | 4 | AUTOMATIC | Sedan | 27150 |
In [ ]python · cell 10
python
Series
In [18]python · cell 12
python
df.Engine HPOutput
[0;36m File [0;32m"/tmp/ipykernel_580/1897567212.py"[0;36m, line [0;32m1[0m [0;31m df.Engine HP[0m [0m ^[0m [0;31mSyntaxError[0m[0;31m:[0m invalid syntax
In [19]python · cell 13
python
df['Engine HP']Output
0 138.0 1 NaN 2 218.0 3 194.0 4 261.0 Name: Engine HP, dtype: float64
In [20]python · cell 14
python
df[['Make', 'Model', 'MSRP']]Output
Make Model MSRP 0 Nissan Stanza 2000 1 Hyundai Sonata 27150 2 Lotus Elise 54990 3 GMC Acadia 34450 4 Nissan Frontier 32340
.dataframe tbody tr th:only-of-type {
vertical-align: middle;
}
.dataframe tbody tr th {
vertical-align: top;
}
.dataframe thead th {
text-align: right;
}
| Make | Model | MSRP | |
|---|---|---|---|
| 0 | Nissan | Stanza | 2000 |
| 1 | Hyundai | Sonata | 27150 |
| 2 | Lotus | Elise | 54990 |
| 3 | GMC | Acadia | 34450 |
| 4 | Nissan | Frontier | 32340 |
In [23]python · cell 15
python
df['id'] = [1, 2, 3, 4, 5]In [26]python · cell 16
python
df['id'] = [10, 20, 30, 40, 50]In [27]python · cell 17
python
dfOutput
Make Model Year Engine HP Engine Cylinders Transmission Type \ 0 Nissan Stanza 1991 138.0 4 MANUAL 1 Hyundai Sonata 2017 NaN 4 AUTOMATIC 2 Lotus Elise 2010 218.0 4 MANUAL 3 GMC Acadia 2017 194.0 4 AUTOMATIC 4 Nissan Frontier 2017 261.0 6 MANUAL Vehicle_Style MSRP id 0 sedan 2000 10 1 Sedan 27150 20 2 convertible 54990 30 3 4dr SUV 34450 40 4 Pickup 32340 50
.dataframe tbody tr th:only-of-type {
vertical-align: middle;
}
.dataframe tbody tr th {
vertical-align: top;
}
.dataframe thead th {
text-align: right;
}
| Make | Model | Year | Engine HP | Engine Cylinders | Transmission Type | Vehicle_Style | MSRP | id | |
|---|---|---|---|---|---|---|---|---|---|
| 0 | Nissan | Stanza | 1991 | 138.0 | 4 | MANUAL | sedan | 2000 | 10 |
| 1 | Hyundai | Sonata | 2017 | NaN | 4 | AUTOMATIC | Sedan | 27150 | 20 |
| 2 | Lotus | Elise | 2010 | 218.0 | 4 | MANUAL | convertible | 54990 | 30 |
| 3 | GMC | Acadia | 2017 | 194.0 | 4 | AUTOMATIC | 4dr SUV | 34450 | 40 |
| 4 | Nissan | Frontier | 2017 | 261.0 | 6 | MANUAL | Pickup | 32340 | 50 |
In [28]python · cell 18
python
del df['id']In [29]python · cell 19
python
dfOutput
Make Model Year Engine HP Engine Cylinders Transmission Type \ 0 Nissan Stanza 1991 138.0 4 MANUAL 1 Hyundai Sonata 2017 NaN 4 AUTOMATIC 2 Lotus Elise 2010 218.0 4 MANUAL 3 GMC Acadia 2017 194.0 4 AUTOMATIC 4 Nissan Frontier 2017 261.0 6 MANUAL Vehicle_Style MSRP 0 sedan 2000 1 Sedan 27150 2 convertible 54990 3 4dr SUV 34450 4 Pickup 32340
.dataframe tbody tr th:only-of-type {
vertical-align: middle;
}
.dataframe tbody tr th {
vertical-align: top;
}
.dataframe thead th {
text-align: right;
}
| Make | Model | Year | Engine HP | Engine Cylinders | Transmission Type | Vehicle_Style | MSRP | |
|---|---|---|---|---|---|---|---|---|
| 0 | Nissan | Stanza | 1991 | 138.0 | 4 | MANUAL | sedan | 2000 |
| 1 | Hyundai | Sonata | 2017 | NaN | 4 | AUTOMATIC | Sedan | 27150 |
| 2 | Lotus | Elise | 2010 | 218.0 | 4 | MANUAL | convertible | 54990 |
| 3 | GMC | Acadia | 2017 | 194.0 | 4 | AUTOMATIC | 4dr SUV | 34450 |
| 4 | Nissan | Frontier | 2017 | 261.0 | 6 | MANUAL | Pickup | 32340 |
Index
In [30]python · cell 21
python
df.indexOutput
RangeIndex(start=0, stop=5, step=1)
In [32]python · cell 22
python
df.Make.indexOutput
RangeIndex(start=0, stop=5, step=1)
In [37]python · cell 23
python
df.index = ['a', 'b', 'c', 'd', 'e']In [38]python · cell 24
python
dfOutput
Make Model Year Engine HP Engine Cylinders Transmission Type \ a Nissan Stanza 1991 138.0 4 MANUAL b Hyundai Sonata 2017 NaN 4 AUTOMATIC c Lotus Elise 2010 218.0 4 MANUAL d GMC Acadia 2017 194.0 4 AUTOMATIC e Nissan Frontier 2017 261.0 6 MANUAL Vehicle_Style MSRP a sedan 2000 b Sedan 27150 c convertible 54990 d 4dr SUV 34450 e Pickup 32340
.dataframe tbody tr th:only-of-type {
vertical-align: middle;
}
.dataframe tbody tr th {
vertical-align: top;
}
.dataframe thead th {
text-align: right;
}
| Make | Model | Year | Engine HP | Engine Cylinders | Transmission Type | Vehicle_Style | MSRP | |
|---|---|---|---|---|---|---|---|---|
| a | Nissan | Stanza | 1991 | 138.0 | 4 | MANUAL | sedan | 2000 |
| b | Hyundai | Sonata | 2017 | NaN | 4 | AUTOMATIC | Sedan | 27150 |
| c | Lotus | Elise | 2010 | 218.0 | 4 | MANUAL | convertible | 54990 |
| d | GMC | Acadia | 2017 | 194.0 | 4 | AUTOMATIC | 4dr SUV | 34450 |
| e | Nissan | Frontier | 2017 | 261.0 | 6 | MANUAL | Pickup | 32340 |
In [42]python · cell 25
python
df.iloc[[1, 2, 4]]Output
Make Model Year Engine HP Engine Cylinders Transmission Type \ b Hyundai Sonata 2017 NaN 4 AUTOMATIC c Lotus Elise 2010 218.0 4 MANUAL e Nissan Frontier 2017 261.0 6 MANUAL Vehicle_Style MSRP b Sedan 27150 c convertible 54990 e Pickup 32340
.dataframe tbody tr th:only-of-type {
vertical-align: middle;
}
.dataframe tbody tr th {
vertical-align: top;
}
.dataframe thead th {
text-align: right;
}
| Make | Model | Year | Engine HP | Engine Cylinders | Transmission Type | Vehicle_Style | MSRP | |
|---|---|---|---|---|---|---|---|---|
| b | Hyundai | Sonata | 2017 | NaN | 4 | AUTOMATIC | Sedan | 27150 |
| c | Lotus | Elise | 2010 | 218.0 | 4 | MANUAL | convertible | 54990 |
| e | Nissan | Frontier | 2017 | 261.0 | 6 | MANUAL | Pickup | 32340 |
In [47]python · cell 26
python
df = df.reset_index(drop=True)In [48]python · cell 27
python
dfOutput
Make Model Year Engine HP Engine Cylinders Transmission Type \ 0 Nissan Stanza 1991 138.0 4 MANUAL 1 Hyundai Sonata 2017 NaN 4 AUTOMATIC 2 Lotus Elise 2010 218.0 4 MANUAL 3 GMC Acadia 2017 194.0 4 AUTOMATIC 4 Nissan Frontier 2017 261.0 6 MANUAL Vehicle_Style MSRP 0 sedan 2000 1 Sedan 27150 2 convertible 54990 3 4dr SUV 34450 4 Pickup 32340
.dataframe tbody tr th:only-of-type {
vertical-align: middle;
}
.dataframe tbody tr th {
vertical-align: top;
}
.dataframe thead th {
text-align: right;
}
| Make | Model | Year | Engine HP | Engine Cylinders | Transmission Type | Vehicle_Style | MSRP | |
|---|---|---|---|---|---|---|---|---|
| 0 | Nissan | Stanza | 1991 | 138.0 | 4 | MANUAL | sedan | 2000 |
| 1 | Hyundai | Sonata | 2017 | NaN | 4 | AUTOMATIC | Sedan | 27150 |
| 2 | Lotus | Elise | 2010 | 218.0 | 4 | MANUAL | convertible | 54990 |
| 3 | GMC | Acadia | 2017 | 194.0 | 4 | AUTOMATIC | 4dr SUV | 34450 |
| 4 | Nissan | Frontier | 2017 | 261.0 | 6 | MANUAL | Pickup | 32340 |
Accessing elements
In [ ]python · cell 29
python
Element-wise operations
In [51]python · cell 31
python
df['Engine HP'] * 2Output
0 276.0 1 NaN 2 436.0 3 388.0 4 522.0 Name: Engine HP, dtype: float64
In [53]python · cell 32
python
df['Year'] >= 2015Output
0 False 1 True 2 False 3 True 4 True Name: Year, dtype: bool
Filtering
In [55]python · cell 34
python
df[
df['Make'] == 'Nissan'
]Output
Make Model Year Engine HP Engine Cylinders Transmission Type \ 0 Nissan Stanza 1991 138.0 4 MANUAL 4 Nissan Frontier 2017 261.0 6 MANUAL Vehicle_Style MSRP 0 sedan 2000 4 Pickup 32340
.dataframe tbody tr th:only-of-type {
vertical-align: middle;
}
.dataframe tbody tr th {
vertical-align: top;
}
.dataframe thead th {
text-align: right;
}
| Make | Model | Year | Engine HP | Engine Cylinders | Transmission Type | Vehicle_Style | MSRP | |
|---|---|---|---|---|---|---|---|---|
| 0 | Nissan | Stanza | 1991 | 138.0 | 4 | MANUAL | sedan | 2000 |
| 4 | Nissan | Frontier | 2017 | 261.0 | 6 | MANUAL | Pickup | 32340 |
In [56]python · cell 35
python
df[
(df['Make'] == 'Nissan') & (df['Year'] >= 2015)
]Output
Make Model Year Engine HP Engine Cylinders Transmission Type \ 4 Nissan Frontier 2017 261.0 6 MANUAL Vehicle_Style MSRP 4 Pickup 32340
.dataframe tbody tr th:only-of-type {
vertical-align: middle;
}
.dataframe tbody tr th {
vertical-align: top;
}
.dataframe thead th {
text-align: right;
}
| Make | Model | Year | Engine HP | Engine Cylinders | Transmission Type | Vehicle_Style | MSRP | |
|---|---|---|---|---|---|---|---|---|
| 4 | Nissan | Frontier | 2017 | 261.0 | 6 | MANUAL | Pickup | 32340 |
String operations
In [68]python · cell 37
python
'machine learning zoomcamp'.replace(' ', '_')Output
'machine_learning_zoomcamp'
In [67]python · cell 38
python
df['Vehicle_Style'].str.lower()Output
0 sedan 1 sedan 2 convertible 3 4dr suv 4 pickup Name: Vehicle_Style, dtype: object
In [72]python · cell 39
python
df['Vehicle_Style'] = df['Vehicle_Style'].str.replace(' ', '_').str.lower()In [74]python · cell 40
python
dfOutput
Make Model Year Engine HP Engine Cylinders Transmission Type \ 0 Nissan Stanza 1991 138.0 4 MANUAL 1 Hyundai Sonata 2017 NaN 4 AUTOMATIC 2 Lotus Elise 2010 218.0 4 MANUAL 3 GMC Acadia 2017 194.0 4 AUTOMATIC 4 Nissan Frontier 2017 261.0 6 MANUAL Vehicle_Style MSRP 0 sedan 2000 1 sedan 27150 2 convertible 54990 3 4dr_suv 34450 4 pickup 32340
.dataframe tbody tr th:only-of-type {
vertical-align: middle;
}
.dataframe tbody tr th {
vertical-align: top;
}
.dataframe thead th {
text-align: right;
}
| Make | Model | Year | Engine HP | Engine Cylinders | Transmission Type | Vehicle_Style | MSRP | |
|---|---|---|---|---|---|---|---|---|
| 0 | Nissan | Stanza | 1991 | 138.0 | 4 | MANUAL | sedan | 2000 |
| 1 | Hyundai | Sonata | 2017 | NaN | 4 | AUTOMATIC | sedan | 27150 |
| 2 | Lotus | Elise | 2010 | 218.0 | 4 | MANUAL | convertible | 54990 |
| 3 | GMC | Acadia | 2017 | 194.0 | 4 | AUTOMATIC | 4dr_suv | 34450 |
| 4 | Nissan | Frontier | 2017 | 261.0 | 6 | MANUAL | pickup | 32340 |
Summarizing operations
In [81]python · cell 42
python
df.describe().round(2)Output
Year Engine HP Engine Cylinders MSRP count 5.00 4.00 5.00 5.00 mean 2010.40 202.75 4.40 30186.00 std 11.26 51.30 0.89 18985.04 min 1991.00 138.00 4.00 2000.00 25% 2010.00 180.00 4.00 27150.00 50% 2017.00 206.00 4.00 32340.00 75% 2017.00 228.75 4.00 34450.00 max 2017.00 261.00 6.00 54990.00
.dataframe tbody tr th:only-of-type {
vertical-align: middle;
}
.dataframe tbody tr th {
vertical-align: top;
}
.dataframe thead th {
text-align: right;
}
| Year | Engine HP | Engine Cylinders | MSRP | |
|---|---|---|---|---|
| count | 5.00 | 4.00 | 5.00 | 5.00 |
| mean | 2010.40 | 202.75 | 4.40 | 30186.00 |
| std | 11.26 | 51.30 | 0.89 | 18985.04 |
| min | 1991.00 | 138.00 | 4.00 | 2000.00 |
| 25% | 2010.00 | 180.00 | 4.00 | 27150.00 |
| 50% | 2017.00 | 206.00 | 4.00 | 32340.00 |
| 75% | 2017.00 | 228.75 | 4.00 | 34450.00 |
| max | 2017.00 | 261.00 | 6.00 | 54990.00 |
In [85]python · cell 43
python
df.nunique()Output
Make 4 Model 5 Year 3 Engine HP 4 Engine Cylinders 2 Transmission Type 2 Vehicle_Style 4 MSRP 5 dtype: int64
Missing values
In [87]python · cell 45
python
df.isnull().sum()Output
Make 0 Model 0 Year 0 Engine HP 1 Engine Cylinders 0 Transmission Type 0 Vehicle_Style 0 MSRP 0 dtype: int64
Grouping
code
SELECT
transmission_type,
AVG(MSRP)
FROM
cars
GROUP BY
transmission_typeIn [90]python · cell 48
python
df.groupby('Transmission Type').MSRP.max()Output
Transmission Type AUTOMATIC 34450 MANUAL 54990 Name: MSRP, dtype: int64
Getting the NumPy arrays
In [93]python · cell 50
python
df.MSRP.valuesOutput
array([ 2000, 27150, 54990, 34450, 32340])
In [95]python · cell 51
python
df.to_dict(orient='records')Output
[{'Make': 'Nissan',
'Model': 'Stanza',
'Year': 1991,
'Engine HP': 138.0,
'Engine Cylinders': 4,
'Transmission Type': 'MANUAL',
'Vehicle_Style': 'sedan',
'MSRP': 2000},
{'Make': 'Hyundai',
'Model': 'Sonata',
'Year': 2017,
'Engine HP': nan,
'Engine Cylinders': 4,
'Transmission Type': 'AUTOMATIC',
'Vehicle_Style': 'sedan',
'MSRP': 27150},
{'Make': 'Lotus',
'Model': 'Elise',
'Year': 2010,
'Engine HP': 218.0,
'Engine Cylinders': 4,
'Transmission Type': 'MANUAL',
'Vehicle_Style': 'convertible',
'MSRP': 54990},
{'Make': 'GMC',
'Model': 'Acadia',
'Year': 2017,
'Engine HP': 194.0,
'Engine Cylinders': 4,
'Transmission Type': 'AUTOMATIC',
'Vehicle_Style': '4dr_suv',
'MSRP': 34450},
{'Make': 'Nissan',
'Model': 'Frontier',
'Year': 2017,
'Engine HP': 261.0,
'Engine Cylinders': 6,
'Transmission Type': 'MANUAL',
'Vehicle_Style': 'pickup',
'MSRP': 32340}]In [ ]python · cell 52
python