5. Model Monitoring

5.1 Intro to ML monitoring

5.2 Environment setup

5.3 Prepare reference and model

5.4 Evidently metrics calculation

5.5 Evidently Monitoring Dashboard

5.6 Dummy monitoring

5.7 Data quality monitoring

Note: in this video we use Prefect (07:33-11:21). Feel free to skip this part. Also note that Prefect is not officially supported in the 2024 edition of the course.

5.8 Save Grafana Dashboard

5.9 Debugging with test suites and reports

Homework

More information here

Notes

Did you take notes? Add them here:

• Week 5 notes by M. Ayoub C.
• week 5: Monitoring notes Ayoub.B
• Week 5: 2023
• Week5: Why we need to monitor models after deployment? by Hongfan (Amber)
• week-5: Detailed Notes about Monitoring, codes and homework by Muhammad Shifa
• Send a PR, add your notes above this line

Monitoring example

Notes

There were a massive update for Evidently since 0.7.0 version.

To check working example with Evidently >= 0.7.0 go to post-evidently-0.7 folder.

Prerequisites

You need following tools installed:

• docker
• docker-compose (included to Docker Desktop for Mac and Docker Desktop for Windows )

Preparation

Note: all actions expected to be executed in repo folder.

• Create virtual environment and activate it (eg. python -m venv venv && source ./venv/bin/activate or conda create -n venv python=3.11 && conda activate venv)
• Install required packages pip install -r requirements.txt
• Run baseline_model_nyc_taxi_data.ipynb for downloading datasets, training model and creating reference dataset

Monitoring Example

Starting services

To start all required services, execute:

docker-compose up

It will start following services:

• db - PostgreSQL, for storing metrics data
• adminer - database management tool
• grafana - Visual dashboarding tool

Sending data

To calculate evidently metrics with prefect and send them to database, execute:

python evidently_metrics_calculation.py

This script will simulate batch monitoring. Every 10 seconds it will collect data for a daily batch, calculate metrics and insert them into database. This metrics will be available in Grafana in preconfigured dashboard.

Accsess dashboard

• In your browser go to a localhost:3000 The default username and password are admin

• Then navigate to General/Home menu and click on Home.

• In the folder General you will see New Dashboard. Click on it to access preconfigured dashboard.

Ad-hoc debugging

Run debugging_nyc_taxi_data.ipynb to see how you can perform a debugging with help of Evidently TestSuites and Reports

Stopping services

To stop all services, execute:

docker-compose down