Module 5: Monitoring

Offline evaluation can't tell you how your RAG system performs once real people use it. This module covers online monitoring: collecting metrics from real traffic and visualizing them on a dashboard.

We build a Streamlit chat app, capture metrics, store conversations in PostgreSQL, and create Grafana dashboards for real-time monitoring.

Lessons

Work through them in order:

1. Intro - Why monitoring matters, what we'll build
2. Assistant Setup - Setting up the RAG assistant
3. Chat App - Basic Streamlit app with RAG
4. Capturing Metrics - LLMCallRecord, cost tracking
5. Database - PostgreSQL with Docker, saving conversations
6. Querying Data - Fetching stored conversations
7. Streamlit Dashboard - Visualizing metrics in Streamlit
8. User Feedback - Thumbs up/down buttons
9. Built-in Judge - LLM-as-a-judge for automatic relevance evaluation
10. Feedback Dashboard - Adding feedback panels to the Streamlit dashboard
11. Synthetic Data - Generating test data for dashboards
12. Grafana Dashboards - SQL queries and dashboard panels
13. Docker Compose - Running everything together
14. Next Steps - OpenTelemetry, alerting, frameworks to learn more

Homework

• Homework

Original workshop recording

This module was taught as a live workshop, which we chopped into the per-lesson videos above. To watch the full uncut recording:

• Monitoring LLM Applications: Traces, Feedback, and Production Quality

Older content

Earlier cohorts ran this module with a different stack:

• 2024 edition
• 2025 edition