Production AI Agents with MLflow: AgentOps Techniques for Evaluation, Observability, and Deployment
Published 11/2026
By Jonathan Bown
MP4 |
Video: h264, 1280x720 |
Audio: AAC, 44.1 KHz, 2 Ch
Genre: eLearning |
Language: English + subtitle |
Duration: 6h 48m |
Size: 12.2 GB
Build, evaluate, and ship production-grade AI agents with MLflow's open-source observability, evaluation, and deployment stack
Overview
Production AI Agents with MLflow: AgentOps Techniques for Evaluation, Observability, and Deployment is a hands-on, code-driven course that takes you from a single LLM call to a LangGraph workflow to a deployed ReAct agent, using MLflow tracing, evaluation, Prompt Registry, AI Gateway, and Model Registry to make every stage observable, measurable, and production-ready.
Learn How To

Apply Evaluation-Driven Development and the Agent Development Lifecycle to move agents from proof-of-concept to production

Set up a local MLflow tracking server and trace LLM calls, RAG pipelines, and tool usage in the MLflow UI

Run structured evaluations with built-in scorers and custom LLM-as-judge scorers

Capture human feedback and ground truth annotations on traces to build alignment datasets

Align automated judges to expert human feedback using the SIMBA optimizer

Version and alias prompts in the MLflow Prompt Registry using a champion/challenger pattern

Evolve a single LLM call into a LangGraph workflow with node-level evaluation, memory, and memory governance

Build and evaluate ReAct tool-calling agents, including the shift from offline to online evaluation

Route traffic, track usage, and set budgets with the MLflow AI Gateway

Automate prompt optimization and evaluate multi-turn conversations with simulated user personas

Package, register, and serve agents locally with the MLflow Model Registry
Who Should Take This Course
AI and GenAI engineers, machine learning engineers, and data scientists working on agents or LLM applications, as well as product engineers and product leaders who want to understand the Agent Development Lifecycle and lead engineering teams building agents. This course is ideal for practitioners who have built LLM or agent prototypes and want the operational discipline to make them observable, measurable, versioned, and deployable.
Course
Requirements

Intermediate Python proficiency (functions, classes, environment variables, virtual environments)

Basic familiarity with LLMs and the OpenAI API or an equivalent (e.g., Gemini, Anthropic)

LLM API Key recommended but not required - Gemini Free Tier suitable for many aspects of course

Comfort working in and reading Jupyter notebooks

No prior MLflow experience required; all MLflow concepts are introduced from scratch
Homepage
Code:
https://www.oreilly.com/videos/production-ai-agents/9780136173045/