What's new

Welcome to Free download educational resource and Apps from TUTBB

Join us now to get access to all our features. Once registered and logged in, you will be able to create topics, post replies to existing threads, give reputation to your fellow members, get your own private messenger, and so, so much more. It's also quick and totally free, so what are you waiting for?

Production AI Agents with MLflow AgentOps Techniques for Evaluation, Observability, and Deployment

TUTBB

Active member
Joined
Apr 9, 2022
Messages
196,414
Reaction score
20
Points
38

fb331631514d208cd3d7a350b0f43c5f.webp

Production AI Agents with MLflow: AgentOps Techniques for Evaluation, Observability, and Deployment
Published 11/2026
By Jonathan Bown
MP4 | Video: h264, 1280x720 | Audio: AAC, 44.1 KHz, 2 Ch
Genre: eLearning | Language: English + subtitle | Duration: 6h 48m | Size: 12.2 GB
Build, evaluate, and ship production-grade AI agents with MLflow's open-source observability, evaluation, and deployment stack
​


Overview
Production AI Agents with MLflow: AgentOps Techniques for Evaluation, Observability, and Deployment is a hands-on, code-driven course that takes you from a single LLM call to a LangGraph workflow to a deployed ReAct agent, using MLflow tracing, evaluation, Prompt Registry, AI Gateway, and Model Registry to make every stage observable, measurable, and production-ready.
Learn How To
⚡ Apply Evaluation-Driven Development and the Agent Development Lifecycle to move agents from proof-of-concept to production
⚡ Set up a local MLflow tracking server and trace LLM calls, RAG pipelines, and tool usage in the MLflow UI
⚡ Run structured evaluations with built-in scorers and custom LLM-as-judge scorers
⚡ Capture human feedback and ground truth annotations on traces to build alignment datasets
⚡ Align automated judges to expert human feedback using the SIMBA optimizer
⚡ Version and alias prompts in the MLflow Prompt Registry using a champion/challenger pattern
⚡ Evolve a single LLM call into a LangGraph workflow with node-level evaluation, memory, and memory governance
⚡ Build and evaluate ReAct tool-calling agents, including the shift from offline to online evaluation
⚡ Route traffic, track usage, and set budgets with the MLflow AI Gateway
⚡ Automate prompt optimization and evaluate multi-turn conversations with simulated user personas
⚡ Package, register, and serve agents locally with the MLflow Model Registry
Who Should Take This Course
AI and GenAI engineers, machine learning engineers, and data scientists working on agents or LLM applications, as well as product engineers and product leaders who want to understand the Agent Development Lifecycle and lead engineering teams building agents. This course is ideal for practitioners who have built LLM or agent prototypes and want the operational discipline to make them observable, measurable, versioned, and deployable.
Course

Requirements


⚡ Intermediate Python proficiency (functions, classes, environment variables, virtual environments)
⚡ Basic familiarity with LLMs and the OpenAI API or an equivalent (e.g., Gemini, Anthropic)
⚡ LLM API Key recommended but not required - Gemini Free Tier suitable for many aspects of course
⚡ Comfort working in and reading Jupyter notebooks
⚡ No prior MLflow experience required; all MLflow concepts are introduced from scratch

Homepage
Code:
https://www.oreilly.com/videos/production-ai-agents/9780136173045/

 
Top Bottom