Welcome to Free download educational resource and Apps from TUTBB

TUTBB · Sep 12, 2026

Instrument a real agent with OpenTelemetry, catch silent failures, judge live traffic, debug a 3-bug incident.
What you'll learn

Instrument a real tool-calling agent with OpenTelemetry: spans, attributes, events, nesting and context propagation

Stand up Arize Phoenix and Langfuse yourself with Docker, and compare both on one identical trace

Catch silent failures that never raise: empty results, hidden retries, graceful fallbacks and confidently wrong answers

Attribute latency and cost per span, per feature, per user and per tenant, and see why run-level averages hide regressions

Build an LLM-as-judge on sampled live traffic, wire its verdicts back into the trace, and catch the judge being wrong

Detect prompt injection and quality drift from trace patterns rather than from the prompt text

Debug an unfamiliar trace under time pressure, correlate it to a user complaint, and diff a good run against a bad one

Work a multi-cause incident end to end: three seeded bugs, one red herring, a verified fix and an honest postmortem

Price observability at real scale and decide what to sample using arithmetic instead of instinct

Roll this out on a team: ownership, code review for instrumentation, and a runbook built from a real incident
Requirements

Comfortable Python: functions, classes, dictionaries, reading a traceback

Any OpenAI-compatible API key. Total spend across the whole course is a few dollars

Docker and Docker Compose installed, for the two tracing backends

No prior OpenTelemetry or observability experience needed. It is built from the first span
Description
This course contains the use of artificial intelligence.
Your agent returned a confident, well formed, completely wrong answer. Every span in the trace says status OK. Your eval suite passed this morning. Nothing is red, and something is badly broken.
An eval score is a photograph. Production is a video. This course is seventy one lectures about closing that gap, and every number in it was measured on a real system rather than typed because it sounded right.
What you actually build. A real tool-calling agent (search, fetch, calculate, summarise) instrumented with OpenTelemetry from scratch. Two real self-hosted backends, Arize Phoenix and Langfuse, compared on one identical trace. An LLM-as-judge running on sampled live traffic. A review queue. A runbook. Around twenty five small diagnostic tools, each written to answer one question the traces raised.
The failures are seeded and real. A search tool that always succeeds and always returns nothing, so the model rephrases six times and burns its budget. A fetch that serves the right document with stale numbers, where the agent does flawless arithmetic and lands eighty nine percent low. A graceful fallback that hides the truth. A prompt injection arriving through a poisoned document rather than a user prompt. None of them raise an error.
Then a capstone incident. Three bugs of three different kinds, no hints. One is a red herring: the loudest thing in the latency data, and not your problem. You learn to abandon it properly, by measuring what it leaves unexplained. The third bug has no error, no slow span, no anomaly at all, and nothing in any dashboard this course builds will flag it. It is visible only by adding up numbers the system already gives you and noticing they disagree.
Including the parts that did not work. When I scored every detector built in this course against that capstone, seven of thirteen contributed nothing. The LLM judge did not merely miss the bug, it returned grounded equals true and was right to. That result is in the course rather than hidden, because your coverage is always narrower than it feels.
Measured, not asserted. Storing every span for a million agent runs a day costs four dollars and thirty two cents a month, which makes sampling traces to save money the wrong instinct. A judge flipped its verdict ten times out of ten purely because the evidence arrived in a different order. Every figure quoted on screen is read off a measurement ledger that ships with the course, so you can check me.
Ten sections, seventy one lectures, roughly seven and a half hours, built on one small system you can run end to end on a laptop.
Who this course is for

Engineers already shipping something with an LLM in it, who have been told it is broken and had no idea how to find out why

Anyone whose agent passes its evals and still misbehaves in production

Backend and platform engineers who know normal APM but have not traced a non-deterministic tool-calling loop

ML and AI engineers who write evals and want the live half of the picture

Not for beginners looking for a first Python course, and not a prompt engineering course
Homepage

Code:

https://www.udemy.com/course/agent-observability-tracing-debug-production-llm-systems

Recommend Download Link Hight Speed | Please Say Thanks Keep Topic Live

No Password - Links are Interchangeable

Search

Welcome to Free download educational resource and Apps from TUTBB

Agent Observability & Tracing Debug Production LLM Systems

TUTBB

Active member

About Us

Newest members

What's new