AI Observability & Tracing

This course brings observability discipline to AI systems: tracing multi-step calls, structured logging, and debugging chains in production. Learners instrument a working AI app end to end.

What this course covers

The course runs across 8 topics, each one a short adaptive session rather than a recorded lecture. The tutor explains the idea, works an example, checks understanding, and adjusts the next step to the answer given.

Introduction to AI Observability & Tracing Architecture — Understand the unique challenges of observing LLM applications compared to traditional APM, and explore the architecture of modern AI tracing frameworks.
Instrumenting and Tracing LLM Calls — Learn to instrument single and multi-step LLM calls using open-source tracing SDKs to capture inputs, outputs, and execution metadata.
Structured Logging for Generative AI — Implement structured logging schemas to capture system prompts, user queries, token counts, and model parameters in a searchable format.
Monitoring Key LLM Metrics — Define and monitor critical operational metrics, including time-to-first-token (TTFT), latency, error rates, and throughput across different models.
Debugging Complex Chains and Multi-Agent Workflows — Visualize execution graphs and debug nested chains, retrieval-augmented generation (RAG) steps, and agentic tool-use loops.
Cost Tracking and Token Budgeting — Track token consumption in real-time, attribute API costs to specific users or features, and set up alerts for budget overruns.
Evaluating Output Quality and Hallucinations — Set up automated evaluation runs on traces to assess context relevance, faithfulness, and potential hallucinations in production environments.
Capstone: End-to-End Application Instrumentation — Apply your skills to instrument, debug, and optimize a live multi-step AI application, resolving latency bottlenecks and tracking its costs.

Skills you build

Results are measured against named skills in the OneRange taxonomy of more than 10,000 skills, so a manager sees proficiency per skill rather than a completion tick.

Who it is for

ML/platform engineers. The material is pitched at advanced level, and takes roughly 120 minutes at a typical pace. Because every session adapts, someone who already knows a topic moves through it quickly instead of sitting through an explanation they do not need.

It sits in the Category: Deployment track of the Vero AI Catalog, and can be assigned to one person, a team, or the whole company.

How it is delivered and assessed

Delivery is conversational and interactive, including code lab, guided lab exercises. There is no video to sit through and no slide deck to click past.

Understanding is checked with a code assessment of 5 items, with a pass mark of 70%.

Administrators can copy this course into their own library and adapt it — edit the outline, change the duration, swap the assessment format, or ground it in internal documentation so answers cite the company's own source material.

Also in this track

LLMOps & Production Deployment — This course teaches ML and platform engineers to take AI features from prototype to reliable production. It covers deployment patterns, monitoring and logging, versioning, guardrails, and incident response. Learners finish able to run AI in production with confidence.
Modern AI Stack for Builders — Master the integration layer of modern AI systems

Frequently asked questions

How long does this course take?

It runs roughly 120 minutes at a typical pace. Because every session adapts, someone who already knows a topic moves through it quickly instead of sitting through an explanation they do not need.

How is it assigned to a team?

An administrator assigns it from the Vero dashboard to one person, a team, a department or the whole company, and sees progress and results per person and per skill.

Can we customise it with our own documents and terminology?

Yes. Administrators can copy this course into their own library and adapt it — edit the outline, change the duration, swap the assessment format, or ground it in internal documentation so answers cite the company's own source material.

Course code CAT-OBS.