Thank you for sending your enquiry! One of our team members will contact you shortly.
Thank you for sending your booking! One of our team members will contact you shortly.
Course Outline
Introduction to LLMOps
- LLMOps vs MLOps: unique challenges of operating LLMs
- The LLM application lifecycle: prompt, evaluate, deploy, monitor
- Production readiness checklist for GenAI applications
Prompt Management and Versioning
- Prompt templating systems and variable injection
- Semantic versioning for prompts with automated regression testing
- Prompt registries and collaboration workflows
LLM Evaluation at Scale
- Evaluation dimensions: accuracy, relevance, safety, groundedness
- LLM-as-judge metrics and human evaluation pipelines
- Automated eval frameworks: RAGAS, DeepEval, and custom evaluators
- Quality gates in CI/CD for LLM deployments
Safety Guardrails and Content Governance
- Input and output guardrails: NeMo Guardrails and Guardrails AI
- PII detection, toxicity filtering, and topic boundaries
- Jailbreak and prompt injection defense strategies
- Red-teaming LLM applications for safety assurance
LLM Observability and Monitoring
- Telemetry: token usage, latency, cost, and quality metrics
- Drift detection in LLM outputs and embedding spaces
- Session-level tracing for multi-turn agent conversations
- Dashboards and alerting with LangSmith, Arize, and OpenTelemetry
AI Gateway and Model Orchestration
- Multi-provider routing with LiteLLM and Portkey
- Fallback strategies, retry logic, and circuit breakers
- Cost-aware model selection and load balancing
- Rate limiting, quota management, and API key governance
Performance Optimization
- Semantic caching with vector stores and exact-match strategies
- Structured output enforcement with constrained decoding
- Batching, streaming, and concurrency patterns
- Latency optimization across model providers
Governance, Compliance, and Audit
- LLM audit trails: prompt logs, response logs, and decision provenance
- Data residency and privacy considerations for LLM APIs
- Policy-as-code for LLM usage within organizations
- Building an internal LLM operations playbook
Requirements
- Experience building or integrating LLM-powered applications.
- Familiarity with Python and REST APIs.
- Basic understanding of prompt engineering concepts.
Audience
- ML engineers and MLOps practitioners transitioning to LLM operations.
- Platform engineers responsible for LLM infrastructure.
- Technical leads managing production GenAI deployments.
14 Hours
Testimonials (3)
Expanding knowledge about AI.
Kamil Lazowski - Alekseon
Course - Advanced Prompt Engineering for DeepSeek LLM
Machine Translated
The interactive style, the exercises
Tamas Tutuntzisz
Course - Introduction to Prompt Engineering
A great repository of resources for future use, instructor's style (full of good sense of humor, great level of detail)
