Thank you for sending your enquiry! One of our team members will contact you shortly.
Thank you for sending your booking! One of our team members will contact you shortly.
Duration 14 hours
Course Outline
Deployment Options
- Trade-offs between hosted APIs and self-hosted solutions.
- Patterns for on-premises, edge, and air-gapped environments.
- Single model vs. model portfolio approaches.
- Comparison of cost per decision.
Hardware Sizing and Capacity
- Memory footprint relative to parameter count.
- CPU-only vs. GPU acceleration strategies.
- Planning for concurrency and throughput.
- Accounting for headroom in task length growth.
Serving Stacks
- Exploring vLLM and SGLang options.
- Using llama.cpp and GGUF execution formats.
- Leveraging Ollama for local development.
- Selecting stacks based on decision workload characteristics.
Quantization Without Losing Calibration
- Precision formats and associated memory savings.
- Measuring quality and calibration loss.
- Per-tensor and importance-aware quantization variants.
- Acceptance testing procedures post-quantization.
Latency Engineering
- Distinguishing median vs. tail latency.
- Impact of context length on latency growth.
- Optimizing warm start and model caching.
- Batching multiple decision queries.
Edge and Constrained Environments
- Running small decision models on CPU.
- Deployment on Apple Silicon and embedded devices.
- Strategies for offline operation and updates.
- Distributing weights in air-gapped settings.
Containerization and Scaling
- Designing images for model serving.
- Managing volumes and caches.
- Horizontal scaling and routing strategies.
- Setting resource limits and sharing GPUs.
Observability
- Tracking request, latency, and error metrics.
- Monitoring confidence distributions.
- Tracking decision outcomes and escalations.
- Alerting on calibration drift.
Versioning and Rollout
- Versioning models and adapters.
- Implementing shadow and canary deployments.
- Pinning model files for audit purposes.
- Establishing rollback procedures.
Security and Multi-Tenancy
- Implementing authentication and rate limiting.
- Ensuring tenant isolation for prompts and logs.
- Network isolation and egress control.
- Audit logging of decision events.
Requirements
- Experience with Linux administration and containerization.
- Knowledge of model inference servers and GPU management.
- Foundational understanding of REST APIs, load balancing, and monitoring.
Target Audience
- MLOps and platform engineers managing inference infrastructure.
- SRE teams supporting AI services.
- AI engineers deploying decision models on-premises.
- Architects designing air-gapped or regulated AI deployments.
