Thank you for sending your enquiry! One of our team members will contact you shortly.
Thank you for sending your booking! One of our team members will contact you shortly.
Course Outline
Introduction to EXO and Local AI Clustering
- Overview of the EXO framework and the exo-explore ecosystem
- Comparison between centralized cloud inference and distributed local inference
- Architecture deep dive: libp2p device discovery, MLX backend, dashboard, and API layers
- Hardware specifications: Apple Silicon (M3 Ultra, M4 Pro/Max), Thunderbolt 5 connectivity, and shared storage requirements
Installing EXO on macOS
- Setting up prerequisites including Xcode, Metal ToolChain, and macOS configurations.
- Installation of uv, Node.js, and the Rust nightly toolchain.
- Installation of the specific macmon fork tailored for Apple Silicon monitoring.
- Cloning the repository and compiling the dashboard using npm.
- Launching EXO from source and verifying access via the localhost:52415 dashboard.
Installing EXO on Linux
- Installing necessary dependencies through apt or Homebrew on Linux systems.
- Configuring uv, Node.js 18+, and the Rust nightly compiler.
- Building the dashboard and launching EXO in CPU-only mode.
- Understanding the directory structure based on XDG Base Directory standards for configuration, data, cache, and logs.
Automatic Device Discovery and Cluster Formation
- Comprehending libp2p-based auto-discovery mechanisms across local networks.
- Configuring custom namespaces using EXO_LIBP2P_NAMESPACE to ensure cluster isolation.
- Verifying node membership through the dashboard's cluster view.
- Addressing discovery failures and network segmentation challenges.
Enabling RDMA over Thunderbolt 5
- Understanding RDMA architecture and its claim of a 99% latency reduction.
- Enabling RDMA capabilities in macOS Recovery mode using rdma_ctl.
- Requirements for cabling and port topology constraints specific to Mac Studio devices.
- Ensuring version parity for macOS across all cluster nodes.
- Troubleshooting RDMA discovery issues and DHCP configuration problems.
Deploying Frontier Models
- Utilizing the dashboard to load and shard models such as DeepSeek v3.1, Qwen3-235B, and various Llama family models.
- Previewing instance placements via the /instance/previews API endpoint.
- Creating model instances using either pipeline or tensor-parallel sharding techniques.
- Configuring custom model cards sourced from the HuggingFace hub.
Monitoring and Troubleshooting
- Reading EXO logs and comprehending distributed tracing mechanisms.
- Interpreting cluster health metrics within the dashboard's cluster view.
- Diagnosing worker node failures and understanding reconnection behaviors.
- Leveraging EXO_TRACING_ENABLED for performance bottleneck analysis.
Cluster Maintenance and Updates
- Procedures for updating EXO binaries and rebuilding the dashboard.
- Migrating model caches and managing pre-downloaded models over NFS.
- Gracefully removing nodes and rebalancing workloads across the cluster.
Requirements
- A solid understanding of networking fundamentals, including IP addressing, subnetting, and firewalls.
- Practical experience with command-line administration on macOS or Linux.
- Familiarity with Python package management (pip/uv) and Node.js tooling.
Target Audience
- System administrators
- DevOps engineers
- AI infrastructure architects tasked with on-premise LLM deployment
21 Hours
