Course Outline
Fundamentals of the Databricks Platform and Lakehouse Architecture
- Overview of Databricks Lakehouse components and architecture.
- Strategies for organizing workspaces and catalogs.
Databricks Workspace and Notebook Management
- Navigating the workspace and developing through notebooks.
- Structuring code for reusability across notebooks.
Apache Spark Architecture and Execution Mechanics
- Understanding Spark runtime architecture and its execution model.
- Concepts of lazy evaluation and job DAGs.
PySpark DataFrames and the DataFrame API
- DataFrame abstractions and schema definitions.
- Core operations and column expressions within DataFrames.
Translating SQL to PySpark DataFrames
- Mapping core SQL clauses to DataFrame operations.
- Implementing window functions and aggregations in PySpark.
Data Ingestion and Output in Databricks
- Loading data from various file formats and database sources.
- Writing and partitioning data within the Lakehouse.
Delta Lake and Table Administration
- Working with Delta tables and ACID transactions.
- Utilizing time travel features and schema evolution.
Patterns for Data Cleaning and Transformation
- Techniques for data cleaning and type conversion.
- Developing reusable logic for transformations.
User-Defined Functions and Modular Coding
- Implementing Python UDFs and pandas UDFs.
- Encapsulating procedural logic into functions for modularity.
Performance Tuning and Optimization Techniques
- Strategies for partitioning and caching.
- Identifying bottlenecks using the Spark UI.
Fundamentals of Structured Streaming
- Comparing batch processing with streaming models.
- Working with streaming DataFrames and basic aggregations.
Databricks Jobs and Workflow Orchestration
- Scheduling notebooks as jobs and tasks.
- Constructing multi-step workflows with defined dependencies.
Unity Catalog and Data Governance
- Architecture of Unity Catalog and namespaces.
- Managing access control and data lineage.
Testing, Debugging, and Production Standards
- Conducting unit tests for PySpark logic.
- Maintaining debugging practices and code quality standards.
End-to-End Use Cases in Financial Services
- Designing a complete banking ETL pipeline.
- Converting legacy SQL processes to PySpark implementations.
Migrating SQL Workloads to PySpark
- Strategies and planning patterns for migration.
- Incremental conversion of SQL workflows to PySpark.
Requirements
- Proficiency in Python programming, encompassing functions and data types.
- Knowledge of SQL, including joins, aggregations, and subqueries.
- No prior exposure to Databricks or PySpark is necessary.
Target Audience
- Data engineers, data analysts, and related data professionals.
- Teams currently transitioning SQL-based workflows to Databricks and PySpark.
Testimonials (1)
I liked that it was practical. Loved to apply the theoretical knowledge with practical examples.
