Thank you for sending your enquiry! One of our team members will contact you shortly.
Thank you for sending your booking! One of our team members will contact you shortly.
Course Outline
Databricks Platform and Lakehouse Fundamentals
- Databricks Lakehouse architecture and components
- Organizing workspaces and catalogs
Databricks Workspace and Notebooks
- Workspace navigation and notebook-based development
- Structuring code into reusable notebooks
Apache Spark Architecture and Execution
- Spark runtime architecture and execution model
- Lazy evaluation and the job DAG
PySpark DataFrames and the DataFrame API
- DataFrame abstractions and schemas
- Core DataFrame operations and column expressions
Translating SQL to PySpark DataFrames
- Translating core SQL clauses to DataFrame operations
- Window functions and aggregations in PySpark
Reading and Writing Data in Databricks
- Reading from common file and database sources
- Writing and partitioning data in the Lakehouse
Delta Lake and Table Management
- Delta tables and ACID transactions
- Time travel and schema evolution
Data Cleaning and Transformation Patterns
- Data cleaning and type conversion
- Building reusable transformation logic
User-Defined Functions and Modular Code
- Python UDFs and pandas UDFs
- Modularizing procedural logic into functions
Performance Tuning and Optimization
- Partitioning and caching strategies
- Diagnosing bottlenecks with the Spark UI
Structured Streaming Fundamentals
- Batch versus streaming processing models
- Streaming DataFrames and basic aggregations
Databricks Jobs and Workflow Orchestration
- Scheduling notebooks as jobs and tasks
- Building multi-step workflows with dependencies
Unity Catalog and Data Governance
- Unity Catalog architecture and namespaces
- Access control and data lineage
Testing, Debugging, and Production Practices
- Unit testing PySpark logic
- Debugging and code quality standards
End-to-End Financial Services Use Cases
- Building an end-to-end banking ETL pipeline
- Translating legacy SQL processes to PySpark
Migrating SQL Workloads to PySpark
- Migration strategy and planning patterns
- Incremental conversion of SQL workflows to PySpark
Requirements
- Experience with Python programming, including functions and data types
- An understanding of SQL, including joins, aggregations, and subqueries
- No prior experience with Databricks or PySpark is required
Audience
- Data engineers, data analysts, and data professionals
- Teams migrating existing SQL-based workflows to Databricks and PySpark
35 Hours
Testimonials (1)
pre-training survey and the application of its results.
Krzysztof - Alfa Laval
Course - Python and Spark for Big Data (PySpark)
Machine Translated
