2 Candidate Submittal Slots, New High Level Policy
MSP Owner: Rob Finton
Location: Washington, DC Metro Area - Hybrid - strong preference will be given to local candidates to Washington Dc metro area. Need to attend the office in person based on the need.
Duration: 6 months
skill id: 10787972
Competencies: 15+ years experience required
Digital : Databricks
Job description:
Design and develop scalable ETL/ELT pipelines using Databricks including Delta Lake, Auto Loader, and DLT.
Develop batch, streaming, and CDC ingestion pipelines to enable efficient and near real-time data processing.
Prior experience with data migration from various data sources like Redshift, DB2, Oracle, Mysql, PLSSql in to databricks
Implement ingestion patterns using Auto Loader with checkpointing and schema evolution for structured and semi-structured data
Develop and maintain reusable DBX functions for data transformation, data quality checks, and validation logic, ensuring consistent implementation across DLT and notebook-based pipelines.
Leverage Unity Catalog to establish secure data governance, enabling controlled access, lineage tracking, and management of External Locations and Volumes across workspaces.
Partner with cross-functional teams to enable secure data sharing using Delta Sharing across internal domains and external partners.
Integrate Power BI/Tableau/Looker with Databricks using optimized connectors (ODBC/JDBC) and Unity Catalog security controls
Implement Medallion architecture (Bronze, Silver, Gold layers).
Develop incremental and CDC-based ingestion pipelines
Design and implement real-time streaming pipelines using Kafka and Structured Streaming.
Optimize Spark jobs, SQL queries, and streaming pipelines.
Strong SQL and data modeling skills.
Experience with cloud platforms and distributed systems.
Build, package, and deploy Databricks data pipelines using Databricks Asset Bundles (DAB), with automated CI/CD processes managed through GitHub or GitLab., Project Code :
MSP Owner: Rob Finton
Location: Washington, DC Metro Area - Hybrid - strong preference will be given to local candidates to Washington Dc metro area. Need to attend the office in person based on the need.
Duration: 6 months
skill id: 10787972
Competencies: 15+ years experience required
Digital : Databricks
Job description:
Design and develop scalable ETL/ELT pipelines using Databricks including Delta Lake, Auto Loader, and DLT.
Develop batch, streaming, and CDC ingestion pipelines to enable efficient and near real-time data processing.
Prior experience with data migration from various data sources like Redshift, DB2, Oracle, Mysql, PLSSql in to databricks
Implement ingestion patterns using Auto Loader with checkpointing and schema evolution for structured and semi-structured data
Develop and maintain reusable DBX functions for data transformation, data quality checks, and validation logic, ensuring consistent implementation across DLT and notebook-based pipelines.
Leverage Unity Catalog to establish secure data governance, enabling controlled access, lineage tracking, and management of External Locations and Volumes across workspaces.
Partner with cross-functional teams to enable secure data sharing using Delta Sharing across internal domains and external partners.
Integrate Power BI/Tableau/Looker with Databricks using optimized connectors (ODBC/JDBC) and Unity Catalog security controls
Implement Medallion architecture (Bronze, Silver, Gold layers).
Develop incremental and CDC-based ingestion pipelines
Design and implement real-time streaming pipelines using Kafka and Structured Streaming.
Optimize Spark jobs, SQL queries, and streaming pipelines.
Strong SQL and data modeling skills.
Experience with cloud platforms and distributed systems.
Build, package, and deploy Databricks data pipelines using Databricks Asset Bundles (DAB), with automated CI/CD processes managed through GitHub or GitLab., Project Code :