2 Candidate Submittal Slots, New High Level Policy MSP Owner: Rob Finton Location: Washington, DC - Remote position but candidate may have to travel onsite to DC occasionally based on need. Duration: 6 months skill id: 10787969 Need associates with at least 10 years of work exp with a minimum of 5 years of exp as data engineer Skills: 8-10+ years experience required MySQL Databricks Pyspark Job description: Design and develop scalable ETL/ELT pipelines using Databricks and PySpark. Build batch and real-time streaming ingestion frameworks. Develop reusable ingestion and transformation frameworks. Implement Medallion architecture (Bronze, Silver, Gold layers). Develop incremental and CDC-based ingestion pipelines Design and implement real-time streaming pipelines using Kafka and Structured Streaming. Optimize Spark jobs, SQL queries, and streaming pipelines. Implement Delta Lake-based ingestion and transformation frameworks. Tune partitioning, caching, and Spark execution strategies. Strong SQL and data modeling skills. Experience with cloud platforms and distributed systems. Familiarity with CI/CD pipelines and DevOps practices., Project Code :

Information Technology_USA - USA_Developer