Databricks Engineer

York Solutions is actively seeking a highly skilled and experienced Azure Databricks Senior Developer to join our dynamic, fully remote team. This pivotal Databricks Engineer role offers an exciting opportunity to design, build, and optimize cutting-edge cloud-native data engineering solutions, primarily leveraging the robust capabilities of the Azure platform and Databricks. If you possess a strong background in data warehousing, enterprise-grade ETL/ELT pipeline development, and a passion for transforming complex data into actionable insights using Spark, Scala, and SQL, we encourage you to explore this challenging and rewarding position.

As an Azure Databricks Senior Developer, you will be instrumental in architecting and implementing scalable data solutions that drive business intelligence and operational efficiency. Your expertise will be crucial in advancing our data infrastructure and ensuring its performance and reliability within a fast-paced environment.

Key Responsibilities

• Design, develop, and optimize cloud-native data engineering solutions on the Microsoft Azure platform, with a strong focus on Azure Databricks.
• Architect, build, and maintain robust ETL/ELT pipelines for batch and real-time data processing, ensuring data quality and integrity.
• Perform advanced Apache Spark and Databricks development, utilizing Scala for optimal performance and complex data transformations.
• Implement sophisticated SQL-based transformations, performance tuning, and optimization strategies for large-scale relational and non-relational datasets.
• Develop and apply data streaming concepts and patterns, including Spark Structured Streaming, to handle high-velocity data.
• Actively participate in Agile delivery models, collaborating closely with data architects, business analysts, and upstream/downstream application teams to deliver integrated solutions.
• Contribute to best practices, code reviews, and the continuous improvement of our data engineering processes and infrastructure.
• Troubleshoot and resolve complex data-related issues, ensuring minimal disruption and high availability of data services.

Requirements

• Minimum of 5 years of hands-on experience in Data Engineering or Data Warehousing development and operations.
• Proven expertise in Azure Databricks, Azure Data Lake Storage (ADLS), and a deep understanding of Apache Spark, including core concepts and performance tuning.
• Strong proficiency in Scala and SQL for advanced data manipulation, scripting, and querying.
• Demonstrable experience with data streaming technologies and design patterns, such as Spark Structured Streaming, for real-time data ingestion and processing.
• Familiarity with Python and PySpark for data processing and scripting is highly desirable.
• General knowledge of the Microsoft Azure platform and its core services is a significant advantage.
• Prior experience within the Healthcare domain, particularly with claims, clinical data, eligibility, or HIPAA-aware data handling, is strongly preferred.
• Ability to thrive in an Agile development environment and excellent collaboration skills with diverse technical and business teams.
• Exposure to NoSQL databases, specifically MongoDB, is a nice-to-have skill.

What We Offer

• A 100% remote work environment, providing flexibility and a healthy work-life balance.
• Opportunity to work on challenging and impactful projects at the forefront of cloud data engineering and analytics.
• Competitive compensation package designed to attract and retain top-tier talent.
• Access to continuous learning and professional development resources, including certifications and training in cutting-edge technologies.
• A collaborative and supportive company culture that values innovation, technical excellence, and the well-being of its employees.
• Engage with a team of experienced professionals dedicated to solving complex data challenges and driving significant business value.

If you are an experienced Azure Databricks Senior Developer ready to make a significant impact and lead innovative data initiatives, we encourage you to apply today. Join York Solutions and help us build the future of data!