Senior Data Engineer

Are you an experienced Data Engineer passionate about crafting cutting-edge data and AI solutions? Birlasoft is seeking a highly skilled Senior Data Engineer to join our dynamic team in Pune. In this pivotal role, you will be instrumental in building enterprise-scale Data & AI transformation solutions for our global clientele, leveraging advanced Lakehouse platforms and robust data engineering practices. We are looking for an individual who thrives on challenges and is eager to contribute to innovative projects that drive real business value.

As a Senior Data Engineer, you will be responsible for the end-to-end development, optimization, and governance of scalable data pipelines and architectural frameworks. Your expertise will directly impact our ability to deliver high-performance, secure, and reliable data solutions that empower our customers' digital transformation journeys.

Key Responsibilities

• Design, develop, and implement robust and scalable data pipelines using the Databricks Lakehouse Platform.
• Drive PySpark development and performance optimization for large-scale data processing workloads.
• Implement and manage Unity Catalog for comprehensive data governance, security, and access control.
• Architect and evolve data solutions leveraging Delta Lake and the Lakehouse Architecture, including Medallion Architecture patterns.
• Conduct performance tuning and optimization of data pipelines to ensure efficient processing of large datasets.
• Collaborate with cross-functional teams to understand data requirements and translate them into technical specifications.
• Ensure the quality, reliability, and integrity of data across various stages of the data lifecycle.
• Provide technical leadership and mentorship to junior team members, fostering a culture of excellence and continuous learning.

Requirements

• Minimum of 6 to 12 years of hands-on experience in data engineering and solution development.
• Proven expertise with the Databricks Lakehouse Platform is essential.
• Strong proficiency in PySpark development, including advanced knowledge of performance optimization techniques.
• In-depth understanding and practical experience with Unity Catalog for data governance and security.
• Extensive experience with Delta Lake and implementing Lakehouse Architecture principles, including Medallion Architecture.
• Demonstrated ability in designing and developing scalable data pipelines and managing large-scale data processing.
• Solid programming skills in Python for data manipulation and scripting.
• Proficiency in SQL for data querying and analysis.
• Experience with cloud platforms such as AWS or Azure is highly desirable.
• Excellent problem-solving skills and a strong analytical mindset.

What We Offer

• An opportunity to work on cutting-edge Data & AI transformation solutions for global customers.
• A collaborative and innovative work environment that fosters professional growth.
• Exposure to the latest technologies and methodologies in data engineering.
• Competitive compensation package and opportunities for career advancement.
• A chance to be a part of a leading IT services company with a strong commitment to employee development.