AI Data Engineer

IBM is actively seeking an experienced Data Engineer specializing in Data Platforms built on AWS and leveraging Databricks. This crucial role involves designing, developing, and maintaining robust data engineering solutions within the dynamic AWS Cloud ecosystem. If you are passionate about building scalable data pipelines and managing diverse data layers, and thrive in an environment that values innovation and technical excellence, we invite you to join our team in Pune or Chennai.

As a Data Engineer at IBM, you will be instrumental in advising on, developing, and maintaining sophisticated data engineering solutions. Your expertise will be vital in designing, building, and operating both batch and real-time data pipelines, ensuring seamless data flow and integrity across various AWS services. This role demands a strong understanding of data architecture, cloud infrastructure, and a commitment to delivering high-quality, performant data solutions.

Key Responsibilities

• Design, build, and operate advanced batch and real-time data pipelines using a suite of AWS services including, but not limited to, AWS EMR, AWS Glue, and Kinesis, ensuring efficient and reliable data processing.
• Create, manage, and optimize diverse data layers on AWS platforms such as Amazon Redshift, Aurora, and DynamoDB, catering to various data storage and retrieval needs.
• Execute complex data migration projects utilizing AWS Database Migration Service (AWS DMS) to ensure secure and efficient transfer of data across systems.
• Collaborate with cross-functional teams to understand data requirements and translate them into scalable and resilient data engineering solutions.
• Monitor, troubleshoot, and optimize data pipeline performance, ensuring high availability and data quality for critical business operations.
• Provide expert advice and technical guidance on best practices for data platform development and management on AWS.

Requirements

• A minimum of 6 years of proven professional experience in Data Engineering, with a strong focus on cloud-based data platforms.
• Extensive hands-on experience with AWS services, including AWS EMR, AWS Glue, Kinesis, Amazon Redshift, Aurora, DynamoDB, and AWS DMS.
• Demonstrated expertise in designing, building, and optimizing both batch and real-time data pipelines.
• Proficiency in working with data platforms like Databricks for data processing and analytics.
• Solid understanding of data modeling, ETL/ELT processes, and data warehousing concepts.
• Excellent problem-solving skills and the ability to work independently as well as collaboratively within a team.
• Strong communication and advisory skills to effectively guide stakeholders and team members.

What We Offer

• An opportunity to work with IBM, a global leader in technology and consulting, on cutting-edge data platforms and cloud technologies.
• Engagement in impactful projects that drive significant business value and innovation.
• A collaborative and dynamic work environment in our Pune and Chennai offices.
• Continuous learning and professional development opportunities in AWS and Databricks.
• A chance to grow your career as a pivotal Data Engineer within a highly skilled team.