AWS Data Engineer - Databricks

IBM is actively seeking a highly skilled and experienced AWS Data Engineer-Databricks to join our dynamic team. This pivotal role is based in our Chennai and Pune offices, offering an exciting opportunity to contribute to cutting-edge data solutions. As an AWS Data Engineer, you will be instrumental in designing, building, and maintaining scalable data pipelines and architectures on the Amazon Web Services (AWS) platform, leveraging Databricks for advanced analytics and machine learning capabilities. You will play a crucial role in transforming raw data into actionable insights, driving innovation, and solving complex business challenges within a world-renowned technology leader.

We are looking for a motivated professional with a strong background in cloud data engineering, eager to make a significant impact on our data-driven initiatives. If you thrive in a collaborative environment and are passionate about leveraging cloud technologies to unlock the power of data, this opportunity at IBM is for you.

Key Responsibilities

• Design, develop, and maintain robust and scalable data pipelines on the AWS platform, ensuring optimal performance and data integrity.
• Perform complex data manipulations, including loading and extracting data from diverse sources into various schemas and formats.
• Implement data solutions using core AWS services such as S3, Redshift, EMR, Lambda, and Glue.
• Develop and deploy cloud-native applications, leveraging tools like CloudFormation and Service Catalog for infrastructure as code.
• Utilize Python, Spark, and PySpark for data processing, transformation, and analytical tasks within the Databricks environment.
• Monitor, troubleshoot, and debug cloud-based data applications and infrastructure to ensure high availability and reliability.
• Collaborate closely with data scientists, analysts, and other engineering teams to understand data requirements and deliver effective solutions.
• Apply best practices for AWS architecture, security (IAM), and operational excellence (CloudWatch).

Requirements

• Minimum of 5 years of hands-on experience in data engineering, with a strong focus on AWS cloud services.
• Solid understanding of core AWS services and fundamental AWS architecture best practices.
• Proven hands-on experience with AWS services including IAM, Glue, Redshift, EMR, Lambda, ECS, S3, State Machine, and CloudWatch.
• Demonstrated ability to perform data manipulations, including loading and extracting data from various sources into different schemas.
• Proficiency in programming languages such as Python, Spark, and PySpark for data processing.
• Experience in developing, deploying, and debugging cloud-based applications using infrastructure tools like CloudFormation and Service Catalog.
• Basic knowledge of Unix, shell scripting, and SQL for data querying and system interaction.

What We Offer

• An opportunity to work with cutting-edge technologies and contribute to innovative data solutions at a global leader.
• A collaborative and supportive work environment that fosters continuous learning and professional growth.
• Exposure to a diverse range of projects and challenges, enhancing your expertise as an AWS Data Engineer.
• The chance to be part of a team that drives significant impact through data-driven decisions.
• Competitive professional development opportunities and career advancement within IBM.