Atacama DQ Tech Lead - Python/Big Data

Responsibilities : - Works closely with the business owner / Product owner / Cross functional Team and gathering technical requirements - Experience in building and maintaining reliable and scalable ETL pipeline on big data & / or Cloud platform through the collection, storage, processing, and transformation of large datasets - Collaborate with other teams to design, develop and deploy data tools that support both operations and product use cases - Work with the team in solving problems in big data technologies and prototype solutions to improve our data processing architecture Must Have : - Ensuring proper execution of tasks and alignment with business vision and objectives - Oversee activities of the junior data engineering teams - Works closely with the business owner / Product owner / Cross functional Team and gathering technical requirements - Experience in building and maintaining reliable and scalable ETL pipeline on big data/Cloud platform through the collection, storage, processing, and transformation of large data-sets - Experience working with varied forms of data infrastructure inclusive of relational databases such as SQL, Hadoop, Spark - Proficiency in scripting languages in PySpark, python - Experience in AWS cloud & DevOps - Experience in Databricks - Experience in Database design/data modelling - Must have strong experience in data warehouse concepts, understanding Data Lake, Lake House - Concepts, any new Big Data eco system. - Experience in testing and validation in order to support the accuracy of data transformations and data verification - Should be able to independently drive the requirements, have solutioning mindset facing business stakeholders Soft Skills : - Excellent Verbal and Communication skills needed. - Should be a excellent team player. - Good knowledge on Agile principles and experience working in scrum teams using Jira - Should be comfortable to mentor junior engineers in the team (For Senior Data Engineers) - Should be able to operate in an ambiguous environment with minimal guidance. (ref:hirist.tech)

Location: in, IN

Posted Date: 11/22/2024