Data Engineer - ETL/Apache Airflow

Job Description and Responsibilities : Solicit business, functional, non-functional and technical requirements through interviewing and requirements gathering process. Analyze and document above requirements and data definitions, perform data analysis, assist in change management, training and testing efforts. Works with stakeholders to gather requirements on merging, de-duplicating, standardizing data. Develop, support, and refine new data pipelines, data models, business logic, data schemas as code, and analytics to product specifications. Prototype and optimize data type checks to ensure data uniformity prior to load. Develop, and refine batch processing data pipeline frameworks. Maintain, improve, and develop expertise in existing production data models, and algorithms. Learn and utilize business data domain knowledge and its correlation to underlying data sources. Define, document, and maintain a data dictionary including data definitions, data sources, business meaning and usage of information. Identify and validate opportunities to reuse existing data and algorithms. Collaborate on design and implementation of data standardization procedures. Share team responsibilities, such as contributing to development of data warehouses and productizing algorithms created by Data Science team members. Participate in on-call and weekly shift rotation. Skills Set : 4 - 6 years of experience building data pipelines and using ETL tools(Must-have). 2 years of experience in ETL tools like Talend /Jaspersoft ETL tools(Must-have) . 2 years of experience in SQL programming language (Must-have) . Strong in writing stored procedures and sql queries(Must-have). 2 years of experience in python programming (Must-have) . Sound knowledge of distributed systems and data processing with spark. Knowledge of any tool for scheduling and orchestration of data pipelines or workflows (preferred Airflow)(must to have) 1 years experience developing modern, industry standard big data frameworks with AWS or other cloud with common GitHub developer practices and paradigms. Experience working with agile methodologies and cross-functional teams. Knowledge in building AWS data pipelines using python, S3 data lake(nice to have). Knowledge of redshift or any other columnar database is preferred. Experience with AWS services including S3, Redshift, EMR (nice to have) Knowledge of distributed systems as it pertains to data storage and computing Knowledge of specialty pharmaceutical and retail pharmacy is a plus. Good to have knowledge of the Data integration process. Ability to effectively communicate with both business and technical teams Professional Approach : Excellent verbal, written communication. Flexible with the working timing. Required Qualification : Bachelor of Engineering - Bachelor of Technology (B.E./B. Tech.) (ref:hirist.tech)

Location: pune, IN

Posted Date: 11/22/2024