Data Engineer - Python/ETL

Qualifications : Minimum Qualifications: - Bachelor's Degree in Engineering, Computer Science, CIS, or related field (or equivalent work experience in a related field). - 5 years of experience in Data Engineering, Data Lake, Data Mesh, Data Warehousing/ETL. - 5 years of experience working on project(s) involving the implementation of solutions applying development life cycles (SDLC). - Experience working with Continuous Integration/Continuous Deployment tools with Git, Bitbucket, Jenkins. - 5 years of experience in systems analysis, including defining technical requirements and performing high level design for complex solutions. Data Engineering : - 5 years of experience in Hadoop, Google Cloud(GCP) and Bigdata components (specific to the Data Engineering role). - Expertise in advanced Python, advanced SQL, PySpark (Spark Batch and Streaming), Airflow, Kafka. - Exposure to REST API creation and usage. - Experience in functional programming, test first architecture (unit testing and regression testing), OOPS concepts(classes and objects). - Exposure in MicroStrategy, Oracle, MySql, SQL Server, DB2, Teradata is a plus. Key Responsibilities : - Design, implement, and develop data pipelines to collect and process large amounts of data from various sources. - Implement data storage solutions that are scalable, secure, and efficient, such as data warehouses and databases. - Develop and implement data validation and testing processes to ensure that data is processed accurately and efficiently. - Automate data collection, processing, and reporting processes to minimize manual work and improve efficiency. - Create high quality documents to capture problem statements, requirements, solutions and designs. - Contribute to the development of reusable, repeatable collateral for use across the practice. - Implement data quality checks and validation processes to ensure the accuracy, consistency, and security of data. - Monitor data pipelines and processes, troubleshoot issues, and implement solutions to prevent recurrence - Maintain comprehensive documentation of data architectures, processes, and workflows to facilitate knowledge sharing and collaboration. (ref:hirist.tech)

Location: bangalore, IN

Posted Date: 10/10/2024