Azure Data Engineer - ETL Tools

Exp -7 to 10 years. Responsibilities - Design, develop, and maintain data pipelines using Databricks and Spark, and other cloud technologies as needed. - Optimize data pipelines for performance,scalability, and reliability. - Ensure data quality and integrity throughout the data lifecycle. - Collaborate with data scientists, analysts, and other stakeholders to understand and meet their data needs. - Troubleshoot and resolve data-related issues, and provide root cause analysis and recommendations. - Document data pipeline specifications, requirements, and enhancements, and communicate them effectively to the team and management. - Create new data validation methods and data analysis tools, and share best practices and learnings with the data engineering community. - Implement ETL processes and data warehouse solutions, and ensure compliance with data governance and security policies. Qualifications : - Bachelor's degree in Computer Science, Engineering, or related field, or equivalent work experience. - 5 years of experience in data engineering with Databricks and Spark. - Proficient in SQL and Python and Pyspark. - Experience with Azure Databricks Medallion Architecture with DLT, Iceberg. - Financial/Corporate Banking context would be a plus. - Experience with data integration and ETL tools, such as Azure Data Factory. - Experience with Azure cloud platform and services. - Experience with data warehouse and data lake concepts and architectures. - Good to have experience with big data technologies, such as Kafka, Hadoop, Hive, etc. - Strong analytical and problem-solving skills. - Excellent communication and teamwork skills. Requirements : - Strong knowledge on data bricks architecture and tools. - Have experience of task and wf jobs creations in data bricks. - Deep understanding of distributed computing and how to use spark for data processing. - SQL and Pyspark strong command over querying databases and proficiency in Pyspark. Cloud platform : - Preferred Azure for data bricks deployment. (ref:hirist.tech)

Location: in, IN

Posted Date: 11/22/2024