Lead Data Engineer - Microsoft Fabric

Lead Data Engineer-MS Fabric. Exp : 10 years. Type : Contract(remote). Shift Time : 3PM to 12AM IST. Description : We are looking for a Lead Data Engineer with extensive experience in developing ETL processes using PySpark Notebooks and Microsoft Fabric and supporting existing legacy SQL Server environments. The ideal candidate will possess a strong background in Spark-based development, demonstrate a high proficiency in SQL, and be comfortable working independently, collaboratively within a team, or leading other developers when required. - Design, develop, and maintain ETL pipelines using PySpark Notebooks and Microsoft Fabric. - Collaborate with data scientists, analysts, and other stakeholders to understand data requirements and deliver efficient data solutions. - Migrate and integrate data from legacy SQL Server environments into modern data platforms. - Optimize data pipelines and workflows for scalability, efficiency, and reliability. - Provide technical leadership and mentorship to junior developers and other team members. - Troubleshoot and resolve complex data engineering issues related to performance, data quality, and system scalability. - Develop, maintain, and enforce data engineering best practices, coding standards, and documentation. - Conduct code reviews and provide constructive feedback to improve team productivity and code quality. - Support data-driven decision-making processes by ensuring data integrity, availability, and consistency across different platforms. Requirements : - Bachelor's or Master's degree in Computer Science, Data Science, Engineering, or a related field. - Experience with Microsoft Fabric or similar cloud-based data integration platforms is a must. - 10 years of experience in data engineering, with a strong focus on ETL development using PySpark or other Spark-based tools. - Proficiency in SQL with extensive experience in complex queries, performance tuning, and data modeling. - Strong knowledge of data warehousing concepts, ETL frameworks, and big data processing. - Familiarity with other data processing technologies (e.g, Hadoop, Hive, Kafka) is an advantage. - Experience working with both structured and unstructured data sources. - Excellent problem-solving skills and the ability to troubleshoot complex data engineering issues. - Proven ability to work independently, as part of a team, and in leadership roles. - Strong communication skills with the ability to translate complex technical concepts into business terms. Mandate skills : - Experience with Data lake, Data warehouse, Delta lake. - Experience with Azure Data Services, including Azure Data Factory, Azure Synapse, or similar tools. - Knowledge of scripting languages (e.g, Python, Scala) for data manipulation and automation. - Familiarity with DevOps practices, CI/CD pipelines, and containerization (Docker, Kubernetes) is a plus. (ref:hirist.tech)

Location: bangalore, IN

Posted Date: 11/21/2024