Data Engineer - Hive/Spark

Job Description : Key Responsibilities : - Design and implement scalable data pipelines to support big data transformations and analytics. - Work with cloud platforms such as Azure (preferred), AWS, or Google Cloud to deploy and manage data systems. - Utilize orchestration tools like Azure Data Factory and other cloud-native services for data pipeline automation. - Write efficient, high-performance scripts using SQL, PySpark, Scala, and Python (Databricks experience preferred). - Work with big data technologies like Hadoop, Spark, Hive, Snowflake, and Kafka to process and manage large datasets. - Design and implement dimensional data models for data warehouses and data marts. - Use monitoring and logging tools (Prometheus, Grafana, Splunk) to ensure system performance and troubleshoot issues. - Collaborate with cross-functional teams using project management tools like JIRA and Azure DevOps. - Develop reports and dashboards using Power BI or similar visualization tools to deliver actionable insights. Qualifications : - Bachelor's degree in Computer Science, Engineering, or a related field. - 5-7 years of experience in data engineering, specializing in scalable data transformations and pipeline design. - Strong proficiency with cloud platforms (Azure preferred), including experience with services like Azure Data Factory. - Expertise in scripting languages such as SQL, Python, PySpark, and/or Scala; experience with Databricks is a plus. - Experience with big data technologies including Hadoop, Spark, Hive, Snowflake, and Kafka. - Strong understanding of dimensional modeling and experience designing data warehouses and data marts. - Experience with monitoring/logging tools such as Prometheus, Grafana, or Splunk. - Familiarity with project management tools like JIRA and Azure DevOps. - Proficiency in Power BI or similar data visualization tools to create insightful reports and dashboards. Must-Haves : - Acknowledges the presence of choice in every moment and takes personal responsibility for their life. - Possesses an entrepreneurial spirit and continuously innovates to achieve great results. - Communicates with honesty and kindness, creating space for others to do the same. - Leads with courage, knowing the possibility of greatness is bigger than the fear of failure. - Fosters connection by putting people first and building trusting relationships. - Integrates fun and joy as a way of being and working, i.e., doesn't take themselves too seriously. (ref:hirist.tech)

Location: bangalore, IN

Posted Date: 11/9/2024