Data Engineer - Spark/Hadoop

Job Description : Responsibilities : - Develop, construct, test, and maintain data architectures, including databases and large-scale processing systems. - Ensure data architecture will support the requirements of our business teams, product teams, and data scientists. - Implement secure and compliant data processing pipelines for optimal extraction, transformation, and loading of data from a wide variety of data sources. - Work closely with data scientists, business analysts, and IT teams to identify and implement internal process improvements, including automating manual processes, optimizing data delivery, and redesigning infrastructure for greater scalability. - Utilize big data tools and programming languages to handle data-related tasks. Requirements : - Proven experience in a Data Engineering role with at least 2 years of experience. - Good knowledge of data engineering concepts such as ETL pipelines, building a data lake, building data warehouses/ data marts, Stream and event processing, and distributed processing of large-scale data. - In-depth knowledge of data warehousing concepts and familiarity with at least one of the warehousing tools such as Google Bigquery, Amazon Redshift, Snowflake, and more. - Strong SQL knowledge and understanding of performance tuning techniques across data infrastructure. - Strong analytical skills to break down problems and build the right and optimized solutions. - Good knowledge of big data tools such as Hadoop, Apache Spark, Apache Druid, etc. - Good exposure to streaming technologies like Spark Streaming, Kafka, SQS, Kinesis, etc. - Good exposure to data tools: Storage (HDFS, S3), Processing Architecture (EC2 EMR, Glue), Data Repository (Glue), Orchestration and Automation (Airflow, AWS Step Functions), Modern File Formats (Parquet, Delta, Iceberg), Datalake and Datawarehouse (Athena, Redshift, Hive). - Good understanding of SQL and NoSQL databases. - Experience with Cloud Platforms like Amazon Web Services or Google Cloud Platform. - Proficient in Python, Scala, or Java. - Experience in MLOps is a plus. - Continuously evaluate and implement new technologies to improve data infrastructure. - High sense of ownership and strong decision-making skills. (ref:hirist.tech)

Location: bangalore, IN

Posted Date: 11/23/2024