Lead Data Engineer - DataLake/Data Warehousing

Key responsibilities : - Working with clients to understand their data. - Based on the understanding you will be building the data structures and pipelines. - You will be working on the application from end to end collaborating with UI and other development teams. - You will be responsible for building the data pipelines to migrate and load the data into the HDFSeither on-prem or in the cloud. - Developing Data ingestion/process/integration pipelines effectively. - Creating Hive data structures, metadata and loading the data into data lakes / Big Data warehouse environments. - Optimized(Performance tuning) many data pipelines effectively to minimize cost. - Code versioning control and git repository is up to date. - You will be responsible for building and maintaining CI/CD of the data pipelines. - You will be managing the unit testing of all data pipelines Skills & Experience : - Bachelor's degree in computer science or related field. - Minimum of 5years working experience with Spark, Hadoop eco systems. - Minimum of 4years working experience on designing data streaming pipelines. - Should be an expertin either Python/Scala/Java. - Should have experience in Data Ingestion and Integration into data lake using hadoop ecosystem tools such as Sqoop, Spark, SQL, Hive, Airflow, etc - Should have experience optimizing (Performance tuning) data pipelines. - Minimum experience of 3 years on NoSQL and Spark Streaming. - Knowledge of Kubernetes and Docker is a plus. - Should have experience with Cloud services either Azure/AWS. - Should have experience with on-prem distribution such as Cloudera/HortonWorks/MapR. - Basic understanding of CI/CD pipelines. - Basic knowledge of Linux environment and commands. (ref:hirist.tech)

Location: chennai, IN

Posted Date: 2/14/2025