Lead Data Engineer - Spark/Hadoop

Location : Chennai. Shift : 11AM 8 PM or 1PM 10PM. Skills : Python, Big-Data, spark/pyspark, databricks, ADF or ADL, team management experience. Role Description : - As a Technical Lead Big Data Platform, you will be responsible for designing and implementing big data platforms to support large-scale data integration projects. - You will work with data architects and data engineers to define the platform architecture and build the necessary infrastructure. - You will be skilled in big data technologies such as Hadoop, Spark, and Kafka and have experience in cloud computing platforms such as AWS or Azure. - You will be responsible for ensuring the performance, scalability, and security of the big data platform and troubleshooting any issues that arise. Roles & Responsibilities : - Designing, developing and maintaining large-scale big data platforms using technologies like Hadoop, Spark and Kafka. - Creating and managing data warehouses, data lakes and data marts. - Implementing and optimizing ETL processes and data pipelines. - Developing and maintaining security and access controls. - Troubleshooting and resolving big data platform issues. - Collaborating with other teams to ensure the consistency and integrity of data. Technical Skills : Skills Requirements : - Experience with big data processing technologies such as Apache Hadoop, Apache Spark, or Apache Kafka. - Understanding of distributed computing concepts such as MapReduce, Spark RDDs, or Apache Flink data streams. - Familiarity with big data storage solutions such as HDFS, Amazon S3, or Azure Data Lake Storage. - Knowledge of big data processing frameworks such as Apache Hive, Apache Pig, or Apache Impala. - Must have excellent communication skills and be able to communicate complex technical information to non-technical stakeholders in a clear and concise manner. - Must understand the company's long-term vision and align with it. - Should be open to new ideas and be willing to learn and develop new skills. - Should also be able to work well under pressure and manage multiple tasks and priorities. Qualifications : Qualifications : - 7-9 years of work experience in relevant field.Tech/B. (ref:hirist.tech)

Location: chennai, IN

Posted Date: 10/20/2024