Data Engineer - Spark/Hadoop

Project Role : Data Engineer Project Role Description : - Develops applications and systems that utilize AI tools, Cloud AI services, with proper cloud or on-prem application pipeline with production ready quality. Be able to apply GenAI models as part of the solution. Could also include but not limited to deep learning, neural networks, chatbots, image processing. Must have skills : Data Engineering Good to have skills : NA Minimum 0-3 year(s) of experience is required Educational Qualification : BE Summary : As a Data Engineer, you will be responsible for developing applications and systems that utilize AI tools and Cloud AI services. Your typical day will involve building and maintaining a proper cloud or on-prem application pipeline with production-ready quality, applying GenAI models as part of the solution, and working with deep learning, neural networks, chatbots, and image processing. Roles & Responsibilities : - Design, develop, and maintain data pipelines and ETL processes to support data science and machine learning workflows. - Collaborate with data scientists and machine learning engineers to understand their requirements and provide scalable solutions. - Implement and maintain data storage solutions, including data lakes, data warehouses, and NoSQL databases. - Develop and maintain data quality and data governance processes to ensure data accuracy, completeness, and consistency. - Create and maintain documentation for data pipelines, ETL processes, and data storage solutions. Professional & Technical Skills : - Must To Have Skills : Experience with data engineering tools and technologies such as Apache Spark, Hadoop, and SQL. - Must To Have Skills : Experience with cloud-based data storage and processing services such as AWS S3, Redshift, and EMR. - Good To Have Skills : Experience with AI tools and technologies such as TensorFlow, Keras, and PyTorch. - Good To Have Skills : Experience with containerization technologies such as Docker and Kubernetes. - Strong understanding of data modeling, data warehousing, and data governance principles. - Experience with data visualization tools such as Tableau or Power BI. Additional Information: - The candidate should have a minimum of 3 years of experience in Data Engineering. - The ideal candidate will possess a strong educational background in computer science, data science, or a related field, along with a proven track record of delivering impactful data-driven solutions. - This position is based at our Bengaluru office. Qualification : BE (ref:hirist.tech)

Location: bangalore, IN

Posted Date: 11/11/2024