Big Data Engineer - PySpark/Scala

Key Responsibilities : Data Pipeline Development : - Design, develop, and maintain scalable data pipelines using Big Data technologies (e.g., Apache Hadoop, Apache Spark, Kafka). - Implement data ingestion strategies to efficiently load and transform large datasets from various sources. Data Architecture and Modeling : - Collaborate with data architects to design and optimize data models that support analytical workloads. - Create and maintain data schemas, ensuring efficient storage and retrieval of data across multiple platforms. ETL Development : - Develop ETL processes to extract, transform, and load data into data lakes and warehouses. - Utilize tools like Apache NiFi, Talend, or custom-built solutions to automate data workflows. Performance Tuning : - Monitor and optimize the performance of data processing jobs to ensure high availability and reliability. - Identify bottlenecks in data processing and implement solutions to improve efficiency. Data Quality and Governance : - Establish data quality frameworks to ensure accuracy, consistency, and reliability of data. - Implement data governance practices, including metadata management and compliance with data regulations. Collaboration and Stakeholder Engagement : - Work closely with data scientists, analysts, and business stakeholders to understand data requirements and translate them into technical specifications. Participate in cross-functional teams to develop data-driven solutions that support business objectives. Troubleshooting and Support : - Provide ongoing support for data systems, troubleshooting issues, and implementing fixes as necessary. - Develop monitoring tools and dashboards to ensure data pipeline health and performance. Documentation and Knowledge Sharing : - Maintain thorough documentation of data processes, architectures, and standards for future reference. - Share knowledge and best practices with team members through training sessions and collaborative discussions. Continuous Improvement : - Stay current with industry trends and emerging technologies in the Big Data landscape. - Propose and implement enhancements to existing data solutions and practices. Mentorship and Leadership : - Mentor junior team members, providing guidance on technical challenges and career development. - Lead initiatives to improve team processes and efficienc - Document processes, data flows, and system architecture for future reference. (ref:hirist.tech)

Location: in, IN

Posted Date: 11/3/2024