Big Data Engineer - PySpark

Job Description : Big Data Engineer (PySpark) Years of experience : 6 to 13 years Locations : Chennai and Pune ( Candidates from chennai and pune only apply) Notice period : Immediate joiners/15 to 30 days only About the Role : We are seeking a highly skilled and experienced Big Data Engineer with a strong focus on PySpark to join our dynamic team. As a Big Data Engineer, you will play a pivotal role in designing, developing, and maintaining robust data pipelines to extract, transform, and load (ETL) large volumes of data. You will leverage your expertise in PySpark to analyze complex datasets, build scalable data solutions, and drive data-driven : 1. Data Engineering : - Design, develop, and maintain efficient and scalable data pipelines using PySpark on big data platforms like Hadoop, Spark, and Databricks. - Extract, transform, and load data from various sources (e.g., databases, APIs, cloud storage) into data warehouses and data lakes. - Optimize data pipelines for performance, reliability, and scalability. - Implement data quality checks and monitoring mechanisms to ensure data accuracy and integrity. 2. Data Analysis and Modeling : - Analyze large, complex datasets to uncover insights and trends. - Develop predictive models and machine learning algorithms using PySpark. - Collaborate with data scientists and business analysts to understand data requirements and translate them into technical solutions. 3. Cloud Infrastructure : - Deploy and manage big data solutions on cloud platforms (e.g., AWS, GCP, Azure). - Configure and optimize cloud resources for optimal performance and cost-effectiveness. 4. Collaboration and Problem-Solving : - Work closely with cross-functional teams (e.g., data scientists, data analysts, software engineers) to deliver data-driven solutions. - Identify and troubleshoot data quality issues and performance bottlenecks. - Stay up-to-date with the latest trends and technologies in the big data ecosystem. Qualifications : - 6 years of experience in data engineering and data analysis. - Strong proficiency in PySpark and Python programming. - In-depth knowledge of big data technologies (Hadoop, Spark, Hive, HDFS). - Experience with cloud platforms (AWS, GCP, Azure) and cloud-based data warehouses (e.g., Snowflake, Redshift). - Familiarity with data modeling, ETL processes, and data warehousing concepts. - Experience with data visualization tools (e.g., Tableau, Power BI). - Strong problem-solving and analytical skills. - Excellent communication and collaboration skills. Preferred Qualifications : - Experience with machine learning and statistical modeling techniques. - Knowledge of SQL and NoSQL databases. - Certification in big data technologies or cloud platforms. (ref:hirist.tech)

Location: in, IN

Posted Date: 11/17/2024