Big Data engineer - Python/Scala

Responsibilities include : - Hands-on experience designing & managing large data models, writing performant SQL queries, and working with large datasets and related technologies. - Work closely with the Architecture group in delivering technical solutions. - Experience with designing and building large-scale data processing systems. - Experience in analyzing data to identify deliverables, gaps, and inconsistencies in data sets. - Participate in the testing of prototypes & validate test procedures to ensure that they apply to the design. - Perform root-cause analysis (RCA) of complex issues ranging from hardware, operating system, application, network, and information security platforms while working closely with various infrastructure teams and business users to quickly arrive at creative, tactical, and long-term solutions. - Excellent verbal and written communication skills, especially in technical communications - Collaborate with project teams (solution architects, business, QA, and project management) to ensure solutions meet business objectives and fall within timelines and acceptance criteria. - Experience monitoring, troubleshooting, and tuning services and applications and operational expertise such as good troubleshooting skills, understanding of systems capacity, bottlenecks, and basics of memory, CPU, OS, storage, and networks. Expected Technical Skills : - 8 years' experience as a software engineer or equivalent designing large data-heavy distributed systems and/or high-traffic web apps. - Must have 4 experience using big data tools such as Spark / PySpark) - Must have experience in Airflow pipeline orchestration. - At least 4 years of hands-on implementation experience working with a combination of the following technologies : PySpark, Spark, Python, EMR, Airflow, and KAFKA. - Strong experience in two of the following SQL and NoSQL Databases - Demonstrate substantial depth of knowledge and experience in Python, Data Streaming, Big Data, Flink, Spark, Scala, Python, Kubernetes, and wrangling of various data formats like Parquet and JSON. - Hands-on experience with Linux and administration. (ref:hirist.tech)

Location: hyderabad, IN

Posted Date: 10/27/2024