Labcorp - Senior Data Analytics Engineer - Big Data/PySpark

Job Responsibilities : - Designing and implementing highly performant data ingestion pipelines from multiple sources - Delivering and presenting proofs of concept to of key technology components to project stakeholders. - Developing scalable and re-usable frameworks for ingesting of data sets - Integrating the end-to-end data pipeline to take data from source systems to target data repositories ensuring the quality and consistency of data is maintained at all times - Working with event based / streaming technologies to ingest and process data - Do analysis of new requirements and propose a solution - Evaluating the performance and applicability of multiple tools against customer requirements Required Skills : - At least 4 years of Hands-on experience on PySpark, Python, Spark, Scala, Able to perform data validation, delivery, quality and integrity. - At least 4 years of Snowflake /Databricks Cloud Data warehouse Implementation experience, Should possess skill pertaining to handling various file formats (XML,JSON,Avro,Parquet) - Good Experience in Extraction, Transformation and Loading (ETL) data from multiple data sources & various data formats (Structured, Semi-Structured & Unstructured) - Should be able to work with modern schedulers (Airflow, Matillion, etc.) and distributed data systems such as Hadoop, Spark, Kafka etc. - Should have experience in business intelligence, reporting, and/or delivery of modern visualization solutions (Tableau, Power BI) - Good working exposure on cloud technologies of AWS - EC2, S3, Lambda, SQS, SNS - Strong Programming experience using PL/SQL packages, Stored procedures, views, Materialized views, triggers, Indexes etc. - Strong understanding on Data Warehousing concepts using Fact, Dimension tables and various methods of loading data - Hands-on Experience with Big Data model techniques using Python / Spark - Experience in CI/CD domain using Jenkins, Terraform, Docker, AWS, and Kubernetes. - Experience in Pharma industry is a plus - Strong communication and collaboration skills. (ref:hirist.tech)

Location: bangalore, IN

Posted Date: 10/9/2024