Data Engineering - Tableau/Power BI

Data Engineer (3-4 Years Experience) - Real-time & Batch Processing | AWS, Kafka, Click House, Python Location : NOIDA. Experience : 3-4 years. Job Type : Full-Time. About the Role : We are looking for a skilled Data Engineer with 3-4 years of experience to design, build, and maintain real-time and batch data pipelines for handling large-scale datasets. You will work with AWS, Kafka, Cloudflare Workers, Python, Click House, Redis, and other modern technologies to enable seamless data ingestion, transformation, merging, and storage. Bonus: If you have Web Data Analytics or Programmatic Advertising knowledge, it will be a big plus. Responsibilities : Real-Time Data Processing & Transformation : - Build low-latency, high-throughput real-time pipelines using Kafka, Redis, Firehose, Lambda, and Cloudflare Workers. - Perform real-time data transformations like filtering, aggregation, enrichment, and deduplication using Kafka Streams, Redis Streams, or AWS Lambda. - Merge data from multiple real-time sources into a single structured dataset for analytics. Batch Data Processing & Transformation : - Develop batch ETL/ELT pipelines for processing large-scale structured and unstructured data. - Perform data transformations, joins, and merging across different sources in Click House, AWS Glue, or Python. - Optimize data ingestion, transformation, and storage workflows for efficiency and reliability. Data Pipeline Development & Optimization : - Design, develop, and maintain scalable, fault-tolerant data pipelines for real-time & batch processing. - Optimize data workflows to reduce latency, cost, and compute load. Data Integration & Merging : - Combine real-time and batch data streams for unified analytics. - Integrate data from various sources (APIs, databases, event streams, cloud storage). Cloud Infrastructure & Storage : - Work with AWS services (S3, EC2, ECS, Lambda, Firehose, RDS, Redshift, ClickHouse) for scalable data processing. - Implement data lake and warehouse solutions using S3, Redshift, and ClickHouse. Data Visualization & Reporting : - Work with Power BI, Tableau, or Grafana to create real-time dashboards and analytical reports. Web Data Analytics & Programmatic Advertising (Big Plus) : - Experience working with web tracking data, user behavior analytics, and digital marketing datasets. - Knowledge of programmatic advertising, ad impressions, clickstream data, and real-time bidding (RTB) analytics. Monitoring & Performance Optimization: - Implement monitoring & logging of data pipelines using AWS CloudWatch, Prometheus, and Grafana. - Tune Kafka, Click House, and Redis for high performance. Collaboration & Best Practices : - Work closely with data analysts, software engineers, and DevOps teams to enhance data accessibility. - Follow best practices for data governance, security, and compliance. Must-Have Skills : Programming : Strong experience in Python and JavaScript. Real-time Data Processing & Merging : Expertise in Kafka, Redis, Cloudflare Workers, Firehose, Lambda. Batch Processing & Transformation : Experience with Click House, Python, AWS Glue, SQL-based transformations. Data Storage & Integration : Experience with MySQL, Click House, Redshift, and S3-based storage. Cloud Technologies : Hands-on with AWS (S3, EC2, ECS, RDS, Firehose, Click House, Lambda, Redshift). Visualization & Reporting : Knowledge of Power BI, Tableau, or Grafana. CI/CD & Infrastructure as Code (IaC) : Familiarity with Terraform, CloudFormation, Git, Docker, and Kubernetes. (ref:hirist.tech)

Location: in, IN

Posted Date: 4/3/2025