Data Engineer - ETL/Python

Job Description : We are hiring for On-Premise Data Engineer for Bangalore location. Job Summary : We are looking for a skilled On-Premise Data Engineer to join our team. The ideal candidate will have experience in designing, developing, and maintaining scalable data pipelines, data lakes, and databases on on-premise infrastructure. You will work closely with data scientists, analysts, and other stakeholders to ensure efficient data processing and availability across the organization. Requirements : - Experience with on-premise databases and data warehouses SQL Server, Oracle. - Strong knowledge of ETL. - Proficiency in scripting languages : Python, Shell for automating data pipelines. (MUST) - Experience with data processing frameworks (e.g., Apache Hadoop, Apache Spark) in an on-premise environment. - Familiarity with batch (real-time data processing technologies is plus). - Proficiency in SQL and ability to optimize complex queries. - Knowledge of data modeling and schema design principles. - Understanding of data governance, data security, and compliance requirements. - Experience in performance tuning of databases and ETL processes. - Strong problem-solving skills and ability to troubleshoot complex data issues. - Familiarity with version control systems (e.g., Git) and CI/CD tools for data pipeline deployment. - Knowledge of containerization (e.g., Docker) (MUST) and on-prem orchestration (Cron Jobs) is added advantage Key Responsibilities : 1. Data Infrastructure Design and Management : - Design, build, and maintain on-premise data pipelines, ETL processes, and data integration frameworks. - Set up and manage on-premise databases, data warehouses, and data lakes. - Ensure optimal performance of data architecture by maintaining high data throughput, low-latency, and reliable data delivery. 2. Data Integration and ETL : - Develop and maintain ETL (Extract, Transform, Load) processes to ingest data from multiple sources into on-premise systems. - Perform data integration tasks and synchronize data from various sources like relational databases, flat files, APIs, and other enterprise systems. - Automate and optimize ETL pipelines to process large volumes of structured and unstructured data. 3. Data Quality and Governance : - Implement data quality checks to ensure accuracy, integrity, and consistency of the data. - Ensure adherence to data governance policies and standards, including data security and access control. 4. Database Management : - Set up and manage relational database : SQL Server, Oracle on-premise. - Tune database performance and optimize queries for better efficiency and reduced downtime. 5. Data Modeling and Architecture : - Design and maintain data models, schemas, and optimized data structures for high-performance querying and storage. - Collaborate with data architects and business stakeholders to create efficient data models that support various business use cases. 6. Monitoring and Troubleshooting : - Monitor and manage on-premise data infrastructure for availability, performance, and capacity. - Troubleshoot any issues related to data ingestion, processing, and storage. 7. Collaboration : a. Soft Skills : - Strong communication and collaboration skills. - Ability to work independently and within a team. - Analytical mindset and detail-oriented approach to problem-solving (ref:hirist.tech)

Location: chennai, IN

Posted Date: 11/9/2024