Senior Big Data Engineer - Python/Azure

Job Description : - Lead the architecture and design of our big data platform, ensuring scalability, reliability, and performance. - Perform and oversee data ingestion tasks, including writing scripts, calling APIs, web scraping, and writing complex SQL queries. - Design and implement data stores (data lakes, data warehouses) that support the scalable processing and storage of our high-frequency data. - Maintain and optimize our existing data pipeline, ensuring efficient data flow and processing. - Customize and oversee the integration of various data tools, warehouses, databases, and analytical systems. - Configure and ensure the availability of data-access tools used by our team of data scientists. - Collaborate with data scientists and other stakeholders to understand data requirements and translate them into platform solutions. - Implement data quality and governance measures. - Stay up-to-date with the latest trends and technologies in the big data ecosystem. What you'll bring : - Extensive experience in designing and building big data platforms. - Strong proficiency in Python for data engineering tasks. - Deep understanding of data warehousing concepts and architectures. - Hands-on experience with ETL/ELT processes and tools. - Expertise in designing and implementing scalable data stores. - Strong SQL skills for querying and manipulating large datasets. - Experience with various data integration tools and techniques. - Understanding of data security and access control. - Excellent problem-solving and analytical skills. - Strong communication and collaboration abilities. Good-to-Have Skills : - Experience with specific cloud platforms (eg, AWS, Azure, GCP). - Familiarity with big data technologies like Spark, Hadoop, Kafka, etc. - Knowledge of NoSQL databases. - Experience with data visualization tools (ref:hirist.tech)

Location: bangalore, IN

Posted Date: 4/4/2025