Azure Data Engineer

The responsibility of Data Engineer encompasses the entire lifecycle of the Data Warehouse environments that underpin the vital business requirements. This includes the design, development, implementation, operation, and ongoing support of these critical systems.
The individual in this position is tasked with developing a flexible, enterprise-level environment that integrates multiple warehouses to guarantee precise, comprehensive, consistent, and timely data. Their primary goal is to create a cohesive system that fulfills these demands while catering to diverse business requirements.
The role necessitates the capacity to explore and grasp emerging technologies while collaborating closely with peer teams to establish strategic roadmaps and priorities. The ability to swiftly acquire and proficiently apply hands-on administration skills is essential. As a Subject Matter Expert in technical requirements, this position will play a crucial role in supporting and implementing data projects, as well as engaging effectively with users and other IT staff.
The Data Engineer responsibilities include:

• Provide support in designing and overseeing enterprise-grade data pipelines and data stores.
• Implement automation and streamline processes to optimize the entire data and analytics platform, ensuring efficient throughput and high-performance outcomes.
• Recognize, devise, and execute internal process enhancements, including automation of manual tasks, optimizing data delivery, and redesigning architecture or infrastructure to enhance scalability.

This job description does not constitute an employment contract, implied or otherwise, other than an “at will” relationship and is subject to change by the employer at any time without notice.

 

• Collate large, intricate datasets that align with functional and non-functional business demands.
• Develop processes that facilitate data transformation, manage data structures, metadata, dependencies, and workload management.
• Collaborate with business users to understand functional and data requirements, contributing to the enhancement of data models and pipelines.
• Apply analytical and problem-solving skills to diagnose and resolve intricate technical issues.
• Create, maintain, and continuously enhance scalable data pipelines, while also developing new data source integrations to accommodate the growing volume and complexity of data.
• Designing, implementing, and managing data extraction, transformation, and loading (ETL) processes.
• Creating comprehensive technical specification documents and application interface designs.
• Creating data processing and integration solutions for both batch and real-time scenarios, proficiently handling structured and unstructured data.
• Participate in design discussions, code reviews, and project-related team meetings.
• Ensuring data security and compliance with relevant regulations and best practices in all data operations.
• Troubleshooting and resolving data and system issues, stepping in when necessary to address outages and challenges.
• Other duties and responsibilities as assigned or needed.

Key Technical Knowledge, Skills, and Abilities:

• Must have experience deploying modern data solutions leveraging components like Azure functions, Azure Synapse, Azure Data Factory, Data Flows, Azure Data Lake, Azure SQL.
• Strong level of understanding on Azure Synapse, ADLS, and Azure DevOps.
• Exhibit an understanding of Data Lake architectures, including raw, enriched, and curated layer concepts, and ETL/ELT operations.
• Exhibit a solid understanding of database design, data warehousing concepts, big data platforms, and ETL operations.
• Experience working with data integration techniques & self-service data preparation.
• Experience in requirements analysis, design, and prototyping.
• Experience with DevOps tools like Azure DevOps, Jenkins, Maven etc.
• Experience in building/operating/maintaining fault tolerant and scalable data processing integrations.
• Demonstrated experience of turning business use cases and requirements into technical solutions.
• Ability to conduct data profiling, cataloging, and mappings for technical design and construction of data flows.
• Strong collaboration and experience working with remote teams.
• Strong problem-solving skills with emphasis on optimization data pipelines.
• Showcase excellent communication and presentation skills for effective collaboration with technical and non-technical stakeholders.