BAU Lead - Data Engineering

Responsibilities We are looking for a seasoned BAU Lead - Data Engineering to lead our data operations and support function in a cloud-first environment. The ideal candidate will bring strong technical leadership, hands-on expertise in AWS and Snowflake, and proven experience in managing Business-As-Usual (BAU) data platforms, production support, and continuous improvement initiatives. ▪ Develop tools to improve data flows between internal/external systems and the data lake/warehouse. ▪ Work with stakeholders to understand needs for data structure, availability, scalability, and accessibility. ▪ Build robust and reproducible data ingest pipelines to collect, clean, harmonize, merge, and consolidate data sources. ▪ Understanding existing data applications and infrastructure architecture ▪ Build and support new data feeds for various Data Management layers and Data Lakes ▪ Evaluate business needs and requirements. ▪ Support migration of existing data transformation jobs in Oracle, and MS-SQL to Snowflake. ▪ Lead the migration of the existing data transformation jobs in Oracle, Hive, Impala etc. into Spark, Python on Glue etc. ▪ Able to document the processes and steps. ▪ Develop and maintain datasets. ▪ Improve data quality and efficiency. ▪ Lead Business requirements and deliver accordingly. ▪ Collaborate with Data Scientists, Architect and Team on several Data Analytics projects. ▪ Collaborate with DevOps Engineer to improve system deployment and monitoring process. ▪ Experience in critical production support and how the BAU functions is preferred. Key Skills: ▪ Lead a team of data engineers in managing day-to-day BAU operations, production support, incident management, and platform stability. ▪ Strong AWS knowledge in terms of designing new architecture and providing optimized solutions for existing ones. (S3, Glue, DMS, MWAA, AMS, IAM). ▪ In-depth knowledge with respect to Snowflake and its architecture. ▪ Prefer prior Experience in BAU environment. ▪ Good knowledge on Airflow and MWAA. ▪ Hands-on experience in SQL/Python/Pyspark. ▪ Expertise in optimizing techniques in cloud environments. ▪ Should have the vision on data strategy and able to deliver the same. ▪ Should be able to lead the design and implementation of data management processes, including data sourcing, integration, and transformation. ▪ Able to manage and lead a team of data professionals, providing guidance, mentoring and foster a collaborative and innovative team culture focused on continuous improvement. ▪ To evaluate and recommend data-related technologies, tools, and platforms. ▪ Collaborate with IT teams to ensure seamless integration of data solutions. ▪ Should have experience in Implementing and enforcing data security protocols and ensure compliance with relevant regulations.

Required Qualifications: ▪ At least 8+ years of Data Engineer Experience ▪ Bachelor qualification in a computer science or STEM (science, technology, engineering, or mathematics) related field. ▪ At least 5+ years of recent hands-on professional experience (actively coding) working as a Lead handling support & production issue. ▪ 2+ experience with large scale datasets, data lake and data warehouse technologies such as AWS Redshift, Google BigQuery, Snowflake. Snowflake is highly preferred. ▪ At least 3+ years of experience in ETL (AWS Glue), Amazon S3, Amazon RDS, Amazon Kinesis, Amazon Lambda, Apache Airflows, Amazon Step Functions. ▪ Professional experience working in an agile, dynamic and customer-facing environment is required. ▪ Understanding of distributed systems and cloud technologies (AWS) is highly preferred. ▪ Understanding of data streaming and scalable data processing is preferred to have. ▪ Strong knowledge in scripting languages like SQL ,Python, UNIX shell and Spark is required. ▪ Understanding of RDBMS, Data ingestions, Data flows, Data Integrations etc. ▪ Technical expertise with data models, data mining and segmentation techniques. ▪ Experience with full SDLC lifecycle and Lean or Agile development methodologies. ▪ Knowledge of CI/CD and GIT Deployments. ▪ Ability to work in team in diverse/ multiple stakeholder environment. ▪ Ability to communicate complex technology solutions to diverse teams namely, technical, business and management teams

Soft Skills ▪ Ability to work in a collaborative environment and coach other team members on coding practices, design principles, and implementation patterns that lead to high quality maintainable solutions. ▪ Excellent communications and stake-holder management are required. ▪ Ability to handle senior leadership and report to them if required. ▪ Ability to work in a dynamic, agile environment within a geographically distributed team. ▪ Ability to focus on promptly addressing customer needs. ▪ Ability to work within a diverse and inclusive team. ▪ Technically curious, self-motivated, versatile and solution-oriented