Databricks Engineer

About the Role We are looking for a skilled Databricks Engineer todesign, develop, and maintain scalable data engineering solutions using the DatabricksLakehouse Platform. The ideal candidate will have strong hands-on experiencewith Databricks, Apache Spark, Python, SQL, Delta Lake, and cloud dataplatforms, with the ability to build reliable and high-performance datapipelines. Key Responsibilities

• Design, develop, and maintain scalable data pipelines using Databricks and Apache Spark.

• Develop ETL/ELT pipelines using PySpark, Python, and SQL.

• Build and maintain Delta Lake tables and data processing workflows.

• Work with Databricks Lakehouse architecture and related data engineering components.

• Develop batch and, where required, near-real-time data processing solutions.

• Ingest and transform data from databases, APIs, files, and other data sources.

• Implement data cleansing, transformation, validation, and quality checks.

• Optimise Spark jobs and Databricks workloads for performance and cost efficiency.

• Work with cloud storage and data services across Azure, AWS, or GCP.

• Implement data security, access controls, and governance within the data platform.

• Collaborate with Data Architects, Data Scientists, BI Developers, and business stakeholders.

• Troubleshoot data pipeline failures and resolve performance and data-quality issues.

• Develop and maintain technical documentation for data pipelines and solutions.

• Participate in code reviews, testing, deployment, and production support.

• Follow Agile development practices and contribute to continuous improvement.

Required Skills & Experience

• 3–5 years of experience in Data Engineering.

• Strong hands-on experience with Databricks.

• Strong knowledge of:

-

• Apache Spark / PySpark

• Python

• SQL

• Delta Lake

• ETL/ELT concepts

• Experience developing and managing data pipelines.

• Good understanding of data warehousing and data modelling concepts.

• Experience working with cloud platforms, preferably Microsoft Azure.

• Experience with cloud storage such as Azure Data Lake Storage (ADLS), Amazon S3, or Google Cloud Storage.

• Experience with relational and/or NoSQL databases.

• Good understanding of data quality, validation, and governance.

• Familiarity with Git and CI/CD practices.

• Strong troubleshooting and analytical skills.

Good to Have

• Databricks Certified Data Engineer Associate/Professional certification.

• Experience with Azure Data Factory.

• Experience with Microsoft Fabric.

• Knowledge of Unity Catalog and Databricks governance.

• Experience with Databricks Workflows and job orchestration.

• Experience with streaming technologies such as Kafka or Structured Streaming.

• Experience with Power BI or other BI platforms.

• Exposure to Machine Learning workflows on Databricks.

• Experience with Terraform or Infrastructure as Code.

• Knowledge of DevOps and CI/CD pipelines.

Candidate Profile The ideal candidate should be a hands-on Databricks/Data Engineer capable of independently developing data pipelines, troubleshooting production issues, optimising Spark workloads, and collaborating with technical and business teams.