Cloud Engineer Azure Databricks

Responsibilities

• Design, implement, and administer the Azure foundation for the data and AI platform
• Author and maintain reusable Terraform modules for Databricks, ADLS Gen2, and Azure ML
• Implement private networking, hub-and-spoke topology, and security posture
• Manage ADLS Gen2 storage account architecture and access patterns
• Oversee Azure Data Factory integration runtime provisioning and deployment automation
• Provision infrastructure for machine learning workloads including GPU quota and MLflow integration
• Build monitoring layers using Azure Monitor and Log Analytics
• Manage FinOps, cost visibility, and capacity planning for DBU and storage spend
• Establish platform standards and reference configurations for Databricks administration

Requirements

• 6+ years in cloud infrastructure or platform engineering
• 4+ years focused on Microsoft Azure
• Expert-level hands-on experience with Terraform
• Demonstrable experience in Databricks administration, including Unity Catalog and cluster policies
• Deep knowledge of Azure Networking, Entra ID, and RBAC
• Experience with ADLS Gen2 design at enterprise scale
• Platform-side experience with Azure Data Factory
• Proficiency in Python, PowerShell, and/or Bash

Preferred Qualifications

• Experience with Databricks Asset Bundles and workspace-as-code patterns
• Experience leading Unity Catalog migrations
• Knowledge of Azure Machine Learning and MLflow
• Experience with Kubernetes, AKS, and containerized workloads
• Familiarity with Policy as Code (Azure Policy, OPA, Sentinel)
• Experience with Event Hubs, Kafka, or Stream Analytics
• Formal FinOps practice experience