Data Architect + AI

Responsibilities

• Design end-to-end data architectures using Bronze, Silver, and Gold layers for production-ready pipelines
• Troubleshoot and tune large-scale Spark jobs using caching, partitioning, and broadcast joins
• Implement ACID transactions, schema enforcement, and Z-ordering using Delta Lake
• Design unified governance models for data and AI assets including RBAC and data lineage
• Manage cloud infrastructure and native ecosystems (AWS, Azure, or GCP)
• Monitor DBUs and right-size clusters to optimize costs and performance

Requirements

• Expertise in Lakehouse and Medallion Architecture
• Deep understanding of Databricks and Spark internals
• Proficiency in Delta Lake and Unity Catalog
• Strong grasp of cloud infrastructure (S3, ADLS, IAM, VNet)
• Advanced SQL skills and fluency in Python or Scala

Preferred Qualifications

• Databricks Certified Data Architect or Data Engineer Professional certifications
• Experience with Generative AI, MLflow, and Mosaic AI suite
• Proficiency in CI/CD, Git, dbt, Azure Data Factory, or Apache Airflow
• Familiarity with Databricks Structured Streaming and Auto Loader
• Experience with Databricks SQL and BI tools like Power BI