Data Engineer

Responsibilities

• Design, build, and maintain scalable data pipelines to support the knowledge base and agentic AI components.
• Develop robust ETL processes for ingesting structured and unstructured data from diverse internal and external sources.
• Implement data transformation workflows optimized for AI model training, inference, and real-time processing.
• Design and optimize data storage architectures for efficient data retrieval, scalability, and performance.
• Establish and maintain data quality checks, validation rules, and monitoring mechanisms to ensure data integrity.
• Collaborate with data scientists and ML engineers to align data infrastructure with model development requirements.
• Ensure data security, privacy, and regulatory compliance throughout the data lifecycle.
• Monitor, troubleshoot, and continuously improve the reliability and performance of data pipelines and related systems.

Requirements

• Minimum 5 years of experience in data engineering, data infrastructure, or a related field
• Strong proficiency in Python and SQL for large-scale data processing, transformation, and analysis
• Hands-on experience with ETL orchestration tools such as Apache Airflow or Azure Data Factory
• Solid understanding of data modeling, schema design, and both relational and non-relational databases
• Familiarity with big data technologies such as Apache Spark, Hadoop, or similar distributed frameworks
• Experience working with cloud-based data platforms (Azure Data Lake, AWS S3, Google BigQuery) and storage services
• Knowledge of data privacy, governance, and security best practices
• Bachelor’s degree in Computer Science, Data Engineering, Information Systems, or a related technical discipline

Technical Skills

• Programming Languages: Python, SQL, Scala
• ETL & Orchestration Tools: Azure Data Factory, Apache Airflow, dbt
• Data Processing Frameworks: Apache Spark, PySpark, Pandas
• Databases: PostgreSQL, Azure Synapse Analytics, Azure Cosmos DB
• Data Lakes & Storage: Azure Data Lake Storage (ADLS), Delta Lake
• Streaming Platforms: Apache Kafka, Azure Event Hubs
• Version Control & Data Versioning: Git, DVC (Data Version Control)

Benefits

• Be at the forefront of building sovereign AI platforms that drive digital independence and transformation
• Work with forward-thinking clients, engineering minds, and AI infrastructure thought leaders
• Grow your impact in a purpose-driven, innovation-led culture that values agility, inclusion, and continuous learning
• Professional development through continuous learning, mentorship, and a cross-cultural work environment
• Work on cutting-edge technology with real-world impact

About the Company

At Visionary, we help organisations solve complex challenges to unlock growth across strategy, technology, operations, and sustainability. With deep industry expertise and a global footprint, our consulting services are tailored to the evolving needs of modern enterprises.