Skip to content
Back to projects
Primary evidence for Spark, Kafka and Airflow data platform
Data EngineeringApplied academic project2026

Spark, Kafka and Airflow data platform

Reproducible local platform with batch, streaming, Medallion architecture and orchestration.

Technical documentationArchitectureRepository

Problem

Maintain a living metadata catalogue by combining structured, semi-structured and real-time event sources.

Architecture

SQL Server, CSV and Kafka feed Spark jobs that publish Bronze, Silver and Gold layers on Delta Lake and MinIO. Airflow orchestrates the batch flows.

Data flow

Sources → batch or streaming ingestion → Spark → Medallion Delta Lake → verifiable Gold previews.

Results

3

Ingestion patterns

Incremental SQL batch, CSV batch and Kafka streaming

View source

3

Data layers

Bronze, Silver and Gold in every pipeline

View source

Contribution and authorship

  • Use-case and Medallion architecture design
  • Implementation of Spark jobs and Airflow DAGs
  • Reproducible integration of Kafka, SQL Server, MinIO and Delta Lake
Alonso Marcos Muñoz · Author and developer

Evidence

Executive architecture
Executive architecture
Technical architecture
Technical architecture

Limitations

  • • Reproducible local environment with demo data, not an enterprise production platform
  • • Included credentials are local development values only

Stack

PythonApache SparkApache AirflowApache KafkaDelta LakeMinIOSQL ServerDocker Compose

https://github.com/AlonsoMarcosM/spark-kafka-airflow-data-platform