Architecture
Daily Azure deployment pipeline.
An end-to-end cloud data engineering project built with Django, Microsoft Azure, GitHub Actions and Docker.
GitHub Repository Deployment BlogThis project demonstrates a cloud-native data pipeline built around a data producer, a central cloud storage layer and a data consumer. A Dockerised Python pipeline acts as the producer, collecting real-time meteorological data for a configurable list of cities from the OpenWeather public API each day and storing them in Azure Blob Storage. A Django web application then acts as the consumer, reading the latest dataset directly from Blob Storage and visualising it through an interactive Plotly dashboard. Azure Data Factory orchestrates the daily execution of the pipeline, and GitHub Actions automatically builds and deploys the Docker image to Azure Container Instances. While the project uses public meteorological data, the architecture is generic and could be reused with many other data sources.
This dashboard demonstrates how the latest dataset is automatically collected, stored and served through the application. The data is automatically refreshed every day using Azure Data Factory and Azure Container Instances. While the visualisation is intentionally simple, the same architecture could power much richer dashboards built on virtually any data source.
Daily Azure deployment pipeline.