This article explores the design and operation of Predicta’s data warehouse, created to handle the large volume of weather forecasts, measurements and model outputs processed across Europe.
Why use a data warehouse?
A data warehouse is useful when the amount of data becomes too large for a traditional transactional database. This does not make the transactional database slow or bad; it was simply designed for different guarantees and workloads.
Choosing a hosting provider
The initial infrastructure design required a cloud provider that could offer reliability, reasonable cost and servers located within the EU. Hetzner was selected for that combination and for its broad experience with dedicated servers and VPS hosting in Germany.
The available infrastructure expertise also made a self-managed cluster a sensible long-term choice. Running the platform directly on dedicated servers offered more control over the environment and avoided unnecessary layers of operational overhead.
Infrastructure
The operating system is openSUSE MicroOS, designed for cloud infrastructure such as Kubernetes. Kubernetes itself runs on k3s, a lightweight distribution that simplifies administration while retaining the practical features needed for daily operation.
Data storage
The first version of the platform used Hetzner’s managed S3-compatible object storage. A managed service reduced the initial operational burden and appeared to offer a good balance between price and convenience.
After several months of production use, however, the service proved unreliable for the scale of the workload. Object-store requests frequently timed out and sometimes required retries, slowing down data operations. Availability also fluctuated over several days.
The managed object store was therefore replaced with a self-managed store based on Rook-Ceph. The current Ceph cluster runs across three nodes and is fully managed by Predicta. Data is replicated three times, so the loss of two servers would not result in data loss. Daily backups to off-site storage provide an additional recovery path in the event of a major incident such as a data-centre fire.
Apache Iceberg was selected as the table format. Iceberg is well positioned as an open standard for modern data warehouse and lakehouse solutions, has broad support across major providers and is developed under the Apache Foundation. That open governance reduces the risk of disruptive licensing changes by a single commercial owner.
An Iceberg warehouse backed by S3 also needs a catalog and access-control layer. Consumers need to know where tables are stored and who can read or write them. The REST catalog stores this metadata and provides fast access over the REST protocol. Several open-source options were evaluated, with Lakekeeper proving the best fit. Its low memory usage, low response latency, Kubernetes support and official Helm chart suit the platform well.
Access permissions are managed through OpenFGA. Not every user should access every dataset or write to every table. Explicit permissions reduce the risk of accidental production changes, especially when someone mistakes a production environment for development. Good access control avoids both the incident and the stress of restoring deleted data from backup.
Processing and operations
Apache Spark handles processing in the warehouse. Previous project experience made Spark a natural choice, while its ability to distribute workloads across multiple servers is essential for datasets measuring hundreds of gigabytes.
Spark is also used as the processing engine by platforms such as Databricks, which stores data in S3 object storage. The combination therefore provides a familiar and proven foundation for forecasting workloads.
Data warehouse architecture
The platform still has many operational details worth exploring, but covering all of them in one article would make it unnecessarily long. Further lessons can follow in a future instalment.