Skip to content

Architecture

The platform is three independent components, integrated by a superproject that records a compatible commit of each one. The diagram shows who talks to whom. Dashed nodes are planned and are not part of the running system.

flowchart TD
  sim[simulated_customers]
  cluster[rpi_cluster]
  lake[lakehouse]
  ml["ml_platform (planned)"]
  agent["analyst_agent (planned)"]
  sim --> cluster --> lake
  lake -.-> ml
  agent -.-> ml

simulated_customers creates the customers and posts their records. rpi_cluster receives those records and writes the shared analytical snapshot. lakehouse reads that snapshot and does not write it back.

Current components

  • simulated_customers generates synthetic customers, state changes, transactions, usage, and behavioral events.
  • rpi_cluster is the homelab infrastructure and the ingestion and storage services.
  • lakehouse is Trino and dbt. It reads the cluster's Iceberg tables. Silver and gold models stay in its local warehouse. Spark in rpi_cluster remains the writer of the shared Iceberg warehouse.

Planned components

  • ml_platform would train and serve models on lakehouse data.
  • analyst_agent would be an agent that consumes and feeds ml_platform.

Neither planned component is pinned, and neither has a deployment.

What the superproject owns

Architecture notes, version pins, cross-component configuration, shared contracts, setup and validation commands, and integration tests. It does not hold the component implementation.