Data flow¶

A simulated day becomes an analytical table in five hops. Each hop has one writer. Later hops read what that writer stored.

flowchart TD
  sim[Simulator]
  gateway[Cluster gateway]
  stores["Postgres, Cassandra, ClickHouse, Kafka"]
  stream[Flink open-day job]
  batch[Airflow sim_customers_daily]
  iceberg[Iceberg warehouse]
  trino[Trino homelab.sim_customers]
  sim --> gateway --> stores
  stores --> stream
  stores --> batch --> iceberg --> trino

1. Simulator. A local process builds one cohort. It keeps its own copy of the run and posts accounts, licenses, wallets, transactions, and events to the cluster gateway. A row in the local copy does not mean the cluster accepted it.
2. Gateway and stores. The gateway fronts the account, events, license, wallet, and commerce APIs. Postgres holds accounts and the related commercial tables. Cassandra holds clicks. ClickHouse holds open-day counts. Kafka is the event log.
3. Flink. The open-day job maintains the ClickHouse open-day counts from the event stream.
4. Airflow. The DAG sim_customers_daily is the only writer of the shared Iceberg warehouse for this data. It snapshots the Postgres tables and the event topics.
5. Lakehouse. Trino reads those Iceberg tables through the catalog homelab.sim_customers. Silver and gold models are written only to the lakehouse's local warehouse, not back into the shared prefix.

Event schemas live in the simulator and cluster repositories. The superproject compares the pinned commits. The contracts/ directory does not hold those schemas yet.

The planned extension, not built, would continue from the lakehouse into ml_platform, with analyst_agent calling that platform. See the roadmap.