Skip to content

rpi_cluster

rpi_cluster is the homelab that receives the simulator. It is the ingestion path and the system of record for this project.

A gateway fronts five APIs: account, events, license, wallet, and commerce. Behind that gateway:

  • Postgres holds accounts and the related commercial tables, including licenses, wallets, and transactions.
  • Cassandra holds click events.
  • ClickHouse holds open-day counts, filled by a Flink job that reads the event stream.
  • Kafka is the event log the other stores and the daily job consume.
  • Airflow, DAG sim_customers_daily, copies the operational tables and event topics into an Iceberg warehouse. That job is the only writer of the shared warehouse prefix for this data. Object storage holds the Parquet files and Iceberg metadata.

The cluster is the only place that writes the snapshot the lakehouse reads.