Generate Clickstream Data for ClickHouse

Generate clickstream data — user browsing sessions and bounces modeled with a finite state machine — and stream page views into ClickHouse for funnel analysis.

Clickstream data is the record of every page a visitor loads on their way through a site, in the order they loaded it — a landing page, a run of product pages, a cart, sometimes a checkout. A funnel report, a conversion-rate query, or a cohort analysis depends entirely on that order: how many sessions reached each stage, and how many dropped off before the next one. A stream of page-view events generated independently of each other can share a session id by coincidence. It cannot reproduce a visitor's journey — landing, browsing, cart — arriving in that order, which is the one property a funnel query actually reads.

Eventum produces that order with the same finite-state-machine technique covered in Modeling sessions: each stage of the journey is a template, and the template's own logic decides when a visitor is ready to advance. This lesson applies it to a five-stage browsing funnel instead of a three-stage login/action/logout cycle, adds a bounce path for visitors who browse but never buy, and streams the result into ClickHouse for funnel analysis.

Funnel test data and the bounce path

Clickstream analytics reports on stages, not isolated page views: a funnel groups every session by the furthest stage it reached — landing, browsing, cart, checkout — and counts how many sessions made it to each one and how many dropped off before the next. That count only means what it claims to mean if every event in a session shares one identifier and arrives in the order a real visit takes; a checkout with no landing before it, or two unrelated visitors sharing a session id by coincidence, breaks exactly what the funnel is supposed to measure.

A bounce is the funnel's simplest outcome: a visitor who lands, browses for a while, and leaves without adding anything to the cart. Modeling it needs the same session-scoped memory as the rest of the funnel — a generator has to track how long the current visitor has been browsing without converting, not just assign a flat percentage of sessions to "left immediately."

What you'll build

The generator uses:

time-patterns input — a daily traffic curve peaking at midday, shaped with a beta distribution.
FSM picking mode — five states modeling a user journey, each one deciding from its own state when the visitor is ready to advance.
shared state — a session id, a cart count, and a browse counter, all reset at the start of every new session.
ClickHouse output — events inserted as JSON rows into a page_views table.

Prerequisites

Eventum installed
A ClickHouse instance with HTTP interface enabled (default port 8123)

No ClickHouse? Replace the clickhouse output with stdout: {} to preview the JSON events in your terminal.

Project structure

eventum.yml

startup.yml

generator.yml

daily-traffic.yml

landing.jinja

browse.jinja

add-to-cart.jinja

checkout.jinja

exit.jinja

Prepare ClickHouse

Create the target table before running the generator:

CREATE TABLE IF NOT EXISTS page_views (
    timestamp DateTime64(3),
    session_id String,
    user_agent String,
    referrer String,
    page String,
    page_type String,
    duration_ms UInt32,
    items_in_cart UInt8
) ENGINE = MergeTree()
ORDER BY (timestamp, session_id);

For the mechanics behind this insert — the HTTP interface, JSONEachRow, and connection pooling — see Generate test data for ClickHouse.

Build it

Create the project directory

mkdir -p eventum/generators/clickstream/{patterns,templates}
cd eventum

Define the daily traffic pattern

The traffic pattern uses a beta distribution anchored to a full day, the same shape the Windows Event Log lesson uses for a business-hours curve: equal shape parameters center the peak at midday and taper density toward both ends of the day, closer to how site traffic actually rises and falls than a hard-edged ramp.

generators/clickstream/patterns/daily-traffic.yml

label: Daily web traffic
oscillator:
  start: "00:00:00"
  end: "never"
  period: 1
  unit: days
multiplier:
  ratio: 2000
randomizer:
  deviation: 0.25
  direction: mixed
spreader:
  distribution: beta
  parameters:
    a: 4
    b: 4

With start anchored to midnight and a one-day period, the equal shape parameters (a: 4, b: 4) put the peak at noon and thin traffic toward both ends of the day; ratio: 2000 sets about 2,000 page-view timestamps per day before the randomizer's ±25% variance is applied.

Write the session templates

The FSM models a user journey through five stages. Each template renders one page-view event and decides, from its own state, whether the visitor is ready to move on — that decision never compares a running count directly inside a transition; each template computes its own threshold and records the outcome as a boolean flag in shared state, and the transition to the next stage only checks whether that flag is present.

Landing — the entry point. Starts a new session: a fresh session id, an empty cart, and a browse counter reset to zero, plus a random user agent and referrer. It also clears every flag a previous session might have left behind, the same way Modeling sessions' login template clears logout_ready for the next visitor.

generators/clickstream/templates/landing.jinja

{%- set session_id = module.rand.crypto.uuid4() -%}
{%- do shared.set("session_id", session_id) -%}
{%- do shared.set("items_in_cart", 0) -%}
{%- do shared.set("browse_streak", 0) -%}
{%- do shared.pop("ready_to_add", None) -%}
{%- do shared.pop("ready_to_checkout", None) -%}
{%- do shared.pop("session_done", None) -%}
{%- set ua = module.faker.locale.en_US.user_agent() -%}
{%- do shared.set("user_agent", ua) -%}
{%- set ref = module.rand.choice(["https://google.com", "https://bing.com", "https://twitter.com", "direct", "https://reddit.com"]) -%}
{%- do shared.set("referrer", ref) -%}
{
  "timestamp": "{{ timestamp.strftime('%Y-%m-%d %H:%M:%S.%f') }}",
  "session_id": "{{ session_id }}",
  "user_agent": "{{ ua }}",
  "referrer": "{{ ref }}",
  "page": "/",
  "page_type": "landing",
  "duration_ms": {{ module.rand.number.integer(500, 5000) }},
  "items_in_cart": 0
}

Browse — product listing or detail pages. browse_streak counts pages viewed since the last landing or cart addition. Once it reaches 5 with an empty cart, the session is done — a bounce. From 2 pages onward, each additional page rolls a 50% chance of signaling readiness to add to the cart, so conversions land after a variable number of pages instead of a fixed one:

generators/clickstream/templates/browse.jinja

{%- set browse_streak = shared.get("browse_streak", 0) + 1 -%}
{%- do shared.set("browse_streak", browse_streak) -%}
{%- set items_in_cart = shared.get("items_in_cart", 0) -%}
{%- set pages = ["/products", "/products/wireless-mouse", "/products/usb-hub", "/products/keyboard", "/products/webcam", "/categories/electronics", "/categories/home"] -%}
{%- if items_in_cart == 0 and browse_streak >= 5 -%}
{%- do shared.set("session_done", true) -%}
{%- elif browse_streak >= 2 and module.rand.chance(0.5) -%}
{%- do shared.set("ready_to_add", true) -%}
{%- endif -%}
{
  "timestamp": "{{ timestamp.strftime('%Y-%m-%d %H:%M:%S.%f') }}",
  "session_id": "{{ shared.get('session_id') }}",
  "user_agent": "{{ shared.get('user_agent') }}",
  "referrer": "{{ shared.get('referrer') }}",
  "page": "{{ module.rand.choice(pages) }}",
  "page_type": "browse",
  "duration_ms": {{ module.rand.number.integer(1000, 15000) }},
  "items_in_cart": {{ items_in_cart }}
}

A fixed page-count threshold would make every converting session the same length. Rolling a chance instead spreads conversions across a range of session lengths — some visitors add an item after two pages, others after four or five — the same way real browsing sessions vary, while the deterministic ceiling at 5 still guarantees a bounce for anyone who never converts at all.

Add to cart — records the cart addition, then clears ready_to_add and resets browse_streak immediately. Unlike Modeling sessions' logout_ready, which fires once per session, ready_to_add here can fire again on a later loop through browse — up to the cart's two-item target. That's why the flag has to be cleared the moment it's consumed: otherwise the next stretch of browsing would fall straight back into add-to-cart on the stale value instead of earning it again from a fresh count:

generators/clickstream/templates/add-to-cart.jinja

{%- do shared.pop("ready_to_add", None) -%}
{%- do shared.set("browse_streak", 0) -%}
{%- set items_in_cart = shared.get("items_in_cart", 0) + 1 -%}
{%- do shared.set("items_in_cart", items_in_cart) -%}
{%- if items_in_cart >= 2 -%}
{%- do shared.set("ready_to_checkout", true) -%}
{%- endif -%}
{
  "timestamp": "{{ timestamp.strftime('%Y-%m-%d %H:%M:%S.%f') }}",
  "session_id": "{{ shared.get('session_id') }}",
  "user_agent": "{{ shared.get('user_agent') }}",
  "referrer": "{{ shared.get('referrer') }}",
  "page": "/cart",
  "page_type": "add_to_cart",
  "duration_ms": {{ module.rand.number.integer(500, 3000) }},
  "items_in_cart": {{ items_in_cart }}
}

Checkout — completes the purchase, clearing the flag that led here.

generators/clickstream/templates/checkout.jinja

{%- do shared.pop("ready_to_checkout", None) -%}
{
  "timestamp": "{{ timestamp.strftime('%Y-%m-%d %H:%M:%S.%f') }}",
  "session_id": "{{ shared.get('session_id') }}",
  "user_agent": "{{ shared.get('user_agent') }}",
  "referrer": "{{ shared.get('referrer') }}",
  "page": "/checkout/complete",
  "page_type": "checkout",
  "duration_ms": {{ module.rand.number.integer(2000, 10000) }},
  "items_in_cart": {{ shared.get("items_in_cart", 0) }}
}

Exit — the session's last event, reached either after checkout or from a bounce. The FSM transitions back to landing to start a new session.

generators/clickstream/templates/exit.jinja

{
  "timestamp": "{{ timestamp.strftime('%Y-%m-%d %H:%M:%S.%f') }}",
  "session_id": "{{ shared.get('session_id') }}",
  "user_agent": "{{ shared.get('user_agent') }}",
  "referrer": "{{ shared.get('referrer') }}",
  "page": "{{ module.rand.choice(['/products', '/', '/categories/electronics']) }}",
  "page_type": "exit",
  "duration_ms": {{ module.rand.number.integer(100, 1000) }},
  "items_in_cart": {{ shared.get("items_in_cart", 0) }}
}

Configure the generator

Every transition below checks a single boolean flag, or falls back to a transition that always fires:

generators/clickstream/generator.yml

input:
  - time_patterns:
      patterns:
        - patterns/daily-traffic.yml

event:
  template:
    mode: fsm
    templates:
      - landing:
          template: templates/landing.jinja
          initial: true
          transitions:
            - to: browse
              when: { always: }
      - browse:
          template: templates/browse.jinja
          transitions:
            - to: exit
              when: { defined: shared.session_done }
            - to: add-to-cart
              when: { defined: shared.ready_to_add }
            - to: browse
              when: { always: }
      - add-to-cart:
          template: templates/add-to-cart.jinja
          transitions:
            - to: checkout
              when: { defined: shared.ready_to_checkout }
            - to: browse
              when: { always: }
      - checkout:
          template: templates/checkout.jinja
          transitions:
            - to: exit
              when: { always: }
      - exit:
          template: templates/exit.jinja
          transitions:
            - to: landing
              when: { always: }

output:
  - stdout:
      formatter:
        format: json
  - clickhouse:
      host: ${params.clickhouse_host}
      port: ${params.clickhouse_port}
      database: default
      table: page_views
      username: ${params.clickhouse_user}
      password: ${secrets.clickhouse_password}

The session flow:

From	To	Condition	Meaning
landing	browse	`always`	Every visit starts with a landing page
browse	exit	`defined: shared.session_done`	Bounced: browsed 5 pages with an empty cart
browse	add-to-cart	`defined: shared.ready_to_add`	Ready to add an item
browse	browse	`always` (fallback)	Keep browsing
add-to-cart	checkout	`defined: shared.ready_to_checkout`	Cart reached its two-item target
add-to-cart	browse	`always` (fallback)	Keep shopping
checkout	exit	`always`	Session complete
exit	landing	`always`	New session starts

Transitions are evaluated in order, and the first one whose condition holds wins. On browse and add-to-cart, the flag checks are listed before the always fallback — reversing that order would make the fallback fire first every time, since always never fails, and Eventum would never reach the flag check at all.

Configure the application

eventum.yml

server:
  host: "0.0.0.0"
  port: 9474

path:
  startup: /home/user/eventum/startup.yml
  generators_dir: /home/user/eventum/generators
  logs: /home/user/eventum/logs
  keyring_cryptfile: /home/user/eventum/cryptfile.cfg

generation:
  timezone: UTC
  batch:
    size: 500

All path.* values must be absolute paths. Adjust to match your actual project location.

startup.yml

- id: clickstream
  path: clickstream/generator.yml
  params:
    clickhouse_host: "localhost"
    clickhouse_port: 8123
    clickhouse_user: "default"

Store the ClickHouse password in the keyring:

eventum-keyring set clickhouse_password

Run it

eventum run -c eventum.yml

The sessions below were produced by running this generator directly with eventum generate in sample mode (--live-mode false), with a short bound temporarily added to the traffic pattern so a handful of complete sessions render at once for inspection. eventum run in the application built above drops that bound and keeps end: "never", pacing sessions to the traffic pattern's actual daily curve in live mode instead of producing them all at once.

A converting session — three pages, a first item, two more pages, a second item, then checkout:

A converting session

{"timestamp": "2026-07-17 23:52:18.575101", "session_id": "e6199a85-3ed6-40b3-932c-3a0bff12b580", "user_agent": "Mozilla/5.0 (Windows; U; Windows NT 10.0) AppleWebKit/531.24.4 (KHTML, like Gecko) Version/4.0.1 Safari/531.24.4", "referrer": "https://twitter.com", "page": "/", "page_type": "landing", "duration_ms": 2690, "items_in_cart": 0}
{"timestamp": "2026-07-17 23:57:40.974241", "session_id": "e6199a85-3ed6-40b3-932c-3a0bff12b580", "user_agent": "Mozilla/5.0 (Windows; U; Windows NT 10.0) AppleWebKit/531.24.4 (KHTML, like Gecko) Version/4.0.1 Safari/531.24.4", "referrer": "https://twitter.com", "page": "/products/webcam", "page_type": "browse", "duration_ms": 14323, "items_in_cart": 0}
{"timestamp": "2026-07-18 00:05:33.659582", "session_id": "e6199a85-3ed6-40b3-932c-3a0bff12b580", "user_agent": "Mozilla/5.0 (Windows; U; Windows NT 10.0) AppleWebKit/531.24.4 (KHTML, like Gecko) Version/4.0.1 Safari/531.24.4", "referrer": "https://twitter.com", "page": "/products/webcam", "page_type": "browse", "duration_ms": 3793, "items_in_cart": 0}
{"timestamp": "2026-07-18 00:21:50.329135", "session_id": "e6199a85-3ed6-40b3-932c-3a0bff12b580", "user_agent": "Mozilla/5.0 (Windows; U; Windows NT 10.0) AppleWebKit/531.24.4 (KHTML, like Gecko) Version/4.0.1 Safari/531.24.4", "referrer": "https://twitter.com", "page": "/products", "page_type": "browse", "duration_ms": 8547, "items_in_cart": 0}
{"timestamp": "2026-07-18 00:40:17.632793", "session_id": "e6199a85-3ed6-40b3-932c-3a0bff12b580", "user_agent": "Mozilla/5.0 (Windows; U; Windows NT 10.0) AppleWebKit/531.24.4 (KHTML, like Gecko) Version/4.0.1 Safari/531.24.4", "referrer": "https://twitter.com", "page": "/cart", "page_type": "add_to_cart", "duration_ms": 2378, "items_in_cart": 1}
{"timestamp": "2026-07-18 00:50:17.960856", "session_id": "e6199a85-3ed6-40b3-932c-3a0bff12b580", "user_agent": "Mozilla/5.0 (Windows; U; Windows NT 10.0) AppleWebKit/531.24.4 (KHTML, like Gecko) Version/4.0.1 Safari/531.24.4", "referrer": "https://twitter.com", "page": "/products/usb-hub", "page_type": "browse", "duration_ms": 11509, "items_in_cart": 1}
{"timestamp": "2026-07-18 00:56:20.483416", "session_id": "e6199a85-3ed6-40b3-932c-3a0bff12b580", "user_agent": "Mozilla/5.0 (Windows; U; Windows NT 10.0) AppleWebKit/531.24.4 (KHTML, like Gecko) Version/4.0.1 Safari/531.24.4", "referrer": "https://twitter.com", "page": "/categories/home", "page_type": "browse", "duration_ms": 11637, "items_in_cart": 1}
{"timestamp": "2026-07-18 00:57:13.339342", "session_id": "e6199a85-3ed6-40b3-932c-3a0bff12b580", "user_agent": "Mozilla/5.0 (Windows; U; Windows NT 10.0) AppleWebKit/531.24.4 (KHTML, like Gecko) Version/4.0.1 Safari/531.24.4", "referrer": "https://twitter.com", "page": "/cart", "page_type": "add_to_cart", "duration_ms": 1597, "items_in_cart": 2}
{"timestamp": "2026-07-18 00:59:05.632261", "session_id": "e6199a85-3ed6-40b3-932c-3a0bff12b580", "user_agent": "Mozilla/5.0 (Windows; U; Windows NT 10.0) AppleWebKit/531.24.4 (KHTML, like Gecko) Version/4.0.1 Safari/531.24.4", "referrer": "https://twitter.com", "page": "/checkout/complete", "page_type": "checkout", "duration_ms": 2722, "items_in_cart": 2}
{"timestamp": "2026-07-18 01:02:06.693648", "session_id": "e6199a85-3ed6-40b3-932c-3a0bff12b580", "user_agent": "Mozilla/5.0 (Windows; U; Windows NT 10.0) AppleWebKit/531.24.4 (KHTML, like Gecko) Version/4.0.1 Safari/531.24.4", "referrer": "https://twitter.com", "page": "/", "page_type": "exit", "duration_ms": 711, "items_in_cart": 2}

Filtering the same run for a session that never added anything to its cart shows the bounce path: browse_streak reaches 5 with the cart still empty, and session_done sends it straight to exit instead of add-to-cart:

A bounced session

{"timestamp": "2026-07-18 01:46:49.808270", "session_id": "a056e002-3805-4c65-b8b4-a17f6a7c10f9", "user_agent": "Mozilla/5.0 (iPhone; CPU iPhone OS 1_1_5 like Mac OS X) AppleWebKit/532.2 (KHTML, like Gecko) CriOS/63.0.875.0 Mobile/90Y946 Safari/532.2", "referrer": "https://reddit.com", "page": "/", "page_type": "landing", "duration_ms": 4311, "items_in_cart": 0}
{"timestamp": "2026-07-18 01:48:51.048712", "session_id": "a056e002-3805-4c65-b8b4-a17f6a7c10f9", "user_agent": "Mozilla/5.0 (iPhone; CPU iPhone OS 1_1_5 like Mac OS X) AppleWebKit/532.2 (KHTML, like Gecko) CriOS/63.0.875.0 Mobile/90Y946 Safari/532.2", "referrer": "https://reddit.com", "page": "/products/usb-hub", "page_type": "browse", "duration_ms": 11673, "items_in_cart": 0}
{"timestamp": "2026-07-18 01:49:14.151790", "session_id": "a056e002-3805-4c65-b8b4-a17f6a7c10f9", "user_agent": "Mozilla/5.0 (iPhone; CPU iPhone OS 1_1_5 like Mac OS X) AppleWebKit/532.2 (KHTML, like Gecko) CriOS/63.0.875.0 Mobile/90Y946 Safari/532.2", "referrer": "https://reddit.com", "page": "/products", "page_type": "browse", "duration_ms": 10861, "items_in_cart": 0}
{"timestamp": "2026-07-18 01:49:48.756966", "session_id": "a056e002-3805-4c65-b8b4-a17f6a7c10f9", "user_agent": "Mozilla/5.0 (iPhone; CPU iPhone OS 1_1_5 like Mac OS X) AppleWebKit/532.2 (KHTML, like Gecko) CriOS/63.0.875.0 Mobile/90Y946 Safari/532.2", "referrer": "https://reddit.com", "page": "/products", "page_type": "browse", "duration_ms": 7044, "items_in_cart": 0}
{"timestamp": "2026-07-18 01:59:13.147119", "session_id": "a056e002-3805-4c65-b8b4-a17f6a7c10f9", "user_agent": "Mozilla/5.0 (iPhone; CPU iPhone OS 1_1_5 like Mac OS X) AppleWebKit/532.2 (KHTML, like Gecko) CriOS/63.0.875.0 Mobile/90Y946 Safari/532.2", "referrer": "https://reddit.com", "page": "/products/webcam", "page_type": "browse", "duration_ms": 11445, "items_in_cart": 0}
{"timestamp": "2026-07-18 02:02:56.904621", "session_id": "a056e002-3805-4c65-b8b4-a17f6a7c10f9", "user_agent": "Mozilla/5.0 (iPhone; CPU iPhone OS 1_1_5 like Mac OS X) AppleWebKit/532.2 (KHTML, like Gecko) CriOS/63.0.875.0 Mobile/90Y946 Safari/532.2", "referrer": "https://reddit.com", "page": "/products/wireless-mouse", "page_type": "browse", "duration_ms": 11214, "items_in_cart": 0}
{"timestamp": "2026-07-18 02:03:05.691342", "session_id": "a056e002-3805-4c65-b8b4-a17f6a7c10f9", "user_agent": "Mozilla/5.0 (iPhone; CPU iPhone OS 1_1_5 like Mac OS X) AppleWebKit/532.2 (KHTML, like Gecko) CriOS/63.0.875.0 Mobile/90Y946 Safari/532.2", "referrer": "https://reddit.com", "page": "/categories/electronics", "page_type": "exit", "duration_ms": 459, "items_in_cart": 0}

Of the four sessions this run touched, two converted, one bounced, and one was still browsing when the bounded run ended. Small samples vary, but the chance roll's math puts the theoretical bounce rate at one session in eight — three independent 50% rolls have to fail in a row before the fifth unconverted page view forces the exit.

Query the conversion funnel in ClickHouse:

SELECT
    page_type,
    count() AS views,
    uniqExact(session_id) AS sessions
FROM page_views
GROUP BY page_type
ORDER BY views DESC;

Going further

Tune the bounce rate — raise or lower browse.jinja's browse_streak ceiling or the chance roll's probability to match a real funnel's bounce rate; add a direct landing → exit transition for visitors who leave without browsing at all, a different and simpler bounce shape than the one built here.
A/B testing — use tags to label timestamps as variant A or B, then branch the FSM based on has_tags.
Multi-device sessions — run two generators in startup.yml with different user agent pools (mobile vs. desktop) writing to the same table.
Real-time dashboards — connect Grafana to ClickHouse and build a live funnel dashboard showing conversion rates as events stream in.

The Modeling sessions lesson for the finite-state-machine and state-flag technique this funnel reuses, applied there to a three-stage login/action/logout cycle instead of a five-stage browsing session
The Correlated events lesson for tracking many concurrent visitors in a pool instead of one session at a time
The Generate test data for ClickHouse lesson for the HTTP insert mechanics, connection pooling, and generateRandom comparison behind this output
The Streaming vs bulk lesson for live mode's continuous feed versus the bounded sample-mode batch used to inspect this generator
The Scenarios track for the broader set of scenarios synthetic data solves
The Eventum Hub for generators that already model common funnels, instead of starting the FSM from a blank state

Generate Clickstream Data for ClickHouse

Funnel test data and the bounce path

What you'll build

Prerequisites

Project structure

Prepare ClickHouse

Build it

Create the project directory

Define the daily traffic pattern

Write the session templates

Configure the generator

Configure the application

Run it

Going further

What's next

FSM picking mode

time-patterns reference

ClickHouse output

FAQ

On this page

Generate Clickstream Data for ClickHouse

FSM picking mode

time-patterns reference

ClickHouse output

What counts as a bounce in this generator?

Why does browse.jinja roll a random chance instead of comparing against a fixed page count?

Can this generator track more than one visitor browsing at the same time?

How do I read the conversion rate out of this data in ClickHouse?

On this page