# FAQ Getting started [#getting-started] Create a minimal generator directory with a template and a `generator.yml`, then run it directly from the command line: ```bash eventum generate --path ./my-generator --id demo ``` No `eventum.yml` or `startup.yml` is needed — [`eventum generate`](/docs/core/cli/eventum-generate) runs a single generator without the full application stack. See the [First run](/docs/core/introduction/first-run) guide for a step-by-step walkthrough. Use [`eventum generate`](/docs/core/cli/eventum-generate) for quick, single-generator runs — testing a template, piping output into another tool, or generating a one-off dataset. Use [`eventum run`](/docs/core/cli/eventum-run) when you need any of: * Multiple generators running concurrently. * The REST API or Studio web UI. * Hot-reload via `SIGHUP`. * Centralized logging, secrets, and configuration. No. Most workflows are covered entirely by YAML configuration and Jinja2 templates. Python is only required if you choose the [script event plugin](/docs/plugins/event/script), which is optional. Eventum requires **Python 3.14** or later. Use the tool that installed Eventum — `uv tool upgrade eventum-generator`, `pip install --upgrade eventum-generator`, or a newer Docker image tag — then restart the application. Configuration files and generators are not modified by an upgrade. See [Upgrading](/docs/core/introduction/upgrading). Templates and event generation [#templates-and-event-generation] Use the `module` context variable. It gives you access to three data-generation libraries: | Library | Access pattern | Strength | | -------------------------------------- | --------------------------------------------- | ---------------------------------- | | [Faker](https://faker.readthedocs.io/) | `module.faker.locale.en.name()` | 70+ locales, hundreds of providers | | [Mimesis](https://mimesis.name/) | `module.mimesis.locale.en.person.full_name()` | High performance, locale-aware | | rand | `module.rand.number.integer(1, 100)` | Lightweight random helpers | `module` is also a gateway to **any installed Python package** — for example, `module.json.dumps(...)` or `module.math.sqrt(...)`. Yes. Three state scopes are available inside templates: * **`locals`** — visible only to the current template. * **`shared`** — visible to all templates within the same generator. * **`globals`** — visible across all generators in the application (thread-safe). ```jinja {# increment a per-template counter #} {% do locals.set('n', locals.get('n', 0) + 1) %} {{ locals.n }} ``` See [Producing events — State](/docs/core/concepts/producing#state-management) for details. A picking mode determines which template(s) render on each event timestamp. Modes range from simple (render all, pick one at random, weighted probability) to advanced (round-robin cycling, fixed sequences, and a finite state machine with conditional transitions). See [Producing events — Picking modes](/docs/core/concepts/producing#picking-modes) for examples and the [template plugin reference](/docs/plugins/event/template) for the full list. Yes. Define [samples](/docs/plugins/event/template#samples) in `generator.yml` and reference them in templates via the `samples` variable. Supported formats: * **CSV** — each row becomes a dict with column headers as keys. * **JSON** — loaded as a Python object. * **Items** — inline list of strings defined directly in the YAML config. Scheduling and input [#scheduling-and-input] Choose the [input plugin](/docs/plugins#input-plugins) that matches your scheduling pattern — cron expressions, fixed intervals, evenly spaced ranges, statistical distributions, explicit datetime lists, or on-demand HTTP triggers. You can combine multiple input plugins in a single generator — their timestamps are [merged](/docs/core/concepts/scheduling#combining-multiple-inputs) in chronological order. UTC by default. Override it per-generator in [`startup.yml`](/docs/core/config/startup-yml), in the [generation defaults](/docs/core/config/eventum-yml#generation) in `eventum.yml`, or with the `--timezone` CLI flag. It depends on the `skip_past` setting: * **`true`** (default) — past timestamps are discarded and generation starts from "now." * **`false`** — past timestamps are emitted immediately as a burst, then generation continues in real time. Set `skip_past: false` when you need to backfill historical data or replay a time range. * **Live mode** (`live_mode: true`) — timestamps are released at their real wall-clock times. The generator waits until each scheduled moment arrives. * **Sample mode** (`live_mode: false`) — all timestamps are released as fast as possible, ignoring the clock. Useful for quickly generating large datasets or running benchmarks. Output and delivery [#output-and-delivery] Yes. Add multiple output plugins to the same generator. Every event is delivered to **all** of them (fan-out). For example, write to a file for local debugging and to ClickHouse for production storage simultaneously. Output plugins use [formatters](/docs/plugins/formatters) to transform events before delivery. Formats range from plain pass-through to JSON encoding and custom Jinja2 templates. See the [Formatters reference](/docs/plugins/formatters) for the full list. Set `keep_order: true` in the generator or generation config. This serializes output writes so events arrive in timestamp order. The default (`false`) allows concurrent writes for higher throughput. Output failures are **non-fatal** — the generator keeps running. Failed writes are counted in the `write_failed` metric and logged. Other output plugins are not affected. You can monitor error counts through the [REST API](/docs/api) or [Studio](/docs/studio/instances#instance-metrics). Configuration and secrets [#configuration-and-secrets] | | Parameters | Secrets | | -------------------- | --------------------------------------------- | ------------------------------------------------------------------------------------------------------ | | **Storage** | Plain text in `startup.yml` or CLI `--params` | Encrypted in the keyring (cryptfile) | | **Reference syntax** | `${params.name}` | `${secrets.name}` | | **Use case** | Hostnames, ports, feature flags, table names | Passwords, API keys, tokens | | **Management** | Edit YAML directly | Use [`eventum-keyring`](/docs/core/cli/eventum-keyring) CLI or [Studio](/docs/studio/settings#secrets) | Use the [`eventum-keyring`](/docs/core/cli/eventum-keyring) CLI to add secrets to the encrypted cryptfile: ```bash eventum-keyring set db_password ``` Then reference the secret in `generator.yml`: ```yaml password: ${secrets.db_password} ``` The cryptfile is protected by a keyring password that you provide at runtime (via the `EVENTUM_KEYRING_PASSWORD` environment variable or interactive prompt). Yes. Use `${params.name}` tokens in the config and pass different values per environment through [`startup.yml`](/docs/core/config/startup-yml) or the `--params` CLI flag. This keeps generator configs portable — the same template works for dev, staging, and production. Eventum raises an error at load time and the generator does not start. This prevents incomplete configuration from running silently. The error message names the missing variable. Paths in `eventum.yml` — both `path.*` and `server.ssl.*` — must be **absolute**. Inside `generator.yml`, relative paths (e.g. `templates/event.jinja`) are resolved from the directory containing the generator config file, not from the working directory. See [Project structure — Path resolution](/docs/core/config/project-structure#path-resolution). Performance and tuning [#performance-and-tuning] Several knobs affect throughput: * **`count`** (input plugins) — how many timestamps per tick. * **`batch.size`** — how many events are grouped before passing to output. Larger batches reduce per-event overhead. * **`batch.delay`** — maximum time span of the event timestamps one batch covers. It applies to timestamps still ahead of real time in live mode; timestamps that have already passed, and every timestamp in sample mode, are grouped by `batch.size` instead. * **`max_concurrency`** — limit on parallel output write operations. Backpressure prevents memory overflow when one pipeline stage is slower than the one feeding it. If output plugins are slow, the event queue fills up and event production pauses. If event production is slow, the timestamp queue fills up and input pauses. Queue depths are controlled by `queue.max_timestamp_batches` and `queue.max_event_batches`. Use **sample mode** (`live_mode: false`). It generates events as fast as possible without waiting for wall-clock alignment, which maximizes throughput for load testing and dataset generation. Deployment and operations [#deployment-and-operations] Send `SIGHUP` to the Eventum process: ```bash kill -HUP $(pidof eventum) ``` This triggers a hot reload: the whole application stops and starts again from `eventum.yml` and `startup.yml`, without the process being replaced. The server stops with everything else, so the REST API and Studio are unreachable for a moment. Yes. Generators with `autostart: false` are registered but not started on launch. Start them later through the [REST API](/docs/api) or [Studio Instances page](/docs/studio/instances#lifecycle-controls). Within a single Eventum application — yes, through the `globals` variable, which is thread-safe. Across separate Eventum processes — no. Cross-process state requires external coordination (a shared database, message queue, etc.). Several approaches, from fastest to most thorough: 1. **Studio debugger** — open the [event debugger](/docs/studio/projects#debugger) in the project console, set a timestamp, and click Produce to see output instantly. 2. **Stdout output** — add a `stdout` output plugin and inspect events in the terminal. 3. **Verbose logging** — run with `-vvvvv` to enable DEBUG-level logs on stderr. 4. **Error counters** — check `produce_failed`, `format_failed`, and `write_failed` metrics via the REST API or Studio. * **Studio dashboard** — the [Monitoring page](/docs/studio/monitoring) shows pipeline flow, throughput, failures and resource usage in real time. * **REST API** — query per-instance metrics (generated, written, failed counters, EPS) programmatically. * **Logs** — structured JSON logs (`log.format: json` in `eventum.yml`) can be ingested by your existing log aggregation stack. Troubleshooting [#troubleshooting] 1. **YAML syntax** — run a YAML linter on your config files. 2. **Missing variables** — verify all `${params.*}` and `${secrets.*}` tokens have corresponding values. 3. **File paths** — confirm template, sample, and pattern file paths exist relative to the generator directory. 4. **Logs** — check the error message in the logs. Validation errors include the field name and expected type. By default, output plugins write concurrently. Set `keep_order: true` in the generation config to serialize writes and maintain strict chronological ordering. Note that this reduces throughput. 1. Check the template in the [Studio debugger](/docs/studio/projects#debugger) — it shows the exact output for a given timestamp. 2. Verify your Jinja2 syntax — missing `{{ }}` delimiters or incorrect filter usage are common issues. 3. Check that sample data files exist and are in the expected format (CSV with headers, valid JSON). 4. If using state (`locals`, `shared`, `globals`), use the [State view](/docs/studio/projects#state) of the project console to inspect current values. Check the per-plugin error counters: | Counter | Meaning | | ---------------- | ------------------------------------------------------------------------------------- | | `format_failed` | The formatter rejected the event (e.g., invalid JSON for a `json` formatter). | | `write_failed` | The output plugin failed to deliver the event (network error, timeout, auth failure). | | `produce_failed` | The event plugin failed to render the template (Jinja2 error, script exception). | These counters are available through the [REST API](/docs/api) and the [Instance metrics dialog](/docs/studio/instances#instance-metrics) in Studio. Identify the bottleneck stage: * **Input slow** — timestamp generation is the limiter. Simplify the scheduling pattern or reduce `count`. * **Event production slow** — template rendering is expensive. Simplify templates, reduce sample dataset sizes, or use the `script` plugin for compute-heavy logic. * **Output slow** — the destination can't keep up. Increase `max_concurrency`, enlarge `batch.size`, raise or drop `batch.delay` so batches are not cut short of that size, or check the destination's health. Monitor throughput and per-stage failures on the [Monitoring dashboard](/docs/studio/monitoring) to pinpoint which stage is falling behind. # Glossary
A B C E F G H I K L M O P Q R S T V
A [#a] Autostart [#autostart] A boolean flag in [`startup.yml`](/docs/core/config/startup-yml) that controls whether a generator instance starts automatically when the Eventum application launches. Instances with `autostart: false` are registered but remain inactive until started manually through the REST API or Studio. B [#b] Backpressure [#backpressure] A flow-control mechanism that prevents memory overflow when one pipeline stage is slower than the one feeding it. If the event queue is full, event production pauses. If the timestamp queue is full, input generation pauses. Queue depths are controlled by `queue.max_timestamp_batches` and `queue.max_event_batches`. Batch [#batch] A group of events processed together as a unit. Batching amortizes per-event overhead and is controlled by two parameters: `batch.size` (maximum events per batch) and `batch.delay` (maximum time span of the event timestamps one batch covers). See [eventum.yml — Generation](/docs/core/config/eventum-yml#generation). C [#c] Chain [#chain] A [picking mode](/docs/core/concepts/producing#picking-modes) that cycles through templates in a fixed, user-defined sequence. Unlike `spin` (which uses the natural order), `chain` lets you specify an explicit order with optional repeats. Chance [#chance] A [picking mode](/docs/core/concepts/producing#picking-modes) that selects a template using weighted probabilities. Each template's `chance` is a relative weight — only the ratio between templates matters, so the values need not sum to 1. Cryptfile [#cryptfile] An AES-encrypted file that stores secrets, managed by the `keyrings.cryptfile` library. Protected by a keyring password provided at runtime. See [Secrets](/docs/core/config/secrets). E [#e] Event plugin [#event-plugin] The second stage of the [generator pipeline](/docs/core/concepts/generator#the-three-stage-pipeline). Receives timestamps from input plugins and produces event strings. See [event plugins](/docs/plugins#event-plugins) for available types. Eventum Studio [#eventum-studio] The built-in web interface shipped with Eventum. Provides a visual dashboard, generator management, plugin editors, event debugger, and administration tools — all accessible from a browser. See [Eventum Studio](/docs/studio). eventum.yml [#eventumyml] The main application configuration file. Defines server settings (host, port, SSL, authentication), generation defaults (timezone, batching, concurrency), logging, and file paths. See [eventum.yml reference](/docs/core/config/eventum-yml). F [#f] Fan-out [#fan-out] The delivery model used by output plugins. Every event produced by the event plugin is sent to **all** configured output plugins. There is no routing or filtering — each output receives a copy of every event. Finite State Machine (FSM) [#finite-state-machine-fsm] A [picking mode](/docs/core/concepts/producing#picking-modes) for stateful event sequences. Each template is a state with defined transitions. Transitions can be unconditional or conditional (evaluated at runtime using Jinja2 expressions). Useful for modeling workflows like user sessions. Formatter [#formatter] A transformation layer on output plugins that reshapes event strings before delivery. See [Formatters](/docs/plugins/formatters) for available formats. G [#g] Generator [#generator] The core execution unit in Eventum. A generator is a self-contained pipeline that produces events through three stages: input (when), event (what), and output (where). Each generator runs independently with its own configuration, state, and metrics. See [Generator concepts](/docs/core/concepts/generator). generator.yml [#generatoryml] The configuration file for a single generator. Defines the input plugin(s), event plugin, and output plugin(s) that form the generator pipeline. Located inside a generator directory. See [generator.yml reference](/docs/core/config/generator-yml). Globals [#globals] A state scope shared across all generators in the application. Accessible as `globals` in templates and in the `produce` function of the `script` plugin. Thread-safe — reads and writes are synchronized. Useful for cross-generator coordination. See [Producing events — State](/docs/core/concepts/producing#state-management). H [#h] Hot reload [#hot-reload] Stopping the whole application and starting it again from its configuration files, without ending the process. Triggered by sending `SIGHUP` to the Eventum process or clicking **Restart** in Studio. I [#i] Input plugin [#input-plugin] The first stage of the [generator pipeline](/docs/core/concepts/generator#the-three-stage-pipeline). Produces timestamps that define when events should be generated. See [input plugins](/docs/plugins#input-plugins) for available types. Instance [#instance] A running (or finished) copy of a generator. Each instance has a unique ID, its own metrics, logs, and lifecycle. Multiple instances can use the same generator project with different parameters. Managed through the [Instances page](/docs/studio/instances) in Studio or the REST API. K [#k] Keep order [#keep-order] A generation flag (`keep_order: true`) that serializes output writes to maintain strict chronological order. When disabled (default), output plugins write concurrently for higher throughput. Keyring [#keyring] The encrypted credential store used by Eventum. Secrets are added with [`eventum-keyring set`](/docs/core/cli/eventum-keyring) and referenced in configs via `${secrets.name}` tokens. See [Secrets](/docs/core/config/secrets). L [#l] Live mode [#live-mode] The default operating mode (`live_mode: true`). Timestamps are released at their real wall-clock times — the generator waits until each scheduled moment arrives before emitting the event. Contrast with [sample mode](#sample-mode). Locals [#locals] A state scope private to a single template. Each template has its own `locals` dictionary, invisible to other templates. Useful for per-template counters and accumulators. See [Producing events — State](/docs/core/concepts/producing#state-management). M [#m] Merging [#merging] The process of combining timestamps from multiple input plugins in a single generator. Timestamps are interleaved in chronological order so the event plugin receives a single unified stream. See [Scheduling — Merging](/docs/core/concepts/scheduling#merging-multiple-inputs). Module [#module] A context variable available in templates that acts as a gateway to Python packages. Provides access to [Faker](https://faker.readthedocs.io/) (`module.faker`), [Mimesis](https://mimesis.name/) (`module.mimesis`), the built-in `rand` helper (`module.rand`), and any installed Python package (`module.json`, `module.math`, etc.). Multiplier [#multiplier] A component in [time-patterns](/docs/plugins/input/time-patterns) scheduling that sets the baseline number of events per period. Combined with a [randomizer](#randomizer) to add variance. See [Scheduling — Multiplier](/docs/core/concepts/scheduling#multiplier). O [#o] Oscillator [#oscillator] A component in [time-patterns](/docs/plugins/input/time-patterns) scheduling that divides time into repeating periods (e.g., every hour, every day). Defined by a start time, end time, period length, and period unit. See [Scheduling — Oscillator](/docs/core/concepts/scheduling#oscillator). Output plugin [#output-plugin] The third stage of the [generator pipeline](/docs/core/concepts/generator#the-three-stage-pipeline). Receives formatted events and writes them to a destination. See [output plugins](/docs/plugins#output-plugins) for available types. P [#p] Parameters [#parameters] Runtime values injected into configuration files through `${params.name}` tokens. Defined in [`startup.yml`](/docs/core/config/startup-yml) per-generator or passed via the `--params` CLI flag. See [Parameters](/docs/core/config/parameters). Picking mode [#picking-mode] A strategy that determines which template(s) render on each event timestamp. See [Producing events — Picking modes](/docs/core/concepts/producing#picking-modes) for available modes. Pipeline [#pipeline] The three-stage data flow inside a generator: Input (timestamps) → Event (strings) → Output (delivery). Each stage is handled by one or more plugins. See [Generator — The three-stage pipeline](/docs/core/concepts/generator#the-three-stage-pipeline). Plugin [#plugin] A swappable, self-contained component that handles one stage of the generator pipeline. Plugins are identified by type (input, event, output) and name (e.g., `cron`, `template`, `clickhouse`). See [Plugins concepts](/docs/core/concepts/plugins). Project [#project] In Studio, a project corresponds to a generator directory on disk — a `generator.yml` file and its supporting resources (templates, scripts, samples). The [Projects page](/docs/studio/projects) provides visual editors for each project. Q [#q] Queue [#queue] An internal buffer between pipeline stages. Two queues exist per generator: a timestamp queue (input → event) and an event queue (event → output). Queue sizes control [backpressure](#backpressure). R [#r] Rand [#rand] A lightweight built-in helper available in templates as `module.rand`. Provides random choices, weighted selection, numbers (integer, float, gaussian), and string generation (hex, digits, letters). See [Producing events — rand](/docs/core/concepts/producing#rand). Randomizer [#randomizer] A component in [time-patterns](/docs/plugins/input/time-patterns) scheduling that adds natural variance to the baseline event count set by the [multiplier](#multiplier). Controlled by a deviation parameter. See [Scheduling — Randomizer](/docs/core/concepts/scheduling#randomizer). Replay plugin [#replay-plugin] An [event plugin](/docs/plugins/event/replay) that reads events from existing log or data files instead of generating new ones. Optionally replaces embedded timestamps with current values using regex patterns. REST API [#rest-api] HTTP endpoints for programmatic control of Eventum — managing generators, querying metrics, reading logs, and modifying configuration. Enabled via `api.enabled: true` in `eventum.yml`. See [API reference](/docs/api). S [#s] Sample mode [#sample-mode] An operating mode (`live_mode: false`) where all timestamps are released as fast as possible, ignoring the wall clock. Used for quickly generating large datasets, benchmarking, or backfilling. Contrast with [live mode](#live-mode). Samples [#samples] External datasets loaded into template context from CSV, JSON, or inline item lists. Defined in `generator.yml` and accessed in templates via the `samples` variable. See [Template plugin — Samples](/docs/plugins/event/template#samples). Script plugin [#script-plugin] An [event plugin](/docs/plugins/event/script) that executes a Python function to produce events. The function receives the timestamp, tags, and parameters, and returns one or more event strings. Used when Jinja2 templates cannot express the required logic. Secrets [#secrets] Encrypted credentials stored in the [keyring](#keyring) and referenced in configs via `${secrets.name}` tokens. Managed with the [`eventum-keyring`](/docs/core/cli/eventum-keyring) CLI or the [Secrets page](/docs/studio/settings#secrets) in Studio. See [Secrets](/docs/core/config/secrets). Shared [#shared] A state scope visible to all templates within the same generator. Accessible as `shared` in templates. Useful for cross-template coordination like maintaining a shared counter or session pool. See [Producing events — State](/docs/core/concepts/producing#state-management). Skip past [#skip-past] A boolean flag (`skip_past`) that controls whether timestamps in the past are discarded when a generator starts. When `true` (default), only future timestamps are emitted. When `false`, past timestamps are released immediately as a burst. Spin [#spin] A [picking mode](/docs/core/concepts/producing#picking-modes) that cycles through templates in round-robin order — template 1, template 2, ..., template N, template 1, and so on. Spreader [#spreader] A component in [time-patterns](/docs/plugins/input/time-patterns) scheduling that distributes events within each period using a probability distribution (uniform, triangular, beta). Controls the intra-period traffic shape. See [Scheduling — Spreader](/docs/core/concepts/scheduling#spreader). startup.yml [#startupyml] The configuration file that lists which generators to run, with per-generator overrides for mode, parameters, and generation settings. See [startup.yml reference](/docs/core/config/startup-yml). T [#t] Tags [#tags] Arbitrary string labels attached by input plugins to timestamps. Tags are propagated through the pipeline and available in templates via the `tags` variable. Useful for varying event content based on the scheduling source. Template plugin [#template-plugin] The primary [event plugin](/docs/plugins/event/template) that renders Jinja2 templates with access to data-generation libraries, parameters, samples, and persistent state. Supports multiple templates with configurable [picking modes](#picking-mode). Timestamp [#timestamp] A timezone-aware datetime value produced by an input plugin. Represents the scheduled moment for an event. Passed to the event plugin as the `timestamp` context variable. V [#v] Variable substitution [#variable-substitution] A pre-processing step that replaces `${params.name}` and `${secrets.name}` tokens in YAML configuration files before parsing. Uses Jinja2 syntax internally. See [Parameters](/docs/core/config/parameters) and [Secrets](/docs/core/config/secrets). VersatileDatetime [#versatiledatetime] A flexible datetime format accepted by input plugins. Supports ISO 8601 (`2025-01-01T00:00:00Z`), human-readable strings (`January 1, 2025`), keywords (`now`, `never`), and relative expressions (`+1h`, `-30m`). See [generator.yml — VersatileDatetime](/docs/core/config/generator-yml#versatiledatetime). # Getting started
Eventum
Welcome to Eventum
A data generation platform that produces synthetic events and delivers them anywhere — in real time or in sample. Describe what your data looks like and when it should appear, Eventum takes care of the rest.
*** Why Eventum [#why-eventum] Building and testing data-driven systems usually means waiting for real data to accumulate or writing throwaway scripts. Eventum gives you a declarative way to generate realistic data on demand.
Realistic content
Templates powered by Faker and Mimesis produce believable names, IPs, timestamps, and domain-specific values.
Precise timing
Cron schedules, fixed intervals, uniform ranges, or statistical patterns that mimic real-world traffic.
Multiple destinations
stdout, files, ClickHouse, OpenSearch, or any HTTP endpoint — all at the same time.
Live and sample modes
Stream events at their actual timestamps, or generate an entire dataset as fast as possible.
*** How it works [#how-it-works] Every generator runs a three-stage pipeline. You describe each stage in YAML — no code required.
Input
Defines when events occur — schedules, timers, cron, or time patterns.
Event
Defines what events look like — Jinja2 templates, Python scripts, or log replay.
Output
Defines where events go — stdout, files, databases, or HTTP endpoints.
Swap any part independently: change the schedule without touching the template, or add a new output without modifying anything else. *** Quick example [#quick-example] Create a template [#create-a-template] Templates use Jinja2 syntax with built-in modules like `faker` for realistic data. Each render receives a `timestamp` variable. ```jinja title="events.jinja" {{ timestamp }} INFO user={{ module.faker.locale.en.user_name() }} action=login ip={{ module.faker.locale.en.ipv4() }} ``` Create a generator config [#create-a-generator-config] Wire the three pipeline stages together — `cron` input firing every second, `template` event plugin, and `stdout` output. ```yaml title="generator.yml" input: - cron: expression: "* * * * * *" count: 1 event: template: mode: all templates: - my_event: template: events.jinja output: - stdout: {} ``` Run the generator [#run-the-generator] ```bash eventum generate --path generator.yml --live-mode ``` Events start printing at one per second: ``` 2025-06-15 12:00:01+00:00 INFO user=jsmith action=login ip=192.168.44.12 2025-06-15 12:00:02+00:00 INFO user=amiller action=login ip=10.0.128.55 2025-06-15 12:00:03+00:00 INFO user=kwilson action=login ip=172.16.0.91 ``` This example runs a single generator from the command line. To run multiple generators with a server, API, and web UI, see the [`eventum run`](/docs/core/cli/eventum-run) command. *** What's next [#whats-next] # Overview {/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */} # Overview
Core documentation
Install Eventum, understand the pipeline model, configure generators, and master the CLI. Follow in order or jump to what you need.
*** Introduction [#introduction] Install Eventum, learn what it does, and run your first generator from zero to output in under five minutes. *** Concepts [#concepts] How Eventum works under the hood — the generator pipeline, plugin system, scheduling strategies, event production, and output delivery. *** Configuration [#configuration] YAML config files, project layout, and runtime variables — everything that controls how generators behave. *** CLI [#cli] Four commands that cover every runtime scenario — single-generator runs, full application mode, systemd service management, and secret management. # What's Next
You've mastered the core
The generator pipeline, plugin system, scheduling, event production, output delivery, configuration files, and the CLI — all covered. Now pick a direction and start building.
✓ Generators
✓ Plugins
✓ Scheduling
✓ Producing
✓ Output
✓ Configuration
✓ CLI
✓ Secrets
*** Plugins reference [#plugins-reference] The core docs described **how** the pipeline works. The plugin reference tells you **what's available** — every parameter, default, and constraint for each plugin. This is your go-to reference when writing `generator.yml` files. *** Eventum Studio [#eventum-studio] The built-in web UI for managing everything visually. Browse your project, edit configs in the browser, preview events in real time, and monitor running generators — no terminal required. Best for iterative template development and debugging. *** REST API [#rest-api] Manage generators programmatically — start, stop, create, update, and monitor over HTTP. Use this when you're integrating Eventum into a larger system, building automation, or controlling generators from external tools. *** Tutorials [#tutorials] Theory is covered — now build something real. Each tutorial walks through a complete project from an empty directory to running output, covering a different combination of plugins and patterns. # 🎉 2.0.0 Released **February 20, 2026** 🎉 Eventum 2.0 is a complete rewrite from the ground up! New plugin-based architecture, batch processing, a built-in web UI, and much more. This is a major release — configurations from 1.x will need to be updated. 🚀 New features [#-new-features] Input plugins [#input-plugins] * **[`http`](/docs/plugins/input/http) input plugin** — trigger event generation from external systems via HTTP requests. * **Live & sample modes** for all input plugins — run in real-time or generate as fast as possible. * **Human-readable dates** — write `"January 1, 2025"`, `"+1h"`, or `"now"` instead of strict ISO formats. * **Multiple input merging** — combine several input plugins in one generator, timestamps are merged automatically in chronological order. Event plugins [#event-plugins] * **[`script`](/docs/plugins/event/script) plugin** — write event logic as a Python function when templates aren't enough. * **[`replay`](/docs/plugins/event/replay) plugin** — replay events from existing log files with optional timestamp replacement. Template plugin enhancements [#template-plugin-enhancements] * **Faker & Mimesis** — two powerful data generation libraries available directly in templates (70+ locales, hundreds of data providers). * **`module` gateway** — access any installed Python package in templates via `module.`. * **Global state** — new `globals` scope for sharing state across all generators (thread-safe). * **New state methods** — `update`, `clear`, and `as_dict` for all state scopes. * **New picking modes** — [`fsm`](/docs/core/concepts/producing#fsm) (finite state machine) and [`chain`](/docs/core/concepts/producing#chain) (fixed sequence). * **New sample types** — `json` and `items` (inline lists in YAML). * **Timezone-aware timestamps** — `timestamp` is now a proper `datetime` object, not a string. * **Better subprocesses** — new `cwd`, `env`, and `timeout` options. Output plugins [#output-plugins] * **[`clickhouse`](/docs/plugins/output/clickhouse) plugin** — write events directly to ClickHouse. * **[`http`](/docs/plugins/output/http) plugin** — send events to any HTTP endpoint. * **Formatters** — transform events before delivery with `plain`, `json`, `json-batch`, `template`, or `template-batch`. Existing output plugin improvements [#existing-output-plugin-improvements] * **File** — new `flush_interval`, `cleanup_interval`, `file_mode`, `write_mode`, `encoding`, and `separator` options. * **Stdout** — new `flush_interval`, `stream`, `encoding`, and `separator` options. * **OpenSearch** — new `connect_timeout`, `request_timeout`, `client_cert`, `client_cert_key`, and `proxy_url` options. *** ⚡ Performance [#-performance] Batch processing across the entire pipeline — events are grouped into configurable batches between stages, dramatically reducing overhead and improving throughput compared to 1.x. *** 🧪 Testing [#-testing] Expanded test coverage for all plugins, the core executor, configuration loading, and the CLI. *** 🏗️ Architecture [#️-architecture] The entire project has been rebuilt from scratch: * **Plugin system** — self-registering plugins with a consistent structure. * **Async pipeline** — `uvloop` event loop with `janus` queues for efficient stage-to-stage communication. * **Configuration** — Pydantic-based validation with `${params.*}` and `${secrets.*}` variable substitution. * **CLI** — rebuilt with Click, options auto-generated from config models. * **REST API** — new FastAPI-based API for programmatic control. * **Eventum Studio** — brand-new React web UI for visual editing, debugging, and monitoring. *** 📝 Other changes [#-other-changes] * 🔄 `sample` input plugin renamed to [`static`](/docs/plugins/input/static). * 🔄 `jinja` event plugin renamed to [`template`](/docs/plugins/event/template). * 📋 Structured logging via [structlog](https://www.structlog.org/) — supports plain-text and JSON output. * 🛡️ Better error diagnostics — exceptions now carry structured context for easier troubleshooting. # 🛠️ 2.0.1 Released **February 21, 2026** A patch release with bug fixes, significantly expanded test coverage, and CI improvements. 🐛 Bug fixes [#-bug-fixes] * **`--params` CLI option** — fixed JSON input not being parsed correctly for dict-type Click parameters. * **Path extension validation** — fixed pydantic validation error when validating file path extensions (`.csv`, `.json`, `.jinja`) by replacing `Field(pattern=...)` with `@field_validator` on `Path` fields. *** 🧪 Testing [#-testing] Added comprehensive tests across all packages: * **API** — auth, generators, configs, instances, startup, secrets, file tree, timestamps aggregation. * **App** — generator and parameter models. * **CLI** — keyring commands and pydantic converter. * **Core** — config loader, generator, initializer, and parameters. * **Plugins** — ClickHouse and stdout output plugin configs. * **Server** — server main and UI routes. *** 📝 Other changes [#-other-changes] * Updated app slogan in CLI splash screen. * Updated documentation links. * Added Codecov badge to CI. * Added HTML report export to CI. * Fixed Docker build. # 🛠️ 2.0.2 Released **February 21, 2026** A patch release that fixes an issue with reading generator configs containing parameter and secret placeholders through the REST API. 🐛 Bug fixes [#-bug-fixes] * **Generator config API** — fixed the GET endpoint returning a validation error when a generator config contained `${params.*}` or `${secrets.*}` placeholders. The endpoint now uses loose validation for reading configs while keeping strict validation for creating and updating. *** 🧪 Testing [#-testing] * Added test for reading generator configs with placeholders via the API. *** 📝 Other changes [#-other-changes] * Updated GitHub URLs in README and pyproject.toml to match the new organization. * Improved release script with detailed usage instructions and phase handling. # 🚩 2.1.0 Released **February 21, 2026** A minor release with a significant performance improvement, new features for working with sample data and placeholders, and several bug fixes. 🚀 New features [#-new-features] * **Named sample access** — access CSV and JSON sample data by column name in templates (e.g., `sample.column_name` instead of `sample[0]`). * **Placeholder support in Studio** — plugin config forms in [Eventum Studio](/docs/studio/overview) now accept `${params.*}` and `${secrets.*}` placeholders. *** ⚡ Performance [#-performance] * **Timezone handling** — migrated from `pytz` to Python's built-in `zoneinfo` module, resulting in up to 2x faster event producing. *** 🐛 Bug fixes [#-bug-fixes] * **Missing dataset headers** — CSV and JSON samples without headers now get automatically generated default column names. * **YAML comments in configs** — full-line YAML comments no longer break config loading when used with template variables. * **File output plugin** — fixed file not being properly closed before reopening. * **Heterogeneous JSON samples** — samples with inconsistent keys across records are now handled correctly. * **Stdout output plugin** — fixed a platform-specific bug by switching from `writelines()` to `write()`. *** 🧪 Testing [#-testing] * Expanded test coverage for placeholder support, sample data handling, config loading, and API model validation. *** 📝 Other changes [#-other-changes] * Added Eventum Improvement Proposals document. * Formatted default values in plugin parameter schemas for better readability. # 🚩 2.2.0 Released **February 27, 2026** A minor release introducing systemd service management, weighted sampling for templates, per-template variables, expanded random distribution functions, and several quality-of-life fixes. 🚀 New features [#-new-features] * **Systemd service management** — new CLI commands to install, uninstall, start, stop, restart, and check the status of Eventum as a systemd service. * **Weighted sampling** — CSV and JSON samples can now include a weight column for non-uniform random selection of rows in templates. * **Per-template variables** — templates can define their own local variables alongside shared ones, enabling more modular generator configurations. * **Random distribution functions** — new `rand.gauss`, `rand.triangular`, `rand.expo`, `rand.lognorm`, `rand.beta`, and `rand.pareto` methods for realistic data generation. * **Dict input for `rand.weighted_choice`** — pass weight mappings directly without needing a separate sample file. *** 🐛 Bug fixes [#-bug-fixes] * **CSV quoting** — added `quotechar` config to the CSV sample reader with better error messages for inconsistent column counts. * **File output directories** — the file output plugin now creates intermediate directories automatically. * **Studio links** — updated community links to GitHub Discussions in Eventum Studio. # 🚩 2.3.0 Released **March 3, 2026** A minor release bringing three new output plugins for network-based event delivery, a move to Python 3.14t (free-threaded), and a reworked multithreaded core architecture. 🚀 New features [#-new-features] * **[Kafka](/docs/plugins/output/kafka) output plugin** — full Apache Kafka integration with SASL authentication, SSL/mTLS, compression (gzip, snappy, lz4, zstd), and batching. * **[TCP](/docs/plugins/output/tcp) output plugin** — send events over persistent TCP connections with SSL/TLS support and automatic reconnection. * **[UDP](/docs/plugins/output/udp) output plugin** — send events as UDP datagrams. *** ⚡ Performance [#-performance] * **Python 3.14t** — migrated to free-threaded Python 3.14 for improved concurrency. * **Multithreaded architecture** — the core pipeline has been reworked to leverage multithreading for better performance and reliability. *** 🧪 Testing [#-testing] * Expanded test coverage with integration and performance tests. *** 📝 Other changes [#-other-changes] * Expanded [template plugin](/docs/plugins/event/template) documentation. * Added [blog](/blog) to the documentation site. # 🛠️ 2.3.1 Released **March 24, 2026** A patch release fixing the broken Docker image. 🐛 Bug fixes [#-bug-fixes] * **Fix Docker image startup** — the container failed with `exec /app/.venv/bin/eventum: no such file or directory` due to issues in the Dockerfile. # 🚩 2.4.0 Released **April 2, 2026** 🚀 New features [#-new-features] * **[Scenarios](/docs/studio/scenarios)** — group generator instances into named workflows with shared global state, bulk start/stop, and an interactive Data Flow diagram * **[Instance metrics](/docs/studio/instances#instance-metrics)** — metrics modal redesigned as an interactive pipeline graph * **[Home page](/docs/studio/overview)** — new home page with quick actions and recent projects; [Monitoring](/docs/studio/monitoring) moved to its own page * **[State management](/docs/studio/projects#state)** — project State tab is now an editable key-value table with a full JSON editor *** 🐛 Bug fixes [#-bug-fixes] * Fix error when listing generators with non-standard directory paths * Fix global-state usage endpoint conflict # 🚩 2.5.0 Released **May 14, 2026** 🚀 New features [#-new-features] * **[Template dispatch API](/docs/plugins/event/template/dispatch)** — route each event across multiple sub-templates, end generation early, or drop unwanted events; dropped events are now visible in the [pipeline metrics](/docs/studio/instances#instance-metrics) and the monitoring panel * **[`rand.network.ip_v4_in_subnet`](/docs/plugins/event/template/modules)** — pick a random IPv4 host address inside a CIDR subnet *** 🐛 Bug fixes [#-bug-fixes] * Fix shutdown hang when a log-stream WebSocket stays open — termination no longer blocks on active streams # 🚩 2.6.0 Released **June 11, 2026** 🚀 New features [#-new-features] * **[MCP server](/docs/mcp)** — connect an AI agent (Claude Code, Cursor, Codex, and others) to Eventum and have it build, validate, preview, and run generators from a plain-language description. Runs over stdio for [local authoring](/docs/mcp/connect#local-authoring-over-stdio) and as an HTTP service on a running server for [live generator management](/docs/mcp/connect#live-management-over-http) behind authentication. The agent checks every step against Eventum itself, so the result is a working generator, not a guess — see [how it works](/docs/mcp/how-it-works) and the full [tool reference](/docs/mcp/tools) * **[`samples..where(**conditions)`](/docs/plugins/event/template/samples)** — filter sample rows by several equality conditions in one call, replacing verbose `selectattr` chains; the result supports further `where` and `pick` calls * **[`pick(default=...)` and `weighted_pick(weight, default=...)`](/docs/plugins/event/template/samples)** — return a fallback value when a sample selection is empty instead of failing the event * **[`rand.network.ip_v6` family](/docs/plugins/event/template/modules)** — random IPv6 addresses: the full space, global unicast, link-local, or unique local * **[`rand.network.mac(oui=..., vendor=...)`](/docs/plugins/event/template/modules)** — fix the MAC prefix to a specific OUI or pick one from a built-in table of 20 vendors (Apple, Cisco, Dell, Intel, VMware, and others) * **[`rand.network.ip_v4_private()`](/docs/plugins/event/template/modules)** — random RFC 1918 private IPv4 addresses with realistic class weights * **[`rand.string.pattern(format_string)`](/docs/plugins/event/template/modules)** — random strings from a printf-like pattern, e.g. `pattern("ORD-%A{3}-%d{6}")` * **[`rand.crypto.sha1()`](/docs/plugins/event/template/modules)** — random SHA-1-length hex strings * **[ClickHouse output: `pool_maxsize`](/docs/plugins/output/clickhouse)** — size the connection pool toward ClickHouse to avoid pool-exhaustion warnings under bursts of concurrent writes 🐛 Bug fixes [#-bug-fixes] * Dot-separated config keys now work at any depth in every YAML file — `eventum.yml`, generator configs, `startup.yml`, and time patterns accept `server: {mcp.enabled: true}`-style spellings, mixed freely with nested blocks; defining the same key twice fails with the exact conflicting path * A failed server startup (for example, when the port is already in use) now stops the app with a clear error instead of hanging until interrupted # 🚩 2.7.0 Released **August 1, 2026** Eventum Studio is refreshed across every screen on a single design system: the project page works as a development studio, the instance and Monitoring pages follow the pipeline live, and every list carries runtime stats and filters. Projects, instances, scenarios and secrets can now be renamed instead of recreated — from Studio, over the REST API, and through MCP, where agents also gain tools for scenarios, global state, settings and instance control. Three changes alter how an existing setup behaves: * **Dot-separated keys in a generator configuration** are no longer expanded into nested blocks. Write `formatter: {format: plain}` instead of `formatter.format: plain`. The shorthand keeps working in `eventum.yml`, `startup.yml` and time-pattern files. * **TLS certificate verification** is on by default in the [`opensearch`](/docs/plugins/output/opensearch) and [`http`](/docs/plugins/output/http) outputs. For a self-signed or internal-CA certificate, set `ca_cert` to the issuing CA, or `verify: false` to connect without the check. * **`server.ui_enabled` and `server.api_enabled` are deprecated** in favour of [`server.ui.enabled` and `server.api.enabled`](/docs/core/config/eventum-yml). The flat keys warn at startup and go away in 2.8. 🚀 New features [#-new-features] Eventum Studio [#eventum-studio] * **[Refreshed interface](/docs/studio)** — every screen follows one design system, with matched dark and light themes * **[Project page as a development studio](/docs/studio/projects)** — edit the configuration and every project file in one workspace, with the explorer, stage inspector and debug console docked around the editor and saved together * **[Live instance page](/docs/studio/instances#instance-page)** — watch throughput move through the pipeline with per-plugin counters, read status and uptime from the header, and stream the logs in the page * **[Monitoring dashboard](/docs/studio/monitoring)** — follow the pipeline flow, throughput and failure charts, each instance's share of the output load, and host resources on one screen * **[Management console](/docs/studio/settings#management)** — read the application and host identity, a resource snapshot and the application log without leaving the page * **[Runtime stats and filters in the tables](/docs/studio/instances#filtering-and-selection)** — sort instances by Flow, Errors and Written, and share a filtered view as a link * **[Renaming for projects, instances, scenarios and secrets](/docs/studio/instances#per-instance-actions)** — change the name of an object instead of recreating it * **[Instance cloning](/docs/studio/instances#per-instance-actions)** — create an instance from an existing one, reusing its project and parameters * **Link-based navigation** — open a record, a sidebar item or a breadcrumb in a new tab with a middle- or Ctrl/Cmd-click * **[File sizes in the project file tree](/docs/studio/projects#explorer)** — see how large a file is before opening it; anything over 10 MB stays out of the editor * **[Reworked editor search](/docs/studio/projects#editor)** — search a file from a compact panel that counts the matches and toggles case, regex and whole-word matching * **[Unsaved-changes guard](/docs/studio/projects#saving)** — confirm before navigating away, refreshing or closing a tab with unsaved work * **[Lit status indicator](/docs/studio/instances#statuses)** — tell a running, starting or stopping instance from one at rest at a glance AI agents [#ai-agents] * **[Rename tools](/docs/mcp/tools)** — have an agent rename a project, generator, scenario or secret, and list the projects reading a secret before it does * **[Scenario, global state, settings and instance-control tools](/docs/mcp/tools)** — let an agent manage scenarios, read and edit the shared global state, read and patch the settings, and stop or restart the instance; write tools stay gated behind `server.mcp.allow_write` * **Bounded file reads** — page a large file through the agent instead of handing it over in one piece API [#api] * **[Rename endpoints](/docs/api)** — rename a project, instance, scenario or secret over HTTP 🐛 Bug fixes [#-bug-fixes] Generators [#generators] * A generator configuration is read exactly as written — a state machine condition, a template parameter or a sample name spelled with a dot works as it stands * `${params.*}` and `${secrets.*}` resolve a dotted name as a path of nested names * The [`clickhouse`](/docs/plugins/output/clickhouse), [`opensearch`](/docs/plugins/output/opensearch) and [`http`](/docs/plugins/output/http) outputs connect over certificate-based TLS — `ca_cert`, `client_cert` and `client_cert_key` take the path to the certificate (Thanks to [Sai Asish Y](https://github.com/SAY-5) for the PR!) * Events lost by an output or rejected by a formatter are counted in the failure metrics, so the gap between Produced and Written is accounted for * The [`http` input](/docs/plugins/input/http) reports a failed bind with the address and the exit code of its server * The `json-batch` formatter writes only what it accepts, and leaves the destination untouched when a whole batch is rejected * Generators run in parallel alongside the [`clickhouse` output](/docs/plugins/output/clickhouse), and several of them start reliably at the same moment Eventum Studio [#eventum-studio-1] * A large project file opens and uploads however long the transfer takes * A finished or failed instance is told apart from a running one at a glance * The MCP settings are editable in the Server section and keep their values when settings are saved * The [cron](/docs/plugins/input/cron) field reads seconds in the generator's order and accepts random values and `${params.*}` * The HTTP output form is validated against its own rules * The plugin switches show what a field resolves to * A project created with the name of a deleted one starts from a clean configuration * The Management page reports a restart and a stop * A long file name in the file tree is trimmed to the panel and shown in full on hover Core and API [#core-and-api] * Ctrl+C exits at once with a live log view or a connected MCP client * The disk and network figures measure the Eventum process rather than the whole host * A file that a generator is writing to transfers in full — it arrives as it stood at the moment it was requested * A scenario page opens faster, however many generators it holds * The websocket API schema describes the running instance, and Eventum starts on a read-only filesystem ⚡ Performance [#-performance] * The editor responds faster while typing, most noticeably in a large project file 📝 Other changes [#-other-changes] * The status of an instance that has not run is now named **Idle** * The web UI and REST API toggles moved under [`server.ui.enabled` and `server.api.enabled`](/docs/core/config/eventum-yml), with the flat keys kept as deprecated aliases # 🚩 2.8.0 Released **August 29, 2026** Studio connects git repositories that publish ready-made generators and installs one of them as a project, so a new data source no longer starts from an empty configuration. Projects export and import as ZIP archives, the log is split into a file per component, and every running instance reports the processor, threads, disk, network and queue memory it occupies. Five changes alter how an existing setup behaves: * **`server.ui_enabled` and `server.api_enabled` are gone.** Deprecated in 2.7.0, they are now rejected as unknown settings — move the value under [`server.ui.enabled` and `server.api.enabled`](/docs/core/config/eventum-yml#server). * **The log files changed.** A record goes to `main.log`, `server.log`, `server_access.log`, `mcp.log` or `generator_.log` by the part of the application it came from, and `server_error.log` is gone. Anything collecting these files needs the [new set](/docs/core/config/eventum-yml#log-files). * **The events queue is bounded in memory.** It holds at most [`generation.queue.max_event_bytes`](/docs/core/config/eventum-yml#generationqueue), 256 MiB by default. The limit is set for the application, overridden per instance, or set to `null` to lift it. * **Sample mode ignores `batch.delay` when `batch.size` is set.** Batches are formed by size alone, so a per-batch formatter such as `json-batch` writes larger arrays than before — see [Batching](/docs/core/concepts/generator#batching). * **A secret name is checked when it is written.** It must be words of lowercase letters, digits and `_`, separated by `.` — see [Naming a secret](/docs/core/config/secrets#naming-a-secret). Names already in a keyring are left as they are. 🚀 New features [#-new-features] Eventum Studio [#eventum-studio] * **[Generator repositories](/docs/studio/repositories)** — connect a git repository of ready-made generators and install any entry of its catalog as a project * **[Repository discovery](/docs/studio/repositories#finding-a-repository)** — find a repository by searching GitHub for the ones carrying the `eventum-generators` topic, and connect it from the list * **[Project export and import](/docs/studio/projects#moving-a-project-between-instances)** — export a project as a ZIP archive and import it as a new project, on the same instance or another one * **[Monitoring rebuilt around the running instances](/docs/studio/monitoring#instances)** — the load chart and the instance table share one search, one set of filters and one selection * **[Instance overview](/docs/studio/instances#overview)** — an instance opens on its rates, its totals and what the run costs, which used to be a grid of figures at the foot of the page * **Release highlights** — what an upgrade brought opens by itself on the first load after it, and stays reachable from the user menu * **[Log channels on the Management page](/docs/studio/settings#management)** — stream the Main, Server, Access and MCP logs of the application separately * **[Download in the project file tree](/docs/studio/projects#explorer)** — save a project file to your machine, including generator output too large for the editor to open * **[A keyring picker beside password fields](/docs/core/config/secrets)** — pick a secret from the keyring instead of typing the `${secrets.}` reference by hand * **[Scripts in the scenario data flow diagram](/docs/studio/scenarios#data-flow)** — the diagram carries the global keys a `script` event plugin reads and writes, beside the templates that do the same * **A rebuilt About dialog** — the version, the runtime, the host and the build read as one aligned list * **Error details ordered by diagnostic value** — the dialog leads with what the server reported, and the raw request and response sit behind one toggle Core [#core] * **[A log file per component](/docs/core/config/eventum-yml#log-files)** — a generator writes a log file of its own, down to the traffic its output plugins produce, instead of sharing one with the application * **[Per-instance resource accounting](/docs/studio/monitoring#instances)** — an instance reports the threads, processor time, waiting, disk and network bytes and queue levels it occupies, and Monitoring ranks the running ones by each of them * **[`generation.queue.max_event_bytes`](/docs/core/config/eventum-yml#generationqueue)** — limit the events queue by the memory it holds, not only by the number of batches in it * **[`log.third_party_level`](/docs/core/config/eventum-yml#log)** — set the level of third-party libraries separately from `log.level`, `warning` by default * **The state of the GIL** — the instance information reports whether generators run in parallel, and [Management](/docs/studio/settings#management) warns when the GIL came back after startup * **The caller of a served request** — everything logged while serving a request carries the address of the client that caused it Generators [#generators] * **[Global state in the `script` event plugin](/docs/plugins/event/script)** — a script reads and writes the same global state a template does, through a `globals` key of its parameters * **[`concurrency` in the `http` output](/docs/plugins/output/http)** — cap how many requests the plugin performs at a time, 100 by default AI agents [#ai-agents] * **[Repository tools](/docs/mcp/tools)** — an agent lists the connected repositories, reads the catalog one publishes and installs a generator from it * **[`export_generator` and `import_generator`](/docs/mcp/tools)** — an agent moves a whole project in or out as a ZIP archive instead of copying it file by file * **[`get_instance_logs`](/docs/mcp/tools)** — an agent reads any log channel of the application, not just the log of a generator * **Resources in `get_generator_stats`** — the tool carries the same per-instance resource figures the API does API [#api] * **[Repository, export and import endpoints](/docs/api)** — connect repositories and export or import projects over HTTP 🐛 Bug fixes [#-bug-fixes] Eventum Studio [#eventum-studio-1] * A page whose code or stylesheet never arrived reloads the document once, instead of dropping Studio on the error screen * The Studio shell is no longer served from the browser cache, so it cannot outlive an upgrade and ask for files the new build does not carry * Wide tables scroll inside their own panel on a narrow window, instead of putting their rightmost columns out of reach behind the page scrollbar * The editor keeps its width while the explorer and the inspector give way, instead of absorbing every bit of narrowing * The plugin parameters form follows the plugin rather than its position in the list, so deleting one no longer writes its values over another * Switching off the events-queue memory limit, or batching by `batch.size` or `batch.delay` alone, now reaches the instance: the settings form dropped the value that asks for it, so the choice was never applied and the switch came back on * The instance settings are reread after saving, so the page stops showing the values it held before the write * An instance registered from outside the workspace names the directory it sits in, instead of linking to a project page that cannot exist * The projects table no longer re-renders without end when it is opened without an instance filter * The template editor completes `pop` among the state calls it offers * The table sort controls, the instance switches and the delimiter shortcuts carry names a screen reader can announce Core and API [#core-and-api] * A secret named `keys`, `items` or `aws.get` no longer resolves to a method of the collection holding the secrets, for `${params.*}` as well * [What uses a secret](/docs/studio/settings#secrets) counts connected repositories as well as projects, and renaming it repoints every one of them * Non-ASCII values in a configuration are stored as they were entered, not as `\uXXXX` escape sequences * Configurations, templates, samples, time patterns and log files are read and written as UTF-8 whatever the host locale is * The pipeline backpressure warnings name what actually happened — a cancelled write, or a queue that filled up — instead of blaming the generation rate * Repeated write timeouts of an output plugin are reported as one line with a running count, in place of one line per cancelled write * A path that leaves the project directory through a symlink is refused for every file operation Generators [#generators-1] * A [`tcp` output](/docs/plugins/output/tcp) whose target stopped reading fails the write at once, instead of growing the memory of the application and holding up the shutdown of the instance * A template that acquired the global state lock and never released it no longer freezes every other generator * A batch written through the [`http` output](/docs/plugins/output/http) under a per-event formatter no longer fires a request per event at once, which used to cancel the write on its timeout and count every event as failed * A [`subprocess.run`](/docs/plugins/event/template/subprocess) call runs under a timeout of 30 seconds by default and 300 at most, and fails rather than holding more than 8 MiB of output ⚡ Performance [#-performance] * A generator catching up — started on a past range, or fallen behind real time — merges the batches whose timestamps are already due up to `batch.size`, instead of publishing each of the small batches `batch.delay` cut them into * Starting many instances at once no longer compiles all of their templates at the same moment, so a large scenario comes up without the spike on the host 📝 Other changes [#-other-changes] * The body of `POST /preview/{name}/event-plugin/produce` is described by a request model of its own and rejects anything else * The preview global state endpoints moved from `/preview/{name}/event-plugin/template/state/global` to `/preview/{name}/event-plugin/state/global`, since the state they read belongs to every event plugin * The disk figures of the application count the bytes handed to the system calls, the same measure an instance is reported by, so the two can be compared; the application used to count what reached the disk instead # Connect your agent There are two ways to connect your agent to Eventum: * **stdio** — your MCP client runs Eventum locally. Best for building, previewing, and running generators on your own machine. * **HTTP** — your client connects to a running [Eventum server](/docs/core/cli/eventum-run), and can also manage the generators running there. Start with stdio. Move to HTTP once you need to manage a live server. Prerequisites [#prerequisites] * **Eventum installed.** See [Installation](/docs/core/introduction/installation). * **A generators directory** — the folder Eventum reads your generators from. Use an existing one, or create an empty folder for the agent to populate: ```bash mkdir -p /path/to/generators ``` The commands below run Eventum with `uv run`. If Eventum is already on your `PATH`, omit the `uv run` prefix. Local authoring over stdio [#local-authoring-over-stdio] Over stdio your client launches Eventum itself. There is no server to run, and every MCP client supports it. Add Eventum to your client [#add-eventum-to-your-client] Add an `eventum` entry to your client's MCP configuration, using an **absolute** path to your generators directory: ```json title="mcp.json" { "mcpServers": { "eventum": { "command": "uv", "args": ["run", "eventum", "mcp", "--generators-dir", "/path/to/generators"] } } } ``` Most clients — including Claude Desktop and Windsurf — accept this same `mcpServers` format. The file's location varies between clients, so check your client's documentation for the exact path. ```bash claude mcp add eventum -- uv run eventum mcp --generators-dir /path/to/generators ``` By default the server is registered only for the project directory you run the command from. Add `-s user` to make it available in all your projects. ```bash codex mcp add eventum -- uv run eventum mcp --generators-dir /path/to/generators ``` ```json title="~/.cursor/mcp.json" { "mcpServers": { "eventum": { "command": "uv", "args": ["run", "eventum", "mcp", "--generators-dir", "/path/to/generators"] } } } ``` Use `.cursor/mcp.json` inside a repository to register the server for that project only. ```json title="~/.gemini/settings.json" { "mcpServers": { "eventum": { "command": "uv", "args": ["run", "eventum", "mcp", "--generators-dir", "/path/to/generators"] } } } ``` ```json title=".vscode/mcp.json" { "servers": { "eventum": { "type": "stdio", "command": "uv", "args": ["run", "eventum", "mcp", "--generators-dir", "/path/to/generators"] } } } ``` Reload your client [#reload-your-client] Restart your client, or reload its MCP configuration. Eventum should appear among its connected MCP servers, with its tools available to the agent. Start building [#start-building] Describe the data you want — for example: > build a generator that emits one JSON web access-log event per second, then preview ten events The [usage scenarios](/docs/mcp/scenarios) cover more of what you can ask for. By default the agent can read and write files in your generators directory. Add `--read-only` to let it explore and preview without changing anything. See the [`eventum mcp`](/docs/core/cli/eventum-mcp) reference for all options. Live management over HTTP [#live-management-over-http] Over HTTP your client connects to a running Eventum server. The agent can do everything it does over stdio, and also manage the generators running on that server. Access is protected by the server's credentials. Enable the MCP server in `eventum.yml` [#enable-the-mcp-server-in-eventumyml] ```yaml title="eventum.yml" server.mcp.enabled: true server.mcp.allow_write: true server.mcp.path: "/mcp" ``` Managing live generators requires `allow_write: true`. Leave it `false` — the default — to limit the agent to exploring and previewing. See the [configuration reference](/docs/core/config/eventum-yml#server-mcp) for every option. Start the server [#start-the-server] ```bash eventum run -c eventum.yml ``` With the defaults, the MCP endpoint is available at `http://localhost:9474/mcp/`. Point your client at the server [#point-your-client-at-the-server] Add the endpoint and the server's credentials to your client. The `Authorization` value is `Basic ` followed by `username:password` in base64; the examples below encode the default `eventum:eventum`. ```json title="mcp.json" { "mcpServers": { "eventum": { "url": "http://localhost:9474/mcp/", "headers": { "Authorization": "Basic ZXZlbnR1bTpldmVudHVt" } } } } ``` Most clients accept this same `mcpServers` format. The file's location varies between clients, so check your client's documentation for the exact path. ```bash claude mcp add --transport http eventum http://localhost:9474/mcp/ --header "Authorization: Basic ZXZlbnR1bTpldmVudHVt" ``` By default the server is registered only for the project directory you run the command from. Add `-s user` to make it available in all your projects. ```toml title="~/.codex/config.toml" [mcp_servers.eventum] url = "http://localhost:9474/mcp/" http_headers = { "Authorization" = "Basic ZXZlbnR1bTpldmVudHVt" } ``` ```json title="~/.cursor/mcp.json" { "mcpServers": { "eventum": { "url": "http://localhost:9474/mcp/", "headers": { "Authorization": "Basic ZXZlbnR1bTpldmVudHVt" } } } } ``` ```bash gemini mcp add --transport http --header "Authorization: Basic ZXZlbnR1bTpldmVudHVt" eventum http://localhost:9474/mcp/ ``` ```json title=".vscode/mcp.json" { "servers": { "eventum": { "type": "http", "url": "http://localhost:9474/mcp/", "headers": { "Authorization": "Basic ZXZlbnR1bTpldmVudHVt" } } } } ``` Reload the client to enable the live-management tools alongside the authoring ones. Change the default `eventum:eventum` credentials before exposing the server on a network, and leave `allow_write` off unless the agent needs to write files or control generators. Next [#next] # How it works The agent turns your request into a working generator over a few passes. An agent working alone can easily produce a configuration that reads correctly but does not run, or runs and produces the wrong data. What makes the result trustworthy is that the agent never relies on its own prediction of how Eventum behaves: at every step it loads its work into Eventum and inspects the actual output. The loop [#the-loop] Those passes form a loop, each one grounded in what Eventum reports back. It begins by asking Eventum what's available — every plugin and the exact settings each one takes — so it builds on current facts rather than training data that may be out of date. Then it writes the generator's files and loads them into Eventum. Anything wrong comes back as Eventum's own error, exact rather than approximate, and the agent fixes it and loads it again. Once the generator loads cleanly, Eventum generates a small sample of events, and the agent shows you those events and their timing — before anything is saved or started. Because Eventum generated that sample itself, it is exactly what the generator will produce when you run it — nothing is simulated. The agent repeats the cycle until the generator loads cleanly and the sample looks right. The tools behind each step are listed in [Tools & resources](/docs/mcp/tools). Two ways to run it [#two-ways-to-run-it] The loop is the same over either transport. What differs is whether the agent can manage long-lived generators on a server, and what you let it change. | | Local (stdio) | Live (HTTP) | | ------------------------- | ------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------- | | How it runs | A local process beside your agent | Mounted into a running [Eventum server](/docs/core/cli/eventum-run) | | Best for | Building, previewing, and running on your machine | Operating the generators on a shared or remote instance | | Build, preview, and run | Yes | Yes | | Manage running generators | — | Yes — register, start, stop, and monitor them | | Access | Your local files | Gated by the server's [authentication](/docs/mcp/connect#live-management-over-http); read-only unless you enable writes | Managing the generators on a running server — registering, starting, and stopping them — requires that server, so it is available only over HTTP. See [Connect your agent](/docs/mcp/connect) to set up either transport. Safety [#safety] Letting an agent write files and start processes deserves care, so Eventum draws the boundaries tightly by default: * **No model, no credentials.** Eventum runs no language model and stores no keys of its own — your agent brings the model, so there are no API keys here and no per-token cost. * **Writes stay under your control.** Over HTTP the agent can discover, validate, and preview freely, but it cannot change a file or affect a running generator until you enable writes. Over stdio it reads and writes your generators directory directly; pass `--read-only` to prevent all changes. * **Your filesystem and secrets stay hidden.** Eventum strips file paths and secret values out of every error and log before the agent sees them, so a failing run reveals its cause without exposing your filesystem or your credentials. The agent can see which secret names a generator refers to, but never their values, and it can neither add nor change them. Writes over HTTP include code execution, not just file access — generators run code by design: a `template` plugin executes Python, a `script` plugin runs a file you provide. Anyone with the server's credentials could run code on your host, so turn writes on only for a trusted network and a trusted agent. What it does not do [#what-it-does-not-do] It is the entry point for an agent, not a replacement for [Eventum Studio](/docs/studio) and its visual editor. Nor does it generate data itself: it builds and runs generators, and the data is produced by the normal [pipeline](/docs/core/concepts/generator) — the same output you would get had you written the generator by hand. # Overview
Eventum MCP server
Describe the data you need in plain language, and your agent builds the Eventum generator that produces it.
AI agents like Claude Code, Cursor, and Codex can do more than answer questions — connected to the right tools, they take action on your behalf. The [**Model Context Protocol (MCP)**](https://modelcontextprotocol.io) is the open standard for those connections: an MCP server offers a set of capabilities, and any MCP-compatible agent can connect to it and use them. The Eventum **MCP server** is such a server: it gives your agent everything it needs to work with Eventum. Instead of learning Eventum's configuration and writing generators by hand, you describe the data you want and let the agent build it. For example, you might ask for: > a week of Apache access logs with a traffic spike at noon, mostly 200s but a burst of 500s during the spike The result is a working generator that produces exactly that. See [how it works](/docs/mcp/how-it-works) for the loop that builds and checks it. No model required [#no-model-required] Eventum has **no language model** of its own, no API keys, no per-token cost — your agent already brings the model. The MCP server adds the tools to discover, build, check, run, and operate generators: * **Discover** — what Eventum can do and how to configure it, with worked examples to start from. * **Install** — take a ready-made generator from a [connected repository](/docs/studio/repositories) into the workspace. * **Build** — create, edit, and remove generators. * **Check** — confirm a generator works, by validating it and previewing its output. * **Run** — run a generator to its configured outputs. * **Operate** — start, stop, and manage generators on a live server. Two ways to connect [#two-ways-to-connect] Both give your agent the same tools to build, check, and run generators. HTTP adds management of the generators running on a server. Read next [#read-next] # Usage scenarios Each scenario below is a task for an agent you have [connected](/docs/mcp/connect), ordered from the most common to the more specialized. The prompts are illustrative; phrase your own requests freely. Build a generator from a description [#build-a-generator-from-a-description] The primary use case: you describe the data you want, and the agent assembles a generator that produces it. > Build a generator that emits one JSON web access-log event per second: a random client IP, an HTTP method, a request path drawn from a small sample, a status code, and a response size. Then preview ten events. The agent discovers the right plugins, drafts the files, and [checks each step against Eventum](/docs/mcp/how-it-works) until the generator loads cleanly and the preview matches your request. The result is a saved, validated generator you can run with [`eventum run`](/docs/core/cli/eventum-run) or [`eventum generate`](/docs/core/cli/eventum-generate). The `create_generator` prompt guides the agent through this. Ask to preview at any point before saving — the schedule first, then the rendered events. Nothing is written or started until you approve it. Manage live generators over HTTP [#manage-live-generators-over-http] With the [HTTP transport](/docs/mcp/connect#live-management-over-http) enabled and writes allowed, the agent can operate the generators on a running server, not just author them. > What's running right now? Start the `web-access-log` generator and tell me when it's producing events. The agent can list the running generators and start or stop them. It can also register one it just authored, then start it — and the server restores it after a restart. Every operation is protected by your server's authentication. The `live_ops` prompt walks through these operations. Diagnose a failing generator [#diagnose-a-failing-generator] When a live run fails, the agent can find the cause without you opening a log file. > The `payments-stream` generator stopped — what went wrong? The agent checks the generator's status, sees that the run ended in failure, and reads its recent logs to find the cause — a bad configuration value, an unreachable output. It then proposes a fix, rewrites the file, and restarts the generator. Coordinate a fleet with scenarios and shared state [#coordinate-a-fleet-with-scenarios-and-shared-state] Beyond single generators, the agent can organize the whole fleet on a running server: group related generators into named [scenarios](/docs/core/config/startup-yml) and inspect the [`globals`](/docs/plugins/event/template/state) state they share at runtime. > Group `web-access-log` and `payments-stream` into a `storefront` scenario, then show me the shared global state. The agent tags both generators into the scenario, lists what the scenario contains, and reads or edits the shared global state generators use to coordinate. Over the same HTTP connection it can also read the running settings and host metrics, patch the settings file, and restart the instance to apply a change — each write behind your server's authentication and `allow_write` gate. Where to go next [#where-to-go-next] # Tools & resources This page lists everything the agent can use: the **tools** it calls to take action, the **resources** it reads to ground itself, and the **prompts** that guide it through common tasks. In the tool tables, the **Access** column marks each tool **Read** or **Write**. Write tools change files or control generators, and are gated: over HTTP they require `server.mcp.allow_write`, and over stdio they are available unless you pass `--read-only`. Discovery tools [#discovery-tools] Read-only, and available on both transports. They let the agent learn what Eventum offers before it writes a config. | Tool | Description | | ------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `list_plugins` | List the available input, event, and output plugins. | | `get_plugin_schema` | Return the configuration schema for one plugin. | | `list_formatters` | List the available output [formatters](/docs/plugins/formatters). | | `get_formatter_schema` | Return the configuration schema for one formatter. | | `describe_sample` | Describe a CSV or JSON [sample](/docs/plugins/event/template/samples) file used by a generator. | | `list_secret_names` | List the secret names in the keyring, so a config can reference them. The agent cannot read or change their values — that is done with [`eventum-keyring`](/docs/core/cli/eventum-keyring). | | `list_secret_references` | List what reads a given secret — the projects whose configuration reads it, and the connected repositories authenticating with it — to see what a rename or a removal would break. | Workspace & authoring tools [#workspace--authoring-tools] Available on both transports. They operate on the generators in your generators directory. | Tool | Access | Description | | ----------------------- | ------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `list_generators` | Read | List the saved generators. | | `list_generator_files` | Read | List the files in one generator. | | `read_generator_file` | Read | Read a file in a generator. A file larger than the read limit comes back in windows the agent pages through, so a generator output file cannot flood its context. | | `write_generator_file` | Write | Create or overwrite a file in a generator. | | `delete_generator_file` | Write | Delete a file from a generator. | | `delete_generator` | Write | Delete a whole generator and its files. | | `export_generator` | Read | Pack a whole generator into a ZIP archive. Named top-level entries can be left out, which is how generated output stays behind. | | `import_generator` | Write | Create a generator from a ZIP archive. An archive that wraps the generator in directories imports the same way as one holding it at the top level. | | `validate_generator` | Read | Validate a generator and return any errors. | | `preview_timestamps` | Read | Preview the timestamps a generator would produce. | | `preview_events` | Read | Preview the events a generator would produce. | | `run_generator` | Write | Run a saved generator once to its configured outputs, stopping when it finishes or reaches a time or event limit. | `validate_generator` and the `preview_*` tools load the generator into the same pipeline a real run uses, so the agent sees exactly what a run would produce. See [how it works](/docs/mcp/how-it-works#the-loop). An archive travels inside the tool call, so it lands in the agent's context. An agent that can make HTTP requests and write files is better off taking the archive from the [REST API](/docs/api) instead, which moves it as a file and carries a project of any size. `export_generator` and `import_generator` say so themselves, and refuse an archive over 128 KiB, where the REST API is the only route left. Repository tools [#repository-tools] Available on both transports. They read the [connected repositories](/docs/studio/repositories) of the instance — git repositories that publish ready-made generators — and install what one of them publishes into your generators directory. | Tool | Access | Description | | ------------------------ | ------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `list_repositories` | Read | List the connected repositories, each with whether it answered the last time it was asked. The password of a private repository comes back as the secret it references, or as `***` when the value is kept in the repositories file itself. | | `discover_repositories` | Read | List the repositories that publish generators in the open, so the agent can name one for the user to connect. The content of a listed repository is not reviewed. | | `get_repository_catalog` | Read | List the generators one repository publishes — what each of them produces, what it consists of, and the projects it is already installed as. Reads the repository anew on request. | | `install_generator` | Write | Install a published generator as a project of the generators directory. An existing project is never overwritten. | Connecting and disconnecting repositories is deliberately not exposed: it names credentials and decides what the instance trusts. Do it on the [Repositories](/docs/studio/repositories) page of Studio, or in the repositories file itself. Over stdio the agent reads that same file — `repositories.yml` next to the generators directory, or the one [`--repositories`](/docs/core/cli/eventum-mcp) names. Live-management tools [#live-management-tools] Available only over HTTP. They control the generators the server manages, and rename the objects an instance holds. | Tool | Access | Description | | ------------------------- | ------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `list_generators_live` | Read | List the generators the server manages, with their status. | | `get_generator_status` | Read | Return the current status of one managed generator. | | `get_generator_stats` | Read | Return runtime statistics for one running generator, including the resources it occupies. | | `start_generator` | Write | Start a managed generator. | | `stop_generator` | Write | Stop a managed generator. | | `register_generator` | Write | Register an authored generator with the running server, so it can be started and is restored after a server restart. | | `unregister_generator` | Write | Remove a generator from the running server, and stop restoring it on restart. | | `rename_generator` | Write | Rename a stopped generator, keeping its project, parameters and scenario membership. | | `get_generator_logs` | Read | Return the recent log lines for a managed generator, to diagnose a run. | | `list_startup_generators` | Read | List the generators configured to start with the server. | | `rename_generator_config` | Write | Rename a project, moving its directory and repointing the generators that use it. All of them must be stopped first. | | `rename_secret` | Write | Rename a secret, keeping its value under the new name. Everything referring to it follows — the `${secrets.*}` token is rewritten in each project configuration, and connected repositories authenticating with it are repointed. Refused when a repository already authenticates with the new name. | Scenario tools [#scenario-tools] Available only over HTTP. A **scenario** is a named group of generators — a tag on their [startup](/docs/core/config/startup-yml) entries — used to operate related generators together. | Tool | Access | Description | | -------------------------------- | ------ | -------------------------------------------------------------- | | `list_scenarios` | Read | List the defined scenarios. | | `get_scenario` | Read | Return a scenario and the ids of the generators in it. | | `add_generator_to_scenario` | Write | Add a generator to a scenario. | | `remove_generator_from_scenario` | Write | Remove a generator from a scenario. | | `rename_scenario` | Write | Rename a scenario, rewriting the tag on every generator in it. | | `delete_scenario` | Write | Delete a scenario, untagging every generator in it. | Global-state tools [#global-state-tools] Available only over HTTP. Generators coordinate at runtime through a shared **global state**; these tools read and edit it. The values are runtime data written by templates and are returned as-is. | Tool | Access | Description | | ------------------------- | ------ | ----------------------------------------- | | `get_global_state` | Read | Return the whole shared global state. | | `get_global_state_key` | Read | Return one value from the global state. | | `set_global_state` | Write | Set one or more keys in the global state. | | `delete_global_state_key` | Write | Remove one key from the global state. | | `clear_global_state` | Write | Remove every key from the global state. | Instance tools [#instance-tools] Available only over HTTP. They read and control the running instance itself. | Tool | Access | Description | | ------------------- | ------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `get_instance_logs` | Read | Return the recent log lines of one [log channel](/docs/core/config/eventum-yml#log-files) of the instance — its core, the server, the requests it served, or this MCP server. | | `update_settings` | Write | Patch the instance [settings](/docs/core/config/eventum-yml) file. Auth credentials cannot be changed, and the change applies on the next restart. | | `stop_instance` | Write | Stop the instance. | | `restart_instance` | Write | Restart the instance, applying any pending settings change. | The MCP server runs inside the instance, so `stop_instance` and `restart_instance` end the agent's own connection — the call returns as the server goes down. Resources [#resources] Documents the agent reads to ground itself. Read-only, on both transports. | Resource | Description | | -------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `eventum://templating/reference` | The in-template API the [`template`](/docs/plugins/event/template) plugin exposes, always current with the installed version. | | `eventum://schema/generator` | The JSON Schema of the top-level [generator.yml](/docs/core/config/generator-yml) document. | | `eventum://examples/generators` | Bundled, validated worked examples to start from, plus links to the public [content-packs](https://github.com/eventum-generator/content-packs) repository and the [generator hub](https://eventum.run/hub). | | `eventum://workspace/configs` | The generators currently saved in your generators directory. | | `eventum://instance/info` | Version, runtime, and host metrics of the running instance. **HTTP only.** | | `eventum://instance/settings` | The running instance settings, with auth credentials redacted and absolute paths reduced to file names. **HTTP only.** | Prompts [#prompts] Ready-made task guides the agent can invoke. | Prompt | Description | | ------------------ | ----------------------------------------------------------------------------------- | | `create_generator` | Guide the agent through the full [authoring loop](/docs/mcp/how-it-works#the-loop). | | `live_ops` | Operate the generators on a running server. **HTTP only.** | # Formatters Every [output plugin](/docs/plugins#output-plugins) has a **formatter** that transforms event strings before writing them to the destination. Formatters are configured with the `formatter` field inside the output plugin config. ```yaml output: - file: path: output/events.log formatter: format: json indent: 2 ``` If you omit the `formatter` field, the plugin uses its default format. Default formatters [#default-formatters] | Plugin | Default format | Reason | | --------------------------------------------- | -------------- | ---------------------------------------------------- | | [stdout](/docs/plugins/output/stdout) | `plain` | Human-readable console output | | [file](/docs/plugins/output/file) | `plain` | Raw log lines, one per row | | [http](/docs/plugins/output/http) | `json-batch` | Send an entire batch in one HTTP request | | [opensearch](/docs/plugins/output/opensearch) | `json` | One JSON document per line for the bulk API | | [clickhouse](/docs/plugins/output/clickhouse) | `json` | One JSON row per line for `JSONEachRow` input format | You only need to set `formatter` when you want something different from the default. Available formats [#available-formats] plain [#plain] Passes event strings through without any transformation. Each event is delivered exactly as produced by the event plugin. | Parameter | Type | Default | Description | | --------- | ------ | ------- | ------------------ | | `format` | string | — | Must be `"plain"`. | ```yaml formatter: format: plain ``` Suppose the event plugin produces these three events: ```text title="Events from event plugin" 2026-02-20T10:00:00 GET /api/users 200 12ms 2026-02-20T10:00:01 POST /api/orders 201 45ms 2026-02-20T10:00:02 GET /api/products 200 8ms ``` The formatter outputs them unchanged — three strings, one per event: ```text title="Output (3 strings)" 2026-02-20T10:00:00 GET /api/users 200 12ms 2026-02-20T10:00:01 POST /api/orders 201 45ms 2026-02-20T10:00:02 GET /api/products 200 8ms ``` json [#json] Validates each event as JSON and optionally pretty-prints it. If an event is not valid JSON, it is skipped and counted as a format error — other events in the batch are not affected. | Parameter | Type | Default | Constraints | Description | | --------- | ------- | ------- | ----------------- | --------------------------------------------------------- | | `format` | string | — | Must be `"json"`. | Format discriminator. | | `indent` | integer | `0` | >= 0 | Indentation level. `0` produces compact single-line JSON. | Compact JSON (indent: 0) [#compact-json-indent-0] ```yaml formatter: format: json ``` Events are validated and compacted to a single line: ```json title="Events from event plugin" {"user": "alice", "action": "login", "ip": "10.0.0.1"} {"user": "bob", "action": "logout", "ip": "10.0.0.2"} ``` ```json title="Output (2 strings, collapsed to one line each)" {"user": "alice", "action": "login", "ip": "10.0.0.1"} {"user": "bob", "action": "logout", "ip": "10.0.0.2"} ``` Pretty-printed JSON (indent: 2) [#pretty-printed-json-indent-2] ```yaml formatter: format: json indent: 2 ``` ```json title="Events from event plugin" {"user":"alice","action":"login","ip":"10.0.0.1"} ``` ```json title="Output (1 string, pretty-printed)" { "user": "alice", "action": "login", "ip": "10.0.0.1" } ``` Invalid JSON handling [#invalid-json-handling] Events that are not valid JSON are skipped. Each rejection is written to the generator log and counted in the `format_failed` metric; the remaining events are still formatted normally: ```text title="Events from event plugin (3 events, one invalid)" {"user": "alice"} not valid json {"user": "bob"} ``` ```json title="Output (2 strings — invalid event skipped)" {"user":"alice"} {"user":"bob"} ``` json-batch [#json-batch] Collects all events in a batch into a single JSON array. Useful for HTTP endpoints that accept a batch payload. Each event must be valid JSON — invalid events are excluded from the array and counted in the `format_failed` metric. When every event of a batch is invalid, the batch produces no output at all. | Parameter | Type | Default | Constraints | Description | | --------- | ------- | ------- | ----------------------- | --------------------------------------------------------- | | `format` | string | — | Must be `"json-batch"`. | Format discriminator. | | `indent` | integer | `0` | >= 0 | Indentation level. `0` produces compact single-line JSON. | Compact batch [#compact-batch] ```yaml formatter: format: json-batch ``` Three events are collected into a single JSON array: ```json title="Events from event plugin (3 separate events)" {"user": "alice", "action": "login"} {"user": "bob", "action": "view"} {"user": "charlie", "action": "purchase"} ``` ```json title="Output (1 string — a JSON array)" [{"user":"alice","action":"login"},{"user":"bob","action":"view"},{"user":"charlie","action":"purchase"}] ``` Pretty-printed batch [#pretty-printed-batch] ```yaml formatter: format: json-batch indent: 2 ``` ```json title="Output (1 string — a pretty-printed JSON array)" [ { "user": "alice", "action": "login" }, { "user": "bob", "action": "view" }, { "user": "charlie", "action": "purchase" } ] ``` This is the default formatter for the [http](/docs/plugins/output/http) output plugin — it lets you send an entire batch as a single HTTP request body. template [#template] Renders a Jinja2 template for **each event** individually. Use this when you need to reshape events — adding fields, wrapping in a custom envelope, or converting formats. | Parameter | Type | Default | Constraints | Description | | --------------- | -------------- | ------- | --------------------- | ------------------------------- | | `format` | string | — | Must be `"template"`. | Format discriminator. | | `template` | string or null | `null` | Non-empty if set. | Inline Jinja2 template string. | | `template_path` | path or null | `null` | — | Path to a Jinja2 template file. | Exactly one of `template` or `template_path` must be provided. Inside the template, the variable `event` holds the raw event string. Wrapping events in an envelope [#wrapping-events-in-an-envelope] Add metadata to each event before sending it to the output: ```yaml output: - http: url: https://ingest.example.com/events formatter: format: template template: '{"source": "eventum", "environment": "staging", "payload": {{ event }}}' ``` ```json title="Events from event plugin" {"user": "alice", "action": "login"} {"user": "bob", "action": "logout"} ``` ```json title="Output (2 strings, each wrapped)" {"source": "eventum", "environment": "staging", "payload": {"user": "alice", "action": "login"}} {"source": "eventum", "environment": "staging", "payload": {"user": "bob", "action": "logout"}} ``` Converting JSON to CSV [#converting-json-to-csv] Transform JSON events into CSV rows: ```yaml formatter: format: template template: '{{ (event | fromjson).timestamp }},{{ (event | fromjson).user }},{{ (event | fromjson).action }}' ``` ```json title="Events from event plugin" {"timestamp": "2026-02-20T10:00:00", "user": "alice", "action": "login"} {"timestamp": "2026-02-20T10:00:05", "user": "bob", "action": "purchase"} ``` ```text title="Output (2 CSV rows)" 2026-02-20T10:00:00,alice,login 2026-02-20T10:00:05,bob,purchase ``` Using an external template file [#using-an-external-template-file] For complex formatting, use a separate template file: ```yaml formatter: format: template template_path: formatters/syslog.jinja ``` ```jinja title="formatters/syslog.jinja" <14>1 {{ (event | fromjson).timestamp }} eventum app - - - {{ event }} ``` ```json title="Events from event plugin" {"timestamp": "2026-02-20T10:00:00", "level": "INFO", "msg": "Request processed"} ``` ```text title="Output" <14>1 2026-02-20T10:00:00 eventum app - - - {"timestamp": "2026-02-20T10:00:00", "level": "INFO", "msg": "Request processed"} ``` template-batch [#template-batch] Renders a single Jinja2 template with **all events in a batch**. The variable `events` holds the list of event strings. This produces one output string per batch, giving you full control over how events are aggregated. | Parameter | Type | Default | Constraints | Description | | --------------- | -------------- | ------- | --------------------------- | ------------------------------- | | `format` | string | — | Must be `"template-batch"`. | Format discriminator. | | `template` | string or null | `null` | Non-empty if set. | Inline Jinja2 template string. | | `template_path` | path or null | `null` | — | Path to a Jinja2 template file. | Exactly one of `template` or `template_path` must be provided. Newline-delimited batch [#newline-delimited-batch] Join events with newlines for line-based protocols: ```yaml formatter: format: template-batch template: '{{ events | join("\n") }}' ``` ```json title="Events from event plugin (3 separate events)" {"id": 1, "msg": "start"} {"id": 2, "msg": "process"} {"id": 3, "msg": "done"} ``` ```text title="Output (1 string — all events joined)" {"id": 1, "msg": "start"} {"id": 2, "msg": "process"} {"id": 3, "msg": "done"} ``` Custom XML envelope [#custom-xml-envelope] Wrap a batch of events in an XML document for SOAP or legacy endpoints: ```yaml formatter: format: template-batch template_path: formatters/xml-batch.jinja ``` ```jinja title="formatters/xml-batch.jinja" {%- for event in events %} {{ event }} {%- endfor %} ``` ```text title="Events from event plugin" user=alice action=login ip=10.0.0.1 user=bob action=logout ip=10.0.0.2 ``` ```xml title="Output (1 string — XML document)" user=alice action=login ip=10.0.0.1 user=bob action=logout ip=10.0.0.2 ``` Summary report [#summary-report] Aggregate a batch into a summary instead of forwarding individual events: ```yaml formatter: format: template-batch template_path: formatters/summary.jinja ``` ```jinja title="formatters/summary.jinja" {%- set parsed = [] -%} {%- for e in events -%} {%- do parsed.append(e | fromjson) -%} {%- endfor -%} {%- set errors = parsed | selectattr("status", "ge", 400) | list -%} {"total": {{ events | length }}, "errors": {{ errors | length }}, "error_rate": {{ "%.2f" | format(errors | length / events | length) }}} ``` ```json title="Events from event plugin (5 events)" {"path": "/api/users", "status": 200} {"path": "/api/orders", "status": 201} {"path": "/api/users", "status": 500} {"path": "/api/products", "status": 404} {"path": "/api/health", "status": 200} ``` ```json title="Output (1 string — a summary)" {"total": 5, "errors": 2, "error_rate": 0.40} ``` Per-event vs per-batch [#per-event-vs-per-batch] | Format | Granularity | Result per write | | ---------------- | ----------- | ------------------------- | | `plain` | Per event | N strings (one per event) | | `json` | Per event | N JSON strings | | `template` | Per event | N rendered strings | | `json-batch` | Per batch | 1 JSON array | | `template-batch` | Per batch | 1 rendered string | Per-event formats produce one output string for each input event. Per-batch formats aggregate all events into a single output string, which reduces the number of I/O calls at the cost of sending larger payloads. Combining a batch into one string also costs memory while it happens — several times the size of the batch itself, since the events and the document assembled from them are held at the same time. Lower `batch.size` when that peak matters more than the number of requests; see [how much memory a generator holds](/docs/core/concepts/generator#how-much-memory-a-generator-holds). **When to use per-batch formats:** * The destination expects a single payload (e.g., HTTP API accepting a JSON array) * You want to reduce I/O overhead by writing once per batch * You need to aggregate or summarize events before delivery **When to use per-event formats:** * The destination processes events one at a time (e.g., line-based log files) * You want each event independently validated (invalid events are skipped, not the whole batch) * Downstream systems need individual records (e.g., OpenSearch bulk API, ClickHouse JSONEachRow) # Overview
Plugin reference
Every parameter, default, and constraint for each input, event, and output plugin — your go-to reference when writing configs.
*** Plugins are the building blocks of an Eventum [generator](/docs/core/concepts/generator). Every generator is a three-stage pipeline where each stage is handled by one or more plugins. For the conceptual overview of how plugins fit together, see [Concepts — Plugins](/docs/core/concepts/plugins). Input plugins [#input-plugins] Input plugins define **when** events happen. A generator can have multiple input plugins — their timestamp streams are merged chronologically. Every input plugin supports an optional `tags` parameter that attaches string labels to timestamps, making them available in event templates via the `tags` variable. | Plugin | Description | | --------------------------------------------------- | ------------------------------------------------- | | [cron](/docs/plugins/input/cron) | Cron expressions with second-level precision | | [timer](/docs/plugins/input/timer) | Fixed-interval ticks | | [linspace](/docs/plugins/input/linspace) | Evenly spaced timestamps across a date range | | [static](/docs/plugins/input/static) | All at once, using the current time | | [timestamps](/docs/plugins/input/timestamps) | An explicit list of datetimes | | [time\_patterns](/docs/plugins/input/time-patterns) | Statistical distributions that mimic real traffic | | [http](/docs/plugins/input/http) | On demand, triggered by HTTP requests | Event plugins [#event-plugins] Event plugins define **what** events look like. A generator has exactly one event plugin. | Plugin | Description | | ---------------------------------------- | ---------------------------------------------------------------------- | | [template](/docs/plugins/event/template) | Renders Jinja2 templates with Faker, Mimesis, random helpers, and more | | [script](/docs/plugins/event/script) | Runs a Python function for full programmatic control | | [replay](/docs/plugins/event/replay) | Reads events from an existing log file | Output plugins [#output-plugins] Output plugins define **where** events go. A generator can have multiple output plugins — every event is delivered to all of them (fan-out). Every output plugin supports a [formatter](/docs/plugins/formatters) that controls how events are serialized before delivery. | Plugin | Default formatter | Description | | --------------------------------------------- | ----------------- | ---------------------------------------------- | | [stdout](/docs/plugins/output/stdout) | `plain` | Prints to the console or stderr | | [file](/docs/plugins/output/file) | `plain` | Writes to a local file | | [http](/docs/plugins/output/http) | `json-batch` | Sends to any HTTP endpoint | | [opensearch](/docs/plugins/output/opensearch) | `json` | Indexes into an OpenSearch cluster | | [clickhouse](/docs/plugins/output/clickhouse) | `json` | Inserts into a ClickHouse database | | [kafka](/docs/plugins/output/kafka) | `json` | Produces to Apache Kafka topics | | [tcp](/docs/plugins/output/tcp) | `plain` | Sends events over a raw TCP socket | | [udp](/docs/plugins/output/udp) | `plain` | Sends events as UDP datagrams to a remote host | Configuration syntax [#configuration-syntax] Each plugin is configured as a named key inside the corresponding section of [generator.yml](/docs/core/config/generator-yml). The key is the plugin name; the value holds plugin-specific parameters: ```yaml title="generator.yml" input: - cron: # plugin name expression: "* * * * * *" # plugin parameter count: 1 event: template: mode: all templates: - my_event: template: templates/event.jinja output: - stdout: {} # empty config is valid - file: path: output/events.jsonl formatter: format: json ``` Rules: * `input` and `output` are **lists** — each item is a single-key mapping naming one plugin. * `event` is a **single mapping** — one plugin name with its config. * Unknown fields cause a validation error before any events are generated. * Relative paths are resolved from the directory containing the generator config file. # Eventum Studio
Eventum Studio
Edit configs, preview events in real time, and monitor running generators — all from your browser, no terminal required.
*** Eventum Studio is the web-based management UI that ships with Eventum. It gives you a visual interface for everything you can do through config files and the CLI — browsing projects, configuring plugins, debugging event output, monitoring running generators, and managing application settings. Studio is served automatically when you run Eventum in application mode: ```bash eventum run -c eventum.yml ``` Then open your browser at [http://localhost:9474](http://localhost:9474) (or whichever host/port you configured). What you can do in Studio [#what-you-can-do-in-studio] | Area | What it covers | | ----------------------------------------- | ------------------------------------------------------------------------------------- | | [Home](/docs/studio/overview) | Where you left off, and whether anything is failing | | [Monitoring](/docs/studio/monitoring) | The whole application as one dashboard — flow, throughput, failures, load | | [Projects](/docs/studio/projects) | Building a generator: its files, its plugins, and a console for previewing each stage | | [Instances](/docs/studio/instances) | Running generators — lifecycle, live metrics, settings and logs | | [Scenarios](/docs/studio/scenarios) | Groups of generators that work together and share state | | [Repositories](/docs/studio/repositories) | Repositories of ready-made generators, installed into the workspace as projects | | [Settings](/docs/studio/settings) | The application itself — configuration, secrets, lifecycle | The sidebar follows the same split: the **Home** and **Monitoring** entries on their own, then a **Generators** group for day-to-day work and a **Management** group for the application. Enabling and disabling Studio [#enabling-and-disabling-studio] Studio is controlled by two settings in [eventum.yml](/docs/core/config/eventum-yml): ```yaml title="eventum.yml" server: ui: enabled: true # enables the web UI api: enabled: true # enables the REST API (required for Studio to function) ``` Setting `ui.enabled: false` disables the web interface while keeping the REST API available. Setting `api.enabled: false` disables both — Studio needs the API to communicate with the backend. Authentication [#authentication] Studio uses HTTP basic authentication. The default credentials are: | Field | Default | | -------- | --------- | | Username | `eventum` | | Password | `eventum` | Change these in `eventum.yml` before exposing Eventum on a network: ```yaml title="eventum.yml" server: auth: user: admin password: s3cret ``` Theme [#theme] Studio supports light and dark themes. Toggle between them using the sun/moon icon in the top-right corner of the top bar. The theme preference is saved in the browser. What's next [#whats-next] # Instances An **instance** is a registered copy of a [project](/docs/studio/projects) with its own identifier, its own runtime parameters and its own log. A project describes how events are generated; an instance is one running of it. Instances are held in [startup.yml](/docs/core/config/startup-yml), so they survive a restart of the application. Instance list [#instance-list] | Column | Contents | | ------------------- | ---------------------------------------------------------------------------- | | **Instance** | The identifier, unique across the application | | **Project** | The project it runs, with a link into its workspace | | **Status** | See below | | **Flow** | Events per second written, averaged over the time since the instance started | | **CPU** | Share of one processor core, averaged over that same time | | **Errors** | Failures since that start — red once above zero | | **Written** | Events delivered since that start | | **Last start time** | When the instance was last started | The last five columns are runtime figures and exist only while an instance runs. For the rest they are empty and sort last. Intentional drops are not failures and are not counted as errors. The **Flow** and **CPU** columns are averages over the whole run, so a generator that produced a burst and went quiet keeps reporting both. For the figures right now, use the [Monitoring](/docs/studio/monitoring) dashboard, which measures over a moving window. Statuses [#statuses] | Status | Meaning | | ------------ | ------------------------------------------------------------------------------------------------- | | **Active** | Running and generating | | **Idle** | Registered, and not run yet | | **Finished** | Ran to completion — the input stage exhausted its timestamps and the generator stopped on its own | | **Failed** | Stopped because of an error; the reason is in the instance log | The **Starting** and **Stopping** statuses appear while a lifecycle action is in flight. Filtering and selection [#filtering-and-selection] Search by instance or by project and separate what is running from what is not. The filters are held in the page address, so a filtered list can be bookmarked or passed on. Selecting rows enables the **Start**, **Stop** and **Delete** actions over the whole selection, so a simulation of a dozen generators comes up in one action. Stopping drains an instance rather than cutting it off. The **Create new** button registers an instance: an identifier, the project it runs, and the parameters that project expects. Per-instance actions [#per-instance-actions] | Action | Available | | -------------------- | ---------------------------------------------------------------------- | | **Edit** | Always — opens the instance page | | **Rename** | While stopped | | **Clone** | Always — copies the parameters onto a new instance of the same project | | **Show metrics** | While running | | **Show logs** | Always | | **Start** / **Stop** | According to the current state | | **Delete** | While stopped | Cloning is the shortest way to run one project at two rates, or against two sets of parameters, at the same time. Instance metrics [#instance-metrics] Totals for the instance sit above a diagram of its pipeline, laid out from the input plugins through the event plugin to the outputs. Each node carries its own counters, including what it lost: events dropped or failed while being produced, and events a destination refused or a formatter rejected. The breakdown names the stage that is losing events instead of leaving it to be inferred from a total. In the dialog above the output writes everything it is handed and the event plugin fails on a fraction of its renders, so the whole gap between what the input enqueued and what was written belongs to the event stage. *** Instance page [#instance-page] An instance opens onto a page of its own, with the lifecycle controls in the header and three tabs below. Overview [#overview] The panel across the top leads with the rate events enter and leave the pipeline at, how many have failed and how many were dropped, and the totals those rates come from. Under it, what the run costs, in three groups: | Group | Contents | | -------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- | | **Processor** | Share of one core over the last interval, the processor time used, the time its threads spent waiting for a processor, and how many of them there are | | **Memory in queues** | How full the timestamps and the events queue are, in batches and — for the events queue — in bytes against the limit they may occupy | | **Input and output** | Bytes read and written through the file system, and bytes sent and received over the network | Below, the **Throughput** chart plots input against output over a moving window, and the **Pipeline** graph shows the same per-plugin counters the metrics dialog holds, updating while the page is open. A stopped instance leaves both empty. The panel beside them states what the instance is — the project, the emission mode, whether it starts with the application, its timezone and its last run — and the [scenarios](/docs/studio/scenarios) it belongs to, which are added and removed there. Settings [#settings] The configuration of this instance, over the fields [startup.yml](/docs/core/config/startup-yml) holds. The **Runtime** section decides how events are emitted — live, at the moments the timestamps name, or sample, as fast as the pipeline allows — whether past timestamps are skipped, whether the instance starts with the application, and the **Parameters** a configuration reads through `${params.name}`. One project serves several instances this way, differing only in a hostname or an output path. The **Generation** section overrides the application-wide defaults for this instance alone: batching, queue depths, ordering, write concurrency. These values are read when a generator starts, so saving a running instance offers to restart it. Logs [#logs] The log this instance writes, as it is written to disk. A **Failed** instance explains itself here, and a rendering failure names the offending event and the line of the template that raised on it. # Monitoring Monitoring is the dashboard for the whole application: every running instance added up, and each of them apart. It polls every five seconds and holds a moving window of **2.5**, **10** or **30 minutes**, chosen in the page header. With nothing running, the sections that depend on generation are replaced by a note pointing at [Instances](/docs/studio/instances), while the host and process resources stay live. The line of state [#the-line-of-state] The row under the title is the application in one line: the instances running, the rate each stage of generation is moving events at, the rate at which events are failing, and how full the fullest queue of any instance is. The totals since those instances started close the row — timestamps generated, events produced, events written and events dropped. | Reading | Meaning | | ----------------- | ----------------------------------------------------------------------------- | | **Input** | Timestamps entering the pipeline per second | | **Event** | Events produced from them per second | | **Output** | Events written to their destinations per second | | **Failing** | Events lost per second, at the event stage and the output stage together | | **Fullest queue** | The most loaded queue of any running instance, as a share of what it may hold | Read left to right, the rates locate where events are going missing without opening a single instance: an event rate under the input rate is the event stage dropping or failing to render, and an output rate under the event rate is a destination refusing them. Throughput and failures [#throughput-and-failures] The **Throughput** chart plots input against output as rates measured between polls — the rate now, rather than the average since a generator started that the [Instances](/docs/studio/instances) table reports. The **Failures** chart covers the same window beside it, split into the event stage and the output stage. Resources [#resources] CPU and memory for the host, disk and network for the Eventum process, each with its history over the window. The memory reading also names how much of the host's memory the Eventum process itself holds. Read next to the throughput chart, these separate a slow generator from a saturated machine: output that plateaus against a CPU ceiling is a different problem from output that plateaus with the machine idle. Instances [#instances] Everything about the running instances is one section: a chart of the load over the window, and the table of figures behind it. **By instance** stacks the output rate of each instance, with the smallest of them folded into a single band, so an instance that dominates the pipeline — or has quietly stopped contributing to it — is visible in the shape of the chart. **By stage** stacks the same window by pipeline stage instead. The search and the quick filters — **Failing**, **At the limit**, **Idle** — narrow the chart and the table together, and colour follows a row from one to the other. Selecting a row opens what that instance is doing beside it: the totals behind its rates, what each of its plugins moved, and the state of both its queues, one link away from [its own page](/docs/studio/instances#instance-page). The table ranks the instances by what they occupy, and sorts by every figure it shows. | Column | Contents | | ---------------- | ---------------------------------------------------------------------------------------------------- | | **CPU** | Share of one processor core over the last poll | | **Wait** | Share of that time spent ready to run while waiting for a processor | | **Output** | Events written per second | | **Failures** | Events lost per second | | **Disk write** | Bytes per second written through the file system | | **Network out** | Bytes per second sent | | **Events queue** | Memory the batches waiting between the event and the output stage occupy, against the limit they may | | **Threads** | Threads the instance runs | **Wait** is what separates an instance the machine cannot keep up with from a slow one: a figure that climbs across every instance means more of them are running than the host has processors to run, while a single hot instance next to calm ones names the generator to stop. Outside Linux the operating system does not account for waiting or file system bytes per thread, so **Wait** and **Disk write** read as zero there. The rest are reported on every platform. # Home Home is the screen Studio opens on. It answers two questions at once: what you were last working on, and whether anything is currently wrong. The **Recent projects** list holds the eight generator projects whose files changed most recently, and the **New project** button creates one. The rail on the right counts every registered instance by state and lists the five most recently started with their uptime. A failure is visible here without going looking for it, and the **Monitoring** link leads to the [dashboard](/docs/studio/monitoring) behind the counts. The **Explore** row below leads to the four sections generators are built and run in — [Projects](/docs/studio/projects), [Instances](/docs/studio/instances), [Scenarios](/docs/studio/scenarios) and [Repositories](/docs/studio/repositories), the last of which installs a ready-made generator into the workspace. The footer states the Eventum, Python and platform versions — the first thing worth quoting in a bug report. # Projects A **project** is a generator directory on disk: a [generator.yml](/docs/core/config/generator-yml) file together with the templates, scripts, samples and pattern files it refers to. Studio lists the directories found under the configured `path.generators_dir` and opens each of them in a workspace that edits the configuration and its files side by side. Project list [#project-list] Every project carries a badge for each [instance](/docs/studio/instances) registered against it, colored by that instance's status. The filters search by project name or by instance name, and narrow the list to projects that are in use or to those that no instance references. The **Create new** button writes a new directory with a minimal `generator.yml` in it. Renaming a project also updates the instances that point at it; deleting one is refused while any instance still does. Moving a project between instances [#moving-a-project-between-instances] A project travels as a ZIP archive of its directory. **Export** in the row menu opens a dialog listing every entry at the top of the project with its size — untick the ones the archive should leave out, which is where the directories holding generated output usually go, and the archive downloads under the project's name. **Import** takes such an archive - dropped onto the dialog or chosen from disk - and unpacks it into a new project. The archive may hold `generator.yml` at its top level or nested in directories, so an archive downloaded from a repository imports as it is. The name of the project is proposed from the directory the archive carries it in, falling back to the file name, and can be changed; a name already taken is reported before the import starts, and an existing project is never overwritten. A generator published by a git repository takes a shorter path: connect the repository once on the [Repositories](/docs/studio/repositories) page and install any generator it publishes straight into the workspace. *** The workspace [#the-workspace] Four panels divide the workspace, and every divider between them can be dragged. | Panel | Purpose | | ------------- | --------------------------------------------------------- | | **Explorer** | The files of the project — create, upload, move, rename | | **Editor** | Editing those files | | **Inspector** | The plugins of the selected stage and their parameters | | **Console** | Previewing the selected stage without running a generator | Pipeline stages [#pipeline-stages] The strip in the header is the generator's [three-stage pipeline](/docs/core/concepts/generator#the-three-stage-pipeline), and each entry names what the stage currently holds. Selecting a stage retargets the Inspector and the Console. Open files are unaffected. So is the work in the Console. Generated timestamps, a running debugger session and typed-in sample events are all still there after switching stages and coming back. Saving [#saving] The configuration and the project's files are edited in one place but saved as separate things. * The **Save** button in the header writes the configuration together with every modified file. While anything is unsaved, the badge beside the button names what is pending — the configuration, a number of files, or both. * The **Save file** button in the Editor header, or `Ctrl/Cmd+S`, writes only the file currently in front of you. Navigating away with anything unsaved asks for confirmation first. *** Explorer [#explorer] Files and directories are created from the panel header or from the right-click menu, and existing files are added with the **Upload files** button or by dropping them onto the tree from outside the browser — a drop lands in the directory under the cursor. Dragging an entry inside the tree moves it. `generator.yml` is what makes the directory a project, so it cannot be renamed, moved or deleted here. **Download** in the right-click menu saves a file to the machine you are browsing from, and the entry states the size before you start. It is the way to reach a generator's output, which grows past what the Editor opens within minutes of a run. *** Editor [#editor] Files open as tabs and stay open until closed, including across a change of stage. Jinja, Python, JSON, YAML and Markdown files are highlighted; anything else opens as plain text. In a `.jinja` file, autocomplete covers the [context variables and modules](/docs/plugins/event/template#template-context) a template can reach. `Ctrl/Cmd+F` opens find and replace, with case, whole-word and regular-expression matching. `Ctrl/Cmd+Alt+G` jumps to a line. Files larger than 10 MB are not opened — the tab reports the size and offers the file for download instead. Sample data of that size is better edited outside Studio. *** Inspector [#inspector] The Inspector holds the same three sections at every stage, filled from the stage selected in the pipeline strip. The **Plugins** section lists what the stage is configured with, and adds or removes entries. The input and output stages accept any number of plugins; the event stage accepts exactly one. The **Parameters** section holds the form for the selected plugin, validated against that [plugin's configuration](/docs/plugins) as you type. Fields that are not free text still accept `${params.name}` and `${secrets.name}` placeholders, which are resolved when an instance loads the configuration. The **Configuration** section shows the YAML the form produces, exactly as it will be written into `generator.yml`. *** Console [#console] The Console exercises the selected stage on its own, without registering or starting a generator. Which tool it offers follows that stage. | Stage | Tool | What it produces | | ------ | ------------------ | ---------------------------------------------------------------------------- | | Input | Timestamps preview | The timestamps the input plugins would emit, as a distribution and as a list | | Event | Event debugger | Events rendered on demand, with the failures behind them | | Event | Template state | The live state of the plugin the debugger is running | | Output | Formatter preview | Sample events passed through a formatter | The panel can be collapsed to its header, or maximized over the rest of the workspace. Timestamps preview [#timestamps-preview] The **Generate** button runs the input plugins and returns the timestamps they produce. The fields beside it decide what takes part and how the result is shaped: the **Plugins** field selects which of them run — left empty it runs all of them merged, as a generator would, and a subset isolates it — **Count** caps how many timestamps are generated, **Time span** sets the width of one histogram bin, and the **Skip past** switch starts the result at the first timestamp that is not in the past. Each plugin has its own color in the distribution, stacked where they overlap. Four of them contribute to the run above: a workday curve, a per-minute heartbeat, an evenly spaced backfill and a burst at start. Beside the chart are the timestamps themselves; a long run lists the first and the last of them with the number skipped in between. Event debugger [#event-debugger] The **Start** button starts an event plugin from the current configuration and keeps it running; the **Produce** button renders events with it. The **Event timestamp** and **Tags** fields stand in for what an input plugin would have supplied, so a template that branches on either is exercised through them. The **Count** field sets how many events one run produces from those same values, up to a hundred. With the **Auto** checkbox on, the timestamp is reset to the current time after each run. Produced events are listed on the left, with an optional syntax to highlight them by. Failures are listed on the right, each naming the event it happened on, the reason and the traceback. The plugin reads its templates when it starts and keeps them for the whole session. After editing a template, press **Stop** and then **Start** for the next run to use the new version. Template state [#template-state] The [template plugin](/docs/plugins/event/template) keeps state across renders, and the **State** view reads and writes that state on the plugin the debugger is running. It becomes available once a debugger session has been started. | Scope | Reach | | ---------------- | ------------------------------------- | | **Local state** | One template — pick it in the toolbar | | **Shared state** | Every template of this generator | | **Global state** | Every generator in the application | Values are edited as JSON, individual keys can be deleted, and a scope can be cleared as a whole. Writing a key directly puts a template into a state a short debugging run would not otherwise reach. Global state is shared with every running instance, not only the generator open in the workspace. [Scenarios](/docs/studio/scenarios) show which generators read and write which keys. Formatter preview [#formatter-preview] Add raw events, choose a [format](/docs/plugins/formatters) with its options, and the **Format** button shows the payload a destination would receive, together with the errors the formatter raised and the event behind each of them. The formatter is configured in the tool itself and is independent of the output plugins on this stage. The preview answers what a format does to an event, not what a particular destination is currently set to send. *** When the configuration cannot be loaded [#when-the-configuration-cannot-be-loaded] A `generator.yml` that fails to parse or validate leaves the stages with nothing to describe, so the pipeline strip, the Inspector and the Console are withdrawn and the workspace keeps the Explorer and the Editor. The alert reports what the validation rejected. Fix the file in the editor, save it, then press the **Reload** button to bring the rest of the workspace back. # Repositories A **connected repository** is a git repository that publishes ready-made generators. Studio reads the catalog it offers and installs any entry of that catalog as a [project](/docs/studio/projects) under `path.generators_dir` — without cloning the repository by hand and pointing the CLI at a path inside it. The repository Eventum publishes is [content-packs](https://github.com/eventum-generator/content-packs), the source of the generators listed in the [Hub](/hub). Any repository laid out the same way works the same. Finding a repository [#finding-a-repository] **Discover** lists the repositories that publish generators in the open. A repository appears there by carrying the `eventum-generators` topic on GitHub — there is nothing to register, and the content of a listed repository is not reviewed. Each entry states what its authors say about it: the description, the stars, the license and when it was last pushed to, with a link to the repository itself. **Connect** opens the same dialog as connecting by hand, filled in with the address and a free name. Community repositories are not reviewed by Eventum. A generator can carry templates and scripts that are executed on your machine when the generator runs, so review what you install and connect only repositories you trust. The search is anonymous — nothing is configured, and no token is asked for. GitHub allows a small number of anonymous searches a minute per address, so what was read is kept for ten minutes and answered from; a search refused because that allowance is spent is reported as such and works again once it resets. Connecting a repository [#connecting-a-repository] **Connect** asks for the address of the repository and, for a private one, the credentials to reach it. The repository is checked before it is connected, so an address that leads nowhere, a branch that does not exist and credentials the host refuses are reported in the form rather than when the catalog is first read. A repository that is temporarily out of reach can be connected anyway from the same dialog. | Field | Required | Description | | ----------------- | -------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Name** | Yes | The name the repository is listed under. Letters, digits, `-`, `_` and `.`. | | **URL** | Yes | The address the repository is fetched from. Only `http://` and `https://` are accepted. | | **Branch or tag** | No | What to fetch. The default branch of the repository when left empty. | | **User name** | No | The user name of a private repository. | | **Password** | No | The password or access token of a private repository, either as the value itself or as a `${secrets.}` [reference](/docs/core/config/secrets) read from the keyring. | A reference keeps the token out of the repositories file: the keyring is read for the duration of a fetch and nothing else holds the value. A password written as the value is kept in the file as it is written, and is the reason the list shows `***` in its place rather than what was typed. Either way a URL carrying credentials — `https://user:token@host/repo.git` — is rejected; give the two parts separately. Because the field holds a name, the [Secrets](/docs/studio/settings#secrets) page counts the repository among what uses that secret, and renaming it there repoints the repository at the new name. Add the access token to the keyring before connecting the repository, either on the [Secrets](/docs/core/config/secrets) page or with `eventum-keyring set`, and the key beside the **Password** field offers it by name. GitHub and GitLab both accept a personal access token in place of a password. The same repository may be connected more than once to follow two of its branches; what is refused is a name already taken, or that repository at a branch or tag already connected — the address is compared by what it points at, so `…/packs` and `…/packs.git` count as one. Each repository carries a badge for whether it answered the last time it was asked: **Reachable**, **Unreachable** with the reason, or **Not checked** until the page asks. Reading a catalog counts as an answer, so a refresh keeps the badge current. The check is a request for the references the repository publishes and transfers nothing of it. **Disconnect** removes a repository from the list. Generators already installed from it stay in the workspace, and each of them keeps the record of where it came from, so connecting the repository again marks them as installed once more. What a repository publishes [#what-a-repository-publishes] A repository publishes its generators from a `generators` directory at its root. Every subdirectory holding a generator configuration is one catalog entry; a subdirectory without one is ignored, so documentation and tooling can live alongside. The entry is named after its directory. Its title and the sentence describing it come from the `README.md` of the generator — the first heading and the paragraph under it — and its size is the size of the files it consists of. Reading the catalog [#reading-the-catalog] Opening a repository reads its catalog. The reading is a fetch of the repository, so it happens on request rather than in the background, and what was read is kept for as long as the instance runs. The line above the table names how many generators the repository publishes, the revision the catalog was read from with its author, and the moment it was read. The search beside it narrows the table by name, title and summary. **Refresh** reads the catalog again, which is how a generator added to the repository after the catalog was first read appears in the list. Selecting a generator opens what the repository states about it: its size and file count, the path it lives at with a link to it on GitHub or GitLab, the commit the catalog was read from, and the projects it is already installed as. Installing a generator [#installing-a-generator] **Install** asks for the name of the project to install into and writes the generator there. The name is proposed from the entry and can be changed; a name already taken is reported before anything is written, and an existing project is never overwritten. A generator the workspace already holds is marked **installed** and offers **Open** instead, which goes to the project it was installed as. Installing it a second time is a step deeper, in the card of the generator: **Install another copy** writes a separate project beside the first, under a free name, and leaves the one you have untouched. That is also how a generator the repository has changed is taken — the new version arrives as its own project, to be compared against the one in use rather than overwriting it. What the repository publishes is written into the workspace under the same rules as an [imported archive](/docs/studio/projects#moving-a-project-between-instances): only regular files are installed, so a symbolic link or a submodule the repository carries is left behind rather than followed, and a generator holding more than 10 000 files or unpacking to more than 512 MiB is refused. The installed project is a copy like any other. It can be edited, renamed, run and exported, and a later change in the repository does not reach it — install it again under another name to compare. An agent connected over [MCP](/docs/mcp) installs from the same repositories: it lists what is connected, reads a catalog and installs an entry, while connecting a repository stays here. See [Tools & resources](/docs/mcp/tools#repository-tools). What a project remembers [#what-a-project-remembers] An installed project carries a `.eventum-source.yml` file naming where it came from: the repository, the generator, the commit, and the content of that generator at that commit. The catalog reads it back, so a generator already in the workspace is recognized by its origin rather than by a name that happens to match — a project renamed afterwards is still recognized, and an unrelated project sharing a name is not mistaken for one. What the workspace already holds is marked beside the name of the generator: **installed**, or **update available** when the repository publishes the generator with content different from what was installed. The mark follows the repository, not the project — editing an installed project never raises it. The card names the projects it is installed as, each with whether it is still up to date and a link that opens it. ```yaml title=".eventum-source.yml" repository: content-packs url: https://github.com/eventum-generator/content-packs.git ref: null entry: web-nginx revision: 29b728c556c04f9d1c8a8e9aaa2a5a27bdfdee17 tree: 1b2071b4d4997c29a2f223b5242a4d553c66d7ff installed_at: '2026-08-19T18:30:44.198256Z' ``` Deleting the file only takes the mark away; the project itself is unaffected. Publishing your own repository [#publishing-your-own-repository] Any git repository laid out the way described above can be connected by address. To have it listed under **Discover** as well, publish it on GitHub and: **Lay the generators out.** Every generator goes in its own directory under `generators/`, with a `generator.yml` in it — that is what makes a directory a catalog entry — and a `README.md` whose first heading and first paragraph become the title and the summary shown in the catalog. **Add the topic.** On the repository page on GitHub, open the settings gear beside **About** and add `eventum-generators` to **Topics**. That is the whole registration: the list is a search for that topic. **Describe the repository.** The **About** description and the license are what an entry shows besides the name, so a repository without a description is listed without one. A repository appears in the list once GitHub has indexed the topic, usually within minutes. It stays listed for as long as it carries the topic, and removing the topic takes it off the list — while anyone who already connected it keeps their connection, since a connection is an address, not an entry in a registry. Where the list is kept [#where-the-list-is-kept] The connected repositories are kept in a file of their own, `repositories.yml` next to the [startup file](/docs/core/config/startup-yml) unless `path.repositories` in [eventum.yml](/docs/core/config/eventum-yml#path) names another location: ```yaml title="repositories.yml" - name: content-packs url: https://github.com/eventum-generator/content-packs.git - name: internal url: https://git.example.com/platform/generators.git ref: stable username: eventum password: ${secrets.internal_git_token} ``` The file can be written by hand as well; the list is read on every request, so an instance picks up an edit without a restart. Two things are worth knowing before editing it: connecting or disconnecting a repository from Studio rewrites the whole file, which does not preserve comments or the order of keys, and a file that no longer parses fails the whole Repositories page rather than being partly read. # Scenarios A **scenario** is a named group of generator instances that run as one unit. [Projects](/docs/studio/projects) define what a generator does; a scenario defines which instances belong together at runtime. A corporate network simulation, for example, may consist of an authentication service tracking user sessions, a web proxy generating traffic for those users, a DNS resolver publishing the domains it refuses, and a firewall logging connections from the same population. Those generators exchange data through [global state](/docs/plugins/event/template/state) and are started and stopped together. Configuration [#configuration] Scenarios come from the `scenarios` field of each instance in [startup.yml](/docs/core/config/startup-yml). An instance can belong to several of them: ```yaml title="startup.yml" - id: corp-auth-service path: corp-auth-service/generator.yml scenarios: - corporate-network - security-monitoring - id: corp-web-proxy path: corp-web-proxy/generator.yml scenarios: - corporate-network - id: corp-dns-resolver path: corp-dns-resolver/generator.yml scenarios: - corporate-network - id: corp-edge-firewall path: corp-edge-firewall/generator.yml scenarios: - corporate-network - security-monitoring ``` Here `corporate-network` groups all four instances and `security-monitoring` groups two of them. There is no separate scenario file: a scenario exists as soon as an instance names it. Studio writes the same field. Scenarios can be created on the Scenarios page, and membership can be changed from a scenario or from an [instance](/docs/studio/instances#overview). Scenario list [#scenario-list] Each row holds the number of instances in a scenario and the states they are in, so one that is only half running is distinguishable from one that is fully up. Rows can be started, stopped or deleted one at a time or as a selection. The **Create new** button takes a name and the instances that belong to it. Deleting a scenario removes the grouping and leaves the instances themselves registered. Scenario page [#scenario-page] The header carries the aggregate status of the group with the **Start all** and **Stop all** buttons. Where the [Instances](/docs/studio/instances) page acts on instances one at a time, this brings a whole simulation up or takes it down in one action. Data flow [#data-flow] The diagram maps the global state the instances exchange: which generator writes a key, and which ones read it. It is present once any generator in the scenario reads or writes a key — from a Jinja template or from the script of a `script` event plugin, both of which are analysed the same way. Hovering an instance or a key dims everything it does not touch, which keeps a key written by one generator and read by three others legible. Instances [#instances] Each instance is a card with the actions the Instances page offers for it, plus a **Remove** action, which takes it out of the group without deleting it. A card for a generator that uses global state expands into its templates and scripts and, for each file, the keys it writes and the keys it reads. Hovering an entry highlights the matching edge in the diagram above. Opening a file name shows its source read-only, inside the card, at the place where the state is read and written. Global state [#global-state] The panel on the right is the live `globals` dictionary the diagram describes, searchable over keys and values, with objects and arrays expandable rather than truncated. Keys can be added, edited as JSON and deleted, which puts a scenario into a state that would otherwise take a long run to reach. Global state is application-wide. Editing it affects every running instance, not only the ones in this scenario. Check the data flow diagram for what depends on a key before changing it. # Settings and management Three pages administer the application itself rather than the generators running in it: the **Settings** page for its configuration, the **Secrets** page for the credentials that configuration refers to, and the **Management** page for what the process is doing. Settings [#settings] The Settings page edits the application configuration — the same parameters [eventum.yml](/docs/core/config/eventum-yml) holds, written back to that file on save. | Section | Covers | | -------------- | -------------------------------------------------------------------------- | | **Server** | How the instance is reached — API, web interface, TLS, authentication, MCP | | **Generation** | The defaults every generator inherits | | **Paths** | Where generators, logs, the startup file and the keyring live | | **Logging** | Level, the level of third-party libraries, format, rotation | A section holding unsaved edits is marked in the rail, so changes spread across several of them stay visible from wherever you are. Batch size and queue depths decide how much memory generation holds at once. The Generation section works that figure out for you: give it an assumed event size and it reports what full queues will occupy for one generator. **Limit memory of events queue** bounds it directly: **Maximum event bytes** caps what the events queue holds whatever the batches in it weigh. The **Save** button appears in the page header once anything is edited and asks for confirmation first. These parameters are read when the application starts, so applying them triggers the same [restart](#danger-zone) the Management page offers, and the interface is briefly unavailable. *** Secrets [#secrets] The Secrets page manages the encrypted keyring, the same one [`eventum-keyring`](/docs/core/cli/eventum-keyring) manages from the terminal. A secret is referenced from any configuration as `${secrets.name}` and resolved when a generator loads. Values stay masked until asked for. Secrets are added, edited and removed in place, and a new one is available to generator configurations immediately, with no restart. Renaming and removing a secret both state what refers to it first. A rename carries every referrer over to the new name: the `${secrets.name}` token is rewritten in the configuration of each project reading it, and a [connected repository](/docs/studio/repositories) authenticating with the secret is repointed. A configuration is rewritten as text, so its comments and formatting stay as they were written; a generator already running holds the configuration it loaded and reads the new name the next time it starts. A removal carries nothing over — both kinds keep the name and stop working until a secret of that name exists again. A rename is refused when a repository already authenticates with the new name, and the repositories holding it are named. The keyring holds one value per name, so the repository left on that name would otherwise start authenticating with the value that just moved under it — the credential of one host reaching another. Removing a secret is irreversible. Any configuration referring to it fails to load until a secret of the same name exists again, and a repository authenticating with it stops answering. *** Management [#management] The cards at the top identify the instance and the machine under it, and link through to [Monitoring](/docs/studio/monitoring) for the history behind the load. The **Application** card also reports whether the GIL is enabled. Generators run on threads, so the [free-threaded build](/docs/core/introduction/requirements#python) is what lets them run in parallel, and on that build the GIL is normally disabled. It can be enabled back after the application starts — by the `PYTHON_GIL=1` environment variable, by the `-X gil=1` interpreter option, or by an extension module without free-threading support that a plugin imports. Generators keep running, but no longer in parallel, so the row reports that state as a warning. On a standard build the GIL is always enabled and the row is neutral. The **Instance logs** panel holds the log of the application itself, split into the channels it writes: | Channel | Contents | | ---------- | --------------------------------------------------------------------------------------------------------- | | **Main** | The application core — configuration loading, which generators were started, and which of them refused to | | **Server** | The API and the HTTP server | | **Access** | The requests the server served | | **MCP** | The [MCP server](/docs/mcp), when it is mounted | A single generator's own log is on its [instance page](/docs/studio/instances#logs). Danger zone [#danger-zone] The **Restart** button stops the whole application and starts it again from [eventum.yml](/docs/core/config/eventum-yml) and [startup.yml](/docs/core/config/startup-yml), without ending the process — what `SIGHUP` does, and the way a change made to either file outside Studio is picked up. The server stops with everything else, so the interface is unavailable for a moment. The **Stop** button shuts the generators down gracefully and ends the process. The **Stop** button ends the whole Eventum process, including the interface you are using. Starting it again means going back to the terminal and running `eventum run -c eventum.yml`. # Seed a Database with Realistic Test Data A development database seeded with three hand-typed rows and a staging environment running against an empty table hide the same class of bug: a pagination control never tested past page one, a query that returns instantly against a dozen rows and takes seconds against ten thousand, an edge case that only exists because a real distribution has a long tail. Copying a snapshot of production data fixes the volume and shape problem but trades it for a worse one — real customer names, payment details, and order histories sitting in an environment that was never built to hold them, reachable by anyone with access to dev. Eventum closes that gap by generating the dataset instead of copying it. Build a generator that produces an e-commerce transaction dataset as a CSV file: purchases, refunds, and chargebacks in a realistic 80/15/5 mix, with correlated fields — customer, product, amount — instead of independently random values. The result is a file shaped like a real `transactions` export, ready to load into whatever database backs dev or staging today. Seed dev database with realistic data [#seed-dev-database-with-realistic-data] Three rows of hand-typed fixture data confirm that a `transactions` table accepts the right columns and nothing more. They don't confirm that a paginated transaction list still responds quickly on page 40, that a monthly revenue report holds up once chargebacks and refunds are mixed in at their real, rare frequency instead of one of each by hand, or that a report which only ever saw positive amounts survives its first negative refund row. All three failure modes need the volume and the shape a handful of manual rows can't provide — which is exactly what the generator built below produces: ten thousand transactions with a realistic distribution, not ten thousand copies of the same row. Copying a production snapshot instead would supply that volume, but it drags real customer names, card details, and order histories into a database any developer on the team can query — a liability a synthetic dataset never carries, since none of it describes a real person or a real purchase. It also goes stale the moment it's taken; regenerating a synthetic dataset costs one command, not a new export. Why CSV for database seeding [#why-csv-for-database-seeding] A CSV file is the most portable way to get that dataset into a database, because every mainstream engine already has a bulk-import path built around it: PostgreSQL's `COPY`, MySQL's `LOAD DATA INFILE`, and the CSV import wizard built into most GUI database clients all read the same file format. The dataset generated below drops into whatever engine currently backs dev — no custom loader script, no per-database export format to maintain. A destination that accepts inserts directly instead of a file can skip the CSV step entirely. Eventum's [`clickhouse` output](/docs/tutorials/delivery/clickhouse) inserts each transaction into a table over HTTP as it's generated, so an analytical store built for that kind of write never needs a file in between — the Going further section revisits this once the CSV build below is in place. What you'll build [#what-youll-build] The generator uses: * [**linspace**](/docs/plugins/input/linspace) input — 10,000 timestamps spread evenly across a date range. * [**chance**](/docs/plugins/event/template/modes#chance) picking mode — weighted random selection between three transaction types. * [**file**](/docs/plugins/output/file) output with the default [`plain` formatter](/docs/plugins/formatters#plain) — each template already renders a complete CSV row, so nothing needs reshaping before it's written. * [**JSON samples**](/docs/plugins/event/template/samples) — product catalog loaded from a file. * [**shared state**](/docs/core/concepts/producing#state-management) — running revenue counter across all events. The entire dataset is generated in one burst using [sample mode](/docs/core/concepts/generator#sample-mode) (`live_mode: false`), so 10,000 rows complete in seconds. Prerequisites [#prerequisites] * [Eventum installed](/docs/core/introduction/installation) Project structure [#project-structure] Build it [#build-it] Create the project directory [#create-the-project-directory] ```bash mkdir -p ecommerce-csv/{templates,data} cd ecommerce-csv ``` Create the product catalog [#create-the-product-catalog] A JSON sample file with product names, categories, and price ranges. Each transaction picks a random product from this list. ```json title="data/products.json" [ { "name": "Wireless Mouse", "category": "Electronics", "min_price": 15, "max_price": 45 }, { "name": "USB-C Hub", "category": "Electronics", "min_price": 25, "max_price": 80 }, { "name": "Desk Lamp", "category": "Home", "min_price": 20, "max_price": 60 }, { "name": "Notebook Set", "category": "Office", "min_price": 8, "max_price": 25 }, { "name": "Bluetooth Speaker", "category": "Electronics", "min_price": 30, "max_price": 120 }, { "name": "Coffee Mug", "category": "Home", "min_price": 10, "max_price": 30 }, { "name": "Backpack", "category": "Accessories", "min_price": 35, "max_price": 90 }, { "name": "Webcam", "category": "Electronics", "min_price": 40, "max_price": 150 }, { "name": "Plant Pot", "category": "Home", "min_price": 12, "max_price": 35 }, { "name": "Mechanical Keyboard", "category": "Electronics", "min_price": 60, "max_price": 200 } ] ``` Write transaction templates [#write-transaction-templates] Each template produces a single CSV row. All three templates share the same column order: `transaction_id, timestamp, type, customer_id, customer_name, product, category, amount, currency`. The `{%- -%}` markers around each `set`/`do` line trim the newline that would otherwise follow it, so the rendered event is exactly one line — without them, every rendered row would carry three blank lines ahead of it. **Purchase** — the most common transaction. Picks a random product and generates a price within its range. ```jinja title="templates/purchase.jinja" {%- set product = module.rand.choice(samples.products) -%} {%- set amount = module.rand.number.floating(product.min_price, product.max_price) -%} {%- do shared.set("revenue", shared.get("revenue", 0) + amount) -%} {{ module.rand.crypto.uuid4() }},{{ timestamp.strftime("%Y-%m-%d %H:%M:%S") }},purchase,C-{{ module.rand.number.integer(10000, 99999) }},{{ module.faker.locale.en.name() }},{{ product.name }},{{ product.category }},{{ "%.2f" | format(amount) }},USD ``` JSON sample rows support named access via object keys: `product.name`, `product.category`, `product.min_price`, `product.max_price`. **Refund** — returns a product. The amount is negative. ```jinja title="templates/refund.jinja" {%- set product = module.rand.choice(samples.products) -%} {%- set amount = module.rand.number.floating(product.min_price, product.max_price) -%} {%- do shared.set("revenue", shared.get("revenue", 0) - amount) -%} {{ module.rand.crypto.uuid4() }},{{ timestamp.strftime("%Y-%m-%d %H:%M:%S") }},refund,C-{{ module.rand.number.integer(10000, 99999) }},{{ module.faker.locale.en.name() }},{{ product.name }},{{ product.category }},-{{ "%.2f" | format(amount) }},USD ``` **Chargeback** — disputed transaction. Rare (5% of total). ```jinja title="templates/chargeback.jinja" {%- set product = module.rand.choice(samples.products) -%} {%- set amount = module.rand.number.floating(product.min_price, product.max_price) -%} {%- do shared.set("revenue", shared.get("revenue", 0) - amount) -%} {{ module.rand.crypto.uuid4() }},{{ timestamp.strftime("%Y-%m-%d %H:%M:%S") }},chargeback,C-{{ module.rand.number.integer(10000, 99999) }},{{ module.faker.locale.en.name() }},{{ product.name }},{{ product.category }},-{{ "%.2f" | format(amount) }},USD ``` The `shared.set("revenue", ...)` call tracks a running total across all events. This doesn't appear in the CSV — it's an example of cross-template coordination. You could use it in a summary template or inspect it in the [State view](/docs/studio/projects#state) of the project console in Studio. Configure the generator [#configure-the-generator] The [chance](/docs/plugins/event/template/modes#chance) picking mode assigns each template a relative weight, distributing timestamps across the three transaction types. ```yaml title="generator.yml" input: - linspace: start: "2025-01-01T00:00:00" end: "2025-12-31T23:59:59" count: 10000 endpoint: true event: template: mode: chance samples: products: type: json source: data/products.json templates: - purchase: template: templates/purchase.jinja chance: 0.80 - refund: template: templates/refund.jinja chance: 0.15 - chargeback: template: templates/chargeback.jinja chance: 0.05 output: - file: path: transactions.csv write_mode: overwrite ``` Key decisions: * **linspace** spreads 10,000 timestamps evenly across the entire year 2025 — about one transaction every 53 minutes. * **chance** mode: 80% of timestamps render the `purchase` template, 15% `refund`, 5% `chargeback`. * **chance values are relative weights** — 0.80/0.15/0.05 need not sum to 1; only the ratio between them matters (see [chance picking mode](/docs/plugins/event/template/modes#chance)). * The file output uses `write_mode: overwrite` so each run starts fresh. The default [`plain` formatter](/docs/plugins/formatters#plain) passes each rendered template through unchanged — since every template already renders a full CSV row, there's nothing left to reshape. Run it [#run-it] Use [`eventum generate`](/docs/core/cli/eventum-generate) in sample mode for fast, one-shot generation: ```bash eventum generate --path generator.yml --id ecommerce --live-mode false ``` The `--live-mode false` flag releases all 10,000 timestamps instantly instead of waiting for their scheduled wall-clock times. The entire dataset generates in a few seconds. Add a CSV header and check the output: ```bash sed -i '1i transaction_id,timestamp,type,customer_id,customer_name,product,category,amount,currency' transactions.csv head -6 transactions.csv ``` A real run produced this — three purchases followed by two refunds: ```csv transaction_id,timestamp,type,customer_id,customer_name,product,category,amount,currency 5d842291-9c9e-4c88-a9b0-76d3ae3e0ccb,2024-12-31 21:00:00,purchase,C-86918,Rachel Hill,Desk Lamp,Home,53.31,USD df9d7962-6a0d-467a-888b-7221ab42b51f,2024-12-31 22:45:07,purchase,C-94340,David Evans,Mechanical Keyboard,Electronics,168.20,USD c120dd12-cfe3-4e95-b585-d273e52b626e,2024-12-31 21:52:33,purchase,C-34622,Jacob Smith,Webcam,Electronics,71.57,USD 8ef4f5d0-99a9-47d3-9920-e387d699987e,2024-12-31 23:37:41,refund,C-16314,Ricky Ewing DVM,Mechanical Keyboard,Electronics,-128.78,USD 4fc84fe8-be85-4920-8681-f9c389e67b68,2025-01-01 00:30:15,refund,C-72557,Andrea Barnes,Wireless Mouse,Electronics,-43.52,USD ``` Elsewhere in the same run, the rarer chargeback case shows up too: ```csv 8b11c3e4-2c40-4d20-8063-82fcb0a7bc6d,2025-01-01 17:09:00,chargeback,C-36012,Justin Russell,Plant Pot,Home,-17.73,USD ``` Check the distribution across the full file: ```bash tail -n +2 transactions.csv | cut -d',' -f3 | sort | uniq -c | sort -rn ``` ```text 7977 purchase 1560 refund 463 chargeback ``` That's 79.8% / 15.6% / 4.6% against a configured 80/15/5 — close, with the small spread expected from a chance-weighted pick rather than a fixed count. Seed staging environment from the CSV [#seed-staging-environment-from-the-csv] The header written in the previous step already names every column a `transactions` table needs, so the file drops straight into a bulk-import command instead of a custom loader script. PostgreSQL reads it with `\copy`, the client-side form of `COPY` that works against a local file without needing server-side filesystem access: ```sql \copy transactions FROM 'transactions.csv' WITH (FORMAT csv, HEADER true) ``` MySQL uses `LOAD DATA`: ```sql LOAD DATA LOCAL INFILE 'transactions.csv' INTO TABLE transactions FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"' LINES TERMINATED BY '\n' IGNORE 1 LINES; ``` Most GUI database clients — pgAdmin, DBeaver, TablePlus, MySQL Workbench — wrap the same operation behind an "Import from CSV" wizard, for a one-off load that doesn't need a saved command. None of that changes when the target shifts: the same command seeds a staging environment exactly the way it seeds dev, pointed at a different connection string. Regenerating a fresh CSV whenever a schema change or a new test scenario calls for different data costs one command — waiting on the next production snapshot doesn't. Going further [#going-further] * **Add customer loyalty** — use `shared` state to track repeat customers by keeping a customer pool and reusing IDs with some probability. * **Seasonal patterns** — replace `linspace` with [time-patterns](/docs/plugins/input/time-patterns) using a triangular distribution to create Black Friday and holiday spikes. * **Multiple currencies** — add `params` for currency and run multiple generators in parallel with different currency settings. * **Seed an analytical store directly** — swap the `file` output for [ClickHouse](/docs/tutorials/delivery/clickhouse) and every transaction inserts straight into a table over HTTP as it's generated, with no CSV or import step in between. What's next [#whats-next] FAQ [#faq] No. `chance` values are relative weights, not percentages of a fixed total — Eventum's picker normalizes them internally, the same way Python's `random.choices` does, so `0.80`/`0.15`/`0.05` and `80`/`15`/`5` pick with identical odds. The only constraint is that each `chance` value must be greater than 0; what matters is the ratio between them, not what they add up to. No. The same shaped, correlated CSV works as an analytics fixture or an ML training set — anywhere a realistic file beats a handful of hand-written rows. Seeding a database just adds one step after generation, a bulk-import command; analytics and ML workloads typically read the file directly instead. A related question, but not the same one. [Load testing](/docs/tutorials/load-testing) measures how much request volume a target can sustain before it degrades. [Pipeline testing](/docs/tutorials/test-data-pipeline) measures whether one realistic event survives a specific transport and lands intact. Seeding, this lesson's subject, asks neither — it asks whether a database already holding realistic volume and distribution behaves correctly at rest: pagination past page one, query performance at scale, and reports that only break against a rare row a handful of fixtures never produce. Related [#related] * The [Realistic values](/docs/tutorials/realism/values) lesson for the distributions and weighted choices behind the price ranges and the 80/15/5 transaction mix * The [ClickHouse delivery lesson](/docs/tutorials/delivery/clickhouse) for seeding an analytical database directly, with no CSV in between * The [Test data pipeline](/docs/tutorials/test-data-pipeline) lesson for testing transport correctness instead of seeding a database at rest * The [Scenarios pillar](/docs/tutorials/scenarios) for the course's other synthetic-data scenarios * The [Eventum Hub](/hub) for datasets already shaped for common sources, ready to load without building one first # Test Sigma Rules with Synthetic Attack Telemetry Shipping a detection rule means answering two questions with evidence, not intuition: does it fire on the activity it was written to catch, and does it stay silent on everything else. A live attack answers both, but only after standing up an isolated lab and running the technique itself — expensive to repeat, and never something to point at a production identity provider or domain controller. A static log sample pulled from a public repository answers the question once, for one host, one account, one moment, and stops answering it the day naming conventions, account names, or timing change. Eventum generates the telemetry a detection rule actually inspects — the event IDs, fields, and repetition a technique leaves behind — parameterized across as many source addresses, accounts, and volumes as a test needs, blended into the ordinary traffic the rule has to ignore. This lesson generates telemetry shaped like a specific attack pattern — event IDs, field values, and repetition — to exercise a detection rule. It runs no exploit code, payload, or attacker tooling — every event below is an ordinary Windows Security log entry, mechanically identical to the benign traffic surrounding it. Attack telemetry for detection testing [#attack-telemetry-for-detection-testing] Attack telemetry, in this sense, is not an attack. It is the sequence of ordinary log events a technique produces as a side effect — repeated authentication failures from one address, a process spawned by an unexpected parent, a registry key written by a process that doesn't normally write it. Reproducing that sequence exercises a rule exactly the way the real technique would, without executing anything resembling the technique itself. Two open, vendor-neutral standards make that reproduction checkable rather than improvised. [Sigma](https://sigmahq.io/) is a generic, structured format for writing a detection rule once — a YAML document naming a log source, the field values that must match, and the condition that must hold — and converting it to any backend's native query language. [MITRE ATT\&CK](https://attack.mitre.org/) is a taxonomy of adversary techniques; a Sigma rule tags the techniques it's meant to catch, so the telemetry that tests it can be described the same way — this stream contains a Brute Force (T1110) pattern, not just this stream contains some failed logons. Testing a rule against telemetry like this means generating both halves of the stream at once: an attack-like pattern shaped to the rule's own logic, and enough ordinary noise around it to prove the rule doesn't fire on the noise too. A [`chance`](/docs/plugins/event/template/modes#chance)-weighted mix of templates produces exactly that split — a low-probability pattern buried in a high-probability background — without a state machine tracking which stage of an attack is underway. What you'll build [#what-youll-build] A single Eventum generator that produces a Windows Security authentication stream with a brute-force pattern deliberately planted in it, alongside the Sigma rule that catches it: * **Three [`template`](/docs/plugins/event/template) files** — a benign logon that dominates the stream, plus the two halves of the attack: a repeated failed logon and the occasional success that follows it. * **[`chance`](/docs/plugins/event/template/modes#chance) picking mode** weights the mix so attack-shaped events stay a small minority, concentrated onto a fixed pool of source addresses held in [`shared`](/docs/plugins/event/template/state) state. * **A bounded [`static`](/docs/plugins/input/static) input** produces a fixed, rerunnable batch — the shape a repeatable detection test wants. * **A [`file`](/docs/plugins/output/file) output** writes the stream as newline-delimited JSON, ready to grep or forward to a SIEM. * **A Sigma [correlation rule](https://github.com/SigmaHQ/sigma-specification/blob/main/specification/sigma-correlation-rules-specification.md)** that fires on the planted pattern and stays quiet on the benign background. Prerequisites [#prerequisites] * [Eventum installed](/docs/core/introduction/installation) Project structure [#project-structure] Build it [#build-it] Create the project directory [#create-the-project-directory] ```bash mkdir -p detection-lab/templates cd detection-lab ``` Write the event templates [#write-the-event-templates] The example below reproduces a classic [Brute Force](https://attack.mitre.org/techniques/T1110/) (T1110) pattern against Windows authentication: many failed logons from one source address, occasionally followed by a success once a guessed credential lands. All three templates share the JSON shape from the [Windows Event Log lesson](/docs/tutorials/formats/windows-event-log) — `event.code`, `winlog.event_id`, and the native `event_data` fields a real Security-channel event carries. `templates/benign-logon.jinja` covers the overwhelming majority of traffic: an ordinary successful logon, with both the source address and the account drawn fresh at random every time. ```jinja title="detection-lab/templates/benign-logon.jinja" {%- set username = module.rand.choice(["jsmith", "ajohnson", "mwilliams", "kbrown", "tpatel"]) -%} {%- set src_ip = module.rand.network.ip_v4_public() -%} {%- set record_id = shared.get('record_id', 1) -%} { "@timestamp": "{{ timestamp.isoformat() }}", "event": { "code": "4624", "action": "logged-in", "outcome": "success" }, "user": { "name": "{{ username }}", "domain": "{{ params.domain }}" }, "source": { "ip": "{{ src_ip }}" }, "winlog": { "channel": "Security", "provider_name": "Microsoft-Windows-Security-Auditing", "event_id": "4624", "record_id": "{{ record_id }}", "logon": { "type": "Network" }, "event_data": { "TargetUserName": "{{ username }}", "TargetDomainName": "{{ params.domain }}", "LogonType": "3", "IpAddress": "{{ src_ip }}" } } } {%- do shared.set('record_id', record_id + 1) -%} ``` `templates/brute-force-attempt.jinja` covers the failed side of the pattern. Instead of a fresh address per event, it draws from a small pool of three addresses generated once and kept in [`shared`](/docs/plugins/event/template/state) state — the same three addresses recur across many events, which is exactly what makes them stand out against the benign traffic's high-cardinality noise. The targeted account also comes from a small, fixed set, since a real credential-guessing campaign aims at known or predictable account names rather than the general user directory: ```jinja title="detection-lab/templates/brute-force-attempt.jinja" {%- if not shared.get('attacker_ips') -%} {%- do shared.set('attacker_ips', [module.rand.network.ip_v4_public(), module.rand.network.ip_v4_public(), module.rand.network.ip_v4_public()]) -%} {%- endif -%} {%- set src_ip = module.rand.choice(shared.get('attacker_ips')) -%} {%- set username = module.rand.choice(["administrator", "admin", "svc-sql", "backup-admin"]) -%} {%- set status = module.rand.weighted_choice(["0xC000006A", "0xC0000064"], [90, 10]) -%} {%- set record_id = shared.get('record_id', 1) -%} { "@timestamp": "{{ timestamp.isoformat() }}", "event": { "code": "4625", "action": "logon-failed", "outcome": "failure" }, "user": { "name": "{{ username }}", "domain": "{{ params.domain }}" }, "source": { "ip": "{{ src_ip }}" }, "winlog": { "channel": "Security", "provider_name": "Microsoft-Windows-Security-Auditing", "event_id": "4625", "record_id": "{{ record_id }}", "event_data": { "TargetUserName": "{{ username }}", "TargetDomainName": "{{ params.domain }}", "Status": "{{ status }}", "IpAddress": "{{ src_ip }}" } } } {%- do shared.set('record_id', record_id + 1) -%} ``` `templates/brute-force-success.jinja` covers the rare case where a guess lands: the same address pool, the same account pool, a successful outcome instead of a failure. It reads the same `attacker_ips` key from `shared` state that the attempt template above writes, so both draw from the same three addresses regardless of which template happens to render first: ```jinja title="detection-lab/templates/brute-force-success.jinja" {%- if not shared.get('attacker_ips') -%} {%- do shared.set('attacker_ips', [module.rand.network.ip_v4_public(), module.rand.network.ip_v4_public(), module.rand.network.ip_v4_public()]) -%} {%- endif -%} {%- set src_ip = module.rand.choice(shared.get('attacker_ips')) -%} {%- set username = module.rand.choice(["administrator", "admin", "svc-sql", "backup-admin"]) -%} {%- set record_id = shared.get('record_id', 1) -%} { "@timestamp": "{{ timestamp.isoformat() }}", "event": { "code": "4624", "action": "logged-in", "outcome": "success" }, "user": { "name": "{{ username }}", "domain": "{{ params.domain }}" }, "source": { "ip": "{{ src_ip }}" }, "winlog": { "channel": "Security", "provider_name": "Microsoft-Windows-Security-Auditing", "event_id": "4624", "record_id": "{{ record_id }}", "logon": { "type": "Network" }, "event_data": { "TargetUserName": "{{ username }}", "TargetDomainName": "{{ params.domain }}", "LogonType": "3", "IpAddress": "{{ src_ip }}" } } } {%- do shared.set('record_id', record_id + 1) -%} ``` Configure the generator [#configure-the-generator] [`mode: chance`](/docs/plugins/event/template/modes#chance) mixes the three templates by weight: successful logons dominate at 87, failed attempts sit at 10, and the rare successful guess at 3 — roughly one attack-shaped event in ten, and about one in thirty of those a completed compromise. A bounded [`static`](/docs/plugins/input/static) input produces a fixed, rerunnable batch instead of an open-ended stream, and a [`file`](/docs/plugins/output/file) output writes one JSON object per line: ```yaml title="detection-lab/generator.yml" input: - static: count: 3000 event: template: mode: chance params: domain: CONTOSO templates: - benign_logon: template: templates/benign-logon.jinja chance: 87 - brute_force_attempt: template: templates/brute-force-attempt.jinja chance: 10 - brute_force_success: template: templates/brute-force-success.jinja chance: 3 output: - file: path: output/security.jsonl formatter: format: json ``` Run it [#run-it] [`eventum generate`](/docs/core/cli/eventum-generate) in [sample mode](/docs/core/concepts/generator#sample-mode) (`--live-mode false`) runs the whole batch immediately instead of pacing it against the clock: ```bash eventum generate \ --path detection-lab/generator.yml \ --id detection-lab \ --live-mode false ``` The batch writes in under a second. `static` assigns every event the same current timestamp — fine for this exercise, since the rule below groups events by source address rather than by time; the [realistic timing lesson](/docs/tutorials/realism/timing) covers spreading a stream like this across real time instead. A slice from the start of an actual run — ordinary, unrelated logons: ```json {"@timestamp": "2026-07-12T15:23:15.761004+00:00", "event": {"code": "4624", "action": "logged-in", "outcome": "success"}, "user": {"name": "tpatel", "domain": "CONTOSO"}, "source": {"ip": "200.12.121.73"}, "winlog": {"channel": "Security", "provider_name": "Microsoft-Windows-Security-Auditing", "event_id": "4624", "record_id": "1", "logon": {"type": "Network"}, "event_data": {"TargetUserName": "tpatel", "TargetDomainName": "CONTOSO", "LogonType": "3", "IpAddress": "200.12.121.73"}}} {"@timestamp": "2026-07-12T15:23:15.761004+00:00", "event": {"code": "4624", "action": "logged-in", "outcome": "success"}, "user": {"name": "jsmith", "domain": "CONTOSO"}, "source": {"ip": "192.147.22.147"}, "winlog": {"channel": "Security", "provider_name": "Microsoft-Windows-Security-Auditing", "event_id": "4624", "record_id": "2", "logon": {"type": "Network"}, "event_data": {"TargetUserName": "jsmith", "TargetDomainName": "CONTOSO", "LogonType": "3", "IpAddress": "192.147.22.147"}}} {"@timestamp": "2026-07-12T15:23:15.761004+00:00", "event": {"code": "4624", "action": "logged-in", "outcome": "success"}, "user": {"name": "ajohnson", "domain": "CONTOSO"}, "source": {"ip": "195.120.95.182"}, "winlog": {"channel": "Security", "provider_name": "Microsoft-Windows-Security-Auditing", "event_id": "4624", "record_id": "3", "logon": {"type": "Network"}, "event_data": {"TargetUserName": "ajohnson", "TargetDomainName": "CONTOSO", "LogonType": "3", "IpAddress": "195.120.95.182"}}} ``` Across the full 3,000-event batch, 302 events came out as EventID 4625 — close to the configured 1-in-10 weight — and 2,698 as 4624. Three source addresses account for nearly all of the 4625s, with 97 to 107 each. Every other address, drawn fresh per benign event, turns up far less often — usually once, occasionally a few times where the public-IP pool's small reserved range happens to collide. None comes anywhere near those three. Test detection rules against the stream [#test-detection-rules-against-the-stream] Sigma expresses "many events from one source within a time window" as a **correlation** — a separate document that references a base rule and adds the grouping, the time span, and the threshold, rather than folding an aggregation into the base rule's own condition. The rule below, built from the [Sigma correlation rules specification](https://github.com/SigmaHQ/sigma-specification/blob/main/specification/sigma-correlation-rules-specification.md), matches the pattern the generator above produces: five or more EventID 4625 events from the same address within 10 minutes, followed by an EventID 4624 from that same address: ```yaml title="detection-lab/mr_brute_force_logon.yml" title: Multiple failed logons followed by a successful logon id: c5e49499-ab06-43a7-b423-2783681d1945 status: experimental description: Detects repeated failed logons from one source followed by a successful logon from the same source correlation: type: temporal_ordered rules: - failed_logon_burst - successful_logon group-by: - IpAddress timespan: 10m falsepositives: - Bulk password resets or account migrations that briefly fail before succeeding - A misconfigured service retrying stale credentials level: high tags: - attack.credential_access - attack.t1110 --- title: Five or more failed logons from one source id: b86a5346-d45f-4bbf-9c3d-b47da5137e34 name: failed_logon_burst description: Detects five or more failed logon attempts from the same source address within 10 minutes correlation: type: event_count rules: - failed_logon group-by: - IpAddress timespan: 10m condition: gte: 5 --- title: Failed logon id: 9668233f-0dd6-4827-8a35-24079b398ae0 name: failed_logon logsource: product: windows service: security detection: selection: EventID: 4625 condition: selection --- title: Successful logon id: 9b5fc948-abcf-49f2-b5ee-7c0d0149bdb3 name: successful_logon logsource: product: windows service: security detection: selection: EventID: 4624 condition: selection ``` The two innermost documents are ordinary Sigma rules — one matches any EventID 4625, the other any EventID 4624 — and say nothing about volume or timing on their own. `failed_logon_burst` correlates the first one: it counts EventID 4625 events grouped by `IpAddress` and fires once a single address crosses 5 within a 10-minute span. The outermost rule chains that burst to a successful logon from the *same* address within the same window — the two-stage pattern the templates above generate, expressed as detection logic instead of Jinja. Filtering the generated stream for one of the three attacker addresses shows exactly the shape the rule is looking for — five failures against different targeted accounts, then a success: ```bash grep '"IpAddress": "178.161.163.143"' output/security.jsonl ``` ```json {"@timestamp": "2026-07-12T15:23:15.761004+00:00", "event": {"code": "4625", "action": "logon-failed", "outcome": "failure"}, "user": {"name": "administrator", "domain": "CONTOSO"}, "source": {"ip": "178.161.163.143"}, "winlog": {"channel": "Security", "provider_name": "Microsoft-Windows-Security-Auditing", "event_id": "4625", "record_id": "202", "event_data": {"TargetUserName": "administrator", "TargetDomainName": "CONTOSO", "Status": "0xC000006A", "IpAddress": "178.161.163.143"}}} {"@timestamp": "2026-07-12T15:23:15.761004+00:00", "event": {"code": "4625", "action": "logon-failed", "outcome": "failure"}, "user": {"name": "backup-admin", "domain": "CONTOSO"}, "source": {"ip": "178.161.163.143"}, "winlog": {"channel": "Security", "provider_name": "Microsoft-Windows-Security-Auditing", "event_id": "4625", "record_id": "234", "event_data": {"TargetUserName": "backup-admin", "TargetDomainName": "CONTOSO", "Status": "0xC000006A", "IpAddress": "178.161.163.143"}}} {"@timestamp": "2026-07-12T15:23:15.761004+00:00", "event": {"code": "4625", "action": "logon-failed", "outcome": "failure"}, "user": {"name": "administrator", "domain": "CONTOSO"}, "source": {"ip": "178.161.163.143"}, "winlog": {"channel": "Security", "provider_name": "Microsoft-Windows-Security-Auditing", "event_id": "4625", "record_id": "236", "event_data": {"TargetUserName": "administrator", "TargetDomainName": "CONTOSO", "Status": "0xC000006A", "IpAddress": "178.161.163.143"}}} {"@timestamp": "2026-07-12T15:23:15.761004+00:00", "event": {"code": "4625", "action": "logon-failed", "outcome": "failure"}, "user": {"name": "svc-sql", "domain": "CONTOSO"}, "source": {"ip": "178.161.163.143"}, "winlog": {"channel": "Security", "provider_name": "Microsoft-Windows-Security-Auditing", "event_id": "4625", "record_id": "269", "event_data": {"TargetUserName": "svc-sql", "TargetDomainName": "CONTOSO", "Status": "0xC000006A", "IpAddress": "178.161.163.143"}}} {"@timestamp": "2026-07-12T15:23:15.761004+00:00", "event": {"code": "4625", "action": "logon-failed", "outcome": "failure"}, "user": {"name": "admin", "domain": "CONTOSO"}, "source": {"ip": "178.161.163.143"}, "winlog": {"channel": "Security", "provider_name": "Microsoft-Windows-Security-Auditing", "event_id": "4625", "record_id": "281", "event_data": {"TargetUserName": "admin", "TargetDomainName": "CONTOSO", "Status": "0xC000006A", "IpAddress": "178.161.163.143"}}} {"@timestamp": "2026-07-12T15:23:15.761004+00:00", "event": {"code": "4624", "action": "logged-in", "outcome": "success"}, "user": {"name": "svc-sql", "domain": "CONTOSO"}, "source": {"ip": "178.161.163.143"}, "winlog": {"channel": "Security", "provider_name": "Microsoft-Windows-Security-Auditing", "event_id": "4624", "record_id": "284", "logon": {"type": "Network"}, "event_data": {"TargetUserName": "svc-sql", "TargetDomainName": "CONTOSO", "LogonType": "3", "IpAddress": "178.161.163.143"}}} ``` Running the same filter against any of the other benign source addresses in this batch turns up only a few matches — under the rule's threshold of 5, and nowhere near the attacker pool's hundred-plus, which is exactly why that threshold separates the two without also catching ordinary coincidental reuse. Sigma rule testing across formats [#sigma-rule-testing-across-formats] The same brute-force shape generalizes past Windows Security. The [OCSF format lesson](/docs/tutorials/formats/ocsf) generates Authentication-class events (`class_uid` 3002) for the same kind of logon: `activity_id: 1` ("Logon") on every attempt, with `status_id`/`status` distinguishing a failure from a success — the identical pair of facts a Sigma rule targeting an OCSF-normalized backend would group and count, just named `status_id` and `src_endpoint.ip` instead of `EventID` and `IpAddress`. Both examples even land on the same `logon_type_id`/`LogonType` value (`3`, "Network") for the same underlying reason: this is a remote authentication attempt either way, whatever the schema calls the fields that describe it. The technique-to-telemetry mapping this lesson builds on carries over to any other Sigma rule with a `logsource` and a `condition` — a Sysmon process-ancestry rule, a DNS-tunneling detection, an unusual PowerShell command line. Decide what the rule's `detection` block is matching, then shape a template's fields — and, where volume matters, a `shared`-backed pool like the one above — to produce exactly that. Going further [#going-further] * Widen the benign side — more usernames, mixed logon types, a second protocol — so the contrast with the attacker pool stays realistic at higher volumes. * Replace the `static` input with [`time_patterns`](/docs/tutorials/realism/timing) to spread the burst across real minutes instead of one instant, closer to how a correlation rule's `timespan` behaves against production traffic. * Point a third template at a different Sigma-mapped pattern — process-ancestry detections over Sysmon telemetry are a natural next target, and the [windows-sysmon](/hub/windows-sysmon) Hub generator is a ready-made source of that event shape. What's next [#whats-next] FAQ [#faq] No. The templates above reproduce the event IDs, field values, and repetition a technique leaves in a log — the same evidence a real Brute Force attempt would leave in Windows Security — without executing a credential-guessing tool, touching a real account, or running any code resembling the technique itself. A rule that fires on this telemetry fires on the pattern; whether that pattern originated from a real attacker or from `brute-force-attempt.jinja` is not something the rule, or the SIEM behind it, can tell. Start from the technique's own description of what it leaves behind, then work backward to the fields that would carry it. [T1110 Brute Force](https://attack.mitre.org/techniques/T1110/) leaves repeated authentication failures from a limited set of sources against a limited set of accounts, which is why the rule above groups by `IpAddress` and counts `EventID: 4625`. The right sub-technique tag depends on the exact pattern: `attack.t1110.001` (Password Guessing) is many passwords against one known account, `attack.t1110.003` (Password Spraying) is few attempts spread across many accounts to avoid lockouts. The generator above reproduces neither precisely — one source, several accounts, many attempts against each — so the rule tags the parent `attack.t1110` rather than a sub-technique that doesn't quite fit. Whatever ratio the rule is meant to survive. The `chance` weights above (87/10/3) put roughly one attack-shaped event in ten in the stream, concentrated onto three addresses — enough to cross the rule's threshold of 5 within a 3,000-event batch. Lowering the attack templates' `chance` values relative to the benign one pushes that ratio down, but the batch (or the running time, under `time_patterns`) needs to grow to match — what makes the pool detectable is the absolute count each address accumulates within the rule's `timespan`, not the percentage of the stream it represents. Related [#related] * The [Windows Event Log & Sysmon format lesson](/docs/tutorials/formats/windows-event-log) for the EventID model and JSON field shape behind these templates * The [OCSF format lesson](/docs/tutorials/formats/ocsf) for generating the same brute-force pattern as an Authentication-class event * The [SIEM test data lesson](/docs/tutorials/siem-events) for modeling stateful Windows Security sessions rather than a single detection pattern * The [test data pipeline lesson](/docs/tutorials/test-data-pipeline) for delivering this telemetry to a real SIEM and confirming receipt * The [Alert simulation: scheduled Telegram alerts](/docs/tutorials/telegram-alerts) tutorial for delivering a matching rule's output as a real alert notification, instead of a line grepped out of a file * The [Scenarios track](/docs/tutorials/scenarios) for the other use cases synthetic test data covers * The [windows-security](/hub/windows-security) and [windows-sysmon](/hub/windows-sysmon) generators in the Eventum Hub for production-tuned sources of this telemetry # Learn
Eventum Learn
Learn the formats your data uses, how to make it realistic, how to deliver it to your stack, and how to prove a pipeline works before touching production.
*** Foundations [#foundations] Start here: what synthetic event and log data is, and why structure matters. Formats & schemas [#formats--schemas] Understand the shape of your data — Windows events, CEF, LEEF, syslog, NDJSON, OCSF, ECS, Apache/Nginx access logs, AWS CloudTrail, Suricata EVE JSON, Linux auditd — and generate a compliant sample of each. Realism [#realism] Techniques that turn a flat stream into data that behaves like production. Delivery [#delivery] Stream synthetic data into the backend you actually use. Scenarios [#scenarios] End-to-end projects that prove a use case from an empty directory to working output. # IoT Test Data: Synthetic Sensor Telemetry A device fleet does not ship until its readings have been tested against something — a dashboard's thresholds, a time-series pipeline's downsampling, an alert that should fire when a value drifts out of range. A flat random number won't exercise any of it: a real thermometer's next reading starts from wherever the last one left off and moves a small step, not a fresh roll every time. Sensor telemetry is only believable when consecutive readings relate to each other. The generator below drifts three sensors — temperature, humidity, and pressure — each within a realistic range, and prints every reading as [NDJSON](/docs/tutorials/formats/ndjson) to standard output. One [`eventum generate`](/docs/core/cli/eventum-generate) command runs it — no `eventum.yml`, `startup.yml`, or backend to stand up first. Sensor telemetry test data: drift instead of flat random [#sensor-telemetry-test-data-drift-instead-of-flat-random] A sensor reading is a **metric** in the sense [Logs vs metrics vs events](/docs/tutorials/foundations/logs-vs-metrics-events) uses the word — a numeric measurement sampled at an instant, without a log's descriptive sentence or a structured event's discrete triggering action. `temperature`, `humidity`, and `pressure` below are exactly that: a name, a value, a unit, a timestamp, and nothing else. What separates a believable stream of those readings from an obviously generated one is whether consecutive readings relate to each other. Picking each one fresh from a realistic range — `module.rand.number.gauss(22.0, 5.0)` for temperature, say — keeps every single reading plausible on its own, and still lets two readings five seconds apart land 15 degrees apart, something no real thermometer does. A real sensor's next reading starts from wherever the last one left off and moves a small step from there. The generator below reads each sensor's last value back out of [state](/docs/core/concepts/producing#state-management) before rendering the next one, nudges it by a small step, and clamps the result to the range a real sensor can physically report — the same drift pattern introduced in that lesson for a single CPU-usage metric, applied here across three independent sensors. Sizing that step is itself a distribution choice: [Generate realistic fake data](/docs/tutorials/realism/values) names `gauss` as the fit for exactly this kind of field — a reading that varies around a baseline — so each step below is drawn from a Gaussian centered on zero instead of spread evenly across a fixed range, keeping most ticks small and only rarely letting one run larger, the way real sensor noise actually clusters. What you'll build [#what-youll-build] The generator uses: * [**timer**](/docs/plugins/input/timer) input — emits a timestamp every 5 seconds. * [**spin**](/docs/core/concepts/producing#picking-modes) picking mode — cycles through three sensor templates round-robin. * [**locals state**](/docs/core/concepts/producing#state-management) — each sensor's last value persists between renders, nudged by a Gaussian step and clamped to a realistic range (see [Generate realistic fake data](/docs/tutorials/realism/values)). * [**stdout output**](/docs/plugins/output/stdout) with `json` formatter — validates each event and writes it as [NDJSON](/docs/tutorials/formats/ndjson), one compact JSON object per line. Prerequisites [#prerequisites] * [Eventum installed](/docs/core/introduction/installation) Project structure [#project-structure] Build it [#build-it] Create the project directory [#create-the-project-directory] ```bash mkdir -p iot-sensors/templates cd iot-sensors ``` Write the sensor templates [#write-the-sensor-templates] Each template simulates a different sensor type, using the drift technique above: read the sensor's last value from `locals`, nudge it with a Gaussian step, clamp the result to a realistic range, and write the new value back before rendering. **Temperature** — drifts around 22°C, nudged by a Gaussian step (σ = 0.1) each reading. ```jinja title="templates/temperature.jinja" {% set prev = locals.get("value", 22.0) %} {% set delta = module.rand.number.gauss(0, 0.1) %} {% set value = module.rand.number.clamp(prev + delta, 15.0, 35.0) %} {% do locals.set("value", value) %} { "sensor_id": "sensor-temp-01", "metric": "temperature", "value": {{ "%.2f" | format(value) }}, "unit": "celsius", "timestamp": "{{ timestamp.isoformat() }}" } ``` `module.rand.number.clamp(value, 15.0, 35.0)` keeps the result between 15°C and 35°C regardless of how a run of same-direction steps would otherwise add up, preventing unrealistic drift over long runs. **Humidity** — drifts around 55%, nudged by a Gaussian step (σ = 0.5) each reading. ```jinja title="templates/humidity.jinja" {% set prev = locals.get("value", 55.0) %} {% set delta = module.rand.number.gauss(0, 0.5) %} {% set value = module.rand.number.clamp(prev + delta, 20.0, 90.0) %} {% do locals.set("value", value) %} { "sensor_id": "sensor-hum-01", "metric": "humidity", "value": {{ "%.1f" | format(value) }}, "unit": "percent", "timestamp": "{{ timestamp.isoformat() }}" } ``` **Pressure** — drifts around 1013 hPa, nudged by a Gaussian step (σ = 0.17) each reading. ```jinja title="templates/pressure.jinja" {% set prev = locals.get("value", 1013.0) %} {% set delta = module.rand.number.gauss(0, 0.17) %} {% set value = module.rand.number.clamp(prev + delta, 990.0, 1040.0) %} {% do locals.set("value", value) %} { "sensor_id": "sensor-pres-01", "metric": "pressure", "value": {{ "%.1f" | format(value) }}, "unit": "hPa", "timestamp": "{{ timestamp.isoformat() }}" } ``` Each sensor template has its own `locals` — the temperature drift is independent of humidity drift. The value persists between calls, creating smooth, correlated time-series data. Configure the generator [#configure-the-generator] The `timer` input emits a timestamp every 5 seconds; with `spin` mode cycling through three templates, each sensor reports once every 15 seconds — one full cycle through temperature, humidity, and pressure every three ticks. ```yaml title="generator.yml" input: - timer: seconds: 5 count: 1 event: template: mode: spin templates: - temperature: template: templates/temperature.jinja - humidity: template: templates/humidity.jinja - pressure: template: templates/pressure.jinja output: - stdout: formatter: format: json ``` The `json` formatter validates each event and writes it on a single line — the [NDJSON](/docs/tutorials/formats/ndjson) shape. Run it [#run-it] No `eventum.yml` or `startup.yml` needed — [`eventum generate`](/docs/core/cli/eventum-generate) runs a single generator directly: ```bash eventum generate --path generator.yml --id sensors ``` The readings below were captured by running this generator in [sample mode](/docs/core/concepts/generator#sample-mode) (`--live-mode false`) with a temporary `repeat` bound on the `timer` input, so the run completed immediately instead of pacing to the clock. The command above drops that bound and runs in [live mode](/docs/core/concepts/generator#live-mode-default) instead, so readings arrive every 5 seconds in real time rather than all at once. Readings stream out, cycling through sensors: ```json {"sensor_id": "sensor-temp-01", "metric": "temperature", "value": 22.15, "unit": "celsius", "timestamp": "2026-07-18T12:38:09.150579+00:00"} {"sensor_id": "sensor-hum-01", "metric": "humidity", "value": 54.9, "unit": "percent", "timestamp": "2026-07-18T12:38:14.150579+00:00"} {"sensor_id": "sensor-pres-01", "metric": "pressure", "value": 1012.9, "unit": "hPa", "timestamp": "2026-07-18T12:38:19.150579+00:00"} {"sensor_id": "sensor-temp-01", "metric": "temperature", "value": 22.08, "unit": "celsius", "timestamp": "2026-07-18T12:38:24.150579+00:00"} {"sensor_id": "sensor-hum-01", "metric": "humidity", "value": 55.3, "unit": "percent", "timestamp": "2026-07-18T12:38:29.150579+00:00"} ``` Temperature drifts from 22.15 to 22.08 across two readings fifteen seconds apart; humidity from 54.9 to 55.3 over the same span — small steps in both directions, never a jump. Across the full run, every reading from all three sensors stayed inside its clamp range. Generate a finite dataset [#generate-a-finite-dataset] This generator's `timer` has no `repeat` set, so in live mode it never stops on its own — the same unbounded case [Streaming vs bulk](/docs/tutorials/foundations/streaming-vs-bulk) describes for a `cron` tick with no `end`: streaming, by design, for as long as the process keeps running. Turning it into a finite dataset takes the same two things that lesson names for any input — a bound, and sample mode to skip the wait: ```yaml title="generator.yml" input: - timer: seconds: 5 count: 1 repeat: 900 ``` ```bash eventum generate --path generator.yml --id sensors-batch --live-mode false > dataset.ndjson ``` `repeat: 900` stops the timer after 900 ticks — 300 full cycles through the three sensors in `spin` order, 900 readings total — and `--live-mode false` produces all of them immediately instead of waiting out the 5-second interval between each one. Pipe to other tools [#pipe-to-other-tools] The NDJSON output works directly with standard Unix tools: ```bash # Filter by sensor type eventum generate --path generator.yml --id sensors | jq 'select(.metric == "temperature")' # Write the live stream to a file (Ctrl+C to stop) eventum generate --path generator.yml --id sensors > readings.ndjson # Count readings per sensor so far (run for a while, then Ctrl+C) eventum generate --path generator.yml --id sensors | jq -r '.sensor_id' | sort | uniq -c ``` Going further [#going-further] * **Anomaly injection** — use `module.rand.chance(0.02)` to occasionally spike a value far outside its normal range, simulating a sensor fault. * **Multiple sensor sites** — run this generator several times with a different `--id` and a distinct `site` value baked into each template, simulating a fleet of installations instead of one. * **A real backend instead of standard output** — swap the `stdout` output for [kafka](/docs/plugins/output/kafka), [opensearch](/docs/plugins/output/opensearch), or any other output plugin; see [Stream synthetic data to your stack](/docs/tutorials/delivery) for wiring a generator directly to the systems you run. * **Historical dataset** — replace `timer` with [linspace](/docs/plugins/input/linspace) — `start`/`end` spanning a year, `count` set so consecutive timestamps land five minutes apart — to generate a full year of readings in sample mode. * **Alert thresholds** — swap `locals` for `globals` state, visible across every generator in the process, so a second generator can watch each sensor's latest value and produce alerts when it crosses a threshold. What's next [#whats-next] FAQ [#faq] A value generated fresh every time — even a realistic one — has no relationship to the reading before it, so a chart of the stream can jump anywhere in the plausible range from one tick to the next, which no physical sensor does. Reading the last value out of `locals` and nudging it by a small step, as every template here does, keeps consecutive readings close together the way a real measurement actually behaves. [Logs vs metrics vs events](/docs/tutorials/foundations/logs-vs-metrics-events) covers the metric shape this generator produces, and [Generate realistic fake data](/docs/tutorials/realism/values) covers choosing a step's distribution — Gaussian here — instead of a flat range. No. `module.rand.number.clamp(value, min, max)` runs on every render after the Gaussian step is added, so the value stored in `locals` can never leave the configured range no matter how many same-direction steps happen to land in a row. Widening or narrowing the range for a sensor only changes the two numbers passed to `clamp` — the drift and clamp mechanics stay the same. Bound the `timer` input with `repeat` and run with `--live-mode false`, exactly as [Streaming vs bulk](/docs/tutorials/foundations/streaming-vs-bulk) describes for any input: a `timer` with no `repeat` keeps producing for as long as the process runs, and sample mode alone doesn't stop an input that never stops on its own — both the bound and the mode are needed together. Yes. Add another template alias — `temperature_2`, say — with its own `sensor_id` and starting baseline; `locals` is scoped per template alias, so the new sensor drifts independently of `sensor-temp-01` without any extra configuration. Add it to `mode: spin`'s `templates` list and the cycle simply grows by one. Related [#related] * The [Logs vs metrics vs events](/docs/tutorials/foundations/logs-vs-metrics-events) lesson for the metric shape this generator produces, and the single-sensor drift example this tutorial builds into a full three-sensor generator * The [Generate realistic fake data](/docs/tutorials/realism/values) lesson for Gaussian and other skewed distributions, and clamping a value to a realistic range * The [Streaming vs bulk](/docs/tutorials/foundations/streaming-vs-bulk) lesson for live mode's continuous feed versus a bounded sample-mode batch, the general rule behind this generator's finite-dataset step * The [Stream synthetic data to your stack](/docs/tutorials/delivery) pillar for delivering these readings to a real backend instead of standard output * The [NDJSON format](/docs/tutorials/formats/ndjson) lesson for the newline-delimited JSON shape this generator writes * The [Scenarios track](/docs/tutorials/scenarios) for the wider picture of applied synthetic data * Ready-made generators in the [Eventum Hub](/hub) # Generate Realistic Load Test Data for an API Load test data is the request stream a load test actually needs: a realistic mix of operations, identifiers, and body shapes, released at whatever rate the test calls for — never the same payload sent as fast as possible. A load test built on the wrong kind of data measures the wrong thing — how fast a target answers the same cached request a thousand times, not how it holds up against the shape of traffic a real deployment receives. Build a generator that produces that stream: a burst of diverse API requests fired at a target endpoint as fast as possible, to benchmark throughput, uncover race conditions, and validate rate limiters — all without writing test scripts or capturing real traffic first. Generating realistic load test data [#generating-realistic-load-test-data] Replaying one fixed request in a `curl` loop is the obvious way to put load on an endpoint, and it does generate traffic — but every iteration hits the same cache entry, the same validator branch, the same row in a database, because every iteration sends the exact same body. A target under that kind of load looks fast for the wrong reason: it keeps answering the same question, not the varied mix a real deployment actually receives. Realistic load test data breaks that pattern on three axes: **operation mix** — a spread of reads and writes instead of one endpoint hammered in isolation; **identifiers** — a different user, record, or resource on every request, so caches see a realistic mix of hits and misses and lookups touch more than one row; and **payload shape** — field values drawn from [distributions and weighted choices](/docs/tutorials/realism/values) instead of one constant body, so a validator's optional-field branches, length checks, and type coercions all get exercised, not just the one path a fixed payload happens to take. None of this needs a captured production sample. `module.rand` and `module.faker` generate a fresh, plausible value on every render, and `weighted_choice` shapes the mix of operations and values to match what a real deployment's traffic actually looks like — the same techniques the [Realistic values](/docs/tutorials/realism/values) lesson covers for log events, applied here to API requests instead. Load testing with diverse payloads [#load-testing-with-diverse-payloads] The build below turns that theory into a concrete pattern: four templates, one per CRUD operation, mixed by [chance](/docs/plugins/event/template/modes#chance) weights that approximate a typical web application's read-heavy traffic — reads dominate, creates and updates follow, deletes stay rare. Every template draws its own identifiers and field values fresh per request, so the 5,000 requests in one run touch different pages, different records, different names and roles — not one payload sent 5,000 times. That volume of realistic requests answers a different question than a [pipeline correctness test](/docs/tutorials/test-data-pipeline) does. Throughput testing — this lesson — asks how much traffic a target can sustain before latency climbs or requests start failing; pipeline testing asks whether one realistic event survives a specific transport and lands intact at the other end. A target can pass one and fail the other: an endpoint that holds up under thousands of requests a second can still silently reject a payload shape a correctness check would have caught, and a pipeline that handles a single well-formed event perfectly can still fall over the moment real concurrency hits it. What you'll build [#what-youll-build] The generator uses: * [**static**](/docs/plugins/input/static) input — produces a burst of N timestamps instantly. * [**Sample mode**](/docs/core/concepts/generator#sample-mode) (`--live-mode false`) — releases all timestamps without waiting for the clock. * [**chance**](/docs/plugins/event/template/modes#chance) picking mode — weighted mix of request types (GET, POST, PUT, DELETE). * [**HTTP output**](/docs/plugins/output/http) — sends each event as a request to the target API; see [Send test data to an API endpoint](/docs/tutorials/delivery/http) for batching and delivery mechanics in depth. * **Batch tuning** — controls how many requests are in flight at once. Prerequisites [#prerequisites] * [Eventum installed](/docs/core/introduction/installation) * A target API endpoint to test (this tutorial uses a placeholder URL — replace it with your own) Only load-test APIs you own or have explicit permission to test. Sending high-volume traffic to third-party services may violate their terms of service. Project structure [#project-structure] Build it [#build-it] Create the project directory [#create-the-project-directory] ```bash mkdir -p load-test/templates cd load-test ``` Write request templates [#write-request-templates] Each template produces a JSON request body. The [HTTP output](/docs/plugins/output/http) sends it as a POST to the target endpoint. The template includes the HTTP method and path so the API can route accordingly (adjust the format to match your API's expectations). **GET users** — list endpoint, no body needed. `page` is drawn from a clamped [exponential distribution](/docs/tutorials/realism/values) instead of a flat range, so most requests land on the first few pages and only a thin tail paginates deep — the way real API clients actually browse a list; `limit` is a weighted pick favoring the default page size over larger ones. ```jinja title="templates/get-users.jinja" { "method": "GET", "path": "/api/v1/users", "params": { "page": {{ module.rand.number.clamp(module.rand.number.exponential(1 / 5), 1, 100) | round | int }}, "limit": {{ module.rand.weighted_choice({10: 60, 25: 30, 50: 10}) }} } } ``` **Create user** — POST with a realistic payload. ```jinja title="templates/create-user.jinja" { "method": "POST", "path": "/api/v1/users", "body": { "name": "{{ module.faker.locale.en.name() }}", "email": "{{ module.faker.locale.en.email() }}", "role": "{{ module.rand.weighted_choice(["viewer", "editor", "admin"], [0.6, 0.3, 0.1]) }}", "department": "{{ module.rand.choice(["engineering", "marketing", "sales", "support", "hr"]) }}" } } ``` **Update user** — PUT with partial changes. ```jinja title="templates/update-user.jinja" { "method": "PUT", "path": "/api/v1/users/{{ module.rand.number.integer(1, 10000) }}", "body": { "name": "{{ module.faker.locale.en.name() }}", "role": "{{ module.rand.choice(["viewer", "editor", "admin"]) }}" } } ``` **Delete user** — DELETE by ID. ```jinja title="templates/delete-user.jinja" { "method": "DELETE", "path": "/api/v1/users/{{ module.rand.number.integer(1, 10000) }}" } ``` Every template above draws its identifiers, names, and roles fresh per render, so the batch below produces a genuinely varied stream instead of one fixture repeated 5,000 times. Configure the generator [#configure-the-generator] The `static` input generates 5,000 timestamps at once. The `chance` mode distributes requests across the four types with a realistic CRUD mix. ```yaml title="generator.yml" input: - static: count: 5000 event: template: mode: chance templates: - get-users: template: templates/get-users.jinja chance: 0.50 - create-user: template: templates/create-user.jinja chance: 0.25 - update-user: template: templates/update-user.jinja chance: 0.15 - delete-user: template: templates/delete-user.jinja chance: 0.10 output: - http: url: "http://localhost:8080/api/v1/batch" method: POST success_code: 200 headers: Content-Type: "application/json" Authorization: "Bearer test-token" formatter: format: json-batch ``` Key settings: * **static count: 5000** — the total number of requests to generate. * **chance distribution** — 50% reads, 25% creates, 15% updates, 10% deletes — mimics a typical web application workload. * **chance values are relative weights** — 0.50/0.25/0.15/0.10 need not sum to 1; only the ratio between them matters (see [chance picking mode](/docs/plugins/event/template/modes#chance)). * **json-batch formatter** — groups events into a JSON array per batch, reducing HTTP round-trips. Run it [#run-it] Use [`eventum generate`](/docs/core/cli/eventum-generate) in sample mode with tuned batch settings: ```bash eventum generate \ --path generator.yml \ --id load-test \ --live-mode false \ --batch.size 100 \ --max-concurrency 10 ``` Flags explained: | Flag | Effect | | ---------------------- | ------------------------------------------------------------------------- | | `--live-mode false` | Releases all 5,000 timestamps instantly — maximum throughput | | `--batch.size 100` | Groups 100 events per batch before sending | | `--max-concurrency 10` | Up to 10 concurrent requests in flight at once — one per batch, see below | `--max-concurrency` limits [concurrent write operations across all output plugins](/docs/core/concepts/output#concurrency) — here, that's the `http` plugin's batch writes. The `json-batch` formatter configured above turns each batch into a single HTTP request carrying the whole batch as one JSON array (see [batch vs per-request delivery](/docs/tutorials/delivery/http#batch-vs-per-request-delivery)). With `json-batch`, one write is one request, so `--max-concurrency 10` caps the run at 10 requests in flight — not 10 batches' worth of per-event requests. The 5,000 events split into 50 batches of 100, with up to 10 of those batches' requests in flight at once. Swap the formatter to plain `json` and each batch fans out into one request per event instead, multiplying `--batch.size` by `--max-concurrency` into the real ceiling (1,000 here). Adjust `--batch.size` and `--max-concurrency` together to find your API's breaking point without overshooting it by accident. Monitor results [#monitor-results] `eventum generate` runs a single generator with no application server behind it, so there's nothing to query while it runs — and it prints no summary when it exits, even at the highest verbosity (`-vvvvv`). What you can check depends on what you're looking for: **Payload diversity, before pointing at a real target.** Swap the `http` block for `stdout` (keep the same `json-batch` formatter) and the exact batches that would have been sent print instead — useful for confirming the CRUD mix and field values look right before a single request reaches the target API. No test API to point at yet? The `http` output creates its client lazily and never tests `url` at startup, so an unreachable endpoint doesn't stop the run. Every write just fails in the background, logged rather than raised, and at the default verbosity that happens quietly — pass `-v` if you want it to show. The `stdout` swap above works whether or not `url` resolves to anything real. **Delivery counters, against a real target.** A one-shot `eventum generate` run has no running application to expose these from, so this needs the full app instead: list the generator in [startup.yml](/docs/core/config/startup-yml) and start it with [`eventum run`](/docs/core/cli/eventum-run). Once it's running, the same counters [every output plugin tracks](/docs/core/concepts/output#error-tracking) become available live: ```bash curl -u eventum:eventum http://localhost:9474/api/generators/load-test/stats ``` | Counter | What it tells you | | --------------- | ----------------------------------------------------------- | | `written` | Requests the target accepted | | `write_failed` | Write attempts that failed outright | | `format_failed` | Events that failed to format before a request was even sent | See the [Get Generator Stats](/docs/api/generators/id/stats/get) reference for the full response shape, or open Studio's [Instance metrics](/docs/studio/instances#instance-metrics) panel for the same numbers without a `curl` call. These counters aren't equally reliable for the `http` output used here. `write_failed` only increases when a write is cancelled outright — a [`write_timeout`](/docs/core/config/startup-yml#overridable-generation-parameters) expiring before a batch finishes sending, for example — because the plugin otherwise handles every response itself and never raises on a bad status code or a timed-out request. And because `json-batch` ships a whole batch as one request, a rejected or dropped batch leaves both `written` and `write_failed` unchanged — nothing marks the failure. Watch `written` against the number of events you actually sent (5,000 in this run) and treat any shortfall as failed requests. Run with `-v` so those failures show up in the logs instead of going unrecorded. Going further [#going-further] * **Ramp-up pattern** — replace `static` with [linspace](/docs/plugins/input/linspace) over a 5-minute window in live mode to gradually increase load. * **Error injection** — add a template that sends intentionally malformed requests (missing required fields, invalid types) to test API error handling. * **Multi-endpoint testing** — run several generators in parallel via `startup.yml`, each targeting a different API service. * **Throughput measurement** — pipe `stdout` output through `pv -l` to count events per second in real time. What's next [#whats-next] FAQ [#faq] These aren't interchangeable checks. This lesson measures capacity — how much request volume a target can sustain before latency climbs or requests start failing. [Testing the pipeline](/docs/tutorials/test-data-pipeline) measures correctness — whether one realistic event survives a specific transport and lands intact at the other end. A target can pass one check and fail the other, so neither replaces the other. Because that's what production traffic actually looks like, and because a single repeated request measures a narrow, often cached code path rather than a target's real capacity. The [chance](/docs/plugins/event/template/modes#chance) mix above spreads load across reads and writes the way a typical web application actually receives them; each template also draws its own identifiers and field values fresh per request, so caches, validators, and database lookups all see a realistic spread of inputs instead of the same one repeated 5,000 times. Related [#related] * The [Send test data to an API endpoint](/docs/tutorials/delivery/http) lesson for HTTP delivery, batching, and authentication in depth * The [Realistic values](/docs/tutorials/realism/values) lesson for the distributions and weighted choices behind a diverse payload * The [Test data pipeline](/docs/tutorials/test-data-pipeline) lesson for confirming correctness instead of capacity * The [Scenarios track](/docs/tutorials/scenarios) for how this scenario relates to the rest of the course * The [Eventum Hub](/hub) for pre-built generators covering common API traffic patterns # SIEM Test Data: Index Windows Security Events in OpenSearch A detection rule tuned against a handful of hand-written events, a SIEM parser wired to a new log source, an analyst who has never seen an incident play out — each needs the same thing before anyone trusts it: security telemetry that behaves like a real environment's. Logons, privilege use, and process activity, in realistic sequences and volume, with no live attack and no production system anywhere in the test. The generator below produces Windows Security Event Log sessions — logon, privilege assignment, process creation, and logoff — with realistic timing and user behavior, indexed into OpenSearch and ready for dashboards and detection rules. **No OpenSearch?** Replace the `opensearch` output with `stdout: {}` in the generator config. Everything else stays the same. What you'll build [#what-youll-build] The generator uses a **finite state machine** (FSM) to model user sessions. Each session follows a fixed flow: 1. **Logon** (Event ID 4624) — a user logs in, session context is established. 2. **Privilege assignment** (Event ID 4672) — special privileges are assigned. 3. **Process creation** (Event ID 4688) — the user launches several processes. The FSM loops on this state. 4. **Logoff** (Event ID 4634) — the session ends, and the cycle restarts. A [time-patterns](/docs/plugins/input/time-patterns) input controls the arrival rate, producing timestamps at natural intervals. Each timestamp advances the FSM by one step. Prerequisites [#prerequisites] * [Eventum installed](/docs/core/introduction/installation) * An OpenSearch instance accessible over HTTP(S) *(optional — stdout works for local testing)* Project structure [#project-structure] Build it [#build-it] Create the project directory [#create-the-project-directory] ```bash mkdir -p eventum/generators/winevents/{patterns,templates,data} cd eventum ``` Define the traffic pattern [#define-the-traffic-pattern] The [time-patterns](/docs/plugins/input/time-patterns) plugin generates timestamps using a statistical model. The pattern file has four components: * **Oscillator** — the repeating time window (every 10 seconds). * **Multiplier** — how many timestamps per period (7 ≈ one full session). * **Randomizer** — adds ±20% variance to the count. * **Spreader** — distributes timestamps within each period (uniform = evenly spaced). ```yaml title="generators/winevents/patterns/traffic.yml" label: Security events oscillator: start: "now" end: "never" period: 10 unit: seconds multiplier: ratio: 7 randomizer: deviation: 0.2 direction: mixed spreader: distribution: uniform parameters: low: 0.0 high: 1.0 ``` This produces roughly 7 timestamps every 10 seconds — enough to complete one user session per cycle. Create sample data [#create-sample-data] The user pool is a JSON file loaded as a [sample](/docs/plugins/event/template/samples). Each session randomly picks a user from this list. JSON sample rows support named access via object keys — `user.name`, `user.domain`. ```json title="generators/winevents/data/users.json" [ { "name": "jsmith", "domain": "CORP" }, { "name": "ajohnson", "domain": "CORP" }, { "name": "mwilliams", "domain": "CORP" }, { "name": "kbrown", "domain": "CORP" }, { "name": "admin", "domain": "CORP" }, { "name": "svc-backup", "domain": "CORP" } ] ``` Write the session templates [#write-the-session-templates] Each template produces one JSON event following [Elastic Common Schema](https://www.elastic.co/guide/en/ecs/current/index.html) field naming — a widely used format for SIEM ingestion. **Logon** — initializes the session. Picks a random user, generates a session ID, and stores everything in [`shared`](/docs/plugins/event/template/state) state so subsequent templates can access it. ```jinja title="generators/winevents/templates/logon.jinja" {% set user = module.rand.choice(samples.users) %} {% set session_id = module.rand.crypto.uuid4() %} {% set src_ip = module.faker.locale.en.ipv4_private() %} {% set workstation = "WS-" ~ module.rand.number.integer(1000, 9999) %} {% set logon_type = module.rand.weighted_choice(["Interactive", "Network", "RemoteInteractive"], [0.5, 0.3, 0.2]) %} {% do shared.set("session_id", session_id) %} {% do shared.set("username", user.name) %} {% do shared.set("domain", user.domain) %} {% do shared.set("src_ip", src_ip) %} {% do shared.set("workstation", workstation) %} {% do shared.set("process_count", 0) %} {% do shared.pop("logoff_ready", None) %} { "@timestamp": "{{ timestamp.isoformat() }}", "event": { "code": 4624, "action": "logged-in", "provider": "Microsoft-Windows-Security-Auditing" }, "host": { "name": "{{ workstation }}" }, "source": { "ip": "{{ src_ip }}" }, "user": { "name": "{{ user.name }}", "domain": "{{ user.domain }}" }, "winlog": { "logon": { "id": "{{ session_id }}", "type": "{{ logon_type }}" } }, "message": "An account was successfully logged on." } ``` **Privilege assignment** — follows logon, reads session context from `shared`. ```jinja title="generators/winevents/templates/privilege.jinja" { "@timestamp": "{{ timestamp.isoformat() }}", "event": { "code": 4672, "action": "assigned-special-privileges", "provider": "Microsoft-Windows-Security-Auditing" }, "host": { "name": "{{ shared.get('workstation') }}" }, "user": { "name": "{{ shared.get('username') }}", "domain": "{{ shared.get('domain') }}" }, "winlog": { "logon": { "id": "{{ shared.get('session_id') }}" } }, "message": "Special privileges assigned to new logon." } ``` **Process creation** — the FSM loops on this state. Each call increments a counter in `shared`; once it reaches 4, the template itself sets a flag that tells the FSM the session is ready for logoff. ```jinja title="generators/winevents/templates/process.jinja" {% set process_count = shared.get("process_count", 0) + 1 %} {% do shared.set("process_count", process_count) %} {% if process_count >= 4 %} {% do shared.set("logoff_ready", true) %} {% endif %} {% set proc = module.rand.choice(["cmd.exe", "powershell.exe", "notepad.exe", "chrome.exe", "svchost.exe", "taskmgr.exe"]) %} { "@timestamp": "{{ timestamp.isoformat() }}", "event": { "code": 4688, "action": "created-process", "provider": "Microsoft-Windows-Security-Auditing" }, "host": { "name": "{{ shared.get('workstation') }}" }, "user": { "name": "{{ shared.get('username') }}", "domain": "{{ shared.get('domain') }}" }, "process": { "name": "{{ proc }}", "executable": "C:\\Windows\\System32\\{{ proc }}", "pid": {{ module.rand.number.integer(1000, 65535) }} }, "winlog": { "logon": { "id": "{{ shared.get('session_id') }}" } }, "message": "A new process has been created." } ``` **Logoff** — ends the session. The FSM transitions back to logon, starting a new cycle. ```jinja title="generators/winevents/templates/logoff.jinja" { "@timestamp": "{{ timestamp.isoformat() }}", "event": { "code": 4634, "action": "logged-off", "provider": "Microsoft-Windows-Security-Auditing" }, "host": { "name": "{{ shared.get('workstation') }}" }, "user": { "name": "{{ shared.get('username') }}", "domain": "{{ shared.get('domain') }}" }, "winlog": { "logon": { "id": "{{ shared.get('session_id') }}" } }, "message": "An account was logged off." } ``` The `logon` template writes session context into `shared` state, and every later template reads from it — the same user, workstation, and session ID appear across every event in the session. Configure the generator [#configure-the-generator] The generator config wires the pipeline together. The [FSM](/docs/core/concepts/producing#picking-modes) `transitions` define the session flow: ```yaml title="generators/winevents/generator.yml" input: - time_patterns: patterns: - patterns/traffic.yml event: template: mode: fsm samples: users: type: json source: data/users.json templates: - logon: template: templates/logon.jinja initial: true transitions: - to: privilege when: { always: } - privilege: template: templates/privilege.jinja transitions: - to: process when: { always: } - process: template: templates/process.jinja transitions: - to: logoff when: { defined: shared.logoff_ready } - to: process when: { always: } - logoff: template: templates/logoff.jinja transitions: - to: logon when: { always: } output: - stdout: formatter: format: json - opensearch: hosts: - ${params.opensearch_host} username: ${params.opensearch_user} password: ${secrets.opensearch_password} index: winevents verify: false ``` Reading the transitions: | From | To | Condition | Meaning | | --------- | --------- | ------------------------------ | --------------------------------- | | logon | privilege | `always` | Every logon gets privileges | | privilege | process | `always` | Start creating processes | | process | logoff | `defined: shared.logoff_ready` | End the session after 4 processes | | process | process | `always` (fallback) | Keep creating processes | | logoff | logon | `always` | Start a new session | Transitions are evaluated **in order**, and the first one whose condition holds wins. On `process`, the `logoff_ready` check is listed before the `always` fallback — reversed, the fallback would fire first every time, since `always` never fails, and the FSM would never reach the flag check. Configure the application [#configure-the-application] ```yaml title="eventum.yml" server: host: "0.0.0.0" port: 9474 path: startup: /home/user/eventum/startup.yml generators_dir: /home/user/eventum/generators logs: /home/user/eventum/logs keyring_cryptfile: /home/user/eventum/cryptfile.cfg generation: timezone: UTC batch: size: 100 ``` All `path.*` values must be **absolute paths**. Adjust to match your actual project location. The startup file registers the generator and passes connection [parameters](/docs/core/config/parameters): ```yaml title="startup.yml" - id: winevents path: winevents/generator.yml params: opensearch_host: "https://localhost:9200" opensearch_user: admin ``` Replace the values with your actual OpenSearch connection details. Store the OpenSearch password [#store-the-opensearch-password] Use the [keyring](/docs/core/cli/eventum-keyring) to encrypt the password: ```bash eventum-keyring set opensearch_password ``` You'll be prompted for the keyring password and the secret value. The generator config references it as `${secrets.opensearch_password}`. Using stdout only? Skip this step and remove the `opensearch` block from `generator.yml`. Run it [#run-it] ```bash eventum run -c eventum.yml ``` JSON events stream to stdout — each one is a step in a user session: ```json {"@timestamp":"2025-06-15T14:23:01+00:00","event":{"code":4624,"action":"logged-in","provider":"Microsoft-Windows-Security-Auditing"},"host":{"name":"WS-4821"},"source":{"ip":"10.12.44.198"},"user":{"name":"jsmith","domain":"CORP"},"winlog":{"logon":{"id":"a1b2c3d4-e5f6-7890-abcd-ef1234567890","type":"Interactive"}},"message":"An account was successfully logged on."} {"@timestamp":"2025-06-15T14:23:02+00:00","event":{"code":4672,"action":"assigned-special-privileges", ...}} {"@timestamp":"2025-06-15T14:23:03+00:00","event":{"code":4688,"action":"created-process", ...},"process":{"name":"powershell.exe", ...}} {"@timestamp":"2025-06-15T14:23:05+00:00","event":{"code":4688,"action":"created-process", ...},"process":{"name":"chrome.exe", ...}} {"@timestamp":"2025-06-15T14:23:06+00:00","event":{"code":4688,"action":"created-process", ...},"process":{"name":"cmd.exe", ...}} {"@timestamp":"2025-06-15T14:23:07+00:00","event":{"code":4688,"action":"created-process", ...},"process":{"name":"svchost.exe", ...}} {"@timestamp":"2025-06-15T14:23:09+00:00","event":{"code":4634,"action":"logged-off", ...}} ``` A complete session: logon → privilege → 4 processes → logoff. Then a new session starts with a different user. Going further [#going-further] * **Scale up traffic** — increase `multiplier.ratio` in `traffic.yml` to 100+ and reduce the oscillator period for high-volume SIEM testing. * **Add failed logons** — create a `logon_failed.jinja` template for Event ID 4625 and add it as an FSM branch with a probability-based condition. * **Vary session length** — store a random target in `shared` during logon (`module.rand.number.integer(2, 8)`) and have the process template compare its counter against that target instead of the fixed 4, setting `logoff_ready` once it's reached. * **Multiple workstations** — add a second generator in `startup.yml` with different parameters to simulate traffic from separate network segments. What's next [#whats-next] FAQ [#faq] No. The generator produces ordinary logon, privilege-use, process-creation, and logoff activity shaped like what a real Windows environment logs — nothing here simulates an exploit, a malicious payload, or an attacker technique. Point the output at any OpenSearch-compatible index, or swap it for `stdout: {}` to inspect events locally without a running cluster at all. Yes. OpenSearch is API-compatible with Elasticsearch on bulk indexing, so pointing `hosts` at an Elasticsearch cluster and adjusting credentials indexes the same events the same way — see the [OpenSearch delivery lesson](/docs/tutorials/delivery/opensearch) for TLS and authentication details. For a different backend entirely, replace the `opensearch` block in `generator.yml` with another output plugin and keep the input and templates unchanged. Add a new Jinja template and wire it into the FSM as an additional state or transition branch. The Going further section above sketches one example — a `logon_failed.jinja` template for Event ID 4625, added as a probability-weighted branch off the logon state. The `shared` values this generator's templates already set are what any new event type would read to stay consistent with the rest of the session. Related [#related] * The [scenarios track](/docs/tutorials/scenarios) for the wider applied picture of synthetic test data * The [detection testing lesson](/docs/tutorials/detection-testing) for turning telemetry like this into Sigma-rule tests * The [Windows Event Log & Sysmon format lesson](/docs/tutorials/formats/windows-event-log) for the EVTX event model and EventID structure behind these templates * The [OpenSearch delivery lesson](/docs/tutorials/delivery/opensearch) for indexing, the bulk API, and TLS/authentication in depth * The [windows-security generator](/hub/windows-security) in the Eventum Hub for a production-tuned, ready-made version of this build # Alert Simulation: Scheduled Telegram Alerts A critical alert that never reaches the on-call phone, a dashboard tile that miscounts what arrived, an escalation that never fires on an unacknowledged warning — each of these bugs waits for a real incident to surface, and surfaces at the cost of one. Firing realistic alerts on a schedule surfaces them on demand instead, exercising every piece downstream of the trigger: the chat, the paging rotation, the dashboard count, the escalation path. The generator below fires exactly those alerts — randomized severity, service, and metric value, on a cron schedule — and delivers each over HTTP to a live Telegram chat through the Bot API. A miniature alerting pipeline in a few files, hitting a real notification channel instead of a mock. Alert simulation and the pipeline it exercises [#alert-simulation-and-the-pipeline-it-exercises] A [detection rule](/docs/tutorials/detection-testing) that matches, a threshold that trips, or a health check that fails all produce the same next step: an alert, structured no differently from any other [event](/docs/tutorials/foundations/logs-vs-metrics-events) Eventum generates — a severity, a service, a message, a timestamp. Testing that a rule fires correctly says nothing about what happens after — the notification routing, on-call paging, dashboard counting, and escalation the opening paragraph lists, everything downstream of the alert itself. A real incident exercises that whole chain eventually — at whatever severity and service it happens to produce. Firing realistic alerts on a schedule exercises the same chain on demand, at whatever severity mix and service spread a test needs, instead of waiting on a rule to match or an incident to happen. Telegram is one concrete chat to land these in — nothing below depends on Telegram past the request body and the endpoint URL, and the same pipeline reaches Slack, Discord, or PagerDuty just as directly. What you'll build [#what-youll-build] The generator uses: * [**cron**](/docs/plugins/input/cron) input — fires every 2 minutes. * [**all**](/docs/plugins/event/template/modes#all) picking mode — renders the one alert template on every tick, using conditional Jinja2 logic to vary message format and value range by severity. * [**HTTP output**](/docs/plugins/output/http) — posts each rendered alert to the Telegram Bot API as its own request. See [Send test data to an API endpoint](/docs/tutorials/delivery/http) for the delivery mechanics behind it. * [**Secrets**](/docs/core/config/secrets) — bot token stored securely in the keyring. * [**Parameters**](/docs/core/config/parameters) — chat ID passed through `startup.yml`. Prerequisites [#prerequisites] * [Eventum installed](/docs/core/introduction/installation) * A Telegram bot token (create one via [@BotFather](https://t.me/BotFather)) * The chat ID of the target chat (send a message to your bot, then query `https://api.telegram.org/bot/getUpdates` to find it) Project structure [#project-structure] Build it [#build-it] Create the project directory [#create-the-project-directory] ```bash mkdir -p eventum/generators/alerts/templates cd eventum ``` Write the alert template [#write-the-alert-template] The template produces a JSON body for the Telegram [sendMessage](https://core.telegram.org/bots/api#sendmessage) API. It uses `module.rand` to pick a random severity and service, then builds a Markdown-formatted message with an emoji indicator. ```jinja title="generators/alerts/templates/alert.jinja" {% set severity = module.rand.weighted_choice(["info", "warning", "critical", "resolved"], [0.4, 0.35, 0.2, 0.05]) %} {% set service = module.rand.choice(["api-gateway", "auth-service", "db-primary", "cache-redis", "worker-queue", "nginx-lb"]) %} {% set icons = {"info": "ℹ️", "warning": "⚠️", "critical": "🔴", "resolved": "✅"} %} {% set metric = module.rand.choice(["CPU usage", "memory usage", "request latency", "error rate", "disk I/O", "connection pool"]) %} {% if severity == "critical" %} {% set value = module.rand.number.integer(90, 100) %} {% elif severity == "warning" %} {% set value = module.rand.number.integer(70, 89) %} {% else %} {% set value = module.rand.number.integer(10, 69) %} {% endif %} { "chat_id": {{ params.chat_id }}, "parse_mode": "Markdown", "text": "{{ icons[severity] }} *{{ severity | upper }}* | `{{ service }}`\n\n{{ metric }}: *{{ value }}%*\nHost: `{{ module.faker.locale.en.hostname() }}`\nTime: {{ timestamp.strftime('%H:%M:%S UTC') }}" } ``` The message looks like this in Telegram: > 🔴 **CRITICAL** | `nginx-lb` > > connection pool: **95%** > Host: `desktop-00.miller.info` > Time: 13:34:09 UTC Configure the generator [#configure-the-generator] The cron expression `*/2 * * * * 0` fires at second 0 of every 2nd minute — seconds are the last field, after the weekday, so the trailing `0` is the second and the leading `*/2` the minute. The HTTP output posts to the Telegram API endpoint, with `success_code: 200` matching what `sendMessage` actually returns on success — the [http output](/docs/plugins/output/http)'s own default of `201` assumes an endpoint that treats each POST as creating a new resource, which Telegram's does not. ```yaml title="generators/alerts/generator.yml" input: - cron: expression: "*/2 * * * * 0" count: 1 event: template: mode: all params: chat_id: ${params.chat_id} templates: - alert: template: templates/alert.jinja output: - stdout: {} - http: url: "https://api.telegram.org/bot${secrets.telegram_token}/sendMessage" method: POST success_code: 200 headers: Content-Type: "application/json" formatter: format: plain ``` The `plain` formatter passes the template output as-is — the template already produces a valid JSON request body. Swapping in the `http` output's own default of `json-batch` here would collect every alert a batch accumulates into a single JSON array and post that array as one request, which Telegram's `sendMessage` endpoint does not accept; see [Send test data to an API endpoint](/docs/tutorials/delivery/http) for the full per-event versus per-batch tradeoff behind that choice. The `stdout` output is included for local debugging; remove it in production. Configure the application [#configure-the-application] ```yaml title="eventum.yml" server: host: "0.0.0.0" port: 9474 path: startup: /home/user/eventum/startup.yml generators_dir: /home/user/eventum/generators logs: /home/user/eventum/logs keyring_cryptfile: /home/user/eventum/cryptfile.cfg ``` All `path.*` values must be **absolute paths**. Adjust to match your actual project location. The startup file passes the chat ID as a [parameter](/docs/core/config/parameters): ```yaml title="startup.yml" - id: alerts path: alerts/generator.yml params: chat_id: "123456789" ``` Replace `123456789` with your actual Telegram chat ID. Store the bot token [#store-the-bot-token] The bot token is sensitive — store it in the [keyring](/docs/core/cli/eventum-keyring): ```bash eventum-keyring set telegram_token ``` Enter your bot token when prompted (e.g., `110201543:AAHdqTcvCH1vGWJxfSeofSAs0K5PALDsaw`). Run it [#run-it] ```bash eventum run -c eventum.yml ``` Every 2 minutes, an alert message appears in your Telegram chat. The terminal shows the same JSON body on stdout: The alert below was captured by pointing this generator's output at `stdout` alone and swapping the `cron` input for a bounded [static](/docs/plugins/input/static) input, so it rendered immediately in [sample mode](/docs/core/concepts/generator#sample-mode) (`--live-mode false`) instead of waiting on the real 2-minute cadence. The deployed config above, run with `eventum run`, produces the identical JSON shape at that real cadence, delivered to Telegram instead of the terminal. ```json { "chat_id": 123456789, "parse_mode": "Markdown", "text": "⚠️ *WARNING* | `api-gateway`\n\nconnection pool: *81%*\nHost: `srv-41.cole-williams.com`\nTime: 13:34:09 UTC" } ``` Going further [#going-further] * **Any HTTP-reachable destination** — as the intro notes, the request body and the URL are the only Telegram-specific parts. Point the same `http` output at a [Slack Incoming Webhook](https://api.slack.com/messaging/webhooks), a [Discord Webhook](https://discord.com/developers/docs/resources/webhook), or a [PagerDuty Events API](https://developer.pagerduty.com/docs/events-api-v2/overview/) integration instead, reshaping the template's JSON to match the destination's expected payload. See [Send test data to an API endpoint](/docs/tutorials/delivery/http) for the delivery mechanics this generalizes. * **Escalation schedule** — use [multiple inputs](/docs/core/concepts/scheduling#combining-multiple-inputs) with different cron expressions: check every 2 minutes during business hours, every 10 minutes overnight. * **Incident sequences** — switch to [FSM mode](/docs/core/concepts/producing#picking-modes) to model alert → acknowledged → investigating → resolved workflows. * **Silence window** — use the cron `start`/`end` fields to suppress alerts during maintenance windows. What's next [#whats-next] FAQ [#faq] Covered above: everything downstream of the alert firing is what's actually at stake, not the condition that triggers it. A real incident only ever exercises one severity/service combination at a time — a scheduled alert can cycle through all of them in an afternoon. See [Test Sigma rules with synthetic attack telemetry](/docs/tutorials/detection-testing) for the same argument one layer upstream, applied to the detection rule that would normally trigger an alert like this one in the first place. [`mode: all`](/docs/plugins/event/template/modes#all) renders every configured template on each input tick; with a single `alert` template listed, that's equivalent to always firing it. The randomization here — severity, service, metric — happens inside the template itself, not through template selection. The mode still matters as a choice: a second, always-fires template — a heartbeat or health-check event running alongside every alert, say — slots in by adding it to the `templates` list, with no change to `mode`. Contrast [`chance`](/docs/plugins/event/template/modes#chance) or [`spin`](/docs/plugins/event/template/modes#spin), which pick one of several templates per tick instead of rendering all of them. `json-batch` — the [http](/docs/plugins/output/http) output's own default — collects every event a batch accumulates into a single JSON array and posts that array as one request body; Telegram's `sendMessage` endpoint expects one JSON object per call, not an array. `json-batch` always wraps its output in an array regardless of batch size — even a single alert becomes a one-element array — so it would break delivery no matter how many alerts a batch holds. `plain` sends one request per event instead, and the template above already renders a complete `sendMessage` body per alert, which is exactly what that one request needs to be. See [Send test data to an API endpoint](/docs/tutorials/delivery/http) for the full per-event versus per-batch tradeoff behind this choice. Yes. See [Going further](#going-further) below for pointing the same `http` output at Slack, Discord, or PagerDuty instead. The cron schedule, the severity/service randomization, and the `${secrets.*}`-backed credential all carry over unchanged either way — only the URL, the body shape, and whatever authentication the new destination expects need to change. Related [#related] * The [Test Sigma rules with synthetic attack telemetry](/docs/tutorials/detection-testing) tutorial for generating the detection match an alert like this one would normally be triggered by * The [Logs vs metrics vs events](/docs/tutorials/foundations/logs-vs-metrics-events) lesson for treating an alert as a structured event like any other, not a special case * The [Send test data to an API endpoint](/docs/tutorials/delivery/http) lesson for the `success_code`, formatter, and batching mechanics behind the HTTP delivery this scenario relies on * The [Secrets](/docs/core/config/secrets) reference for the keyring and `${secrets.*}` substitution storing the bot token * The [Scenarios track](/docs/tutorials/scenarios) for the wider picture of applied synthetic data * Ready-made generators in the [Eventum Hub](/hub) # Test Data Pipeline End to End A new log source, a rebuilt parser, and a schema migration all face the same test before they ship: real-looking events, flowing through the actual pipeline, landing where a dashboard or an alert expects to find them. Waiting for production traffic to run that test finds a broken parser or a rejected field mapping only after real data has already hit it — the most expensive place to discover either. A handful of sample documents pasted directly into a test index confirms that a query runs; it says nothing about whether the transport in front of that index survives a realistic volume of traffic, or whether the source's own event shape still matches what the parser expects. Testing the pipeline instead of just the endpoint means generating a realistic stream, sending it through the same input, transport, and backend production uses, and then confirming — not assuming — that the backend received it. Eventum generates that stream and delivers it with the same output plugins a live deployment would use, so the whole chain gets a rehearsal before real traffic runs it for the first time. Test data pipeline before production [#test-data-pipeline-before-production] A pipeline earns trust by handling realistic conditions correctly, not by having a few lines pasted into a test environment and called good. Three moments call for exactly that kind of rehearsal: * **Onboarding a new source** — a fresh integration, log format, or event schema needs to reach its destination correctly before it carries real traffic. * **Changing a parser or a mapping** — a regression here doesn't announce itself; it silently drops or reshapes fields until a dashboard comes up empty or a detection stops firing. * **Validating a dashboard or an alert at volume** — a panel or a rule that looks right against ten hand-written events can still fail against ten thousand, or against a shape of traffic those ten never represented. Synthetic data fits this job because none of it is real: no user data, credentials, or business records ever sit in a test cluster. It isn't a fixed sample either — the same generator reruns in CI on every change instead of going stale after one export — and nothing here is hand-authored, so its volume scales to whatever the test needs, from a dozen events to a sustained stream. End-to-end data pipeline testing [#end-to-end-data-pipeline-testing] Testing a pipeline end to end means exercising all three of its stages together, not just the one under change, and then checking what came out the other side: The first two stages are a single Eventum generator: an input plugin paces the timestamps, an event plugin — typically [template](/docs/plugins/event/template) — renders each one into a realistic event, and an output plugin delivers it through whatever transport the real pipeline uses: OpenSearch, Kafka, ClickHouse, or an HTTP endpoint. None of that is specific to testing; it's the same three-stage pipeline every Eventum generator runs. The third stage is what turns a generator run into a pipeline test: checking, after the run, that the destination actually holds what was sent, rather than inferring it from the fact that the generator didn't crash. Eventum's own per-plugin counters confirm its half of the handoff — that it handed a formatted event to the destination and didn't get an error back. A query against the backend confirms the other half — that the documents, rows, or messages it describes are actually there and intact. What you'll build [#what-youll-build] The generator below models connection events from a network perimeter — most connections allowed, a smaller share denied — and sends them to OpenSearch, chosen here because a document count is the simplest possible receipt check: index the events, then ask the cluster how many landed. The same method works unchanged against [Kafka](/docs/tutorials/delivery/kafka), [ClickHouse](/docs/tutorials/delivery/clickhouse), or [any HTTP endpoint](/docs/tutorials/delivery/http) — the pipeline layers section further down covers what changes when you swap backends. Two templates cover the two outcomes: * **Allowed** (the common case) — a connection with a realistic transport protocol and byte count. * **Denied** (the rare case) — the same shape, minus the transferred bytes, with a reason attached. A [`chance`](/docs/plugins/event/template/modes#chance) pick weights the two roughly 92/8, and a bounded [`static`](/docs/plugins/input/static) input produces a fixed batch instead of an open-ended stream — the shape a repeatable pipeline test actually wants, run the same way every time in a CI job or a terminal. Prerequisites [#prerequisites] * [Eventum installed](/docs/core/introduction/installation) * An OpenSearch instance reachable over HTTP(S) *(optional — the generator also prints to stdout, so nothing here requires a live cluster)* Project structure [#project-structure] Build it [#build-it] Create the project directory [#create-the-project-directory] ```bash mkdir -p network-events/templates cd network-events ``` Write the event templates [#write-the-event-templates] Both templates share the same ECS shape — [`event.category: ["network"]`](/docs/tutorials/formats/ecs), a source and destination, and a `related.ip` list for pivoting — so the only real difference between them is what happened to the connection. `templates/connection-allowed.jinja` covers the common case. [`module.rand.weighted_choice`](/docs/tutorials/realism/values) picks a destination port and transport with realistic odds, and a clamped [`lognormal`](/docs/tutorials/realism/values) draw gives the transferred bytes a skewed, realistic spread instead of a flat range: ```jinja title="network-events/templates/connection-allowed.jinja" {%- set src_ip = module.rand.network.ip_v4_public() -%} {%- set dst_ip = module.rand.network.ip_v4_private_a() -%} {%- set src_port = module.rand.number.integer(1024, 65535) -%} {%- set dst_port = module.rand.weighted_choice({443: 70, 80: 20, 22: 8, 3389: 2}) -%} {%- set transport = module.rand.weighted_choice({"tcp": 85, "udp": 12, "icmp": 3}) -%} {%- set bytes_sent = module.rand.number.clamp(module.rand.number.lognormal(7.5, 0.8), 200, 50000) | round | int -%} {%- set event = { "@timestamp": timestamp.isoformat(), "ecs": {"version": "9.4.0"}, "event": { "kind": "event", "category": ["network"], "type": ["allowed"], "action": "connection-attempt", "outcome": "success" }, "network": { "transport": transport, "bytes": bytes_sent }, "source": {"ip": src_ip, "port": src_port}, "destination": {"ip": dst_ip, "port": dst_port}, "host": {"name": "edge-fw-01"}, "related": {"ip": [src_ip, dst_ip]} } -%} {{ event | tojson }} ``` `templates/connection-denied.jinja` covers the rarer case — a different, riskier port distribution, no transferred bytes, and a reason: ```jinja title="network-events/templates/connection-denied.jinja" {%- set src_ip = module.rand.network.ip_v4_public() -%} {%- set dst_ip = module.rand.network.ip_v4_private_a() -%} {%- set src_port = module.rand.number.integer(1024, 65535) -%} {%- set dst_port = module.rand.weighted_choice({3389: 40, 22: 30, 445: 20, 23: 10}) -%} {%- set transport = module.rand.weighted_choice({"tcp": 90, "udp": 10}) -%} {%- set event = { "@timestamp": timestamp.isoformat(), "ecs": {"version": "9.4.0"}, "event": { "kind": "event", "category": ["network"], "type": ["denied"], "action": "connection-attempt", "outcome": "failure", "reason": "no matching allow rule" }, "network": { "transport": transport }, "source": {"ip": src_ip, "port": src_port}, "destination": {"ip": dst_ip, "port": dst_port}, "host": {"name": "edge-fw-01"}, "related": {"ip": [src_ip, dst_ip]} } -%} {{ event | tojson }} ``` Configure the generator [#configure-the-generator] A [`static`](/docs/plugins/input/static) input produces a fixed batch of 1,000 timestamps at once — a rerunnable test dataset, not a live stream. [`mode: chance`](/docs/plugins/event/template/modes#chance) picks between the two templates, and the output fans out to both `stdout`, for a local, human-readable copy of everything generated, and [`opensearch`](/docs/tutorials/delivery/opensearch), for the real delivery: ```yaml title="network-events/generator.yml" input: - static: count: 1000 event: template: mode: chance templates: - connection_allowed: template: templates/connection-allowed.jinja chance: 92 - connection_denied: template: templates/connection-denied.jinja chance: 8 output: - stdout: formatter: format: json - opensearch: hosts: - ${params.opensearch_host} username: ${params.opensearch_user} password: ${secrets.opensearch_password} index: network-events verify: false ``` Fan-out delivery is a core [output](/docs/core/concepts/output#fan-out-delivery) behavior, not a special case: every output plugin receives every event independently, so adding `stdout` alongside `opensearch` costs nothing and gives you a local copy of the exact stream being delivered. **No OpenSearch cluster?** You don't need to delete the `opensearch` block to follow along. This plugin only reaches out to the cluster when it actually has events to write, not when the generator starts, so an unreachable cluster doesn't stop the run — the `stdout` output above still prints every event either way. What you lose without a live cluster is delivery itself: writes to `opensearch` fail quietly in the background, counted as `write_failed` and logged, while the rest of the pipeline keeps going. Run it [#run-it] [`eventum generate`](/docs/core/cli/eventum-generate) runs a single generator without the application server — the quickest way to produce a batch and inspect it directly. [Sample mode](/docs/core/concepts/generator#execution-modes) (`--live-mode false`) releases the full batch immediately instead of pacing it against the clock: ```bash eventum generate \ --path network-events/generator.yml \ --id network-events \ --live-mode false ``` The full 1,000-event batch prints and the command exits in under a second. A slice of an actual run — three allowed connections followed by a denied one: ```json {"@timestamp": "2026-07-12T14:24:17.070047+00:00", "destination": {"ip": "10.162.7.193", "port": 80}, "ecs": {"version": "9.4.0"}, "event": {"action": "connection-attempt", "category": ["network"], "kind": "event", "outcome": "success", "type": ["allowed"]}, "host": {"name": "edge-fw-01"}, "network": {"bytes": 3345, "transport": "udp"}, "related": {"ip": ["171.164.170.62", "10.162.7.193"]}, "source": {"ip": "171.164.170.62", "port": 45881}} {"@timestamp": "2026-07-12T14:24:17.070047+00:00", "destination": {"ip": "10.164.61.141", "port": 443}, "ecs": {"version": "9.4.0"}, "event": {"action": "connection-attempt", "category": ["network"], "kind": "event", "outcome": "success", "type": ["allowed"]}, "host": {"name": "edge-fw-01"}, "network": {"bytes": 3459, "transport": "udp"}, "related": {"ip": ["86.129.32.16", "10.164.61.141"]}, "source": {"ip": "86.129.32.16", "port": 6037}} {"@timestamp": "2026-07-12T14:24:17.070047+00:00", "destination": {"ip": "10.83.158.120", "port": 443}, "ecs": {"version": "9.4.0"}, "event": {"action": "connection-attempt", "category": ["network"], "kind": "event", "outcome": "success", "type": ["allowed"]}, "host": {"name": "edge-fw-01"}, "network": {"bytes": 1756, "transport": "tcp"}, "related": {"ip": ["3.164.166.80", "10.83.158.120"]}, "source": {"ip": "3.164.166.80", "port": 18625}} {"@timestamp": "2026-07-12T14:24:17.070047+00:00", "destination": {"ip": "10.140.58.19", "port": 22}, "ecs": {"version": "9.4.0"}, "event": {"action": "connection-attempt", "category": ["network"], "kind": "event", "outcome": "failure", "reason": "no matching allow rule", "type": ["denied"]}, "host": {"name": "edge-fw-01"}, "network": {"transport": "tcp"}, "related": {"ip": ["192.139.130.16", "10.140.58.19"]}, "source": {"ip": "192.139.130.16", "port": 25701}} ``` Across that run, 942 of the 1,000 events came out allowed and 58 denied — close to the configured 92/8 split, with the small spread expected from a chance-weighted pick. Validate data pipeline end to end [#validate-data-pipeline-end-to-end] The run above already proves the first stage: a realistic, ECS-shaped stream came out the other end of the templates, matching what the `opensearch` block would deliver, since both outputs receive identical events from the same fan-out. The second stage, delivery, is the same generator run with a reachable cluster: nothing about the templates, the picking mode, or the batch changes, only whether `opensearch.hosts` resolves to something real. The [OpenSearch delivery lesson](/docs/tutorials/delivery/opensearch) covers connection, authentication, and TLS in depth. The third stage, confirming receipt, happens outside the generator, through two independent checks: **Eventum's own counters.** Every output plugin tracks `written`, `write_failed`, and `format_failed` for its own deliveries: | Counter | What it tells you | | --------------- | ------------------------------------------------------------------------- | | `written` | Events the plugin delivered successfully | | `write_failed` | Events the destination rejected (connection error, timeout, auth failure) | | `format_failed` | Events that failed formatting before delivery even started | A one-shot `eventum generate` run like the one above doesn't expose these — there's no running application to ask. List the generator in [`startup.yml`](/docs/core/config/startup-yml) and start the full application with [`eventum run`](/docs/core/cli/eventum-run) instead, and the counters become available live: ```bash curl -u eventum:eventum http://localhost:9474/api/generators/network-events/stats ``` The response nests one entry per output plugin under `output`, each carrying the three counters above, alongside a `total_written` rollup — see the [Get Generator Stats](/docs/api/generators/id/stats/get) reference for the full response shape. Studio's [Instance metrics](/docs/studio/instances#instance-metrics) dialog shows the same numbers on a pipeline diagram, without a single `curl` call. **A query against the backend itself.** A plugin's `written` count only confirms that OpenSearch acknowledged the request — it doesn't rule out a mapping conflict or a dropped shard swallowing documents afterward. Asking the index directly closes that gap: ```bash curl -u admin: "https://localhost:9200/network-events/_count" ``` A count that matches `total_written` — or one that keeps climbing at the expected rate, for a long-running generator — is the actual proof: the pipeline generated the data, delivered it over the same transport production uses, and the backend genuinely holds it. The pipeline, layer by layer [#the-pipeline-layer-by-layer] This lesson connects four tracks, each covering one layer of the pipeline in depth: Not every destination behaves the same way when it's unreachable, which matters for a test built to run without one. OpenSearch and [HTTP](/docs/tutorials/delivery/http) only reach out when there's something to write, so a pipeline test still runs end to end without a live target — delivery just fails quietly, per event, as shown above. [Kafka](/docs/tutorials/delivery/kafka) and [ClickHouse](/docs/tutorials/delivery/clickhouse) connect as soon as the generator starts, so an unreachable broker or database stops the run before it produces anything. Neither behavior is wrong, but know which one your backend has before building a test around it. What to run through the pipeline [#what-to-run-through-the-pipeline] The method above — realistic stream, real transport, confirmed receipt — applies to any source. A few to start from: What's next [#whats-next] FAQ [#faq] Two independent checks, both covered above. Eventum's own `written`, `write_failed`, and `format_failed` counters, available through the [REST API](/docs/api/generators/id/stats/get) or [Studio](/docs/studio/instances#instance-metrics) once the generator runs under `eventum run`, confirm that Eventum handed the data off successfully. A direct query against the backend — a document count, a row count, a message offset — confirms the backend actually holds it, which catches failures the write-side counters can't see, such as a mapping conflict that surfaces after acknowledgement. Nothing about the templates, the picking mode, or the batch. Add a `stdout` output alongside (or instead of) the real one, as this lesson does, and every event that would have been delivered prints locally. For an OpenSearch or HTTP destination specifically, you don't even need to remove the block — an unreachable target doesn't stop the run, since neither plugin checks connectivity until it has something to write. Kafka and ClickHouse do check at startup, so those need either a reachable target or the block removed. Not the same measurement. This lesson checks correctness — whether a realistic event reaches the destination intact, in the format it expects. [Load testing](/docs/tutorials/load-testing) checks capacity — how much throughput the destination or the API in front of it can sustain before it degrades. A pipeline that passes this lesson's check can still fall over under load, and a pipeline that survives a load test can still be silently dropping fields a correctness check would have caught. Related [#related] * The [Scenarios pillar](/docs/tutorials/scenarios) for the wider picture of what synthetic data tests * The [Eventum Hub](/hub) for generators built for other pipelines and sources, ready to point at yours # Generate Clickstream Data for ClickHouse Clickstream data is the record of every page a visitor loads on their way through a site, in the order they loaded it — a landing page, a run of product pages, a cart, sometimes a checkout. A funnel report, a conversion-rate query, or a cohort analysis depends entirely on that order: how many sessions reached each stage, and how many dropped off before the next one. A stream of page-view events generated independently of each other can share a session id by coincidence. It cannot reproduce a visitor's journey — landing, browsing, cart — arriving in that order, which is the one property a funnel query actually reads. Eventum produces that order with the same finite-state-machine technique covered in [Modeling sessions](/docs/tutorials/realism/sessions): each stage of the journey is a template, and the template's own logic decides when a visitor is ready to advance. This lesson applies it to a five-stage browsing funnel instead of a three-stage login/action/logout cycle, adds a bounce path for visitors who browse but never buy, and streams the result into ClickHouse for funnel analysis. Funnel test data and the bounce path [#funnel-test-data-and-the-bounce-path] Clickstream analytics reports on stages, not isolated page views: a funnel groups every session by the furthest stage it reached — landing, browsing, cart, checkout — and counts how many sessions made it to each one and how many dropped off before the next. That count only means what it claims to mean if every event in a session shares one identifier and arrives in the order a real visit takes; a checkout with no landing before it, or two unrelated visitors sharing a session id by coincidence, breaks exactly what the funnel is supposed to measure. A bounce is the funnel's simplest outcome: a visitor who lands, browses for a while, and leaves without adding anything to the cart. Modeling it needs the same session-scoped memory as the rest of the funnel — a generator has to track how long the current visitor has been browsing without converting, not just assign a flat percentage of sessions to "left immediately." What you'll build [#what-youll-build] The generator uses: * [**time-patterns**](/docs/plugins/input/time-patterns) input — a daily traffic curve peaking at midday, shaped with a beta distribution. * [**FSM**](/docs/core/concepts/producing#picking-modes) picking mode — five states modeling a user journey, each one deciding from its own state when the visitor is ready to advance. * [**shared state**](/docs/core/concepts/producing#state-management) — a session id, a cart count, and a browse counter, all reset at the start of every new session. * [**ClickHouse output**](/docs/plugins/output/clickhouse) — events inserted as JSON rows into a `page_views` table. Prerequisites [#prerequisites] * [Eventum installed](/docs/core/introduction/installation) * A ClickHouse instance with HTTP interface enabled (default port 8123) **No ClickHouse?** Replace the `clickhouse` output with `stdout: {}` to preview the JSON events in your terminal. Project structure [#project-structure] Prepare ClickHouse [#prepare-clickhouse] Create the target table before running the generator: ```sql CREATE TABLE IF NOT EXISTS page_views ( timestamp DateTime64(3), session_id String, user_agent String, referrer String, page String, page_type String, duration_ms UInt32, items_in_cart UInt8 ) ENGINE = MergeTree() ORDER BY (timestamp, session_id); ``` For the mechanics behind this insert — the HTTP interface, `JSONEachRow`, and connection pooling — see [Generate test data for ClickHouse](/docs/tutorials/delivery/clickhouse). Build it [#build-it] Create the project directory [#create-the-project-directory] ```bash mkdir -p eventum/generators/clickstream/{patterns,templates} cd eventum ``` Define the daily traffic pattern [#define-the-daily-traffic-pattern] The traffic pattern uses a beta distribution anchored to a full day, the same shape the [Windows Event Log lesson](/docs/tutorials/formats/windows-event-log) uses for a business-hours curve: equal shape parameters center the peak at midday and taper density toward both ends of the day, closer to how site traffic actually rises and falls than a hard-edged ramp. ```yaml title="generators/clickstream/patterns/daily-traffic.yml" label: Daily web traffic oscillator: start: "00:00:00" end: "never" period: 1 unit: days multiplier: ratio: 2000 randomizer: deviation: 0.25 direction: mixed spreader: distribution: beta parameters: a: 4 b: 4 ``` With `start` anchored to midnight and a one-day period, the equal shape parameters (`a: 4`, `b: 4`) put the peak at noon and thin traffic toward both ends of the day; `ratio: 2000` sets about 2,000 page-view timestamps per day before the randomizer's ±25% variance is applied. Write the session templates [#write-the-session-templates] The FSM models a user journey through five stages. Each template renders one page-view event and decides, from its own state, whether the visitor is ready to move on — that decision never compares a running count directly inside a transition; each template computes its own threshold and records the outcome as a boolean flag in `shared` state, and the transition to the next stage only checks whether that flag is present. **Landing** — the entry point. Starts a new session: a fresh session id, an empty cart, and a browse counter reset to zero, plus a random user agent and referrer. It also clears every flag a previous session might have left behind, the same way [Modeling sessions](/docs/tutorials/realism/sessions)' `login` template clears `logout_ready` for the next visitor. ```jinja title="generators/clickstream/templates/landing.jinja" {%- set session_id = module.rand.crypto.uuid4() -%} {%- do shared.set("session_id", session_id) -%} {%- do shared.set("items_in_cart", 0) -%} {%- do shared.set("browse_streak", 0) -%} {%- do shared.pop("ready_to_add", None) -%} {%- do shared.pop("ready_to_checkout", None) -%} {%- do shared.pop("session_done", None) -%} {%- set ua = module.faker.locale.en_US.user_agent() -%} {%- do shared.set("user_agent", ua) -%} {%- set ref = module.rand.choice(["https://google.com", "https://bing.com", "https://twitter.com", "direct", "https://reddit.com"]) -%} {%- do shared.set("referrer", ref) -%} { "timestamp": "{{ timestamp.strftime('%Y-%m-%d %H:%M:%S.%f') }}", "session_id": "{{ session_id }}", "user_agent": "{{ ua }}", "referrer": "{{ ref }}", "page": "/", "page_type": "landing", "duration_ms": {{ module.rand.number.integer(500, 5000) }}, "items_in_cart": 0 } ``` **Browse** — product listing or detail pages. `browse_streak` counts pages viewed since the last landing or cart addition. Once it reaches 5 with an empty cart, the session is done — a bounce. From 2 pages onward, each additional page rolls a 50% chance of signaling readiness to add to the cart, so conversions land after a variable number of pages instead of a fixed one: ```jinja title="generators/clickstream/templates/browse.jinja" {%- set browse_streak = shared.get("browse_streak", 0) + 1 -%} {%- do shared.set("browse_streak", browse_streak) -%} {%- set items_in_cart = shared.get("items_in_cart", 0) -%} {%- set pages = ["/products", "/products/wireless-mouse", "/products/usb-hub", "/products/keyboard", "/products/webcam", "/categories/electronics", "/categories/home"] -%} {%- if items_in_cart == 0 and browse_streak >= 5 -%} {%- do shared.set("session_done", true) -%} {%- elif browse_streak >= 2 and module.rand.chance(0.5) -%} {%- do shared.set("ready_to_add", true) -%} {%- endif -%} { "timestamp": "{{ timestamp.strftime('%Y-%m-%d %H:%M:%S.%f') }}", "session_id": "{{ shared.get('session_id') }}", "user_agent": "{{ shared.get('user_agent') }}", "referrer": "{{ shared.get('referrer') }}", "page": "{{ module.rand.choice(pages) }}", "page_type": "browse", "duration_ms": {{ module.rand.number.integer(1000, 15000) }}, "items_in_cart": {{ items_in_cart }} } ``` A fixed page-count threshold would make every converting session the same length. Rolling a chance instead spreads conversions across a range of session lengths — some visitors add an item after two pages, others after four or five — the same way real browsing sessions vary, while the deterministic ceiling at 5 still guarantees a bounce for anyone who never converts at all. **Add to cart** — records the cart addition, then clears `ready_to_add` and resets `browse_streak` immediately. Unlike [Modeling sessions](/docs/tutorials/realism/sessions)' `logout_ready`, which fires once per session, `ready_to_add` here can fire again on a later loop through `browse` — up to the cart's two-item target. That's why the flag has to be cleared the moment it's consumed: otherwise the next stretch of browsing would fall straight back into `add-to-cart` on the stale value instead of earning it again from a fresh count: ```jinja title="generators/clickstream/templates/add-to-cart.jinja" {%- do shared.pop("ready_to_add", None) -%} {%- do shared.set("browse_streak", 0) -%} {%- set items_in_cart = shared.get("items_in_cart", 0) + 1 -%} {%- do shared.set("items_in_cart", items_in_cart) -%} {%- if items_in_cart >= 2 -%} {%- do shared.set("ready_to_checkout", true) -%} {%- endif -%} { "timestamp": "{{ timestamp.strftime('%Y-%m-%d %H:%M:%S.%f') }}", "session_id": "{{ shared.get('session_id') }}", "user_agent": "{{ shared.get('user_agent') }}", "referrer": "{{ shared.get('referrer') }}", "page": "/cart", "page_type": "add_to_cart", "duration_ms": {{ module.rand.number.integer(500, 3000) }}, "items_in_cart": {{ items_in_cart }} } ``` **Checkout** — completes the purchase, clearing the flag that led here. ```jinja title="generators/clickstream/templates/checkout.jinja" {%- do shared.pop("ready_to_checkout", None) -%} { "timestamp": "{{ timestamp.strftime('%Y-%m-%d %H:%M:%S.%f') }}", "session_id": "{{ shared.get('session_id') }}", "user_agent": "{{ shared.get('user_agent') }}", "referrer": "{{ shared.get('referrer') }}", "page": "/checkout/complete", "page_type": "checkout", "duration_ms": {{ module.rand.number.integer(2000, 10000) }}, "items_in_cart": {{ shared.get("items_in_cart", 0) }} } ``` **Exit** — the session's last event, reached either after checkout or from a bounce. The FSM transitions back to `landing` to start a new session. ```jinja title="generators/clickstream/templates/exit.jinja" { "timestamp": "{{ timestamp.strftime('%Y-%m-%d %H:%M:%S.%f') }}", "session_id": "{{ shared.get('session_id') }}", "user_agent": "{{ shared.get('user_agent') }}", "referrer": "{{ shared.get('referrer') }}", "page": "{{ module.rand.choice(['/products', '/', '/categories/electronics']) }}", "page_type": "exit", "duration_ms": {{ module.rand.number.integer(100, 1000) }}, "items_in_cart": {{ shared.get("items_in_cart", 0) }} } ``` Configure the generator [#configure-the-generator] Every transition below checks a single boolean flag, or falls back to a transition that always fires: ```yaml title="generators/clickstream/generator.yml" input: - time_patterns: patterns: - patterns/daily-traffic.yml event: template: mode: fsm templates: - landing: template: templates/landing.jinja initial: true transitions: - to: browse when: { always: } - browse: template: templates/browse.jinja transitions: - to: exit when: { defined: shared.session_done } - to: add-to-cart when: { defined: shared.ready_to_add } - to: browse when: { always: } - add-to-cart: template: templates/add-to-cart.jinja transitions: - to: checkout when: { defined: shared.ready_to_checkout } - to: browse when: { always: } - checkout: template: templates/checkout.jinja transitions: - to: exit when: { always: } - exit: template: templates/exit.jinja transitions: - to: landing when: { always: } output: - stdout: formatter: format: json - clickhouse: host: ${params.clickhouse_host} port: ${params.clickhouse_port} database: default table: page_views username: ${params.clickhouse_user} password: ${secrets.clickhouse_password} ``` The session flow: | From | To | Condition | Meaning | | ----------- | ----------- | ----------------------------------- | ------------------------------------------- | | landing | browse | `always` | Every visit starts with a landing page | | browse | exit | `defined: shared.session_done` | Bounced: browsed 5 pages with an empty cart | | browse | add-to-cart | `defined: shared.ready_to_add` | Ready to add an item | | browse | browse | `always` (fallback) | Keep browsing | | add-to-cart | checkout | `defined: shared.ready_to_checkout` | Cart reached its two-item target | | add-to-cart | browse | `always` (fallback) | Keep shopping | | checkout | exit | `always` | Session complete | | exit | landing | `always` | New session starts | Transitions are evaluated **in order**, and the first one whose condition holds wins. On `browse` and `add-to-cart`, the flag checks are listed before the `always` fallback — reversing that order would make the fallback fire first every time, since `always` never fails, and Eventum would never reach the flag check at all. Configure the application [#configure-the-application] ```yaml title="eventum.yml" server: host: "0.0.0.0" port: 9474 path: startup: /home/user/eventum/startup.yml generators_dir: /home/user/eventum/generators logs: /home/user/eventum/logs keyring_cryptfile: /home/user/eventum/cryptfile.cfg generation: timezone: UTC batch: size: 500 ``` All `path.*` values must be **absolute paths**. Adjust to match your actual project location. ```yaml title="startup.yml" - id: clickstream path: clickstream/generator.yml params: clickhouse_host: "localhost" clickhouse_port: 8123 clickhouse_user: "default" ``` Store the ClickHouse password in the [keyring](/docs/core/cli/eventum-keyring): ```bash eventum-keyring set clickhouse_password ``` Run it [#run-it] ```bash eventum run -c eventum.yml ``` The sessions below were produced by running this generator directly with [`eventum generate`](/docs/core/cli/eventum-generate) in [sample mode](/docs/tutorials/foundations/streaming-vs-bulk) (`--live-mode false`), with a short bound temporarily added to the traffic pattern so a handful of complete sessions render at once for inspection. `eventum run` in the application built above drops that bound and keeps `end: "never"`, pacing sessions to the traffic pattern's actual daily curve in [live mode](/docs/core/concepts/generator#live-mode-default) instead of producing them all at once. A converting session — three pages, a first item, two more pages, a second item, then checkout: ```json title="A converting session" {"timestamp": "2026-07-17 23:52:18.575101", "session_id": "e6199a85-3ed6-40b3-932c-3a0bff12b580", "user_agent": "Mozilla/5.0 (Windows; U; Windows NT 10.0) AppleWebKit/531.24.4 (KHTML, like Gecko) Version/4.0.1 Safari/531.24.4", "referrer": "https://twitter.com", "page": "/", "page_type": "landing", "duration_ms": 2690, "items_in_cart": 0} {"timestamp": "2026-07-17 23:57:40.974241", "session_id": "e6199a85-3ed6-40b3-932c-3a0bff12b580", "user_agent": "Mozilla/5.0 (Windows; U; Windows NT 10.0) AppleWebKit/531.24.4 (KHTML, like Gecko) Version/4.0.1 Safari/531.24.4", "referrer": "https://twitter.com", "page": "/products/webcam", "page_type": "browse", "duration_ms": 14323, "items_in_cart": 0} {"timestamp": "2026-07-18 00:05:33.659582", "session_id": "e6199a85-3ed6-40b3-932c-3a0bff12b580", "user_agent": "Mozilla/5.0 (Windows; U; Windows NT 10.0) AppleWebKit/531.24.4 (KHTML, like Gecko) Version/4.0.1 Safari/531.24.4", "referrer": "https://twitter.com", "page": "/products/webcam", "page_type": "browse", "duration_ms": 3793, "items_in_cart": 0} {"timestamp": "2026-07-18 00:21:50.329135", "session_id": "e6199a85-3ed6-40b3-932c-3a0bff12b580", "user_agent": "Mozilla/5.0 (Windows; U; Windows NT 10.0) AppleWebKit/531.24.4 (KHTML, like Gecko) Version/4.0.1 Safari/531.24.4", "referrer": "https://twitter.com", "page": "/products", "page_type": "browse", "duration_ms": 8547, "items_in_cart": 0} {"timestamp": "2026-07-18 00:40:17.632793", "session_id": "e6199a85-3ed6-40b3-932c-3a0bff12b580", "user_agent": "Mozilla/5.0 (Windows; U; Windows NT 10.0) AppleWebKit/531.24.4 (KHTML, like Gecko) Version/4.0.1 Safari/531.24.4", "referrer": "https://twitter.com", "page": "/cart", "page_type": "add_to_cart", "duration_ms": 2378, "items_in_cart": 1} {"timestamp": "2026-07-18 00:50:17.960856", "session_id": "e6199a85-3ed6-40b3-932c-3a0bff12b580", "user_agent": "Mozilla/5.0 (Windows; U; Windows NT 10.0) AppleWebKit/531.24.4 (KHTML, like Gecko) Version/4.0.1 Safari/531.24.4", "referrer": "https://twitter.com", "page": "/products/usb-hub", "page_type": "browse", "duration_ms": 11509, "items_in_cart": 1} {"timestamp": "2026-07-18 00:56:20.483416", "session_id": "e6199a85-3ed6-40b3-932c-3a0bff12b580", "user_agent": "Mozilla/5.0 (Windows; U; Windows NT 10.0) AppleWebKit/531.24.4 (KHTML, like Gecko) Version/4.0.1 Safari/531.24.4", "referrer": "https://twitter.com", "page": "/categories/home", "page_type": "browse", "duration_ms": 11637, "items_in_cart": 1} {"timestamp": "2026-07-18 00:57:13.339342", "session_id": "e6199a85-3ed6-40b3-932c-3a0bff12b580", "user_agent": "Mozilla/5.0 (Windows; U; Windows NT 10.0) AppleWebKit/531.24.4 (KHTML, like Gecko) Version/4.0.1 Safari/531.24.4", "referrer": "https://twitter.com", "page": "/cart", "page_type": "add_to_cart", "duration_ms": 1597, "items_in_cart": 2} {"timestamp": "2026-07-18 00:59:05.632261", "session_id": "e6199a85-3ed6-40b3-932c-3a0bff12b580", "user_agent": "Mozilla/5.0 (Windows; U; Windows NT 10.0) AppleWebKit/531.24.4 (KHTML, like Gecko) Version/4.0.1 Safari/531.24.4", "referrer": "https://twitter.com", "page": "/checkout/complete", "page_type": "checkout", "duration_ms": 2722, "items_in_cart": 2} {"timestamp": "2026-07-18 01:02:06.693648", "session_id": "e6199a85-3ed6-40b3-932c-3a0bff12b580", "user_agent": "Mozilla/5.0 (Windows; U; Windows NT 10.0) AppleWebKit/531.24.4 (KHTML, like Gecko) Version/4.0.1 Safari/531.24.4", "referrer": "https://twitter.com", "page": "/", "page_type": "exit", "duration_ms": 711, "items_in_cart": 2} ``` Filtering the same run for a session that never added anything to its cart shows the bounce path: `browse_streak` reaches 5 with the cart still empty, and `session_done` sends it straight to `exit` instead of `add-to-cart`: ```json title="A bounced session" {"timestamp": "2026-07-18 01:46:49.808270", "session_id": "a056e002-3805-4c65-b8b4-a17f6a7c10f9", "user_agent": "Mozilla/5.0 (iPhone; CPU iPhone OS 1_1_5 like Mac OS X) AppleWebKit/532.2 (KHTML, like Gecko) CriOS/63.0.875.0 Mobile/90Y946 Safari/532.2", "referrer": "https://reddit.com", "page": "/", "page_type": "landing", "duration_ms": 4311, "items_in_cart": 0} {"timestamp": "2026-07-18 01:48:51.048712", "session_id": "a056e002-3805-4c65-b8b4-a17f6a7c10f9", "user_agent": "Mozilla/5.0 (iPhone; CPU iPhone OS 1_1_5 like Mac OS X) AppleWebKit/532.2 (KHTML, like Gecko) CriOS/63.0.875.0 Mobile/90Y946 Safari/532.2", "referrer": "https://reddit.com", "page": "/products/usb-hub", "page_type": "browse", "duration_ms": 11673, "items_in_cart": 0} {"timestamp": "2026-07-18 01:49:14.151790", "session_id": "a056e002-3805-4c65-b8b4-a17f6a7c10f9", "user_agent": "Mozilla/5.0 (iPhone; CPU iPhone OS 1_1_5 like Mac OS X) AppleWebKit/532.2 (KHTML, like Gecko) CriOS/63.0.875.0 Mobile/90Y946 Safari/532.2", "referrer": "https://reddit.com", "page": "/products", "page_type": "browse", "duration_ms": 10861, "items_in_cart": 0} {"timestamp": "2026-07-18 01:49:48.756966", "session_id": "a056e002-3805-4c65-b8b4-a17f6a7c10f9", "user_agent": "Mozilla/5.0 (iPhone; CPU iPhone OS 1_1_5 like Mac OS X) AppleWebKit/532.2 (KHTML, like Gecko) CriOS/63.0.875.0 Mobile/90Y946 Safari/532.2", "referrer": "https://reddit.com", "page": "/products", "page_type": "browse", "duration_ms": 7044, "items_in_cart": 0} {"timestamp": "2026-07-18 01:59:13.147119", "session_id": "a056e002-3805-4c65-b8b4-a17f6a7c10f9", "user_agent": "Mozilla/5.0 (iPhone; CPU iPhone OS 1_1_5 like Mac OS X) AppleWebKit/532.2 (KHTML, like Gecko) CriOS/63.0.875.0 Mobile/90Y946 Safari/532.2", "referrer": "https://reddit.com", "page": "/products/webcam", "page_type": "browse", "duration_ms": 11445, "items_in_cart": 0} {"timestamp": "2026-07-18 02:02:56.904621", "session_id": "a056e002-3805-4c65-b8b4-a17f6a7c10f9", "user_agent": "Mozilla/5.0 (iPhone; CPU iPhone OS 1_1_5 like Mac OS X) AppleWebKit/532.2 (KHTML, like Gecko) CriOS/63.0.875.0 Mobile/90Y946 Safari/532.2", "referrer": "https://reddit.com", "page": "/products/wireless-mouse", "page_type": "browse", "duration_ms": 11214, "items_in_cart": 0} {"timestamp": "2026-07-18 02:03:05.691342", "session_id": "a056e002-3805-4c65-b8b4-a17f6a7c10f9", "user_agent": "Mozilla/5.0 (iPhone; CPU iPhone OS 1_1_5 like Mac OS X) AppleWebKit/532.2 (KHTML, like Gecko) CriOS/63.0.875.0 Mobile/90Y946 Safari/532.2", "referrer": "https://reddit.com", "page": "/categories/electronics", "page_type": "exit", "duration_ms": 459, "items_in_cart": 0} ``` Of the four sessions this run touched, two converted, one bounced, and one was still browsing when the bounded run ended. Small samples vary, but the chance roll's math puts the theoretical bounce rate at one session in eight — three independent 50% rolls have to fail in a row before the fifth unconverted page view forces the exit. Query the conversion funnel in ClickHouse: ```sql SELECT page_type, count() AS views, uniqExact(session_id) AS sessions FROM page_views GROUP BY page_type ORDER BY views DESC; ``` Going further [#going-further] * **Tune the bounce rate** — raise or lower `browse.jinja`'s `browse_streak` ceiling or the chance roll's probability to match a real funnel's bounce rate; add a direct `landing → exit` transition for visitors who leave without browsing at all, a different and simpler bounce shape than the one built here. * **A/B testing** — use [tags](/docs/core/concepts/plugins#tags-input-plugins) to label timestamps as variant A or B, then branch the FSM based on `has_tags`. * **Multi-device sessions** — run two generators in `startup.yml` with different user agent pools (mobile vs. desktop) writing to the same table. * **Real-time dashboards** — connect Grafana to ClickHouse and build a live funnel dashboard showing conversion rates as events stream in. What's next [#whats-next] FAQ [#faq] A session that lands, browses, and leaves without ever adding anything to the cart. `browse.jinja` tracks how many pages the current visitor has viewed since the session started, or since the last cart addition, and sets `session_done` once that count reaches 5 with the cart still empty, sending the FSM to `exit` instead of `add-to-cart` on the next check. A visitor who leaves from the landing page itself, before browsing at all, is a different and simpler bounce shape that this generator doesn't model directly. So conversion timing varies between visitors instead of landing on the same page count every time — the reasoning is in `browse.jinja` above. What the chance keeps is the deterministic ceiling: once a visitor's streak hits 5, the funnel forces the outcome, so no session browses forever without either converting or bouncing. No. One `fsm` machine holds exactly one current session in `shared` state — the same limitation [Modeling sessions](/docs/tutorials/realism/sessions) has for its login/action/logout cycle — so sessions here play out one after another rather than concurrently. Modeling many visitors browsing at once needs the pool-in-state pattern from [Correlated events](/docs/tutorials/realism/correlated-events) instead: a dict of in-flight sessions keyed by session id, with each render advancing one entry from the pool. The query shown above already groups by `page_type` and counts distinct `session_id` values per stage; the conversion rate from landing to checkout is the `checkout` row's session count divided by the `landing` row's. Every event in a session carries that session's `session_id`, so the same grouping works for any stage pair — landing to add-to-cart, or add-to-cart to checkout — without a separate query per transition. Related [#related] * The [Modeling sessions](/docs/tutorials/realism/sessions) lesson for the finite-state-machine and state-flag technique this funnel reuses, applied there to a three-stage login/action/logout cycle instead of a five-stage browsing session * The [Correlated events](/docs/tutorials/realism/correlated-events) lesson for tracking many concurrent visitors in a pool instead of one session at a time * The [Generate test data for ClickHouse](/docs/tutorials/delivery/clickhouse) lesson for the HTTP insert mechanics, connection pooling, and `generateRandom` comparison behind this output * The [Streaming vs bulk](/docs/tutorials/foundations/streaming-vs-bulk) lesson for live mode's continuous feed versus the bounded sample-mode batch used to inspect this generator * The [Scenarios track](/docs/tutorials/scenarios) for the broader set of scenarios synthetic data solves * The [Eventum Hub](/hub) for generators that already model common funnels, instead of starting the FSM from a blank state # List Generators {/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */} List ids of all generators # List Generator Dirs {/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */} List all generator directory names inside `path.generators_dir` with generator configs. # List Repositories {/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */} List connected generator repositories, each with the result of the last check made in this process. A password naming a keyring secret is answered as written, while one holding the credential itself is answered redacted. # Add Repository {/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */} Connect a generator repository. The repository is checked before it is connected, and the catalog it publishes is read on the first request for it. # List Secret Names {/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */} List all secrets names # List Scenarios {/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */} List all scenarios # Get Generators In Startup {/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */} Get list of generator definitions in the startup file # $ eventum generate Runs a single generator without the application server. This is the quickest way to test a generator or produce events in a script or CI pipeline. All execution parameters that would normally come from [eventum.yml](/docs/core/config/eventum-yml) and [startup.yml](/docs/core/config/startup-yml) are passed as CLI flags instead. ```bash eventum generate [OPTIONS] ``` Options [#options] Generator [#generator] | Option | Type | Default | Required | Description | | ------------- | ------- | ------- | -------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `--id` | string | — | Yes | Unique identifier for the generator. Used in log messages. | | `--path` | path | — | Yes | Path to the generator config file (`generator.yml`). | | `--live-mode` | boolean | `true` | No | `true` — generate events at their scheduled timestamps in real time. `false` — generate all events as fast as possible (sample mode). | | `--skip-past` | boolean | `true` | No | In live mode, skip timestamps that have already passed when the generator starts. When `false`, past timestamps are generated immediately before catching up to real time. | | `--params` | JSON | `{}` | No | JSON object of [parameters](/docs/core/config/parameters) to substitute into the generator config. Example: `--params '{"host": "localhost", "port": 9200}'`. | Generation parameters [#generation-parameters] These correspond to the `generation` section of [eventum.yml](/docs/core/config/eventum-yml#generation): | Option | Type | Default | Description | | ------------------------------- | ------- | ------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `--timezone` | string | `"UTC"` | IANA timezone for timestamp generation (e.g. `UTC`, `America/New_York`, `Europe/London`). | | `--batch.size` | integer | `10000` | Maximum number of events per batch. At least one of `--batch.size` or `--batch.delay` must be set. | | `--batch.delay` | float | `1.0` | Maximum time span, in seconds, of the event timestamps one batch covers — see [Batching](/docs/core/concepts/generator#batching) for when it applies. At least one of `--batch.size` or `--batch.delay` must be set. | | `--queue.max-timestamp-batches` | integer | `10` | Maximum timestamp batches in the input→event queue. | | `--queue.max-event-batches` | integer | `10` | Maximum event batches in the event→output queue. | | `--keep-order` | boolean | `false` | Process output batches sequentially to preserve chronological order. Disables concurrent writes. | | `--max-concurrency` | integer | `100` | Maximum concurrent write operations across all output plugins. | | `--write-timeout` | integer | `10` | Timeout in seconds for a single write operation. | Logging and security [#logging-and-security] | Option | Type | Default | Description | | --------------- | ----- | ------- | ------------------------------------------------------------------------------------------------------------------------------ | | `-v, --verbose` | count | `0` | Verbosity level for log output. Repeat the flag to increase (up to 5 times). Logs are written to stderr. | | `--cryptfile` | path | — | Path to the keyring cryptfile for [secret](/docs/core/config/secrets) retrieval. If omitted, uses the system default location. | Verbosity levels [#verbosity-levels] | Flag | Level | What gets logged | | -------- | -------- | ------------------------------------------------------------ | | *(none)* | Disabled | No log output | | `-v` | CRITICAL | Only critical errors | | `-vv` | ERROR | Errors and critical | | `-vvv` | WARNING | Warnings, errors, and critical | | `-vvvv` | INFO | Informational messages and above (recommended for debugging) | | `-vvvvv` | DEBUG | Everything, including internal details | Logging is disabled by default so that log messages don't mix with generated events on stdout. Pass `-v` one or more times to enable logs — they are written to stderr. Signal handling [#signal-handling] | Signal | Behavior | | ----------------- | ----------------------------------------- | | `SIGINT` (Ctrl+C) | Stops the generator gracefully and exits. | | `SIGTERM` | Stops the generator gracefully and exits. | Exit codes [#exit-codes] | Code | Meaning | | ----- | -------------------------------------------------------------------------- | | `0` | Generator completed successfully. | | `1` | Generator failed to start (invalid config, missing secrets, plugin error). | | `130` | Terminated by SIGINT (128 + 2). | | `143` | Terminated by SIGTERM (128 + 15). | Examples [#examples] Minimal invocation — generate events in real time with console output: ```bash eventum generate --id my-gen --path ./my-generator/generator.yml ``` Sample mode with verbose logging — generate all events at once: ```bash eventum generate \ --id test-gen \ --path ./generators/access-logs/generator.yml \ --live-mode false \ -vvvv ``` Custom timezone and batch size: ```bash eventum generate \ --id web-logs \ --path ./generators/web/generator.yml \ --timezone America/New_York \ --batch.size 50000 \ --batch.delay 2.0 ``` With secrets from a keyring cryptfile: ```bash eventum generate \ --id prod-gen \ --path ./generators/prod/generator.yml \ --cryptfile ./cryptfile.cfg \ -vvvv ``` With parameters to inject values into the generator config: ```bash eventum generate \ --id web-logs \ --path ./generators/web/generator.yml \ --params '{"opensearch_host": "https://localhost:9200", "index_name": "dev-logs"}' ``` Ordered output for deterministic results: ```bash eventum generate \ --id ordered-gen \ --path ./generator.yml \ --keep-order true \ --max-concurrency 10 \ --write-timeout 30 ``` Pipe output to another tool: ```bash eventum generate \ --id pipe-gen \ --path ./generator.yml \ --live-mode false | jq '.' ``` When piping output, make sure the generator uses the [`stdout`](/docs/plugins/output/stdout) output plugin. Logs go to stderr (when enabled with `-v`), so they won't interfere with piped event data. # $ eventum-keyring A standalone tool for managing secrets stored in an encrypted keyring file. Secrets added here are available in generator configs via `${secrets.name}` tokens. See [Secrets](/docs/core/config/secrets) for the full picture of how secrets work in Eventum. `eventum-keyring` is a separate executable from `eventum` — it is installed alongside it but invoked independently. ```bash eventum-keyring [OPTIONS] ``` Commands [#commands] set [#set] Stores or updates a secret in the keyring. ```bash eventum-keyring set [] [--cryptfile ] ``` | Argument | Required | Description | | -------- | -------- | --------------------------------------------------------------------------------------- | | `name` | Yes | Name of the secret. This is the key used in `${secrets.name}` tokens. | | `value` | No | Secret value. If omitted, you are prompted to enter it interactively (input is hidden). | | Option | Type | Description | | ------------- | ---- | ---------------------------------------------------------------------------------------------------------------- | | `--cryptfile` | path | Path to the cryptfile. If it doesn't exist, a new file is created. If omitted, uses the system default location. | ```bash # Inline value eventum-keyring set db_password "s3cret" # Interactive prompt (hidden input) eventum-keyring set db_password # Enter password of `db_password`: ******** # Done # Custom cryptfile location eventum-keyring set api_key "tok_abc123" --cryptfile ./project/cryptfile.cfg ``` Prints `Done` to stderr on success. *** get [#get] Retrieves a secret from the keyring and prints it to stdout. ```bash eventum-keyring get [--cryptfile ] ``` | Argument | Required | Description | | -------- | -------- | ------------------------------- | | `name` | Yes | Name of the secret to retrieve. | | Option | Type | Description | | ------------- | ---- | -------------------------------------------------------------------------------- | | `--cryptfile` | path | Path to the cryptfile. Must exist. If omitted, uses the system default location. | ```bash eventum-keyring get db_password # s3cret eventum-keyring get api_key --cryptfile ./project/cryptfile.cfg # tok_abc123 ``` If the secret doesn't exist, prints an error and exits with code 1. *** remove [#remove] Deletes a secret from the keyring. ```bash eventum-keyring remove [--cryptfile ] ``` | Argument | Required | Description | | -------- | -------- | ----------------------------- | | `name` | Yes | Name of the secret to delete. | | Option | Type | Description | | ------------- | ---- | -------------------------------------------------------------------------------- | | `--cryptfile` | path | Path to the cryptfile. Must exist. If omitted, uses the system default location. | ```bash eventum-keyring remove db_password # Done ``` Prints `Done` to stderr on success. If the secret doesn't exist, prints an error and exits with code 1. Exit codes [#exit-codes] All three commands share the same exit code conventions: | Code | Meaning | | ---- | ------------------------------------------------------------------------------- | | `0` | Operation completed successfully. | | `1` | Error — secret not found, blank name, cryptfile access issue, or other failure. | Environment variables [#environment-variables] EVENTUM_KEYRING_PASSWORD [#eventum_keyring_password] The encryption password used to read and write the cryptfile. Set this before running any keyring command: ```bash export EVENTUM_KEYRING_PASSWORD="your-strong-password" ``` If the variable is not set, Eventum uses the default password `eventum` and logs a warning. For production use, always set a custom password. The same `EVENTUM_KEYRING_PASSWORD` must be set when running `eventum generate` or `eventum run` — otherwise Eventum cannot decrypt the secrets stored in the cryptfile. The cryptfile [#the-cryptfile] The cryptfile is an AES-encrypted file managed by [keyrings.cryptfile](https://pypi.org/project/keyrings.cryptfile/). All secrets are stored under the service name `eventum`. The `--cryptfile` flag on each command controls which file is used. This should match the path configured elsewhere: | Context | Where the path is set | | ------------------ | ----------------------------------------------------------------------------- | | `eventum run` | `path.keyring_cryptfile` in [eventum.yml](/docs/core/config/eventum-yml#path) | | `eventum generate` | `--cryptfile` flag | | `eventum-keyring` | `--cryptfile` flag on each subcommand | When `--cryptfile` is omitted, all three tools fall back to the system default keyring location. Examples [#examples] Setting up secrets for a project: ```bash export EVENTUM_KEYRING_PASSWORD="project-key" # Store credentials eventum-keyring set opensearch_password "prod-password" --cryptfile ./cryptfile.cfg eventum-keyring set ch_password "clickhouse-secret" --cryptfile ./cryptfile.cfg eventum-keyring set auth_password "admin-password" --cryptfile ./cryptfile.cfg # Verify eventum-keyring get opensearch_password --cryptfile ./cryptfile.cfg # prod-password ``` Then reference them in configs: ```yaml title="generator.yml" output: - opensearch: hosts: - https://opensearch:9200 username: admin password: ${secrets.opensearch_password} ``` ```bash eventum generate \ --id my-gen \ --path ./generator.yml \ --cryptfile ./cryptfile.cfg ``` Rotating a secret: ```bash # Update the value — same name overwrites the previous secret eventum-keyring set opensearch_password "new-password" --cryptfile ./cryptfile.cfg # Restart or hot-reload to pick up the change kill -HUP $(pgrep -f "eventum run") ``` # $ eventum mcp Runs Eventum as an [MCP server](/docs/mcp) over standard input/output — the transport every MCP client supports. An agent connected to it can discover plugins, write and read generator files, and validate and preview generators against the real engine. For live generator management — start, stop, register, unregister, and read logs — mount the server over HTTP instead; see [Connect your agent](/docs/mcp/connect#live-management-over-http). ```bash eventum mcp --generators-dir [OPTIONS] ``` `stdout` is reserved for the protocol stream: all logs go to `stderr`, and no startup banner is printed, so the channel stays clean for the client. Options [#options] | Option | Type | Default | Description | | --------------------- | ------ | -------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `--generators-dir` | path | — | **Required.** Directory holding generator subdirectories. The agent reads and writes generators here. Must be an existing directory. | | `--config-filename` | string | `generator.yml` | Name of the config file inside each generator directory. Set it when your generators use a non-default filename. | | `--repositories` | path | `repositories.yml` beside the generators directory | File listing the [connected repositories](/docs/studio/repositories) the agent reads and installs generators from. The default is where an instance keeps it, since that file lives next to the startup file. | | `--read-only` | flag | `false` | Block all writes. The agent can discover, validate, and preview, but cannot create or modify files. | | `--keyring-cryptfile` | path | — | Path to the [keyring](/docs/core/config/secrets) cryptfile. When set, `list_secret_names` reports the **names** of stored secrets (never their values); without it, `list_secret_names` returns nothing. | | `--log-level` | choice | `WARNING` | Log verbosity on `stderr`: `DEBUG`, `INFO`, `WARNING`, `ERROR`, or `CRITICAL`. | Examples [#examples] Author generators in a local directory: ```bash eventum mcp --generators-dir ./generators ``` Expose generators read-only — discovery, validation, and preview, but no changes: ```bash eventum mcp --generators-dir ./generators --read-only ``` Read the connected repositories from a file of your own: ```bash eventum mcp --generators-dir ./generators --repositories ./repositories.yml ``` Point at a keyring so the agent can list the names of stored secrets: ```bash eventum mcp --generators-dir ./generators --keyring-cryptfile ./cryptfile.cfg ``` Register it from a client's command line (for example, with Claude Code): ```bash claude mcp add eventum -- uv run eventum mcp --generators-dir ./generators ``` See [Connect your agent](/docs/mcp/connect) for the config-file format used by most clients. See also [#see-also] * [MCP overview](/docs/mcp) — what the server is and what it's for. * [Tools & resources](/docs/mcp/tools) — everything the agent can call. * [`server.mcp`](/docs/core/config/eventum-yml#server-mcp) — enabling the HTTP transport on a running server. # $ eventum run Starts Eventum as a long-running application. This mode launches the web server (UI and REST API), loads all generators defined in [startup.yml](/docs/core/config/startup-yml), and keeps running until stopped. Use this for production deployments and any scenario where you need multiple generators, runtime management, or monitoring. ```bash eventum run -c ``` Options [#options] | Option | Type | Default | Required | Description | | -------------- | ---- | ------- | -------- | ------------------------------------------------------------------------------------------------------------- | | `-c, --config` | path | — | Yes | Path to the main [eventum.yml](/docs/core/config/eventum-yml) configuration file. Must exist and be readable. | All other settings — server, logging, generation defaults, file paths — are defined inside the config file rather than as CLI flags. See the [eventum.yml](/docs/core/config/eventum-yml) reference for the full schema. What happens on startup [#what-happens-on-startup] 1. The application loads all generators listed in [startup.yml](/docs/core/config/startup-yml). 2. Generators with `autostart: true` (the default) begin generating events immediately. Generators with `autostart: false` are registered but remain idle until started via the API or UI. 3. The web server starts on the configured `host:port`, exposing the UI and REST API. Signal handling [#signal-handling] | Signal | Behavior | | ---------------------------- | ------------------------------------------------------------------------------------------------------------------------------------ | | `SIGTERM`, `SIGINT` (Ctrl+C) | Stops all generators and the server, then exits. | | `SIGHUP` | **Hot reload** — stops the whole application and starts it again from the config file and the startup list. The process stays alive. | `SIGHUP` applies a config change without replacing the process. The server stops with everything else, so the API and Studio are unreachable for a moment: ```bash # Edit eventum.yml or startup.yml, then: kill -HUP $(pgrep -f "eventum run") ``` Exit codes [#exit-codes] | Code | Meaning | | ---- | -------------------------------------------------------------------------------------------------------- | | `0` | Normal shutdown (not typically seen — signal-based termination uses code 1). | | `1` | Error during startup (invalid config, YAML parse error, validation failure) or signal-based termination. | Error messages [#error-messages] **YAML parsing failure:** ``` Error: Failed to parse configuration YAML content:
``` **Schema validation failure:** ``` Error: Failed to validate settings: --server.port: Input should be a valid integer --log.level: Input should be 'debug', 'info', 'warning', 'error' or 'critical' ``` Option names in validation errors use `--` prefix and kebab-case to match CLI conventions. Examples [#examples] Basic usage: ```bash eventum run -c eventum.yml ``` Running in the background: ```bash eventum run -c eventum.yml & # Later, hot reload after config changes: kill -HUP %1 # Graceful shutdown: kill %1 ``` # $ eventum service Manages Eventum as a systemd service. This command group handles the full lifecycle — creating directories, generating default configs, installing a systemd unit file, and tearing everything down. Designed for production deployments where Eventum should start on boot and run continuously via [`eventum run`](/docs/core/cli/eventum-run). ```bash eventum service [OPTIONS] ``` Mode detection [#mode-detection] Eventum automatically picks system or user mode based on who runs the command: | Context | Mode | Config dir | Log dir | Unit file | | ------------------- | ------ | -------------------- | ------------------------------ | ---------------------------------------- | | Running as root | System | `/etc/eventum/` | `/var/log/eventum/` | `/etc/systemd/system/eventum.service` | | Running as non-root | User | `~/.config/eventum/` | `~/.local/state/eventum/logs/` | `~/.config/systemd/user/eventum.service` | | Root with `--user` | User | (same as above) | (same as above) | (same as above) | The `--user` flag forces user mode even when running as root. Commands [#commands] install [#install] Creates directories, generates default configuration files, and installs a systemd unit. ```bash eventum service install [--config-dir ] [--log-dir ] [--user] [--no-ask] ``` | Option | Type | Default | Description | | -------------- | ---- | ------------------------------------- | ----------------------------------------------------------------- | | `--config-dir` | path | See [mode detection](#mode-detection) | Directory for configuration files. | | `--log-dir` | path | See [mode detection](#mode-detection) | Directory for log files. | | `--user` | flag | `false` | Install as a user service even when running as root. | | `--no-ask` | flag | `false` | Skip all prompts — use defaults and proceed without confirmation. | When `--config-dir` or `--log-dir` are omitted and `--no-ask` is not set, the installer prompts for each directory with the default pre-filled. **What it creates:** | Path | Contents | | ---------------------------- | ------------------------------------------------------------------------------ | | `/` | Configuration directory | | `/generators/` | Generator configs (referenced by [startup.yml](/docs/core/config/startup-yml)) | | `/eventum.yml` | Main config with sensible defaults — see [generated config](#generated-config) | | `/startup.yml` | Empty generator list (`[]`) | | `/cryptfile.cfg` | Empty keyring cryptfile | | `/` | Log directory | | Unit file | systemd service unit | Existing files are never overwritten. If `eventum.yml` or `startup.yml` already exist, they are preserved and a message is printed. After installation, the service is **not started automatically**. Follow the printed next steps to enable and start it. **Interactive example:** ```bash $ eventum service install Configuration directory [/etc/eventum]: Log directory [/var/log/eventum]: Eventum service will be installed with the following settings: Mode: system Binary: /usr/local/bin/eventum Config dir: /etc/eventum Log dir: /var/log/eventum Unit file: /etc/systemd/system/eventum.service Proceed? [Y/n]: y Created /etc/eventum/ Created /etc/eventum/generators/ Created /var/log/eventum/ Generated /etc/eventum/eventum.yml Generated /etc/eventum/startup.yml Created /etc/eventum/cryptfile.cfg Installed /etc/systemd/system/eventum.service Reloaded systemd daemon Done! Next steps: 1. Review configuration: cat /etc/eventum/eventum.yml 2. Enable on boot: sudo systemctl enable eventum 3. Start the service: sudo systemctl start eventum 4. Check status: eventum service status ``` **Non-interactive (scripted):** ```bash sudo eventum service install \ --config-dir /opt/eventum/config \ --log-dir /opt/eventum/logs \ --no-ask ``` *** uninstall [#uninstall] Stops the service, removes the systemd unit file, and optionally purges configuration and log directories. ```bash eventum service uninstall [--user] [--purge] ``` | Option | Type | Default | Description | | --------- | ---- | ------- | ------------------------------------------------------------------------- | | `--user` | flag | `false` | Uninstall the user service even when running as root. | | `--purge` | flag | `false` | Also remove configuration and log directories (prompts for confirmation). | Without `--purge`, configuration files are preserved and a hint is printed showing how to remove them later. ```bash $ sudo eventum service uninstall Stopping eventum service... Disabling eventum service... Removed /etc/systemd/system/eventum.service Reloaded systemd daemon Done! Configuration files preserved in /etc/eventum/ To also remove configuration and logs, run: sudo eventum service uninstall --purge ``` With `--purge`, each directory is confirmed individually: ```bash $ sudo eventum service uninstall --purge Stopping eventum service... Disabling eventum service... Removed /etc/systemd/system/eventum.service Reloaded systemd daemon Remove configuration directory /etc/eventum/ and all its contents? [y/N]: y Removed /etc/eventum/ Remove log directory /var/log/eventum/ and all its contents? [y/N]: y Removed /var/log/eventum/ Done! ``` *** status [#status] Shows the current state of the Eventum service. ```bash eventum service status [--user] ``` | Option | Type | Default | Description | | -------- | ---- | ------- | ------------------------------------------------- | | `--user` | flag | `false` | Check the user service even when running as root. | ```bash $ eventum service status Service: installed Unit: /etc/systemd/system/eventum.service State: active (running) Enabled: yes Config: /etc/eventum/eventum.yml ``` If the service is not installed: ``` Service: not installed ``` Generated config [#generated-config] The `install` command generates an [eventum.yml](/docs/core/config/eventum-yml) with all required sections and sensible defaults: ```yaml title="eventum.yml" # Eventum Configuration # Generated by: eventum service install # Full reference: https://eventum.run/docs/configuration server.ui.enabled: true server.api.enabled: true server.host: 0.0.0.0 server.port: 9474 server.auth.user: eventum server.auth.password: eventum generation.timezone: UTC log.level: info log.format: plain path.logs: /var/log/eventum path.startup: /etc/eventum/startup.yml path.generators_dir: /etc/eventum/generators path.keyring_cryptfile: /etc/eventum/cryptfile.cfg ``` The default credentials (`eventum`/`eventum`) are for initial setup only. Change `server.auth.user` and `server.auth.password` before exposing the server to a network. Path values are resolved to absolute paths based on the directories chosen during installation. The config uses [dot notation](/docs/core/config/eventum-yml) which Eventum expands to nested YAML at load time. Systemd unit [#systemd-unit] The generated unit file runs `eventum run -c ` as a `Type=simple` service: * **Restart** on failure with a 5-second delay * **Hot reload** via `systemctl reload eventum` (sends SIGHUP — see [signal handling](/docs/core/cli/eventum-run#signal-handling)) * **Journal logging** — view with `journalctl -u eventum` Standard systemd commands work as expected: ```bash # Start / stop / restart sudo systemctl start eventum sudo systemctl stop eventum sudo systemctl restart eventum # Hot reload (re-reads config without downtime) sudo systemctl reload eventum # Enable on boot sudo systemctl enable eventum # View logs journalctl -u eventum -f ``` For user services, add `--user` and drop `sudo`: ```bash systemctl --user start eventum journalctl --user -u eventum -f ``` Exit codes [#exit-codes] | Code | Meaning | | ---- | ------------------------------------------------------------------------------------------ | | `0` | Command completed successfully. | | `1` | Error — service not installed, permission denied, systemd not available, or other failure. | # eventum.yml The main application config controls how Eventum runs as a service with `eventum run`. It defines four sections: **server** settings, file **paths**, **logging**, and default **generation** parameters that all generators inherit. ```yaml title="eventum.yml" server: host: "0.0.0.0" port: 9474 path: startup: /etc/eventum/startup.yml generators_dir: /etc/eventum/generators logs: /var/log/eventum keyring_cryptfile: /etc/eventum/cryptfile.cfg log: level: info format: plain generation: timezone: UTC batch: size: 10000 ``` The config supports both **nested YAML** and **dot notation**. For example, `server.host: "0.0.0.0"` is equivalent to the nested `server: { host: "0.0.0.0" }`. Both formats can be mixed in the same file. Extra fields are **forbidden** — any unrecognized key will cause a validation error at load time. *** server [#server] Controls the built-in web server that exposes the UI and REST API. | Parameter | Type | Default | Constraints | Description | | -------------------- | ------- | ----------- | ----------- | ----------------------------------- | | `server.ui.enabled` | boolean | `true` | — | Enable the web-based management UI. | | `server.api.enabled` | boolean | `true` | — | Enable the REST API endpoints. | | `server.host` | string | `"0.0.0.0"` | Non-empty. | Address the server binds to. | | `server.port` | integer | `9474` | >= 1 | Port the server binds to. | server.ssl [#serverssl] TLS/SSL settings for the server. | Parameter | Type | Default | Constraints | Description | | ------------------------ | -------------- | ------- | -------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `server.ssl.enabled` | boolean | `false` | — | Enable HTTPS. When `true`, `cert` and `cert_key` are required. | | `server.ssl.verify_mode` | string or null | `null` | `"none"`, `"optional"`, or `"required"` | Client certificate verification mode. `"none"` — no client certificate required. `"optional"` — request but don't require. `"required"` — reject connections without a valid client certificate. | | `server.ssl.ca_cert` | path or null | `null` | Must be an absolute path. | CA certificate for verifying client certificates. | | `server.ssl.cert` | path or null | `null` | Must be an absolute path. Required when SSL is enabled. Must be provided together with `cert_key`. | Server certificate file. | | `server.ssl.cert_key` | path or null | `null` | Must be an absolute path. Required when SSL is enabled. Must be provided together with `cert`. | Server certificate private key file. | All SSL paths must be **absolute**. ```yaml server: ssl: enabled: true cert: /etc/eventum/server.crt cert_key: /etc/eventum/server.key verify_mode: optional ca_cert: /etc/eventum/ca.crt ``` server.auth [#serverauth] HTTP basic authentication credentials for the UI and API. | Parameter | Type | Default | Constraints | Description | | ---------------------- | ------ | ----------- | ----------- | ----------- | | `server.auth.user` | string | `"eventum"` | Non-empty. | Username. | | `server.auth.password` | string | `"eventum"` | Non-empty. | Password. | Change the default credentials before exposing Eventum on a network. server.mcp [#servermcp] Mounts the [MCP server](/docs/mcp) into the running server so an AI agent can build, validate, preview, and manage generators over HTTP. Disabled by default. The endpoint is served at `:` and protected by the [`server.auth`](#server-auth) credentials. | Parameter | Type | Default | Constraints | Description | | -------------------------- | --------------- | -------- | ----------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `server.mcp.enabled` | boolean | `false` | — | Mount the MCP server over HTTP. | | `server.mcp.allow_write` | boolean | `false` | — | Allow write tools — file writes and deletes, one-shot runs, and starting, stopping, registering, and unregistering generators. When `false`, the endpoint is read-only: discovery, validation, preview, and live status/log reads, but no changes. Enabling writes grants code execution on the host (see the warning below). | | `server.mcp.path` | string | `"/mcp"` | Leading `/`, no trailing `/`. | Mount path for the MCP endpoint. | | `server.mcp.allowed_hosts` | list of strings | `[]` | — | Allowed `Host` header values (DNS-rebinding protection). Empty disables the check — suitable behind a trusted reverse proxy; a non-empty list enables it and rejects other hosts. Allowed `Origin` values are derived from this list, so browser-based clients on a listed host pass while foreign origins are rejected. | ```yaml server: mcp: enabled: true allow_write: false path: /mcp allowed_hosts: [] ``` `allow_write` over HTTP grants code execution: a connected agent can write a generator and preview it, and some plugins execute code on the host by design (the `template` plugin runs arbitrary Python; a `script` plugin runs a script you provide). Enable it only on a trusted network with a trusted agent, and change the default `server.auth` credentials first. Local authoring doesn't need the server — run [`eventum mcp`](/docs/core/cli/eventum-mcp) for a stdio server next to your agent. See [Connect your agent](/docs/mcp/connect). *** path [#path] File system paths used by the application. All paths must be **absolute**. | Parameter | Type | Default | Constraints | Description | | -------------------------------- | ---- | ----------------- | ------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------- | | `path.logs` | path | — | Required. Must be an absolute path. | Directory for application log files. Created automatically if it doesn't exist. | | `path.startup` | path | — | Required. Must be an absolute path. | Path to the [startup.yml](/docs/core/config/startup-yml) file. | | `path.generators_dir` | path | — | Required. Must be an absolute path. | Directory containing generator subdirectories. Generator paths in `startup.yml` are resolved relative to this directory. | | `path.keyring_cryptfile` | path | — | Required. Must be an absolute path. | Path to the encrypted keyring file used for [secrets](/docs/core/config/secrets). | | `path.repositories` | path | `null` | Must be an absolute path when set. | Path to the file listing the [connected repositories](/docs/studio/repositories). Defaults to `repositories.yml` next to `path.startup`. | | `path.generator_config_filename` | path | `"generator.yml"` | Single filename. Must end with `.yml` or `.yaml`. | The expected config filename inside each generator directory. Used by the API to auto-detect valid generator directories. | ```yaml path: startup: /etc/eventum/startup.yml generators_dir: /etc/eventum/generators logs: /var/log/eventum keyring_cryptfile: /etc/eventum/cryptfile.cfg generator_config_filename: generator.yml ``` *** log [#log] Controls application logging — separate from the events that generators produce. | Parameter | Type | Default | Constraints | Description | | ----------------------- | ------- | ------------------- | ------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `log.level` | string | `"info"` | `"debug"`, `"info"`, `"warning"`, `"error"`, or `"critical"` | Minimum severity level for log messages. | | `log.third_party_level` | string | `"warning"` | `"debug"`, `"info"`, `"warning"`, `"error"`, or `"critical"` | Minimum severity level for messages of third-party libraries. Independent of `log.level`, so running Eventum at `"debug"` doesn't bring the debug output of every dependency with it. | | `log.format` | string | `"plain"` | `"plain"` or `"json"` | Log output format. `"plain"` is human-readable, `"json"` is structured. | | `log.max_bytes` | integer | `10485760` (10 MiB) | >= 1024 | Maximum size per log file before rotation. | | `log.backups` | integer | `5` | >= 1 | Number of rotated log files to keep. | ```yaml log: level: info third_party_level: warning format: json max_bytes: 52428800 # 50 MiB backups: 10 ``` Log files [#log-files] Each message goes to one file in `path.logs`, named after the part of the application it came from. Standard output receives all of them, so `docker logs` stays the combined view. | File | Contents | | -------------------- | --------------------------------------------------------------------------------- | | `main.log` | Application startup, shutdown and everything not attributed below. | | `server.log` | REST API and HTTP server, including its startup and errors. | | `server_access.log` | One line per HTTP request. | | `mcp.log` | MCP server. | | `generator_.log` | Everything one generator does, including the delivery its output plugins perform. | With `log.format: json` the same files carry the `.json` extension instead. *** generation [#generation] Default generation parameters inherited by all generators. Individual generators can override any of these in [startup.yml](/docs/core/config/startup-yml#overridable-generation-parameters). | Parameter | Type | Default | Constraints | Description | | --------------------- | ------ | ------- | ----------------------------------- | ------------------------------------------- | | `generation.timezone` | string | `"UTC"` | Valid IANA timezone. Min length: 3. | Default timezone for generating timestamps. | generation.batch [#generationbatch] Controls how events are grouped before being passed to output plugins: | Parameter | Type | Default | Constraints | Description | | ------------------------ | --------------- | --------- | ----------- | ------------------------------------------------------------------------ | | `generation.batch.size` | integer or null | `10000`\* | >= 1 | Maximum number of events per batch. | | `generation.batch.delay` | float or null | `1.0`\* | >= 0.1 | Maximum time span, in seconds, of the event timestamps one batch covers. | \*The defaults of `10000` and `1.0` apply as a pair when **neither** field is specified. If you set only one, the other defaults to `null`. At least one of `size` or `delay` must be set. `size` is the primary limit. `delay` bounds the lag batching adds to delivery in [live mode](/docs/core/concepts/generator#live-mode-default), so it forms the batches of timestamps that are still ahead of real time; timestamps that have already passed, and every timestamp in [sample mode](/docs/core/concepts/generator#sample-mode), are grouped by `size`. With no `size` set, `delay` is the only limit on how large a batch grows and therefore always applies — see [Batching](/docs/core/concepts/generator#batching) for the full breakdown. generation.queue [#generationqueue] Controls the internal queues between pipeline stages. These act as backpressure buffers — when a downstream stage is slower, the upstream stage will block once the queue is full. | Parameter | Type | Default | Constraints | Description | | ---------------------------------------- | --------------- | ----------- | ----------- | --------------------------------------------------------------------------------------------------------------------------- | | `generation.queue.max_timestamp_batches` | integer | `10` | >= 1 | Maximum timestamp batches in the input→event queue. | | `generation.queue.max_event_batches` | integer | `10` | >= 1 | Maximum event batches in the event→output queue. | | `generation.queue.max_event_bytes` | integer or null | `268435456` | >= 1 | Maximum memory the event batches in the event→output queue occupy together, 256 MiB by default. `null` leaves it unlimited. | A batch is as large as the events in it, so the number of batches alone does not bound how much the queue holds: ten batches of ten thousand events come to twenty megabytes with small events and half a gigabyte with large ones. Both limits of the events queue apply, whichever is reached first, so `max_event_bytes` keeps that figure predictable no matter what the templates render. A single batch larger than the whole limit still passes through — holding it back would stall the pipeline — so keep `batch.size` in proportion to the size of an event. What a generator occupies while it produces and writes a batch is a separate figure, and it follows `batch.size` — see [how much memory a generator holds](/docs/core/concepts/generator#how-much-memory-a-generator-holds). The memory each queue currently holds is reported on the instance page next to the batches it holds. Concurrency and ordering [#concurrency-and-ordering] | Parameter | Type | Default | Constraints | Description | | ---------------------------- | ------- | ------- | ----------- | ----------------------------------------------------------------------------------------- | | `generation.keep_order` | boolean | `false` | — | When `true`, output plugins process batches sequentially to preserve chronological order. | | `generation.max_concurrency` | integer | `100` | >= 1 | Maximum number of concurrent write operations across all output plugins. | | `generation.write_timeout` | integer | `10` | >= 1 | Timeout in seconds for a single write operation. | ```yaml generation: timezone: America/New_York batch: size: 20000 delay: 2.0 queue: max_timestamp_batches: 20 max_event_batches: 20 max_event_bytes: 536870912 keep_order: false max_concurrency: 200 write_timeout: 30 ``` *** Complete example [#complete-example] ```yaml title="eventum.yml" # Server server: ui: enabled: true api: enabled: true host: "0.0.0.0" port: 9474 ssl: enabled: true cert: /etc/eventum/server.crt cert_key: /etc/eventum/server.key verify_mode: none auth: user: admin password: s3cret # Paths path: startup: /etc/eventum/startup.yml generators_dir: /etc/eventum/generators logs: /var/log/eventum keyring_cryptfile: /etc/eventum/cryptfile.cfg # Logging log: level: info format: json max_bytes: 52428800 backups: 10 # Default generation parameters generation: timezone: UTC batch: size: 10000 delay: 1.0 queue: max_timestamp_batches: 10 max_event_batches: 10 max_event_bytes: 268435456 keep_order: false max_concurrency: 100 write_timeout: 10 ``` What's next [#whats-next] # generator.yml The generator config file defines the three-stage pipeline that produces events. It has exactly three top-level keys — `input`, `event`, and `output` — each configuring one stage: ```yaml title="generator.yml" input: - : event: : output: - : ``` All relative paths inside the file are resolved from the **directory containing `generator.yml`**, not from the working directory. This makes generator directories fully portable. See [Project structure — Path resolution](/docs/core/config/project-structure#path-resolution) for details. Extra fields are **forbidden** — any unrecognized key will cause a validation error at load time. An AI agent can fetch this document's machine-readable JSON Schema over MCP through the [`eventum://schema/generator`](/docs/mcp/tools#resources) resource. Top-level structure [#top-level-structure] | Key | Type | Required | Description | | -------- | ---------------------- | -------- | ----------------------------------------------------- | | `input` | list of plugin configs | Yes | One or more input plugins. At least one is required. | | `event` | single plugin config | Yes | Exactly one event plugin. | | `output` | list of plugin configs | Yes | One or more output plugins. At least one is required. | Each plugin config is a single-key mapping where the key is the plugin name and the value is a mapping of plugin-specific parameters: ```yaml - cron: # plugin name expression: "* * * * * *" # plugin parameter count: 1 # plugin parameter ``` Variable substitution [#variable-substitution] Generator config file supports **variable substitution** using `${ }` syntax. This lets you parameterize configs without hardcoding values: ```yaml title="generator.yml" output: - opensearch: hosts: - ${params.opensearch_host} username: ${params.opensearch_user} password: ${secrets.opensearch_password} ``` Two types of variables are available: | Syntax | Source | Example | | ----------------- | --------------------------------------------------------- | ------------------------ | | `${params.name}` | Passed via CLI `--params` or `startup.yml` `params` field | `${params.host}` | | `${secrets.name}` | Loaded from the encrypted keyring | `${secrets.db_password}` | See [Parameters](/docs/core/config/parameters) and [Secrets](/docs/core/config/secrets) for details on how to pass and manage these values. Input plugins [#input-plugins] Input plugins define **when** events are generated. Multiple input plugins can be combined — their timestamp streams are merged chronologically. Every input plugin supports an optional `tags` field. See the [input plugin reference](/docs/plugins#input-plugins) for all available plugins and their parameters. *** Event plugins [#event-plugins] Event plugins define **what** events look like. Exactly one event plugin is configured per generator. See the [event plugin reference](/docs/plugins#event-plugins) for all available plugins and their parameters. *** Output plugins [#output-plugins] Output plugins define **where** events are sent. Multiple output plugins can be configured — every event is delivered to all of them (fan-out). Every output plugin supports a [formatter](/docs/plugins/formatters). See the [output plugin reference](/docs/plugins#output-plugins) for all available plugins and their parameters. *** Versatile datetime [#versatile-datetime] Several input plugins accept datetime values in a flexible format called **versatile datetime**. The following formats are accepted: | Format | Example | Description | | ------------------- | -------------------------------------------- | ---------------------------------------------------------------------------------------------- | | ISO 8601 | `"2024-06-15T14:30:00"` | Standard datetime string. | | Human-readable | `"June 15, 2024 2:30 PM"`, `"yesterday 3pm"` | Parsed with [dateparser](https://dateparser.readthedocs.io/). | | Time of day | `"14:30:00"` | Interpreted as today at the given time. | | `now` | `now` | Current time at evaluation. | | `never` | `never` | No time limit (used for `end`). | | Relative expression | `+1h`, `-30m`, `+1d12h30m` | Offset from `start` (for `end`) or from current time (for `start`). Units: `d`, `h`, `m`, `s`. | See [Scheduling — Date ranges and versatile datetime](/docs/core/concepts/scheduling#date-ranges-and-versatile-datetime) for more details and examples. *** Complete example [#complete-example] ```yaml title="generator.yml" input: - cron: expression: "* * * * * *" count: 1 tags: [web] - cron: expression: "* * * * * */10" count: 1 tags: [error] event: template: mode: chance samples: users: type: csv source: samples/users.csv header: true templates: - access: template: templates/access.jinja chance: 95 - error: template: templates/error.jinja chance: 5 output: - stdout: formatter: format: plain - file: path: output/events.jsonl formatter: format: json - opensearch: hosts: - ${params.opensearch_host} username: ${params.opensearch_user} password: ${secrets.opensearch_password} index: logs ``` # Parameters Parameters let you inject values into a [generator config](/docs/core/config/generator-yml) at load time without editing the YAML file itself. A single generator directory can serve development, staging, and production by swapping the parameters passed to it. How it works [#how-it-works] Any value in `generator.yml` can contain a `${params.name}` token. Before the YAML is parsed, Eventum replaces every token with the matching value from the parameters you provide: ```yaml title="generator.yml" output: - opensearch: hosts: - ${params.opensearch_host} username: ${params.opensearch_user} index: ${params.index_name} ``` The substitution uses [Jinja2](https://jinja.palletsprojects.com/) under the hood with `${` / `}` delimiters, so the full Jinja2 expression syntax is available if needed. Providing parameters [#providing-parameters] Via startup.yml [#via-startupyml] When running with [`eventum run`](/docs/core/cli/eventum-run), parameters are set per generator in the `params` field of each [startup.yml](/docs/core/config/startup-yml) entry: ```yaml title="startup.yml" - id: access-logs path: access-logs/generator.yml params: opensearch_host: https://opensearch.prod:9200 opensearch_user: admin index_name: access-logs ``` Each key in `params` maps to a `${params.}` token in the generator config. Parameters can also be grouped, and the token then names the path to the value: ```yaml title="startup.yml" - id: access-logs path: access-logs/generator.yml params: opensearch: host: https://opensearch.prod:9200 user: admin ``` The generator config reads them as `${params.opensearch.host}` and `${params.opensearch.user}`. A parameter named with a dot in it — `opensearch.host: https://opensearch.prod:9200` — is read by the same token. Via CLI flags [#via-cli-flags] When running with [`eventum generate`](/docs/core/cli/eventum-generate), pass parameters as a JSON object with the `--params` flag: ```bash eventum generate \ --id access-logs \ --path access-logs/generator.yml \ --params '{"opensearch_host": "https://localhost:9200", "opensearch_user": "admin", "index_name": "dev-logs"}' ``` The JSON keys map to the same `${params.}` tokens, and nested objects are read by the same path form. This is useful for one-off runs and CI pipelines where you don't want to create a `startup.yml`. Type handling [#type-handling] Parameter values retain the type defined in YAML. Standard YAML type inference applies: `8443` is an integer, `"8443"` is a string, `true` is a boolean. Since substitution happens as text replacement before YAML parsing, the final type depends on what the YAML parser sees after substitution. In practice this rarely matters, but if you need an exact type, quote accordingly in `startup.yml`. For example, to ensure a port is parsed as an integer: ```yaml params: port: 8443 # integer port_str: "8443" # string enabled: true # boolean ``` Missing parameters [#missing-parameters] If a generator config references `${params.name}` but no matching parameter is provided, Eventum raises an error at load time and the generator does not start: ``` Error: Failed to obtain params used in configuration Reason: Parameters {'index_name'} are missing ``` This is intentional — it prevents generators from running with incomplete configuration. When to use parameters [#when-to-use-parameters] **Environment-specific values** — hosts, ports, index names, file paths that differ between dev and prod: ```yaml title="startup.yml" # Development - id: web-logs path: web-logs/generator.yml params: opensearch_host: https://localhost:9200 index_name: dev-logs # Production - id: web-logs path: web-logs/generator.yml params: opensearch_host: https://opensearch.prod:9200 index_name: prod-logs ``` **Shared generators with different targets** — the same generator directory reused for different outputs: ```yaml title="startup.yml" - id: logs-to-file path: web-logs/generator.yml params: output_path: /tmp/events.jsonl - id: logs-to-opensearch path: web-logs/generator.yml params: output_path: /var/log/events.jsonl ``` **Tunable thresholds** — values you want to adjust without editing the generator: ```yaml title="generator.yml" input: - cron: expression: "${params.schedule}" count: 1 output: - http: url: ${params.endpoint} success_code: ${params.expected_code} ``` For sensitive values like passwords and API keys, use [secrets](/docs/core/config/secrets) instead of parameters. Parameters are stored in plain text in `startup.yml`. # Project structure Eventum has two levels of configuration: the **application** level (for `eventum run`) and the **generator** level (for both `eventum run` and `eventum generate`). Understanding the structure makes it easier to navigate the config pages that follow. Application level [#application-level] When running Eventum as a full application with `eventum run`, the project root contains two config files and a directory of generators: | File | Purpose | | ---------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | [`eventum.yml`](/docs/core/config/eventum-yml) | Main application config — server settings, default generation parameters, logging, and file paths. | | [`startup.yml`](/docs/core/config/startup-yml) | List of generators to load at startup, with per-generator overrides for execution parameters. | | `repositories.yml` | Repositories of ready-made generators the instance is [connected to](/docs/studio/repositories). Written when the first one is connected — an instance that connects none never has it. Its location is `path.repositories`, defaulting to the directory of `startup.yml`. | | `generators/` | Directory containing generator subdirectories. Each subdirectory holds a self-contained generator. | | `logs/` | Application log files (path configurable in `eventum.yml`). | How the files relate [#how-the-files-relate] `eventum.yml` points to `startup.yml` and to the generators directory: ```yaml title="eventum.yml" path: startup: /opt/eventum/startup.yml generators_dir: /opt/eventum/generators logs: /opt/eventum/logs keyring_cryptfile: /opt/eventum/cryptfile.cfg ``` All `path.*` values must be absolute paths. `startup.yml` lists which generators to start and can override any generation parameter from `eventum.yml` on a per-generator basis: ```yaml title="startup.yml" - id: access-logs path: access-logs/generator.yml # relative to generators_dir autostart: true timezone: America/New_York - id: error-logs path: error-logs/generator.yml live_mode: false # override: run in sample mode batch: size: 50000 # override: larger batches ``` Parameters cascade: `eventum.yml` sets the defaults → `startup.yml` overrides per generator. Any field not specified in `startup.yml` inherits from `eventum.yml`. Generator level [#generator-level] A generator is a self-contained directory with a config file and its supporting resources: Only `generator.yml` is required. The other directories depend on which plugins you use: | Directory | Used by | Contains | | ------------ | --------------------------------- | ------------------------------ | | `templates/` | `template` event plugin | Jinja2 `.jinja` files | | `scripts/` | `script` event plugin | Python `.py` files | | `samples/` | `template` event plugin (samples) | CSV, JSON, or other data files | | `patterns/` | `time_patterns` input plugin | Time pattern YAML definitions | generator.yml [#generatoryml] The generator config defines the three-stage pipeline — input, event, and output: ```yaml title="generator.yml" input: - cron: expression: "* * * * * *" count: 1 event: template: mode: all templates: - access_log: template: templates/access.jinja output: - stdout: {} - file: path: output/events.log ``` See the [generator.yml](/docs/core/config/generator-yml) reference for the full schema. Path resolution [#path-resolution] All relative paths inside `generator.yml` are resolved from the **directory containing the config file**, not from the working directory. This means a generator directory is fully portable — you can move it anywhere and it works as long as its internal structure is intact. For example, if `generator.yml` is at `/opt/eventum/generators/access-logs/generator.yml`, then `templates/access.jinja` resolves to `/opt/eventum/generators/access-logs/templates/access.jinja`. Single-generator mode [#single-generator-mode] When using `eventum generate`, you don't need `eventum.yml` or `startup.yml` — just a generator directory with its config: ```bash eventum generate --id my-gen --path my-generator/generator.yml ... ``` All execution parameters (timezone, batch size, live mode, etc.) are passed as CLI flags instead of being defined in `startup.yml`. See the [`eventum generate`](/docs/core/cli/eventum-generate) reference for all available flags. What's next [#whats-next] # Secrets Secrets provide a secure way to inject sensitive values — passwords, API keys, tokens — into generator configs without storing them in plain text. Values are kept in an encrypted keyring file and referenced in configs with `${secrets.name}` tokens. How it works [#how-it-works] Secrets follow the same substitution model as [parameters](/docs/core/config/parameters), but the values come from an encrypted file instead of `startup.yml` or the `--params` CLI flag: ```yaml title="generator.yml" output: - opensearch: hosts: - ${params.opensearch_host} username: ${params.opensearch_user} password: ${secrets.opensearch_password} # from the keyring ``` When Eventum loads the config, it extracts all `${secrets.*}` tokens, looks up each name in the encrypted keyring, decrypts the values, and substitutes them into the config before YAML parsing. Renaming a secret on the [Secrets](/docs/studio/settings#secrets) page rewrites the token in every config that reads it, so the reference follows the name. The rewrite is textual — comments and formatting are left as they were. The keyring and cryptfile [#the-keyring-and-cryptfile] Secrets are stored in an encrypted file called the **cryptfile**, managed by the [keyrings.cryptfile](https://pypi.org/project/keyrings.cryptfile/) library. The cryptfile is AES-encrypted and protected by a password. Keyring password [#keyring-password] The keyring password is read from the `EVENTUM_KEYRING_PASSWORD` environment variable. If the variable is not set, Eventum falls back to the default password `eventum` and logs a warning. For production use, always set a custom password: ```bash export EVENTUM_KEYRING_PASSWORD="your-strong-password" ``` Cryptfile location [#cryptfile-location] The cryptfile path depends on how you run Eventum: | Mode | How the path is set | | ------------------ | ------------------------------------------------------------------------------------------------------ | | `eventum run` | `path.keyring_cryptfile` in [eventum.yml](/docs/core/config/eventum-yml#path) | | `eventum generate` | `--cryptfile` CLI flag. If omitted, uses the system default location. | | `eventum-keyring` | `--cryptfile` flag on each subcommand. If omitted, uses the system default location. | | `eventum mcp` | `--keyring-cryptfile` flag. Enables the agent's `list_secret_names` to report stored secret **names**. | An [MCP](/docs/mcp) agent can call `list_secret_names` to discover which secret **names** exist — names only, never values — so it references them correctly in a config. It cannot add or read secret values; manage those with `eventum-keyring` (below). Renaming one is the exception - an agent may move a value to another name without seeing it. Point the stdio server at the cryptfile with [`eventum mcp --keyring-cryptfile`](/docs/core/cli/eventum-mcp). Naming a secret [#naming-a-secret] A configuration reads a secret as `${secrets.}` and is rendered to substitute it, so the name is read as an expression: words of lowercase letters, digits and `_`, separated by `.`, each word starting with a letter or `_`. | Name | Accepted | | ----------------------------------------------------- | -------- | | `db_password`, `aws.access_key`, `_internal` | Yes | | `db-password`, `1token`, `my key`, `a/b`, `API_TOKEN` | No | Lowercase is part of the rule rather than a convention: the keyring stores a name in a form that does not keep its case, while the value is encrypted against the name as it was given, so a name holding a capital would be listed in a spelling that cannot read it back. A name outside that could be stored but never read back, so it is refused wherever a secret is written — the CLI, the API, Studio and the rename tool of an [MCP](/docs/mcp) agent. Names already in a keyring are left as they are; rename one to bring it back into use. Managing secrets with eventum-keyring [#managing-secrets-with-eventum-keyring] The `eventum-keyring` CLI tool lets you add, read, and remove secrets from the cryptfile. Set a secret [#set-a-secret] ```bash # Provide the value inline eventum-keyring set db_password "s3cret" # Or omit the value to be prompted interactively (input is hidden) eventum-keyring set db_password # Enter password of `db_password`: ******** ``` Get a secret [#get-a-secret] ```bash eventum-keyring get db_password # s3cret ``` Remove a secret [#remove-a-secret] ```bash eventum-keyring remove db_password ``` Custom cryptfile location [#custom-cryptfile-location] Every subcommand accepts a `--cryptfile` flag to work with a specific file instead of the system default: ```bash eventum-keyring set api_key "tok_abc123" --cryptfile ./my-project/cryptfile.cfg eventum-keyring get api_key --cryptfile ./my-project/cryptfile.cfg ``` This is the same file referenced by `path.keyring_cryptfile` in `eventum.yml` or `--cryptfile` on `eventum generate`. Using secrets in configs [#using-secrets-in-configs] Reference secrets with `${secrets.name}` anywhere in `generator.yml`: ```yaml title="generator.yml" output: - http: url: https://api.example.com/ingest headers: Authorization: "Bearer ${secrets.api_token}" - opensearch: hosts: - https://opensearch.prod:9200 username: admin password: ${secrets.opensearch_password} - clickhouse: host: clickhouse.prod username: ${secrets.ch_user} password: ${secrets.ch_password} ``` Using secrets in the repositories file [#using-secrets-in-the-repositories-file] The password of a [connected repository](/docs/studio/repositories) is read the same way. Written as a reference, it is resolved from the keyring every time the repository is fetched, so the token itself never enters the file: ```yaml title="repositories.yml" - name: internal url: https://git.example.com/platform/generators.git username: eventum password: ${secrets.internal_git_token} ``` Missing secrets [#missing-secrets] If a config references `${secrets.name}` but the secret is not in the keyring, Eventum raises an error at load time and the generator does not start: ``` Error: Failed to obtain secrets used in configuration Reason: Cannot obtain secret `api_token`: Secret is missing ``` This prevents generators from running with empty credentials. When to use secrets [#when-to-use-secrets] **Credentials and sensitive data** — passwords for OpenSearch, ClickHouse; API keys and tokens for authentication to HTTP endpoints or any other output destination: ```bash eventum-keyring set opensearch_password "prod-password-here" eventum-keyring set ch_password "clickhouse-secret" eventum-keyring set api_token "tok_abc123xyz" ``` ```yaml title="generator.yml" output: - opensearch: password: ${secrets.opensearch_password} - clickhouse: password: ${secrets.ch_password} - http: url: https://api.example.com/events headers: Authorization: "Bearer ${secrets.api_token}" ``` Secrets vs parameters [#secrets-vs-parameters] | | Parameters | Secrets | | ---------------- | ----------------------------------------------- | --------------------------- | | **Storage** | Plain text in `startup.yml` | Encrypted in the cryptfile | | **Syntax** | `${params.name}` | `${secrets.name}` | | **Managed with** | `startup.yml` or `--params` CLI flag | `eventum-keyring` CLI | | **Nesting** | Flat or grouped, read as `${params.group.name}` | Flat key-value pairs | | **Use for** | Hosts, ports, paths, thresholds | Passwords, API keys, tokens | For the full CLI reference see [`$ eventum-keyring`](/docs/core/cli/eventum-keyring) command page. # startup.yml The startup file is a YAML list of generators that Eventum loads when it starts with `eventum run`. Each entry identifies a generator directory and can override any [generation parameter](#overridable-generation-parameters) from [eventum.yml](/docs/core/config/eventum-yml). ```yaml title="startup.yml" - id: access-logs path: access-logs/generator.yml autostart: true - id: error-logs path: error-logs/generator.yml live_mode: false batch: size: 50000 ``` Parameter cascade [#parameter-cascade] Startup entries inherit generation parameters from `eventum.yml` and can override them individually. The cascade works in one direction: ``` eventum.yml (defaults) → startup.yml (per-generator overrides) ``` Any field not specified in a startup entry inherits the value from `eventum.yml`. This means you only need to specify what differs from the defaults. For example, if `eventum.yml` sets `timezone: UTC` and `batch.size: 10000`, a startup entry with `batch.size: 50000` will use UTC timezone (inherited) but a batch size of 50000 (overridden). Generator entry fields [#generator-entry-fields] Core fields [#core-fields] | Parameter | Type | Default | Constraints | Description | | ----------- | --------------- | ------- | ------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `id` | string | — | Required. Non-empty. Must be unique across all entries. | Unique identifier for the generator. Used in API endpoints and logs. | | `path` | path | — | Required. | Path to the generator config file. Relative paths are resolved against the `generators_dir` from [eventum.yml](/docs/core/config/eventum-yml#path). | | `autostart` | boolean | `true` | — | Whether to start this generator automatically when the application launches. If `false`, the generator is registered but must be started manually via the API. | | `scenarios` | list of strings | `[]` | — | Scenario names this generator belongs to. Scenarios group generators for coordinated lifecycle control and global state visualization in [Studio](/docs/studio/scenarios). A generator can belong to multiple scenarios. | Execution mode [#execution-mode] | Parameter | Type | Default | Constraints | Description | | ----------- | ------- | ------- | ----------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `live_mode` | boolean | `true` | — | `true` — events are generated at their scheduled timestamps in real time (live mode). `false` — all events are generated as fast as possible without waiting (sample mode). | | `skip_past` | boolean | `true` | — | In live mode, whether to skip timestamps that have already passed when the generator starts. When `false`, Eventum will generate events for past timestamps immediately before catching up to real time. | Variable substitution parameters [#variable-substitution-parameters] | Parameter | Type | Default | Description | | --------- | ------- | ------- | --------------------------------------------------------------------------------------------------------- | | `params` | mapping | `{}` | Key-value pairs substituted into the generator config via `${params.name}` tokens. Can be flat or nested. | ```yaml - id: web-logs path: web-logs/generator.yml params: opensearch_host: https://opensearch.prod:9200 opensearch_user: admin log_level: info ``` These values replace `${params.opensearch_host}`, `${params.opensearch_user}`, and `${params.log_level}` in the generator's `generator.yml`. See [Parameters](/docs/core/config/parameters) for more details. Overridable generation parameters [#overridable-generation-parameters] These fields correspond to the `generation` section of [eventum.yml](/docs/core/config/eventum-yml#generation). Any of them can be set per generator in startup.yml to override the application-level defaults. timezone [#timezone] | Parameter | Type | Default | Constraints | Description | | ---------- | ------ | ------- | -------------------------------------------------------------------------------------- | --------------------------------------------------------- | | `timezone` | string | `"UTC"` | Valid IANA timezone (e.g., `UTC`, `America/New_York`, `Europe/London`). Min length: 3. | Timezone used for interpreting and generating timestamps. | batch [#batch] Controls how events are grouped before being passed to output plugins: | Parameter | Type | Default | Constraints | Description | | ------------- | --------------- | --------- | ----------- | ------------------------------------------------------------------------ | | `batch.size` | integer or null | `10000`\* | >= 1 | Maximum number of events per batch. | | `batch.delay` | float or null | `1.0`\* | >= 0.1 | Maximum time span, in seconds, of the event timestamps one batch covers. | \*The defaults of `10000` and `1.0` apply as a pair when **neither** field is specified. If you set only one, the other defaults to `null`. At least one of `size` or `delay` must be set. When both are set, the batch is flushed when either limit is reached — whichever comes first. See [eventum.yml — Generation](/docs/core/config/eventum-yml#generation) for what `delay` bounds while a generator catches up on past timestamps. queue [#queue] Controls the internal queues between pipeline stages: | Parameter | Type | Default | Constraints | Description | | ----------------------------- | --------------- | ----------- | ----------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `queue.max_timestamp_batches` | integer | `10` | >= 1 | Maximum number of timestamp batches held in the input→event queue. Controls backpressure when the event plugin is slower than input. | | `queue.max_event_batches` | integer | `10` | >= 1 | Maximum number of event batches held in the event→output queue. Controls backpressure when output plugins are slower than event processing. | | `queue.max_event_bytes` | integer or null | `268435456` | >= 1 | Maximum memory the event batches in the event→output queue occupy together, 256 MiB by default. Applies alongside the limit on their number, whichever is reached first. `null` leaves it unlimited. | Concurrency and ordering [#concurrency-and-ordering] | Parameter | Type | Default | Constraints | Description | | ----------------- | ------- | ------- | ----------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `keep_order` | boolean | `false` | — | When `true`, output plugins process batches sequentially to preserve chronological order. When `false`, batches may be written concurrently and arrive out of order. | | `max_concurrency` | integer | `100` | >= 1 | Maximum number of concurrent write operations across all output plugins. Acts as a semaphore to prevent overloading destinations. | | `write_timeout` | integer | `10` | >= 1 | Timeout in seconds for a single write operation. If an output plugin takes longer than this, the write is cancelled. | Format variants [#format-variants] Startup entries support both **nested YAML** and **dot notation** for overriding nested parameters: Both formats produce identical results — use whichever is more readable for your case. ```yaml title="Nested YAML" - id: my-generator path: my-generator/generator.yml batch: size: 5000 delay: 0.5 queue: max_timestamp_batches: 20 ``` ```yaml title="Dot notation" - id: my-generator path: my-generator/generator.yml batch.size: 5000 batch.delay: 0.5 queue.max_timestamp_batches: 20 ``` Complete example [#complete-example] ```yaml title="startup.yml" # High-throughput access log generator — sample mode, large batches - id: access-logs path: access-logs/generator.yml autostart: true live_mode: false timezone: America/New_York batch: size: 50000 max_concurrency: 50 write_timeout: 30 scenarios: - web-traffic params: opensearch_host: https://opensearch.prod:9200 opensearch_user: admin # Real-time error log generator — live mode, ordered output - id: error-logs path: error-logs/generator.yml autostart: true live_mode: true skip_past: true keep_order: true max_concurrency: 10 scenarios: - web-traffic - alerting params: severity: error # Metrics generator — registered but not started automatically - id: metrics path: metrics/generator.yml autostart: false timezone: UTC ``` # Generator A **generator** is the central building block of Eventum. It is a self-contained unit that knows *when* to produce events, *what* those events look like, and *where* to send them. You can run a single generator from the command line or dozens in parallel through the application server. The three-stage pipeline [#the-three-stage-pipeline] Every generator runs a pipeline with three stages connected in sequence: ``` Input → Event → Output ``` Each stage is handled by a [plugin](/docs/core/concepts/plugins) — a swappable component that you pick and configure in YAML. The stages are deliberately independent: changing when events happen doesn't affect what they contain, and adding a new destination doesn't require touching the schedule or the template. Input — *when* events happen [#input--when-events-happen] An input plugin produces **timestamps** — the moments in time when events should occur. Different [input plugins](/docs/plugins#input-plugins) offer different scheduling strategies: cron expressions, fixed intervals, evenly spaced ranges, statistical distributions, and more. A generator can have **multiple input plugins** working simultaneously. Their timestamps are merged into a single stream before reaching the event stage. Each input plugin can attach **tags** to its timestamps so the event stage knows which source produced them. See [Scheduling](/docs/core/concepts/scheduling) for the full reference. Event — *what* events look like [#event--what-events-look-like] An event plugin takes each timestamp and turns it into one or more event strings. Eventum provides three event plugins: * **template** — renders [Jinja2](https://jinja.palletsprojects.com/) templates with access to [Faker](https://faker.readthedocs.io/), [Mimesis](https://mimesis.name/), random helpers, sample datasets, and persistent state. This is the primary way to generate events and covers most use cases. * **script** — executes a Python function, giving you full programmatic control when templates aren't enough. * **replay** — reads events from an existing log file and optionally replaces timestamps. A generator has exactly **one event plugin**. See [Producing events](/docs/core/concepts/producing) for the full reference. Output — *where* events go [#output--where-events-go] An [output plugin](/docs/plugins#output-plugins) receives rendered events and writes them to a destination — the console, a local file, an HTTP endpoint, or a database. A generator can have **multiple output plugins**. Every event is delivered to all configured outputs — you don't need to duplicate generators to write the same data to a file and an HTTP endpoint. Each output plugin can apply a [formatter](/docs/plugins/formatters) to transform events before writing. See [Outputting events](/docs/core/concepts/output) for the full reference. Generator configuration [#generator-configuration] A generator is defined by a single YAML file with three top-level keys matching the pipeline stages: ```yaml title="generator.yml" input: - cron: expression: "* * * * * *" count: 1 event: template: mode: all templates: - my_event: template: templates/event.jinja output: - stdout: {} - file: path: output/events.log flush_interval: 1 ``` * **`input`** is a list — you can combine multiple input plugins. * **`event`** is a single object — one plugin with its settings. * **`output`** is a list — events go to every output in the list. Each plugin is identified by its name (`cron`, `template`, `stdout`, etc.) and configured with a nested object of plugin-specific settings. See [Configuration files](/docs/core/config/files) for the full reference. Variable substitution [#variable-substitution] Generator configs support `${params.name}` and `${secrets.name}` tokens. Parameters are passed at runtime (via CLI flags or the startup config), while secrets are resolved from the encrypted [keyring](/docs/core/cli/eventum-keyring). This lets you reuse the same config across environments without hard-coding values: ```yaml output: - opensearch: hosts: - ${params.opensearch_host} password: ${secrets.opensearch_password} ``` See [Variables](/docs/core/config/vars) for details. Execution modes [#execution-modes] A generator can run in one of two modes: Live mode (default) [#live-mode-default] Events are emitted **at the actual moments** defined by their timestamps, synchronized with the wall clock. If the input plugin says "event at 12:00:05", the generator waits until 12:00:05 to produce and deliver it. This mode is designed for: * Simulating real-time traffic against a live system * Feeding a SIEM or monitoring tool with a continuous event stream * Stress-testing a pipeline at a controlled, realistic rate By default, timestamps in the past are skipped (`skip_past: true`), so the generator starts producing from "now" forward. Sample mode [#sample-mode] All events are generated **as fast as possible**, regardless of what the timestamps say. A cron expression that would take an hour in live mode can produce its full output in seconds. This mode is designed for: * Seeding a database with historical data * Creating test datasets * Backfilling a time range after an outage Switch between modes with a single flag: ```bash # Live mode (default) eventum generate --id my-gen --path generator.yml ... --live-mode # Sample mode eventum generate --id my-gen --path generator.yml ... --live-mode false ``` How it works under the hood [#how-it-works-under-the-hood] Understanding the internal flow isn't required to use Eventum, but it helps when tuning performance or debugging. A generator runs its pipeline in a dedicated thread. Inside that thread, an async executor manages three concurrent tasks connected by two queues: 1. The **input task** collects timestamps from all input plugins, merges them in chronological order, groups them into batches, and (in live mode) waits until the right moment before releasing each batch into the timestamps queue. 2. The **event task** reads timestamp batches, calls the event plugin's `produce` function for each timestamp, and puts the resulting event strings into the events queue. 3. The **output task** reads event batches and writes them to all output plugins concurrently, respecting a configurable concurrency limit. The two queues provide natural **backpressure**: if the event plugin is slower than the input, the timestamps queue fills up and the input task pauses. If the output is slower than the event plugin, the events queue fills up and event production pauses. This prevents memory from growing unboundedly without dropping events. How much memory a generator holds [#how-much-memory-a-generator-holds] The memory a generator occupies follows the size of a batch rather than the rate of events. A batch is in more than one place at once — one is produced while the previous one is written — and each destination formats its own copy of what it delivers, so the total comes to several times the size of a single batch. For events of a few hundred bytes and the default batch of 10 000, a generator writing JSON to one destination settles around 20 MiB above what the process itself needs, and every further destination adds about 6 MiB. A batch of 1 000 events brings the whole figure under 2 MiB. Lower `batch.size` when a generator has to fit into a smaller footprint — `queue.max_event_bytes` bounds a different part, the batches that pile up in the queue when a destination stops keeping up. The memory reported for a process is the largest amount it has needed so far, so the figure climbs in steps while the first batches pass through and then stays level. A generator that reaches that level and holds it is behaving as expected. Batching [#batching] Timestamps travel through the pipeline in batches, and a batch is the unit of work for everything downstream: one pass of event rendering, one write per output plugin. Two parameters decide where a batch ends — `batch.size`, how many timestamps it holds, and `batch.delay`, the time span of the timestamps it covers. `size` is the primary limit. What `delay` buys is a bound on the lag batching adds to delivery: a batch is released once its last timestamp comes due, so the first event in it waits at most `delay` seconds past its own moment. Where nothing is waited for, `delay` has no lag to bound and would only cut the run into batches the size of the event rate, so the batch is formed by `size`: | Configuration | Mode and situation | What ends the batch | | ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------- | | `size` only | any mode | `size` is reached | | `size` and `delay` | live, timestamps still ahead of real time | whichever of the two limits is reached first | | `size` and `delay` | live, timestamps that have already passed — a generator started on a past range with `skip_past` disabled, or one running behind real time | `size` is reached | | `size` and `delay` | sample mode | `size` is reached | | `delay` only | any mode | the `delay` window — with no `size` set it is the only limit on how large a batch grows | Tuning parameters [#tuning-parameters] | Parameter | Default | What it controls | | ----------------------------- | ----------- | -------------------------------------------------------------------------------------------------------------------------------------------------- | | `batch.size` | `10000` | Max timestamps per batch — larger batches reduce overhead but increase latency | | `batch.delay` | `1.0` | Max time span of the timestamps one batch covers — applies to timestamps still ahead of real time in live mode, everything else is grouped by size | | `queue.max_timestamp_batches` | `10` | Timestamps queue depth — more buffering between input and event stages | | `queue.max_event_batches` | `10` | Events queue depth — more buffering between event and output stages | | `queue.max_event_bytes` | `268435456` | Memory the events queue may hold, 256 MiB by default — `null` leaves it unlimited | | `max_concurrency` | `100` | Max parallel output write operations | | `write_timeout` | `10` | Seconds before canceling a slow output write | | `keep_order` | `false` | When `true`, output writes are serialized to maintain strict chronological order (slower but ordered) | These parameters are set either via CLI flags (`eventum generate --batch.size 5000`) or in the application config under `generation.*`. See [eventum.yml](/docs/core/config/files/eventum-yml) for details. Running multiple generators [#running-multiple-generators] When you need more than one generator — different event types, different schedules, different destinations — use `eventum run` to launch the full application. Each generator runs in its own thread with independent pipelines, queues, and plugins. They don't share state unless you explicitly use the [`globals`](/docs/plugins/event/template/state) state, available to templates and scripts alike. The [startup config](/docs/core/config/files/startup-yml) lists all generators and lets you override parameters per generator: ```yaml title="startup.yml" - id: access-logs path: generators/access-logs/generator.yml live_mode: true - id: error-logs path: generators/error-logs/generator.yml live_mode: true - id: backfill path: generators/backfill/generator.yml autostart: false live_mode: false ``` Generators with `autostart: false` are loaded but not started — you can start them later through the [REST API](/docs/api) or [Studio](/docs/studio). What's next [#whats-next] # Outputting events Output plugins are the final stage of the [generator pipeline](/docs/core/concepts/generator#the-three-stage-pipeline). They receive event strings from the event plugin and write them to a destination — the console, a file, an HTTP endpoint, or a database. This page covers the mechanics that make output flexible: formatters, fan-out delivery, concurrency, and error handling. For per-plugin parameters and configuration, see the [output plugin reference](/docs/plugins#output-plugins). Fan-out delivery [#fan-out-delivery] A generator can have multiple output plugins. Every event is delivered to **all** of them — there is no filtering or routing at the output stage. If you need different events to go to different destinations, use separate generators with different event plugins. This design is intentional: the output stage is a fan-out, not a router. It's common to combine a local file (for debugging or archival) with a remote destination (for production use): ```yaml output: # Local debug log - file: path: output/events.jsonl formatter: format: json # Production endpoint - opensearch: hosts: - https://opensearch.example.com:9200 index: events username: admin password: ${secrets.opensearch_password} ``` Both plugins receive every event independently. If one fails, the other is not affected. Formatters [#formatters] Before an event reaches the plugin's write method, it passes through a **formatter** that transforms the raw event string. Formats range from plain pass-through to JSON encoding and custom Jinja2 templates. Each plugin ships with a sensible default. You only need to set the `formatter` field when you want something different. For the full list of formats and their parameters, see the [Formatters reference](/docs/plugins/formatters). Async lifecycle [#async-lifecycle] Output plugins follow an **asynchronous lifecycle** — `open`, `write`, and `close` are all async. This allows multiple plugins to write concurrently without blocking each other. The lifecycle runs once per generator execution: 1. **`open`** — Establishes connections, opens file handles, resets metrics. 2. **`write`** (repeated) — Receives a batch of formatted events and writes them to the destination. 3. **`close`** — Flushes buffers, closes connections, releases resources. Formatting itself also runs off the main thread, so a slow formatter doesn't block other plugins. Concurrency [#concurrency] When a generator has multiple output plugins, their writes run **concurrently** by default. Two parameters control this behavior: **`max_concurrency`** — limits the total number of concurrent write operations across all output plugins. Defaults to `100`. If your destinations can't handle high parallelism, lower this value: ```bash eventum generate ... --max-concurrency 10 ``` **`keep_order`** — when set to `true`, the generator waits for all output plugins to finish writing a batch before starting the next one. This guarantees that events arrive at every destination in chronological order, at the cost of throughput: ```bash eventum generate ... --keep-order ``` With `keep_order: false` (the default), plugins write independently — a fast plugin won't wait for a slow one. Error tracking [#error-tracking] Each output plugin independently tracks three counters: | Counter | What it counts | | ------------------ | ---------------------------------------------------------------------- | | **written** | Events successfully delivered | | **write\_failed** | Events that failed to write (network error, rejected by destination) | | **format\_failed** | Events that failed formatting (e.g. invalid JSON for `json` formatter) | A formatting failure skips the event but doesn't stop the batch. A write failure is logged but doesn't affect other output plugins. These counters are available through the [REST API](/docs/api) and [Studio](/docs/studio) for monitoring. What's next [#whats-next] # Plugins Everything a generator does — scheduling timestamps, producing events, writing output — is handled by **plugins**. A plugin is a self-contained component that implements one stage of the pipeline. You pick a plugin by name, configure it in YAML, and Eventum takes care of the rest. Plugin types [#plugin-types] Plugins are grouped into three types, one per pipeline stage: | Type | Role | Cardinality | | ---------- | ------------------------------------------------------------- | ------------------------- | | **Input** | Produces timestamps — *when* events happen | One or more per generator | | **Event** | Turns timestamps into event strings — *what* events look like | Exactly one per generator | | **Output** | Writes events to a destination — *where* events go | One or more per generator | Each type has its own set of available plugins. You can mix and match within a type (e.g. two input plugins feeding the same generator), but you cannot use a plugin outside its type. Input plugins [#input-plugins] Input plugins produce timestamps using different scheduling strategies — cron expressions, fixed intervals, evenly spaced ranges, statistical distributions, and more. A generator can combine **multiple input plugins**; their timestamps are merged into a single chronological stream before reaching the event stage. Event plugins [#event-plugins] Event plugins turn timestamps into event strings. The primary plugin renders Jinja2 templates; alternatives include running a Python function or replaying events from an existing log file. A generator has exactly **one event plugin**. Output plugins [#output-plugins] Output plugins write events to a destination — the console, a local file, an HTTP endpoint, or a database. A generator can have **multiple output plugins**; every event is delivered to all of them. For the full list of available plugins and their parameters, see the [Plugins](/docs/plugins) reference. Configuration [#configuration] Each plugin is configured as a named key inside the corresponding pipeline section. The key is the plugin name; the value is an object with plugin-specific settings: ```yaml title="generator.yml" input: - cron: expression: "* * * * * *" count: 1 event: template: mode: all templates: - my_event: template: templates/event.jinja output: - stdout: {} ``` A few rules: * **`input`** and **`output`** are lists — each item is a single-key object naming the plugin. * **`event`** is a single object — one plugin name with its config. * Plugin-specific fields are validated at startup. An unknown field or a wrong type causes a clear error before any events are generated. * Relative file paths (e.g. `templates/event.jinja`) are resolved from the directory containing the generator config file. Empty configs [#empty-configs] Some plugins require no settings. Use an empty object: ```yaml output: - stdout: {} ``` Common properties [#common-properties] While each plugin has its own settings, some properties are shared across all plugins of a type. Tags (input plugins) [#tags-input-plugins] Any input plugin can attach **tags** — arbitrary string labels — to its timestamps. Tags are carried through the pipeline and are available inside the event plugin (as the `tags` variable in templates or the `tags` field in script params). This lets a single event plugin produce different output depending on which input triggered it: ```yaml input: - cron: expression: "* * * * * */5" count: 1 tags: [heartbeat] - cron: expression: "* * * * *" count: 1 tags: [minutely] ``` Formatter (output plugins) [#formatter-output-plugins] Every output plugin has a **formatter** that controls how event strings are shaped before writing. The formatter is configured with the `formatter` field: ```yaml output: - file: path: output/events.log formatter: format: json ``` For the full list of formats and parameters, see the [Formatters reference](/docs/plugins/formatters). Lifecycle [#lifecycle] Plugins go through a predictable lifecycle managed by the generator: 1. **Configuration** — The YAML is parsed and validated against the plugin's schema. Invalid config produces an error before anything runs. 2. **Instantiation** — A plugin instance is created with the validated config. Each plugin receives a sequential ID (useful for logging) and a unique GUID. 3. **Execution** — The generator calls the plugin's core method repeatedly: * Input: `generate` — yields batches of timestamps * Event: `produce` — turns a timestamp (and tags) into event strings * Output: `open` → `write` (repeated) → `close` 4. **Shutdown** — Output plugins are closed gracefully (flushing buffers, closing connections). Input and event plugins require no explicit cleanup. Output plugins follow an **async lifecycle** — `open`, `write`, and `close` are asynchronous, which allows concurrent writes to multiple destinations without blocking. Input and event plugins run synchronously. Error handling [#error-handling] Eventum tracks errors at the plugin level without crashing the pipeline: * If an **event plugin** fails to produce for a given timestamp, the failure is counted, logged and the pipeline moves on to the next timestamp. * If an **output plugin** fails to write a batch, the failure is counted and logged. Other output plugins in the same generator are not affected. * If a **formatter** cannot format an event, the event is skipped for that output. The failure count is tracked separately from write failures. Error counts are exposed through the [REST API](/docs/api) and [Studio](/docs/studio), so you can monitor plugin health at runtime. Choosing a plugin [#choosing-a-plugin] The [Plugins reference](/docs/plugins) lists every available plugin with its parameters. As a rule of thumb: * **Input** — use `cron` or `timer` for repeating schedules, `linspace` for evenly spaced ranges, `time_patterns` for realistic traffic shapes, or `http` for on-demand triggers. * **Event** — `template` covers the vast majority of use cases. Reach for `script` only when you need full programmatic control, and `replay` to reprocess existing logs. * **Output** — pick the plugin that matches your destination. You can combine multiple outputs to write to several destinations at once. What's next [#whats-next] # Producing events An event plugin sits in the middle of the [generator pipeline](/docs/core/concepts/generator#the-three-stage-pipeline). It receives a **timestamp** and **tags** from the input stage and returns one or more **event strings** that the output stage delivers. This page explains how that process works for each of the three event plugins. For per-plugin parameters and configuration, see the [event plugin reference](/docs/plugins#event-plugins). What an event plugin receives [#what-an-event-plugin-receives] Every time the event plugin is called, it gets two values: * **`timestamp`** — a timezone-aware datetime representing *when* this event happens. * **`tags`** — a tuple of strings attached by the [input plugin](/docs/core/concepts/plugins#tags-input-plugins) that produced the timestamp. The plugin returns a list of strings. Each string is a complete event — a log line, a JSON object, a CSV row, or whatever format your use case requires. The output stage treats each string as an opaque payload. Template plugin [#template-plugin] The `template` plugin is the primary way to produce events. It renders [Jinja2](https://jinja.palletsprojects.com/) templates with a rich context of data generation tools. Template context [#template-context] Inside a `.jinja` template, you have access to the following variables: | Variable | What it provides | | ------------ | ------------------------------------------------------------ | | `timestamp` | The current event's datetime | | `tags` | Tags from the input plugin | | `module` | Data generation libraries (Faker, Mimesis, random utilities) | | `params` | User-defined parameters passed at runtime | | `samples` | Data loaded from external files (CSV, JSON, or inline lists) | | `locals` | State scoped to the current template | | `shared` | State shared across all templates in the same generator | | `globals` | State shared across all generators (thread-safe) | | `subprocess` | Run shell commands and capture output | Generating data with `module` [#generating-data-with-module] The `module` object gives you three built-in libraries for producing realistic synthetic data. It also acts as a gateway to any Python package — if the name isn't one of the built-in three, `module` imports it from the Python stdlib or the installed environment: **`module.faker`** — the [Faker](https://faker.readthedocs.io/) library, accessed by locale: ```jinja {{ module.faker.locale.en.name() }} {{ module.faker.locale.en.ipv4() }} {{ module.faker.locale.de.city() }} ``` **`module.mimesis`** — the [Mimesis](https://mimesis.name/) library, accessed by locale or specialty: ```jinja {{ module.mimesis.locale.en.person.full_name() }} {{ module.mimesis.locale.en.internet.ip_v4() }} ``` **`module.rand`** — built-in random utilities for common data types: ```jinja {# Numbers #} {{ module.rand.number.integer(1, 1000) }} {{ module.rand.number.floating(0.0, 1.0) }} {# Strings and crypto #} {{ module.rand.string.hex(16) }} {{ module.rand.crypto.uuid4() }} {{ module.rand.crypto.md5() }} {# Network #} {{ module.rand.network.ip_v4() }} {{ module.rand.network.ip_v4_private_a() }} {{ module.rand.network.mac() }} {# Random selection #} {{ module.rand.choice(['GET', 'POST', 'PUT', 'DELETE']) }} {{ module.rand.weighted_choice({200: 0.7, 404: 0.2, 500: 0.1}) }} {# Chance (returns true/false with given probability) #} {{ module.rand.chance(0.05) }} {# Random timestamp in range #} {{ module.rand.datetime.timestamp('2025-01-01', '2025-12-31') }} ``` **Any other name** — imported as a Python module. You can use stdlib packages or anything installed in the environment: ```jinja {# Standard library #} {{ module.json.dumps({"key": "value"}) }} {{ module.math.ceil(3.2) }} {# Third-party packages #} {{ module.hashlib.sha256(b"data").hexdigest() }} ``` A complete template example [#a-complete-template-example] Here's a template that produces an Apache-style access log line: ```jinja title="templates/access_log.jinja" {{ module.rand.network.ip_v4_public() }} - {{ module.faker.locale.en.user_name() }} [{{ timestamp.strftime('%d/%b/%Y:%H:%M:%S %z') }}] "{{ module.rand.choice(['GET', 'POST', 'PUT']) }} {{ module.rand.choice(['/api/users', '/api/orders', '/health', '/login']) }} HTTP/1.1" {{ module.rand.weighted_choice({200: 0.7, 301: 0.05, 404: 0.15, 500: 0.1}) }} {{ module.rand.number.integer(200, 15000) }} ``` The generator config to use it: ```yaml title="generator.yml" input: - cron: expression: "* * * * * *" count: 5 event: template: mode: all templates: - access_log: template: templates/access_log.jinja output: - stdout: {} ``` Each second, 5 timestamps arrive, and each one renders the template with fresh random values — producing 5 unique log lines per second. Picking modes [#picking-modes] When a generator has **multiple templates**, the picking mode controls which templates render on each call. Modes range from simple (render all, pick one at random) to advanced (finite state machine with conditional transitions). See the [template plugin reference](/docs/plugins/event/template) for the full list of modes and their parameters. **`all` mode** is the default and the simplest — every template produces an event for every timestamp. It's the right choice when you want to generate multiple correlated event types from the same schedule. **`chance` mode** is useful when different event types have different frequencies: ```yaml event: template: mode: chance templates: - success: template: templates/success.jinja chance: 0.85 - error: template: templates/error.jinja chance: 0.15 ``` 85% of timestamps produce a success event, 15% produce an error. **`fsm` mode** lets you model stateful event sequences where the next event type depends on past events. Transitions between templates are driven by conditions on state variables: ```yaml event: template: mode: fsm templates: - login: template: templates/login.jinja initial: true transitions: - to: browse when: { always: } - browse: template: templates/browse.jinja transitions: - to: checkout when: { ge: { "locals.page_views": 3 } } - checkout: template: templates/checkout.jinja transitions: - to: login when: { always: } ``` This models a user session: login → browse (until 3+ page views) → checkout → back to login. The `locals` state persists between calls, so the template can increment a counter and the FSM can check it. State management [#state-management] Templates can store and read state across multiple `produce` calls. There are three levels: **`locals`** — scoped to one template alias. Each template has its own `locals` that other templates cannot see: ```jinja title="templates/counter.jinja" {% do locals.set('n', locals.get('n', 0) + 1) %} Event #{{ locals.get('n') }} ``` **`shared`** — visible to all templates in the same generator. Useful when templates need to coordinate: ```jinja title="templates/login.jinja" {% set user_id = module.rand.crypto.uuid4() %} {% do shared.set('current_user', user_id) %} User {{ user_id }} logged in ``` ```jinja title="templates/action.jinja" User {{ shared.get('current_user') }} performed an action ``` **`globals`** — shared across all generators in the application. Thread-safe. Useful for global counters or inter-generator coordination. Use [Scenarios](/docs/studio/scenarios) in Studio to visualize and manage global state: ```jinja {% do globals.set('total', globals.get('total', 0) + 1) %} ``` Loading external data with samples [#loading-external-data-with-samples] You can load data from files and use it in templates. Supported sources: ```yaml event: template: mode: all samples: users: type: csv source: data/users.csv header: true endpoints: type: json source: data/endpoints.json status_codes: type: items source: [200, 201, 301, 404, 500] templates: - request: template: templates/request.jinja ``` Access in templates: ```jinja {# Pick a random user — named access via CSV headers #} {%- set user = samples.users | random -%} {{ user.name }} {# JSON rows also support named access via object keys #} {%- set ep = samples.endpoints | random -%} {{ ep.host }} {# Inline items #} {{ module.rand.choice(samples.status_codes) }} ``` CSV samples with `header: true` and JSON samples support named field access (e.g. `user.name`). All sample types support index access (e.g. `user[0]`). See the [template plugin reference](/docs/plugins/event/template#accessing-sample-rows) for details. Running shell commands [#running-shell-commands] The `subprocess` variable lets you call external commands from a template: ```jinja {{ subprocess.run('hostname').stdout | trim }} ``` The `run` method accepts `command`, `cwd`, `env`, and `timeout` parameters and returns an object with `stdout`, `stderr`, and `exit_code`. Subprocess calls run synchronously and block event production. Use them sparingly and with short timeouts to avoid stalling the pipeline. Script plugin [#script-plugin] When Jinja2 templates aren't flexible enough, the `script` plugin lets you write event production logic in Python: ```yaml title="generator.yml" event: script: path: scripts/produce.py ``` The script must define a `produce` function: ```python title="scripts/produce.py" from datetime import datetime def produce(params: dict) -> str | list[str]: ts: datetime = params['timestamp'] tags: tuple[str, ...] = params['tags'] return f'{ts.isoformat()} - event from tags: {tags}' ``` The function receives a dict with `timestamp` and `tags`, and returns either a single string or a list of strings. Returning an empty list skips the event for that timestamp. Use `script` when you need: * Complex control flow that Jinja2 can't express * External API calls or database lookups * Heavy computation or data transformation * Third-party Python libraries Replay plugin [#replay-plugin] The `replay` plugin reads events line-by-line from an existing log file instead of generating new ones: ```yaml title="generator.yml" event: replay: path: logs/access.log ``` Each `produce` call returns the next line from the file. When the file is exhausted, the generator stops — unless `repeat: true` is set, in which case it loops back to the beginning. Timestamp replacement [#timestamp-replacement] By default, replayed events keep their original content. To inject the current timestamp into each line, provide a regex pattern with a named `timestamp` group: ```yaml event: replay: path: logs/access.log timestamp_pattern: '(?P\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2})' timestamp_format: '%Y-%m-%dT%H:%M:%S' ``` The plugin finds the match in each line and replaces it with the current timestamp formatted according to `timestamp_format`. Lines that don't match the pattern are returned unchanged. This is useful for replaying historical logs against a live system while making the events look current. What's next [#whats-next] # Scheduling The [Generator](/docs/core/concepts/generator) page introduced the idea that input plugins produce timestamps and the generator delivers events at those moments. This page goes deeper into how that process actually works and how to take advantage of it. For per-plugin parameters and configuration, see the [input plugin reference](/docs/plugins#input-plugins). From timestamps to events [#from-timestamps-to-events] An input plugin doesn't emit events directly — it emits **timestamps**. A timestamp is a point in time that answers one question: "when should an event happen?" The event plugin later decides *what* happens at that moment. This separation is the key to flexible scheduling. The same template can produce an event every second (via `cron`), at irregular real-world patterns (via `time_patterns`), or at 10,000 evenly spaced moments across a date range (via `linspace`) — without any changes to the template itself. The `count` parameter [#the-count-parameter] Most input plugins support a `count` field: the number of timestamps to produce per tick. A cron expression that fires once per second with `count: 5` produces **5 timestamps at the same moment** — which means 5 separate calls to the event plugin, each potentially generating a different event (thanks to randomization in templates). This is the simplest way to control event throughput without changing the schedule itself: ```yaml input: - cron: expression: "* * * * * *" # every second count: 10 # 10 events per tick ``` Date ranges and versatile datetime [#date-ranges-and-versatile-datetime] Most input plugins accept `start` and `end` parameters to define when timestamps should be produced. These fields use a **versatile datetime** format — you can specify them in several ways: | Format | Example | Meaning | | ------------------- | ----------------------------- | ---------------------------------------------------- | | ISO 8601 | `"2025-06-15T09:00:00"` | An exact moment | | Human-readable | `"1st August 2025"` | Parsed naturally | | Keyword `now` | `"now"` | Current system time | | Keyword `never` | `"never"` | No bound (run indefinitely) | | Relative expression | `"+1h"`, `"-30m"`, `"+1d12h"` | Offset from now (or from `start` when used in `end`) | | Time of day | `"14:30:00"` | Anchored to the current date | Relative expressions use a compact syntax: combine `d` (days), `h` (hours), `m` (minutes), and `s` (seconds) with an optional `+`/`-` sign. A few examples: ```yaml start: "now" # from this moment start: "-1h" # one hour ago start: "2025-01-01" # exact date end: "+24h" # 24 hours after start end: "+7d12h" # 7 days and 12 hours after start end: "never" # run indefinitely ``` When used in the `end` field, relative expressions are calculated from the `start` value, not from the current time. So `start: "2025-01-01"` with `end: "+7d"` means January 1 through January 8. Not all input plugins support these fields. Schedule-based plugins like `cron`, `timer`, `linspace`, and `time_patterns` accept `start` and/or `end` to define the active window. Other plugins (`timestamps`, `static`, `http`) determine their timing differently. See each plugin's [reference page](/docs/plugins#input-plugins) for which fields are available. Combining multiple inputs [#combining-multiple-inputs] A generator can have any number of input plugins. Their timestamps are **merged chronologically** into a single stream before reaching the event stage. This lets you build complex, layered schedules from simple building blocks. Example: baseline + spikes [#example-baseline--spikes] A monitoring simulation might need steady background traffic with occasional bursts: ```yaml input: # Steady stream: 1 event/sec - cron: expression: "* * * * * *" count: 1 tags: [baseline] # Burst: 50 events every 5 minutes - cron: expression: "*/5 * * * * 0" count: 50 tags: [spike] ``` Both plugins run independently. The merger interleaves their timestamps in time order, so the event plugin sees a smooth stream of moments — most one second apart, with clusters of 50 at five-minute marks. How merging works [#how-merging-works] When multiple inputs run simultaneously, the merger collects timestamps from each plugin and combines them in chronological order. If two plugins produce timestamps at the same instant, their relative order within that instant is stable but not guaranteed between plugins. Each timestamp carries the **ID** of the input plugin that produced it. This ID is used internally to look up the plugin's [tags](/docs/core/concepts/plugins#tags-input-plugins), which are then passed to the event plugin as the `tags` variable. Finite and infinite inputs [#finite-and-infinite-inputs] Some inputs are **finite** — they produce a bounded number of timestamps and then stop. `linspace`, `timestamps`, and `static` are always finite. `cron` and `timer` are finite when configured with bounds (`end` or `repeat`) and infinite otherwise. When all input plugins in a generator have exhausted their timestamps, the generator finishes. If *some* plugins are finite and others are infinite, the finite ones simply stop contributing — the infinite ones keep the generator running. Wall-clock alignment [#wall-clock-alignment] In [live mode](/docs/core/concepts/generator#live-mode-default), the generator doesn't just produce timestamps — it **waits** for the real clock to reach each timestamp before releasing it downstream. The scheduler looks at the **latest timestamp in each batch** and sleeps until the wall clock catches up. This means the event plugin receives timestamps only after they've "happened" in real time, creating a faithful simulation of a live system. In [sample mode](/docs/core/concepts/generator#sample-mode), the scheduler is bypassed entirely — timestamps flow as fast as the pipeline can process them. Skipping past timestamps [#skipping-past-timestamps] When a generator starts in live mode, there may be a gap between the schedule's start time and "now." By default (`skip_past: true`), input plugins skip all timestamps that fall before the current time and begin producing from the first future moment. This prevents a startup burst of historical events. If you *want* to replay past timestamps in live mode (e.g. to catch up after a restart), set `skip_past: false` — but be aware that all past timestamps will be released immediately since the scheduler only waits for future ones. Timezone [#timezone] All timestamps in a generator are interpreted in the generator's **timezone**, which defaults to `UTC`. The timezone affects: * How input plugins calculate "now" (for `skip_past` and relative time expressions) * How the scheduler compares timestamps to the wall clock * The `timestamp` value passed to the event plugin Set the timezone via CLI flag or in the startup config: ```bash eventum generate ... --timezone America/New_York ``` ```yaml title="startup.yml" - id: my-gen path: generator.yml timezone: Europe/Berlin ``` Timestamps are stored internally as timezone-naive values that represent moments in the generator's timezone. Changing the timezone shifts when events appear on the wall clock without modifying the schedule. On-demand scheduling [#on-demand-scheduling] The `http` input plugin doesn't follow a schedule at all. Instead, it starts an HTTP server and waits for external requests to trigger event generation: ```yaml input: - http: host: 0.0.0.0 port: 8081 ``` A `POST /generate` request with a body like `{"count": 10}` produces 10 timestamps at the current time. This is useful when events should be triggered by an external system — a webhook, a CI pipeline, or a manual curl command. The HTTP input can be combined with scheduled inputs in the same generator. Scheduled timestamps flow as usual; HTTP-triggered timestamps are injected into the stream on demand. Practical patterns [#practical-patterns] Generate a fixed dataset [#generate-a-fixed-dataset] Use `linspace` to spread timestamps evenly across a range, then run in sample mode to produce them instantly: ```yaml input: - linspace: start: "2025-01-01" end: "2025-01-31" count: 100000 ``` ```bash eventum generate ... --live-mode false ``` Simulate business-hours traffic [#simulate-business-hours-traffic] Combine two `cron` inputs — one for work hours, one for off-hours — with different throughput: ```yaml input: # Business hours: high rate - cron: expression: "* 9-17 * * MON-FRI *" count: 20 tags: [business] # Off-hours: low rate - cron: expression: "* 0-8,18-23 * * * */10" count: 1 tags: [quiet] ``` Simulate realistic traffic with time patterns [#simulate-realistic-traffic-with-time-patterns] Cron and timer produce perfectly regular timestamps. Real systems aren't like that — traffic clusters, fluctuates, and follows distributions. The `time_patterns` plugin lets you model this by combining four controls: * **Oscillator** — divides time into repeating periods (e.g. one hour each) * **Multiplier** — sets a baseline event count per period * **Randomizer** — adds natural variance (±deviation) to that count * **Spreader** — distributes events *within* each period using a probability distribution (uniform, triangular, or beta) For example, to simulate API traffic that peaks in the middle of each hour with ±20% variation: ```yaml title="patterns/api-traffic.yml" label: API traffic oscillator: start: "now" end: +24h period: 1 unit: hours multiplier: ratio: 3000 # ~3,000 requests/hour baseline randomizer: deviation: 0.2 # ±20% hour-to-hour variation direction: mixed spreader: distribution: beta parameters: a: 5 # bell-shaped, clustered toward mid-hour b: 5 ``` ```yaml title="generator.yml" input: - time_patterns: patterns: - patterns/api-traffic.yml ``` You can load **multiple pattern files** in the same plugin to layer different traffic shapes — for instance a steady baseline pattern plus a periodic spike pattern. Their timestamps are merged just like multiple input plugins would be. Replay with adjusted timing [#replay-with-adjusted-timing] Use `timestamps` to replay events at their original times, optionally in live mode to re-create the original pace: ```yaml input: - timestamps: path: original_timestamps.csv ``` ```bash # Replay at original pace eventum generate ... --live-mode --skip-past false # Replay as fast as possible eventum generate ... --live-mode false ``` What's next [#whats-next] # Features Eventum packs a wide range of capabilities into a single tool. This page gives you a bird's-eye view of the major features so you know what's available before diving into the details. Jinja2 template engine [#jinja2-template-engine] The primary way to define events in Eventum is through [Jinja2](https://jinja.palletsprojects.com/) templates. Every template has access to a rich set of context variables including the event `timestamp`, data generation libraries, parameters, sample datasets, and persistent state. ```jinja title="access_log.jinja" {{ timestamp.strftime('%d/%b/%Y:%H:%M:%S %z') }} {{ module.faker.locale.en.ipv4() }} - {{ module.faker.locale.en.user_name() }} "GET /{{ module.faker.locale.en.uri_path() }} HTTP/1.1" {{ module.rand.weighted_choice([200, 301, 404, 500], [85, 5, 8, 2]) }} {{ module.rand.number.integer(200, 15000) }} ``` Built-in data libraries make templates realistic out of the box: * **[Faker](https://faker.readthedocs.io/)** — names, addresses, emails, IPs, user agents, credit cards, and hundreds of other providers across 70+ locales. * **[Mimesis](https://mimesis.name/)** — high-performance alternative with locale-aware data for people, addresses, dates, networking, and more. * **rand** — lightweight helper for random choices, weighted selection, numbers (integer, float, gaussian), and string generation (hex, digits, letters). Templates also support **persistent state** — `locals` for per-template state, `shared` for cross-template state within a generator, and `globals` for state shared across all generators. This lets you build stateful sequences like user sessions or correlated event chains. Templates support multiple [picking modes](/docs/core/concepts/producing#picking-modes) for selecting which template to render — from simple random selection to a full finite state machine with conditional transitions. Python scripts [#python-scripts] When templates aren't enough, write event logic as a Python script. Your script receives the timestamp, tags, and parameters, and returns one or more event strings. This gives you the full power of Python — external libraries, system calls, database lookups, or any custom logic. ```python title="scripts/firewall_status.py" import json import subprocess def produce(params: dict) -> str | list[str]: result = subprocess.run( ['/bin/systemctl', 'is-active', 'ufw'], stdout=subprocess.PIPE, stderr=subprocess.DEVNULL, text=True, ) event = { 'timestamp': params['timestamp'].isoformat(), 'status': result.stdout.strip(), } return json.dumps(event) ``` Flexible scheduling [#flexible-scheduling] Eventum gives you precise control over *when* events happen. Choose from several [input plugins](/docs/plugins#input-plugins) — cron expressions, fixed intervals, evenly spaced ranges, statistical time patterns, and more. You can combine multiple input plugins in a single generator — for example, a `cron` plugin for steady baseline traffic and a `time_patterns` plugin for periodic spikes. Each plugin can carry **tags** that propagate to templates, so you can conditionally vary event content based on the source. Multiple output destinations [#multiple-output-destinations] Every generator can send events to one or more [output plugins](/docs/plugins#output-plugins) simultaneously. Write to a local file for archival while also pushing to a database for analysis — no duplicated generators needed. Destinations include the console, local files, HTTP endpoints, and databases. All output plugins support [formatters](/docs/plugins/formatters) to transform events before delivery. You can also control **event ordering** — run outputs in parallel for throughput, or serialize them to maintain strict chronological order. Live and sample modes [#live-and-sample-modes] Eventum supports two execution modes that cover both real-time streaming and bulk generation: * **Live mode** — events are emitted at the exact moments defined by their timestamps, synchronized with the wall clock. This is ideal for simulating real-time traffic, feeding a SIEM, or stress-testing a pipeline with a realistic event rate. * **Sample mode** — all events are generated as fast as possible regardless of their timestamps. Use this when you need to seed a database, create a historical dataset, or backfill a time range. Both modes use the same generator config — switch between them with a single flag: ```bash # Real-time streaming eventum generate --path generator.yml ... --live-mode # As fast as possible eventum generate --path generator.yml ... --live-mode false ``` Log replay [#log-replay] Not every event needs to be synthesized from scratch. The **replay** plugin reads events from an existing log file and optionally substitutes timestamps to make historical data look current. This is useful for reproducing production incidents, replaying known traffic patterns, or migrating data with fresh timestamps. ```yaml title="generator.yml" event: replay: path: access.log timestamp_pattern: '(?P\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2})' timestamp_format: '%Y-%m-%dT%H:%M:%S' ``` Declarative YAML configuration [#declarative-yaml-configuration] Everything in Eventum is configured through YAML files — no code required for most use cases. A generator config wires together an input, event, and output section. A startup config lists which generators to run. A main application config controls the server, logging, and paths. ```yaml title="generator.yml" input: - cron: expression: "* * * * * */5" count: 3 tags: [web] event: template: mode: any templates: - access_event: template: templates/access_log.jinja - error_event: template: templates/error_log.jinja params: app_name: my-service samples: countries: type: csv source: data/countries.csv header: true output: - stdout: {} - file: path: output/events.log flush_interval: 1 ``` Generator configs support **variable substitution** with `${params.name}` for user-defined parameters and `${secrets.name}` for credentials stored in the secure keyring. Web UI — Eventum Studio [#web-ui--eventum-studio] Eventum ships with a built-in web interface called **Studio** for visual generator management. Studio lets you: * Browse and edit generator projects, with a file explorer and a code editor beside the plugin forms * Configure input, event, and output plugins through forms * Preview scheduling distributions as histograms * Debug event production — render events on demand, inspect output and state, and catch errors before going live * Preview the payload a formatter delivers before a destination ever sees it * Start, stop and watch generator instances, with live throughput and per-stage failure counts * Group generators into scenarios and see the state they exchange REST API [#rest-api] Every operation available in Studio is also accessible through a REST API, so you can manage generators programmatically. Start and stop generators, update configs, and query status — all over HTTP. The API is served alongside the web UI when you run Eventum in server mode: ```bash eventum run -c eventum.yml ``` See the [API reference](/docs/api) for the full list of endpoints. Secure secrets management [#secure-secrets-management] Credentials for output plugins (database passwords, API tokens, TLS certificates) can be stored in an encrypted **keyring** instead of being hard-coded in config files. Eventum uses `keyrings.cryptfile` for local encrypted storage. ```bash # Store a secret eventum-keyring set my_db_password # Reference it in config # password: ${secrets.my_db_password} ``` Scalable by design [#scalable-by-design] Eventum is built for running many generators in parallel. Each generator runs in its own thread with independent input, event, and output pipelines. Backpressure is handled through bounded queues between stages, and you can tune batch sizes, concurrency limits, and write timeouts per generator. The server process manages the full lifecycle — start, stop, and restart individual generators without affecting others. Signal handling (`SIGINT`, `SIGTERM`, `SIGHUP`) provides graceful shutdown and hot reload. # First run This guide walks you through two ways to run Eventum. Start with a **single generator** to see results immediately, then move to the **full application** when you need multiple generators, a REST API, or the Studio web UI. Make sure Eventum is installed before continuing. See [Installation](/docs/core/introduction/installation) for options. Run a single generator [#run-a-single-generator] The `eventum generate` command runs one generator directly from the command line — no server, no config files beyond the generator itself. This is the fastest way to see Eventum in action. Create a project directory [#create-a-project-directory] ```bash mkdir my-generator && cd my-generator mkdir templates ``` Write a template [#write-a-template] Create a Jinja2 template that defines what each event looks like. The `timestamp` variable is provided automatically by the input plugin, and `module.faker` gives you access to [Faker](https://faker.readthedocs.io/) for realistic data. ```jinja title="templates/event.jinja" { "timestamp": "{{ timestamp.isoformat() }}", "level": "{{ module.rand.weighted_choice(['INFO', 'WARN', 'ERROR'], [80, 15, 5]) }}", "service": "api-gateway", "user": "{{ module.faker.locale.en.user_name() }}", "action": "{{ module.rand.choice(['login', 'logout', 'request', 'timeout']) }}", "ip": "{{ module.faker.locale.en.ipv4() }}" } ``` Write a generator config [#write-a-generator-config] Create a YAML file that wires the three pipeline stages together: **input** (when), **event** (what), and **output** (where). ```yaml title="generator.yml" input: - cron: expression: "* * * * * *" count: 1 event: template: mode: all templates: - event: template: templates/event.jinja output: - stdout: {} ``` This config generates one event per second using a cron expression, renders it with the template, and prints it to stdout. Run it [#run-it] ```bash eventum generate --id my-gen --path generator.yml ``` Events start printing to your terminal: ```json {"timestamp": "2026-02-18T12:00:01+00:00", "level": "INFO", "service": "api-gateway", "user": "jsmith", "action": "login", "ip": "192.168.44.12"} {"timestamp": "2026-02-18T12:00:02+00:00", "level": "INFO", "service": "api-gateway", "user": "amiller", "action": "request", "ip": "10.0.128.55"} {"timestamp": "2026-02-18T12:00:03+00:00", "level": "WARN", "service": "api-gateway", "user": "kwilson", "action": "timeout", "ip": "172.16.0.91"} ``` Press `Ctrl+C` to stop. For the full list of available flags see the [`$ eventum generate`](/docs/core/cli/eventum-generate) CLI reference. *** Run as application [#run-as-application] The `eventum run` command starts the full Eventum application: multiple generators managed by a central process, an optional REST API, and the Studio web UI. This is the way to run Eventum in production or when you need more than a single generator. Set up the project structure [#set-up-the-project-structure] Eventum expects a directory layout where each generator lives in its own folder: Create the generators [#create-the-generators] Each generator has its own config and templates — the same format as the single-generator example above. For instance: ```yaml title="generators/access-logs/generator.yml" input: - cron: expression: "* * * * * *" count: 3 event: template: mode: all templates: - access: template: templates/access.jinja output: - stdout: {} - file: path: ./output/access.log flush_interval: 1 ``` See [Generator](/docs/core/concepts/generator) and [Configuration files](/docs/core/config/files) for the full reference. Create the startup config [#create-the-startup-config] The startup file lists which generators to run and lets you override parameters per generator: ```yaml title="startup.yml" - id: access-logs path: generators/access-logs/generator.yml autostart: true live_mode: true - id: error-logs path: generators/error-logs/generator.yml autostart: true live_mode: true ``` * **`id`** — unique name for the generator * **`path`** — relative path to the generator config (resolved from the startup file's directory) * **`autostart`** — start automatically when the app launches (default: `true`). Generators with `autostart: false` can be started later through the API or Studio * **`live_mode`** — override the execution mode per generator See [startup.yml](/docs/core/config/files/startup-yml) for the full reference. Create the main application config [#create-the-main-application-config] The `eventum.yml` file configures the server, logging, default generation parameters, and paths: ```yaml title="eventum.yml" server.host: "0.0.0.0" server.port: 9474 server.api.enabled: true server.ui.enabled: true server.auth.user: eventum server.auth.password: eventum generation.timezone: UTC generation.batch.size: 10000 generation.batch.delay: 1.0 log.level: info log.format: plain path.logs: /home/user/my-project/logs/ path.startup: /home/user/my-project/startup.yml path.generators_dir: /home/user/my-project/generators/ path.keyring_cryptfile: /home/user/my-project/cryptfile.cfg ``` Key sections: | Section | What it controls | | ------------------ | --------------------------------------------------------------------------------------- | | **`server.*`** | Host, port, SSL, basic auth, and which services to enable (API, Studio UI) | | **`generation.*`** | Default parameters for all generators — timezone, batch size, queue limits, concurrency | | **`log.*`** | Log level, format (`plain` or `json`), rotation settings | | **`path.*`** | Directories for generators, logs, startup file, and keyring | See [eventum.yml](/docs/core/config/files/eventum-yml) for the full reference. Start the application [#start-the-application] ```bash eventum run -c eventum.yml ``` Eventum starts all generators marked with `autostart: true` and launches the server. You'll see log output confirming each component: ```log 2026-02-18T19:51:13.290066Z [info ] Starting generators [eventum.app.main] 2026-02-18T19:51:13.290250Z [warning ] Generators are running [eventum.app.main] count=2 non_running_generators=[] running_generators=['access-logs', 'error-logs'] 2026-02-18T19:51:13.290940Z [info ] Starting Server [eventum.app.main] host=0.0.0.0 port=9474 2026-02-18T19:51:13.467938Z [info ] Starting REST API service [eventum.server.main] 2026-02-18T19:51:14.076947Z [info ] Starting web UI service [eventum.server.main] ``` * **Studio UI** — open [http://localhost:9474](http://localhost:9474) in your browser to manage generators visually * **REST API** — available at the same address under `/api` route (and specs under `/api/swagger`, `/api/asyncapi` and `/api/redoc` routes); see the [API reference](/docs/api) for endpoints * **Graceful shutdown** — press `Ctrl+C` or send `SIGTERM` * **Hot reload** — send `SIGHUP` to restart with updated configuration What's next [#whats-next] # Installation Prerequisites [#prerequisites] Eventum runs natively on **Linux** (including WSL2) and on **any OS via Docker**; native installation requires **Python 3.14+**. See [Requirements](/docs/core/introduction/requirements) for supported platforms, system libraries, and resource usage. Check your Python version [#check-your-python-version] ```bash python --version ``` If you see `Python 3.14.x` or higher, you're ready. If not, install the required version: * **Ubuntu/Debian:** `sudo apt install python3.14` * **With uv:** `uv python install 3.14` (uv manages Python versions for you) Choose an installation method [#choose-an-installation-method] | Method | Best for | | --------------------------------- | ---------------------------------------------------------------- | | [uv tool](#install-with-uv) | Easiest install — one command, no virtual environment management | | [pip](#install-with-pip) | When you already have a Python environment set up | | [Docker](#docker) | Containerized deployments without a local Python installation | | [From source](#build-from-source) | Development or running unreleased changes | *** Install with uv [#install-with-uv] [uv](https://docs.astral.sh/uv/) is a fast Python package manager. If you don't have it yet: ```bash curl -LsSf https://astral.sh/uv/install.sh | sh ``` Install Eventum as a tool — this makes the `eventum` and `eventum-keyring` commands globally available: ```bash uv tool install eventum-generator ``` `uv tool install` creates an isolated environment for Eventum automatically — no project setup or virtual environment management needed. Verify the installation: ```bash eventum --version ``` Install with pip [#install-with-pip] ```bash pip install eventum-generator ``` We recommend installing into a virtual environment to avoid dependency conflicts: ```bash python -m venv .venv && source .venv/bin/activate pip install eventum-generator ``` Verify the installation: ```bash eventum --version ``` *** Set up a project [#set-up-a-project] After installing Eventum, you need to create a project directory with configuration files. The structure depends on how you plan to run Eventum. Single generator mode [#single-generator-mode] For quick experiments and one-off generation with [`eventum generate`](/docs/core/cli/eventum-generate), all you need is a generator config file and its resources (e.g. templates): ```bash mkdir -p my-project/templates ``` Create a `generator.yml` that defines the three-stage pipeline — input, event, and output. See the [generator.yml](/docs/core/config/generator-yml) reference for the full schema. Then run it directly: ```bash eventum generate --id my-gen --path my-project/generator.yml ``` See [First run](/docs/core/introduction/first-run) for a complete step-by-step example with template content. Application mode [#application-mode] For production use with [`eventum run`](/docs/core/cli/eventum-run) — multiple generators, a REST API, and the Studio web UI — you need three config files and a directory for your generators: ```bash mkdir -p eventum/{generators,logs} ``` On Linux with systemd, [`eventum service install`](/docs/core/cli/eventum-service) creates this entire structure automatically — directories, default config, empty startup file, and a systemd unit — in a single command. | File | Purpose | Reference | | ------------- | -------------------------------------------------------------------------------------------------- | ---------------------------------------------------------- | | `eventum.yml` | Main application config — server settings, default generation parameters, logging, and file paths. | [eventum.yml reference](/docs/core/config/eventum-yml) | | `startup.yml` | List of generators to load at startup with per-generator parameter overrides. | [startup.yml reference](/docs/core/config/startup-yml) | | `generators/` | Each subdirectory is a self-contained generator with its own `generator.yml` and resources. | [generator.yml reference](/docs/core/config/generator-yml) | | `logs/` | Application log files (path configured in `eventum.yml`). | — | Create a minimal `eventum.yml` — update the paths to match your project location: ```yaml title="eventum/eventum.yml" server: host: "0.0.0.0" port: 9474 auth: user: eventum password: eventum generation: timezone: UTC batch: size: 10000 log: level: info path: startup: /absolute/path/to/eventum/startup.yml generators_dir: /absolute/path/to/eventum/generators logs: /absolute/path/to/eventum/logs keyring_cryptfile: /absolute/path/to/eventum/cryptfile.cfg ``` All values in the `path` section must be **absolute paths**. Replace `/absolute/path/to/my-project` with the actual path on your system. Create a `startup.yml` — this is where you register your generators. Start with an empty list and add generators as you create them: ```yaml title="eventum/startup.yml" [] ``` Then start the application: ```bash eventum run -c eventum/eventum.yml ``` Once running, open [http://localhost:9474](http://localhost:9474) in your browser to access the Studio UI. See [First run — Run as application](/docs/core/introduction/first-run#run-as-application) for a complete walkthrough with generator examples, and [Project structure](/docs/core/config/project-structure) for a detailed breakdown of how the files relate. *** Docker [#docker] Official images are available on [Docker Hub](https://hub.docker.com/r/rnv812/eventum-generator). Docker does not require a local Python installation. Project layout for Docker [#project-layout-for-docker] Prepare your project directory with the similar structure as the application mode above. The Docker container expects the config at `/app/config/eventum.yml` and generators in the directory you mount: Since the files will be mounted into the container, use the **container paths** in `eventum.yml`: ```yaml title="config/eventum.yml" server: host: "0.0.0.0" port: 9474 auth: user: eventum password: eventum generation: timezone: UTC batch: size: 10000 log: level: info path: startup: /app/config/startup.yml generators_dir: /app/generators logs: /app/logs keyring_cryptfile: /app/config/cryptfile.cfg ``` Run with Docker [#run-with-docker] Mount your project directories and expose port `9474` for the API and Studio UI: ```bash docker run --rm \ -v $(pwd)/config:/app/config \ -v $(pwd)/generators:/app/generators \ -v $(pwd)/logs:/app/logs \ -p 9474:9474 \ rnv812/eventum-generator:latest ``` The container starts with `eventum run -c /app/config/eventum.yml` by default. Run with Docker Compose [#run-with-docker-compose] For a more reproducible setup, use a `docker-compose.yml`: ```yaml title="docker-compose.yml" services: eventum: image: rnv812/eventum-generator:latest ports: - "9474:9474" volumes: - ./config:/app/config - ./generators:/app/generators - ./logs:/app/logs ``` ```bash docker compose up -d ``` Once running, open [http://localhost:9474](http://localhost:9474) in your browser to access the Studio UI. *** Build from source [#build-from-source] For development or running the latest unreleased changes: Clone the repository [#clone-the-repository] ```bash git clone https://github.com/eventum-generator/eventum.git cd eventum ``` Install dependencies [#install-dependencies] [uv](https://docs.astral.sh/uv/) is required for dependency management: ```bash uv sync ``` Build the Studio UI [#build-the-studio-ui] The web interface is a React application that needs to be compiled separately. [Node.js](https://nodejs.org/) is required: ```bash cd eventum/ui npm ci --legacy-peer-deps npm run build cd ../.. ``` Verify [#verify] ```bash uv run eventum --version ``` *** What's next [#whats-next] # Requirements Eventum runs natively on Linux and in Docker on any host. Before [installing](/docs/core/introduction/installation), check the supported platforms, the required Python version, and the resources a running instance consumes. Supported platforms [#supported-platforms] | Platform | Support | Notes | | ------------------------ | ------------- | ---------------------------------------------------------------------------------------------------------------------------------------- | | Linux | Native | Full support — install from PyPI, from source, or run with Docker. | | WSL2 (Windows) | Native | Runs exactly as on Linux — the recommended way to run on a Windows host. [Set up WSL2](https://learn.microsoft.com/windows/wsl/install). | | Docker (any OS) | Full | Official images run on Windows, macOS, and Linux hosts. | | Windows / macOS (native) | Not supported | Use Docker, or WSL2 on Windows. | Python [#python] Eventum requires **Python 3.14** or later. This applies to native installations (PyPI or source) — the Docker image already bundles Python. For best performance, use the **free-threaded build** (Python 3.14t). It disables the GIL, allowing multiple generators to run truly in parallel across CPU cores. Install it with `uv python install 3.14t`. Resource usage [#resource-usage] Resource use depends on the run mode and your workload. The figures below are a baseline to size from. | Scenario | CPU | RAM | Notes | | ----------- | ------ | ---- | ---------------------------------------------------------------------------------------------------------------------------- | | Minimal | 1 vCPU | 1 GB | Can run a single generator with [`eventum generate`](/docs/core/cli/eventum-generate). | | Recommended | 2 vCPU | 4 GB | Can run the full application with [`eventum run`](/docs/core/cli/eventum-run) — multiple generators plus the API and Studio. | | Optimal | 4 vCPU | 8 GB | Can run many generators at high throughput, with room to spare. | * **CPU** — Driven mainly by the event stage, which is CPU-bound; how much it needs depends on the event plugin you choose, how you configure it, and your target throughput. Each generator runs three pipeline stages — input, event, and output — in parallel on the free-threaded build, so throughput scales with available cores. * **RAM** — Scales with the number of generators and their batching. Each generator buffers timestamp and event batches sized by `batch.size` and `queue.*`, so larger batches or more generators raise memory use. * **Disk** — Allow around 5 GB for the application, its caches, and logs. * **Network** — Used only when output plugins deliver over the network (Kafka, HTTP, OpenSearch, ClickHouse). Batching, queue depth, and concurrency are configurable — tune them in [eventum.yml](/docs/core/config/eventum-yml) to trade memory for throughput. What's next [#whats-next] # Upgrading Eventum is upgraded with the tool that installed it. An upgrade replaces the application only: `eventum.yml`, `startup.yml`, the generator projects and the keyring stay in place and are read by the new version. Before you upgrade [#before-you-upgrade] * **Read the changelog.** The [changelog](https://github.com/eventum-generator/eventum/blob/master/CHANGELOG.md) has one section per release. When several releases are skipped, read every section in between. Renamed or removed settings are listed under *Other changes* — `eventum.yml` rejects a key the new version does not know, and the application does not start until the file is updated. * **Check the requirements.** A release may raise the minimum Python version — 2.3.0 moved to Python 3.14. [Requirements](/docs/core/introduction/requirements) states the current minimum. The Docker image bundles its own interpreter. * **Plan a restart.** A running `eventum run` process keeps the version it was started with. Hot reload keeps the process, and with it the old version, so the new one takes effect only after the process is stopped and started again. uv [#uv] ```bash uv tool upgrade eventum-generator ``` The upgrade respects the constraints of the original installation. To move to a specific version, or to change a pin, install it explicitly: ```bash uv tool install eventum-generator== ``` When a release raises the minimum Python version, name the interpreter to use: ```bash uv tool upgrade --python 3.14 eventum-generator ``` pip [#pip] Activate the environment Eventum was installed into, then: ```bash pip install --upgrade eventum-generator ``` A specific version: ```bash pip install eventum-generator== ``` Docker [#docker] Each release is published under four tags: the full version (for example `2.8.0`), the minor line (`2.8`), the major line (`2`), and `latest`. Pin the full version in production, so that an upgrade is a deliberate change of the tag rather than a side effect of a pull. Pull the new image: ```bash docker pull rnv812/eventum-generator: ``` Then stop the running container and start it again from the new tag, with the same volume mounts as in [Run with Docker](/docs/core/introduction/installation#run-with-docker). The mounted `config/`, `generators/` and `logs/` directories are not modified by the upgrade. With Docker Compose, change the tag in `image:` and recreate the service: ```yaml title="docker-compose.yml" services: eventum: image: rnv812/eventum-generator: ``` ```bash docker compose pull docker compose up -d ``` From source [#from-source] Check out the release tag and reinstall the dependencies: ```bash git fetch --tags git checkout v uv sync ``` Then rebuild the Studio UI as described in [Build from source](/docs/core/introduction/installation#build-from-source) — the bundle belongs to the version it was built from. After upgrading [#after-upgrading] Confirm the version: ```bash eventum --version ``` * **systemd service.** Restart the service installed by [`eventum service install`](/docs/core/cli/eventum-service): `sudo systemctl restart eventum`, or `systemctl --user restart eventum` for a user service. The unit runs the same `eventum` binary, which the upgrade replaced in place, so the unit itself needs no change. * **Studio.** The footer shows the running version. The first load after an upgrade opens the release highlights, which stay reachable from the user menu. * **Configuration.** If the application refuses to start, the error names the setting it rejected. The changelog entry of the release states what replaced it. What's next [#whats-next] # replay Replays events from an existing log file. Each line of the file becomes one event. Optionally replaces original timestamps with the timestamps provided by the input plugin, so you can "re-schedule" historical data. Parameters [#parameters] | Parameter | Type | Default | Constraints | Description | | ------------------- | -------------- | --------- | ---------------------------------------------- | ------------------------------------------------------------------------- | | `path` | path | — | Required | Path to the log file. | | `timestamp_pattern` | string or null | `null` | Must contain a `timestamp` named group if set. | Regex pattern to locate the timestamp in each line. | | `timestamp_format` | string or null | `null` | C89 strftime format. | Format of the timestamp in the log file. Defaults to ISO 8601 if omitted. | | `repeat` | boolean | `false` | — | Whether to loop back to the start after reaching the end of the file. | | `chunk_size` | integer | `1048576` | >= 0 | Bytes to read per chunk. `0` reads the entire file at once. | | `encoding` | string | `"utf_8"` | Valid Python codec name. | File encoding. | How it works [#how-it-works] The plugin reads the log file line by line using buffered chunks. Each call to the plugin consumes the **next line** from the file: 1. On startup, the plugin opens the file and begins reading from position 0. 2. For each incoming timestamp, it reads the next line and either returns it as-is or replaces the original timestamp. 3. When the file ends, the plugin either stops producing events or loops back to the start (if `repeat: true`). The file is not loaded entirely into memory — it's read in chunks controlled by `chunk_size`, making it safe for large files. Timestamp replacement [#timestamp-replacement] When both `timestamp_pattern` and `timestamp_format` are provided, the plugin finds the original timestamp in each line using the regex pattern and replaces it with the timestamp from the input plugin. This lets you replay historical logs at a different time or rate. The regex must include a named group called `timestamp`: ```yaml timestamp_pattern: '^\[(?P[^\]]+)\]' timestamp_format: "%d/%b/%Y:%H:%M:%S %z" ``` If these fields are omitted, lines are emitted as-is without modification. Examples [#examples] Simple replay — emit lines as-is [#simple-replay--emit-lines-as-is] The simplest use case: feed an existing log file as events without any modification. Each line becomes one event. ```yaml title="generator.yml" event: replay: path: logs/access.log ``` Given this input file: ```text title="logs/access.log" 127.0.0.1 - - [01/Dec/2023:12:34:56 +0000] "GET /index.html HTTP/1.1" 200 1024 127.0.0.1 - - [01/Dec/2023:12:35:01 +0000] "POST /form HTTP/1.1" 201 512 192.168.1.1 - - [01/Dec/2023:12:35:15 +0000] "GET /about.html HTTP/1.1" 200 2048 ``` Each timestamp from the input plugin produces one event — the next line from the file, unchanged: ```text title="Output (3 events)" 127.0.0.1 - - [01/Dec/2023:12:34:56 +0000] "GET /index.html HTTP/1.1" 200 1024 127.0.0.1 - - [01/Dec/2023:12:35:01 +0000] "POST /form HTTP/1.1" 201 512 192.168.1.1 - - [01/Dec/2023:12:35:15 +0000] "GET /about.html HTTP/1.1" 200 2048 ``` The original timestamps stay intact. The input plugin controls *when* events are emitted, but the content is untouched. Replay with timestamp replacement [#replay-with-timestamp-replacement] Re-schedule historical logs to new timestamps. The plugin finds the original timestamp in each line using a regex pattern and replaces it with the current timestamp from the input plugin. ```yaml title="generator.yml" input: - cron: expression: "* * * * * */5" # every 5 seconds count: 1 event: replay: path: logs/access.log timestamp_pattern: '\[(?P[^\]]+)\]' timestamp_format: "%d/%b/%Y:%H:%M:%S %z" ``` Given the same input file, the original timestamps get replaced with the schedule-generated ones: ```text title="Input file" 127.0.0.1 - - [01/Dec/2023:12:34:56 +0000] "GET /index.html HTTP/1.1" 200 1024 127.0.0.1 - - [01/Dec/2023:12:35:01 +0000] "POST /form HTTP/1.1" 201 512 192.168.1.1 - - [01/Dec/2023:12:35:15 +0000] "GET /about.html HTTP/1.1" 200 2048 ``` ```text title="Output (timestamps replaced by input plugin schedule)" 127.0.0.1 - - [20/Feb/2026:10:00:00 +0000] "GET /index.html HTTP/1.1" 200 1024 127.0.0.1 - - [20/Feb/2026:10:00:05 +0000] "POST /form HTTP/1.1" 201 512 192.168.1.1 - - [20/Feb/2026:10:00:10 +0000] "GET /about.html HTTP/1.1" 200 2048 ``` The log content stays the same — only the timestamp portion matched by the regex is swapped. Replay JSON logs with ISO timestamps [#replay-json-logs-with-iso-timestamps] Works with any log format. Here's an example with JSON logs where timestamps appear in a different position: ```yaml title="generator.yml" event: replay: path: logs/app.jsonl timestamp_pattern: '"time":"(?P[^"]+)"' timestamp_format: "%Y-%m-%dT%H:%M:%S" ``` ```json title="logs/app.jsonl" {"level":"INFO","time":"2023-11-15T08:30:00","msg":"Server started","port":8080} {"level":"INFO","time":"2023-11-15T08:30:01","msg":"Connected to database","db":"postgres"} {"level":"WARN","time":"2023-11-15T08:30:05","msg":"Slow query detected","duration_ms":1200} ``` ```json title="Output (timestamps replaced)" {"level":"INFO","time":"2026-02-20T10:00:00","msg":"Server started","port":8080} {"level":"INFO","time":"2026-02-20T10:00:05","msg":"Connected to database","db":"postgres"} {"level":"WARN","time":"2026-02-20T10:00:10","msg":"Slow query detected","duration_ms":1200} ``` Continuous replay for load testing [#continuous-replay-for-load-testing] Loop a small sample file to generate a continuous stream. The plugin resets to the beginning of the file when it reaches the end, pairing each line with the next timestamp from the input plugin. ```yaml title="generator.yml" input: - cron: expression: "* * * * * *" # every second count: 10 # 10 timestamps per tick event: replay: path: logs/sample-requests.log repeat: true ``` ```text title="logs/sample-requests.log (5 lines)" GET /api/users 200 POST /api/orders 201 GET /api/products 200 DELETE /api/orders/42 404 GET /health 200 ``` With 10 timestamps per second and a 5-line file, the file loops every 0.5 seconds: ```text title="Output (first 10 events — file loops after line 5)" GET /api/users 200 POST /api/orders 201 GET /api/products 200 DELETE /api/orders/42 404 GET /health 200 GET /api/users 200 ← file loops back to the start POST /api/orders 201 GET /api/products 200 DELETE /api/orders/42 404 GET /health 200 ``` This is useful for load testing: take a small representative sample and replay it at whatever rate the input plugin generates timestamps. Replay syslog with custom encoding [#replay-syslog-with-custom-encoding] Handle log files from legacy systems with non-UTF-8 encodings: ```yaml title="generator.yml" event: replay: path: logs/legacy-syslog.log timestamp_pattern: '^(?P\w{3}\s+\d+\s+\d{2}:\d{2}:\d{2})' timestamp_format: "%b %d %H:%M:%S" encoding: latin_1 chunk_size: 4194304 # 4 MiB chunks for large files ``` ```text title="logs/legacy-syslog.log" Dec 1 12:34:56 server01 sshd[1234]: Accepted publickey for admin Dec 1 12:35:01 server01 kernel: [UFW BLOCK] IN=eth0 SRC=10.0.0.5 Dec 1 12:35:15 server01 nginx: 192.168.1.50 "GET /dashboard" 200 ``` ```text title="Output (timestamps replaced)" Feb 20 10:00:00 server01 sshd[1234]: Accepted publickey for admin Feb 20 10:00:05 server01 kernel: [UFW BLOCK] IN=eth0 SRC=10.0.0.5 Feb 20 10:00:10 server01 nginx: 192.168.1.50 "GET /dashboard" 200 ``` # script Runs a Python function to produce events. Use this plugin when your event logic requires control flow, external API calls, or computation that doesn't fit Jinja2 templates. For most use cases, the [template](/docs/plugins/event/template) plugin is simpler and sufficient. Reach for `script` only when you need full Python expressiveness. Parameters [#parameters] | Parameter | Type | Constraints | Description | | --------- | ---- | ----------- | ------------------------------- | | `path` | path | Required | Path to the Python script file. | Function signature [#function-signature] The script must define a `produce` function. It receives a dictionary with the current timestamp, tags and global state, and returns one or more event strings: ```python title="scripts/produce.py" from datetime import datetime def produce(params: dict) -> str | list[str]: timestamp: datetime = params['timestamp'] tags: tuple[str, ...] = params['tags'] return f'{{"timestamp": "{timestamp.isoformat()}", "tags": {list(tags)}}}' ``` The `params` dictionary contains: | Key | Type | Description | | ----------- | ------------------ | -------------------------------------------------------------------------------------------------- | | `timestamp` | `datetime` | Timezone-aware datetime of the current event. | | `tags` | `tuple[str, ...]` | Tags attached by the input plugin. | | `globals` | `MultiThreadState` | [Global state](/docs/plugins/event/template/state) shared with every generator in the application. | **Return type:** a single `str` (treated as one event) or a `list[str]` (each element is a separate event). An empty list means no events are produced for this timestamp. Examples [#examples] Basic — single JSON event [#basic--single-json-event] ```yaml title="generator.yml" event: script: path: scripts/produce.py ``` ```python title="scripts/produce.py" import json from datetime import datetime def produce(params: dict) -> str: ts: datetime = params['timestamp'] event = { 'timestamp': ts.isoformat(), 'level': 'INFO', 'message': 'Request processed', } return json.dumps(event) ``` For a timestamp `2026-02-20T10:00:00+00:00`, this produces: ```json title="Output (1 event)" {"timestamp": "2026-02-20T10:00:00+00:00", "level": "INFO", "message": "Request processed"} ``` Multiple events per timestamp [#multiple-events-per-timestamp] Return a list to emit multiple events for each timestamp. This is useful when a single "tick" should produce a batch of related records. ```python title="scripts/batch_metrics.py" import json import random from datetime import datetime SERVICES = ['api-gateway', 'auth-service', 'order-service', 'payment-service'] def produce(params: dict) -> list[str]: ts: datetime = params['timestamp'] events = [] for service in SERVICES: event = { 'timestamp': ts.isoformat(), 'service': service, 'cpu_percent': round(random.uniform(5, 95), 1), 'memory_mb': random.randint(128, 2048), 'request_count': random.randint(0, 500), } events.append(json.dumps(event)) return events ``` Each timestamp produces 4 events — one metric per service: ```json title="Output (4 events per timestamp)" {"timestamp": "2026-02-20T10:00:00+00:00", "service": "api-gateway", "cpu_percent": 42.3, "memory_mb": 512, "request_count": 187} {"timestamp": "2026-02-20T10:00:00+00:00", "service": "auth-service", "cpu_percent": 12.7, "memory_mb": 256, "request_count": 43} {"timestamp": "2026-02-20T10:00:00+00:00", "service": "order-service", "cpu_percent": 67.8, "memory_mb": 1024, "request_count": 312} {"timestamp": "2026-02-20T10:00:00+00:00", "service": "payment-service", "cpu_percent": 23.1, "memory_mb": 384, "request_count": 89} ``` Using tags for conditional logic [#using-tags-for-conditional-logic] Tags from the input plugin let you vary behavior based on which schedule triggered the event. ```yaml title="generator.yml" input: - cron: expression: "* * * * * */5" count: 1 tags: [regular] - cron: expression: "0 * * * *" count: 1 tags: [hourly-summary] event: script: path: scripts/tagged.py ``` ```python title="scripts/tagged.py" import json import random from datetime import datetime def produce(params: dict) -> str: ts: datetime = params['timestamp'] tags: tuple[str, ...] = params['tags'] if 'hourly-summary' in tags: return json.dumps({ 'timestamp': ts.isoformat(), 'type': 'summary', 'total_requests': random.randint(1000, 5000), 'error_rate': round(random.uniform(0.01, 0.1), 3), 'p99_latency_ms': random.randint(50, 500), }) return json.dumps({ 'timestamp': ts.isoformat(), 'type': 'request', 'method': random.choice(['GET', 'POST', 'PUT', 'DELETE']), 'path': random.choice(['/api/users', '/api/orders', '/api/products']), 'status': random.choice([200, 200, 200, 201, 400, 404, 500]), }) ``` Stateful script — accumulating data across calls [#stateful-script--accumulating-data-across-calls] Module-level variables persist across `produce()` calls, letting you build up state over time. The script is loaded once at startup, so top-level variables act as persistent storage. ```python title="scripts/session_tracker.py" import json import random from datetime import datetime from uuid import uuid4 # Module-level state — persists across all produce() calls active_sessions: dict[str, dict] = {} session_counter = 0 def produce(params: dict) -> list[str]: global session_counter ts: datetime = params['timestamp'] events = [] # Randomly start new sessions if random.random() < 0.3: session_counter += 1 session_id = str(uuid4()) active_sessions[session_id] = { 'user': f'user_{session_counter}', 'started': ts.isoformat(), 'page_views': 0, } events.append(json.dumps({ 'timestamp': ts.isoformat(), 'event': 'session_start', 'session_id': session_id, 'user': active_sessions[session_id]['user'], })) # Active sessions generate page views for sid, session in list(active_sessions.items()): session['page_views'] += 1 events.append(json.dumps({ 'timestamp': ts.isoformat(), 'event': 'page_view', 'session_id': sid, 'user': session['user'], 'page': random.choice(['/home', '/products', '/cart', '/checkout']), 'total_views': session['page_views'], })) # End session after enough page views if session['page_views'] > random.randint(5, 20): events.append(json.dumps({ 'timestamp': ts.isoformat(), 'event': 'session_end', 'session_id': sid, 'user': session['user'], 'total_views': session['page_views'], })) del active_sessions[sid] return events ``` Over time, sessions overlap and interact: ```json title="Output (events accumulate across calls)" {"timestamp": "...", "event": "session_start", "session_id": "a1b2...", "user": "user_1"} {"timestamp": "...", "event": "page_view", "session_id": "a1b2...", "user": "user_1", "page": "/home", "total_views": 1} {"timestamp": "...", "event": "page_view", "session_id": "a1b2...", "user": "user_1", "page": "/products", "total_views": 2} {"timestamp": "...", "event": "session_start", "session_id": "c3d4...", "user": "user_2"} {"timestamp": "...", "event": "page_view", "session_id": "a1b2...", "user": "user_1", "page": "/cart", "total_views": 3} {"timestamp": "...", "event": "page_view", "session_id": "c3d4...", "user": "user_2", "page": "/home", "total_views": 1} ``` Correlating with other generators through global state [#correlating-with-other-generators-through-global-state] Module-level variables live inside one generator. To coordinate with other generators — a payment stream consuming the orders a storefront generator emits, for example — use the `globals` state, which every generator in the application shares. The same state is available to templates, so a script and a template can exchange keys. ```python title="scripts/payments.py" import json import random from datetime import datetime METHODS = ['card', 'wallet', 'transfer'] def produce(params: dict) -> str | list[str]: ts: datetime = params['timestamp'] state = params['globals'] # orders published by the storefront generator orders = state.get('recent_orders', []) if not orders: return [] # count payments across generators without losing an increment state.acquire() try: sequence = state.get('payment_sequence', 0) + 1 state.set('payment_sequence', sequence) finally: state.release() return json.dumps({ 'timestamp': ts.isoformat(), 'event': 'payment', 'sequence': sequence, 'order_id': random.choice(orders), 'method': random.choice(METHODS), 'amount': round(random.uniform(5, 500), 2), }) ``` Individual operations (`get`, `set`, `pop`) are already thread-safe. `acquire()` and `release()` are needed only when several operations must run as one, as above. A hold left behind — including one left by a call that raised before reaching `release()` — is released once the event is over, so it cannot block other generators. The keys a script reads and writes appear in the data flow diagram of [Scenarios](/docs/studio/scenarios), next to those of the templates, as long as they are written as literals — a key built at runtime is reported as a caveat instead. The values themselves are in the global state panel of a scenario the generator belongs to. Distributed tracing — correlated microservice spans [#distributed-tracing--correlated-microservice-spans] This example generates realistic [OpenTelemetry](https://opentelemetry.io/)-style distributed traces across a microservice architecture. Each incoming timestamp triggers a full request flow through multiple services, producing properly correlated spans with parent-child relationships, realistic latencies, and occasional error propagation. This is a scenario where `script` shines — managing trace context, propagating errors through a service graph, and computing dependent latencies requires logic that would be difficult to express in templates. ```yaml title="generator.yml" input: - cron: expression: "* * * * * */2" # a request every 2 seconds count: 1 event: script: path: scripts/tracing.py output: - stdout: formatter: format: plain ``` ```python title="scripts/tracing.py" import json import random from datetime import datetime, timedelta from uuid import uuid4 # Service dependency graph: service → list of downstream calls TOPOLOGY = { 'api-gateway': ['auth-service', 'order-service'], 'auth-service': ['user-db'], 'order-service': ['inventory-service', 'payment-service'], 'inventory-service': ['product-db'], 'payment-service': ['payment-provider'], 'user-db': [], 'product-db': [], 'payment-provider': [], } # Base latencies per service (ms) BASE_LATENCY = { 'api-gateway': 2, 'auth-service': 5, 'order-service': 3, 'inventory-service': 4, 'payment-service': 8, 'user-db': 10, 'product-db': 8, 'payment-provider': 50, } # Error probability per service ERROR_RATE = { 'payment-provider': 0.05, 'product-db': 0.02, } METHODS = ['POST /orders', 'GET /orders', 'POST /checkout', 'GET /products'] def _make_span( trace_id: str, parent_id: str | None, service: str, start: datetime, duration_ms: float, error: bool, method: str, ) -> str: span = { 'trace_id': trace_id, 'span_id': uuid4().hex[:16], 'parent_span_id': parent_id, 'service': service, 'operation': method, 'start_time': start.isoformat(), 'duration_ms': round(duration_ms, 2), 'status': 'ERROR' if error else 'OK', } if error: span['error_message'] = f'{service}: internal error' return json.dumps(span) def _traverse( service: str, trace_id: str, parent_id: str | None, start: datetime, method: str, ) -> tuple[list[str], float, bool]: """Walk the service graph depth-first, collecting spans.""" spans = [] own_latency = max(1, random.gauss(BASE_LATENCY[service], 2)) cursor = start + timedelta(milliseconds=own_latency * 0.3) child_error = False for downstream in TOPOLOGY[service]: child_spans, child_dur, errored = _traverse( downstream, trace_id, uuid4().hex[:16], cursor, method, ) spans.extend(child_spans) cursor += timedelta(milliseconds=child_dur) child_error = child_error or errored total = (cursor - start).total_seconds() * 1000 + own_latency * 0.7 is_error = child_error or (random.random() < ERROR_RATE.get(service, 0)) spans.append( _make_span(trace_id, parent_id, service, start, total, is_error, method) ) return spans, total, is_error def produce(params: dict) -> list[str]: ts: datetime = params['timestamp'] trace_id = uuid4().hex method = random.choice(METHODS) spans, _, _ = _traverse('api-gateway', trace_id, None, ts, method) return spans ``` Each timestamp produces a full trace — 8 correlated spans across the service graph: ```json title="Output (8 spans per request, sharing one trace_id)" {"trace_id": "a1b2c3d4...", "span_id": "f1e2d3c4...", "parent_span_id": "9a8b7c6d...", "service": "user-db", "operation": "POST /orders", "start_time": "2026-02-20T10:00:00.003...", "duration_ms": 12.34, "status": "OK"} {"trace_id": "a1b2c3d4...", "span_id": "9a8b7c6d...", "parent_span_id": "5e6f7a8b...", "service": "auth-service", "operation": "POST /orders", "start_time": "2026-02-20T10:00:00.001...", "duration_ms": 18.56, "status": "OK"} {"trace_id": "a1b2c3d4...", "span_id": "1c2d3e4f...", "parent_span_id": "7g8h9i0j...", "service": "product-db", "operation": "POST /orders", "start_time": "2026-02-20T10:00:00.022...", "duration_ms": 9.87, "status": "OK"} {"trace_id": "a1b2c3d4...", "span_id": "7g8h9i0j...", "parent_span_id": "3k4l5m6n...", "service": "inventory-service", "operation": "POST /orders", "start_time": "2026-02-20T10:00:00.020...", "duration_ms": 15.23, "status": "OK"} {"trace_id": "a1b2c3d4...", "span_id": "2o3p4q5r...", "parent_span_id": "3k4l5m6n...", "service": "payment-provider", "operation": "POST /orders", "start_time": "2026-02-20T10:00:00.038...", "duration_ms": 53.41, "status": "OK"} {"trace_id": "a1b2c3d4...", "span_id": "6s7t8u9v...", "parent_span_id": "3k4l5m6n...", "service": "payment-service", "operation": "POST /orders", "start_time": "2026-02-20T10:00:00.036...", "duration_ms": 62.18, "status": "OK"} {"trace_id": "a1b2c3d4...", "span_id": "3k4l5m6n...", "parent_span_id": "5e6f7a8b...", "service": "order-service", "operation": "POST /orders", "start_time": "2026-02-20T10:00:00.019...", "duration_ms": 80.92, "status": "OK"} {"trace_id": "a1b2c3d4...", "span_id": "5e6f7a8b...", "parent_span_id": null, "service": "api-gateway", "operation": "POST /orders", "start_time": "2026-02-20T10:00:00+00:00", "duration_ms": 102.45, "status": "OK"} ``` Key behaviors: * All spans in a request share the same `trace_id`. * Each span records its `parent_span_id`, forming a tree. * Latencies accumulate realistically — a parent span's duration includes its children. * Errors in leaf services (e.g., `payment-provider`) propagate up to parent spans. * The service topology, latencies, and error rates are all configurable by editing the module-level constants. Pipe this into an output plugin that writes to [Elasticsearch](/docs/plugins/output) or a file, then visualize the traces in Jaeger or Grafana Tempo. # cron Generates timestamps on a cron schedule. Supports second-level precision, year fields, and random values via the extended syntax provided by [croniter](https://github.com/kiorky/croniter). Parameters [#parameters] | Parameter | Type | Default | Constraints | Description | | ------------ | -------------------------------------------------------------------------------------- | ------- | -------------------------------- | --------------------------------------------------------------------- | | `expression` | string | — | Required. Valid cron expression. | Cron expression defining the schedule. | | `count` | integer | — | Required. > 0 | Number of timestamps to emit per cron tick. | | `start` | [VersatileDatetime](/docs/core/concepts/scheduling#date-ranges-and-versatile-datetime) | `null` | — | Start of the active range. Defaults to current time if omitted. | | `end` | [VersatileDatetime](/docs/core/concepts/scheduling#date-ranges-and-versatile-datetime) | `null` | — | End of the active range. `null` means run indefinitely. | | `tags` | list of strings | `[]` | — | Tags attached to every timestamp. Accessible in templates via `tags`. | When both `start` and `end` are specified, `start` must be earlier than or equal to `end`. Cron expression format [#cron-expression-format] The standard format is `minute hour day month weekday`, but croniter extends it with optional seconds and year fields: | Field | Position | Allowed values | | ------------ | -------------- | ---------------- | | Minute | 1st | 0–59 | | Hour | 2nd | 0–23 | | Day of month | 3rd | 1–31 | | Month | 4th | 1–12 | | Day of week | 5th | 0–6 (0 = Sunday) | | Second | 6th (optional) | 0–59 | | Year | 7th (optional) | 1970–2099 | Both extra fields come after the standard five, seconds first — so `35 10 * * * 3` fires at 10:35:03 every day, and the year field is only available once seconds are given. Special characters: `*` (any), `,` (list), `-` (range), `/` (step), `R` (random value within range). Examples [#examples] Every second: ```yaml input: - cron: expression: "* * * * * *" count: 1 ``` Every 5 minutes during business hours: ```yaml input: - cron: expression: "*/5 9-17 * * 1-5" count: 1 start: "2024-01-01" end: "2024-12-31" ``` Burst of 100 events at the top of each hour, tagged: ```yaml input: - cron: expression: "0 * * * *" count: 100 tags: [hourly-burst] ``` # http Opens an HTTP endpoint that accepts POST requests. Each incoming request triggers timestamp generation on demand, making this plugin useful for event-driven or API-triggered workflows. Parameters [#parameters] | Parameter | Type | Default | Constraints | Description | | ---------------------- | --------------- | ----------- | ----------------- | --------------------------------------------------------------------- | | `port` | integer | — | Required. 1–65535 | Port to bind to. | | `host` | string | `"0.0.0.0"` | Non-empty | Address to bind to. | | `max_pending_requests` | integer | `100` | >= 1 | Maximum number of queued requests before new ones are rejected. | | `tags` | list of strings | `[]` | — | Tags attached to every timestamp. Accessible in templates via `tags`. | Examples [#examples] Basic on-demand endpoint: ```yaml input: - http: port: 8080 ``` With a higher queue limit and custom bind address: ```yaml input: - http: host: 127.0.0.1 port: 9090 max_pending_requests: 500 ``` See [Scheduling — On-demand scheduling](/docs/core/concepts/scheduling#on-demand-scheduling-http) for usage details. # linspace Distributes a fixed number of timestamps evenly across a time range — similar to NumPy's `linspace`. Useful for generating datasets where events need to be uniformly spaced. Parameters [#parameters] | Parameter | Type | Default | Constraints | Description | | ---------- | -------------------------------------------------------------------------------------- | ------- | ------------------------- | --------------------------------------------------------------------- | | `start` | [VersatileDatetime](/docs/core/concepts/scheduling#date-ranges-and-versatile-datetime) | — | Required. Cannot be null. | Start of the range. | | `end` | [VersatileDatetime](/docs/core/concepts/scheduling#date-ranges-and-versatile-datetime) | — | Required. Cannot be null. | End of the range. | | `count` | integer | — | Required. >= 1 | Total number of timestamps to distribute. | | `endpoint` | boolean | `true` | — | Whether to include `end` as the last timestamp. | | `tags` | list of strings | `[]` | — | Tags attached to every timestamp. Accessible in templates via `tags`. | `start` must be earlier than or equal to `end`. Neither can be `null` or `never`. Examples [#examples] One event per minute across a full day: ```yaml input: - linspace: start: "2024-01-01" end: "2024-01-02" count: 1440 endpoint: false ``` 100 events in the next hour: ```yaml input: - linspace: start: now end: +1h count: 100 ``` # static Emits a fixed number of timestamps all at once — no scheduling, no time range. All timestamps use the current time. This is the simplest input plugin, useful for generating datasets in [sample mode](/docs/core/concepts/generator#execution-modes). Parameters [#parameters] | Parameter | Type | Default | Constraints | Description | | --------- | --------------- | ------- | ------------- | --------------------------------------------------------------------- | | `count` | integer | — | Required. > 0 | Number of timestamps to emit. | | `tags` | list of strings | `[]` | — | Tags attached to every timestamp. Accessible in templates via `tags`. | Examples [#examples] Generate a dataset of 50,000 events: ```yaml input: - static: count: 50000 ``` Tagged for use with multiple event templates: ```yaml input: - static: count: 1000 tags: [batch] ``` # time_patterns Generates timestamps using time-pattern definitions — oscillators, multipliers, randomizers, and statistical distributions. Each pattern is defined in a separate YAML file, making it easy to compose complex traffic shapes by combining multiple patterns. Parameters [#parameters] | Parameter | Type | Default | Constraints | Description | | ---------- | --------------- | ------- | ---------------------------- | --------------------------------------------------------------------- | | `patterns` | list of paths | — | Required. At least one path. | Paths to time-pattern YAML files. | | `tags` | list of strings | `[]` | — | Tags attached to every timestamp. Accessible in templates via `tags`. | ```yaml input: - time_patterns: patterns: - patterns/traffic.yml - patterns/errors.yml ``` Pattern file schema [#pattern-file-schema] Each pattern file defines a pipeline of four stages that shape the timestamp distribution: ```yaml title="patterns/traffic.yml" label: api-traffic oscillator: period: 1 unit: days start: "2024-01-01" end: "2024-12-31" multiplier: ratio: 500 randomizer: deviation: 0.3 direction: mixed spreader: distribution: beta parameters: a: 2 b: 5 ``` oscillator [#oscillator] Defines the base periodic signal — the repeating time window. | Parameter | Type | Constraints | Description | | --------- | -------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------- | -------------------------- | | `period` | float | > 0 | Length of one cycle. | | `unit` | string | `weeks`, `days`, `hours`, `minutes`, `seconds`, `milliseconds`, or `microseconds` | Unit for the period. | | `start` | [VersatileDatetime](/docs/core/concepts/scheduling#date-ranges-and-versatile-datetime) | Required. Cannot be null. | Start of the active range. | | `end` | [VersatileDatetime](/docs/core/concepts/scheduling#date-ranges-and-versatile-datetime) | Required. Cannot be null. | End of the active range. | multiplier [#multiplier] Scales the number of timestamps per period. | Parameter | Type | Constraints | Description | | --------- | ------- | ----------- | ------------------------------------------------ | | `ratio` | integer | >= 1 | Base number of timestamps per oscillator period. | randomizer [#randomizer] Adds noise to the multiplied count, so each period generates a slightly different number of timestamps. | Parameter | Type | Default | Constraints | Description | | ----------- | ------- | ------- | ---------------------------------- | ------------------------------------------------- | | `deviation` | float | — | 0.0–1.0 | Maximum percentage deviation from the base count. | | `direction` | string | — | `decrease`, `increase`, or `mixed` | Direction of the deviation. | | `sampling` | integer | `1024` | >= 16 | Number of samples for the noise distribution. | spreader [#spreader] Controls **where within each period** the timestamps land, using a statistical distribution. **uniform** — timestamps spread evenly: | Parameter | Type | Constraints | Description | | ----------------- | ------ | ------------------------- | --------------------------------- | | `distribution` | string | Must be `"uniform"`. | Distribution type. | | `parameters.low` | float | 0 ≤ low, low \< 1 | Lower bound of the uniform range. | | `parameters.high` | float | 0 \< high ≤ 1, high > low | Upper bound of the uniform range. | **triangular** — timestamps cluster around a peak: | Parameter | Type | Constraints | Description | | ------------------ | ------ | ----------------------- | ------------------ | | `distribution` | string | Must be `"triangular"`. | Distribution type. | | `parameters.left` | float | 0 ≤ left, left \< 1 | Left bound. | | `parameters.mode` | float | left ≤ mode ≤ right | Peak position. | | `parameters.right` | float | 0 \< right ≤ 1 | Right bound. | **beta** — flexible shape for skewed or bimodal patterns: | Parameter | Type | Constraints | Description | | -------------- | ------ | ----------------- | ---------------------- | | `distribution` | string | Must be `"beta"`. | Distribution type. | | `parameters.a` | float | >= 0 | Alpha shape parameter. | | `parameters.b` | float | >= 0 | Beta shape parameter. | See [Scheduling — Simulate realistic traffic with time patterns](/docs/core/concepts/scheduling#simulate-realistic-traffic-with-time-patterns) for a practical walkthrough. # timer Generates timestamps at a fixed interval. Simpler than [cron](/docs/plugins/input/cron) when you just need a steady tick rate. Parameters [#parameters] | Parameter | Type | Default | Constraints | Description | | --------- | -------------------------------------------------------------------------------------- | ------- | ---------------- | --------------------------------------------------------------------- | | `seconds` | float | — | Required. >= 0.1 | Interval between ticks in seconds. | | `count` | integer | — | Required. >= 1 | Number of timestamps per tick. | | `start` | [VersatileDatetime](/docs/core/concepts/scheduling#date-ranges-and-versatile-datetime) | `null` | — | When to start. Defaults to current time if omitted. | | `repeat` | integer or null | `null` | >= 1 or null | Number of cycles. `null` means repeat indefinitely. | | `tags` | list of strings | `[]` | — | Tags attached to every timestamp. Accessible in templates via `tags`. | Examples [#examples] 10 events every half second, forever: ```yaml input: - timer: seconds: 0.5 count: 10 ``` One event per second, 60 times total: ```yaml input: - timer: seconds: 1 count: 1 repeat: 60 ``` Delayed start — begin 5 minutes from now: ```yaml input: - timer: seconds: 2 count: 1 start: +5m ``` # timestamps Emits timestamps from a predefined list or a file. Use this when you need events at specific, known times — for example, to reproduce a scenario or fill gaps in a dataset. Parameters [#parameters] | Parameter | Type | Default | Constraints | Description | | --------- | ------------------------- | ------- | ----------- | ---------------------------------------------------------------------------------- | | `source` | list of datetimes or path | — | Required | Inline list of ISO 8601 timestamps, or path to a file with one timestamp per line. | | `tags` | list of strings | `[]` | — | Tags attached to every timestamp. Accessible in templates via `tags`. | Timestamps must be in ascending order. Examples [#examples] Inline list: ```yaml input: - timestamps: source: - "2024-06-01T09:00:00" - "2024-06-01T09:15:00" - "2024-06-01T09:30:00" - "2024-06-01T10:00:00" ``` From a file (one ISO 8601 timestamp per line): ```yaml input: - timestamps: source: data/timestamps.txt ``` ```txt title="data/timestamps.txt" 2024-06-01T09:00:00 2024-06-01T09:15:00 2024-06-01T09:30:00 ``` # clickhouse Inserts events into a ClickHouse table using the HTTP interface. Supports configurable input formats, DSN connection strings, and TLS. Parameters [#parameters] | Parameter | Type | Default | Constraints | Description | | ------------------ | ------------------------------------- | --------------- | ------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `host` | string | — | Required. Non-empty. | ClickHouse server hostname or IP. | | `port` | integer | `8123` | 1–65535 | HTTP/HTTPS port. | | `protocol` | string | `"http"` | `"http"` or `"https"` | Connection protocol. | | `database` | string | `"default"` | Non-empty. | Target database. | | `table` | string | — | Required. Non-empty. | Target table. | | `username` | string | `"default"` | Non-empty. | Authentication username. | | `password` | string | `""` | — | Authentication password. | | `dsn` | ClickHouse DSN or null | `null` | — | DSN string (e.g., `clickhouse+http://user:pass@host:8123/db`). | | `connect_timeout` | integer | `10` | >= 1 | Connection timeout in seconds. | | `request_timeout` | integer | `300` | >= 1 | Full request timeout in seconds. | | `client_name` | string or null | `null` | Non-empty if set. | Client name for the ClickHouse query log. | | `verify` | boolean | `true` | — | Whether to verify the server's TLS certificate. | | `ca_cert` | path or null | `null` | — | Path to CA certificate file. | | `client_cert` | path or null | `null` | — | Path to client certificate. Must be provided together with `client_cert_key`. | | `client_cert_key` | path or null | `null` | — | Path to client certificate key. Must be provided together with `client_cert`. | | `server_host_name` | string or null | `null` | Non-empty if set. | Expected server hostname in the TLS certificate. | | `tls_mode` | string or null | `null` | `"proxy"`, `"strict"`, or `"mutual"` | TLS verification mode. | | `proxy_url` | URL or null | `null` | Valid HTTP/HTTPS URL. | Proxy address. | | `pool_maxsize` | integer | `32` | >= 1 | Maximum number of HTTP connections kept in the pool toward the ClickHouse host. Raise alongside `generation.max_concurrency` to avoid pool exhaustion under bursts of concurrent writes. | | `input_format` | string | `"JSONEachRow"` | Valid ClickHouse input format. | ClickHouse [input format](https://clickhouse.com/docs/en/interfaces/formats) for the INSERT query. | | `header` | string | `""` | — | String prepended before all events in each batch. | | `footer` | string | `""` | — | String appended after all events in each batch. | | `separator` | string | `"\n"` | — | String inserted between events. | | `formatter` | [formatter](/docs/plugins/formatters) | `json` | — | How events are serialized before insertion. | Behavior [#behavior] * Events are sent via ClickHouse's raw insert API using the specified `input_format`. * The default combination of `json` formatter + `JSONEachRow` input format sends one JSON object per line. * For non-JSON formats (CSV, TabSeparated, etc.), use `header`, `footer`, and `separator` to control event wrapping and the `plain` formatter. * `dsn` fills only the connection values that are not set otherwise. Since `host` is required and `port`, `database` and `username` carry defaults, a DSN contributes the password when `password` is left empty. Examples [#examples] Basic JSONEachRow insertion: ```yaml output: - clickhouse: host: clickhouse.example.com database: analytics table: events username: ${params.ch_user} password: ${secrets.ch_password} ``` HTTPS with custom port: ```yaml output: - clickhouse: host: clickhouse.prod port: 8443 protocol: https database: analytics table: events verify: true input_format: JSONEachRow ``` CSV input format with plain formatter: ```yaml output: - clickhouse: host: localhost table: events input_format: CSV separator: "\n" formatter: format: plain ``` # file Writes events to a local file. Automatically reopens the file if it is deleted or rotated, and closes the file handle after a period of inactivity to prevent handle leaks. Parameters [#parameters] | Parameter | Type | Default | Constraints | Description | | ------------------ | ------------------------------------- | -------------------------- | --------------------------- | ------------------------------------------------------- | | `path` | path | — | Required | Output file path. | | `flush_interval` | float | `1` | >= 0 | Seconds between buffer flushes. | | `cleanup_interval` | float | `10` | >= 1.0 | Seconds of inactivity before the file handle is closed. | | `file_mode` | integer | `640` | 0–7777 | Unix file permissions (octal notation). | | `write_mode` | string | `"append"` | `"append"` or `"overwrite"` | Whether to append to or overwrite the file. | | `encoding` | string | `"utf_8"` | Valid Python codec name. | File encoding. | | `separator` | string | `"\n"` (OS line separator) | — | String inserted between events. | | `formatter` | [formatter](/docs/plugins/formatters) | `plain` | — | How events are serialized before writing. | Behavior [#behavior] * **Auto-reopen**: If the output file is deleted or becomes inaccessible while the generator runs (e.g., by an external log rotator), the plugin automatically recreates and reopens it on the next write. * **Auto-close**: After `cleanup_interval` seconds with no writes, the file handle is closed to prevent resource leaks. It reopens automatically on the next write. * **Write modes**: `append` adds to the end of the file; `overwrite` truncates the file on each open. Examples [#examples] Append JSON lines to a log file: ```yaml output: - file: path: output/events.jsonl formatter: format: json ``` Overwrite with pretty-printed JSON: ```yaml output: - file: path: output/latest.json write_mode: overwrite formatter: format: json indent: 2 ``` Custom separator and permissions: ```yaml output: - file: path: /var/log/eventum/events.log file_mode: 644 separator: "\n---\n" ``` # http Sends events to an HTTP endpoint. By default, events are batched into a single JSON array (`json-batch` formatter) and sent as one request per batch. Parameters [#parameters] | Parameter | Type | Default | Constraints | Description | | ----------------- | ------------------------------------- | ------------ | ------------------------------------------------------------- | ------------------------------------------------------------------------------------------ | | `url` | URL | — | Required. Valid HTTP/HTTPS URL. | Target endpoint URL. | | `method` | string | `"POST"` | `GET`, `HEAD`, `OPTIONS`, `POST`, `PUT`, `PATCH`, or `DELETE` | HTTP method. | | `success_code` | integer | `201` | >= 100 | Expected response status code. | | `headers` | mapping | `{}` | — | Custom request headers. | | `username` | string or null | `null` | Non-empty if set. | Username for HTTP basic auth. | | `password` | string or null | `null` | Non-empty if set. | Password for HTTP basic auth. | | `connect_timeout` | integer | `10` | >= 1 | Connection timeout in seconds. | | `request_timeout` | integer | `300` | >= 1 | Full request timeout in seconds. | | `verify` | boolean | `true` | — | Whether to verify the server's TLS certificate. | | `ca_cert` | path or null | `null` | — | Path to CA certificate file. | | `client_cert` | path or null | `null` | — | Path to client certificate. Must be provided together with `client_cert_key`. | | `client_cert_key` | path or null | `null` | — | Path to client certificate key. Must be provided together with `client_cert`. | | `proxy_url` | URL or null | `null` | Valid HTTP/HTTPS URL. | Proxy address. | | `concurrency` | integer | `100` | >= 1 | Maximum number of requests performed at a time. Also sets the size of the connection pool. | | `formatter` | [formatter](/docs/plugins/formatters) | `json-batch` | — | How events are serialized before sending. | Behavior [#behavior] * Events are batched by the generator's `batch.size` / `batch.delay` settings. How many requests a batch costs depends on the formatter: a batch formatter such as the default `json-batch` sends the whole batch in one request, while a formatter that serializes each event on its own (`plain`, `json`, `template`) sends one request per event. * The default `json-batch` formatter wraps all events in a JSON array — ideal for endpoints that accept batch payloads. * No more than `concurrency` requests are performed at a time; the rest wait for a free slot. Under a per-event formatter, keep `batch.size` within what the endpoint accepts before the generator's `write_timeout` cancels the write and counts its events as failed. * If the response code doesn't match `success_code`, the write is counted as failed. * On an `https://` URL the server's certificate is verified by default. For an endpoint with a self-signed or internal-CA certificate, set `ca_cert` to the issuing CA, or set `verify: false` to connect without the check. Examples [#examples] Basic POST to a webhook: ```yaml output: - http: url: https://api.example.com/events method: POST success_code: 200 headers: Content-Type: application/json ``` With authentication and custom API key: ```yaml output: - http: url: https://api.example.com/ingest headers: Authorization: "Bearer ${secrets.api_token}" Content-Type: application/json success_code: 200 ``` Mutual TLS with client certificates: ```yaml output: - http: url: https://secure.example.com/events verify: true ca_cert: certs/ca.pem client_cert: certs/client.pem client_cert_key: certs/client-key.pem ``` Via proxy: ```yaml output: - http: url: https://api.example.com/events proxy_url: http://proxy.internal:8080 ``` # kafka Produces events to Apache Kafka topics using async producer. Supports SASL/SSL authentication, compression, idempotent and transactional delivery. Parameters [#parameters] Connection [#connection] | Parameter | Type | Default | Constraints | Description | | ------------------------- | --------------- | -------- | --------------------- | ---------------------------------------------------- | | `bootstrap_servers` | list of strings | — | Required. Min 1 item. | Kafka broker addresses in `host:port` format. | | `client_id` | string or null | `null` | Non-empty if set. | Client name passed in each request to brokers. | | `metadata_max_age_ms` | integer | `300000` | >= 0 | Period after which metadata is force-refreshed (ms). | | `request_timeout_ms` | integer | `40000` | >= 1 | Produce request timeout (ms). | | `connections_max_idle_ms` | integer | `540000` | >= 0 | Close idle connections after this time (ms). | Topic & Message [#topic--message] | Parameter | Type | Default | Constraints | Description | | ---------- | -------------- | --------- | -------------------- | -------------------------------------------------------- | | `topic` | string | — | Required. Non-empty. | Target Kafka topic. | | `key` | string or null | `null` | Non-empty if set. | Message key applied to all produced messages. | | `encoding` | string | `"utf-8"` | Non-empty. | Encoding for converting event strings and keys to bytes. | Performance & Reliability [#performance--reliability] | Parameter | Type | Default | Constraints | Description | | ------------------------ | -------------- | --------- | ------------------------------------------ | -------------------------------------------------------------------- | | `acks` | integer | `1` | `0`, `1`, or `-1` | Acknowledgments: `0`=fire-and-forget, `1`=leader, `-1`=all replicas. | | `compression_type` | string or null | `null` | `"gzip"`, `"snappy"`, `"lz4"`, or `"zstd"` | Compression codec. | | `max_batch_size` | integer | `16384` | >= 1 | Max buffered data per partition (bytes). | | `max_request_size` | integer | `1048576` | >= 1 | Max produce request size (bytes). | | `linger_ms` | integer | `0` | >= 0 | Artificial delay for batching (ms). | | `retry_backoff_ms` | integer | `100` | >= 0 | Backoff between retries (ms). | | `enable_idempotence` | boolean | `false` | — | Exactly-once delivery guarantee. | | `transactional_id` | string or null | `null` | Non-empty if set. | Transactional producer identifier. | | `transaction_timeout_ms` | integer | `60000` | >= 1 | Transaction timeout (ms). | Security [#security] | Parameter | Type | Default | Constraints | Description | | ---------------------------- | -------------- | ------------- | ----------------------------------------------------------- | ------------------------------ | | `security_protocol` | string | `"PLAINTEXT"` | `"PLAINTEXT"`, `"SSL"`, `"SASL_PLAINTEXT"`, or `"SASL_SSL"` | Broker communication protocol. | | `sasl_mechanism` | string or null | `null` | `"PLAIN"`, `"SCRAM-SHA-256"`, or `"SCRAM-SHA-512"` | SASL authentication mechanism. | | `sasl_plain_username` | string or null | `null` | Non-empty if set. Must pair with `sasl_plain_password`. | SASL username. | | `sasl_plain_password` | string or null | `null` | Non-empty if set. Must pair with `sasl_plain_username`. | SASL password. | | `sasl_kerberos_service_name` | string | `"kafka"` | Non-empty. | Kerberos service name. | | `sasl_kerberos_domain_name` | string or null | `null` | Non-empty if set. | Kerberos domain name. | SSL/TLS [#ssltls] | Parameter | Type | Default | Constraints | Description | | -------------- | ------------ | ------- | ------------------------------ | --------------------------- | | `ssl_cafile` | path or null | `null` | — | Path to CA certificate. | | `ssl_certfile` | path or null | `null` | Must pair with `ssl_keyfile`. | Path to client certificate. | | `ssl_keyfile` | path or null | `null` | Must pair with `ssl_certfile`. | Path to client key. | Formatter [#formatter] | Parameter | Type | Default | Description | | ----------- | ------------------------------------- | ------- | ------------------------------------------- | | `formatter` | [formatter](/docs/plugins/formatters) | `json` | How events are serialized before producing. | Behavior [#behavior] * Events are produced to the specified Kafka topic asynchronously. * The default `json` formatter serializes each event as a single-line JSON string, then encodes it to bytes using the configured `encoding`. * When `enable_idempotence` is `true`, the producer ensures exactly-once delivery semantics. * The producer batches messages internally based on `linger_ms` and `max_batch_size` for throughput optimization. Examples [#examples] Basic production to a topic: ```yaml output: - kafka: bootstrap_servers: - broker1:9092 - broker2:9092 topic: events ``` SASL\_SSL authentication: ```yaml output: - kafka: bootstrap_servers: - kafka.prod:9093 topic: security-events security_protocol: SASL_SSL sasl_mechanism: SCRAM-SHA-256 sasl_plain_username: ${params.kafka_user} sasl_plain_password: ${secrets.kafka_password} ssl_cafile: certs/ca.pem ``` High-throughput with compression and batching: ```yaml output: - kafka: bootstrap_servers: - broker1:9092 - broker2:9092 - broker3:9092 topic: high-volume-events key: my-partition-key acks: 1 compression_type: lz4 linger_ms: 50 max_batch_size: 65536 enable_idempotence: true ``` # opensearch Indexes events into an OpenSearch (or Elasticsearch-compatible) cluster. Uses the **bulk API** for efficient batch indexing and supports load balancing across multiple nodes. Parameters [#parameters] | Parameter | Type | Default | Constraints | Description | | ----------------- | ------------------------------------- | ------- | -------------------------------------------------------- | ----------------------------------------------------------------------------- | | `hosts` | list of URLs | — | Required. At least one. Format: `https://:`. | Cluster node addresses. | | `username` | string | — | Required. Non-empty. | Authentication username. | | `password` | string | — | Required. Non-empty. | Authentication password. | | `index` | string | — | Required. Non-empty. | Target index name. | | `connect_timeout` | integer | `10` | >= 1 | Connection timeout in seconds. | | `request_timeout` | integer | `300` | >= 1 | Full request timeout in seconds. | | `verify` | boolean | `true` | — | Whether to verify the server's TLS certificate. | | `ca_cert` | path or null | `null` | — | Path to CA certificate file. | | `client_cert` | path or null | `null` | — | Path to client certificate. Must be provided together with `client_cert_key`. | | `client_cert_key` | path or null | `null` | — | Path to client certificate key. Must be provided together with `client_cert`. | | `proxy_url` | URL or null | `null` | Valid HTTP/HTTPS URL. | Proxy address. | | `formatter` | [formatter](/docs/plugins/formatters) | `json` | — | How events are serialized before indexing. | Behavior [#behavior] * Events are indexed using the OpenSearch bulk API for efficient batch writes. * When multiple `hosts` are listed, the plugin **round-robins** requests across them for load balancing. * Each event must be valid JSON — the default `json` formatter ensures this. * Certificates of the cluster nodes are verified by default. For nodes with a self-signed or internal-CA certificate, set `ca_cert` to the issuing CA, or set `verify: false` to connect without the check. Examples [#examples] Single-node cluster: ```yaml output: - opensearch: hosts: - https://opensearch:9200 username: admin password: ${secrets.opensearch_password} index: application-logs ``` Multi-node cluster with TLS: ```yaml output: - opensearch: hosts: - https://node1:9200 - https://node2:9200 - https://node3:9200 username: ${params.opensearch_user} password: ${secrets.opensearch_password} index: events verify: true ca_cert: /etc/ssl/opensearch-ca.pem ``` # stdout Writes events to the standard output or error stream. The simplest output plugin — useful for debugging, piping to another tool, or quick inspection. Parameters [#parameters] | Parameter | Type | Default | Constraints | Description | | ---------------- | ------------------------------------- | -------------------------- | ------------------------ | ----------------------------------------- | | `stream` | string | `"stdout"` | `"stdout"` or `"stderr"` | Target stream. | | `flush_interval` | float | `1` | >= 0 | Seconds between buffer flushes. | | `encoding` | string | `"utf_8"` | Valid Python codec name. | Stream encoding. | | `separator` | string | `"\n"` (OS line separator) | — | String inserted between events. | | `formatter` | [formatter](/docs/plugins/formatters) | `plain` | — | How events are serialized before writing. | Behavior [#behavior] * Buffered output is flushed every `flush_interval` seconds. Set to `0` for immediate output — useful during debugging, but reduces throughput. * Writing to `stderr` is useful when `stdout` is piped to another tool. Examples [#examples] Minimal — write events to stdout with defaults: ```yaml output: - stdout: {} ``` Immediate flush to stderr: ```yaml output: - stdout: stream: stderr flush_interval: 0 ``` JSON output to console: ```yaml output: - stdout: formatter: format: json indent: 2 ``` # tcp Sends events over a TCP connection. Useful for forwarding events to syslog receivers, SIEM collectors, or any service that accepts data over raw TCP. Parameters [#parameters] | Parameter | Type | Default | Constraints | Description | | ----------------- | ------------------------------------- | ----------------- | --------------------------- | ----------------------------------------------------------------------------- | | `host` | string | — | Required. Non-empty. | Hostname or IP address to connect to. | | `port` | integer | — | Required. 1–65535. | TCP port number. | | `encoding` | string | `"utf_8"` | Valid Python encoding name. | Encoding used to encode events before sending. | | `separator` | string | OS line separator | — | Separator appended after each event. | | `connect_timeout` | integer | `10` | >= 1 | Connection timeout in seconds. | | `ssl` | boolean | `false` | — | Whether to use SSL/TLS for the connection. | | `verify` | boolean | `true` | — | Whether to verify the server's TLS certificate. | | `ca_cert` | path or null | `null` | — | Path to CA certificate file. | | `client_cert` | path or null | `null` | — | Path to client certificate. Must be provided together with `client_cert_key`. | | `client_cert_key` | path or null | `null` | — | Path to client certificate key. Must be provided together with `client_cert`. | | `formatter` | [formatter](/docs/plugins/formatters) | `plain` | — | How events are serialized before sending. | Behavior [#behavior] * A single TCP connection is opened when the plugin starts and reused for all events within a generation run. * Events are encoded using the configured encoding, with the separator appended after each event. * All events in a batch are concatenated and sent as a single write for efficiency. * If the connection drops during writing, the write is counted as failed. The plugin does not attempt automatic reconnection. Examples [#examples] Send events to a syslog receiver: ```yaml output: - tcp: host: syslog.example.com port: 514 separator: "\n" ``` Forward JSON events to a SIEM collector: ```yaml output: - tcp: host: 10.0.0.50 port: 5044 separator: "\n" formatter: format: json ``` With TLS encryption: ```yaml output: - tcp: host: secure-collector.example.com port: 6514 ssl: true verify: true ca_cert: certs/ca.pem separator: "\n" ``` Mutual TLS with client certificates: ```yaml output: - tcp: host: secure-collector.example.com port: 6514 ssl: true verify: true ca_cert: certs/ca.pem client_cert: certs/client.pem client_cert_key: certs/client-key.pem separator: "\n" ``` # udp Sends each event as a separate UDP datagram. Useful for forwarding events to syslog receivers, SIEM collectors, or any service that accepts data over UDP. Parameters [#parameters] | Parameter | Type | Default | Constraints | Description | | ----------- | ------------------------------------- | ----------------- | --------------------------- | ---------------------------------------------- | | `host` | string | — | Required. Non-empty. | Hostname or IP address to send datagrams to. | | `port` | integer | — | Required. 1–65535. | UDP port number. | | `encoding` | string | `"utf_8"` | Valid Python encoding name. | Encoding used to encode events before sending. | | `separator` | string | OS line separator | — | Separator appended after each event. | | `formatter` | [formatter](/docs/plugins/formatters) | `plain` | — | How events are serialized before sending. | Behavior [#behavior] * A connected UDP socket is created when the plugin starts and reused for all events within a generation run. * Each event is sent as a separate UDP datagram with the separator appended, preserving natural UDP message boundaries. * If a single event fails to encode (e.g., due to encoding mismatch), it is skipped and the remaining events are still sent. * UDP is connectionless — there is no delivery confirmation. Events may be silently dropped by the network. * ICMP errors (e.g., destination unreachable) are logged but do not interrupt event delivery. Examples [#examples] Send events to a syslog receiver: ```yaml output: - udp: host: syslog.example.com port: 514 separator: "\n" ``` Forward JSON events to a log collector: ```yaml output: - udp: host: 10.0.0.50 port: 9000 separator: "\n" formatter: format: json ``` # Generate test data for ClickHouse A materialized view has to prove it keeps up with rows arriving continuously; a `TTL` clause needs rows old enough to actually expire; a dashboard query needs a month of partitions to scan. None of that holds against a table filled once and left static. Copying production data fills it but carries every customer record and payment detail into an environment built for testing, not for holding them. ClickHouse's own shortcuts don't help: `generateRandom()` writes one fixed batch of independently random values, and `INSERT ... SELECT` copies rows that already exist somewhere — usually production, the very table this is meant to avoid touching. Both confirm a column accepts the right type; neither shows whether a view keeps up with a real arrival rate or a partition holds what a dashboard expects to query. The `clickhouse` output plugin inserts each generated event straight into a table over ClickHouse's HTTP interface, at a rate shaped like real traffic and with values drawn from statistical distributions rather than one uniform random draw. A row's shape lives in a template; the plugin writes it continuously, as events are produced. Inserting test data into ClickHouse over HTTP [#inserting-test-data-into-clickhouse-over-http] ClickHouse is a columnar analytical database: it stores each column's values together on disk rather than each row, a layout built for scanning and aggregating billions of rows over a handful of columns — exactly the kind of query a dashboard, a rollup, or a monthly report runs. Two interfaces reach it: a native TCP protocol on port 9000, and an HTTP interface on port 8123 that accepts a query string and a request body over plain HTTP or HTTPS. Eventum's `clickhouse` output uses the HTTP interface exclusively, so nothing beyond an HTTP(S) endpoint needs to be reachable. An `INSERT` sent over HTTP names the target table and an **input format** that describes how the request body maps to rows — ClickHouse supports dozens of them, from `CSV` to `Avro` to `Parquet`. The plugin defaults to `JSONEachRow`: one JSON object per line, its keys matched against column names, which is also exactly what Eventum's own `json` [formatter](/docs/plugins/formatters) renders for each event. The two defaults are already paired to work together, so a batch of events becomes a batch of newline-delimited JSON objects, sent as the body of one insert request: ```text POST /?query=INSERT INTO . FORMAT JSONEachRow HTTP/1.1 { } { } ``` generateRandom vs Eventum [#generaterandom-vs-eventum] ClickHouse ships its own way to manufacture rows without an external tool: the [`generateRandom`](https://clickhouse.com/docs/sql-reference/table-functions/generate) table function produces a fixed number of rows matching a column structure, each value drawn independently at random. ```sql INSERT INTO observability.request_metrics SELECT * FROM generateRandom( 'timestamp DateTime64(6), service String, host String, endpoint String, status UInt16, latency_ms Float32, host_cpu_percent Float32, bytes_out UInt32', 1, 10, 5 ) LIMIT 1000; ``` That statement fills the table with 1,000 rows in one shot and stops. It's a genuinely useful way to confirm the table's column types accept the shape of data a real `INSERT` would send — exactly the smoke test it's built for. What it doesn't produce is anything resembling a real source: * `timestamp` lands on a uniformly random instant across the column's entire representable range instead of clustering near the present. * `status` is a uniformly random 16-bit integer rather than the mostly-`200` mix a real API returns. * `service`, `host`, and `endpoint` are random strings instead of values drawn from a fixed, meaningful set. * Every row is independent, so nothing about one row's `endpoint` says anything about the next row sharing it. None of that matters for a type check. All of it matters for a materialized view that aggregates by `service` over time, a `TTL` policy, or a query that prunes by partition — which is what a continuous, time-aware stream is for. Generate a stream with Eventum [#generate-a-stream-with-eventum] The generator below produces request-level performance metrics for an internal API — latency, a response size, and the host's CPU load alongside each request's outcome — the kind of observability data a team pushes into ClickHouse to back a latency dashboard or an SLO query, at a rate shaped to look like real traffic instead of a flat one. The template [#the-template] Each event is one API request's outcome: the service and host that handled it, the endpoint and status code, and three measurements drawn from a distribution shaped like the real thing instead of a flat range. [`module.rand`](/docs/plugins/event/template/modules) covers all of it. ```jinja title="generators/api-metrics/templates/request_metric.jinja" {%- set service = module.rand.choice(["checkout", "catalog", "auth", "search"]) -%} {%- set host = "api-" ~ module.rand.number.integer(1, 6) ~ ".prod.local" -%} {%- set endpoint = module.rand.choice(["/v1/orders", "/v1/products", "/v1/session", "/v1/search"]) -%} {%- set status = module.rand.weighted_choice({200: 90, 400: 4, 404: 3, 500: 3}) -%} {%- set latency_ms = module.rand.number.exponential(0.02) | round(2) -%} {%- set host_cpu_percent = module.rand.number.clamp(module.rand.number.gauss(42, 12), 0, 100) | round(1) -%} {%- set bytes_out = module.rand.number.lognormal(8.2, 0.9) | round | int -%} { "timestamp": "{{ timestamp.strftime('%Y-%m-%d %H:%M:%S.%f') }}", "service": "{{ service }}", "host": "{{ host }}", "endpoint": "{{ endpoint }}", "status": {{ status }}, "latency_ms": {{ latency_ms }}, "host_cpu_percent": {{ host_cpu_percent }}, "bytes_out": {{ bytes_out }} } ``` `status` is weighted toward `200` rather than spread evenly across four codes, `latency_ms` comes from an exponential distribution so most requests are fast with an occasional long one, `host_cpu_percent` comes from a Gaussian clamped to `[0, 100]` so it clusters around a mid-range load instead of spanning the full scale evenly, and `bytes_out` comes from a log-normal distribution so most responses are a modest size with an occasional much larger one — four different shapes for four different kinds of measurement, real distributions instead of the single uniform draw `generateRandom` uses for every column. `timestamp` renders with `strftime`, not `isoformat()`: ClickHouse's default text parsing for a `DateTime`/`DateTime64` column expects `YYYY-MM-DD HH:MM:SS[.ffffff]`, not the `T`-separated, offset-suffixed shape `isoformat()` produces, so matching ClickHouse's own format here avoids a parsing mismatch at insert time. A realistic arrival rate [#a-realistic-arrival-rate] A flat tick is the giveaway that traffic is synthetic — real API load rises and falls even within an hour. The [time-patterns](/docs/plugins/input/time-patterns) input models that shape instead of a flat rate, combining four stages: * **Oscillator** — a repeating one-hour window. * **Multiplier** — around 3,600 requests per hour (roughly one a second on average). * **Randomizer** — ±25% variation, so no two hours carry an identical count. * **Spreader** — a beta distribution clustering requests toward the middle of each hour and thinning out at the edges. ```yaml title="generators/api-metrics/patterns/request-rate.yml" label: request-rate oscillator: start: "now" end: "+24h" period: 1 unit: hours multiplier: ratio: 3600 randomizer: deviation: 0.25 direction: mixed spreader: distribution: beta parameters: a: 5 b: 5 ``` This window covers one day of traffic; set `end` to `"never"` instead to repeat the same hourly shape indefinitely for a generator meant to run continuously. Reserve that setting for live mode: sample mode ignores wall-clock pacing entirely, so an unbounded pattern there tries to produce its whole infinite range at once instead of spreading it out over real time. The generator config [#the-generator-config] The [clickhouse](/docs/plugins/output/clickhouse) output inserts each rendered event into the `request_metrics` table: ```yaml title="generators/api-metrics/generator.yml" input: - time_patterns: patterns: - patterns/request-rate.yml event: template: mode: all templates: - request_metric: template: templates/request_metric.jinja output: - clickhouse: host: clickhouse database: observability table: request_metrics username: ${params.ch_user} password: ${secrets.ch_password} formatter: format: json ``` `host`, `database`, and `table` name the destination — `database` falls back to `default` and `table` has no default, so it always needs setting explicitly. `formatter` is set to `json` here, which is also the plugin's default, paired with `input_format`, left unset here to keep its own default of `JSONEachRow` — the pairing described above. Every write from this output goes through a connection pool capped at `pool_maxsize` (default `32`) toward the ClickHouse host. [`generation.max_concurrency`](/docs/core/config/eventum-yml#concurrency-and-ordering) defaults to `100`, already above `pool_maxsize`. Raise `pool_maxsize` to match once a generator's actual concurrent write volume grows — otherwise the pool discards connections under a burst it wasn't sized for. No ClickHouse server within reach yet? Eventum opens every configured output plugin before a generator starts producing, so one that cannot reach its target fails the whole run rather than skipping itself. Swap the `clickhouse` block for `stdout` while checking a template and pattern — it renders the identical formatted string `clickhouse` would have sent as one row's JSON body. Point `host` at a real server once one exists; nothing else in the config changes. Store the password in the keyring [#store-the-password-in-the-keyring] `password` above is a `${secrets.*}` reference, not a plaintext value. Set it once in the encrypted keyring before running the generator: ```bash eventum-keyring set ch_password # Enter password of `ch_password`: ******** ``` Then run the generator, pointing at the same cryptfile: ```bash eventum generate --path generator.yml --id api-metrics --cryptfile ./cryptfile.cfg ``` If the secret is missing from the keyring, Eventum reports it and refuses to start rather than connecting with an empty password. See [Secrets](/docs/core/config/secrets) for how the keyring and `${secrets.*}` substitution work. ClickHouse sample data [#clickhouse-sample-data] A full run needs a reachable ClickHouse server to actually receive these rows. The config and template above were validated by pointing the same generator at `stdout` instead of `clickhouse`, since no server was reachable while writing this lesson. Three consecutive rows from an actual run: ```json title="Rows from an actual run" {"timestamp": "2026-07-12 12:15:56.568696", "service": "auth", "host": "api-6.prod.local", "endpoint": "/v1/session", "status": 500, "latency_ms": 63.59, "host_cpu_percent": 50.1, "bytes_out": 11813} {"timestamp": "2026-07-12 12:15:56.576664", "service": "auth", "host": "api-1.prod.local", "endpoint": "/v1/session", "status": 200, "latency_ms": 48.22, "host_cpu_percent": 8.7, "bytes_out": 5016} {"timestamp": "2026-07-12 12:15:56.849137", "service": "checkout", "host": "api-6.prod.local", "endpoint": "/v1/search", "status": 200, "latency_ms": 102.68, "host_cpu_percent": 54.2, "bytes_out": 5330} ``` The `clickhouse` output sends these same lines as the body of one `INSERT INTO observability.request_metrics FORMAT JSONEachRow` request, one line per row, matching a table shaped like this: ```sql title="Matching table shape" CREATE TABLE IF NOT EXISTS request_metrics ( timestamp DateTime64(6), service LowCardinality(String), host String, endpoint LowCardinality(String), status UInt16, latency_ms Float32, host_cpu_percent Float32, bytes_out UInt32 ) ENGINE = MergeTree() PARTITION BY toYYYYMM(timestamp) ORDER BY (service, timestamp) TTL timestamp + INTERVAL 30 DAY; ``` `PARTITION BY toYYYYMM(timestamp)` and the `TTL` clause are exactly the mechanics a one-shot `generateRandom()` sample can't exercise honestly: partitioning only groups rows sensibly when timestamps actually cluster near the present the way this stream's do, and a 30-day `TTL` only proves anything against rows that keep arriving past that window. FAQ [#faq] Reach for `generateRandom()` to confirm a table's column types accept the shape of data a real `INSERT` would send — it fills a table in one statement and needs nothing beyond a running ClickHouse server. Reach for Eventum when the test needs the table to behave like it does in production: rows arriving continuously rather than all at once, a realistic mix of values instead of a uniform random draw across each column's full range, and timestamps clustered near the present instead of scattered across the column's entire range — the difference that actually matters for a materialized view, a `TTL` policy, or a partition-pruning query. `pool_maxsize` (default `32`) caps how many HTTP connections the `clickhouse` output keeps open toward the server; `generation.max_concurrency` (default `100`, set at the top level of [eventum.yml](/docs/core/config/eventum-yml#concurrency-and-ordering)) caps how many write operations across all output plugins run at once. Because the second number is already larger than the first by default, a generator that actually reaches meaningful concurrency — several output plugins, or a `clickhouse` output receiving large batches — can exceed `pool_maxsize` before it exceeds `max_concurrency`, and the pool starts discarding connections instead of reusing them under that load. Raise `pool_maxsize` to match or exceed `max_concurrency` once real concurrent volume is a possibility; the default pairing is fine for the modest, single-output rate shown above. `JSONEachRow` — the default — is the right choice whenever `formatter` is `json`, since the two already agree on shape: one JSON object per line, matched against column names by key. Column order in the `CREATE TABLE` statement doesn't need to match the JSON key order; ClickHouse matches by name. For a non-JSON destination format — `CSV`, `TabSeparated`, and the rest of [ClickHouse's input formats](https://clickhouse.com/docs/en/interfaces/formats) — pair the matching `input_format` with the `plain` [formatter](/docs/plugins/formatters) and use `header`, `footer`, and `separator` to control how rows are wrapped, exactly as the [clickhouse reference](/docs/plugins/output/clickhouse#examples) shows for CSV. Either way, the target table's column types decide what a value needs to look like — a `DateTime64` column expects `YYYY-MM-DD HH:MM:SS[.ffffff]`, not an ISO 8601 string with a `T` and a UTC offset, which is why the template above renders `timestamp` with `strftime` instead of `isoformat()`. Related [#related] * The [delivery track](/docs/tutorials/delivery) for streaming synthetic data to OpenSearch, Kafka, and HTTP alongside ClickHouse * The [Kafka delivery lesson](/docs/tutorials/delivery/kafka) for producing the same kind of stream to a topic instead of a table * The [OpenSearch delivery lesson](/docs/tutorials/delivery/opensearch) for indexing it into a search cluster instead * The [Web clickstream scenario](/docs/tutorials/web-clickstream) for a stateful, session-based generator that streams into ClickHouse using an FSM instead of a flat `mode: all` template * The [Realistic timing](/docs/tutorials/realism/timing) lesson for shaping the arrival rate beyond a single beta spread * The [clickhouse output reference](/docs/plugins/output/clickhouse) for every connection, TLS, and performance parameter # Output formatters: a shape per destination A synthetic event stream rarely has just one destination in mind, and different destinations expect the same data shaped differently: a log file wants one JSON object on its own line so a shipper can tail it, a bulk-ingest API wants every event bundled into a single request body, and a partner endpoint wants its own named wrapper around the batch. Nothing about the event itself changes between these three — only the wire shape does. Baking that shape into the event template itself is the obvious fallback: open a bracket before the first event, comma-join the rest, close it after the last one, wrap the whole thing in whatever key one particular endpoint happens to want. It works for exactly one destination, and breaks the moment a second one needs a different shape — the template now carries wire-format logic that has nothing to do with the data it produces, duplicated and drifting a little more every time another destination joins. Eventum keeps that decision out of the template entirely. Every output plugin applies a **formatter** — a small, swappable step between "here is an event string" and "here is what gets written" — so the same generator config can hand a bulk-ingest API a single JSON array, a log file one compact line per event, and a partner's endpoint its own named envelope, by changing nothing but the `formatter` block on each output. One stream, a shape per destination [#one-stream-a-shape-per-destination] Eventum's pipeline produces one stream of events and hands an identical copy to every configured output plugin — a generator commonly writes to more than one destination at once. Formatting happens per output plugin, not once for the whole generator, so each destination's `formatter` block picks its own shape independently of what any other output plugin does with the same events. Point one output at a local file with `json` and another at a bulk API with `json-batch` in the same `output:` list, and the identical stream leaves as an NDJSON file on disk and a single batched array over HTTP, at the same time. The [formatters reference](/docs/plugins/formatters) documents every format and parameter in full; this lesson is about picking the right one for a given destination. Choosing a formatter [#choosing-a-formatter] Match the formatter to what the destination expects, not to whatever shape the event template happens to render — that stays the same regardless of formatter: | Destination | Formatter | | ------------------------------------------------------------- | ----------------------------------------------------------- | | Raw-text collector — a syslog listener, a plain-text log file | [`plain`](/docs/plugins/formatters#plain) | | NDJSON file, or any per-record index or stream | [`json`](/docs/plugins/formatters#json) | | Bulk-ingest API that accepts one array per call | [`json-batch`](/docs/plugins/formatters#json-batch) | | A custom per-event request body or line | [`template`](/docs/plugins/formatters#template) | | A named envelope, CSV, or XML batch | [`template-batch`](/docs/plugins/formatters#template-batch) | | Another Eventum generator's `http` input | `eventum-http-input` | `plain` and `json` look interchangeable whenever the event template already renders JSON, but they aren't: `plain` passes the string through exactly as produced, untouched and unchecked, while `json` parses it and re-serializes it compactly, catching anything that isn't valid JSON before it ever reaches the destination. NDJSON wants `json`, not `plain`, for exactly that reason — see the [NDJSON lesson](/docs/tutorials/formats/ndjson) for that pairing built out in full, `file`'s own `separator: "\n"` included. Per-event vs per-batch [#per-event-vs-per-batch] The six formats split into two groups by what they operate on. `plain`, `json`, and `template` are per-event: each takes N events and returns up to N formatted strings, one per input event — `json` can skip an individual malformed event without touching the rest of the batch. `json-batch`, `template-batch`, and `eventum-http-input` are per-batch: each takes N events and returns exactly one formatted string for the whole batch, however many events that batch held. That split changes two things a destination-shape decision alone doesn't cover. **Request count**: a per-event formatter means one write — one HTTP request, one Kafka message, one line appended to a file — for every event; a per-batch formatter collapses however many events the generator's own [`batch.size`/`batch.delay`](/docs/core/config/eventum-yml#generation) accumulated into a single write, trading many small writes for fewer, larger ones. **Failure blast radius**: a per-event write stands on its own, so one rejected or malformed event doesn't touch its neighbors; a per-batch write is all-or-nothing once it leaves Eventum — whatever the destination does with that one array or envelope, accept it or reject it, happens to every event the batch was carrying at once. [Batch vs per-request delivery](/docs/tutorials/delivery/http#batch-vs-per-request-delivery) works through that exact tradeoff for one destination in depth. Worked examples [#worked-examples] One event template, reused unchanged below — only the output block's formatter changes between examples. Write the shared template [#write-the-shared-template] A background job finishing on a worker queue: an ID, which queue it ran on, which worker picked it up, how long it took, and how it ended. Building it as a Jinja mapping and serializing it in one step with the `tojson` filter keeps every rendered event on a single line, the same technique the [CloudTrail lesson](/docs/tutorials/formats/cloudtrail) uses for its own nested records — `template` and `template-batch` forward whatever a template renders exactly as written, with no reformatting of their own, so a template spanning multiple lines would carry those line breaks straight into a batched result instead. ```jinja title="generators/job-events/templates/job-completed.jinja" {%- set status = module.rand.weighted_choice({"succeeded": 82, "failed": 11, "retried": 7}) -%} {%- set event = { "job_id": module.rand.crypto.uuid4(), "queue": module.rand.choice(["emails", "exports", "thumbnails", "webhooks"]), "worker_id": "worker-" ~ module.rand.number.integer(1, 12), "duration_ms": module.rand.number.exponential(0.004) | round(1), "status": status, "finished_at": timestamp.isoformat() } -%} {{ event | tojson }} ``` `duration_ms` uses `module.rand.number.exponential`, not a uniform range — most jobs finish quickly and a shrinking few take much longer, the shape a real queue's completion times actually follow. NDJSON file, via json [#ndjson-file-via-json] A [cron](/docs/plugins/input/cron) input ticks once a second, and a [file](/docs/plugins/output/file) output writes each event through [json](/docs/plugins/formatters#json): ```yaml title="generators/job-events/generator.yml" input: - cron: expression: "* * * * * *" count: 1 event: template: mode: all templates: - job_completed: template: templates/job-completed.jinja output: - file: path: output/jobs.ndjson separator: "\n" formatter: format: json ``` `json` validates and compacts each event to one line regardless of how the template rendered it; `file`'s own `separator: "\n"` is what then places each one on its own line — the pairing the [NDJSON lesson](/docs/tutorials/formats/ndjson) covers in full. A bulk array, via json-batch [#a-bulk-array-via-json-batch] A bulk-ingest API that accepts one array per call wants every event from a batch collected into a single JSON array instead of separate lines. Swap the output block for: ```yaml title="generators/job-events/generator.yml (output block only)" output: - file: path: output/jobs-batch.json write_mode: overwrite formatter: format: json-batch ``` Written to a file here, with `write_mode: overwrite`, so the array from the latest batch is easy to inspect on its own — a real bulk-ingest destination is normally an `http` output instead, where this is the same [`json-batch`](/docs/tutorials/delivery/http#batch-vs-per-request-delivery) default the HTTP delivery lesson sends as a single POST body per batch, no overwrite-or-append question involved since nothing is being appended to a local file. Every event in the batch lands inside one pair of brackets, comma-joined; how many events that is comes from the generator's own `batch.size`/`batch.delay`, not from the formatter. A named envelope, via template-batch [#a-named-envelope-via-template-batch] Some destinations don't want a bare array at all — they want it wrapped in a key of their own choosing, the way AWS CloudTrail wraps a batch in `{"Records": [...]}` (covered in full in the [CloudTrail lesson](/docs/tutorials/formats/cloudtrail)). `json-batch` can't produce that: it always writes a bare `[...]`, with no key to name. [`template-batch`](/docs/plugins/formatters#template-batch) hands the whole batch to a template instead and lets it decide the wrapper: ```yaml title="generators/job-events/generator.yml (output block only)" output: - file: path: output/jobs-envelope.json write_mode: overwrite formatter: format: template-batch template: '{"jobs": [{{ events | join(", ") }}]}' ``` Written to a file for the same reason as the previous step. `events` inside a `template-batch` template is the list of raw rendered strings from the first step — each one already valid JSON text, not a parsed object. `events | join(", ")` concatenates them as they are, so wrapping the result in `[...]` reassembles a genuine JSON array under the `jobs` key. Writing `{{ events }}` directly instead, without the filter, renders Jinja's own textual representation of that list — comma-separated and single-quoted, not valid JSON at all. The result [#the-result] Three separate runs of the configs above, eight jobs each. `json` produces eight independent, compact lines — any one of them parses on its own, and a line-oriented tool can start on the first line without waiting for the rest: ```json title="output/jobs.ndjson" {"duration_ms": 109.7, "finished_at": "2026-07-17T21:30:55+00:00", "job_id": "9a2bd766-79f3-4d61-8c1b-6ef9753286e4", "queue": "emails", "status": "succeeded", "worker_id": "worker-3"} {"duration_ms": 20.9, "finished_at": "2026-07-17T21:30:54+00:00", "job_id": "521faeb9-a61a-42e9-b4be-1aa8e7e3cc0a", "queue": "emails", "status": "succeeded", "worker_id": "worker-10"} {"duration_ms": 40.9, "finished_at": "2026-07-17T21:30:56+00:00", "job_id": "1732c98b-179f-46a8-9d59-1bbbd733f833", "queue": "exports", "status": "succeeded", "worker_id": "worker-1"} {"duration_ms": 131.1, "finished_at": "2026-07-17T21:30:57+00:00", "job_id": "b2f9c855-c240-4e58-8446-c78bd48886db", "queue": "thumbnails", "status": "succeeded", "worker_id": "worker-6"} {"duration_ms": 320.7, "finished_at": "2026-07-17T21:30:58+00:00", "job_id": "4bd0ebac-209a-4d67-8a3f-690dafb63533", "queue": "emails", "status": "succeeded", "worker_id": "worker-3"} {"duration_ms": 333.0, "finished_at": "2026-07-17T21:30:59+00:00", "job_id": "29924f02-91f8-4b13-a9df-237f515c897e", "queue": "exports", "status": "succeeded", "worker_id": "worker-5"} {"duration_ms": 368.3, "finished_at": "2026-07-17T21:31:00+00:00", "job_id": "155159f4-2d50-472d-bb08-f531cd5b441b", "queue": "webhooks", "status": "succeeded", "worker_id": "worker-7"} {"duration_ms": 149.3, "finished_at": "2026-07-17T21:31:01+00:00", "job_id": "de950fb0-5f27-4205-872f-4f97acebf154", "queue": "emails", "status": "retried", "worker_id": "worker-2"} ``` `json-batch` collapses the same eight jobs into a single array — this is what would ship as one POST body against a bulk-ingest API: ```json title="output/jobs-batch.json" [{"duration_ms": 14.2, "finished_at": "2026-07-17T21:30:55+00:00", "job_id": "d4b74395-0518-43c1-94f7-b7a54c7730c6", "queue": "thumbnails", "status": "succeeded", "worker_id": "worker-10"}, {"duration_ms": 137.8, "finished_at": "2026-07-17T21:30:56+00:00", "job_id": "bed7d0e8-8c0b-4867-8c1f-715bc502417e", "queue": "webhooks", "status": "succeeded", "worker_id": "worker-2"}, {"duration_ms": 19.4, "finished_at": "2026-07-17T21:30:57+00:00", "job_id": "b224c0b3-3707-45c6-af17-0c1f84fc5fc6", "queue": "thumbnails", "status": "succeeded", "worker_id": "worker-6"}, {"duration_ms": 320.0, "finished_at": "2026-07-17T21:30:58+00:00", "job_id": "bedc9a44-1f6b-44da-b33d-870cad19f052", "queue": "emails", "status": "succeeded", "worker_id": "worker-5"}, {"duration_ms": 67.5, "finished_at": "2026-07-17T21:30:59+00:00", "job_id": "fc0d95d0-6948-47ff-992c-3cb54ce39f49", "queue": "thumbnails", "status": "failed", "worker_id": "worker-9"}, {"duration_ms": 196.1, "finished_at": "2026-07-17T21:31:00+00:00", "job_id": "25602229-e870-4802-80fd-723d185d911e", "queue": "webhooks", "status": "succeeded", "worker_id": "worker-1"}, {"duration_ms": 256.3, "finished_at": "2026-07-17T21:31:01+00:00", "job_id": "eb5fc86e-9bed-4cda-a01d-767cf82accdb", "queue": "thumbnails", "status": "succeeded", "worker_id": "worker-6"}, {"duration_ms": 165.4, "finished_at": "2026-07-17T21:31:02+00:00", "job_id": "2cdf55de-19d3-49d4-8715-681ac3fd424b", "queue": "exports", "status": "succeeded", "worker_id": "worker-10"}] ``` `template-batch` wraps the same eight jobs in the named `jobs` envelope instead of a bare array: ```json title="output/jobs-envelope.json" {"jobs": [{"duration_ms": 116.6, "finished_at": "2026-07-17T21:30:56+00:00", "job_id": "3948edb5-afbe-4278-8ea3-47d0d96ae3bd", "queue": "emails", "status": "succeeded", "worker_id": "worker-8"}, {"duration_ms": 172.0, "finished_at": "2026-07-17T21:30:57+00:00", "job_id": "a613dbc9-f759-40b0-8b01-b2ca71ad96f8", "queue": "webhooks", "status": "succeeded", "worker_id": "worker-10"}, {"duration_ms": 572.8, "finished_at": "2026-07-17T21:30:58+00:00", "job_id": "13361b47-ca1c-4e0b-8cf4-6467561343b1", "queue": "exports", "status": "succeeded", "worker_id": "worker-6"}, {"duration_ms": 125.7, "finished_at": "2026-07-17T21:30:59+00:00", "job_id": "a018f0ad-e6b8-4d50-9b06-7c9fa95fe25f", "queue": "webhooks", "status": "failed", "worker_id": "worker-3"}, {"duration_ms": 158.7, "finished_at": "2026-07-17T21:31:00+00:00", "job_id": "e2395d26-cc3f-4ae5-9011-6eda60754cac", "queue": "thumbnails", "status": "succeeded", "worker_id": "worker-1"}, {"duration_ms": 299.8, "finished_at": "2026-07-17T21:31:01+00:00", "job_id": "90817eed-e7fa-4897-8002-20ef82a342c0", "queue": "webhooks", "status": "failed", "worker_id": "worker-12"}, {"duration_ms": 41.9, "finished_at": "2026-07-17T21:31:02+00:00", "job_id": "bfa16842-493e-4ffe-af9b-f5643dcd4d72", "queue": "exports", "status": "failed", "worker_id": "worker-7"}, {"duration_ms": 435.3, "finished_at": "2026-07-17T21:31:03+00:00", "job_id": "a011ae6a-2045-463b-9dca-bdac9882c5d9", "queue": "exports", "status": "succeeded", "worker_id": "worker-12"}]} ``` Every field above sits at a single space after `:` and `,` — `json` and `json-batch` default to `indent: 0`, which means compact, not stripped; a positive `indent` pretty-prints the same data across multiple lines instead. Three separate runs, one template, one input — only the `formatter` block changed, following the mapping table above. FAQ [#faq] Yes. Batching happens once, upstream of every output plugin — the generator's own `batch.size`/`batch.delay` decide when a batch flushes, and every configured output plugin receives that identical batch to format and write in its own way. Formatting is the only thing that differs per output block: a `file` output set to `json` and an `http` output set to `json-batch` in the same generator both act on the same events at the same batch boundaries, they just package them differently on the way out. It depends on the formatter's own granularity. `json` and `template` skip just that one event — each is formatted independently, so one failure is logged as a format error and the rest of the batch ships normally. `json-batch` also isolates it: an invalid event is simply excluded from the array, not the whole array. `template-batch` is the one exception worth watching — the whole batch renders through a single template call, so a template that only joins the raw strings, like the envelope above, is unaffected by what's inside them, but one that inspects individual events, parsing each with a filter such as `fromjson`, fails for the entire batch the moment one event doesn't parse. Mainly for a file meant for a person to read directly — a sample dataset, a fixture, a config-review artifact — where a positive `indent` spreads each event across multiple lines for readability. Leave it at the default `0` for anything a machine parses next: a pretty-printed array parses identically, it just costs more bytes for no benefit once nothing human is reading it directly, and, as the [NDJSON lesson](/docs/tutorials/formats/ndjson) warns, a non-zero `indent` on `json` breaks the one-event-per-line shape a log shipper or `jq` depends on, since the event itself then spans several physical lines. Chaining one generator's output into a second generator's input, rather than shaping the events themselves. It ignores the batch's content entirely and emits a single `{"count": N}` object — the exact request body Eventum's own [http](/docs/plugins/input/http) input plugin expects on its [on-demand endpoint](/docs/core/concepts/scheduling#on-demand-scheduling): N timestamps generated the moment that request lands. Point an `http` output using this formatter at a second generator's `http` input port, and every batch the first generator produces triggers the second to generate that many events of its own. Related [#related] * The [formatters reference](/docs/plugins/formatters) for every parameter, plus the CSV, XML, and summary-report examples this lesson didn't rebuild * The [NDJSON lesson](/docs/tutorials/formats/ndjson) for the json-to-NDJSON pairing built out with its own two-template generator * The [AWS CloudTrail lesson](/docs/tutorials/formats/cloudtrail) for the Records envelope this lesson's named wrapper is modeled on * The [HTTP delivery lesson](/docs/tutorials/delivery/http) for json-batch vs json request shaping against a real endpoint * The [OpenSearch delivery lesson](/docs/tutorials/delivery/opensearch) for the bulk body a json-shaped stream feeds directly * The [formats field guide](/docs/tutorials/formats) for the log and event shapes formatters end up producing * The [delivery track](/docs/tutorials/delivery) for every destination these shapes actually ship to # Send test data to an API endpoint A webhook receiver only proves itself against requests that actually arrive on the wire, with payloads shaped like whatever a real integration sends and no two requests identical. The system that would normally produce that traffic — a payment processor, a client SDK, a partner's platform — is usually the one piece not available yet, and pointing a new receiver at a live integration just to see whether it parses a payload correctly is not a risk most teams take. Reaching for `curl` in a shell loop is the obvious shortcut, and it does put requests on the wire — but every iteration sends the exact same JSON body, at whatever interval a `sleep` between iterations happens to produce, which is not what traffic from an independent, real source looks like: neither the payload nor the pace varies from one request to the next. The `http` output plugin delivers every generated event as an HTTP request — individually, or batched into a single request per batch — continuously, in whatever shape the template renders. The receiver sees traffic that behaves like a real, independent event source instead of one payload replayed on a timer. Test data for an API endpoint [#test-data-for-an-api-endpoint] Every request the `http` output sends is built from the same handful of pieces: a target `url`, an HTTP `method` — `POST` by default, though `GET`, `PUT`, `PATCH`, `DELETE`, `HEAD`, and `OPTIONS` are all available — and `headers`. `headers` carries whatever the destination expects — typically a content type, plus a credential: a bearer token, or a username and password that Eventum turns into HTTP basic auth automatically. Whether a request counts as delivered comes down to one check: the response's status code against `success_code`, an expected value the plugin compares every response against. Anything else — a different `2xx`, a `4xx`, a `5xx`, or a request that times out before a response arrives — counts as a failed write rather than a successful one. That makes `success_code` a contract with the destination, not a cosmetic setting: it has to match exactly what the real endpoint returns for a call it accepts. That shape covers most of what a synthetic stream needs to reach over HTTP: a webhook receiver waiting on a payment processor's or a partner's event notifications, an ingest API accepting application or analytics events, an HTTP-based log collector such as a Fluentd or Logstash HTTP input, and an API endpoint being [load-tested](/docs/tutorials/load-testing) with realistic traffic instead of one hand-crafted request repeated at maximum speed. Batch vs per-request delivery [#batch-vs-per-request-delivery] Every output plugin shapes events through a [formatter](/docs/plugins/formatters) before writing them, and for `http` the formatter decides something that matters more than usual: how many requests a batch of events turns into. The default, [`json-batch`](/docs/plugins/formatters#json-batch), validates every event in a batch as JSON, joins them into a single JSON array, and sends that array as the body of one request — a batch of fifty events becomes one `POST` carrying all fifty. Setting `formatter` to [`json`](/docs/plugins/formatters#json) instead keeps every event a separate JSON string and sends one request per event, so the same batch of fifty becomes fifty independent requests instead. How many events land in a batch to begin with is not something the `http` block controls. That comes from the generator's `batch.size` and `batch.delay` — the application default in [eventum.yml](/docs/core/config/eventum-yml#generation), a per-generator override in [startup.yml](/docs/core/config/startup-yml#overridable-generation-parameters), or `--batch.size`/`--batch.delay` flags on [`eventum generate`](/docs/core/cli/eventum-generate) — whichever limit is reached first flushes the accumulated events onward to every output plugin, `http` included. The choice of formatter also decides how a single bad response plays out: under `json-batch`, one request carries the whole batch, so a response that doesn't match `success_code` fails every event that batch was carrying at once; under `json`, each event's request stands on its own, so one failure doesn't touch the rest. More on that in the FAQ below. Generate webhook test data with Eventum [#generate-webhook-test-data-with-eventum] The generator below produces the kind of payload a customer-engagement platform's webhook delivers whenever a user signs up, starts a trial, upgrades, or churns — realistic enough to test a webhook receiver or an ingest integration before wiring it to the real platform, with a different, freshly generated user behind every event instead of one fixture replayed forever. The template [#the-template] Each event is one lifecycle notification: an event type weighted toward the action that happens most often in a real product, an identifier, a timestamp, and the user behind it. [`module.rand`](/docs/plugins/event/template/modules) and [`module.faker`](/docs/plugins/event/template/modules#modulefaker) cover all of it. ```jinja title="generators/user-events/templates/user_event.jinja" {%- set event_type = module.rand.weighted_choice({"feature_used": 60, "trial_started": 15, "user_signed_up": 12, "subscription_upgraded": 8, "user_churned": 5}) -%} {%- set plan = module.rand.choice(["free", "pro", "enterprise"]) -%} { "event_id": "{{ module.rand.crypto.uuid4() }}", "event_type": "{{ event_type }}", "occurred_at": "{{ timestamp.isoformat() }}", "user": { "id": "{{ module.rand.crypto.uuid4() }}", "email": "{{ module.faker.locale['en_US'].email() }}", "plan": "{{ plan }}" } } ``` `event_type` is weighted toward `feature_used` — ordinary usage, not a lifecycle milestone — rather than spread evenly across all five outcomes, which mirrors how lopsided a real product's event mix actually is; every other value stays a plausible but rarer milestone. The generator config [#the-generator-config] A [cron](/docs/plugins/input/cron) input ticks once a second, and the [http](/docs/plugins/output/http) output posts each rendered event to the ingest endpoint: ```yaml title="generators/user-events/generator.yml" input: - cron: expression: "* * * * * *" count: 1 event: template: mode: all templates: - user_event: template: templates/user_event.jinja output: - http: url: https://api.example.com/ingest method: POST success_code: 200 headers: Content-Type: application/json Authorization: "Bearer ${secrets.api_token}" formatter: format: json-batch ``` `url` is the only field with no default — everything else here is shown for clarity even where it matches the plugin's own default. `method: POST` is that default; `success_code: 200` overrides the plugin's own default of `201`, matching an ingest endpoint that acknowledges a batch with a plain `200` instead of treating it as a single created resource. `headers` carries a bearer token pulled from the keyring through `${secrets.api_token}`, never written into the file as plain text. `formatter: json-batch` is, again, already the default — collecting whatever events this generator's `batch.size`/`batch.delay` accumulated into one JSON array and sending it as a single request body. No HTTP endpoint within reach yet? The `http` output creates its client lazily and does not test the `url` at startup, so an unreachable endpoint does not stop the run — every write simply fails in the background, logged rather than raised, and at the default verbosity that happens quietly. Swap the `http` block for `stdout` while checking a template and payload shape — paired with the same `json-batch` formatter, it renders the identical batched JSON array `http` would have sent as the request body, where you can see it directly. Point `url` at a real endpoint once one exists; nothing else in the config changes. Store the token in the keyring [#store-the-token-in-the-keyring] `headers.Authorization` above is a `${secrets.*}` reference, not a plaintext token. Set it once in the encrypted keyring before running the generator: ```bash eventum-keyring set api_token # Enter password of `api_token`: ******** ``` Then run the generator, pointing at the same cryptfile: ```bash eventum generate --path generator.yml --id user-events --cryptfile ./cryptfile.cfg ``` If the secret is missing from the keyring, Eventum reports it and refuses to start rather than sending requests with an empty `Authorization` header. See [Secrets](/docs/core/config/secrets) for how the keyring and `${secrets.*}` substitution work. The result [#the-result] A full run needs a reachable HTTP endpoint to actually receive these requests. The config and template above were validated by pointing the same generator at `stdout` instead of `http`, with a small `--batch.size 5` so one batch prints on a single line, since no endpoint was reachable while writing this lesson. One batch from an actual run, exactly as `json-batch` shaped it into a single request body: ```json title="One batch, as it would ship in a single POST body" [{"event_id": "79175abd-5bf2-49c4-bc9e-f616a7684bd1", "event_type": "feature_used", "occurred_at": "2026-07-12T12:40:40+00:00", "user": {"id": "8dd8473d-effd-44b7-8279-a5e70ebc0d71", "email": "egraves@example.net", "plan": "pro"}}, {"event_id": "9dc1afcf-9047-445b-822e-a37611b68fa5", "event_type": "user_churned", "occurred_at": "2026-07-12T12:40:41+00:00", "user": {"id": "83d54093-eb4b-4526-a885-2e336f29a21e", "email": "gregory99@example.net", "plan": "pro"}}, {"event_id": "e8827d04-166f-4bc6-b5d0-755cc9b230e3", "event_type": "user_signed_up", "occurred_at": "2026-07-12T12:40:42+00:00", "user": {"id": "c27a0502-e195-4150-a477-730c21898a51", "email": "teresa27@example.com", "plan": "enterprise"}}, {"event_id": "c2d0f92d-b413-47b2-8fdc-6dd727c03e88", "event_type": "trial_started", "occurred_at": "2026-07-12T12:40:43+00:00", "user": {"id": "7414d44d-15ef-4054-bba8-d61a27e426a2", "email": "harriseric@example.net", "plan": "enterprise"}}, {"event_id": "cf2eec4a-c595-405a-a099-e8117b40dd1b", "event_type": "trial_started", "occurred_at": "2026-07-12T12:40:44+00:00", "user": {"id": "8a24e82c-5803-46ed-b947-2ba1bcc2e45e", "email": "lawrence19@example.com", "plan": "enterprise"}}] ``` The `http` output sends exactly this string as the body of one `POST https://api.example.com/ingest` request, with the configured headers attached. A `200` response counts all five of these events as delivered in one write; anything else fails all five at once — the same tradeoff `json-batch` makes at any batch size, fifty events or five. FAQ [#faq] `json-batch` — the default — is the right choice whenever the destination accepts an array of records in one call, which most ingest APIs and modern webhook receivers document explicitly: fewer requests, lower overhead, and a batch that either lands as a whole or doesn't. Switch to [`json`](/docs/plugins/formatters#json) when the destination's API only accepts one record per call, or when each event's delivery outcome needs to be independent of the others — a `json` request that fails doesn't take the rest of the batch down with it, while a `json-batch` request that fails takes all of them. Either way, how many events end up in a batch comes from the generator's `batch.size`/`batch.delay`, not from the `http` output or its formatter. Three ways, matching whatever the destination expects. A bearer token goes in `headers` as `Authorization: "Bearer ${secrets.api_token}"` — shown above — resolved from the encrypted keyring at load time rather than written as plain text. HTTP basic auth uses the dedicated `username`/`password` fields instead, which can just as well be `${secrets.*}` references; Eventum attaches them as a standard `Authorization: Basic ...` header on every request. Beyond that, `client_cert`/`client_cert_key` cover mutual TLS, and `ca_cert` verifies the server's own certificate once `verify` is set to `true` (it defaults to `false`) — see the [http reference](/docs/plugins/output/http#parameters) for the rest, and [Secrets](/docs/core/config/secrets) for how the keyring stores every credential referenced with `${secrets.*}`. Eventum compares the destination's actual response status code against `success_code` (`201` by default) for every request. A different `2xx`, a `4xx`, a `5xx`, or a timeout before any response arrives — all of these count as a failed write. Eventum logs the response body and does not retry automatically. Which events that failure touches depends on the formatter: under `json-batch`, one response covers the whole batch, so a single non-matching response fails every event that request was carrying; under `json`, only the one event whose request came back wrong fails, and the rest of the batch succeeds or fails independently. Set `success_code` to whatever the real endpoint actually returns on success — many ingest APIs answer a batch with a plain `200` or `202`, not the plugin's own `201` default, which assumes a single resource just got created. Related [#related] * The [delivery track](/docs/tutorials/delivery) for streaming synthetic data to OpenSearch, Kafka, and ClickHouse alongside HTTP * The [OpenSearch delivery lesson](/docs/tutorials/delivery/opensearch) for indexing the same kind of stream into a search cluster instead of posting it to an endpoint * The [Kafka delivery lesson](/docs/tutorials/delivery/kafka) for producing it to a topic instead * The [ClickHouse delivery lesson](/docs/tutorials/delivery/clickhouse) for inserting it into an analytical table instead * The [API Load Testing](/docs/tutorials/load-testing) tutorial for pushing the same kind of HTTP traffic at maximum throughput instead of a steady rate * The [Alert simulation: scheduled Telegram alerts](/docs/tutorials/telegram-alerts) tutorial for a concrete scenario built on this mechanism — posting generated alerts to the Telegram Bot API on a cron schedule * The [http output reference](/docs/plugins/output/http) for every timeout, TLS, and proxy parameter # Deliver synthetic data to a real backend Generating events is one stage; delivering them to a real backend is the next. Eventum fans out the same stream to one or more destinations in parallel, formatting per destination. This track shows how to deliver synthetic data to the systems you run. # Generate test data for Kafka A consumer, a stream-processing job, and a dashboard tracking consumer lag all need messages actually landing on a topic, at a pace that behaves like production traffic instead of a burst or a trickle. Wiring up a real producer just to get there is disproportionate for a test, and pointing a consumer at a mirror of the production topic carries every customer record and payment detail it holds into an environment built for testing, not for holding it. A demo connector replaying a handful of canned Avro records at a fixed interval, or a script publishing JSON in a loop, produces one message shape at a steady tick — never a rate that rises and falls like real traffic. The `kafka` output plugin produces every generated event to the topic as it's created — continuously, in whatever format the template renders, at a rate shaped to look like real traffic rather than a metronome. Point it at a broker and describe a message's shape in a template; the plugin handles the rest. Producing to a Kafka topic [#producing-to-a-kafka-topic] A Kafka **topic** is a named, append-only log — the category a producer publishes to and a consumer subscribes to, conceptually similar to a queue that several readers can replay independently. A topic is split into one or more **partitions**, each an ordered, immutable sequence of messages. Partitions are what let a topic scale, since different partitions can sit on different brokers and be written or read in parallel, and what let a consumer group parallelize reads, since each partition is owned by exactly one consumer in the group at a time. Every message can carry a **key**. The producer hashes the key to pick a partition deterministically, so messages sharing a key always land on the same partition and keep their relative order there — the mechanism behind per-entity ordering guarantees, such as every update for one order arriving on the consumer side in the sequence it was produced. A message with no key is spread across partitions to balance load, with no ordering guarantee across the topic as a whole. The **message value** is the payload itself, and Kafka treats it as an opaque byte string — JSON, Avro, Protobuf, CSV, or plain text are all valid, and nothing about the broker enforces one over another. The format is an agreement between producer and consumer; Eventum's [formatter](/docs/plugins/formatters) is what decides it on the producing side. Generate a stream with Eventum [#generate-a-stream-with-eventum] The generator below produces order-placed events for an online store's checkout service — a high-volume, non-security data source that exercises the same delivery path a clickstream or payment topic would, at a rate shaped to look like real traffic instead of a flat one. The template [#the-template] Each event is one placed order: an ID, the customer, a product category, a quantity and price, and a payment method. All of it comes from [module.rand](/docs/plugins/event/template/modules) and [module.faker](/docs/plugins/event/template/modules#modulefaker), so every run produces different but structurally consistent orders. ```jinja title="generators/orders/templates/order_placed.jinja" {%- set order_id = module.rand.string.pattern("ORD-%A{3}-%d{6}") -%} {%- set customer_name = module.faker.locale['en_US'].name() -%} {%- set customer_email = module.faker.locale['en_US'].email() -%} {%- set category = module.rand.choice(["electronics", "home", "apparel", "sporting-goods", "books"]) -%} {%- set quantity = module.rand.number.integer(1, 5) -%} {%- set unit_price = module.rand.number.floating(4.99, 249.99) | round(2) -%} {%- set payment_method = module.rand.weighted_choice({"card": 70, "paypal": 20, "gift-card": 10}) -%} { "order_id": "{{ order_id }}", "placed_at": "{{ timestamp.isoformat() }}", "customer": { "name": "{{ customer_name }}", "email": "{{ customer_email }}" }, "category": "{{ category }}", "quantity": {{ quantity }}, "unit_price": {{ "%.2f" | format(unit_price) }}, "total": {{ "%.2f" | format(quantity * unit_price) }}, "payment_method": "{{ payment_method }}" } ``` A realistic arrival rate [#a-realistic-arrival-rate] A fixed gap between messages is the giveaway that data is synthetic — real checkout traffic rises and falls even within a single hour. The [time-patterns](/docs/plugins/input/time-patterns) input models that shape instead of a flat rate, combining four stages: * **Oscillator** — a repeating one-hour window. * **Multiplier** — around 1,200 orders per hour (roughly 20 a minute on average). * **Randomizer** — ±20% variation, so no two hours look identical. * **Spreader** — a beta distribution clustering orders toward the middle of each hour and thinning out at the edges. ```yaml title="generators/orders/patterns/checkout-traffic.yml" label: checkout-traffic oscillator: start: "now" end: "+24h" period: 1 unit: hours multiplier: ratio: 1200 randomizer: deviation: 0.2 direction: mixed spreader: distribution: beta parameters: a: 5 b: 5 ``` This window covers one day of traffic; set `end` to `"never"` instead to repeat the same hourly shape indefinitely for a generator meant to run continuously. Reserve that setting for live mode: sample mode ignores wall-clock pacing entirely, so an unbounded pattern there tries to produce its whole infinite range at once instead of spreading it out over real time. The generator config [#the-generator-config] The [kafka](/docs/plugins/output/kafka) output produces each rendered order to the `orders` topic: ```yaml title="generators/orders/generator.yml" input: - time_patterns: patterns: - patterns/checkout-traffic.yml event: template: mode: all templates: - order_placed: template: templates/order_placed.jinja output: - kafka: bootstrap_servers: - kafka:9092 topic: orders key: checkout-service formatter: format: json ``` Four fields here do the real configuration work: * `bootstrap_servers` takes a list, so naming more than one broker lets the plugin discover the rest of the cluster even if the first address it tries is down. * `topic` is the only thing naming where messages land — `orders` here, which needs to already exist on the cluster unless the broker is configured to create topics automatically on first use. * `key` is a single value applied to every message this output plugin produces, not a per-event expression: set it to whatever identifies this generator's own stream, such as the service or shard it represents. Running several generators against the same topic, each with its own `key` — a store ID, a region, a service name — spreads those sources across the topic's partitions while keeping every source's own messages in the order they were produced. * `formatter` is set to `json` explicitly here, which is also the plugin's default: one compact JSON string per event, encoded to bytes with `encoding` (`utf-8` unless set otherwise). No Kafka broker within reach yet? Eventum opens every configured output plugin before a generator starts producing, so one that cannot reach its broker fails the whole run rather than skipping itself. Swap the `kafka` block for `stdout` while checking a template and pattern — it renders the identical formatted string `kafka` would have sent as the message value. Point `bootstrap_servers` at a real cluster once one exists; nothing else in the config changes. The result [#the-result] A full run needs a broker to actually receive these messages. The config and event shape above were validated by pointing the same template and pattern at `stdout` instead of `kafka`, since no cluster was reachable while writing this lesson. Two events from an actual run: ```json title="Events from an actual run" {"order_id": "ORD-FWX-826496", "placed_at": "2026-07-11T15:12:14.578246+00:00", "customer": {"name": "Karen Lopez", "email": "christopherortiz@example.org"}, "category": "sporting-goods", "quantity": 5, "unit_price": 161.76, "total": 808.80, "payment_method": "card"} {"order_id": "ORD-NZZ-582985", "placed_at": "2026-07-11T15:15:43.649283+00:00", "customer": {"name": "George Stevenson", "email": "fishersarah@example.net"}, "category": "books", "quantity": 5, "unit_price": 204.37, "total": 1021.85, "payment_method": "paypal"} ``` The `kafka` output turns each line into one message on the `orders` topic: the JSON string becomes the value, and `checkout-service` — the static `key` configured on the plugin — travels alongside every one of them: ```text title="Message as it lands on the orders topic" Key: checkout-service Value: {"order_id": "ORD-FWX-826496", "placed_at": "2026-07-11T15:12:14.578246+00:00", "customer": {"name": "Karen Lopez", "email": "christopherortiz@example.org"}, "category": "sporting-goods", "quantity": 5, "unit_price": 161.76, "total": 808.80, "payment_method": "card"} ``` Because every message from this generator carries that same key, all of them hash to the same partition and keep the order they were produced in — exactly the guarantee a consumer processing this stream in arrival order depends on. FAQ [#faq] [Kafka Connect Datagen](https://github.com/confluentinc/kafka-connect-datagen) is what most searches for Kafka test data turn up first: a Kafka Connect source connector that replays one of a handful of bundled, Avro-defined schemas — orders, users, pageviews, and a few others — at a fixed interval into a Kafka topic, for as long as the connector keeps running. It needs a Kafka Connect worker, and for Avro, a schema registry, running alongside the broker. Eventum is a single process, not a connector. The order rate above comes from time-patterns — an oscillator, multiplier, randomizer, and spreader — so the topic carries something that rises and falls like real traffic instead of one fixed gap between messages repeating forever. The message is whatever a Jinja2 template renders, in whatever format the formatter is set to, not a fixed schema. And the same generated stream can reach Kafka alongside OpenSearch, ClickHouse, a file, or an HTTP endpoint in the same run, which a Kafka-only connector cannot do. The `kafka` output supports it directly. `security_protocol` selects `SSL`, `SASL_PLAINTEXT`, or `SASL_SSL` in place of the default `PLAINTEXT`; `sasl_mechanism` (`PLAIN`, `SCRAM-SHA-256`, or `SCRAM-SHA-512`) together with `sasl_plain_username` and `sasl_plain_password` cover SASL, and `ssl_cafile`, `ssl_certfile`, and `ssl_keyfile` cover TLS, including mutual TLS once a client certificate and key are both set. Keep the password in the keyring rather than in the config: ```bash eventum-keyring set kafka_password # Enter password of `kafka_password`: ******** ``` ```yaml output: - kafka: bootstrap_servers: - kafka.prod:9093 topic: orders security_protocol: SASL_SSL sasl_mechanism: SCRAM-SHA-256 sasl_plain_username: checkout-service sasl_plain_password: ${secrets.kafka_password} ssl_cafile: certs/ca.pem ``` See [Secrets](/docs/core/config/secrets) for how `${secrets.*}` and the keyring work, and the [plugin reference](/docs/plugins/output/kafka#parameters) for every connection and security parameter. Two different things, both straightforward. For more than one topic from the same event stream, add another `kafka` block to `output` with a different `topic` — every output plugin in the list receives an identical copy of each generated event, whatever the destination. For fan-out to other systems, add an `opensearch`, `http`, `file`, or any other output alongside `kafka` the same way; each formats and delivers the same stream independently. To spread several logical sources across one topic's partitions while keeping each source's own messages in order, run one generator per source with its own `key` — a store ID, a region, a device fleet — using [parameters](/docs/core/config/parameters) to keep a single generator directory reusable across all of them. Related [#related] * The [delivery track](/docs/tutorials/delivery) for streaming synthetic data to OpenSearch, ClickHouse, and HTTP alongside Kafka * The [OpenSearch delivery lesson](/docs/tutorials/delivery/opensearch) for indexing the same kind of stream into a search cluster instead of a topic * The [kafka output reference](/docs/plugins/output/kafka) for every connection, security, and performance parameter * The [SIEM test data lesson](/docs/tutorials/siem-events) for a complete stateful generator — the same finite-state, multi-template approach used there applies to a `kafka` output exactly as it does OpenSearch # Generate logs for OpenSearch A dashboard, a detection rule, or an ingestion benchmark all need a realistic stream of log documents already sitting in an OpenSearch index. Waiting for production traffic to accumulate takes time most projects don't have, and copying real logs into a test cluster carries whatever they contain — IP addresses, session tokens, request bodies — into an environment built for testing, not for holding production data. Most ingestion tutorials assume the logs already exist somewhere; a script faking access-log lines produces text, not a document actually indexed. The `opensearch` output plugin sends every generated event to an index over the bulk API as it's produced — continuously in real time, or as a single finite batch for a one-off test dataset. A document's shape lives in a template, and the plugin indexes each event as it's generated. How log ingestion into OpenSearch works [#how-log-ingestion-into-opensearch-works] An OpenSearch **index** is where related documents live — the rough equivalent of a table, holding every log line that belongs together, typically because it came from the same source or shares a retention policy. A cluster usually holds many indices at once, one per application, log type, or day, and a client picks the index it wants by name whenever it writes or searches. Indexing one document at a time over HTTP works, but a request round trip per log line does not scale to a real log volume. The **bulk API** collects any number of index, update, or delete operations into a single HTTP request: the body is newline-delimited JSON, an action line naming the target index followed by the document itself, repeated for every operation and terminated with a trailing newline: ```text {"index": {"_index": ""}} { } {"index": {"_index": ""}} { } ``` One HTTP call carries any number of documents — the reason nearly every log shipper that talks to OpenSearch indexes through this endpoint rather than one request per event. Every index also has a **mapping** — the schema that says what fields a document may carry and how each one is stored: a full-text `text` field, an exact-match `keyword`, a `date`, a number. Without one, OpenSearch infers a mapping from the first documents it sees, a convenience that also means a field can silently end up the wrong type — an ISO-8601 timestamp string is recognized automatically as `date`, but a non-standard format is not and quietly becomes `text` instead, which breaks range queries and time-based dashboards. Production setups usually create the index with an explicit mapping before sending it any data; for evaluation or a dashboard prototype, the inferred mapping is normally enough to start from. The `opensearch` output plugin issues the bulk requests itself and lets OpenSearch create the index on first write. What's left for a generator config to decide is just which index to target and what a document looks like. Generate and index with Eventum [#generate-and-index-with-eventum] The generator below simulates a web server's access log — a realistic, non-security data source that exercises the same delivery path a SIEM stream would, with none of the session state a security scenario needs. The template [#the-template] Each event is one HTTP request: a method and path, a status code weighted toward success, a response size and latency, the client's address, and its user agent. All of it comes from [`module.rand`](/docs/plugins/event/template/modules) and [`module.faker`](/docs/plugins/event/template/modules#modulefaker), so every run produces different but structurally consistent requests. ```jinja title="generators/web-logs/templates/access_log.jinja" {%- set method = module.rand.weighted_choice({"GET": 75, "POST": 15, "PUT": 6, "DELETE": 4}) -%} {%- set path = module.rand.choice(["/", "/products", "/products/wireless-mouse", "/cart", "/checkout", "/api/search", "/login", "/static/app.css"]) -%} {%- set status = module.rand.weighted_choice({200: 82, 301: 3, 404: 9, 500: 6}) -%} {%- set bytes_sent = module.rand.number.integer(180, 24000) -%} {%- set response_time_ms = module.rand.number.integer(4, 620) -%} {%- set client_ip = module.rand.network.ip_v4_public() -%} {%- set user_agent = module.faker.locale['en_US'].user_agent() -%} {%- set referrer = module.rand.choice(["-", "https://google.com", "https://bing.com", "https://duckduckgo.com"]) -%} { "@timestamp": "{{ timestamp.isoformat() }}", "client_ip": "{{ client_ip }}", "method": "{{ method }}", "path": "{{ path }}", "status": {{ status }}, "bytes_sent": {{ bytes_sent }}, "response_time_ms": {{ response_time_ms }}, "user_agent": "{{ user_agent }}", "referrer": "{{ referrer }}" } ``` `@timestamp` is the field name OpenSearch Dashboards expects for the time field when creating an index pattern, so keeping it avoids remapping the field later. The generator config [#the-generator-config] A [cron](/docs/plugins/input/cron) input ticks once a second and emits four timestamps per tick, so the single `access_log` template renders four events every second. The [opensearch](/docs/plugins/output/opensearch) output indexes them: ```yaml title="generators/web-logs/generator.yml" input: - cron: expression: "* * * * * *" count: 4 event: template: mode: all templates: - access_log: template: templates/access_log.jinja output: - stdout: formatter: format: json - opensearch: hosts: - https://opensearch:9200 username: admin password: ${secrets.opensearch_password} index: web-access-logs ``` `hosts` takes a list, so listing more than one node lets the plugin round-robin requests across them for load balancing. `index` is the only thing naming where documents land — `web-access-logs` here, created automatically on the first write if it doesn't already exist. `formatter` is left unset, so the plugin falls back to its default: one compact JSON document per line, exactly what the bulk API expects on each source line (see the [formatter reference](/docs/plugins/formatters)). No OpenSearch cluster within reach yet? The `stdout` output above renders the same documents `opensearch` would index — every output plugin gets an identical copy of each event, only the destination differs. Point `hosts` at a real cluster's address once one exists; nothing else in the config changes. Store the password in the keyring [#store-the-password-in-the-keyring] `password` above is a `${secrets.*}` reference, not a plaintext value. Set it once in the encrypted keyring before running the generator: ```bash eventum-keyring set opensearch_password # Enter password of `opensearch_password`: ******** ``` Then run the generator, pointing at the same cryptfile: ```bash eventum generate --path generator.yml --id web-logs --cryptfile ./cryptfile.cfg ``` If the secret is missing from the keyring, Eventum reports it and refuses to start rather than connecting with an empty password. See [Secrets](/docs/core/config/secrets) for how the keyring and `${secrets.*}` substitution work. The result [#the-result] Running the generator above sends a steady stream of bulk requests to the cluster — Eventum groups events into batches before handing them to output plugins, so a single request typically carries more than one document. Two events from an actual run, before indexing: ```json {"@timestamp": "2026-07-11T14:29:42+00:00", "client_ip": "200.227.2.140", "method": "GET", "path": "/products/wireless-mouse", "status": 200, "bytes_sent": 16482, "response_time_ms": 100, "user_agent": "Mozilla/5.0 (Windows NT 4.0) AppleWebKit/531.1 (KHTML, like Gecko) Chrome/19.0.817.0 Safari/531.1", "referrer": "-"} {"@timestamp": "2026-07-11T14:29:49+00:00", "client_ip": "192.69.61.124", "method": "GET", "path": "/products/wireless-mouse", "status": 404, "bytes_sent": 1095, "response_time_ms": 524, "user_agent": "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_8_8 rv:6.0; id-ID) AppleWebKit/534.19.6 (KHTML, like Gecko) Version/5.1 Safari/534.19.6", "referrer": "https://google.com"} ``` The `opensearch` output wraps each one in a bulk operation line and posts the pair to `/_bulk`: ```text {"index": {"_index": "web-access-logs"}} {"@timestamp": "2026-07-11T14:29:42+00:00", "client_ip": "200.227.2.140", "method": "GET", "path": "/products/wireless-mouse", "status": 200, "bytes_sent": 16482, "response_time_ms": 100, "user_agent": "Mozilla/5.0 (Windows NT 4.0) AppleWebKit/531.1 (KHTML, like Gecko) Chrome/19.0.817.0 Safari/531.1", "referrer": "-"} ``` The operation line carries no `_id`, so OpenSearch assigns one. Fetching that document back returns it wrapped in the usual envelope: ```json { "_index": "web-access-logs", "_id": "V1ZUB5oB8x4Qa2n3k7dR", "_version": 1, "found": true, "_source": { "@timestamp": "2026-07-11T14:29:42+00:00", "client_ip": "200.227.2.140", "method": "GET", "path": "/products/wireless-mouse", "status": 200, "bytes_sent": 16482, "response_time_ms": 100, "user_agent": "Mozilla/5.0 (Windows NT 4.0) AppleWebKit/531.1 (KHTML, like Gecko) Chrome/19.0.817.0 Safari/531.1", "referrer": "-" } } ``` `_source` is exactly the document the template rendered. OpenSearch stores it unchanged and layers `_index`, the assigned `_id`, and a `_version` around it whenever a client fetches that document back directly. FAQ [#faq] Yes. OpenSearch began as a fork of Elasticsearch 7.10.2 and remains compatible on the core document APIs, including bulk indexing — the [plugin reference](/docs/plugins/output/opensearch) itself describes the target as "an OpenSearch (or Elasticsearch-compatible) cluster." Point `hosts` at an Elasticsearch cluster's HTTP endpoint, adjust `username`/`password` to match its auth setup, and the same config indexes into it the same way. `username` and `password` are required on every request and sent as HTTP basic auth — keep the password in the keyring as shown above, never inline in the config. For TLS, `verify` controls whether the cluster's certificate is checked and defaults to `false`; set it to `true` and point `ca_cert` at your CA bundle once the cluster has a certificate from an internal or public CA. `client_cert` and `client_cert_key` add mutual TLS on top and must be set together. All of these, plus `proxy_url` for routing through an HTTP(S) proxy, are documented on the [plugin reference](/docs/plugins/output/opensearch#parameters). Related [#related] * The [delivery track](/docs/tutorials/delivery) for streaming synthetic data to Kafka, ClickHouse, and HTTP alongside OpenSearch * The [SIEM test data lesson](/docs/tutorials/siem-events) for a complete stateful generator indexing Windows Security events into OpenSearch * The [opensearch output reference](/docs/plugins/output/opensearch) for every connection, authentication, and TLS parameter * The [formats field guide](/docs/tutorials/formats) for other log and event shapes headed to the same or a different destination # Send syslog to a collector over TCP or UDP A syslog line that only ever lands in a file has the right shape but never crosses the wire a real syslog source actually uses. Testing an rsyslog or syslog-ng ingestion rule, a SIEM's syslog listener, or a log shipper's parser end to end means putting bytes on that exact connection — a live UDP datagram or a live TCP stream arriving at the collector's own listening port — not text that merely looks correct sitting in a local file. Eventum ships generated syslog straight to a collector over either transport: the [udp](/docs/plugins/output/udp) output sends one datagram per event, the [tcp](/docs/plugins/output/tcp) output holds a connection open and streams events across it, and either one delivers whatever a template renders — the RFC 3164 or RFC 5424 line from the [syslog format lesson](/docs/tutorials/formats/syslog), or a CEF or LEEF line riding the same header. What changes between the two outputs is the wire behavior this lesson covers: how each transport marks where one message ends and the next begins, and what each one does the moment the collector on the other end isn't there. Syslog on the wire: UDP vs TCP [#syslog-on-the-wire-udp-vs-tcp] Syslog's message format — the PRI value, then either an RFC 3164 or RFC 5424 header — is standardized separately from how that message actually reaches a collector. Two RFCs cover the wire: [RFC 5426](https://datatracker.ietf.org/doc/html/rfc5426) for UDP, published in 2009, and [RFC 6587](https://datatracker.ietf.org/doc/html/rfc6587) for TCP, three years later in 2012. This lesson is about those two — see the [syslog format lesson](/docs/tutorials/formats/syslog) for what actually goes inside the message itself. UDP is the original transport, and it stays close to BSD syslog's own assumptions: RFC 5426 requires exactly one syslog message per datagram, with no acknowledgment and no guarantee a datagram arrives at all, arrives once, or arrives in the order it was sent. That's also exactly why UDP is still the default nearly everywhere — a network appliance, a firewall, or a long-lived Unix daemon has no connection to establish, maintain, or recover if the collector restarts; it just keeps sending into the network and trusts that most of it gets there. RFC 5426 registers UDP port 514 for this, the same port BSD syslog has used informally since long before either RFC existed. TCP arrived later, once environments needed something UDP structurally can't offer: delivery actually acknowledged by the transport, ordering that survives a lossy network, and no practical ceiling on a single message's size. RFC 6587 never registers a standard port for it — it notes plainly that 514/TCP actually belongs to the unrelated Shell protocol — but nearly every collector listens on 514 for TCP anyway, convention outrunning the standard. A TCP connection also raises a question a UDP datagram never has to answer: since TCP is just a byte stream with no built-in message boundaries, where does one syslog message end and the next begin? RFC 6587 calls this framing, and the TCP section below covers the two ways it defines to answer that, alongside the TLS-encrypted variant TCP alone makes possible. Send syslog over UDP [#send-syslog-over-udp] Write the template [#write-the-template] One line, in the RFC 3164 shape the format lesson already covers: a PRI value computed the same way — facility 4 (security/authorization messages) times 8 plus severity 4 (Warning), the same `<36>` from that lesson — followed by a timestamp, a hostname, and a free-text tail. [`module.rand`](/docs/plugins/event/template/modules) fills in the parts that vary between events: ```jinja title="generators/syslog-collector/templates/conn-denied.jinja" {%- set src_ip = module.rand.network.ip_v4_public() -%} {%- set dst_port = module.rand.weighted_choice({22: 40, 3389: 35, 23: 15, 445: 10}) -%} <36>{{ timestamp.strftime('%b %e %H:%M:%S') }} fw-edge-01 netfilter: DENY IN=eth0 SRC={{ src_ip }} DST=10.0.4.12 PROTO=TCP DPT={{ dst_port }} ``` A gateway firewall logging a blocked inbound connection — `dst_port` weighted toward the ports opportunistic scanners hit most, SSH and RDP ahead of Telnet and SMB. See the [syslog format lesson](/docs/tutorials/formats/syslog) for how the PRI value and the rest of the RFC 3164 header are actually put together; this one rendered line is all the transport below needs. Configure the udp output [#configure-the-udp-output] A [cron](/docs/plugins/input/cron) input ticks once a second, and the [udp](/docs/plugins/output/udp) output sends each rendered line as its own datagram: ```yaml title="generators/syslog-collector/generator.yml" input: - cron: expression: "* * * * * *" count: 1 event: template: mode: all templates: - conn_denied: template: templates/conn-denied.jinja output: - udp: host: collector.example.com port: 514 separator: "\n" ``` `port: 514` is [UDP syslog's own IANA-registered port](https://datatracker.ietf.org/doc/html/rfc5426) — most collectors listen there by default. `formatter` is left unset, falling back to `udp`'s own default, [`plain`](/docs/plugins/formatters#plain): the line the template rendered goes out unchanged, nothing reshapes it. `separator: "\n"` is also already the default, shown here for clarity — RFC 5426 already guarantees exactly one message per datagram, so this trailing newline isn't what separates one event from the next; the datagram boundary already does that. It's appended only because most collectors still expect a trailing newline on the payload itself, a holdover convention rather than a framing requirement. Send syslog over TCP [#send-syslog-over-tcp] The `udp` block above sends into the network without ever checking whether anything is listening on the other end — and that is the point of the callout below. Swap the output block for [`tcp`](/docs/plugins/output/tcp), leave `input` and `event` untouched, and delivery behaves fundamentally differently the moment the generator starts: ```yaml title="generators/syslog-collector/generator.yml (output block only)" output: - tcp: host: collector.example.com port: 514 separator: "\n" ``` `tcp` opens a real connection to `host`/`port` before anything is sent, bounded by `connect_timeout` (10 seconds by default), so a closed or unreachable port fails immediately and stops the run with a logged error — a syslog misconfiguration surfaces loudly. `udp` only opens a local socket with no connection to fail, so an unreachable target never stops the run: datagrams are sent into the void and only surface as logged socket errors, if at all. Two ways to frame a message [#two-ways-to-frame-a-message] A UDP datagram carries exactly one message by construction — the network preserves that boundary for free. A TCP connection is just a byte stream; nothing about it marks where one syslog message ends and the next begins, so RFC 6587 defines two ways to add that boundary back: * **Non-transparent framing** — the message followed by a trailing delimiter, almost always `LF`. This is what `separator: "\n"` produces, and it's the default for both `tcp` and `udp` above — the shape nearly every collector accepts. * **Octet-counting** — a decimal length prefix ahead of the message instead of a trailing delimiter: `MSG-LEN SP SYSLOG-MSG`, a byte count, a space, then exactly that many bytes. ```text title="Non-transparent framing (LF invisible at the end of the line)" <36>Jul 17 09:14:02 fw-edge-01 netfilter: DENY IN=eth0 SRC=203.0.113.44 DST=10.0.4.12 PROTO=TCP DPT=22 ``` ```text title="The same message, octet-counting framed instead" 102 <36>Jul 17 09:14:02 fw-edge-01 netfilter: DENY IN=eth0 SRC=203.0.113.44 DST=10.0.4.12 PROTO=TCP DPT=22 ``` `separator` on the `tcp` output only ever appends a trailing string, so it produces non-transparent framing and nothing else — keep it as the default unless a collector's own documentation specifically calls for octet-counting. RFC 6587 itself flags why octet-counting exists at all: a trailing delimiter only works if the message never contains that same character, and nothing stops a syslog message from carrying an embedded `LF`. If a collector needs the length-prefixed form instead, render it directly in the template — replacing this lesson's final template line with: ```jinja {%- set line = "<36>" ~ timestamp.strftime('%b %e %H:%M:%S') ~ " fw-edge-01 netfilter: DENY IN=eth0 SRC=" ~ src_ip ~ " DST=10.0.4.12 PROTO=TCP DPT=" ~ dst_port -%} {{ line | length }} {{ line }} ``` paired with `separator: ""` on the `tcp` block, since the length prefix — not a trailing character — now marks the end of each message. (`length` counts characters here; a message with multi-byte UTF-8 content needs the encoded byte count instead, since RFC 6587 counts octets, not characters.) Encrypting the connection with TLS [#encrypting-the-connection-with-tls] Plain TCP syslog carries every message in cleartext. [RFC 5425](https://datatracker.ietf.org/doc/html/rfc5425) layers TLS over the same TCP transport, traditionally on port 6514 rather than 514. `tcp`'s `ssl: true` turns it on: ```yaml title="generators/syslog-collector/generator.yml (output block only)" output: - tcp: host: collector.example.com port: 6514 ssl: true verify: true ca_cert: certs/ca.pem separator: "\n" ``` `verify: true` is already `tcp`'s own default — it just has no effect until `ssl` is on — and it checks the collector's certificate against `ca_cert` rather than accepting anything presented once TLS is enabled; set it to `false` only against a collector whose certificate can't be validated yet, such as a self-signed one during setup. Mutual TLS, where the collector in turn verifies the generator's own identity, adds `client_cert` and `client_cert_key` alongside the fields above; see the [tcp output reference](/docs/plugins/output/tcp#parameters) for every TLS field. `udp` has no TLS equivalent — RFC 5425 exists only for the TCP transport, since there is no persistent connection for a connectionless datagram to secure the way TLS secures a stream. The result [#the-result] Both configs above were validated against a local listener standing in for a real collector — `nc -u -l` for `udp`, plain `nc -l` for `tcp` — since no actual collector was reachable while writing this lesson. Three consecutive lines exactly as the `udp` output sent them, one datagram each: ```text title="Received over UDP" <36>Jul 17 20:06:23 fw-edge-01 netfilter: DENY IN=eth0 SRC=4.76.136.227 DST=10.0.4.12 PROTO=TCP DPT=22 <36>Jul 17 20:06:24 fw-edge-01 netfilter: DENY IN=eth0 SRC=198.65.83.210 DST=10.0.4.12 PROTO=TCP DPT=23 <36>Jul 17 20:06:24 fw-edge-01 netfilter: DENY IN=eth0 SRC=199.213.45.61 DST=10.0.4.12 PROTO=TCP DPT=22 ``` Swapping in the `tcp` output and pointing the same generator at a plain TCP listener produced the identical shape, LF-terminated exactly as `separator: "\n"` specifies — the only difference is the connection underneath: ```text title="Received over TCP" <36>Jul 17 20:06:43 fw-edge-01 netfilter: DENY IN=eth0 SRC=171.15.15.42 DST=10.0.4.12 PROTO=TCP DPT=22 <36>Jul 17 20:06:44 fw-edge-01 netfilter: DENY IN=eth0 SRC=8.236.67.150 DST=10.0.4.12 PROTO=TCP DPT=23 <36>Jul 17 20:06:45 fw-edge-01 netfilter: DENY IN=eth0 SRC=198.44.68.112 DST=10.0.4.12 PROTO=TCP DPT=22 ``` Pointing the `tcp` config at a closed port instead — nothing listening at all — never gets this far. The run stops before a single event renders: ```text title="tcp output against a closed port" [error ] Failed to connect [eventum.core.stages.output_stage] generator_id=syslog-collector host=127.0.0.1 port=20599 reason="[Errno 111] Connect call failed ('127.0.0.1', 20599)" [error ] Failed to open some of the output plugins [eventum.core.generator] generator_id=syslog-collector ``` The reverse test on `udp` confirms the other half: pointing it at a target that stopped listening mid-run, three sends in a row logged an ICMP rejection instead of raising: ```text title="udp output after the collector stops listening" [error ] UDP socket error received [eventum.plugins.output.plugins.udp.plugin] generator_id=syslog-collector plugin_id=1 plugin_name=udp plugin_type=output reason='[Errno 111] Connection refused' ``` repeated for each of the three events sent while nothing was listening — and the generator kept producing and sending the next one regardless. Nothing about the run itself distinguished those three failed sends from a successful one; only the listener's own absence gave it away. FAQ [#faq] Match whatever the real fleet or the collector's own documentation uses, not a general preference — both are standardized transports, and most collectors (rsyslog, syslog-ng, and effectively every commercial SIEM) accept either on their default listener. UDP stays the default for network hardware, embedded devices, and anything that predates or never adopted RFC 6587 — no connection to establish or maintain, and a collector restart or a brief network blip doesn't need a reconnect. Reach for TCP when a message might not fit comfortably in a single datagram, when delivery and ordering need to survive a lossy network, or simply because the receiving collector requires it — the tradeoff, covered above, is that TCP also means a genuinely unreachable collector stops the generator at startup instead of failing invisibly. It depends entirely on which output is configured — test both before relying on either in an unattended run. `tcp` fails loudly: the connection attempt fails before a single event is generated, the run stops, and the error lands right in the log, naming the host, port, and the underlying OS reason. `udp` fails silently: the process keeps running, keeps "succeeding" by its own accounting, and the only trace of a problem is an ICMP error logged if one happens to come back, which isn't guaranteed on every network path. Don't rely on the absence of errors in Eventum's own log to confirm a `udp` stream actually arrived — check the collector's ingest counters, or capture the wire directly with `tcpdump`, the same way this lesson's own verification used `nc`. Non-transparent is the default (`separator: "\n"` above) unless a collector's documentation specifically calls for octet-counting. Nearly every common target (rsyslog's `imtcp`, syslog-ng's `network()` source, most commercial SIEM TCP listeners) accepts LF-delimited messages out of the configuration shown above, no extra settings needed. Octet-counting mainly matters when a message can itself contain an embedded `LF`. RFC 6587 flags exactly this as non-transparent framing's weak point: a receiver has no way to tell an `LF` inside the message from the one marking its end. A length prefix has no character it needs to avoid. Yes — both outputs send whatever the template renders, one line at a time; nothing about `udp` or `tcp` is specific to the RFC 3164/5424 shape used above. Swap in the [CEF](/docs/tutorials/formats/cef) or [LEEF](/docs/tutorials/formats/leef) lesson's own template and the exact same `udp`/`tcp` config ships it to a SIEM's syslog listener instead — many products, ArcSight and QRadar included, expect their format wrapped inside (or standing in for) a syslog header on the wire. For a complete pipeline behind that listener, the [detection testing](/docs/tutorials/detection-testing) and [SIEM test data](/docs/tutorials/siem-events) lessons build the stateful, correlated telemetry a rule or a dashboard actually needs; either one's output can ride this transport instead of the file or OpenSearch index they use directly. Related [#related] * The [delivery track](/docs/tutorials/delivery) for streaming the same kind of stream to OpenSearch, Kafka, ClickHouse, and HTTP alongside a syslog collector * The [syslog format lesson](/docs/tutorials/formats/syslog) for the PRI value and the RFC 3164/5424 header fields this transport carries * The [CEF](/docs/tutorials/formats/cef) and [LEEF](/docs/tutorials/formats/leef) lessons for two more formats that often ride this same transport into a SIEM * The [detection testing](/docs/tutorials/detection-testing) and [SIEM test data](/docs/tutorials/siem-events) lessons for complete, correlated telemetry to deliver this way * The [tcp output reference](/docs/plugins/output/tcp), [udp output reference](/docs/plugins/output/udp), and [formatters reference](/docs/plugins/formatters) for every connection, TLS, framing, and formatting parameter # Apache & Nginx access logs: CLF and Combined A log shipper's parser, a WAF signature tuned against request-line shape, and a billing job that reconciles bytes served all expect access-log lines shaped exactly the way a real web server writes them. A live Apache or nginx instance can produce them, but its output arrives on whatever schedule its actual traffic sets, and standing one up just to capture a redirect, a 404, or a response with an empty body is slow for what it returns. Both vendors document the format precisely, but neither ships more than a single illustrative line — far short of the mix of status codes and body sizes a parser actually has to handle. Eventum renders that stream directly from a template — client address, request line, status, and byte count in the exact field order Apache's Combined Log Format defines, the same order nginx's matching default uses — and writes each line to a file exactly as produced. Any of the three can then be exercised against realistic access-log traffic without standing up a real web server. Common Log Format and Combined Log Format [#common-log-format-and-combined-log-format] [Apache's HTTP Server documentation](https://httpd.apache.org/docs/2.4/logs.html) defines Common Log Format (CLF) as a single `LogFormat` directive — the baseline nearly every access-log format since, nginx's included, has copied or extended: ```text LogFormat "%h %l %u %t \"%r\" %>s %b" common ``` Every directive names one positional field, delimited by spaces; nothing here is optional or reorderable: | Field | Directive | Meaning | | ------------------ | --------- | ------------------------------------------------------------------------------------------------------------------------ | | Remote host | `%h` | The client's IP address. | | Ident | `%l` | The client's RFC 1413 identity, as reported by `identd` on the client machine. Almost never available — see below. | | Authenticated user | `%u` | The username from HTTP authentication, or `-` when the request carries none. | | Time | `%t` | Receipt time, formatted `[day/Mon/yyyy:HH:MM:SS +zzzz]`. | | Request line | `"%r"` | The client's request line — method, path, and protocol version — quoted. | | Status | `%>s` | The final HTTP status code sent to the client. | | Bytes | `%b` | Response body size, excluding headers. `-` when the response carries no body; `%B` prints `0` for the same case instead. | Here is that example with every field labeled: ```text title="Apache's reference CLF example" 127.0.0.1 - frank [10/Oct/2000:13:55:36 -0700] "GET /apache_pb.gif HTTP/1.0" 200 2326 ``` | Segment | Value | Field | | ------- | ------------------------------- | ------------------------- | | 1 | `127.0.0.1` | Remote host (`%h`) | | 2 | `-` | Ident (`%l`) | | 3 | `frank` | Authenticated user (`%u`) | | 4 | `[10/Oct/2000:13:55:36 -0700]` | Time (`%t`) | | 5 | `"GET /apache_pb.gif HTTP/1.0"` | Request line (`%r`) | | 6 | `200` | Status (`%>s`) | | 7 | `2326` | Bytes (`%b`) | Combined Log Format is CLF with two more quoted fields appended — the same seven fields above, unchanged, plus the inbound `Referer` and `User-agent` request headers: ```text LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-agent}i\"" combined ``` ```text title="Apache's reference Combined example" 127.0.0.1 - frank [10/Oct/2000:13:55:36 -0700] "GET /apache_pb.gif HTTP/1.0" 200 2326 "http://www.example.com/start.html" "Mozilla/4.08 [en] (Win98; I ;Nav)" ``` `%{Referer}i` and `%{User-agent}i` both use Apache's general `%{header}i` directive, which reads any inbound request header by name — quoted because either value can legally contain a space. Combined is what a stock Apache install actually logs; CLF alone now mostly shows up in documentation and in parsers old enough to only expect the first seven fields. Apache vs Nginx defaults [#apache-vs-nginx-defaults] nginx's own [`ngx_http_log_module`](https://nginx.org/en/docs/http/ngx_http_log_module.html) ships a predefined format also named `combined`, modeled directly on Apache's: ```text log_format combined '$remote_addr - $remote_user [$time_local] ' '"$request" $status $body_bytes_sent ' '"$http_referer" "$http_user_agent"'; ``` Rendered, the two formats produce the same nine fields in the same order: | # | Apache Combined | nginx `combined` | Holds | | - | ------------------ | -------------------- | --------------------------------------------- | | 1 | `%h` | `$remote_addr` | Client IP address | | 2 | `%l` | *(literal `-`)* | Ident — see caveat below | | 3 | `%u` | `$remote_user` | Authenticated username, `-` if none | | 4 | `%t` | `$time_local` | Receipt time, `[day/Mon/yyyy:HH:MM:SS +zzzz]` | | 5 | `"%r"` | `"$request"` | Request line, quoted | | 6 | `%>s` | `$status` | Final HTTP status code | | 7 | `%b` | `$body_bytes_sent` | Response body size — see caveat below | | 8 | `"%{Referer}i"` | `"$http_referer"` | Referer header, quoted | | 9 | `"%{User-agent}i"` | `"$http_user_agent"` | User-Agent header, quoted | Visually the two lines are indistinguishable. Three details underneath that surface still catch a parser written against only one vendor: * **Ident is a real field in Apache, a hardcoded character in nginx.** Apache's `%l` is a genuine positional field — an RFC 1413 lookup against `identd` on the client machine — that prints `-` because virtually no client runs `identd` any more, not because the field is fake. nginx's format string has no equivalent variable in that position at all: the `-` between `$remote_addr` and `[$time_local]` in the `log_format combined` definition above is a literal character baked into the format string, not a substitution. * **A zero-byte body prints differently.** Apache's `%b` prints `-` when the response carries no body — a `304 Not Modified`, most commonly. `%B`, a separate directive, prints `0` for that same case instead. nginx's `$body_bytes_sent` follows `%B`'s convention, not `%b`'s: [nginx's own variable reference](https://nginx.org/en/docs/http/ngx_http_core_module.html#var_body_bytes_sent) describes it as "compatible with the `%B` parameter of the `mod_log_config` Apache module," and it prints a literal `0`, never a dash. A parser that treats a bare `-` as the only valid empty-body marker silently mis-parses every zero-byte nginx line. * **Both defaults count body bytes only.** `%b` / `%B` and `$body_bytes_sent` all exclude response headers from the count — [nginx's docs](https://nginx.org/en/docs/http/ngx_http_core_module.html#var_body_bytes_sent) state it directly, describing the variable as "not counting the response header." Apache's `%O` (needs `mod_logio`) and nginx's `$bytes_sent` both count the full response, headers included, but neither ships in the respective vendor's default `combined` format — reconciling either byte count against a packet capture or a proxy's own counter needs the headers-inclusive variable explicitly, not the default. Generate an access log with Eventum [#generate-an-access-log-with-eventum] The generator that follows models a small storefront's traffic — page and asset requests that mostly succeed, a share of `304 Not Modified` responses from conditional requests, and a rarer mix of client and server errors — rendering each one as a single Combined-format line. Apache Combined and nginx's `combined` produce the same line for the same request, with one exception: a response with an empty body, where Apache's `%b` writes `-` and nginx's `$body_bytes_sent` writes `0` (the caveat above). Outside that one field, one template covers what either vendor would actually write. Write the Combined-format templates [#write-the-combined-format-templates] Values come from [`module.rand`](/docs/plugins/event/template/modules), the [template](/docs/plugins/event/template) event plugin's built-in randomization module. `templates/access-success.jinja` renders the common case — mostly `200`s, a share of zero-byte `304`s that exercise the caveat above directly, and an occasional `301`: ```jinja title="generators/access-logs/templates/access-success.jinja" {%- set client_ip = module.rand.network.ip_v4_public() -%} {%- set method = module.rand.weighted_choice({"GET": 85, "POST": 12, "HEAD": 3}) -%} {%- set path = module.rand.choice(["/", "/products", "/products/42", "/cart", "/api/search", "/static/app.css", "/static/logo.png"]) -%} {%- set status = module.rand.weighted_choice({200: 78, 304: 15, 301: 7}) -%} {%- set bytes = 0 if status == 304 else module.rand.number.lognormal(8.5, 1.0) | round | int -%} {%- set referer = module.rand.weighted_choice({"-": 40, "https://example.com/": 35, "https://www.google.com/": 25}) -%} {%- set user_agent = module.rand.choice([ "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36", "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.0 Safari/605.1.15", "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36" ]) -%} {{ client_ip }} - - [{{ timestamp.strftime('%d/%b/%Y:%H:%M:%S %z') }}] "{{ method }} {{ path }} HTTP/1.1" {{ status }} {{ "-" if bytes == 0 else bytes }} "{{ referer }}" "{{ user_agent }}" ``` The ident position is a hardcoded literal character — the first `-` right after `client_ip` — not a variable, since this storefront runs no `identd`; that matches the first caveat above exactly. The second `-`, authuser, is a genuine substitutable field in both vendors (`%u` / `$remote_user`), not a literal one — it renders as a dash here only because this storefront gates nothing behind HTTP authentication, so there's never a real username to substitute; that's a simplification of this generator's traffic, not the same real-vs-literal split as ident's. `bytes` is drawn from [`module.rand.number.lognormal`](/docs/plugins/event/template/modules), the right-skewed distribution the [realistic values lesson](/docs/tutorials/realism/values) recommends for response sizes, forced to `0` for every `304` — and the render line prints that `0` as a bare `-`, Apache's own convention for an empty body. `templates/access-error.jinja` renders the rarer case — `404`, `403`, and `500` responses, each with a real, non-zero error-page body: ```jinja title="generators/access-logs/templates/access-error.jinja" {%- set client_ip = module.rand.network.ip_v4_public() -%} {%- set method = module.rand.weighted_choice({"GET": 80, "POST": 20}) -%} {%- set path = module.rand.choice(["/admin", "/wp-login.php", "/api/orders", "/.env", "/checkout"]) -%} {%- set status = module.rand.weighted_choice({404: 70, 403: 20, 500: 10}) -%} {%- set bytes = module.rand.number.integer(180, 620) -%} {%- set user_agent = module.rand.choice([ "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36", "python-requests/2.31.0", "Mozilla/5.0 (compatible; AhrefsBot/7.0; +http://ahrefs.com/robot/)" ]) -%} {{ client_ip }} - - [{{ timestamp.strftime('%d/%b/%Y:%H:%M:%S %z') }}] "{{ method }} {{ path }} HTTP/1.1" {{ status }} {{ bytes }} "-" "{{ user_agent }}" ``` Configure the generator [#configure-the-generator] Per timestamp, [`mode: chance`](/docs/plugins/event/template/modes#chance) chooses between the two templates, weighted so successful requests dominate and errors stay rare. A [cron](/docs/plugins/input/cron) input ticks once a second, and a [file](/docs/plugins/output/file) output writes each rendered line to `output/access.log` through the [plain formatter](/docs/plugins/formatters#plain): ```yaml title="generators/access-logs/generator.yml" input: - cron: expression: "* * * * * *" count: 1 event: template: mode: chance templates: - access_success: template: templates/access-success.jinja chance: 85 - access_error: template: templates/access-error.jinja chance: 15 output: - file: path: output/access.log formatter: format: plain ``` `plain` is `file`'s own default formatter, shown explicitly here to make the choice deliberate: each template already renders one complete, final access-log line, so nothing needs reshaping before it's written. The [`json` formatter](/docs/plugins/formatters#json) would be wrong for this output — it validates every event as a JSON value and rejects anything that isn't, and a Combined-format line never is one. The result [#the-result] A short burst in [sample mode](/docs/core/concepts/generator#sample-mode) produced consistent Combined-format lines end to end. Seven consecutive lines from an actual run: ```text title="output/access.log" 1.110.236.55 - - [17/Jul/2026:16:30:22 +0000] "GET /static/logo.png HTTP/1.1" 200 24528 "https://example.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.0 Safari/605.1.15" 55.89.112.41 - - [17/Jul/2026:16:30:23 +0000] "GET /static/app.css HTTP/1.1" 200 750 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.0 Safari/605.1.15" 126.52.234.151 - - [17/Jul/2026:16:30:24 +0000] "POST /products HTTP/1.1" 200 1871 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.0 Safari/605.1.15" 201.140.53.101 - - [17/Jul/2026:16:30:25 +0000] "POST /cart HTTP/1.1" 200 10066 "https://www.google.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.0 Safari/605.1.15" 221.59.69.248 - - [17/Jul/2026:16:30:26 +0000] "GET /checkout HTTP/1.1" 403 513 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36" 77.56.158.118 - - [17/Jul/2026:16:30:27 +0000] "POST / HTTP/1.1" 200 20691 "https://www.google.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36" 192.138.23.154 - - [17/Jul/2026:16:30:28 +0000] "GET / HTTP/1.1" 304 - "https://example.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36" ``` All nine Combined fields land in the order the tables above define: client address, the hardcoded `- -` for ident and authuser, the bracketed local timestamp, the quoted request line, status, bytes, and the two quoted headers. The fifth line is a `403` with a real body of `513` bytes; the seventh is a `304` — the bytes column prints a bare `-` instead of a number, exactly the empty-body convention Apache's `%b` defines, the same response nginx's `$body_bytes_sent` would instead print as a literal `0`. Substituting nginx's convention for that same request changes exactly one field: ```text title="The same 304 as nginx would write it" 192.138.23.154 - - [17/Jul/2026:16:30:28 +0000] "GET / HTTP/1.1" 304 0 "https://example.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36" ``` Only the bytes column changes, from a bare `-` to a literal `0`; every other field is unchanged. The generator never rendered this line itself — it always follows Apache's `%b` convention — this is the same request rewritten by hand to show nginx's `$body_bytes_sent` convention instead. FAQ [#faq] Combined, in almost every real case. Referer and User-Agent are the two fields most bot-detection and log-analytics rules actually key on, and a stock install of either server writes Combined by default — bare CLF mostly shows up in documentation and in parsers old enough to only expect the first seven fields. Drop the two trailing quoted fields from the templates above to produce CLF instead; the first seven fields don't change shape between the two formats. Not from the bytes column alone. A bare `-` is Apache's convention (`%b`), a bare `0` is nginx's (`$body_bytes_sent`, compatible with Apache's own `%B`), but nothing in the line names which vendor produced it either way. That comes from whatever metadata already tags the file by host or service. A parser meant to handle both should treat `-` and `0` in that column as the same fact: a response with no body. Related [#related] * The [formats field guide](/docs/tutorials/formats) covering every supported log and event shape * The [ECS lesson](/docs/tutorials/formats/ecs) for the schema-mapped shape a Filebeat or Elastic Agent module turns these lines into downstream * The [NDJSON lesson](/docs/tutorials/formats/ndjson) for emitting one JSON object per line instead of a raw text line * The [Stream to your stack](/docs/tutorials/delivery) lessons for shipping generated access logs to a real backend instead of a local file * The [Web clickstream](/docs/tutorials/web-clickstream) and [API load testing](/docs/tutorials/load-testing) scenarios for other generators built around web traffic * The [web-apache](/hub/web-apache) and [web-nginx](/hub/web-nginx) generators in the Eventum Hub for the downstream ECS-mapped form of these same servers * The [file output](/docs/plugins/output/file) and [formatters](/docs/plugins/formatters) references for every field used above # auditd log format: multi-record events Validating a parser built against `/var/log/audit/audit.log`, or a detection rule written to catch a specific syscall or `execve` pattern, needs audit records shaped exactly the way Linux's own audit subsystem writes them — the `type=`/`msg=audit(...)` envelope, the several lines one logical action actually produces, and the hex-encoded fields mixed in among the plain ones. A real audited host produces the same records, but only after an audit rule is installed and the exact activity is run under it — captured once, from a single host, a single process, a single moment, nowhere near the volume or variety a parser or detection rule has to handle. Eventum renders auditd records directly from a template — the same `type=`/`msg=audit(...)` lines, the same audit ID stitching a multi-record event together, the same hex-encoded fields a real kernel writes — so a parser, a SIEM detection rule, or an ingestion pipeline built against raw `audit.log` syntax can be exercised against a realistic stream with no audited host, no triggered syscall, and no captured log file required. The auditd record format [#the-auditd-record-format] Linux's audit subsystem — configured with `auditctl` or a persistent ruleset under `/etc/audit/rules.d/`, enforced inside the kernel, and written to disk by the `auditd` daemon — logs every matched event to `/var/log/audit/audit.log` as plain, line-oriented text: one record per line, never JSON, never a structured envelope of any kind. [Red Hat's own Auditing documentation](https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/9/html/security_hardening/auditing-the-system_security-hardening) and the `auditd`/`ausearch` manual pages describe the same fixed shape every one of those lines follows: ```text type= msg=audit(.:): = = ... ``` | Part | Meaning | | ---------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `type` | The record's kind — `SYSCALL`, `EXECVE`, `CWD`, `PATH`, `PROCTITLE`, and dozens of others, each with its own set of fields. | | `msg=audit(.:)` | The record's audit ID: `epoch.ms` is when the kernel logged it, to the millisecond; `serial` is a counter that only ever increases, one higher for every record the audit subsystem has logged since it started. | | `= ...` | The record's own fields, space-separated — a bare number or hex string, or a double-quoted string wherever a value might contain a space. | Which keys actually follow the colon depends entirely on `type`. A `SYSCALL` record's fields have nothing in common with a `PROCTITLE` record's beyond that shared envelope — and, as the next section covers, the two routinely describe the very same moment. One event, several records [#one-event-several-records] A single logical action — one command executed, one file opened — rarely produces just one line in `audit.log`. The kernel's audit subsystem splits everything it knows about that action across several records, each a different `type`, and ties them back together with one unmistakable signal: every record belonging to the same event carries the exact same `audit(.:)` id. Nothing else marks them as related — no shared index, no parent-child pointer — just that one repeated id, copy-pasted onto every line in the group. Running a binary an audit rule watches for — one matched on `execve`, tagged with a `key` so the rule that caught it is easy to find later — is the clearest case of this. It produces five distinct record types, six lines total, every one of them sharing the same audit ID: ```text title="One auditd event: six lines, one shared audit ID" type=SYSCALL msg=audit(1784316035.000:24531): arch=c000003e syscall=59 success=yes exit=0 a0=300233cfb52a a1=76acd4955815 a2=08adee2a8c6d a3=c items=2 ppid=13835 pid=65496 auid=1001 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts4 ses=33 comm="cat" exe="/usr/bin/cat" subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 key="privileged_commands" type=EXECVE msg=audit(1784316035.000:24531): argc=2 a0="cat" a1="/etc/passwd" type=CWD msg=audit(1784316035.000:24531): cwd="/root" type=PATH msg=audit(1784316035.000:24531): item=0 name="/usr/bin/cat" inode=953387 dev=fd:00 mode=0100755 ouid=0 ogid=0 nametype=NORMAL type=PATH msg=audit(1784316035.000:24531): item=1 name="/lib64/ld-linux-x86-64.so.2" inode=384068 dev=fd:00 mode=0100755 ouid=0 ogid=0 nametype=NORMAL type=PROCTITLE msg=audit(1784316035.000:24531): proctitle=636174002F6574632F706173737764 ``` Every field that matters for reading this block: | Record | Field | Meaning | | --------- | ------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | SYSCALL | `arch` | The CPU architecture the syscall table belongs to — `c000003e` for x86\_64, needed to know which table `syscall` itself indexes into. | | SYSCALL | `syscall` | The syscall number — `59` is `execve` on x86\_64; the same number names a different call entirely on another architecture. | | SYSCALL | `success` / `exit` | Whether the call completed, and its return value — `0` on a successful `execve`, a negative `errno` (`-13` is `EACCES`) when the kernel refused it. | | SYSCALL | `a0`-`a3` | The syscall's first four arguments, exactly as the kernel received them — raw hex, not decoded text (more on this below). | | SYSCALL | `uid` / `auid` | The process's current user ID, and its audit ID — `auid` is assigned once at login and never changes for the life of the session, surviving exactly the privilege change `uid` doesn't. | | SYSCALL | `pid` / `comm` / `exe` | The process ID, its short command name, and the resolved path of the binary that's actually running. | | SYSCALL | `key` | The label an administrator's own audit rule attached with `-k`, for finding every record that rule produces later. | | EXECVE | `argc` / `a0`, `a1`, ... | How many arguments the new program was given, followed by each one — `a0` is the program as invoked, `a1` onward its actual arguments. | | CWD | `cwd` | The process's working directory at the moment of the syscall — resolves any relative path a `PATH` record below might carry. | | PATH | `item` | Which path this is, when a syscall resolves more than one — `0`, `1`, and so on, matching the count `SYSCALL`'s own `items` field gives. | | PATH | `name` / `inode` | The resolved path itself, and the inode on disk it points to. | | PATH | `mode` / `ouid` | The path's Unix permission bits in octal, and the numeric ID of the user who owns it. | | PROCTITLE | `proctitle` | The process's own full command line — always hex, never plain text (below). | A few more fields round out the block without being central to reading it — `ppid`, `gid`, `euid`/`suid`/`fsuid`, `egid`/`sgid`/`fsgid`, `tty`, `ses`, and `subj` (the process's SELinux context) on `SYSCALL`; `dev`, `ogid`, and `nametype` on `PATH`. The Red Hat documentation and the `ausearch` manual page linked above cover the complete field set for every record type. Three fields above are hex instead of plain text, and each for a different reason. `SYSCALL`'s `a0`-`a3` are the syscall's raw arguments exactly as the kernel received them — for `execve`, that means raw memory addresses inside the calling process, meaningless without also reading that process's memory, so the kernel logs them as opaque hex rather than pretending they're readable text. `EXECVE`'s own `a0`, `a1`, and so on are different: they hold the actual decoded argument text (`a0="cat"` above is a plain quoted string), and stay that way unless one particular argument contains something a space-delimited log line can't carry safely — an embedded space, a quote, a non-printable byte — in which case only that argument switches to hex instead. `PROCTITLE`'s own `proctitle` field is hex unconditionally, on every record, with no plain-text form at all: it's the process's entire command line, its arguments joined with a null byte instead of a space, so `636174002F6574632F706173737764` above decodes to `cat`, a null byte, `/etc/passwd` — the same two arguments `EXECVE` already spelled out, just encoded end to end this time. `items=2` on the `SYSCALL` record above is what tells a reader, or a parser, to expect exactly two `PATH` records for this event — `item=0` for the binary that actually ran, `item=1` for the dynamic linker the kernel also had to resolve to load it. Nothing enforces that the six lines above even arrive in this order inside the log file; matching on the shared audit ID, not position, is what a real coalescer relies on instead — [go-libaudit's `aucoalesce` package](https://github.com/elastic/go-libaudit/blob/main/aucoalesce/normalizations.yaml), the library behind Elastic's own Auditbeat, groups records by exactly that id rather than the order they arrived in. Not every record type joins a group like this one. `USER_LOGIN`, `USER_AUTH`, `USER_CMD`, and the credential-lifecycle types PAM generates around a login or a `sudo` invocation (`CRED_ACQ`, `CRED_DISP`) typically stand alone as a single line under their own audit ID, with no `SYSCALL`/`EXECVE`/`PATH` group attached. The [linux-auditd Hub generator](/hub/linux-auditd) covers this wider set of record types too — see the FAQ below for the shape it actually produces. Generate auditd logs with Eventum [#generate-auditd-logs-with-eventum] What follows models a fleet of admins already elevated to root — each via an earlier `su` or `sudo` — reading `/etc/passwd` under a rule that tags every privileged command with `key="privileged_commands"`, the same event modeled above. Write the multi-record template [#write-the-multi-record-template] Unlike the JSON-based format lessons elsewhere in this track, this template builds no data structure at all — it renders six literal text lines directly, computing the shared audit ID once and reusing it on every line. Randomization comes from [`module.rand`](/docs/plugins/event/template/modules), the [template](/docs/plugins/event/template) event plugin's built-in module; the audit ID's `serial` component persists across renders in [`shared`](/docs/plugins/event/template/state) state, the same technique the [linux-auditd Hub generator](/hub/linux-auditd) itself uses for its own per-host `event.sequence` counter. ```jinja title="generators/auditd-raw/templates/privileged-command.jinja" {%- set admins = [ {"name": "jsmith", "auid": 1000}, {"name": "agarcia", "auid": 1001}, {"name": "mwilson", "auid": 1002} ] -%} {%- set admin = module.rand.choice(admins) -%} {%- set pid = module.rand.number.integer(1000, 65535) -%} {%- set ppid = module.rand.number.integer(1000, 65535) -%} {%- set ses = module.rand.number.integer(1, 50) -%} {%- set tty = "pts" ~ module.rand.number.integer(0, 9) -%} {%- set a0 = module.rand.string.hex(12) -%} {%- set a1 = module.rand.string.hex(12) -%} {%- set a2 = module.rand.string.hex(12) -%} {%- set a3 = module.rand.string.hex(1) -%} {%- set inode_bin = module.rand.number.integer(100000, 999999) -%} {%- set inode_ld = module.rand.number.integer(100000, 999999) -%} {%- set allowed = module.rand.chance(0.9) -%} {%- set success = "yes" if allowed else "no" -%} {%- set exit_code = "0" if allowed else "-13" -%} {%- set serial = shared.get('serial', 24531) -%} {%- set epoch = "%.3f" | format(timestamp.timestamp()) -%} {%- set msg_id = "audit(" ~ epoch ~ ":" ~ serial ~ ")" -%} {%- set argv = ["cat", "/etc/passwd"] -%} {%- set nul = module.builtins.chr(0) -%} {%- set proctitle_hex = (argv | join(nul)).encode().hex().upper() -%} type=SYSCALL msg={{ msg_id }}: arch=c000003e syscall=59 success={{ success }} exit={{ exit_code }} a0={{ a0 }} a1={{ a1 }} a2={{ a2 }} a3={{ a3 }} items=2 ppid={{ ppid }} pid={{ pid }} auid={{ admin.auid }} uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty={{ tty }} ses={{ ses }} comm="cat" exe="/usr/bin/cat" subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 key="privileged_commands" type=EXECVE msg={{ msg_id }}: argc=2 a0="cat" a1="/etc/passwd" type=CWD msg={{ msg_id }}: cwd="/root" type=PATH msg={{ msg_id }}: item=0 name="/usr/bin/cat" inode={{ inode_bin }} dev=fd:00 mode=0100755 ouid=0 ogid=0 nametype=NORMAL type=PATH msg={{ msg_id }}: item=1 name="/lib64/ld-linux-x86-64.so.2" inode={{ inode_ld }} dev=fd:00 mode=0100755 ouid=0 ogid=0 nametype=NORMAL type=PROCTITLE msg={{ msg_id }}: proctitle={{ proctitle_hex }} {%- do shared.set('serial', serial + 1) -%} ``` `epoch` is formatted to exactly three decimal places with the `format` filter, matching the millisecond precision real `audit.log` timestamps carry. `serial` starts wherever `shared` last left it (`24531` the first time this template renders in a fresh generator) and is written back one higher at the very end, so every subsequent event's audit ID keeps climbing instead of resetting. `nul` reaches into Python's own `builtins` module — importable like any other package via `module` — for a literal null byte; `argv | join(nul)` then joins `["cat", "/etc/passwd"]` into one string with that null byte sitting between the two arguments, and `.encode().hex().upper()` turns the whole thing into the same uppercase hex a real kernel writes for `PROCTITLE`. Configure the generator [#configure-the-generator] [`mode: any`](/docs/plugins/event/template/modes#any) is the simplest fit for a single template — every timestamp renders it, with no picking logic needed since there's nothing to pick between. A [cron](/docs/plugins/input/cron) input ticks once a second and emits five timestamps per tick, so each second produces five audit events — six lines each, thirty in total, every event under its own audit ID. A [file](/docs/plugins/output/file) output writes them through the [plain](/docs/plugins/formatters#plain) formatter: ```yaml title="generators/auditd-raw/generator.yml" input: - cron: expression: "* * * * * *" count: 5 event: template: mode: any templates: - privileged_command: template: templates/privileged-command.jinja output: - file: path: output/audit.log formatter: format: plain ``` `plain` is what an already-correctly-shaped text format needs — it passes each event through untouched. Switching to `json` or `json-batch` here would not just be the wrong style choice: since a six-line `type=SYSCALL ...` block is not valid JSON, either formatter would reject every single event as a format error and write nothing to `output/audit.log` at all. The result [#the-result] Running the generator above in [sample mode](/docs/core/concepts/generator#sample-mode) for a moment produced two consecutive events, the audit ID advancing from `:24531` to `:24532` between them and nothing else about the six-line shape changing: ```text title="output/audit.log — two consecutive events, one allowed and one denied" type=SYSCALL msg=audit(1784316035.000:24531): arch=c000003e syscall=59 success=yes exit=0 a0=300233cfb52a a1=76acd4955815 a2=08adee2a8c6d a3=c items=2 ppid=13835 pid=65496 auid=1001 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts4 ses=33 comm="cat" exe="/usr/bin/cat" subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 key="privileged_commands" type=EXECVE msg=audit(1784316035.000:24531): argc=2 a0="cat" a1="/etc/passwd" type=CWD msg=audit(1784316035.000:24531): cwd="/root" type=PATH msg=audit(1784316035.000:24531): item=0 name="/usr/bin/cat" inode=953387 dev=fd:00 mode=0100755 ouid=0 ogid=0 nametype=NORMAL type=PATH msg=audit(1784316035.000:24531): item=1 name="/lib64/ld-linux-x86-64.so.2" inode=384068 dev=fd:00 mode=0100755 ouid=0 ogid=0 nametype=NORMAL type=PROCTITLE msg=audit(1784316035.000:24531): proctitle=636174002F6574632F706173737764 type=SYSCALL msg=audit(1784316035.000:24532): arch=c000003e syscall=59 success=no exit=-13 a0=60d1787f47c3 a1=852e5a26fb25 a2=bf98aa9a9d60 a3=c items=2 ppid=32376 pid=17403 auid=1001 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts8 ses=15 comm="cat" exe="/usr/bin/cat" subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 key="privileged_commands" type=EXECVE msg=audit(1784316035.000:24532): argc=2 a0="cat" a1="/etc/passwd" type=CWD msg=audit(1784316035.000:24532): cwd="/root" type=PATH msg=audit(1784316035.000:24532): item=0 name="/usr/bin/cat" inode=676886 dev=fd:00 mode=0100755 ouid=0 ogid=0 nametype=NORMAL type=PATH msg=audit(1784316035.000:24532): item=1 name="/lib64/ld-linux-x86-64.so.2" inode=231381 dev=fd:00 mode=0100755 ouid=0 ogid=0 nametype=NORMAL type=PROCTITLE msg=audit(1784316035.000:24532): proctitle=636174002F6574632F706173737764 ``` Every line in the first block carries `audit(1784316035.000:24531)`; every line in the second carries `:24532` instead — the exact grouping mechanism the previous section describes, not a special case invented for this example. Both events happen to trace to the same admin here (`auid=1001`), already root (`uid=0`), running the same command under the same `key="privileged_commands"` rule; the only fields that differ are `success` and `exit` — `yes`/`0` on the first, `no`/`-13` (`EACCES`) on the second, as if a policy such as SELinux enforcement let the path resolution complete both times but blocked the operation itself on the second attempt. A real system enforcing that policy would typically log a separate `type=AVC` record alongside a denial like this one, naming which policy rule refused it — a record type outside what this lesson covers. FAQ [#faq] Because they answer different questions, and only one of them changes when a session escalates privilege. `uid` (and `euid` alongside it) is the process's current user — `0` for the root session in the example above. `auid`, the login UID, is assigned once by `pam_loginuid` at the start of a session and stays fixed for its entire lifetime, immune to any later `su` or `sudo` — so a `SYSCALL` record with `uid=0` and `auid=1001` together mean exactly one thing: whoever holds `auid` 1001 escalated to root and is now running commands as it. Drop `auid` from a query or a detection rule built on these records, and every privilege escalation in the log looks identical to a process root started on its own. `ausearch` is the standard tool for it, and it understands the multi-record grouping this page describes natively — `ausearch -k privileged_commands` returns every record sharing an audit ID with any record tagged that key, already grouped into events rather than loose lines. Add `-i` ("interpret") and it also decodes the hex fields this page covers — numeric `uid`/`auid` values resolved to usernames, and `PROCTITLE`'s hex resolved back to the plain command line — instead of leaving them for a human to decode by hand. Related [#related] * The [formats field guide](/docs/tutorials/formats) covering every supported log and event shape * The [ECS lesson](/docs/tutorials/formats/ecs) for the normalized shape a real audit trail gets mapped into once Auditbeat coalesces and parses it * The [detection-testing lesson](/docs/tutorials/detection-testing) for testing a Sigma rule or an ATT\&CK-mapped detection against generated syscall and execve telemetry * The [Stream to your stack](/docs/tutorials/delivery) lessons for delivering a generated log stream to a real collector instead of a local file * The [formatters](/docs/plugins/formatters) and [template event plugin](/docs/plugins/event/template) references for every field and format used above * The [linux-auditd](/hub/linux-auditd) generator in the Eventum Hub for the downstream ECS-mapped form of this same source # CEF format: header and extension Validating a SIEM's CEF parser, or a detection rule written against CEF fields, requires CEF events shaped to exercise the exact case under test. A live firewall, intrusion detection system, or ArcSight SmartConnector can produce them, but standing one up is slow and its output arrives on its own schedule, not the tester's. Public examples are equally scarce: the authoritative specification is a vendor PDF, and most search results are forum threads with a few copied lines and no explanation of the structure behind them. Eventum generates CEF lines directly from a template — a correctly delimited header followed by a correctly escaped extension — and delivers them to whatever the collector listens on. The parser or the detection rule can be exercised against them without a real device in place. What CEF looks like [#what-cef-looks-like] CEF (Common Event Format) is a text-based log format [defined by ArcSight](https://www.microfocus.com/documentation/arcsight/arcsight-smartconnectors-8.3/pdfdoc/cef-implementation-standard/cef-implementation-standard.pdf) — now part of OpenText — so that security products from any vendor could feed a single, predictable event shape into its SIEM. Firewalls, intrusion detection systems, endpoint agents, and other security appliances now emit it natively or through a mapping layer, and most SIEMs and log collectors other than IBM QRadar, which expects LEEF instead, parse it directly. A CEF message is a header followed by an extension, both riding inside a single line of text: ```text CEF:Version|Device Vendor|Device Product|Device Version|Device Event Class ID|Name|Severity|Extension ``` Every field up to `Severity` is mandatory and positional, delimited by unescaped pipes (`|`). `Extension` is optional and carries the event's actual data as space-separated `key=value` pairs. | Field | Description | | --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Version | Format version — `0` or `1`. | | Device Vendor | Identifies the vendor of the device that produced the event. | | Device Product | Identifies the product. | | Device Version | The product's version string. | | Device Event Class ID | A unique identifier per event or signature type — commonly called the Signature ID. | | Name | A short, human-readable description. It should not repeat information already carried by another field: "Connection blocked", not "Connection from 198.51.100.7 blocked on port 22". | | Severity | The event's importance: 0–10, or one of `Unknown`, `Low`, `Medium`, `High`, `Very-High`. | Here is ArcSight's reference line, broken down field by field: ```text title="ArcSight's reference example" CEF:0|Security|threatmanager|1.0|100|worm successfully stopped|10|src=10.0.0.1 dst=2.1.2.2 spt=1232 ``` | Segment | Value | Field | | ------- | ----------------------------------- | --------------------- | | 1 | `0` | Version | | 2 | `Security` | Device Vendor | | 3 | `threatmanager` | Device Product | | 4 | `1.0` | Device Version | | 5 | `100` | Device Event Class ID | | 6 | `worm successfully stopped` | Name | | 7 | `10` | Severity | | — | `src=10.0.0.1 dst=2.1.2.2 spt=1232` | Extension | Severity accepts either form, and a numeric value maps to a string band: | Numeric range | String value | | ------------- | ------------ | | 0–3 | `Low` | | 4–6 | `Medium` | | 7–8 | `High` | | 9–10 | `Very-High` | `Unknown` is also valid as a string, with no corresponding integer. The Extension carries the event's data as `key=value` pairs, space-separated, in any order. Keys come from ArcSight's own Extension Dictionary — more than a hundred predefined names — of which a handful account for most events: | Key | Full name | Meaning | | ----------------------------------------- | ------------------------------------------- | ---------------------------------------------------------------------------------------------------------- | | `src` / `dst` | sourceAddress / destinationAddress | Source / destination IPv4 or IPv6 address. | | `spt` / `dpt` | sourcePort / destinationPort | Source / destination port, 0–65535. | | `shost` / `dhost` | sourceHostName / destinationHostName | Source / destination hostname. | | `suser` / `duser` | sourceUserName / destinationUserName | Source / destination user name. | | `act` | deviceAction | Action taken by the device. | | `proto` | transportProtocol | Layer-4 protocol, e.g. `TCP` or `UDP`. | | `outcome` | eventOutcome | Usually `success` or `failure`. | | `cat` | deviceEventCategory | Vendor-assigned event category. | | `msg` | message | Free-text detail. Multi-line values use `\n` as the line separator. | | `rt` | deviceReceiptTime | When the event was received — `MMM dd yyyy HH:mm:ss` or epoch milliseconds. | | `cnt` | baseEventCount | Number of times this same event was observed; omitted when `1`. | | `cs1` through `cs6` / `cn1` through `cn3` | deviceCustomString\* / deviceCustomNumber\* | Custom string / numeric fields for data with no dedicated key, each paired with a matching `*Label` field. | The header and the extension enforce different escaping rules, and mixing them up produces a line the receiving parser splits incorrectly: * **Pipe (`|`)** must be escaped as `\|` when it appears inside a header value; the seven delimiter pipes themselves are never escaped. A pipe inside an extension value needs no escaping at all. * **Backslash (`\`)** must be escaped as `\\` wherever it appears — in the header and in the extension. * **Equals sign (`=`)** must be escaped as `\=` inside an extension value; a literal `=` inside a header field needs no such treatment. * **Newline** inside an extension value is encoded as a literal `\n` or `\r`; only extension values may span multiple lines this way, never header fields. A Windows file path exercises the backslash rule in both halves of the message at once, and a literal `=` inside an argument exercises the extension-only rule: ```text title="Escaping backslash and equals" CEF:0|Acme|NetGuard|3.1|1007|Blocked path C:\\Temp\\payload.exe|6|act=blocked filePath=C:\\Temp\\payload.exe msg=argument was key\=value ``` Every backslash in `C:\Temp\payload.exe` becomes `\\`, in both the Name field and the `filePath` extension value. The `=` inside `key=value` is escaped as `\=` because it sits inside an extension value rather than acting as a delimiter. Generate CEF with Eventum [#generate-cef-with-eventum] The setup below models a firewall — `Acme NetGuard` — producing two CEF event types: routine policy-blocked connections and rarer, higher-severity intrusion alerts. The templates [#the-templates] Each field is drawn from [`module.rand`](/docs/plugins/event/template/modules), the [template](/docs/plugins/event/template) event plugin's built-in randomization module, so every run produces different but structurally valid lines. `templates/blocked-connection.jinja` renders the routine case, Severity `4` (Medium): ```jinja title="generators/cef-firewall/templates/blocked-connection.jinja" {%- set src_ip = module.rand.network.ip_v4_public() -%} {%- set dst_ip = module.rand.network.ip_v4_private_c() -%} {%- set spt = module.rand.number.integer(1024, 65535) -%} {%- set dpt = module.rand.weighted_choice([22, 80, 443, 3389, 445], [10, 30, 35, 15, 10]) -%} {%- set rt = timestamp.strftime('%b %d %Y %H:%M:%S') -%} CEF:0|Acme|NetGuard|3.1|1000|Connection blocked by policy|4|rt={{ rt }} src={{ src_ip }} dst={{ dst_ip }} spt={{ spt }} dpt={{ dpt }} proto=TCP act=blocked outcome=failure ``` `templates/intrusion-detected.jinja` renders the rarer case, Severity `8` (High): ```jinja title="generators/cef-firewall/templates/intrusion-detected.jinja" {%- set src_ip = module.rand.network.ip_v4_public() -%} {%- set dst_ip = module.rand.network.ip_v4_private_c() -%} {%- set dpt = module.rand.choice([22, 3389, 445, 1433]) -%} {%- set rt = timestamp.strftime('%b %d %Y %H:%M:%S') -%} {%- set attempts = module.rand.number.integer(15, 120) -%} CEF:0|Acme|NetGuard|3.1|2001|Possible port scan detected|8|rt={{ rt }} src={{ src_ip }} dst={{ dst_ip }} dpt={{ dpt }} proto=TCP act=alert cat=Intrusion/PortScan cnt={{ attempts }} ``` Neither template escapes anything, because the IP addresses, ports, and fixed strings above contain no pipe, backslash, or equals sign. A template that interpolates free-form text — a file path, a user-supplied query string — into `msg` or a custom field must apply the rules from the previous section itself before rendering. The generator config [#the-generator-config] [`mode: chance`](/docs/plugins/event/template/modes#chance) picks between the two templates per timestamp, weighted so blocked connections dominate and intrusions stay rare. A [cron](/docs/plugins/input/cron) input ticks once a second, and a [tcp](/docs/plugins/output/tcp) output ships each rendered line to the collector's syslog port: ```yaml title="generators/cef-firewall/generator.yml" input: - cron: expression: "* * * * * *" count: 1 event: template: mode: chance templates: - blocked_connection: template: templates/blocked-connection.jinja chance: 85 - intrusion_detected: template: templates/intrusion-detected.jinja chance: 15 output: - tcp: host: siem.example.com port: 514 separator: "\n" ``` For UDP delivery, replace `tcp` with [`udp`](/docs/plugins/output/udp) and keep the same `host` and `port`. To inspect lines locally before pointing the generator at a real collector, replace the output with [`file`](/docs/plugins/output/file) or [`stdout: {}`](/docs/plugins/output/stdout). The result [#the-result] Running the generator above produces one CEF line per timestamp, alternating between the two event types: ```text CEF:0|Acme|NetGuard|3.1|2001|Possible port scan detected|8|rt=Jul 11 2026 12:01:59 src=192.61.161.51 dst=192.168.14.85 dpt=3389 proto=TCP act=alert cat=Intrusion/PortScan cnt=86 CEF:0|Acme|NetGuard|3.1|1000|Connection blocked by policy|4|rt=Jul 11 2026 12:02:00 src=5.115.188.144 dst=192.168.33.235 spt=61575 dpt=443 proto=TCP act=blocked outcome=failure ``` Both lines are valid CEF: seven pipe-delimited header fields followed by space-separated extension pairs, `rt` in the `MMM dd yyyy HH:mm:ss` format the specification defines, `src` drawn from outside any private range, and `dst` kept inside the internal network the firewall protects. FAQ [#faq] Both are pipe-delimited, vendor-neutral security event formats with a header followed by key-value data, but the header shapes differ. CEF's header carries seven fields, including `Name` and `Severity`; LEEF's carries five — `Version`, `Vendor`, `Product`, `Version`, `EventID` — with no `Name` or `Severity` field at all. The extension delimiter differs too: CEF pairs are space-separated, LEEF pairs are tab-separated by default. CEF is what ArcSight and most SIEMs other than IBM QRadar expect; QRadar uses LEEF. See the [LEEF lesson](/docs/tutorials/formats/leef) for the full structure and how to generate it with Eventum. CEF is a message format, not a transport — it is usually carried inside a syslog message, though many collectors also accept the bare CEF line over TCP or UDP. Prepend a syslog header (an RFC 3164 `timestamp hostname` header, or an RFC 5424 header) to the `CEF:Version|...` line inside the same template, then send the result over [tcp](/docs/plugins/output/tcp) or [udp](/docs/plugins/output/udp) to the collector's syslog port — `514` for plain text, `6514` if the collector requires TLS. Some collectors accept the bare `CEF:Version|...` line directly, without any syslog header, which is the form the generator above produces. See the [syslog lesson](/docs/tutorials/formats/syslog) for both header formats. Either an integer from 0 to 10, or one of five strings: `Unknown`, `Low`, `Medium`, `High`, `Very-High`. When a number maps to a string, 0–3 is `Low`, 4–6 is `Medium`, 7–8 is `High`, and 9–10 is `Very-High`; nothing outside that range or set is valid. Related [#related] * The [formats field guide](/docs/tutorials/formats) covering every supported log and event shape * The [LEEF lesson](/docs/tutorials/formats/leef) for IBM QRadar's equivalent format * The [syslog lesson](/docs/tutorials/formats/syslog) for wrapping any message in a standards-compliant syslog header * The [detection testing lesson](/docs/tutorials/detection-testing) for generating attack-shaped telemetry to test Sigma rules # AWS CloudTrail log format: the Records envelope Validating a CloudTrail parser, or a detection rule written against CloudTrail's own field names, needs event records shaped to the exact case those field names use — `eventName`, not `event_name` or `EventName`; `sourceIPAddress`, not `source_ip_address`. A real AWS account produces those records, but only after the specific API call is triggered and CloudTrail delivers it minutes later, on AWS's schedule rather than a test's — and a single account exercises one code path at a time, not the mix of services and outcomes a parser has to handle. Eventum renders those records straight from a template — the same flat camelCase fields, the same `Records` envelope CloudTrail writes to S3 — and delivers them on whatever schedule a test sets. A parser, a detection rule, or an ingestion pipeline written against the real field names sees the shape it expects, with no AWS account to configure and no delivery delay to wait out. The CloudTrail record format [#the-cloudtrail-record-format] CloudTrail delivers events to an S3 bucket as gzip-compressed JSON files, one object per delivery interval. Unzipped, every object holds exactly one top-level key, `Records`, whose value is an array of event objects — never a bare array on its own, and never one event per file. [AWS's own CloudTrail log file examples](https://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-log-file-examples.html) show the shape directly; the record below is one of theirs, trimmed to the fields this lesson covers: ```json title="A CloudTrail log file delivered to S3 (adapted from AWS's own reference example)" { "Records": [ { "eventVersion": "1.08", "eventTime": "2023-07-19T21:17:28Z", "eventSource": "ec2.amazonaws.com", "eventName": "StartInstances", "awsRegion": "us-east-1", "sourceIPAddress": "192.0.2.0", "userAgent": "aws-cli/2.13.5 Python/3.11.4 Linux/4.14.255-314-253.539.amzn2.x86_64", "userIdentity": { "type": "IAMUser", "principalId": "EXAMPLE6E4XEGITWATV6R", "arn": "arn:aws:iam::123456789012:user/Mateo", "accountId": "123456789012", "accessKeyId": "AKIAIOSFODNN7EXAMPLE", "userName": "Mateo" }, "requestParameters": { "instancesSet": { "items": [{ "instanceId": "i-EXAMPLE56126103cb" }] } }, "responseElements": { "instancesSet": { "items": [{ "instanceId": "i-EXAMPLE56126103cb", "currentState": { "code": 0, "name": "pending" }, "previousState": { "code": 80, "name": "stopped" } }] } }, "eventID": "e755e09c-42f9-4c5c-9064-EXAMPLE228c7", "eventType": "AwsApiCall", "readOnly": false, "managementEvent": true, "recipientAccountId": "123456789012" } ] } ``` Every field inside that one event is flat and camelCase — no nested schema namespaces the way [ECS](/docs/tutorials/formats/ecs) or [OCSF](/docs/tutorials/formats/ocsf) organize their own fields. A handful of fields, `userIdentity` chief among them, nest a small object of their own. A production trail holds many such objects inside one `Records` array per delivered file, not one file per event. Key fields of a CloudTrail event [#key-fields-of-a-cloudtrail-event] [CloudTrail's own record contents reference](https://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-event-reference-record-contents.html) documents every field a record can carry; the ones below are the ones a parser or detection rule keys on most: | Field | Description | | ---------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `eventVersion` | The version of the CloudTrail record format itself, e.g. `1.09` — unrelated to the version of the AWS API the call used. | | `eventTime` | The UTC time of the request, formatted `YYYY-MM-DDTHH:MM:SSZ`. | | `eventSource` | The AWS service the request went to, e.g. `sts.amazonaws.com`, `ec2.amazonaws.com`. | | `eventName` | The specific API action that was called, e.g. `AssumeRole`, `DescribeInstances`. | | `awsRegion` | The AWS Region the request targeted. | | `sourceIPAddress` | The IP address the request came from. | | `userAgent` | The client that made the request — the AWS CLI, an SDK, or the Management Console. | | `userIdentity` | Who made the call — see the table below. | | `requestParameters` | The parameters sent with the request; `null` for calls that take none. | | `responseElements` | The result of a create, update, or delete call; `null` for read-only calls and for calls that return nothing. | | `eventID` | A GUID CloudTrail assigns to uniquely identify the record. | | `eventType` | What kind of activity produced the record — see below. | | `readOnly` | `true` for a call that only reads data, `false` for one that changes something. | | `managementEvent` | `true` for control-plane activity — creating, modifying, or deleting resources and configuration — as opposed to the data-plane activity a resource generates while running. | | `recipientAccountId` | The account that received the event — usually the caller's own account, but can differ for cross-account resource access. | | `errorCode` / `errorMessage` | Present only when the call failed — the AWS error code and a description of what went wrong. | Real CloudTrail records carry more fields than the set above depending on the event type — `requestID`, `eventCategory`, and `tlsDetails` among them. [CloudTrail's record contents reference](https://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-event-reference-record-contents.html) documents the rest. `eventType` identifies what kind of activity produced the record. Most events use one of two values: `AwsApiCall` for an ordinary API call, or `AwsConsoleSignIn` for a console sign-in — CloudTrail defines a few narrower values too, for service-generated and VPC-endpoint activity, that this lesson does not generate. `userIdentity` is itself an object, [documented separately by AWS](https://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-event-reference-user-identity.html): | Field | Description | | ---------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `type` | The kind of identity that made the call — `IAMUser`, `AssumedRole`, `Root`, `FederatedUser`, `AWSService`, and several other values for narrower cases. | | `principalId` | A unique identifier for the identity — for temporary credentials, this includes the session name. | | `arn` | The full ARN of the identity that made the call. | | `accountId` | The account that owns the identity. | | `accessKeyId` | The access key that signed the request — `AKIA`-prefixed for a long-term IAM user key, `ASIA`-prefixed for temporary credentials issued by STS. | | `userName` | The identity's friendly name, when `type` provides one. | | `sessionContext` | Present only when the request used temporary credentials — names how they were obtained (`sessionIssuer`) and whether the session used MFA (`attributes.mfaAuthenticated`). | `type` alone has [more than half a dozen possible values](https://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-event-reference-user-identity.html) beyond the ones named above; `IAMUser` and `AssumedRole` cover the large majority of ordinary activity in most accounts. Generate CloudTrail records with Eventum [#generate-cloudtrail-records-with-eventum] This section builds a small multi-account AWS organization's traffic — deployments that assume a role far more often than anyone signs into the console by hand: Write the CloudTrail event templates [#write-the-cloudtrail-event-templates] [`module.rand`](/docs/plugins/event/template/modules), the [template](/docs/plugins/event/template) event plugin's built-in randomization module, supplies every field. Each template builds the record as a single Jinja mapping and serializes it in one step with the built-in `tojson` filter, the same technique the [ECS lesson](/docs/tutorials/formats/ecs) uses for its own nested event shape — CloudTrail's `userIdentity`, `requestParameters`, and `responseElements` all nest the same way. `templates/assume-role.jinja` renders the common case — an IAM user in one of three accounts assuming a role to run a deployment: ```jinja title="generators/cloudtrail-records/templates/assume-role.jinja" {%- set account_id = module.rand.choice(["111122223333", "444455556666", "777788889999"]) -%} {%- set user_name = module.rand.choice(["morgan.lee", "priya.desai", "diego.alvarez", "hannah.becker"]) -%} {%- set role_name = module.rand.choice(["DevOpsDeployRole", "ReadOnlyAuditRole", "CI-DeployRole"]) -%} {%- set principal_id = "AIDA" ~ module.rand.string.letters_uppercase(8) ~ module.rand.string.digits(8) -%} {%- set access_key_id = "AKIA" ~ module.rand.string.letters_uppercase(8) ~ module.rand.string.digits(8) -%} {%- set temp_access_key_id = "ASIA" ~ module.rand.string.letters_uppercase(8) ~ module.rand.string.digits(8) -%} {%- set assumed_role_id = "AROA" ~ module.rand.string.letters_uppercase(8) ~ module.rand.string.digits(8) -%} {%- set session_name = "deploy-" ~ module.rand.string.hex(8) -%} {%- set src_ip = module.rand.network.ip_v4_public() -%} {%- set record = { "eventVersion": "1.09", "eventTime": timestamp.strftime('%Y-%m-%dT%H:%M:%SZ'), "eventSource": "sts.amazonaws.com", "eventName": "AssumeRole", "awsRegion": "us-east-1", "sourceIPAddress": src_ip, "userAgent": "aws-cli/2.15.22 Python/3.11.8 Linux/6.5.0", "userIdentity": { "type": "IAMUser", "principalId": principal_id, "arn": "arn:aws:iam::" ~ account_id ~ ":user/" ~ user_name, "accountId": account_id, "accessKeyId": access_key_id, "userName": user_name }, "requestParameters": { "roleArn": "arn:aws:iam::" ~ account_id ~ ":role/" ~ role_name, "roleSessionName": session_name, "durationSeconds": 3600 }, "responseElements": { "credentials": { "accessKeyId": temp_access_key_id, "sessionToken": module.rand.string.hex(64) }, "assumedRoleUser": { "assumedRoleId": assumed_role_id ~ ":" ~ session_name, "arn": "arn:aws:sts::" ~ account_id ~ ":assumed-role/" ~ role_name ~ "/" ~ session_name } }, "eventID": module.rand.crypto.uuid4(), "eventType": "AwsApiCall", "readOnly": false, "managementEvent": true, "recipientAccountId": account_id } -%} {{ record | tojson }} ``` `accountId`, `userName`, and `roleName` are drawn from small fixed pools, the same technique the [ECS lesson](/docs/tutorials/formats/ecs) uses for its own hostname pool — enough variety to look like a real multi-account organization without needing a real one. `accessKeyId` follows AWS's own prefix convention: `AKIA` for the caller's long-term IAM user key, `ASIA` for the temporary credentials returned in `responseElements.credentials`. `sessionContext` is left out of `userIdentity` here, since this caller authenticates with that long-term key directly rather than a session already in progress — the Key fields section above covers what it holds when one is present. `templates/console-login.jinja` renders the rarer case — a console sign-in, close to the roughly 1-in-6 share the [cloud-aws-cloudtrail Hub generator](/hub/cloud-aws-cloudtrail) assigns this same action relative to `AssumeRole`: ```jinja title="generators/cloudtrail-records/templates/console-login.jinja" {%- set account_id = module.rand.choice(["111122223333", "444455556666", "777788889999"]) -%} {%- set user_name = module.rand.choice(["morgan.lee", "priya.desai", "diego.alvarez", "hannah.becker"]) -%} {%- set principal_id = "AIDA" ~ module.rand.string.letters_uppercase(8) ~ module.rand.string.digits(8) -%} {%- set src_ip = module.rand.network.ip_v4_public() -%} {%- set mfa_used = module.rand.chance(0.4) -%} {%- set success = module.rand.chance(0.92) -%} {%- set user_agent = module.rand.choice([ "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36", "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.0 Safari/605.1.15" ]) -%} {%- set record = { "eventVersion": "1.09", "eventTime": timestamp.strftime('%Y-%m-%dT%H:%M:%SZ'), "eventSource": "signin.amazonaws.com", "eventName": "ConsoleLogin", "awsRegion": "us-east-1", "sourceIPAddress": src_ip, "userAgent": user_agent, "userIdentity": { "type": "IAMUser", "principalId": principal_id, "arn": "arn:aws:iam::" ~ account_id ~ ":user/" ~ user_name, "accountId": account_id, "userName": user_name }, "requestParameters": none, "responseElements": { "ConsoleLogin": "Success" if success else "Failure" }, "additionalEventData": { "MFAUsed": "Yes" if mfa_used else "No" }, "eventID": module.rand.crypto.uuid4(), "eventType": "AwsConsoleSignIn", "readOnly": false, "managementEvent": true, "recipientAccountId": account_id } -%} {%- if not success -%} {%- do record.update({"errorMessage": "Failed authentication"}) -%} {%- endif -%} {{ record | tojson }} ``` A sign-in succeeds 92% of the time here; `module.rand.chance` decides the outcome and the record's own fields follow it — `responseElements.ConsoleLogin` reads `Success` or `Failure`, and `errorMessage` is added, via `record.update`, only when it does not. `requestParameters` is `null` on every sign-in — a `ConsoleLogin` call carries no request body of its own. Configure the generator [#configure-the-generator] Per timestamp, [`mode: chance`](/docs/plugins/event/template/modes#chance) selects between the two templates, weighted so `AssumeRole` dominates and console sign-ins stay a minority — the same 85/15 split the [ECS](/docs/tutorials/formats/ecs) and [NDJSON](/docs/tutorials/formats/ndjson) lessons use for their own common/rare pairs. A [cron](/docs/plugins/input/cron) input ticks once a second, and a [file](/docs/plugins/output/file) output writes each record through the [json](/docs/plugins/formatters#json) formatter — one complete CloudTrail record per line, the schema a parser or validator checks a record against directly: ```yaml title="generators/cloudtrail-records/generator.yml" input: - cron: expression: "* * * * * *" count: 1 event: template: mode: chance templates: - assume_role: template: templates/assume-role.jinja chance: 85 - console_login: template: templates/console-login.jinja chance: 15 output: - file: path: output/cloudtrail.json formatter: format: json ``` `json` validates each record and compacts it to one line regardless of how the template itself is indented — useful here, since the Jinja mapping above spans many lines for readability. This is the per-record schema — what a parser checks one record at a time — not the batch shape CloudTrail actually delivers to S3; the next step reproduces that. Reproduce the S3 delivery envelope [#reproduce-the-s3-delivery-envelope] CloudTrail never delivers a bare stream of records — every S3 object is the `Records` wrapper from [the section above](#the-cloudtrail-record-format). Two formatters aggregate a whole batch into one output string ([formatter reference](/docs/plugins/formatters#per-event-vs-per-batch)), but only one of them produces a named key: [`json-batch`](/docs/plugins/formatters#json-batch) collects a batch into a bare `[...]` array, while [`template-batch`](/docs/plugins/formatters#template-batch) hands the whole batch to a Jinja template and lets it decide the wrapper — which is what a literal `Records` key needs. Swap the output block for: ```yaml title="generators/cloudtrail-records/generator.yml (output block only — input and event stay the same)" output: - file: path: output/cloudtrail-records.json formatter: format: template-batch template: '{"Records": [{{ events | join(", ") }}]}' ``` `template-batch` exposes the whole batch as `events`, a list of the raw strings each template rendered — already valid JSON text apiece, not parsed objects. `events | join(", ")` concatenates those strings as they are, so wrapping the result in `[...]` reassembles a genuine JSON array; writing `{{ events }}` directly instead would print Jinja's own textual representation of that list — comma-separated and single-quoted, not valid JSON at all. The bracket and the join are what turn a batch of independent records into the array CloudTrail's own `Records` key expects. The result [#the-result] Running the generator from the previous step in live mode for a short burst produces a steady stream, one record per second, `console_login` turning up close to its configured 15% share. Four consecutive lines from an actual run: ```json title="output/cloudtrail.json" {"awsRegion": "us-east-1", "eventID": "6744070a-36de-48d6-a623-1b73eecddb98", "eventName": "AssumeRole", "eventSource": "sts.amazonaws.com", "eventTime": "2026-07-17T17:25:51Z", "eventType": "AwsApiCall", "eventVersion": "1.09", "managementEvent": true, "readOnly": false, "recipientAccountId": "111122223333", "requestParameters": {"durationSeconds": 3600, "roleArn": "arn:aws:iam::111122223333:role/CI-DeployRole", "roleSessionName": "deploy-52428934"}, "responseElements": {"assumedRoleUser": {"arn": "arn:aws:sts::111122223333:assumed-role/CI-DeployRole/deploy-52428934", "assumedRoleId": "AROALPNWYSCZ21373705:deploy-52428934"}, "credentials": {"accessKeyId": "ASIADCENDLJI63843220", "sessionToken": "2470ea39a9f90c4cee58fc3692c658669512b126ba0c259dc880869122e3d4ca"}}, "sourceIPAddress": "201.186.144.205", "userAgent": "aws-cli/2.15.22 Python/3.11.8 Linux/6.5.0", "userIdentity": {"accessKeyId": "AKIADQAMYBPJ36610238", "accountId": "111122223333", "arn": "arn:aws:iam::111122223333:user/hannah.becker", "principalId": "AIDAZGZRGLER73255734", "type": "IAMUser", "userName": "hannah.becker"}} {"awsRegion": "us-east-1", "eventID": "a3278d5a-19dd-43f4-80c8-d2e78a556764", "eventName": "AssumeRole", "eventSource": "sts.amazonaws.com", "eventTime": "2026-07-17T17:25:52Z", "eventType": "AwsApiCall", "eventVersion": "1.09", "managementEvent": true, "readOnly": false, "recipientAccountId": "777788889999", "requestParameters": {"durationSeconds": 3600, "roleArn": "arn:aws:iam::777788889999:role/CI-DeployRole", "roleSessionName": "deploy-fc7af4da"}, "responseElements": {"assumedRoleUser": {"arn": "arn:aws:sts::777788889999:assumed-role/CI-DeployRole/deploy-fc7af4da", "assumedRoleId": "AROAHNSZXTBN47215902:deploy-fc7af4da"}, "credentials": {"accessKeyId": "ASIASGYDEFTA32984751", "sessionToken": "6a91fc62a42ba0e616aa9a49a9192215cc72594364e5168d0c9a2d8d801a74b7"}}, "sourceIPAddress": "196.74.51.180", "userAgent": "aws-cli/2.15.22 Python/3.11.8 Linux/6.5.0", "userIdentity": {"accessKeyId": "AKIAPCLHTJOP70692867", "accountId": "777788889999", "arn": "arn:aws:iam::777788889999:user/morgan.lee", "principalId": "AIDADZHMDQAR94347894", "type": "IAMUser", "userName": "morgan.lee"}} {"additionalEventData": {"MFAUsed": "No"}, "awsRegion": "us-east-1", "eventID": "c9612245-893d-4bd2-9354-83c2784c98bc", "eventName": "ConsoleLogin", "eventSource": "signin.amazonaws.com", "eventTime": "2026-07-17T17:25:53Z", "eventType": "AwsConsoleSignIn", "eventVersion": "1.09", "managementEvent": true, "readOnly": false, "recipientAccountId": "777788889999", "requestParameters": null, "responseElements": {"ConsoleLogin": "Success"}, "sourceIPAddress": "198.61.173.73", "userAgent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.0 Safari/605.1.15", "userIdentity": {"accountId": "777788889999", "arn": "arn:aws:iam::777788889999:user/diego.alvarez", "principalId": "AIDANLPRZBMB13811225", "type": "IAMUser", "userName": "diego.alvarez"}} {"awsRegion": "us-east-1", "eventID": "d5a90865-be9f-4a1f-858d-d3204470b2eb", "eventName": "AssumeRole", "eventSource": "sts.amazonaws.com", "eventTime": "2026-07-17T17:25:54Z", "eventType": "AwsApiCall", "eventVersion": "1.09", "managementEvent": true, "readOnly": false, "recipientAccountId": "111122223333", "requestParameters": {"durationSeconds": 3600, "roleArn": "arn:aws:iam::111122223333:role/DevOpsDeployRole", "roleSessionName": "deploy-1c966eda"}, "responseElements": {"assumedRoleUser": {"arn": "arn:aws:sts::111122223333:assumed-role/DevOpsDeployRole/deploy-1c966eda", "assumedRoleId": "AROASXETSZUD11620091:deploy-1c966eda"}, "credentials": {"accessKeyId": "ASIABFASVOIF73488789", "sessionToken": "53eb4562c21cf50d680e6e67f0d0a114878f9596768ae5b150b228d98b1751ff"}}, "sourceIPAddress": "170.133.244.178", "userAgent": "aws-cli/2.15.22 Python/3.11.8 Linux/6.5.0", "userIdentity": {"accessKeyId": "AKIAGECGOLDL04697431", "accountId": "111122223333", "arn": "arn:aws:iam::111122223333:user/diego.alvarez", "principalId": "AIDAOLFYPPHA59776030", "type": "IAMUser", "userName": "diego.alvarez"}} ``` The first, second, and fourth lines are `AssumeRole` calls against three different accounts; the third is the `AwsConsoleSignIn` in between, `responseElements.ConsoleLogin` reading `Success`. A failed sign-in from a separate run carries the same shape with two fields flipped: ```json title="A failed console sign-in, from a separate run" {"additionalEventData": {"MFAUsed": "Yes"}, "awsRegion": "us-east-1", "errorMessage": "Failed authentication", "eventID": "d74a6938-0417-44d4-9f5c-bbb5d1c3e641", "eventName": "ConsoleLogin", "eventSource": "signin.amazonaws.com", "eventTime": "2026-07-17T17:23:43Z", "eventType": "AwsConsoleSignIn", "eventVersion": "1.09", "managementEvent": true, "readOnly": false, "recipientAccountId": "444455556666", "requestParameters": null, "responseElements": {"ConsoleLogin": "Failure"}, "sourceIPAddress": "136.14.207.24", "userAgent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.0 Safari/605.1.15", "userIdentity": {"accountId": "444455556666", "arn": "arn:aws:iam::444455556666:user/diego.alvarez", "principalId": "AIDAVKIOVJAK71473602", "type": "IAMUser", "userName": "diego.alvarez"}} ``` `responseElements.ConsoleLogin` reads `Failure` and `errorMessage` appears — present only because this attempt did not succeed, exactly as the Key fields section describes. Swapping to the `template-batch` output from the step above produces a different shape entirely — not a per-record stream, but a single JSON object per batch, its `Records` array holding every record that batch contained: ```json title="output/cloudtrail-records.json" {"Records": [{"awsRegion": "us-east-1", "eventID": "2405d5d8-79b9-4e3a-81cd-a3b259d1779a", "eventName": "AssumeRole", "eventSource": "sts.amazonaws.com", "eventTime": "2026-07-17T17:26:42Z", "eventType": "AwsApiCall", "eventVersion": "1.09", "managementEvent": true, "readOnly": false, "recipientAccountId": "777788889999", "requestParameters": {"durationSeconds": 3600, "roleArn": "arn:aws:iam::777788889999:role/DevOpsDeployRole", "roleSessionName": "deploy-d64fadeb"}, "responseElements": {"assumedRoleUser": {"arn": "arn:aws:sts::777788889999:assumed-role/DevOpsDeployRole/deploy-d64fadeb", "assumedRoleId": "AROAOHTMVGOQ97546276:deploy-d64fadeb"}, "credentials": {"accessKeyId": "ASIAPIYHWTMC06659140", "sessionToken": "c997c0f487750f3703c6edc1650e21b378152a18edfe0e7fb7f54c966c2639d4"}}, "sourceIPAddress": "201.16.102.233", "userAgent": "aws-cli/2.15.22 Python/3.11.8 Linux/6.5.0", "userIdentity": {"accessKeyId": "AKIAHJNZPSGY56402318", "accountId": "777788889999", "arn": "arn:aws:iam::777788889999:user/priya.desai", "principalId": "AIDAORJUFOQW84574538", "type": "IAMUser", "userName": "priya.desai"}}, {"awsRegion": "us-east-1", "eventID": "58b3f050-1674-4f1f-8368-751be74fe07a", "eventName": "AssumeRole", "eventSource": "sts.amazonaws.com", "eventTime": "2026-07-17T17:26:42Z", "eventType": "AwsApiCall", "eventVersion": "1.09", "managementEvent": true, "readOnly": false, "recipientAccountId": "111122223333", "requestParameters": {"durationSeconds": 3600, "roleArn": "arn:aws:iam::111122223333:role/DevOpsDeployRole", "roleSessionName": "deploy-35af6c90"}, "responseElements": {"assumedRoleUser": {"arn": "arn:aws:sts::111122223333:assumed-role/DevOpsDeployRole/deploy-35af6c90", "assumedRoleId": "AROAXRJTQXNB15780269:deploy-35af6c90"}, "credentials": {"accessKeyId": "ASIAMMDIHIRE12817713", "sessionToken": "7d5ce0aaaa6fa3ba4e5bffa8ed6fff78df7b34658dee91457590cc4b231042bf"}}, "sourceIPAddress": "208.134.186.179", "userAgent": "aws-cli/2.15.22 Python/3.11.8 Linux/6.5.0", "userIdentity": {"accessKeyId": "AKIAPOEOAVJF59014958", "accountId": "111122223333", "arn": "arn:aws:iam::111122223333:user/diego.alvarez", "principalId": "AIDAOUOLQOBY03768790", "type": "IAMUser", "userName": "diego.alvarez"}}, {"awsRegion": "us-east-1", "eventID": "bbee1d49-b0c3-416d-9b39-a667bafb18cb", "eventName": "AssumeRole", "eventSource": "sts.amazonaws.com", "eventTime": "2026-07-17T17:26:42Z", "eventType": "AwsApiCall", "eventVersion": "1.09", "managementEvent": true, "readOnly": false, "recipientAccountId": "111122223333", "requestParameters": {"durationSeconds": 3600, "roleArn": "arn:aws:iam::111122223333:role/CI-DeployRole", "roleSessionName": "deploy-5ef96124"}, "responseElements": {"assumedRoleUser": {"arn": "arn:aws:sts::111122223333:assumed-role/CI-DeployRole/deploy-5ef96124", "assumedRoleId": "AROAFUPIDPYP64043815:deploy-5ef96124"}, "credentials": {"accessKeyId": "ASIAVQXBEXXX68850070", "sessionToken": "7d4b29105c930a179fb7b1af9bc78a02efc8ceba85c3137b4b8381b7acccb48e"}}, "sourceIPAddress": "188.32.116.163", "userAgent": "aws-cli/2.15.22 Python/3.11.8 Linux/6.5.0", "userIdentity": {"accessKeyId": "AKIANLTUQVUU90706873", "accountId": "111122223333", "arn": "arn:aws:iam::111122223333:user/priya.desai", "principalId": "AIDAJKDHWJPJ65518893", "type": "IAMUser", "userName": "priya.desai"}}]} ``` The result is the same envelope [the section above](#the-cloudtrail-record-format) described — a single JSON object with one `Records` array, holding three complete records — now produced by Eventum instead of quoted from AWS's docs. Three shapes have appeared across this lesson, and a pipeline built against CloudTrail needs to know which one it is looking at: the per-record camelCase schema `json` produces above, the batched `Records` envelope `template-batch` reproduces here, and the ECS-normalized `aws.cloudtrail.*` document a collector like Filebeat's AWS module produces once it parses either one — covered in the [ECS lesson](/docs/tutorials/formats/ecs), not this page. FAQ [#faq] `json-batch` collects a batch into a bare JSON array — `[{...}, {...}]` — with no key wrapping it, because it has no way to know what that key should be called. CloudTrail's own envelope is a JSON object with a `Records` field, not a bare array, so only a formatter that lets you author the wrapper yourself can reproduce it — which is exactly what `template-batch` is for. Any destination that already expects a bare array — an HTTP endpoint accepting a batch payload, among others — is what `json-batch` is for instead. Both fields are always present in the record, but either can be `null`. `requestParameters` is `null` for calls that take none, like the `ConsoleLogin` example above. `responseElements` is `null` for every read-only call (`DescribeInstances`, `GetCallerIdentity`, and similar `Describe`/`List`/`Get` actions) and for write calls that return nothing — AWS's own reference singles out `responseElements` as present only for actions that create, update, or delete something. A parser that assumes both fields always hold an object breaks on exactly these records. Related [#related] * The [formats field guide](/docs/tutorials/formats) covering every supported log and event shape * The [NDJSON lesson](/docs/tutorials/formats/ndjson) for the per-record stream shape on its own terms, independent of any particular schema * The [ECS lesson](/docs/tutorials/formats/ecs) for the schema CloudTrail records get mapped into downstream * The [formatters reference](/docs/plugins/formatters) for every formatter used above, `template-batch` included * The [Stream to your stack](/docs/tutorials/delivery) lessons, and specifically [OpenSearch delivery](/docs/tutorials/delivery/opensearch), for indexing generated records into a real cluster * The [detection-testing lesson](/docs/tutorials/detection-testing) for planting an attack-shaped pattern in a generated stream and testing a Sigma rule against it — `AssumeRole` and `ConsoleLogin` are as common a detection target as Windows Security's own logon events * The [cloud-aws-cloudtrail](/hub/cloud-aws-cloudtrail) generator in the Eventum Hub for the downstream ECS-mapped form of this same source # ECS fields: the Elastic Common Schema A firewall calls it `src_ip`. A cloud identity provider's API calls the same fact `sourceIpAddress`. A home-grown application logs it as `client.address`. All three name the exact same piece of data about the exact same login attempt, and a query, dashboard, or detection rule written against one of those field names finds nothing in the other two — not because the data disagrees, but because nothing forces different sources to agree on what to call it. Elastic Common Schema fixes the field names once, across every source that adopts it, so the same query keeps matching regardless of which source produced the event. Eventum generates events that already carry ECS's real field names and structure directly from a template, and its JSON formatter delivers them ready to index. An index mapping, a Kibana dashboard, or a detection rule built against ECS fields can be exercised against them without a live firewall, identity provider, or application in place. What is Elastic Common Schema? [#what-is-elastic-common-schema] [ECS](https://www.elastic.co/guide/en/ecs/current/index.html) is Elastic's open, versioned field-naming specification: instead of every log source inventing its own name for the same fact, every source that adopts ECS emits the same field name for the same kind of data. This lesson matches **ECS version 9.4.0**, the current release documented in the [ECS field reference](https://www.elastic.co/docs/reference/ecs/ecs-field-reference). ECS does not define fixed event "classes" the way some other normalization schemas do. Instead, it defines **field sets** — Base, Event, Host, User, Source, Related, and dozens more — and a single event mixes in whichever field sets its data actually has. A field set's name becomes a namespace: the Event field set's `category` field is named `event.category`, and once an event is rendered as JSON, that dotted name becomes a nested object, `{"event": {"category": [...]}}`, rather than a flat key: ECS's native home is the Elastic Stack: Elasticsearch stores the fields, and Kibana's dashboards and detection rules query them by name. OpenSearch, having forked from Elasticsearch and remaining compatible on the same document and bulk APIs, indexes and queries the identical ECS-shaped documents the same way, even though ECS itself is an Elastic-maintained specification rather than a formal OpenSearch one. ECS fields [#ecs-fields] `@timestamp` sits at the root of every event, outside any field set — the one required date field every ECS event carries, marking when the source produced the event. `ecs.version` is required on every event too. Its value names which ECS version the event's fields conform to — `9.4.0` for every event generated below. The **event** field set carries the fields that classify what happened. `event.kind` sets the broadest classification, one of eight allowed values. `event` — used by every example below — covers ordinary activity and is by far the most common; `alert` covers a detection firing; the remaining six (`metric`, `state`, `asset`, `enrichment`, `pipeline_error`, `signal`) cover narrower cases like numeric measurements and ingestion diagnostics. `event.category` and `event.type` narrow further, together, and both are constrained to closed lists rather than free text: `event.category` names one of 20 allowed broad buckets — `authentication`, `network`, `file`, `process`, `iam`, and 15 others — and `event.type` names one of 18 allowed sub-buckets within it, such as `start`, `end`, `info`, `creation`, or `deletion`. The two aren't independent: ECS documents which `event.type` values are expected for each `event.category`. The `authentication` category, for instance, expects only `start`, `end`, or `info` — a login attempt is the `start` of the challenge-response process regardless of how it turns out, and `end` marks a logoff. `event.action` stays free text, a short label for what specifically happened (`ssh_login`, `user-password-change`), and `event.outcome` closes the fact off with exactly one of three allowed values — `success`, `failure`, or `unknown`. `host.*` names the machine the event concerns. `host.name` and `host.ip` are the two fields nearly every event populates — `host.ip` is itself documented as an array, since a single host can carry more than one address. `related.*` exists purely to make an event pivotable. `related.user`, `related.ip`, `related.hosts`, and `related.hash` collect every user, address, hostname, or hash that appears anywhere else in the same event, so a search for one value finds it no matter which specific field it originally sat in. All four are documented as arrays, and that holds even when an event only has one of something: an event naming exactly one user still writes `related.user: ["alice"]`, never a bare string. Generate an ECS authentication event with Eventum [#generate-an-ecs-authentication-event-with-eventum] This section generates SSH access to a small fleet of Linux servers — ssh logs are one of the two sources ECS's own authentication category description names as typical, alongside Windows event logs — producing a common successful logon and a rarer failed attempt, each a complete ECS authentication event. The templates [#the-templates] Because an ECS event nests several field sets inside one document — `event`, `host`, `user`, `source`, `related` — each template below builds the event as a single Jinja mapping and serializes the whole structure at once with the built-in `tojson` filter, the same approach the [OCSF lesson](/docs/tutorials/formats/ocsf) uses for its own nested event shape. Every field inside it is drawn from [`module.rand`](/docs/plugins/event/template/modules), the [template](/docs/plugins/event/template) event plugin's built-in randomization module. `templates/ssh-login-success.jinja` renders the common case: ```jinja title="generators/ecs-ssh-auth/templates/ssh-login-success.jinja" {%- set user_name = module.rand.choice(["alice", "bob", "carol", "dave", "frank"]) -%} {%- set user_id = module.rand.number.integer(1000, 1010) -%} {%- set hostname = module.rand.choice(["web-prod-03", "app-prod-11", "db-prod-02", "cache-prod-05"]) -%} {%- set host_ip = module.rand.network.ip_v4_private_a() -%} {%- set src_ip = module.rand.network.ip_v4_public() -%} {%- set src_port = module.rand.number.integer(1024, 65535) -%} {%- set event = { "@timestamp": timestamp.isoformat(), "ecs": {"version": "9.4.0"}, "event": { "kind": "event", "category": ["authentication"], "type": ["start"], "action": "ssh_login", "outcome": "success" }, "host": {"name": hostname, "ip": [host_ip]}, "user": {"name": user_name, "id": user_id | string}, "source": {"ip": src_ip, "port": src_port}, "related": { "user": [user_name], "ip": [src_ip, host_ip], "hosts": [hostname] } } -%} {{ event | tojson }} ``` `templates/ssh-login-failure.jinja` renders the rarer case, the same category and type, a failed outcome and a reason instead: ```jinja title="generators/ecs-ssh-auth/templates/ssh-login-failure.jinja" {%- set user_name = module.rand.choice(["alice", "bob", "carol", "dave", "frank"]) -%} {%- set hostname = module.rand.choice(["web-prod-03", "app-prod-11", "db-prod-02", "cache-prod-05"]) -%} {%- set host_ip = module.rand.network.ip_v4_private_a() -%} {%- set src_ip = module.rand.network.ip_v4_public() -%} {%- set src_port = module.rand.number.integer(1024, 65535) -%} {%- set reason = module.rand.choice(["invalid password", "no such user", "account locked", "connection closed by authenticating user"]) -%} {%- set event = { "@timestamp": timestamp.isoformat(), "ecs": {"version": "9.4.0"}, "event": { "kind": "event", "category": ["authentication"], "type": ["start"], "action": "ssh_login", "outcome": "failure", "reason": reason }, "host": {"name": hostname, "ip": [host_ip]}, "user": {"name": user_name}, "source": {"ip": src_ip, "port": src_port}, "related": { "user": [user_name], "ip": [src_ip, host_ip], "hosts": [hostname] } } -%} {{ event | tojson }} ``` Both templates share `event.category: ["authentication"]` and `event.type: ["start"]` (see above) — only `event.outcome` tells them apart. The failure template adds `event.reason`, a free-text field naming why the action in `event.action` did not succeed. `related.user`, `related.ip`, and `related.hosts` collect the user, both addresses involved (the connecting client and the host being logged into), and the hostname named elsewhere in the same event — each as an array, per the rule above. Neither template populates `related.hash`, since a login event has no file or process hash to relate. Populate it the same way — as an array — on any event that does carry one. The generator config [#the-generator-config] The template config uses [`mode: chance`](/docs/plugins/event/template/modes#chance) to pick between the two templates per timestamp, weighted so successful logons dominate and failures stay rare. A [cron](/docs/plugins/input/cron) input ticks once a second, and a [file](/docs/plugins/output/file) output writes each event through the [json](/docs/plugins/formatters#json) formatter with `indent: 2`, so the nested structure stays readable: ```yaml title="generators/ecs-ssh-auth/generator.yml" input: - cron: expression: "* * * * * *" count: 1 event: template: mode: chance templates: - login_success: template: templates/ssh-login-success.jinja chance: 85 - login_failure: template: templates/ssh-login-failure.jinja chance: 15 output: - file: path: output/events.json formatter: format: json indent: 2 ``` ECS example [#ecs-example] The generator above produces one authentication event per second, mostly successful logons and, at the configured 15% chance, an occasional failure. A successful logon from an actual run: ```json title="A successful SSH logon" { "@timestamp": "2026-07-12T11:02:19+00:00", "ecs": { "version": "9.4.0" }, "event": { "action": "ssh_login", "category": [ "authentication" ], "kind": "event", "outcome": "success", "type": [ "start" ] }, "host": { "ip": [ "10.154.76.9" ], "name": "cache-prod-05" }, "related": { "hosts": [ "cache-prod-05" ], "ip": [ "170.54.164.16", "10.154.76.9" ], "user": [ "alice" ] }, "source": { "ip": "170.54.164.16", "port": 16974 }, "user": { "id": "1007", "name": "alice" } } ``` A failed attempt from the same run: ```json title="A failed SSH logon" { "@timestamp": "2026-07-12T11:02:23+00:00", "ecs": { "version": "9.4.0" }, "event": { "action": "ssh_login", "category": [ "authentication" ], "kind": "event", "outcome": "failure", "reason": "no such user", "type": [ "start" ] }, "host": { "ip": [ "10.255.237.80" ], "name": "db-prod-02" }, "related": { "hosts": [ "db-prod-02" ], "ip": [ "28.140.13.243", "10.255.237.80" ], "user": [ "frank" ] }, "source": { "ip": "28.140.13.243", "port": 57738 }, "user": { "name": "frank" } } ``` Both events carry the same `event.category` and `event.type` — `authentication` and `start`, as before. Only `event.outcome` — and `event.reason` on the failed attempt — changes between the two events shown here. `related.ip` carries two addresses on every event, the connecting client and the host itself, and `related.user` and `related.hosts` still write their single value as a one-element array, exactly as the rule requires. FAQ [#faq] Both standardize field names across sources so one query or detection rule works against all of them, but they organize that standardization differently. ECS namespaces fields into dotted, composable field sets — `event.category`, `host.name`, `related.ip` — that mix into a JSON document as needed, native to the Elastic Stack and, through OpenSearch's API compatibility, usable there as well. [OCSF](/docs/tutorials/formats/ocsf) instead defines a fixed hierarchy of numbered categories, classes, and activities per kind of event, backed by a broader multi-vendor group under the Linux Foundation. Where ECS lets an authentication event mix in whatever field sets apply to it, OCSF instead requires that same event to conform to one predefined Authentication class — a fixed required-field envelope and enumerated values, with optional and recommended fields layered on top. Neither replaces the other universally — which one a pipeline targets usually follows from which backend receives the data. Only two, by ECS's own specification: `@timestamp` and `ecs.version`, which every event must carry regardless of what it describes. Everything else — including every `event.*`, `host.*`, and `related.*` field used above — is documented at a "core" or "extended" level, meaning it is expected wherever it applies rather than mandatory on every event; an event with no file involved simply omits `file.*` fields entirely. In practice, a usable authentication event needs more than the bare minimum to be worth generating — at least enough of `event.category`, `event.type`, and `event.outcome` to be found by a query written against them — but ECS itself only enforces the two root fields. Related [#related] * The [formats field guide](/docs/tutorials/formats) covering every supported log and event shape * The [structured logging lesson](/docs/tutorials/foundations/structured-logging) for the field-per-event foundation ECS's naming layer builds on * The [OCSF lesson](/docs/tutorials/formats/ocsf) for the other widely adopted normalization schema * The [SIEM test data scenario](/docs/tutorials/siem-events) for a complete stateful Windows Security event stream indexed into OpenSearch * The [OpenSearch delivery lesson](/docs/tutorials/delivery/opensearch) for indexing generated events, ECS-shaped or otherwise, over the bulk API * The [Linux Auditd generator](/hub/linux-auditd) in the Eventum Hub for a real ECS-mapped authentication source across a full host fleet * The [template](/docs/plugins/event/template) event plugin and [formatters](/docs/plugins/formatters) reference for every field and format used above # Log and event formats: a field guide Every log pipeline speaks a format. Get it wrong and your parser drops the event; get it right and detections, dashboards, and alerts behave correctly. This track explains the formats you meet most often and shows how to generate a compliant sample of each with Eventum. # LEEF format: header and delimiter Testing a QRadar DSM (Device Support Module) mapping, or a correlation rule built against LEEF fields, requires LEEF events shaped to exercise the exact case under test. A live firewall, intrusion prevention system, or endpoint agent configured to forward LEEF can produce them, but standing one up just to generate test traffic is slow, and its output arrives on its own schedule, not the tester's. Public examples are scarce too: the authoritative specification is an IBM PDF guide, and nearly everything else that ranks for the format's name is vendor integration documentation that assumes the reader already knows the structure it depends on. Eventum renders LEEF lines from a template — header fields in order, attributes correctly delimited — and ships them to the QRadar log source directly, so the DSM mapping or the rule gets tested without standing up an appliance. What LEEF looks like [#what-leef-looks-like] LEEF (Log Event Extended Format) is a text-based log format [defined by IBM](https://www.ibm.com/docs/en/SS42VS_DSM/pdf/b_Leef_format_guide.pdf) so that any vendor's product can send events straight into IBM Security QRadar without a custom parser: a device or application formats its output as LEEF, and QRadar's LEEF DSM already knows how to read it. Firewalls, endpoint agents, and other security products that ship a QRadar integration typically offer LEEF as one of their output formats, alongside syslog or CEF for every other SIEM. LEEF events are plain UTF-8 text. A LEEF message is a header followed by a set of delimited attributes, both riding inside a single line of text. LEEF 1.0's header has five pipe-delimited fields: ```text LEEF:Version|Vendor|Product|Version|EventID| ``` LEEF 2.0 adds a sixth, optional field that lets the event declare its own attribute delimiter instead of accepting the default: ```text LEEF:Version|Vendor|Product|Version|EventID|Delimiter| ``` Every field is mandatory and positional, delimited by unescaped pipes (`|`) — except `Delimiter`, which exists only in 2.0 and can be left empty. | Field | Description | | --------------- | ------------------------------------------------------------------------------------------------------------------------------------------ | | LEEF version | Format version, `1.0` or `2.0`, written as part of the literal `LEEF:` prefix. | | Vendor | Identifies the vendor of the device or application that produced the event. | | Product | Identifies the product. | | Product version | The product's version string. | | EventID | An identifier for the event or signature type — numeric or text, vendor-assigned. | | Delimiter | LEEF 2.0 only. The character that separates the attributes that follow. Empty, or omitted along with its trailing pipe, defaults to a tab. | IBM's format guide includes this LEEF 2.0 line as an example, using a caret instead of the tab default: ```text title="From IBM's LEEF format guide" LEEF:2.0|Lancope|StealthWatch|1.0|41|^|src=10.0.1.8^dst=10.0.0.5^sev=5^srcPort=81^dstPort=21 ``` | Segment | Value | Field | | ------- | ------------------------------------------------------- | ---------------------------------------------- | | 1 | `LEEF:2.0` | LEEF version | | 2 | `Lancope` | Vendor | | 3 | `StealthWatch` | Product | | 4 | `1.0` | Product version | | 5 | `41` | EventID | | 6 | `^` | Delimiter | | — | `src=10.0.1.8^dst=10.0.0.5^sev=5^srcPort=81^dstPort=21` | Attributes, separated by the `^` just declared | LEEF 1.0 hardcodes that separator to a tab character (`0x09`) — the format gives no way to change it. LEEF 2.0's `Delimiter` field exists specifically to lift that restriction: a vendor whose event data might itself contain a tab can declare a different character instead, as StealthWatch does above. The field accepts either a literal character or a hex value prefixed with `0x` or `x` and one to four hex digits, so the tab default can also be spelled out explicitly as `x09`. Leaving the field empty, or dropping it and its trailing pipe entirely, falls back to that same tab — the form most real LEEF 2.0 events actually use, since most attribute values have no reason to contain one. Attributes follow the header as `key=value` pairs, in any order, joined by whichever delimiter the header declares. LEEF defines a set of recommended keys so that QRadar can recognize common fields without vendor-specific configuration: | Key | Meaning | | --------------------- | --------------------------------------------------------------------------------------------------------- | | `devTime` | The event's own timestamp — preferred over the syslog header's timestamp when both are present. | | `devTimeFormat` | The pattern `devTime` is written in, e.g. `MMM dd yyyy HH:mm:ss`. Required whenever `devTime` is present. | | `sev` | Severity, an integer from `1` (lowest) to `10` (highest). | | `src` / `dst` | Source / destination IPv4 or IPv6 address. | | `srcPort` / `dstPort` | Source / destination port. | | `usrName` | The user name associated with the event. | | `proto` | Layer-4 protocol, e.g. `TCP` or `UDP`. | | `cat` | A vendor-assigned category for the event. | None of these keys are required, and LEEF places no limit on adding others beyond them — unlike CEF, which reserves a fixed set of custom slots (`cs1` through `cs6`, `cn1` through `cn3`) for data with no dedicated key. A LEEF vendor adds whatever keys its event carries; QRadar's DSM maps the recognized ones onto its own normalized fields and passes the rest through as-is. The header is where LEEF and CEF differ most. CEF's seven-field header always ends with `Name` and `Severity` — a short human-readable description and a mandatory severity value, both positional and always present. LEEF's header carries neither: there is no name field at all, and severity travels as the `sev` attribute alongside the rest of the event's data, present only if the vendor sends it. The two formats also default to a different attribute delimiter — CEF's extension is always space-separated with no way to change it, while LEEF's is tab-separated by default and, from 2.0 onward, configurable. QRadar recognizes an incoming line as LEEF by its `LEEF:` prefix and hands it to the DSM registered for that line's `Vendor` and `Product`. The DSM is what turns recognized attribute keys into QRadar's own searchable fields, which is why sticking to the recommended keys above — rather than inventing equivalents — is what makes a generated event behave like a real one once it reaches QRadar, instead of arriving as an unparsed raw log. Generate LEEF for QRadar with Eventum [#generate-leef-for-qradar-with-eventum] The generator below models the same firewall as the [CEF lesson](/docs/tutorials/formats/cef) — `Acme NetGuard` — this time emitting LEEF for a QRadar log source instead of CEF for a general SIEM: routine policy-blocked connections and rarer, higher-severity intrusion alerts. The templates [#the-templates] Fields are filled with [`module.rand`](/docs/plugins/event/template/modules), the [template](/docs/plugins/event/template) event plugin's built-in randomization module, exactly as in the CEF example. The one LEEF-specific difficulty is the delimiter: the header above declares `x09`, a tab, so every attribute pair must actually be joined by a tab and nothing else. Typing a literal tab character into a template file is fragile — an editor set to expand tabs to spaces, a code formatter, or a copy-paste through a chat window can silently turn it into something else, and the line stops matching its own header. Building the attributes as a list and joining them with the explicit escape sequence `"\t"` keeps the delimiter visible in the source and immune to that class of mistake. `templates/blocked-connection.jinja` renders the routine case, severity `4`: ```jinja title="generators/leef-firewall/templates/blocked-connection.jinja" {%- set src_ip = module.rand.network.ip_v4_public() -%} {%- set dst_ip = module.rand.network.ip_v4_private_c() -%} {%- set src_port = module.rand.number.integer(1024, 65535) -%} {%- set dst_port = module.rand.weighted_choice([22, 80, 443, 3389, 445], [10, 30, 35, 15, 10]) -%} {%- set dev_time = timestamp.strftime('%b %d %Y %H:%M:%S') -%} {%- set attrs = [ "devTime=" ~ dev_time, "devTimeFormat=MMM dd yyyy HH:mm:ss", "sev=4", "src=" ~ src_ip, "dst=" ~ dst_ip, "srcPort=" ~ src_port, "dstPort=" ~ dst_port, "proto=TCP", ] -%} LEEF:2.0|Acme|NetGuard|3.1|1000|x09|{{ attrs | join("\t") }} ``` `templates/intrusion-detected.jinja` renders the rarer case, severity `8`: ```jinja title="generators/leef-firewall/templates/intrusion-detected.jinja" {%- set src_ip = module.rand.network.ip_v4_public() -%} {%- set dst_ip = module.rand.network.ip_v4_private_c() -%} {%- set dst_port = module.rand.choice([22, 3389, 445, 1433]) -%} {%- set dev_time = timestamp.strftime('%b %d %Y %H:%M:%S') -%} {%- set attempts = module.rand.number.integer(15, 120) -%} {%- set attrs = [ "devTime=" ~ dev_time, "devTimeFormat=MMM dd yyyy HH:mm:ss", "sev=8", "src=" ~ src_ip, "dst=" ~ dst_ip, "dstPort=" ~ dst_port, "proto=TCP", "attempts=" ~ attempts, ] -%} LEEF:2.0|Acme|NetGuard|3.1|2001|x09|{{ attrs | join("\t") }} ``` `attempts` is not one of LEEF's recommended keys — the format places no restriction on additional keys, so this generator adds one of its own, exactly as a real vendor would for data the predefined set does not cover. The generator config [#the-generator-config] [`mode: chance`](/docs/plugins/event/template/modes#chance) picks one of the two templates per timestamp, weighted so blocked connections dominate and intrusions stay rare, same as the CEF generator. A [cron](/docs/plugins/input/cron) input ticks once a second, and a [tcp](/docs/plugins/output/tcp) output ships each rendered line to QRadar's syslog listener: ```yaml title="generators/leef-firewall/generator.yml" input: - cron: expression: "* * * * * *" count: 1 event: template: mode: chance templates: - blocked_connection: template: templates/blocked-connection.jinja chance: 85 - intrusion_detected: template: templates/intrusion-detected.jinja chance: 15 output: - tcp: host: qradar.example.com port: 514 separator: "\n" ``` The default `utf_8` encoding on `tcp` already matches what LEEF requires, so nothing needs to change there. For UDP delivery, replace `tcp` with [`udp`](/docs/plugins/output/udp) and keep the same `host` and `port`. To inspect lines locally before pointing the generator at a real QRadar instance, replace the output with [`file`](/docs/plugins/output/file) or [`stdout: {}`](/docs/plugins/output/stdout). The result [#the-result] Running the generator above produces one LEEF line per timestamp, alternating between the two event types: ```text LEEF:2.0|Acme|NetGuard|3.1|2001|x09|devTime=Jul 11 2026 13:07:32 devTimeFormat=MMM dd yyyy HH:mm:ss sev=8 src=192.96.151.68 dst=192.168.155.81 dstPort=3389 proto=TCP attempts=69 LEEF:2.0|Acme|NetGuard|3.1|1000|x09|devTime=Jul 11 2026 13:07:33 devTimeFormat=MMM dd yyyy HH:mm:ss sev=4 src=179.100.186.150 dst=192.168.161.82 srcPort=6025 dstPort=443 proto=TCP ``` The gaps between attributes above are not spaces — each is a single tab character, matching the `x09` the header declares. Marked explicitly, the first line reads: ```text title="Same line with the tab delimiter marked" LEEF:2.0|Acme|NetGuard|3.1|2001|x09|devTime=Jul 11 2026 13:07:32→devTimeFormat=MMM dd yyyy HH:mm:ss→sev=8→src=192.96.151.68→dst=192.168.155.81→dstPort=3389→proto=TCP→attempts=69 ``` Both lines are valid LEEF: six pipe-delimited header fields ending in the declared delimiter, tab-separated attributes that match it, `devTime` in the exact format `devTimeFormat` names, and `sev` inside the `1`–`10` range the spec defines. FAQ [#faq] Both are pipe-delimited, vendor-neutral security event formats with a header followed by key-value data, but the header shapes differ. LEEF's header carries five fields — `Version`, `Vendor`, `Product`, `Version`, `EventID` — plus an optional sixth, `Delimiter`, in LEEF 2.0; CEF's carries seven, including `Name` and `Severity`, which LEEF has no equivalent for. The attribute delimiter differs too: LEEF's are tab-separated by default and configurable from 2.0 onward, CEF's are always space-separated. LEEF is what IBM QRadar expects; CEF is what ArcSight and most other SIEMs expect. See the [CEF lesson](/docs/tutorials/formats/cef) for the full structure and how to generate it with Eventum. QRadar parses both versions — there is no release where only one works. 2.0 is the better choice for a new integration, since it can declare a delimiter other than tab, which matters if the event data itself might contain one. Absent that need, a 1.0 event is just as valid, and QRadar reads its hardcoded tab delimiter the same way it reads an explicitly declared one. A tab (`0x09`) by default, in both versions. LEEF 2.0 can override it through the header's sixth field, using a literal character or a hex value (`0x` or `x` followed by one to four hex digits) — but unless the event data itself might contain a tab, there is no reason to. Whatever the field declares must exactly match what actually separates the attributes; a mismatch is what turns a well-formed line into a QRadar parsing failure. LEEF has no transport of its own — like CEF, it typically rides inside a syslog message. Prepend a syslog header (an RFC 3164 `timestamp hostname` header, or an RFC 5424 header) to the `LEEF:Version|...` line inside the same template, then send the result over [tcp](/docs/plugins/output/tcp) or [udp](/docs/plugins/output/udp) to the port QRadar's log source listens on — `514` is the common default for plain syslog. Some log sources accept the bare `LEEF:Version|...` line directly, without any syslog header, which is the form the generator above produces. See the [syslog lesson](/docs/tutorials/formats/syslog) for both header formats. Related [#related] * The [formats field guide](/docs/tutorials/formats) covering every supported log and event shape * The [CEF lesson](/docs/tutorials/formats/cef) for the format most other SIEMs expect instead * The [syslog lesson](/docs/tutorials/formats/syslog) for wrapping any message in a standards-compliant syslog header * The [detection testing lesson](/docs/tutorials/detection-testing) for generating attack-shaped telemetry to test Sigma rules # NDJSON format: newline-delimited JSON Testing a log shipper's tailing rules, a bulk-ingest endpoint, or a `jq` pipeline requires a sample file shaped exactly like the stream those tools expect to receive: one JSON record per line, not one record nested inside a single array. A JSON array holding the same events fails that shape on both ends — nothing can start processing it until its closing `]` arrives, and appending one more event means rewriting the file's punctuation rather than simply writing a new line. Hand-writing enough realistic-looking lines to fill that gap is slow, and every line tends to end up shaped identically, which defeats the point of testing against something resembling real, varied traffic. Eventum's [file](/docs/plugins/output/file) output, paired with its [json](/docs/plugins/formatters#json) formatter, writes exactly that shape directly from a template: every event lands on its own line as a compact, independently valid JSON object, with nothing wrapping the stream and no punctuation to rewrite as new events arrive — so a log shipper, a bulk-ingest endpoint, or a `jq` filter reads a realistic NDJSON stream with no live source behind it. What is NDJSON? [#what-is-ndjson] NDJSON stands for Newline Delimited JSON: a text stream holding one JSON value per line, each line separated from the next by a single `\n` character. Every line stands on its own as a complete, independently parseable JSON value — typically an object — instead of all of them sitting inside one shared array. The same convention also goes by JSON Lines, or JSONL for short; all three names describe the identical rule, and nothing meaningful separates them beyond which one a given tool or team happens to use. No IETF or ISO body standardizes NDJSON the way [RFC 5424](https://datatracker.ietf.org/doc/html/rfc5424) standardizes syslog. It is a de facto convention instead, written up independently under two names — NDJSON and JSON Lines — that describe essentially the same rule. Adoption comes from how widely tools already support the convention, not from a numbered standard behind it. NDJSON also says nothing about what fields belong inside each line, only about how the lines themselves are delimited. A schema convention such as [ECS](/docs/tutorials/formats/ecs) or [OCSF](/docs/tutorials/formats/ocsf) fills that separate gap by standardizing field names across sources; NDJSON is the transport underneath either one, or underneath no schema at all. NDJSON example [#ndjson-example] A JSON array holding two events nests both inside one shared structure: ```json title="As a JSON array" [ {"user": "alice", "action": "login"}, {"user": "bob", "action": "logout"} ] ``` The same two events as NDJSON drop the surrounding array and the comma between them; each line is already a complete JSON value on its own: ```json title="As NDJSON" {"user": "alice", "action": "login"} {"user": "bob", "action": "logout"} ``` Visually the difference is small — a pair of brackets and a comma disappear — but it changes how the two shapes behave under everything covered next. NDJSON vs JSON [#ndjson-vs-json] | | JSON array | NDJSON | | ------------------------------------------------ | --------------------------------------------- | -------------------------------------------------------------------------------------------------- | | Structure | One array wrapping every event | One JSON value per line, no wrapper | | Start processing | Only after the closing `]` arrives | As soon as the first line arrives | | Append one more event | Rewrite the closing bracket and add a comma | Write one more line | | Line-oriented tools (`grep`, `tail -f`, `wc -l`) | Operate on raw text, not on individual events | Operate on individual events directly, since one line is one record | | A crash mid-write | Can leave a truncated, invalid array | Leaves every complete line already written still valid; only a half-written final line is affected | Unlike a tabular format such as CSV, neither shape requires every record to carry the same fields — a property of JSON objects in general, not something the array or the newline adds or removes. The second and third rows are why log pipelines settle on NDJSON rather than a JSON array. Log shippers such as Filebeat and Fluent Bit tail a file and forward each new line as its own record the moment it appears, exactly the way they already handle a plain-text log. The `jq` command-line processor reads a stream of top-level JSON values by default, so it processes an NDJSON file exactly as it arrives, one line at a time; a JSON array is a single top-level value instead, so pulling its elements out one at a time means adding a `.[]` filter first. Bulk-ingest and bulk-load APIs use the same shape for the same reason: OpenSearch and Elasticsearch's bulk endpoint takes an NDJSON request body — an action line followed by a document line, repeated for every operation (see the [OpenSearch delivery lesson](/docs/tutorials/delivery/opensearch) for a generator that produces exactly that body) — and BigQuery's load-job API, for one, names newline-delimited JSON as a source format directly. Generate NDJSON with Eventum [#generate-ndjson-with-eventum] The generator below simulates an API gateway's request log: a common successful-request event and a rarer error event, both landing in the same NDJSON file. The templates [#the-templates] Fields are filled with [`module.rand`](/docs/plugins/event/template/modules), the [template](/docs/plugins/event/template) event plugin's built-in randomization module. Each template renders one compact JSON object directly — no array brackets, no trailing comma — since NDJSON's line-per-record shape comes from how the output stage writes events, not from anything the template itself needs to do. `templates/api-request.jinja` renders the common case, level `INFO`: ```jinja title="generators/api-logs/templates/api-request.jinja" {%- set method = module.rand.weighted_choice({"GET": 70, "POST": 20, "PUT": 6, "DELETE": 4}) -%} {%- set path = module.rand.choice(["/api/users", "/api/orders", "/api/products", "/api/search"]) -%} {%- set status = module.rand.weighted_choice({200: 90, 201: 5, 304: 5}) -%} {%- set duration_ms = module.rand.number.integer(5, 250) -%} {"timestamp": "{{ timestamp.isoformat() }}", "level": "INFO", "service": "api-gateway", "method": "{{ method }}", "path": "{{ path }}", "status": {{ status }}, "duration_ms": {{ duration_ms }}} ``` `templates/api-error.jinja` renders the rarer failure, level `ERROR`, with `request_id` and `error` fields in place of `duration_ms`. Nothing requires the two templates to share a structure — each rendered line is independently valid JSON, so a success event and an error event can carry entirely different fields in the same stream: ```jinja title="generators/api-logs/templates/api-error.jinja" {%- set method = module.rand.weighted_choice({"GET": 70, "POST": 30}) -%} {%- set path = module.rand.choice(["/api/users", "/api/orders", "/api/products", "/api/search"]) -%} {%- set status = module.rand.choice([500, 502, 503]) -%} {%- set request_id = module.rand.crypto.uuid4() -%} {%- set error = module.rand.choice(["upstream timeout", "database connection lost", "unhandled exception"]) -%} {"timestamp": "{{ timestamp.isoformat() }}", "level": "ERROR", "service": "api-gateway", "method": "{{ method }}", "path": "{{ path }}", "status": {{ status }}, "request_id": "{{ request_id }}", "error": "{{ error }}"} ``` The generator config [#the-generator-config] [mode: chance](/docs/plugins/event/template/modes#chance) picks one of the two templates per timestamp, weighted so successful requests dominate and errors stay rare. A [cron](/docs/plugins/input/cron) input ticks once a second, and a [file](/docs/plugins/output/file) output writes each rendered event to `output/events.ndjson` through the [json](/docs/plugins/formatters#json) formatter: ```yaml title="generators/api-logs/generator.yml" input: - cron: expression: "* * * * * *" count: 1 event: template: mode: chance templates: - api_request: template: templates/api-request.jinja chance: 85 - api_error: template: templates/api-error.jinja chance: 15 output: - file: path: output/events.ndjson separator: "\n" formatter: format: json ``` The `json` formatter validates each event and compacts it to a single line; `file`'s own `separator` field is what then places each formatted event on its own line in the output file. `separator` defaults to the host operating system's line separator, which is `\n` on Linux and macOS but `\r\n` on Windows — setting it explicitly to `"\n"`, as above, keeps the file's line endings NDJSON-compliant regardless of which platform the generator runs on. Leave `indent` at its default of `0` on the `json` formatter. A pretty-printed value (`indent` greater than `0`) spreads one event across several physical lines, and `file`'s separator only follows the whole formatted string — so the output would mix record boundaries with the JSON value's own internal line breaks, no longer one value per line. Nothing in Eventum requires a particular file extension. `.ndjson` and `.jsonl` are both common choices that signal the file's line-delimited shape to whatever reads it next; `path` accepts either, or any other extension. To index this stream directly into OpenSearch or another destination instead of a local file, swap the `file` output for [opensearch](/docs/plugins/output/opensearch) or [kafka](/docs/plugins/output/kafka) and keep the same `json` formatter. The result [#the-result] Running the generator above for a short stretch produces a steady one-line-per-second stream, `api_request` events dominating and an `api_error` every so often at the configured 15% chance. Five consecutive lines from an actual run: ```json title="output/events.ndjson" {"timestamp": "2026-07-11T22:17:24+00:00", "level": "INFO", "service": "api-gateway", "method": "GET", "path": "/api/products", "status": 200, "duration_ms": 184} {"timestamp": "2026-07-11T22:17:25+00:00", "level": "INFO", "service": "api-gateway", "method": "DELETE", "path": "/api/products", "status": 200, "duration_ms": 178} {"timestamp": "2026-07-11T22:17:26+00:00", "level": "ERROR", "service": "api-gateway", "method": "POST", "path": "/api/orders", "status": 503, "request_id": "9da8bf81-93bf-4e59-9541-0099997b8b1b", "error": "unhandled exception"} {"timestamp": "2026-07-11T22:17:27+00:00", "level": "INFO", "service": "api-gateway", "method": "GET", "path": "/api/search", "status": 200, "duration_ms": 214} {"timestamp": "2026-07-11T22:17:28+00:00", "level": "INFO", "service": "api-gateway", "method": "GET", "path": "/api/search", "status": 200, "duration_ms": 168} ``` Every line parses on its own as a complete JSON object, and the third line proves the point made earlier about shape: it carries `request_id` and `error` instead of `duration_ms`, and still sits in the same file as the `INFO` lines around it without breaking anything downstream that reads the file one line at a time. FAQ [#faq] The file as a whole is not: JSON syntax allows exactly one top-level value, and a file with more than one line has more than one top-level value, which a standards-compliant JSON parser rejects once it reaches the second one. Each line on its own is a complete, valid JSON value, parsed independently — that independence is the entire point of the format. None. All three names describe the same convention: one JSON value per line, separated by `\n`. NDJSON and JSON Lines are two near-identical community write-ups of the same rule, and JSONL is simply the short form of "JSON Lines" commonly used as a file extension. Pick whichever name matches the tooling or team convention already in place; Eventum's output is identical either way. No. Each line is independently valid JSON, so nothing requires the same keys from one line to the next — the result above demonstrates this directly: the `INFO` lines carry `duration_ms`, the `ERROR` line instead carries `request_id` and `error`, and both are equally valid NDJSON. A schema convention such as [ECS](/docs/tutorials/formats/ecs) or [OCSF](/docs/tutorials/formats/ocsf) can standardize field names across sources on top of NDJSON, but NDJSON itself never requires it. Related [#related] * The [formats field guide](/docs/tutorials/formats) covering every supported log and event shape * The [structured logging lesson](/docs/tutorials/foundations/structured-logging) for JSON as a field structure, one layer below the wire shape covered here * The [OpenSearch delivery lesson](/docs/tutorials/delivery/opensearch) for the bulk API that consumes an NDJSON body directly * The [IoT test data](/docs/tutorials/iot-telemetry) tutorial for this same shape written through the `stdout` output instead of `file` * The [ECS](/docs/tutorials/formats/ecs) and [OCSF](/docs/tutorials/formats/ocsf) lessons for schemas that standardize the fields inside each line * The [formatters](/docs/plugins/formatters) and [file output](/docs/plugins/output/file) references for every field used above # OCSF format: classes and activities Every security vendor has historically defined its own event shape: a firewall's own field names, an identity provider's own JSON structure, an EDR agent's own log lines. A detection rule or a query written against one source rarely works against another describing the exact same kind of event — a successful logon, in every case — because nothing forces the two vendors to agree on what that logon's fields should be called, or what values a status field is allowed to hold. OCSF (Open Cybersecurity Schema Framework) exists to remove that disagreement: one vendor-neutral definition per kind of security event, adopted directly by the platforms and products that choose to speak it, instead of translated after the fact. Eventum generates events shaped to OCSF's real class definitions directly from a template — the same field names, enum values, and required structure the current schema publishes — so a SIEM mapping, a detection rule, or a security data lake's ingestion pipeline gets compliant OCSF data to run against, with no identity provider or security product actually running behind it. What is OCSF? [#what-is-ocsf] [OCSF](https://ocsf.io/) was initiated by AWS and Splunk in 2022 and is developed today as a Linux Foundation project by a broader group of contributors. It defines a single schema that any security product can adopt to describe the events it produces, so that a query or detection rule written against one adopter's data works against another's without a translation layer in between. This lesson matches **OCSF schema version 1.8.0**, the current stable release published at the framework's schema browser, [schema.ocsf.io](https://schema.ocsf.io/). The schema organizes events into a fixed hierarchy. Eight top-level **categories** — Identity & Access Management, Network Activity, System Activity, and five others — each group a set of **event classes**, and every class defines its own set of possible **activities**: A class's `class_uid` is built from its category: category `3` (Identity & Access Management) contributes the leading digit, so its classes number `3001`, `3002`, and onward — Authentication is `3002`. Within a class, `activity_id` enumerates the specific stages it recognizes: for Authentication, that means Logon and Logoff plus several Kerberos-specific stages such as ticket requests and renewals. Combining the two produces `type_uid`, calculated as `class_uid × 100 + activity_id` — `300201` names the exact combination "Authentication: Logon" — which is why a consumer can filter on one integer instead of two. Every OCSF event, regardless of class, carries the same mandatory envelope: `time` (milliseconds since the Unix epoch), `metadata` (which itself requires a `product` object and a schema `version` string), `class_uid`, `category_uid`, `activity_id`, `severity_id`, and the derived `type_uid`. A specific class then promotes further fields to required on top of that envelope — Authentication also requires a `user` object, since an authentication event is meaningless without naming who it's about. Most of these identifiers pair with an optional human-readable sibling: `activity_id` with `activity_name`, `severity_id` with `severity`, `status_id` with `status`, and so on. A consumer that only recognizes the numeric taxonomy still gets a working event; a human reading the same event, or a source that has to fall back to `99` ("Other"), gets the sibling field's plain-text label instead of a bare number. Because OCSF is vendor-neutral by design, several platforms have adopted it as their normalization target rather than inventing another schema of their own. Amazon Security Lake automatically converts logs from its natively supported AWS services to OCSF before storing them — CloudTrail management events, for instance, land as API Activity, Account Change, or the Authentication class covered below, depending on what each event actually describes. Splunk, one of the framework's co-founders, supports OCSF-formatted sources in its security products, and Datadog Cloud SIEM ships an OCSF processor that normalizes incoming security logs to the same schema. Generate an OCSF Authentication event with Eventum [#generate-an-ocsf-authentication-event-with-eventum] The generator below models a corporate identity provider issuing Authentication events for network logons: a common successful logon and a rarer failed attempt, both matching OCSF's real Authentication class (`class_uid` `3002`). The templates [#the-templates] Because an OCSF event nests several objects inside one another — `user`, `src_endpoint`, `dst_endpoint`, `metadata.product` — each template below builds the event as a single Jinja mapping and serializes the whole structure at once with the built-in `tojson` filter, instead of typing nested JSON braces by hand as the flatter examples elsewhere in this course do. Fields are filled with [`module.rand`](/docs/plugins/event/template/modules), the [template](/docs/plugins/event/template) event plugin's built-in randomization module. `templates/logon-success.jinja` renders the common case — a successful network logon over Kerberos: ```jinja title="generators/ocsf-idp/templates/logon-success.jinja" {%- set user_name = module.rand.choice(["alice", "bob", "carol", "dave", "frank"]) -%} {%- set hostname = module.rand.choice(["ws-finance-14", "ws-eng-07", "ws-hr-22", "ws-sales-31"]) -%} {%- set dst_ip = module.rand.network.ip_v4_private_c() -%} {%- set src_ip = module.rand.network.ip_v4_private_b() -%} {%- set event = { "time": (timestamp.timestamp() * 1000) | int, "severity_id": 1, "severity": "Informational", "class_uid": 3002, "class_name": "Authentication", "category_uid": 3, "category_name": "Identity & Access Management", "activity_id": 1, "activity_name": "Logon", "type_uid": 300201, "type_name": "Authentication: Logon", "status_id": 1, "status": "Success", "is_mfa": true, "logon_type_id": 3, "logon_type": "Network", "auth_protocol_id": 2, "auth_protocol": "Kerberos", "user": {"name": user_name, "uid": module.rand.crypto.uuid4()}, "src_endpoint": {"ip": src_ip}, "dst_endpoint": {"hostname": hostname, "ip": dst_ip}, "metadata": { "version": "1.8.0", "product": {"name": "Fabrikam Identity Provider", "vendor_name": "Fabrikam"} } } -%} {{ event | tojson }} ``` `templates/logon-failure.jinja` renders the rarer case — the same class and activity, a failed outcome instead: ```jinja title="generators/ocsf-idp/templates/logon-failure.jinja" {%- set user_name = module.rand.choice(["alice", "bob", "carol", "dave", "frank"]) -%} {%- set hostname = module.rand.choice(["ws-finance-14", "ws-eng-07", "ws-hr-22", "ws-sales-31"]) -%} {%- set dst_ip = module.rand.network.ip_v4_private_c() -%} {%- set src_ip = module.rand.network.ip_v4_private_b() -%} {%- set reason = module.rand.choice(["USER_DOES_NOT_EXIST", "INVALID_CREDENTIALS", "ACCOUNT_DISABLED", "ACCOUNT_LOCKED_OUT", "PASSWORD_EXPIRED"]) -%} {%- set event = { "time": (timestamp.timestamp() * 1000) | int, "severity_id": 2, "severity": "Low", "class_uid": 3002, "class_name": "Authentication", "category_uid": 3, "category_name": "Identity & Access Management", "activity_id": 1, "activity_name": "Logon", "type_uid": 300201, "type_name": "Authentication: Logon", "status_id": 2, "status": "Failure", "status_detail": reason, "is_mfa": false, "logon_type_id": 3, "logon_type": "Network", "auth_protocol_id": 2, "auth_protocol": "Kerberos", "user": {"name": user_name, "uid": module.rand.crypto.uuid4()}, "src_endpoint": {"ip": src_ip}, "dst_endpoint": {"hostname": hostname, "ip": dst_ip}, "metadata": { "version": "1.8.0", "product": {"name": "Fabrikam Identity Provider", "vendor_name": "Fabrikam"} } } -%} {{ event | tojson }} ``` Both templates share `activity_id: 1` ("Logon"): the class covers a logon attempt regardless of its outcome, and `status_id`/`status` — not `activity_id` — is what actually distinguishes success from failure, which is also why `type_uid` stays `300201` in both. `logon_type_id` is `3` ("Network") because the event names a separate `src_endpoint` (where the credentials came from) and `dst_endpoint` (the host being logged into); Authentication requires at least one of `service` or `dst_endpoint` to be present, and `dst_endpoint` covers that here. Kerberos (`auth_protocol_id: 2`) is one of the protocols OCSF's own Authentication class description names as typical, and on the failure template, `status_detail` is drawn from that same class's own example failure list. Some OCSF attributes, such as `cloud` or `osint`, belong to optional profiles — reusable attribute bundles a producer opts into for a specific context, like reporting from a cloud provider or attaching threat-intelligence data. The schema browser lists a profile's fields as required wherever that profile applies, but a plain Authentication event like the one above, which opts into no profile, needs none of them. Authentication also defines further optional and recommended attributes not used above — certificate details, HTTP request context, risk scoring, MITRE ATT\&CK mappings — added the same way, as additional keys in the `event` mapping, once the pipeline consuming this data expects them. The generator config [#the-generator-config] [`mode: chance`](/docs/plugins/event/template/modes#chance) picks one of the two templates per timestamp, weighted so successful logons dominate and failures stay rare. A [cron](/docs/plugins/input/cron) input ticks once a second, and a [file](/docs/plugins/output/file) output writes each event through the [json](/docs/plugins/formatters#json) formatter with `indent: 2`, so the nested structure stays readable: ```yaml title="generators/ocsf-idp/generator.yml" input: - cron: expression: "* * * * * *" count: 1 event: template: mode: chance templates: - logon_success: template: templates/logon-success.jinja chance: 85 - logon_failure: template: templates/logon-failure.jinja chance: 15 output: - file: path: output/events.json separator: "\n" formatter: format: json indent: 2 ``` OCSF example [#ocsf-example] Running the generator above produces one Authentication event per second, mostly successful logons and, at the configured 15% chance, an occasional failure. A successful logon from an actual run: ```json title="A successful logon" { "activity_id": 1, "activity_name": "Logon", "auth_protocol": "Kerberos", "auth_protocol_id": 2, "category_name": "Identity \u0026 Access Management", "category_uid": 3, "class_name": "Authentication", "class_uid": 3002, "dst_endpoint": { "hostname": "ws-finance-14", "ip": "192.168.226.133" }, "is_mfa": true, "logon_type": "Network", "logon_type_id": 3, "metadata": { "product": { "name": "Fabrikam Identity Provider", "vendor_name": "Fabrikam" }, "version": "1.8.0" }, "severity": "Informational", "severity_id": 1, "src_endpoint": { "ip": "172.17.117.54" }, "status": "Success", "status_id": 1, "time": 1783811504000, "type_name": "Authentication: Logon", "type_uid": 300201, "user": { "name": "carol", "uid": "de3b0992-6cde-47a5-b70a-c000fff8cc36" } } ``` The `tojson` filter that serializes each event escapes `&` (along with other HTML-significant characters such as `<` and `>`) for safe embedding, so `category_name` reads `Identity \u0026 Access Management` in the output — a standard JSON escape that any parser decodes back to `&`. A failed logon from the same run: ```json title="A failed logon" { "activity_id": 1, "activity_name": "Logon", "auth_protocol": "Kerberos", "auth_protocol_id": 2, "category_name": "Identity \u0026 Access Management", "category_uid": 3, "class_name": "Authentication", "class_uid": 3002, "dst_endpoint": { "hostname": "ws-eng-07", "ip": "192.168.95.167" }, "is_mfa": false, "logon_type": "Network", "logon_type_id": 3, "metadata": { "product": { "name": "Fabrikam Identity Provider", "vendor_name": "Fabrikam" }, "version": "1.8.0" }, "severity": "Low", "severity_id": 2, "src_endpoint": { "ip": "172.27.197.60" }, "status": "Failure", "status_detail": "USER_DOES_NOT_EXIST", "status_id": 2, "time": 1783811507000, "type_name": "Authentication: Logon", "type_uid": 300201, "user": { "name": "dave", "uid": "2dc3a3c0-1050-4de4-a289-ab849e5b2f87" } } ``` Both events carry the same `class_uid`, `category_uid`, and `type_uid`, since both describe the same class and the same activity; only `severity_id`, `status_id`, and their sibling fields change to reflect the actual outcome, exactly as the schema intends. FAQ [#faq] Both are schemas for normalizing event data across sources, but they differ in structure and origin. [ECS](/docs/tutorials/formats/ecs) (Elastic Common Schema) names fields as dotted, nested paths — `event.category`, `event.action` — read primarily by the Elastic Stack. OCSF uses a numeric category/class/activity taxonomy with paired human-readable sibling fields, backed by a broader multi-vendor group under the Linux Foundation and adopted by platforms such as Amazon Security Lake independent of any single vendor's stack. Neither replaces the other universally — which one a pipeline targets usually follows from which backend or data lake receives the events. Amazon Security Lake normalizes the AWS security data it ingests to OCSF before storing it. Splunk, one of the framework's co-founders alongside AWS, supports OCSF-formatted sources in its security products, and Datadog Cloud SIEM ships a processor that maps incoming logs onto the same schema. Adoption has grown past the founding group since the framework's 2022 launch, though how completely any given product implements it still varies. No. Only the base envelope — `time`, `metadata`, `class_uid`, `category_uid`, `activity_id`, `severity_id`, `type_uid` — plus whatever a specific class promotes to required is mandatory; Authentication adds `user` on top of that envelope. Everything else on a class, recommended or optional, deepens the event without being necessary for it to validate. Related [#related] * The [formats field guide](/docs/tutorials/formats) covering every supported log and event shape * The [ECS lesson](/docs/tutorials/formats/ecs) for the other widely adopted normalization schema * The [detection testing lesson](/docs/tutorials/detection-testing) for turning Authentication-style telemetry into data that exercises Sigma rules and ATT\&CK-mapped detections * The [delivery track](/docs/tutorials/delivery) for shipping generated events toward a real data lake or SIEM * The [Okta generator](/hub/identity-okta) in the Eventum Hub for a real identity-provider source in its native shape * The [template](/docs/plugins/event/template) event plugin and [formatters](/docs/plugins/formatters) reference for every field and format used above # Suricata EVE JSON format: event_type and alerts Validating a parser built against Suricata's own alert fields, or a detection rule that keys on `signature_id` and `severity`, needs EVE JSON events shaped exactly the way a real sensor writes them — `event_type` naming which kind of record a line carries, and the alert's own fields sitting inside it exactly where a rule expects to find them. Standing up a live Suricata sensor, feeding it traffic, and waiting for one specific signature to actually fire is slow for what it returns, and a single documentation example or a captured pcap yields one alert, once — far short of the volume and the spread across event types a parser or a detection rule has to handle. Eventum renders EVE JSON lines directly from a template — the same `event_type` discriminator, the same common connection fields, the same nested `alert` object a real sensor writes — so a parser, a SIEM detection rule, or an ingestion pipeline built against Suricata's own field names gets a realistic EVE JSON stream without deploying a sensor, triggering a signature, or capturing a single pcap. What is Suricata EVE JSON [#what-is-suricata-eve-json] Suricata calls its own JSON output EVE — Extensible Event Format — and writes it as NDJSON: one complete JSON object per line, the same newline-delimited convention the [NDJSON lesson](/docs/tutorials/formats/ndjson) covers on its own terms. Every line carries an `event_type` field at its root naming what kind of record the line is — `alert`, `dns`, `flow`, and several others — plus a handful of fields common to every line regardless of type, and a type-named key holding that type's own fields. A `flow` line, for instance, carries the connection's common fields plus a single `flow` object summarizing the connection Suricata just finished tracking: ```json title="A Suricata EVE JSON flow event" {"timestamp": "2026-07-14T09:12:03.184552+0000", "flow_id": 5185340927741163, "in_iface": "eth0", "event_type": "flow", "src_ip": "192.168.1.42", "src_port": 51422, "dest_ip": "203.0.113.77", "dest_port": 443, "proto": "TCP", "app_proto": "tls", "flow": {"pkts_toserver": 42, "pkts_toclient": 51, "bytes_toserver": 6180, "bytes_toclient": 28914, "start": "2026-07-14T09:11:48.041223+0000", "end": "2026-07-14T09:12:03.184552+0000", "age": 15, "state": "closed", "reason": "shutdown", "alerted": false}} ``` An `alert` line shares the same common fields but carries an `alert` object in place of `flow`. That pairing is exclusive in this lesson's generated output — one type-named key per line — though real Suricata can depart from it, as [the alert event section below](#the-alert-event) covers. `event_type` remains the discriminator this whole format turns on: a parser that reads it first knows which type-named key is the line's primary payload, and which fields inside it are valid to read. Common fields and event types [#common-fields-and-event-types] Every EVE JSON line, regardless of `event_type`, carries the same core set of fields identifying the connection it belongs to: | Field | Description | | ----------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `timestamp` | When Suricata logged the event, to the microsecond. | | `flow_id` | A numeric identifier shared by every event belonging to the same connection — an alert, a flow summary, and a protocol record for the same connection all carry the same `flow_id`. | | `in_iface` | The network interface Suricata captured the traffic on. | | `event_type` | The discriminator this lesson teaches — names which type-specific object the line carries. | | `src_ip` / `src_port` | The connection's source address and port. | | `dest_ip` / `dest_port` | The connection's destination address and port. | | `proto` | The transport protocol — `TCP`, `UDP`, and similar. | Two more fields turn up on most lines, but not all. `app_proto` names the application-layer protocol Suricata identified on the connection — `http`, `tls`, `dns`, and so on — once it has seen enough packets to tell. `community_id` is opt-in, off by default: a hash of the connection's five-tuple that lets Suricata and another tool such as Zeek write the same identifier for the same connection, so records from both sides of a pipeline can be correlated by that one value. `event_type` is what actually tells two lines apart. Suricata decodes several dozen protocols in total; the values a general-purpose monitored network produces most often are: | `event_type` | What the line records | | ------------ | ---------------------------------------------------------------------------- | | `alert` | A signature matched — see [the alert event section below](#the-alert-event). | | `flow` | Summary statistics for one connection, logged once it closes. | | `dns` | A DNS query or the answer to one. | | `http` | One HTTP transaction. | | `tls` | A TLS handshake, including the negotiated version and certificate details. | | `ssh` | An SSH handshake, including client and server software banners. | | `smtp` | One SMTP transaction. | | `dhcp` | A DHCP lease transaction. | | `fileinfo` | Metadata about a file transferred over a tracked protocol. | | `anomaly` | A protocol-decoding irregularity Suricata's parsers flagged. | | `stats` | Suricata's own periodic engine counters, unrelated to any single connection. | Suricata decodes well over a dozen further protocols — FTP, SMB, Kerberos, RDP, MQTT, and QUIC among them — each contributing its own `event_type` value the same way. This lesson covers the values a general-purpose monitored network produces most often; [Suricata's own EVE JSON format reference](https://docs.suricata.io/en/latest/output/eve/eve-json-format.html) documents the rest. Every value shares the same envelope: the common fields above, plus a type-named key holding that type's own fields. That single-key rule holds for every `event_type` in the table above, not only `alert` — real Suricata's extended logging is the exception, covered next. The alert event [#the-alert-event] An `alert` event is what Suricata writes when a signature matches. Five fields carry the core of the detection: | Field | Description | | -------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `action` | Whether Suricata allowed the traffic through or blocked it — `allowed` unless a matching rule uses the `drop` action and Suricata is running inline as an IPS; most sensors run passively, so `allowed` is what the large majority of alerts report regardless of how serious the detection is. | | `signature` | The rule's own message text, naming what it detects. | | `signature_id` | The rule's SID — the number that uniquely identifies it within the ruleset it came from: ET Open (Emerging Threats' free ruleset), ET Pro, or a local custom rule. | | `category` | The rule's classification, e.g. `Attempted Administrator Privilege Gain` or `Potentially Bad Traffic` — one of the classifications a ruleset's `classification.config` defines. | | `severity` | The classification's priority — `1` is the highest priority, `3` the lowest, inherited from the same `classification.config` entry as `category`. | Three more fields round out the object without being part of that core five: `gid` groups which rule-matching engine raised the alert — `1` for Suricata's standard signature engine, the value on virtually every alert; `rev` is the signature's revision number, incremented each time the ruleset publishes an update to the same SID; and `metadata` is a nested object of vendor-defined tags a rule can carry — `signature_severity`, `mitre_attack`, `cve`, and similar keys, each holding an array of values, present only on rules whose author supplied them. ```json title="A Suricata alert event (its alert object adapted from Suricata's own EVE JSON format documentation)" { "timestamp": "2026-07-14T09:15:11.442018+0000", "flow_id": 2071558430912467, "in_iface": "eth0", "event_type": "alert", "src_ip": "192.168.1.42", "src_port": 51882, "dest_ip": "203.0.113.77", "dest_port": 80, "proto": "TCP", "app_proto": "http", "alert": { "action": "allowed", "gid": 1, "signature_id": 2024056, "rev": 4, "signature": "ET MALWARE Win32/CryptFile2 / Revenge Ransomware Checkin M3", "category": "Malware Command and Control Activity Detected", "severity": 1 } } ``` This lesson's `alert` events carry only the fields above. A real sensor can carry more: turning on extended alert logging (the `metadata` option under the `alert` entry in `eve-log`, covering `app-layer` and `flow`) makes an alert line also carry the triggering protocol's own object — `http`, `tls`, and similar — plus a `flow` object, alongside `alert`. A parser that assumes an alert line never carries anything but `alert` will break on a sensor configured that way. Generate EVE JSON with Eventum [#generate-eve-json-with-eventum] The generator below models a monitored network's outbound traffic — mostly ordinary flow and DNS activity, an ET Open-style signature firing on the rare connection that looks like it doesn't belong. Write the EVE JSON templates [#write-the-eve-json-templates] Fields are filled with [`module.rand`](/docs/plugins/event/template/modules), the [template](/docs/plugins/event/template) event plugin's built-in randomization module. Each template builds its event as a single Jinja mapping and serializes it in one step with the built-in `tojson` filter, the same technique the [ECS lesson](/docs/tutorials/formats/ecs) uses for its own nested event shape. `templates/alert.json.jinja` renders the event this lesson leads with — a signature match against outbound traffic, drawn from a small pool of ET Open-style signatures spanning the severity range: ```jinja title="generators/suricata-eve/templates/alert.json.jinja" {%- set signatures = [ { "sid": 2034647, "gid": 1, "rev": 2, "signature": "ET EXPLOIT Apache Log4j RCE Attempt - 2021-44228 (CVE-2021-44228)", "category": "Attempted Administrator Privilege Gain", "severity": 1, "metadata": { "signature_severity": ["Critical"], "attack_target": ["Server"], "mitre_tactic_id": ["TA0001"], "mitre_technique_id": ["T1190"] } }, { "sid": 2018358, "gid": 1, "rev": 10, "signature": "ET HUNTING GENERIC SUSPICIOUS POST to Dotted Quad with Fake Browser 1", "category": "Potentially Bad Traffic", "severity": 2, "metadata": { "signature_severity": ["Major"], "attack_target": ["Client_Endpoint"] } }, { "sid": 2013028, "gid": 1, "rev": 12, "signature": "ET POLICY curl User-Agent Outbound", "category": "Not Suspicious Traffic", "severity": 3, "metadata": { "signature_severity": ["Informational"], "attack_target": ["Client_Endpoint"] } } ] -%} {%- set sig = module.rand.choice(signatures) -%} {%- set src_ip = params.internal_subnet ~ "." ~ module.rand.number.integer(2, 254) -%} {%- set src_port = module.rand.number.integer(1024, 65535) -%} {%- set dest_ip = module.rand.network.ip_v4_public() -%} {%- set dest_port = module.rand.weighted_choice([80, 443, 8080], [50, 35, 15]) -%} {%- set action = module.rand.weighted_choice(["allowed", "blocked"], [90, 10]) -%} {%- set flow_id = module.rand.number.integer(1000000000000000, 9999999999999999) -%} {%- set cid_input = (src_ip ~ ":" ~ (src_port | string) ~ "<>" ~ dest_ip ~ ":" ~ (dest_port | string) ~ "/tcp").encode() -%} {%- set community_id = "1:" ~ module.base64.b64encode(module.hashlib.sha1(cid_input).digest()).decode() -%} {%- set event = { "timestamp": timestamp.strftime('%Y-%m-%dT%H:%M:%S.%f') ~ "+0000", "flow_id": flow_id, "in_iface": params.interface, "event_type": "alert", "src_ip": src_ip, "src_port": src_port, "dest_ip": dest_ip, "dest_port": dest_port, "proto": "TCP", "community_id": community_id, "app_proto": "http", "alert": { "action": action, "gid": sig['gid'], "signature_id": sig['sid'], "rev": sig['rev'], "signature": sig['signature'], "category": sig['category'], "severity": sig['severity'], "metadata": sig['metadata'] } } -%} {{ event | tojson }} ``` `action` reads `allowed` for the large majority of alerts here, matching Suricata's own default outside inline IPS mode, and `blocked` only on the rarer draw — this sensor treats a handful of its signatures as inline drop rules. `community_id` is computed the same way the [security-suricata Hub generator](/hub/security-suricata) computes it: a SHA-1 hash of the connection's tuple, formatted with the `1:` version prefix a real Community ID value uses — enough to give the field its real shape without reimplementing Suricata's own binary hashing algorithm. `templates/dns.json.jinja` renders the second event type — a DNS answer for one of a small pool of domains, `NXDOMAIN` on the rarer lookup that doesn't resolve: ```jinja title="generators/suricata-eve/templates/dns.json.jinja" {%- set domains = [ "www.suricata.io", "update.example.com", "api.example.net", "cdn.example.org", "mail.example.com" ] -%} {%- set domain = module.rand.choice(domains) -%} {%- set src_ip = params.internal_subnet ~ "." ~ module.rand.number.integer(2, 254) -%} {%- set src_port = module.rand.number.integer(1024, 65535) -%} {%- set dns_id = module.rand.number.integer(1, 65535) -%} {%- set found = module.rand.chance(0.9) -%} {%- set flow_id = module.rand.number.integer(1000000000000000, 9999999999999999) -%} {%- set cid_input = (src_ip ~ ":" ~ (src_port | string) ~ "<>" ~ params.dns_server_ip ~ ":53/udp").encode() -%} {%- set community_id = "1:" ~ module.base64.b64encode(module.hashlib.sha1(cid_input).digest()).decode() -%} {%- set dns = { "version": 3, "type": "answer", "id": dns_id, "flags": "8180", "qr": true, "rd": true, "ra": true, "rcode": "NOERROR" if found else "NXDOMAIN", "queries": [{"rrname": domain, "rrtype": "A"}] } -%} {%- if found -%} {%- do dns.update({"answers": [{"rrname": domain, "rrtype": "A", "ttl": module.rand.number.integer(30, 3600), "rdata": module.rand.network.ip_v4_public()}]}) -%} {%- endif -%} {%- set event = { "timestamp": timestamp.strftime('%Y-%m-%dT%H:%M:%S.%f') ~ "+0000", "flow_id": flow_id, "in_iface": params.interface, "event_type": "dns", "src_ip": src_ip, "src_port": src_port, "dest_ip": params.dns_server_ip, "dest_port": 53, "proto": "UDP", "community_id": community_id, "app_proto": "dns", "dns": dns } -%} {{ event | tojson }} ``` `dns.type` reads `answer` here — this template renders only the answer side of a DNS transaction, the richer of the two since it carries both the original query and the resolved (or unresolved) result. A real sensor also logs a separate `request` event when the query first goes out, sharing the same `flow_id` as its answer. `dns.version` reads `3` — Suricata 8.0 unified the DNS event shape used across `dns` events and the `dns` object nested inside alerts, naming that query-side event `request`. Older Suricata versions log a flatter shape instead, with `rrname` and `rrtype` sitting directly on the `dns` object and the query-side event named `query` rather than `request`. Suricata's own documentation is less consistent about the answer side's name at version 3 — the FAQ below covers what changed and what a parser should actually check for. `templates/flow.json.jinja` renders the third event type — the summary Suricata logs once a connection closes, `start` and `end` bracketing the same span `age` reports in seconds: ```jinja title="generators/suricata-eve/templates/flow.json.jinja" {%- set profiles = [ {"proto": "http", "port": 80}, {"proto": "tls", "port": 443}, {"proto": "ssh", "port": 22} ] -%} {%- set profile = module.rand.weighted_choice(profiles, [60, 30, 10]) -%} {%- set app_proto = profile['proto'] -%} {%- set dest_port = profile['port'] -%} {%- set src_ip = params.internal_subnet ~ "." ~ module.rand.number.integer(2, 254) -%} {%- set src_port = module.rand.number.integer(1024, 65535) -%} {%- set dest_ip = module.rand.network.ip_v4_public() -%} {%- set age = module.rand.number.integer(1, 300) -%} {%- set start = timestamp - module.datetime.timedelta(seconds=age) -%} {%- set flow_id = module.rand.number.integer(1000000000000000, 9999999999999999) -%} {%- set cid_input = (src_ip ~ ":" ~ (src_port | string) ~ "<>" ~ dest_ip ~ ":" ~ (dest_port | string) ~ "/tcp").encode() -%} {%- set community_id = "1:" ~ module.base64.b64encode(module.hashlib.sha1(cid_input).digest()).decode() -%} {%- set event = { "timestamp": timestamp.strftime('%Y-%m-%dT%H:%M:%S.%f') ~ "+0000", "flow_id": flow_id, "in_iface": params.interface, "event_type": "flow", "src_ip": src_ip, "src_port": src_port, "dest_ip": dest_ip, "dest_port": dest_port, "proto": "TCP", "community_id": community_id, "app_proto": app_proto, "flow": { "pkts_toserver": module.rand.number.integer(2, 400), "pkts_toclient": module.rand.number.integer(2, 400), "bytes_toserver": module.rand.number.lognormal(7.0, 1.2) | round | int, "bytes_toclient": module.rand.number.lognormal(8.5, 1.4) | round | int, "start": start.strftime('%Y-%m-%dT%H:%M:%S.%f') ~ "+0000", "end": timestamp.strftime('%Y-%m-%dT%H:%M:%S.%f') ~ "+0000", "age": age, "state": module.rand.weighted_choice(["closed", "established"], [80, 20]), "reason": module.rand.weighted_choice(["timeout", "shutdown"], [50, 50]), "alerted": module.rand.chance(0.05) } } -%} {{ event | tojson }} ``` `flow` fields describe the connection Suricata just finished tracking, not a single packet — packet and byte counts split by direction, `state` and `reason` naming how the connection ended. None of the three templates share a `flow_id`, `src_ip`, or `community_id` with each other here — correlating an alert with the flow record for the same connection is a deeper technique than this lesson covers, not something EVE JSON's own shape requires. Configure the generator [#configure-the-generator] [`mode: chance`](/docs/plugins/event/template/modes#chance) picks between the three templates per timestamp, weighted so flow and DNS activity dominate and the alert stays rarest, as on any real monitored network — though closer together here than a production sensor's ratio, so an alert shows up quickly in a short run. A [cron](/docs/plugins/input/cron) input ticks once a second, and a [file](/docs/plugins/output/file) output writes each event to `output/eve.json` — the same filename Suricata itself writes by default — through the [json](/docs/plugins/formatters#json) formatter, one compact object per line: ```yaml title="generators/suricata-eve/generator.yml" input: - cron: expression: "* * * * * *" count: 1 event: template: mode: chance params: interface: eth0 internal_subnet: "192.168.1" dns_server_ip: "192.168.1.1" templates: - alert: template: templates/alert.json.jinja chance: 15 - dns: template: templates/dns.json.jinja chance: 40 - flow: template: templates/flow.json.jinja chance: 45 output: - file: path: output/eve.json separator: "\n" formatter: format: json ``` Leave `indent` at its default of `0` on the `json` formatter, the same rule the [NDJSON lesson](/docs/tutorials/formats/ndjson) covers in full — a pretty-printed value spreads one event across several physical lines, breaking the one-event-per-line shape `separator: "\n"` is there to produce. The result [#the-result] Running the generator in [sample mode](/docs/core/concepts/generator#sample-mode) produced a steady stream, `event_type` switching line to line exactly the way a real sensor's own `eve.json` does. Seven consecutive lines from an actual run: ```json title="output/eve.json" {"app_proto": "dns", "community_id": "1:dC/rvjgk+IK/cAhXl9gW82JZjVo=", "dest_ip": "192.168.1.1", "dest_port": 53, "dns": {"flags": "8180", "id": 42142, "qr": true, "queries": [{"rrname": "cdn.example.org", "rrtype": "A"}], "ra": true, "rcode": "NXDOMAIN", "rd": true, "type": "answer", "version": 3}, "event_type": "dns", "flow_id": 7207020877667736, "in_iface": "eth0", "proto": "UDP", "src_ip": "192.168.1.3", "src_port": 10560, "timestamp": "2026-07-17T18:49:11.000000+0000"} {"app_proto": "dns", "community_id": "1:KADPHSxsm9bhdt8447LsHDzWEKU=", "dest_ip": "192.168.1.1", "dest_port": 53, "dns": {"flags": "8180", "id": 27593, "qr": true, "queries": [{"rrname": "mail.example.com", "rrtype": "A"}], "ra": true, "rcode": "NXDOMAIN", "rd": true, "type": "answer", "version": 3}, "event_type": "dns", "flow_id": 2702901642521440, "in_iface": "eth0", "proto": "UDP", "src_ip": "192.168.1.99", "src_port": 1568, "timestamp": "2026-07-17T18:49:12.000000+0000"} {"alert": {"action": "allowed", "category": "Potentially Bad Traffic", "gid": 1, "metadata": {"attack_target": ["Client_Endpoint"], "signature_severity": ["Major"]}, "rev": 10, "severity": 2, "signature": "ET HUNTING GENERIC SUSPICIOUS POST to Dotted Quad with Fake Browser 1", "signature_id": 2018358}, "app_proto": "http", "community_id": "1:6BVTNcY4j2bsghS0UaWAAALxRA0=", "dest_ip": "142.29.169.216", "dest_port": 443, "event_type": "alert", "flow_id": 6839898014437394, "in_iface": "eth0", "proto": "TCP", "src_ip": "192.168.1.110", "src_port": 46119, "timestamp": "2026-07-17T18:49:13.000000+0000"} {"app_proto": "dns", "community_id": "1:/Bly0w0gc96i3m+Dcwi0L55HgYg=", "dest_ip": "192.168.1.1", "dest_port": 53, "dns": {"answers": [{"rdata": "159.17.190.233", "rrname": "update.example.com", "rrtype": "A", "ttl": 2595}], "flags": "8180", "id": 56411, "qr": true, "queries": [{"rrname": "update.example.com", "rrtype": "A"}], "ra": true, "rcode": "NOERROR", "rd": true, "type": "answer", "version": 3}, "event_type": "dns", "flow_id": 1172549065297974, "in_iface": "eth0", "proto": "UDP", "src_ip": "192.168.1.187", "src_port": 53560, "timestamp": "2026-07-17T18:49:14.000000+0000"} {"app_proto": "dns", "community_id": "1:9Ft+zL0dAMw6BOgp6itjVOO3llY=", "dest_ip": "192.168.1.1", "dest_port": 53, "dns": {"answers": [{"rdata": "192.125.227.241", "rrname": "update.example.com", "rrtype": "A", "ttl": 2347}], "flags": "8180", "id": 4772, "qr": true, "queries": [{"rrname": "update.example.com", "rrtype": "A"}], "ra": true, "rcode": "NOERROR", "rd": true, "type": "answer", "version": 3}, "event_type": "dns", "flow_id": 3491226459074512, "in_iface": "eth0", "proto": "UDP", "src_ip": "192.168.1.176", "src_port": 52804, "timestamp": "2026-07-17T18:49:15.000000+0000"} {"app_proto": "tls", "community_id": "1:+AbTlx0odOfbTj9Hte3BQsuoBNw=", "dest_ip": "199.218.49.47", "dest_port": 443, "event_type": "flow", "flow": {"age": 112, "alerted": false, "bytes_toclient": 242, "bytes_toserver": 2215, "end": "2026-07-17T18:49:16.000000+0000", "pkts_toclient": 283, "pkts_toserver": 312, "reason": "timeout", "start": "2026-07-17T18:47:24.000000+0000", "state": "closed"}, "flow_id": 6925782777063857, "in_iface": "eth0", "proto": "TCP", "src_ip": "192.168.1.19", "src_port": 8093, "timestamp": "2026-07-17T18:49:16.000000+0000"} {"app_proto": "dns", "community_id": "1:28AOvsUoQKKIVHcZVHFOcRw3mwk=", "dest_ip": "192.168.1.1", "dest_port": 53, "dns": {"answers": [{"rdata": "188.94.112.112", "rrname": "api.example.net", "rrtype": "A", "ttl": 745}], "flags": "8180", "id": 10348, "qr": true, "queries": [{"rrname": "api.example.net", "rrtype": "A"}], "ra": true, "rcode": "NOERROR", "rd": true, "type": "answer", "version": 3}, "event_type": "dns", "flow_id": 9727542245832573, "in_iface": "eth0", "proto": "UDP", "src_ip": "192.168.1.157", "src_port": 37779, "timestamp": "2026-07-17T18:49:17.000000+0000"} ``` `event_type` names a different record across this window. Two `dns` misses (`NXDOMAIN`) open it, then an `alert` carrying all five core fields plus `gid`, `rev`, and `metadata`. Two more `dns` answers resolve normally, a `flow` closes out a 112-second `tls` connection, and a final `dns` answer ends the window. Every line is independently valid JSON and carries exactly one of `alert`, `dns`, or `flow` here — this lesson's templates don't reproduce the extra protocol objects real extended alert logging can add to a line. A separate run's alert turned up the pairing the fields table above describes only in the abstract — the highest-severity signature in the pool, this time actually blocked: ```json title="A blocked, severity-1 alert from a separate run" {"alert": {"action": "blocked", "category": "Attempted Administrator Privilege Gain", "gid": 1, "metadata": {"attack_target": ["Server"], "mitre_tactic_id": ["TA0001"], "mitre_technique_id": ["T1190"], "signature_severity": ["Critical"]}, "rev": 2, "severity": 1, "signature": "ET EXPLOIT Apache Log4j RCE Attempt - 2021-44228 (CVE-2021-44228)", "signature_id": 2034647}, "app_proto": "http", "community_id": "1:hoCHfciYPBWQ0sAFoXchwdkb0dc=", "dest_ip": "8.167.82.46", "dest_port": 443, "event_type": "alert", "flow_id": 9928557118489585, "in_iface": "eth0", "proto": "TCP", "src_ip": "192.168.1.171", "src_port": 24445, "timestamp": "2026-07-17T18:15:44.000000+0000"} ``` `severity: 1` and `action: blocked` are independent facts about the same event here, exactly as the alert event section above explains: this connection matched the pool's most serious signature, and this sensor happens to run that one as an inline drop rule. FAQ [#faq] No — `allowed` only means Suricata didn't block the traffic, not that nothing happened. A passive IDS sensor can only ever write `allowed`, since it has no way to drop a packet in the first place; `action` reads `blocked` only when Suricata runs inline as an IPS and the specific signature that fired is also configured with the `drop` action. A signature with `severity: 1` and `action: allowed` is exactly as serious as the same signature would be if it had been blocked instead — `action` describes what the sensor did with the packet, `severity` describes how serious the detection is, and neither implies the other. Suricata's DNS logging has gone through three EVE schema versions. The original names the query-side event `type: query`, with `rrname` and `rrtype` sitting directly on the `dns` object. Version 3 — introduced in Suricata 8.0 and unified with the `dns` object nested inside alert events — renames the query-side event to `type: request`. The answer side may have renamed too: Suricata's own migration notes for 8.0 state `answer` becomes `response` at version 3, but the main EVE JSON format reference page's own version-3 example still shows `type: answer`, the value this lesson generates. Suricata's documentation disagrees with itself here, so a parser should check what its own sensor actually writes rather than assume either name. Related [#related] * The [formats field guide](/docs/tutorials/formats) covering every supported log and event shape * The [NDJSON lesson](/docs/tutorials/formats/ndjson) for the newline-delimited shape EVE JSON itself uses, independent of Suricata's own fields * The [ECS lesson](/docs/tutorials/formats/ecs) for the schema a real EVE line gets mapped into once an ingest pipeline parses it * The [detection-testing lesson](/docs/tutorials/detection-testing) for testing a Sigma rule or an ATT\&CK-mapped detection against generated telemetry, the same evidence-over-intuition approach an alert-matching rule needs * The [OpenSearch delivery lesson](/docs/tutorials/delivery/opensearch) for indexing a generated EVE JSON stream into a real cluster instead of a local file * The [formatters](/docs/plugins/formatters) and [template event plugin](/docs/plugins/event/template) references for every field and format used above * The [security-suricata](/hub/security-suricata) generator in the Eventum Hub for the downstream ECS-mapped form of this same source # syslog format: PRI, RFC 3164 vs RFC 5424 Testing a collector's parsing rules, an rsyslog or syslog-ng ingestion pipeline, or a SIEM's syslog listener requires messages shaped like what a real server, network appliance, or application actually sends. A live host can produce them, but its output arrives on its own schedule, and most environments run a mix of two incompatible message shapes at once: an older format still emitted by routers, firewalls, and most Unix daemons by default, and a newer one used by modern applications, cloud logging agents, and SIEMs that want structured metadata attached to a message. A parser built or tested against only one of them breaks silently the day a device speaking the other one gets connected. Eventum renders both shapes directly from a template — the header fields in the exact order each format defines, the PRI value computed from a real facility and severity — and delivers the result over TCP or UDP to whatever the collector listens on, so the parsing rules for either format get tested with no real device plugged in to produce them. The two syslog headers [#the-two-syslog-headers] syslog is standardized twice, and both versions remain in active use. [RFC 3164](https://datatracker.ietf.org/doc/html/rfc3164) (2001) wrote down the format BSD Unix systems — and the network hardware that copied them — had already been using informally for years. [RFC 5424](https://datatracker.ietf.org/doc/html/rfc5424) (2009) replaces it with a stricter, extensible message meant to fix that older format's loosest edges: an unspecified character encoding, a timestamp with no year or time zone, and no structured way to carry metadata beyond free text. Both wrap a single line of text in the same bracketed PRI value, but everything after it — field order, what is mandatory, what a field is allowed to contain — differs enough that a parser built for one will mis-split or reject a message from the other. Every syslog message, either version, opens with the PRI value: a decimal number in angle brackets, ``, encoding two things at once. Facility says what kind of process produced the message — the kernel, the mail system, an FTP daemon, one of eight generic slots reserved for local use — and severity says how urgent it is, from `0` (Emergency) down to `7` (Debug). The two combine through a single formula, shared by both RFCs: ```text PRI = Facility * 8 + Severity ``` | Facility | Meaning | | -------- | ---------------------------------------- | | 0 | kernel messages | | 1 | user-level messages | | 2 | mail system | | 3 | system daemons | | 4 | security/authorization messages | | 5 | messages generated internally by syslogd | | 6 | line printer subsystem | | 7 | network news subsystem | | 8 | UUCP subsystem | | 9 | clock daemon | | 10 | security/authorization messages | | 11 | FTP daemon | | 12 | NTP subsystem | | 13 | log audit | | 14 | log alert | | 15 | clock daemon | | 16-23 | local use 0 through 7 | Facilities 4 and 10, and 9 and 15, share the same description in the specification's own table — not a mistake in the table above, just two device classes the RFCs never fully separated. | Severity | Level | | -------- | ---------------------------------------- | | 0 | Emergency: system is unusable | | 1 | Alert: action must be taken immediately | | 2 | Critical: critical conditions | | 3 | Error: error conditions | | 4 | Warning: warning conditions | | 5 | Notice: normal but significant condition | | 6 | Informational: informational messages | | 7 | Debug: debug-level messages | A failed SSH login is worth flagging but nowhere near an emergency: facility `4` (security/authorization messages) and severity `4` (Warning) give `PRI = 4 * 8 + 4 = 36`, written `<36>`. Both examples below reuse this exact combination. RFC 3164 documents what's typically called BSD syslog: the format nearly every router, firewall, and long-lived Unix daemon still emits by default. A message has three parts — PRI, then a HEADER carrying just a timestamp and a hostname, then a MSG that carries everything else as free text, conventionally shaped as a TAG followed by CONTENT: ```text Mmm dd hh:mm:ss HOSTNAME TAG: CONTENT ``` TIMESTAMP uses a fixed English month abbreviation, no year, and no time zone — local time, implicitly. A day of the month under 10 is padded with a space rather than a zero, so the 1st of July is `Jul 1`, not `Jul 01`. HOSTNAME is the device's bare name, never its domain. Everything after it is the MSG part, which the specification itself leaves unstructured: convention, not the RFC, shapes its first token as a TAG of alphanumeric characters naming the process, cut off at the first character that isn't alphanumeric — commonly a `:` or a `[` opening a PID — with everything from that character onward read as CONTENT. Applied to a failed SSH login from a gateway host: ```text title="RFC 3164 (BSD) syslog message" <36>Jul 11 12:00:00 web-gw-03 sshd[8419]: Failed password for admin from 203.0.113.7 port 51422 ssh2 ``` | Part | Value | Meaning | | --------- | -------------------------------------------------------------------- | ---------------------------------------------------------------------- | | PRI | `<36>` | facility 4 (security/authorization messages) x8 + severity 4 (Warning) | | TIMESTAMP | `Jul 11 12:00:00` | `Mmm dd hh:mm:ss`, local time, no year, no zone | | HOSTNAME | `web-gw-03` | device name only, never a domain | | TAG | `sshd` | alphanumeric process name; stops at the first `[` | | CONTENT | `[8419]: Failed password for admin from 203.0.113.7 port 51422 ssh2` | PID convention plus the message text | TAG and CONTENT together are the MSG part. RFC 5424 replaces that free-text tail with a fully structured, positional header, plus an optional block of machine-parseable metadata ahead of the message: ```text VERSION TIMESTAMP HOSTNAME APP-NAME PROCID MSGID STRUCTURED-DATA MSG ``` VERSION is currently always `1`. TIMESTAMP is a full [RFC 3339](https://datatracker.ietf.org/doc/html/rfc3339) date and time — four-digit year, `T` as the date/time separator, up to six digits of sub-second precision, and an explicit offset (`Z` for UTC, or `+hh:mm` / `-hh:mm`) — a stricter, unambiguous replacement for RFC 3164's zoneless timestamp. HOSTNAME, APP-NAME, PROCID, and MSGID identify the device, the originating program, its process ID or instance, and an optional message-type identifier; the specification allows any of them, TIMESTAMP included, to be replaced with a single `-` — the NILVALUE — when the sender has nothing to put there. STRUCTURED-DATA, itself replaceable with `-` when empty, holds zero or more bracketed elements of machine-parseable metadata, each an ID followed by space-separated `key="value"` pairs: ```text [SD-ID key1="value1" key2="value2"] ``` Multiple elements concatenate directly, `[id1 ...][id2 ...]`. A literal `"`, `\`, or `]` inside a value must be escaped as `\"`, `\\`, or `\]`, since left unescaped any of the three would be read as the end of the value or the element instead. The ID commonly carries a Private Enterprise Number after an `@` — `exampleSDID@32473` in the specification's own examples — so two vendors' element names never collide. The same login failure, now as RFC 5424: ```text title="RFC 5424 syslog message" <36>1 2026-07-11T12:00:00+00:00 web-gw-03.example.com sshd 8419 - [sshAuth@32473 srcIp="203.0.113.7" user="admin" result="failed"] Failed password for admin from 203.0.113.7 port 51422 ssh2 ``` | Part | Value | Meaning | | --------------- | ------------------------------------------------------------------ | ------------------------------------------------------ | | PRI | `<36>` | same facility and severity as the 3164 version | | VERSION | `1` | fixed | | TIMESTAMP | `2026-07-11T12:00:00+00:00` | RFC 3339 profile: date, `T`, time, explicit UTC offset | | HOSTNAME | `web-gw-03.example.com` | FQDN is preferred here, unlike RFC 3164 | | APP-NAME | `sshd` | originating program | | PROCID | `8419` | OS process ID | | MSGID | `-` | NILVALUE — no message-type identifier assigned | | STRUCTURED-DATA | `[sshAuth@32473 srcIp="203.0.113.7" user="admin" result="failed"]` | one element, ID `sshAuth@32473`, three key-value pairs | | MSG | `Failed password for admin from 203.0.113.7 port 51422 ssh2` | free-text message, same content as the 3164 version | RFC 3164 is still what most infrastructure emits by default — routers, firewalls, and the syslog daemons on most Linux distributions send it unless reconfigured — so a collector aimed at a real environment has to accept it. RFC 5424 is the better target for anything new: cloud logging agents, modern application frameworks, and any SIEM integration that needs a real home for key-value metadata instead of a private convention bolted onto free text. Generate syslog with Eventum [#generate-syslog-with-eventum] The generator below simulates a gateway host's `sshd` reporting failed logins — the same underlying event, rendered as either an RFC 3164 line or an RFC 5424 line depending on which template is picked for a given timestamp. The templates [#the-templates] Both templates share the same facility and severity — `4` and `4`, the PRI `<36>` from the worked example above — computed inline rather than hardcoded, so changing either value in one place keeps the header correct. Fields are filled with [`module.rand`](/docs/plugins/event/template/modules), the [template](/docs/plugins/event/template) event plugin's built-in randomization module. `templates/auth-failure-3164.jinja` renders the legacy header. Its timestamp uses Python's `%e` day-of-month directive, which pads a single-digit day with a space exactly as RFC 3164 requires, rather than `%d`, which would pad it with a zero: ```jinja title="generators/syslog-auth/templates/auth-failure-3164.jinja" {%- set pri = 4 * 8 + 4 -%} {%- set host = "web-gw-03" -%} {%- set pid = module.rand.number.integer(1000, 65535) -%} {%- set user = module.rand.choice(["admin", "root", "deploy", "oracle"]) -%} {%- set src_ip = module.rand.network.ip_v4_public() -%} {%- set port = module.rand.number.integer(1024, 65535) -%} <{{ pri }}>{{ timestamp.strftime('%b %e %H:%M:%S') }} {{ host }} sshd[{{ pid }}]: Failed password for {{ user }} from {{ src_ip }} port {{ port }} ssh2 ``` `templates/auth-failure-5424.jinja` renders the same event with the modern header. `timestamp` is already timezone-aware, so its own `isoformat()` is already a valid RFC 3339 TIMESTAMP — no manual formatting needed: ```jinja title="generators/syslog-auth/templates/auth-failure-5424.jinja" {%- set pri = 4 * 8 + 4 -%} {%- set host = "web-gw-03.example.com" -%} {%- set pid = module.rand.number.integer(1000, 65535) -%} {%- set user = module.rand.choice(["admin", "root", "deploy", "oracle"]) -%} {%- set src_ip = module.rand.network.ip_v4_public() -%} {%- set port = module.rand.number.integer(1024, 65535) -%} <{{ pri }}>1 {{ timestamp.isoformat() }} {{ host }} sshd {{ pid }} - [sshAuth@32473 srcIp="{{ src_ip }}" user="{{ user }}" result="failed"] Failed password for {{ user }} from {{ src_ip }} port {{ port }} ssh2 ``` The generator config [#the-generator-config] [`mode: chance`](/docs/plugins/event/template/modes#chance) picks one of the two templates per timestamp, weighted so the legacy format dominates — matching a real fleet where most devices still default to RFC 3164. A [cron](/docs/plugins/input/cron) input ticks once a second, and a [udp](/docs/plugins/output/udp) output ships each line to the collector's syslog port: ```yaml title="generators/syslog-auth/generator.yml" input: - cron: expression: "* * * * * *" count: 1 event: template: mode: chance templates: - auth_failure_3164: template: templates/auth-failure-3164.jinja chance: 70 - auth_failure_5424: template: templates/auth-failure-5424.jinja chance: 30 output: - udp: host: siem.example.com port: 514 separator: "\n" ``` For reliable, ordered delivery instead of UDP's best-effort datagrams, replace `udp` with [`tcp`](/docs/plugins/output/tcp) and keep the same `host` and `port` — most rsyslog and syslog-ng installations listen on both by default. To inspect lines locally before pointing the generator at a real collector, replace the output with [`file`](/docs/plugins/output/file) or [`stdout: {}`](/docs/plugins/output/stdout). The result [#the-result] Running the generator above produces a mix of both formats, each line independently valid: ```text <36>Jul 11 13:39:16 web-gw-03 sshd[10666]: Failed password for admin from 13.162.114.145 port 43966 ssh2 ``` ```text <36>1 2026-07-11T13:39:22+00:00 web-gw-03.example.com sshd 15350 - [sshAuth@32473 srcIp="192.1.17.245" user="root" result="failed"] Failed password for root from 192.1.17.245 port 35952 ssh2 ``` Both carry the same PRI, `<36>` — facility `4` (security/authorization messages) x8 plus severity `4` (Warning) — because both describe the same kind of event; only the header shape around it differs. The first has no year or time zone and a bare hostname, true to RFC 3164. The second carries a version, an offset-qualified timestamp, an FQDN, and a structured-data element repeating the source IP and username the message text already states in prose — redundant here to show the syntax, but in a real integration this is exactly where fields a receiver should parse without scanning free text belong. FAQ [#faq] RFC 5424 for anything new — it is unambiguous, timezone-aware, and gives structured metadata a real place to live instead of being crammed into free text. Reach for RFC 3164 only when the receiving collector or a downstream parser expects it specifically, which today mostly means older network hardware and its matching monitoring stack. The two are not equally tolerant of a mismatch: a strict RFC 5424 parser rejects a 3164 message outright, while a 3164-only parser accepts a 5424 message but mis-splits its fields, since RFC 3164 places no real constraint on what follows the PRI value. Point the generator's output at UDP port 514 (or TCP, if the collector's input module is configured for it) on the host running rsyslog or syslog-ng, exactly as the [udp](/docs/plugins/output/udp) example above does. rsyslog accepts either format on the same listener without extra configuration: its default parser chain tries the RFC 5424 parser first, looking for the sequence `1` immediately after the PRI value, and falls back to the RFC 3164 parser — which accepts nearly any text as a valid message — the moment that check fails. syslog-ng also reads both, though its default source driver usually separates them by port rather than sniffing content the way rsyslog does; check the driver configuration if a collector needs to be pointed at a specific listener per format. 514 is the traditional port for plain-text syslog. UDP/514 is a formal IANA registration ([RFC 5426](https://datatracker.ietf.org/doc/html/rfc5426)); TCP/514 is convention only — [RFC 6587](https://datatracker.ietf.org/doc/html/rfc6587), which governs syslog over TCP, explicitly declines to assign it a port and notes that 514/TCP is actually allocated to the unrelated Shell protocol, but nearly every collector listens there anyway. 6514 is [RFC 5425](https://datatracker.ietf.org/doc/html/rfc5425)'s registered port for syslog over TLS, used when the transport itself needs to be encrypted and authenticated — regardless of whether RFC 3164 or RFC 5424 rides inside it. Eventum's [tcp](/docs/plugins/output/tcp) output supports `ssl` and certificate options for that case; [udp](/docs/plugins/output/udp) has no TLS equivalent, since UDP carries no connection to secure. Related [#related] * The [formats field guide](/docs/tutorials/formats) covering every supported log and event shape * The [syslog over TCP/UDP delivery lesson](/docs/tutorials/delivery/syslog-transport) for the transport, framing, and TLS choices behind actually shipping these lines to a collector * The [CEF lesson](/docs/tutorials/formats/cef) for wrapping a CEF line inside a syslog header * The [LEEF lesson](/docs/tutorials/formats/leef) for wrapping a LEEF line inside a syslog header * The [detection testing lesson](/docs/tutorials/detection-testing) for generating attack-shaped telemetry to test Sigma rules # Windows Event Log & Sysmon Testing a detection rule, a log parser, or a SIEM pipeline against Windows telemetry usually means one of two options: point the tool at a production domain controller, or pull a static EVTX sample from a public forensics repository. The first risks the environment the tool is meant to protect. The second is frozen at capture time, rarely matches the target environment's naming conventions or host inventory, and is the same file every other team already downloaded from the same repository. Eventum generates fresh Windows Event Log and Sysmon events on demand — parameterized, reproducible, and shaped to the traffic curve the pipeline expects — without installing anything on a Windows machine. What Windows Event Log looks like [#what-windows-event-log-looks-like] Windows Event Log is the logging subsystem built into Windows since Windows Vista. Every event is stored in the binary [EVTX](https://forensics.wiki/windows_xml_event_log_\(evtx\)/) format and organized by channel: a named stream of events for a given audience. Event Viewer groups the built-in channels under Windows Logs — Application, Security, Setup, System, and Forwarded Events — while individual products get their own channel under Applications and Services Logs, as Sysmon does. Three of those channels account for most of what a pipeline ingests. Application carries events from user-mode software that is not part of the operating system itself. System carries events from OS components — drivers, services, the kernel. Security is the audit log, written exclusively by the Local Security Authority: logons and logoffs, privilege use, object access, account and group changes. It is also the channel most Windows-focused SIEM detections are built against, and the one the example below generates. Every event carries the same envelope, whatever channel or provider produced it. A `System` block identifies where the event came from: Provider (name and GUID — older tooling and the pre-Vista `.evt` format called this the Source), EventID, Version, Level (severity), Task and Opcode (sub-classification within the provider), Keywords (bit flags such as Audit Success or Audit Failure), TimeCreated, a per-channel EventRecordID, the Computer, and the Channel itself. An `EventData` block carries the payload specific to that EventID as a flat list of `Data` elements, each named by a `Name` attribute — name/value pairs, not deeply nested XML. Here is a trimmed version of a real Security-channel event, [4624 — An account was successfully logged on](https://learn.microsoft.com/en-us/previous-versions/windows/it-pro/windows-10/security/threat-protection/auditing/event-4624): ```xml 4624 2 0 12544 0 0x8020000000000000 211 Security WIN-GG82ULGC9GO S-1-5-18 WIN-GG82ULGC9GO$ WORKGROUP S-1-5-21-1377283216-344919071-3415362939-500 Administrator WIN-GG82ULGC9GO 2 Negotiate WIN-GG82ULGC9GO 127.0.0.1 ``` [Sysmon](https://learn.microsoft.com/en-us/sysinternals/downloads/sysmon) (System Monitor) is a free Sysinternals tool — a Windows system service paired with a kernel driver — that watches process, network, file, and registry activity beyond what Security auditing covers. It writes to its own channel, `Applications and Services Logs/Microsoft/Windows/Sysmon/Operational`, and is configured through an XML rule file that decides which of its roughly thirty event types to record and for which processes. The event IDs referenced most often: 1 (Process Create), 3 (Network Connection), 5 (Process Terminated), 11 (FileCreate), 13 (Registry Value Set), and 22 (DNS Query). Every process-related event carries a ProcessGuid that survives PID reuse, so a process's full lifecycle — creation, network connections, file creation, termination — stays correlated even after Windows recycles its process ID. Generate it with Eventum [#generate-it-with-eventum] Eventum does not reproduce the binary EVTX form directly — nothing downstream of Windows consumes events that way. In a real pipeline a shipper such as winlogbeat or Elastic Agent reads each event and re-emits it as an [ECS](/docs/tutorials/formats/ecs)-mapped JSON document under a `winlog.*` namespace; that shipped shape is what a SIEM actually indexes. The generator below produces that winlog/ECS JSON directly — the same shape a winlogbeat feed delivers, with no Windows host or agent to run. A generator that produces Windows Security events needs three pieces: an input that shapes when events happen, a [template](/docs/plugins/event/template) event plugin that renders one of several Windows-shaped Jinja templates per timestamp, and an output. The example below produces `4624` (logon success) and `4625` (logon failure) events on a daily curve that ramps up through the morning, peaks around midday, and tapers off overnight. The traffic pattern [#the-traffic-pattern] [time-patterns](/docs/plugins/input/time-patterns) builds a curve from four stages: an oscillator that defines a repeating period, a multiplier that sets a baseline count per period, a randomizer that adds period-to-period variance, and a spreader that places events within each period using a statistical distribution. A beta spreader with equal shape parameters concentrates events around the middle of the day and thins them toward both edges — a symmetric bell that traces a business-hours-like shape: ```yaml title="generators/winlog/patterns/business-hours.yml" label: business-hours-logons oscillator: start: "00:00:00" end: "never" period: 1 unit: days multiplier: ratio: 2000 randomizer: deviation: 0.25 direction: mixed spreader: distribution: beta parameters: a: 4 b: 4 ``` With `start` anchored to midnight and a one-day period, the equal shape parameters (`a: 4`, `b: 4`) center the day's peak at noon and taper density smoothly toward both ends — a rounded bell rather than a triangle's straight sides, so activity thins through the early morning and late evening instead of stopping at a hard cutoff. `ratio: 2000` sets about 2,000 logon-family events per day before the randomizer's ±25% variance is applied. The event templates [#the-event-templates] [`mode: chance`](/docs/plugins/event/template/modes#chance) picks one template per timestamp with weighted probability — a higher `chance` value makes a template more frequent. Logon success and logon failure share a small inline [sample](/docs/plugins/event/template/samples) of usernames and a `domain` parameter: ```jinja title="generators/winlog/templates/logon-success.jinja" {%- set username = (samples.usernames | random)[0] -%} {%- set src_ip = module.rand.network.ip_v4_private_c() -%} {%- set logon = module.rand.weighted_choice([["Network", "3"], ["Service", "5"], ["RemoteInteractive", "10"], ["Interactive", "2"]], [55, 20, 8, 5]) -%} {%- set record_id = shared.get('record_id', 1) -%} { "@timestamp": "{{ timestamp.isoformat() }}", "event": { "action": "logged-in", "category": ["authentication"], "code": "4624", "kind": "event", "module": "security", "outcome": "success", "provider": "Microsoft-Windows-Security-Auditing", "type": ["start"] }, "host": { "name": "{{ params.host }}", "os": { "type": "windows" } }, "user": { "name": "{{ username }}", "domain": "{{ params.domain }}" }, "source": { "ip": "{{ src_ip }}" }, "winlog": { "channel": "Security", "provider_name": "Microsoft-Windows-Security-Auditing", "event_id": "4624", "record_id": "{{ record_id }}", "keywords": ["Audit Success"], "logon": { "type": "{{ logon[0] }}" }, "event_data": { "TargetUserName": "{{ username }}", "TargetDomainName": "{{ params.domain }}", "LogonType": "{{ logon[1] }}", "IpAddress": "{{ src_ip }}" } } } {%- do shared.set('record_id', record_id + 1) -%} ``` Logon failure follows the same shape, using the `Status` field Windows itself sets on failed attempts — `0xC000006A` (bad password) and `0xC0000064` (bad username) account for most real-world failures, and `0xC0000072` (account disabled) covers the remainder: ```jinja title="generators/winlog/templates/logon-failure.jinja" {%- set username = (samples.usernames | random)[0] -%} {%- set src_ip = module.rand.network.ip_v4_public() -%} {%- set status = module.rand.weighted_choice(["0xC000006A", "0xC0000064", "0xC0000072"], [70, 20, 10]) -%} {%- set record_id = shared.get('record_id', 1) -%} { "@timestamp": "{{ timestamp.isoformat() }}", "event": { "action": "logon-failed", "category": ["authentication"], "code": "4625", "kind": "event", "module": "security", "outcome": "failure", "provider": "Microsoft-Windows-Security-Auditing", "type": ["start"] }, "host": { "name": "{{ params.host }}", "os": { "type": "windows" } }, "user": { "name": "{{ username }}", "domain": "{{ params.domain }}" }, "source": { "ip": "{{ src_ip }}" }, "winlog": { "channel": "Security", "provider_name": "Microsoft-Windows-Security-Auditing", "event_id": "4625", "record_id": "{{ record_id }}", "keywords": ["Audit Failure"], "event_data": { "TargetUserName": "{{ username }}", "TargetDomainName": "{{ params.domain }}", "Status": "{{ status }}", "IpAddress": "{{ src_ip }}" } } } {%- do shared.set('record_id', record_id + 1) -%} ``` The generator config [#the-generator-config] Wiring the pieces together: the pattern feeds [`time_patterns`](/docs/plugins/input/time-patterns), the two templates are weighted the same 3:1 ratio the Hub's own `windows-security` generator uses, and events are appended to a [file](/docs/plugins/output/file) as they are produced: ```yaml title="generators/winlog/generator.yml" input: - time_patterns: patterns: - patterns/business-hours.yml event: template: mode: chance params: domain: CONTOSO host: WIN-DC-01 samples: usernames: type: items source: [jsmith, ajohnson, mwilliams, kbrown, tpatel] templates: - logon_success: template: templates/logon-success.jinja chance: 150 - logon_failure: template: templates/logon-failure.jinja chance: 50 output: - file: path: output/security.jsonl formatter: format: json ``` To emit Sysmon telemetry instead of Security events, keep the same input and output and swap in Sysmon-shaped templates: set `winlog.channel` to `Microsoft-Windows-Sysmon/Operational`, use Sysmon's own EventIDs, and add a `ProcessGuid` to correlate related events. The [Eventum Hub](/hub) ships both as ready-made, production-tuned generators: [windows-security](/hub/windows-security) covers 12 correlated event types across a 120-host Active Directory fleet, and [windows-sysmon](/hub/windows-sysmon) covers 15 Sysmon event types with process-lifecycle correlation across the same fleet. The result [#the-result] Running the generator above produces one JSON object per event, in the winlog/ECS shape a shipper would emit from the underlying Security event: ```json { "@timestamp": "2026-03-04T09:41:07.203112+00:00", "event": { "action": "logged-in", "category": ["authentication"], "code": "4624", "kind": "event", "module": "security", "outcome": "success", "provider": "Microsoft-Windows-Security-Auditing", "type": ["start"] }, "host": { "name": "WIN-DC-01", "os": { "type": "windows" } }, "user": { "name": "ajohnson", "domain": "CONTOSO" }, "source": { "ip": "192.168.14.203" }, "winlog": { "channel": "Security", "provider_name": "Microsoft-Windows-Security-Auditing", "event_id": "4624", "record_id": "482", "keywords": ["Audit Success"], "logon": { "type": "Network" }, "event_data": { "TargetUserName": "ajohnson", "TargetDomainName": "CONTOSO", "LogonType": "3", "IpAddress": "192.168.14.203" } } } ``` `winlog.event_id` (mirrored in `event.code`) carries the EventID, `@timestamp` is the rendered TimeCreated, `winlog.provider_name` is the Provider, and `winlog.event_data` holds the same Data Name/value pairs — `TargetUserName`, `LogonType`, `IpAddress` — that the real EventData block would carry. The `event.*` classification (`category`, `type`, `kind`) and `host.name` are the ECS fields a shipper adds on top — the ones authentication detections and cross-host correlation actually key on. FAQ [#faq] EVTX is the binary container Windows actually writes to `C:\Windows\System32\winevt\Logs` — event records encoded as binary XML and grouped into 64 KB chunks alongside the XML templates they reference, not human-readable text. The XML shown in Event Viewer, in Microsoft's own event documentation, and in the example above is a rendering of that binary data, not the storage format itself. Most non-Windows destinations — SIEMs, Elastic Agent, Eventum — skip the binary layer entirely and work with the rendered fields as JSON, which is what the templates above produce directly; no EVTX file is created or parsed at any point. Point the output plugin at the SIEM's ingestion path instead of a file: [opensearch](/docs/plugins/output/opensearch) or [http](/docs/plugins/output/http) for Elastic or OpenSearch, [kafka](/docs/plugins/output/kafka) for a streaming pipeline, [tcp](/docs/plugins/output/tcp) or [udp](/docs/plugins/output/udp) for a syslog-style collector. The input, templates, and sample data stay the same — only the `output` block changes. For a complete, stateful build that indexes correlated Windows Security sessions into OpenSearch, see the [SIEM test data lesson](/docs/tutorials/siem-events). Related [#related] * The full [formats field guide](/docs/tutorials/formats) covering every supported log and event shape * The [detection testing lesson](/docs/tutorials/detection-testing) for exercising Sigma rules with this telemetry * The [Windows Security Event Log generator](/hub/windows-security) in the Eventum Hub # Synthetic data for event and log pipelines Testing a pipeline, tuning a detection, or demonstrating a dashboard all depend on the same thing: event and log data that behaves like production traffic. The real thing is rarely available — production systems are off-limits to test against, a dataset exported once goes stale as soon as traffic patterns shift, and a set of fake rows from a general-purpose data generator does not arrive the way a live system delivers events, continuously and over time. Synthetic event and log data closes that gap: generated records that describe something happening at a point in time, produced on demand, in the format a real pipeline expects. What synthetic event and log data is [#what-synthetic-event-and-log-data-is] A login, an HTTP request, a sensor reading, a security alert — each is a record of something happening at a specific point in time, not a row in a fixed table of unrelated samples. Three traits set that apart from a flat export out of a general-purpose data generator such as Faker or Mockaroo: * **Time-aware** — each record carries a timestamp that reflects when it occurred, and the sequence of timestamps follows a pattern, such as a steady rate or a realistic daily curve, instead of being stamped all at once when a file is written. * **Continuous** — a generator keeps producing new records for as long as it runs, at a pace you control, rather than producing one export and stopping. * **Parameterizable** — the format, volume, and content of the data are configured to match a specific schema and rate, rather than fixed to a generic table of sample values. A synthetic event or log feed can stand in for a live source; a static export can only stand in for a data structure. Where it is used [#where-it-is-used] Different teams use synthetic event and log data for different reasons, though the underlying data is the same kind of thing. A data engineer needs realistic traffic to test a pipeline before real data exists; an SRE, sustained load to see how ingestion and storage hold up under pressure. A detection engineer needs telemetry shaped like an attack, without running one against a live system — a backend developer, just a believable set of records to seed a staging database. What changes across these cases is the format, the destination, and the volume, not the generator underneath. How Eventum generates it [#how-eventum-generates-it] Eventum generates event and log data through a [three-stage pipeline](/docs/core/concepts/generator#the-three-stage-pipeline): an input stage decides when each record occurs, an event stage decides what it contains, and an output stage delivers it to a destination — a file, a database, a message queue, or any HTTP endpoint you configure. Related [#related] * Continue to [log and event formats](/docs/tutorials/formats) — the shape a record takes on the wire. * Then [making synthetic data realistic](/docs/tutorials/realism) — timing, sessions, and value distributions that mimic production traffic. * [Getting started](/docs) — install Eventum and generate your first events. * Ready-made generators for common data sources in the [Eventum Hub](/hub). # Logs vs metrics vs events A ticket asks for the metrics for the checkout service, and that request alone could mean a log line written each time an order fails, a number sampled every few seconds, or a structured record carrying a dozen named fields — a different thing to generate, a different destination, a different volume — three problems wearing one word. *Log*, *metric*, and *event* get used as if they name the same kind of thing, but a log store, a time-series database, and an event index each expect a specific shape, and a feed built for one doesn't load cleanly into another. Choosing what to generate and where to send it starts with knowing which of the three you need, before a single template gets written. Logs, metrics, and events: three shapes of telemetry [#logs-metrics-and-events-three-shapes-of-telemetry] Three terms that get used interchangeably, but name three specific things: * A **log** is a timestamped textual record of a discrete occurrence — a line of text describing something that happened, written to be read in order, top to bottom. * A **metric** is a numeric measurement of a system, sampled or aggregated over time — a single number taken at an instant or rolled up over a window, not a description of an occurrence. * An **event** is a structured record of a discrete occurrence carrying named fields — the same occurrence a log describes, kept as data instead of folded into a sentence. Each still describes a single occurrence — what differs is the shape it's given and where that shape ends up living. How they differ [#how-they-differ] The differences show up concretely in what each one is, where it lives, and how it gets read back: | Signal | Shape | Typical example | Where it's stored | How it's queried | | ------- | ------------------------------------------------- | ---------------------------------------------------------------------------- | ------------------------------------------------------ | ----------------------------------------------------------------------- | | Logs | A timestamped line of text | `2026-07-11T17:44:01+00:00 ERROR order 39402 failed: payment declined` | A log store — a file, or a log-oriented search backend | Full-text or field search, filtered by a time range | | Metrics | A timestamped number, often pre-aggregated | `cpu.usage{host="web-01"} 0.72` | A time-series database | Range queries and aggregation over a window — average, rate, percentile | | Events | A timestamped structured record with named fields | `{"event": "order_failed", "order_id": 39402, "reason": "payment declined"}` | An event stream or a document index | Field-level filters and aggregations against the structured payload | Every row above starts with the same word, timestamped, because log, metric, and event aren't independent categories: a log is an event rendered as a line of text, a metric is an event whose payload is a measurement, and an event is the general shape the other two specialize from — one suited to being read on its own, the other to being measured and compared across thousands of samples. Where events fit as the common abstraction [#where-events-fit-as-the-common-abstraction] Eventum's [three-stage pipeline](/docs/core/concepts/generator#the-three-stage-pipeline) produces events: the input stage decides when one occurs, the event stage decides what it contains, and the output stage decides where it goes. Nothing about that pipeline is specific to a log or a metric — the event stage renders whatever a [template](/docs/plugins/event/template) defines, and the same mechanism that fills in a JSON object with named fields renders one sentence of plain text or a single labeled number, depending only on what the template writes and which [formatter](/docs/plugins/formatters) serializes the result on the way out. A log line and a metric reading built with Eventum are the same event, shaped differently by the template — one as a sentence, one as a labeled number. The template and the formatter make that call; nothing else in the pipeline does. Generate each with Eventum [#generate-each-with-eventum] The pipeline underneath is identical for all three; only the template and the formatter change. Each fragment below uses the [template](/docs/plugins/event/template) event plugin and is illustrative rather than a full project — the linked lesson after each one builds the complete generator. A plain log line [#a-plain-log-line] A one-line, human-readable sentence with the variable data folded directly into it — the shape a service typically writes to its own console or log file: ```jinja title="templates/order.jinja" {%- set order_id = module.rand.number.integer(10000, 99999) -%} {%- set user = module.rand.choice(["alice", "bob", "carol"]) -%} {%- set duration_ms = module.rand.number.integer(80, 400) -%} {{ timestamp.isoformat() }} INFO order {{ order_id }} processed for {{ user }} in {{ duration_ms }}ms ``` Rendered through the [file](/docs/plugins/output/file) output's default `plain` formatter, a line reads: ```text 2026-07-11T17:44:01+00:00 INFO order 39402 processed for bob in 205ms ``` `order_id`, `user`, and `duration_ms` above are flat, uniform picks — fine for this illustration, but not for a value meant to resemble production traffic. [Generate realistic fake data](/docs/tutorials/realism/values) covers drawing a duration like this one from a skewed distribution instead of a flat range. A metric reading over time [#a-metric-reading-over-time] A single numeric measurement, sampled on a fixed interval, carrying none of a log's descriptive text — only what's being measured, where, and the value itself. Eventum has no dedicated metrics format — a metric-shaped event is a regular JSON object built around a name and a value, validated the same way as any other event, through the [json](/docs/plugins/formatters#json) formatter: ```jinja title="templates/cpu-usage.jinja" {%- set previous = locals.get("value", 0.35) -%} {%- set step = module.rand.number.floating(-0.05, 0.05) -%} {%- set value = module.rand.number.clamp(previous + step, 0.0, 1.0) -%} {%- do locals.set("value", value) -%} {"metric": "cpu.usage", "host": "web-01", "value": {{ "%.2f" | format(value) }}, "timestamp": "{{ timestamp.isoformat() }}"} ``` Each reading adjusts the previous one by a small step instead of picking a fresh, unrelated number, so consecutive samples drift the way a real metric does: ```json {"metric": "cpu.usage", "host": "web-01", "value": 0.36, "timestamp": "2026-07-11T17:44:00+00:00"} {"metric": "cpu.usage", "host": "web-01", "value": 0.33, "timestamp": "2026-07-11T17:44:05+00:00"} ``` [IoT sensor telemetry](/docs/tutorials/iot-telemetry) builds this same drift technique into a full generator sampling on a [timer](/docs/plugins/input/timer) input, across several sensors. A structured event [#a-structured-event] A record with several named fields, each describing one part of the same occurrence — the shape an application or an API actually emits, rather than a sentence written for a person to read: ```jinja title="templates/login.jinja" {%- set user = module.rand.choice(["alice", "bob", "carol"]) -%} {%- set method = module.rand.weighted_choice({"password": 80, "sso": 15, "mfa_backup": 5}) -%} {%- set success = module.rand.chance(0.95) -%} {"timestamp": "{{ timestamp.isoformat() }}", "event": "user_login", "user": "{{ user }}", "method": "{{ method }}", "success": {{ success | tojson }}} ``` Through the [json](/docs/plugins/formatters#json) formatter, each render is a validated JSON object: ```json {"timestamp": "2026-07-11T17:44:01+00:00", "event": "user_login", "user": "bob", "method": "password", "success": true} ``` [Structured logging](/docs/tutorials/foundations/structured-logging) builds this same technique — named fields, a weighted pick between outcomes — into a full generator, using an order-processing example with a common and a rare case. FAQ [#faq] Yes — a log is what an event looks like once its fields are folded into a sentence. [Structured logging](/docs/tutorials/foundations/structured-logging) covers what's gained by keeping the fields separate instead. Eventum generates metric-shaped events — a template renders a named measurement and a value on the same schedule a real metric would be sampled, as in [the example above](#generate-each-with-eventum), validated by the same [json](/docs/plugins/formatters#json) formatter used for any other event. It is not a metrics aggregator or a time-series database: every value is generated or drifted by the template on each render, not computed as a rollup — a rate, a percentile, an average — from data already produced, and there's no counter, histogram, or query engine behind it. Traces are the third signal in the observability taxonomy: a tree of correlated spans following one request as it crosses services, used to see where time went in a call chain. Eventum's core focus is events and logs, not traces — there's no dedicated span or trace-context primitive, though nothing stops a template or the [script](/docs/plugins/event/script) event plugin from rendering trace-shaped output by hand. Related [#related] * The [Foundations](/docs/tutorials/foundations) pillar for the broader path from synthetic event data to a working pipeline * The [Structured logging](/docs/tutorials/foundations/structured-logging) lesson for turning a plain log line into named JSON fields * The [IoT test data](/docs/tutorials/iot-telemetry) tutorial for a full generator producing drifting metric readings * The [Alert simulation](/docs/tutorials/telegram-alerts) tutorial for an alert delivered as an event over HTTP to a chat * The [Generate realistic fake data](/docs/tutorials/realism/values) lesson for skewed distributions and weighted choice instead of flat random values * The [Log and event formats field guide](/docs/tutorials/formats) for the wire formats these shapes are wrapped in * Ready-made generators in the [Eventum Hub](/hub) # Streaming vs bulk: live vs sample mode Run `eventum generate` against a fresh generator config with every default left in place, and the command does not return. Nothing prints, no file fills up, and checking back a minute later still finds no finished dataset — the generator is doing exactly what it was configured to do: waiting for each timestamp's real moment on the wall clock before producing it, one event at a time, the way a live system would. Add a single flag to the same command, and it finishes in under a second with a complete file on disk. Those are the two ways any Eventum generator can produce test data. Streaming keeps producing for as long as it runs, pacing every event to the clock so the feed behaves like a live source. Bulk ignores that pacing, produces a fixed amount of data as fast as the pipeline can move, and finishes with a dataset instead of an ongoing feed. The same generator config can do either — picking the wrong one for a given test means waiting on a stream that was never going to finish in time, or handing a live dashboard a file that stopped updating the moment it was written. Streaming vs bulk: two ways to produce test data [#streaming-vs-bulk-two-ways-to-produce-test-data] Two properties of a run are independent of what the generator actually produces: whether it keeps going, and whether it waits for the clock. Every run combines them one of two ways: * **Streaming** is continuous and paced — the generator keeps producing for as long as it runs, releasing each event at (or near) the moment its timestamp names, so the feed behaves like a live source a downstream system can tail indefinitely. * **Bulk** is finite and unpaced — the generator produces a fixed amount of data, ignores what the timestamps say about timing, and finishes as fast as the pipeline can move; the result is a dataset you keep, not an ongoing feed. Neither property depends on the event content or the destination. The same JSON event, built from the same template, can arrive as a live tap a monitoring tool watches or as a file sitting on disk before anything reads it — streaming and bulk decide which, not the shape of the data itself. When to use each [#when-to-use-each] Which one fits depends on what the data feeds into, not on how the generator is built: | Streaming | Bulk | | ------------------------------- | -------------------------- | | Real-time pipeline test | Seed a database | | SIEM or monitoring feed | Build a fixed test dataset | | Stress test at a realistic rate | Backfill a time range | | Live dashboard demo | CI fixture | "Stress test" is not one exercise. Pacing a stream at a realistic rate to see how a live pipeline holds up over time — the row above — is streaming. Firing as many requests as possible to find an endpoint's breaking point is a different exercise entirely: a bulk burst aimed at a live target instead of a file. See [API load testing](/docs/tutorials/load-testing) for the latter. Live mode and sample mode in Eventum [#live-mode-and-sample-mode-in-eventum] Eventum implements this choice as a single flag on how a generator is run, not as something configured in the generator itself. [Live mode](/docs/core/concepts/generator#live-mode-default) is the default, and it is streaming: each event is produced and delivered at the actual moment its timestamp names, synchronized to the wall clock — a tick scheduled for 12:00:05 is produced at 12:00:05, not before. [Sample mode](/docs/core/concepts/generator#sample-mode) (`--live-mode false`) is bulk: it ignores what the timestamps say about timing and produces all of them as fast as the pipeline can move, so a schedule that would span a week in live mode completes in however long rendering and writing every event actually takes. ```bash # Streaming - live mode, the default eventum generate --path generator.yml --id my-gen # Bulk - sample mode eventum generate --path generator.yml --id my-gen --live-mode false ``` The same toggle exists as the `live_mode` field on a generator entry in [startup.yml](/docs/core/config/startup-yml), for running many generators through `eventum run` instead of one at a time on the command line. One more flag matters only in live mode: [`--skip-past`](/docs/core/cli/eventum-generate) defaults to `true`, so a schedule that technically began before the generator was launched doesn't dump its entire backlog the moment it starts — streaming picks up from "now," not from wherever the schedule's `start` happens to fall. Sample mode has no equivalent concern, since a bulk run was never paced to "now" in the first place. Making a run finite [#making-a-run-finite] Streaming only makes sense for an input that keeps producing. A [cron](/docs/plugins/input/cron) tick with no `end`, a [timer](/docs/plugins/input/timer) with no `repeat` limit, a [time\_patterns](/docs/plugins/input/time-patterns) oscillator with `end: "never"` — none of these ever produce a finished run on their own; left in live mode, each is, by design, still going the next time you check. Bulk needs the opposite: a run that actually stops, which is a property of the input plugin, not of sample mode itself. [linspace](/docs/plugins/input/linspace), [timestamps](/docs/plugins/input/timestamps), and [static](/docs/plugins/input/static) are finite by construction — a range, a list, or a fixed count. Only `cron` and `timer` need to be told where to stop — `cron`'s `end` field, `timer`'s `repeat` count. Left unset, those same fields are exactly what makes them run forever. Before: a per-second cron with no bound keeps producing for as long as the generator runs, in either mode: ```yaml input: - cron: expression: "* * * * * *" count: 1 ``` After, the same schedule bounded to one hour — still one event per second, but now a run that ends: ```yaml input: - cron: expression: "* * * * * *" count: 1 end: "+1h" ``` Run the bounded version in sample mode and it is a finite dataset; run it in live mode and it is a finite stream — one hour long instead of unbounded, but still paced to the clock while it lasts. Same generator, both modes [#same-generator-both-modes] Wire the bounded schedule above into a complete generator, and the flag alone decides whether the result is a dataset already sitting on disk or a feed still arriving: ```yaml title="generators/heartbeat/generator.yml" input: - cron: expression: "* * * * * *" count: 1 end: "+1h" event: template: mode: all templates: - heartbeat: template: templates/heartbeat.jinja output: - file: path: output/events.jsonl write_mode: overwrite formatter: format: json ``` ```jinja title="generators/heartbeat/templates/heartbeat.jinja" {"timestamp": "{{ timestamp.isoformat() }}", "status": "ok"} ``` Run it once in sample mode and the full hour completes immediately — a finished file, ready to load into a dev database or hand to a test as a fixture: ```bash eventum generate --path generator.yml --id seed --live-mode false ``` ```json title="output/events.jsonl — 3,600 lines, produced in under a second" {"timestamp": "2026-07-17T12:55:29+00:00", "status": "ok"} {"timestamp": "2026-07-17T12:55:30+00:00", "status": "ok"} {"timestamp": "2026-07-17T12:55:31+00:00", "status": "ok"} ... {"timestamp": "2026-07-17T13:55:26+00:00", "status": "ok"} {"timestamp": "2026-07-17T13:55:27+00:00", "status": "ok"} {"timestamp": "2026-07-17T13:55:28+00:00", "status": "ok"} ``` Run the exact same config again with no flag added — live mode is the default — and the file grows one line a second instead of appearing all at once, the way a real heartbeat would arrive at a monitoring tool tailing it. Eight real seconds in: ```json title="output/events.jsonl — after 8 real seconds" {"timestamp": "2026-07-17T12:55:42+00:00", "status": "ok"} {"timestamp": "2026-07-17T12:55:43+00:00", "status": "ok"} {"timestamp": "2026-07-17T12:55:44+00:00", "status": "ok"} {"timestamp": "2026-07-17T12:55:45+00:00", "status": "ok"} {"timestamp": "2026-07-17T12:55:46+00:00", "status": "ok"} {"timestamp": "2026-07-17T12:55:47+00:00", "status": "ok"} {"timestamp": "2026-07-17T12:55:48+00:00", "status": "ok"} ``` Same `generator.yml`, same template, same output path. What changes is only whether the run has already finished or is still going — the difference between a dataset and a feed is the flag it started with, not the config that describes it. FAQ [#faq] Same idea, different name: "batch" is what most tools call what Eventum calls bulk — see [above](#streaming-vs-bulk-two-ways-to-produce-test-data) for how the two differ from streaming. Two things need to be true: the input plugin has to actually stop — [linspace](/docs/plugins/input/linspace), [timestamps](/docs/plugins/input/timestamps), and [static](/docs/plugins/input/static) always do, `cron` and `timer` only if `end` or `repeat` is set — and the run needs to be in [sample mode](/docs/core/concepts/generator#sample-mode) (`--live-mode false`) so it completes immediately instead of taking as long in wall-clock time as the schedule spans. A finite input left in live mode still finishes eventually, just not before the last timestamp's real moment arrives. Not by default. [`--skip-past`](/docs/core/cli/eventum-generate) defaults to `true`, so live mode skips every timestamp earlier than "now" and starts from the next one still in the future. Setting `--skip-past false` replays that backlog instead — but immediately, not at the original pace, since the scheduler only ever waits for timestamps that are still ahead of the clock. Related [#related] * The [Foundations](/docs/tutorials/foundations) pillar for the broader path from synthetic event data to a working pipeline * The [Scheduling](/docs/core/concepts/scheduling) concept page for how finite and infinite inputs combine and how timestamps from multiple inputs merge * The [Stream synthetic data to your stack](/docs/tutorials/delivery) pillar for delivering a live feed to a real backend instead of a file * The [API load testing](/docs/tutorials/load-testing) lesson for a bulk burst fired at a live endpoint to find its breaking point * The [Seed a database with realistic test data](/docs/tutorials/csv-dataset) lesson for a bulk dataset built to load into a database * The [Web clickstream scenario](/docs/tutorials/web-clickstream) tutorial for a continuous session stream in live mode, inspected from a bounded sample-mode batch while building it * The [IoT test data](/docs/tutorials/iot-telemetry) tutorial for bounding a `timer` input with `repeat` to turn a live sensor stream into a finite dataset * Ready-made generators in the [Eventum Hub](/hub) # Structured logging: from plain text to JSON A service that logs a single sentence per event is simple to write and easy to read on a terminal, until someone has to answer a real question with that log: how many payment failures affected a specific customer in the last hour, or which requests exceeded a set duration. Answering that from a sentence means writing a pattern that matches its exact wording — and that pattern breaks the moment the sentence is reworded, a field moves, or a new one appears. The only contract between writer and reader is prose, and nothing enforces it. Structured logging replaces that contract with one a program can rely on: each piece of data is emitted as its own named field, in a consistent shape, instead of folded into a sentence. Eventum generates events this way directly from a template — every field rendered on its own — and its JSON formatter validates the result before delivery, so a parser, a search query, or a detection rule can be exercised against realistic structured data without a live service producing it. Structured logging vs plain text [#structured-logging-vs-plain-text] Plain-text logging composes a human-readable sentence and interpolates the event's data directly into it. The line below and the structured event after it carry the same information — the same order, the same customer, the same failure reason: ```text 2026-07-11 17:44:01 ERROR order 39402 failed for user bob: payment declined ``` ```json {"timestamp": "2026-07-11T17:44:01+00:00", "level": "ERROR", "service": "orders", "message": "order failed", "order_id": 39402, "user": "bob", "reason": "payment declined"} ``` Reading the order ID back out of the first line means matching digits that happen to follow the word "order" — a pattern tied to that exact sentence. Reading it out of the second means reading the `order_id` field directly, regardless of where other data sits in the line or how the message text happens to be worded. That difference is the entire point of structured logging: a field sits at the same name every time, so software queries it directly instead of reverse-engineering it from a sentence written for a person to read. Structured logging formats: key-value and JSON [#structured-logging-formats-key-value-and-json] A key-value line keeps the flat, single-line feel of plain text but separates each piece of data into its own `key=value` pair. JSON goes further and nests and types the data as a real object instead of flat text, so a number stays a number and a list stays a list rather than everything being a string that happens to look like one: | Shape | Example | Typical use | | ---------- | ----------------------------------------------------------------------------- | -------------------------------------------------------------------------- | | Plain text | `order 39402 failed for user bob: payment declined` | Read by a person; matched by a pattern tuned to this exact wording | | Key-value | `level=ERROR order_id=39402 user=bob reason="payment declined"` | Application logs read by tools that split on whitespace and `=` | | JSON | `{"level":"ERROR","order_id":39402,"user":"bob","reason":"payment declined"}` | Log shippers, search backends, and SIEMs that parse a JSON object natively | Fixing the shape of a log line does not fix what each field is called: two structured loggers can both emit JSON and still name the same piece of data differently — `user`, `username`, `actor.name` — so a query written against one does not match the other. A schema convention closes that gap by standardizing field names across every source that adopts it, so one query or detection rule matches all of them. [ECS](/docs/tutorials/formats/ecs) and [OCSF](/docs/tutorials/formats/ocsf) are the two most widely adopted schemas built this way: structured logging is the mechanism, a schema is the shared vocabulary layered on top of it. Rules that keep a structured log queryable [#rules-that-keep-a-structured-log-queryable] A structured log is only as useful as the discipline behind it: * **Pick one format per service, and default to JSON** — most log shippers, search backends, and SIEMs already parse it, so nothing extra needs configuring downstream. * **Keep a field's name and type stable** — a field that is a string in one event and a number in the next breaks any query or detection rule written against it. * **Keep variable data out of the message text** — a `message` field should read the same regardless of which order or customer triggered the event; the order ID, the customer, and the failure reason belong in their own fields. * **Attach the fields that make one event usable on its own** — a timestamp, a severity level, and whatever identifies the source, such as a service name or host. * **Adopt a field-naming schema, such as ECS or OCSF, instead of inventing names per service** — the payoff is a query or detection rule that already works across every source that follows the same schema. Generate structured logs with Eventum [#generate-structured-logs-with-eventum] The generator below simulates an order-processing service logging both outcomes as structured JSON: a common success case and a rarer failure, rendered from two separate templates, each carrying only the fields its event needs. The templates [#the-templates] Fields are filled with [module.rand](/docs/plugins/event/template/modules), the [template](/docs/plugins/event/template) event plugin's built-in randomization module. `templates/order-processed.jinja` renders the common case, level `INFO`: ```jinja title="generators/orders-service/templates/order-processed.jinja" {%- set order_id = module.rand.number.integer(10000, 99999) -%} {%- set user = module.rand.choice(["alice", "bob", "carol", "dave"]) -%} {%- set duration_ms = module.rand.number.integer(80, 400) -%} {"timestamp": "{{ timestamp.isoformat() }}", "level": "INFO", "service": "orders", "message": "order processed", "order_id": {{ order_id }}, "user": "{{ user }}", "duration_ms": {{ duration_ms }}} ``` `templates/order-failed.jinja` renders the rarer failure, level `ERROR`, with a `reason` field in place of `duration_ms` — a failure and a success do not describe themselves with the same data, so each template only carries the fields it needs: ```jinja title="generators/orders-service/templates/order-failed.jinja" {%- set order_id = module.rand.number.integer(10000, 99999) -%} {%- set user = module.rand.choice(["alice", "bob", "carol", "dave"]) -%} {%- set reason = module.rand.choice(["payment declined", "inventory unavailable", "address validation failed"]) -%} {"timestamp": "{{ timestamp.isoformat() }}", "level": "ERROR", "service": "orders", "message": "order failed", "order_id": {{ order_id }}, "user": "{{ user }}", "reason": "{{ reason }}"} ``` The generator config [#the-generator-config] [mode: chance](/docs/plugins/event/template/modes#chance) picks one of the two templates per timestamp, weighted so successes dominate and failures stay rare. A [cron](/docs/plugins/input/cron) input ticks once a second, and a [file](/docs/plugins/output/file) output writes each rendered event to a local file through the [json](/docs/plugins/formatters#json) formatter, which validates every line as JSON before it is written: ```yaml title="generators/orders-service/generator.yml" input: - cron: expression: "* * * * * *" count: 1 event: template: mode: chance templates: - order_processed: template: templates/order-processed.jinja chance: 90 - order_failed: template: templates/order-failed.jinja chance: 10 output: - file: path: output/events.jsonl formatter: format: json ``` To deliver these events to a real backend instead of a local file, replace `file` with [opensearch](/docs/plugins/output/opensearch) or [kafka](/docs/plugins/output/kafka) and keep the same `json` formatter. The result [#the-result] Running the generator above produces one structured event per second, mostly `order_processed` and, at the configured 10% chance, an occasional `order_failed` — three consecutive lines from an actual run: ```json title="output/events.jsonl" {"timestamp": "2026-07-11T17:44:00+00:00", "level": "INFO", "service": "orders", "message": "order processed", "order_id": 74824, "user": "carol", "duration_ms": 205} {"timestamp": "2026-07-11T17:44:01+00:00", "level": "ERROR", "service": "orders", "message": "order failed", "order_id": 39402, "user": "bob", "reason": "payment declined"} {"timestamp": "2026-07-11T17:44:02+00:00", "level": "INFO", "service": "orders", "message": "order processed", "order_id": 91759, "user": "dave", "duration_ms": 148} ``` Every line is a complete, independently valid JSON object — the `json` formatter checked each one before writing it, so a malformed line would have been dropped rather than corrupt the file. Filtering this file for `"level":"ERROR"` or `"user":"bob"` is a direct field match, the same query a search backend or a detection rule runs once the data is ingested, with no pattern tuned to a particular sentence's wording required. FAQ [#faq] JSON is one common structured logging format, not the only one. What makes logging "structured" is that each piece of data is its own named field rather than text folded into a sentence; JSON expresses that as a nested, typed object, and a key-value line (`level=ERROR order_id=39402 user=bob`) expresses the same idea as flat text. JSON is the more common choice today because most log shippers, search backends, and SIEMs parse a JSON object directly, without a separate line grammar configured for it first. Unstructured logging composes a sentence and buries the data inside it; structured logging keeps each piece as its own field. See [above](#structured-logging-vs-plain-text) for what that costs a query written against the sentence form. No. Structured logging only requires a consistent shape, such as JSON — a service that logs `{"level": "ERROR", "order_id": 39402}` is already structured. A schema such as ECS or OCSF adds a second layer on top: a shared set of field names, so events from different sources use the same name for the same kind of data. That layer matters once more than one source needs to be queried the same way. Related [#related] * The [Foundations](/docs/tutorials/foundations) pillar for the broader path from synthetic event data to a working pipeline * The [ECS](/docs/tutorials/formats/ecs) lesson for a schema built on top of structured logging that standardizes field names across sources * The [log and event formats field guide](/docs/tutorials/formats) for the formats structured events are wrapped in on the wire * The [template](/docs/plugins/event/template) event plugin and [formatters](/docs/plugins/formatters) reference for every field and format used above * Ready-made generators that already emit structured JSON events in the [Eventum Hub](/hub) # Correlated events: shared IDs and event.sequence A generator that renders every timestamp on its own produces events that share a field by coincidence, not because they belong together. Real activity rarely works that way: a network connection opens, moves data across a handful of packets, and closes, all under one flow; a request gets exactly one response somewhere downstream; a lateral-movement attempt leaves a trail across three different hosts in the order it actually happened. None of that shows up in a stream of independently generated lines, however many fields they share, and a detection rule, a distributed trace, or a funnel query is built entirely around following the thread that ties events like these together. Eventum ties correlated events together the same way a real source does: a shared identifier, invented once and copied unchanged into every event that follows from it, plus a sequence number that orders them exactly, independent of whatever precision the timestamp itself happens to carry. Unlike a single actor moving through one fixed path, a correlated stream usually has several of these threads open at once, so the generator needs to track a whole pool of what is currently active, not just one running value. What correlated events look like [#what-correlated-events-look-like] Three properties separate a correlated stream from one where events merely happen to share a field: * **A correlation id** — one value, invented once at the start and copied unchanged into everything that follows from it, whether it gets called a flow id, a request id, a trace id, or a session id. * **Causal order** — an open precedes every transfer that follows it, and a transfer precedes the close, because that is the order the underlying activity actually happened in, not an order the generator picked for convenience. * **Sequence numbers**, where a real source publishes them, climb per host or per source, giving a second, independent way to notice a gap or a reordering beyond whatever the timestamp alone can show. This lesson's shape is deliberately different from [Modeling sessions](/docs/tutorials/realism/sessions). A session follows one actor down one fixed path, one stage at a time, with only one session ever open in that generator's shared state. A correlated stream holds many threads open at the same time — several connections mid-transfer, several requests still waiting on a reply — interleaved in whatever order the generator happens to render them, and often spanning more than one host. The mechanism needs a whole pool of active threads, not a single current one, and on every render it picks which thread to advance. Correlation identifiers and sequence numbers [#correlation-identifiers-and-sequence-numbers] A correlation id is the simplest of the three properties: one value, generated when a thread starts — a UUID is the usual choice, since a fixed-format string only has to be unique, never meaningful — and copied into every event that follows from that start, unchanged. Nothing computes it or looks it up a second time; it is set once and carried, the same way a session id is set once at login and read back on every action afterward. `event.sequence` is [ECS](/docs/tutorials/formats/ecs)'s field for exactly this kind of ordering: a number that climbs strictly per host or per source, published by the source itself so a consumer can detect a gap or a reordering that a timestamp's precision might hide. That scope matters — the count belongs to the host or source, not to a correlation id or the stream as a whole. A gateway handling six connections at once still produces one running count for itself, shared across every one of those connections, while tracking how far along any single connection is stays an entirely separate concern. `related.*` fields close the loop for search: `related.ip`, `related.hosts`, `related.user`, and `related.hash` collect every address, hostname, user, or hash appearing anywhere else in the same event, always as an array — even an event naming exactly one host still writes `related.hosts: ["gw-eu-01"]`, never a bare string. A query against `related.ip` finds a value regardless of whether it turned up in `source.ip`, `destination.ip`, or somewhere else in the document entirely. Correlate events in Eventum [#correlate-events-in-eventum] Modeling this in Eventum means keeping the pool of active threads in [shared](/docs/plugins/event/template/state) state — a dict keyed by correlation id — and deciding, on every render, whether to start a new thread or advance one already in the pool. [mode: fsm](/docs/plugins/event/template/modes#fsm) doesn't fit this shape: a state machine tracks exactly one current state, built for one sequential actor rather than many concurrent, interleaved ones. A single template under [mode: all](/docs/plugins/event/template/modes#all) that decides everything about its own render fits a pool of concurrent threads far better — exactly the alternative design [Modeling sessions](/docs/tutorials/realism/sessions) itself calls for, the moment more than one session needs to run at once. The template [#the-template] `flow.jinja` models a network flow — a connection that opens, transfers data over one or more renders, and eventually closes — across a small fleet of gateways, with several flows open at once: ```jinja title="generators/network-flows/templates/flow.jinja" {%- set hosts = ["gw-eu-01", "gw-eu-02", "gw-us-01"] -%} {%- set pool = shared.get("pool", {}) -%} {%- set open_ids = pool.keys() | list -%} {%- set pool_full = (open_ids | length) >= 6 -%} {%- set act_on_existing = open_ids and (pool_full or module.rand.chance(0.7)) -%} {%- if act_on_existing -%} {%- set flow_id = open_ids[0] if pool_full else module.rand.choice(open_ids) -%} {%- set flow = pool[flow_id] -%} {%- set host = flow["host"] -%} {%- set src_ip = flow["src_ip"] -%} {%- set dst_ip = flow["dst_ip"] -%} {%- set chunk = module.rand.number.integer(300, 4000) -%} {%- set total_bytes = flow["bytes"] + chunk -%} {%- set transfers = flow["transfers"] + 1 -%} {%- if pool_full or transfers >= 3 -%} {%- set action = "flow_close" -%} {%- do pool.pop(flow_id) -%} {%- else -%} {%- set action = "flow_transfer" -%} {%- do pool.update({flow_id: {"host": host, "src_ip": src_ip, "dst_ip": dst_ip, "transfers": transfers, "bytes": total_bytes}}) -%} {%- endif -%} {%- else -%} {%- set flow_id = module.rand.crypto.uuid4() -%} {%- set host = module.rand.choice(hosts) -%} {%- set src_ip = module.rand.network.ip_v4_private_a() -%} {%- set dst_ip = module.rand.network.ip_v4_public() -%} {%- set total_bytes = module.rand.number.integer(80, 200) -%} {%- set action = "flow_open" -%} {%- do pool.update({flow_id: {"host": host, "src_ip": src_ip, "dst_ip": dst_ip, "transfers": 0, "bytes": total_bytes}}) -%} {%- endif -%} {%- do shared.set("pool", pool) -%} {%- set seq_by_host = shared.get("seq_by_host", {}) -%} {%- set sequence = seq_by_host.get(host, 0) + 1 -%} {%- do seq_by_host.update({host: sequence}) -%} {%- do shared.set("seq_by_host", seq_by_host) -%} {%- set event = { "timestamp": timestamp.isoformat(), "flow_id": flow_id, "event": {"action": action, "sequence": sequence}, "host": {"name": host}, "source": {"ip": src_ip}, "destination": {"ip": dst_ip}, "network": {"bytes": total_bytes}, "related": {"ip": [src_ip, dst_ip], "hosts": [host]} } -%} {{ event | tojson }} ``` * **`pool`** — the dict of every flow currently in flight, keyed by `flow_id`, read out of `shared` at the top of every render and written back at the end. Each entry carries the flow's host, both addresses, how many transfers it has had, and its running byte count. * **`act_on_existing`** — true when there is at least one flow to advance and either the pool is full or a 70% roll says to work an existing flow rather than open a new one; false, and a new flow opens instead, whenever the pool is empty or the remaining 30% roll lands. * **Advancing a flow** — `module.rand.choice(open_ids)` picks which one at random, or `open_ids[0]` — the oldest entry — once the pool is full (see below). Its running byte total grows by a random chunk; once its transfer count would reach three, this render closes the flow instead of transferring again, and removes it from the pool. * **Opening a flow** — a fresh `flow_id`, host, and address pair, added to the pool with zero transfers and a small initial byte count for the handshake. * **`pool_full`** — caps the pool at six flows in flight. Once that many are open, the render always advances the oldest entry (`open_ids[0]`, since dict keys preserve insertion order) and forces it to close instead of admitting a seventh — the same rule any long-lived pool needs: cap it before it grows unbounded, applied here by closing the oldest thread rather than silently dropping it. * **`seq_by_host`** — a second, separate counter from anything tracked per flow: one running count per host, incremented on every render regardless of which flow it belongs to or which stage that flow is in. This is what becomes `event.sequence`. The same pool-in-state pattern works with [globals](/docs/plugins/event/template/state) in place of `shared` when the correlated entities span more than one generator: thread-safe, and visible to every generator in the process rather than just one. The generator config [#the-generator-config] One template, [mode: all](/docs/plugins/event/template/modes#all), and a steady per-second tick — the config supplies the cadence, and the template's own state decides what actually renders on each one: ```yaml title="generators/network-flows/generator.yml" input: - cron: expression: "* * * * * *" count: 1 event: template: mode: all templates: - flow: template: templates/flow.jinja output: - file: path: output/events.jsonl write_mode: overwrite formatter: format: json ``` The events below were produced by running this generator in [sample mode](/docs/core/concepts/generator#sample-mode) (`--live-mode false`) with a short bound added to the cron input, so forty events render at once for inspection, and with `--keep-order true` so [output writes are serialized](/docs/core/concepts/output#concurrency) and each host's `event.sequence` reads in order top-to-bottom below. A generator left running normally drops both the bound and the ordering flag, ticking indefinitely in [live mode](/docs/core/concepts/generator#live-mode-default) with output writes running concurrently again — the sequence values are always correct, but only `keep_order: true` guarantees the file's physical line order matches them. The result [#the-result] Running this generator for forty timestamps opened twelve flows, closed eight of them, and left four still transferring when the bound was reached — no two ever collided on an identifier, since each is a freshly generated UUID. A slice from early in the run shows one flow open, transfer twice, and close on one gateway, then the other two gateways each open their first flow while a second flow starts back on the first: ```json title="output/events.jsonl" {"destination": {"ip": "192.0.1.127"}, "event": {"action": "flow_open", "sequence": 1}, "flow_id": "44c711fc-182c-43e6-a7a5-c0dcb10c7932", "host": {"name": "gw-eu-02"}, "network": {"bytes": 93}, "related": {"hosts": ["gw-eu-02"], "ip": ["10.115.249.42", "192.0.1.127"]}, "source": {"ip": "10.115.249.42"}, "timestamp": "2026-07-17T14:09:47+00:00"} {"destination": {"ip": "192.0.1.127"}, "event": {"action": "flow_transfer", "sequence": 2}, "flow_id": "44c711fc-182c-43e6-a7a5-c0dcb10c7932", "host": {"name": "gw-eu-02"}, "network": {"bytes": 2241}, "related": {"hosts": ["gw-eu-02"], "ip": ["10.115.249.42", "192.0.1.127"]}, "source": {"ip": "10.115.249.42"}, "timestamp": "2026-07-17T14:09:48+00:00"} {"destination": {"ip": "192.0.1.127"}, "event": {"action": "flow_transfer", "sequence": 3}, "flow_id": "44c711fc-182c-43e6-a7a5-c0dcb10c7932", "host": {"name": "gw-eu-02"}, "network": {"bytes": 5416}, "related": {"hosts": ["gw-eu-02"], "ip": ["10.115.249.42", "192.0.1.127"]}, "source": {"ip": "10.115.249.42"}, "timestamp": "2026-07-17T14:09:49+00:00"} {"destination": {"ip": "192.0.1.127"}, "event": {"action": "flow_close", "sequence": 4}, "flow_id": "44c711fc-182c-43e6-a7a5-c0dcb10c7932", "host": {"name": "gw-eu-02"}, "network": {"bytes": 8559}, "related": {"hosts": ["gw-eu-02"], "ip": ["10.115.249.42", "192.0.1.127"]}, "source": {"ip": "10.115.249.42"}, "timestamp": "2026-07-17T14:09:50+00:00"} {"destination": {"ip": "2.236.12.0"}, "event": {"action": "flow_open", "sequence": 1}, "flow_id": "2d397e76-699d-48e9-a37f-68abce096415", "host": {"name": "gw-eu-01"}, "network": {"bytes": 91}, "related": {"hosts": ["gw-eu-01"], "ip": ["10.241.88.202", "2.236.12.0"]}, "source": {"ip": "10.241.88.202"}, "timestamp": "2026-07-17T14:09:51+00:00"} {"destination": {"ip": "112.150.73.240"}, "event": {"action": "flow_open", "sequence": 1}, "flow_id": "06db2745-ba87-4a55-a18c-2094d72350ec", "host": {"name": "gw-us-01"}, "network": {"bytes": 132}, "related": {"hosts": ["gw-us-01"], "ip": ["10.90.172.49", "112.150.73.240"]}, "source": {"ip": "10.90.172.49"}, "timestamp": "2026-07-17T14:09:52+00:00"} {"destination": {"ip": "2.236.12.0"}, "event": {"action": "flow_transfer", "sequence": 2}, "flow_id": "2d397e76-699d-48e9-a37f-68abce096415", "host": {"name": "gw-eu-01"}, "network": {"bytes": 434}, "related": {"hosts": ["gw-eu-01"], "ip": ["10.241.88.202", "2.236.12.0"]}, "source": {"ip": "10.241.88.202"}, "timestamp": "2026-07-17T14:09:53+00:00"} {"destination": {"ip": "70.156.64.89"}, "event": {"action": "flow_open", "sequence": 5}, "flow_id": "b5db8ad8-aa76-45b5-83fe-80dc1781fa44", "host": {"name": "gw-eu-02"}, "network": {"bytes": 117}, "related": {"hosts": ["gw-eu-02"], "ip": ["10.217.206.168", "70.156.64.89"]}, "source": {"ip": "10.217.206.168"}, "timestamp": "2026-07-17T14:09:54+00:00"} {"destination": {"ip": "70.156.64.89"}, "event": {"action": "flow_transfer", "sequence": 6}, "flow_id": "b5db8ad8-aa76-45b5-83fe-80dc1781fa44", "host": {"name": "gw-eu-02"}, "network": {"bytes": 3048}, "related": {"hosts": ["gw-eu-02"], "ip": ["10.217.206.168", "70.156.64.89"]}, "source": {"ip": "10.217.206.168"}, "timestamp": "2026-07-17T14:09:55+00:00"} {"destination": {"ip": "112.150.73.240"}, "event": {"action": "flow_transfer", "sequence": 2}, "flow_id": "06db2745-ba87-4a55-a18c-2094d72350ec", "host": {"name": "gw-us-01"}, "network": {"bytes": 1099}, "related": {"hosts": ["gw-us-01"], "ip": ["10.90.172.49", "112.150.73.240"]}, "source": {"ip": "10.90.172.49"}, "timestamp": "2026-07-17T14:09:56+00:00"} ``` Four different `flow_id` values interleave across these lines, each opened, continued, or closed independently of the others, yet `event.sequence` for `gw-eu-02` still climbs 1, 2, 3, 4, 5, 6 without a gap or a repeat — the count belongs to the host, not to any one flow, so it keeps incrementing across every flow passing through that host, including the second one that opens right after the first closes. `gw-eu-01` and `gw-us-01` each start their own count at 1 the moment their first flow opens, entirely independent of whatever `gw-eu-02` has already reached. Filtering the same run down to one `flow_id` shows the complete lifecycle a single connection went through — scattered across the file among other flows' lines, but tied together by the id every step repeats: ```json title="output/events.jsonl — one flow_id, filtered from the run above" {"destination": {"ip": "61.173.247.170"}, "event": {"action": "flow_open", "sequence": 5}, "flow_id": "dca4021e-6f52-476b-b4a1-6e059e5715b6", "host": {"name": "gw-us-01"}, "network": {"bytes": 182}, "related": {"hosts": ["gw-us-01"], "ip": ["10.144.221.252", "61.173.247.170"]}, "source": {"ip": "10.144.221.252"}, "timestamp": "2026-07-17T14:10:01+00:00"} {"destination": {"ip": "61.173.247.170"}, "event": {"action": "flow_transfer", "sequence": 8}, "flow_id": "dca4021e-6f52-476b-b4a1-6e059e5715b6", "host": {"name": "gw-us-01"}, "network": {"bytes": 3514}, "related": {"hosts": ["gw-us-01"], "ip": ["10.144.221.252", "61.173.247.170"]}, "source": {"ip": "10.144.221.252"}, "timestamp": "2026-07-17T14:10:09+00:00"} {"destination": {"ip": "61.173.247.170"}, "event": {"action": "flow_transfer", "sequence": 9}, "flow_id": "dca4021e-6f52-476b-b4a1-6e059e5715b6", "host": {"name": "gw-us-01"}, "network": {"bytes": 5884}, "related": {"hosts": ["gw-us-01"], "ip": ["10.144.221.252", "61.173.247.170"]}, "source": {"ip": "10.144.221.252"}, "timestamp": "2026-07-17T14:10:11+00:00"} {"destination": {"ip": "61.173.247.170"}, "event": {"action": "flow_close", "sequence": 12}, "flow_id": "dca4021e-6f52-476b-b4a1-6e059e5715b6", "host": {"name": "gw-us-01"}, "network": {"bytes": 8505}, "related": {"hosts": ["gw-us-01"], "ip": ["10.144.221.252", "61.173.247.170"]}, "source": {"ip": "10.144.221.252"}, "timestamp": "2026-07-17T14:10:14+00:00"} ``` Every line carries `dca4021e-6f52-476b-b4a1-6e059e5715b6`, `network.bytes` climbs from 182 to 8,505 as the running total grows at every step from open through close, and this host's `event.sequence` advances 5, 8, 9, 12 for this flow specifically — the gaps belong to other flows' events interleaved in between, not to anything missing from this one. Elsewhere in the same run, as many as five flows were open across the three gateways at once at the busiest point, just short of the pool's configured ceiling of six. FAQ [#faq] `event.sequence` numbers a stream per host or per source, exactly as [ECS](/docs/tutorials/formats/ecs) defines it, so a consumer can catch a gap or a reordering that the timestamp's own precision might miss. Its scope is covered above: per host or source, never per correlation id or across the whole stream. In practice, a gap in this counter is often the more reliable signal, since a clock adjustment or two events landing in the same millisecond can hide inside a timestamp while a broken sequence stands out immediately. Yes, with [globals](/docs/plugins/event/template/state) instead of `shared` — the identical dict-keyed-by-correlation-id pattern works there too, thread-safe and visible to every generator in the process rather than just one. That fits a source split across several generators producing different vantage points on the same activity: a firewall generator and a NAT gateway generator, each logging the same connection from its own side, coordinated through one pool of correlation ids in `globals` instead of two that never agree. [Scenarios](/docs/studio/scenarios) in Eventum Studio visualizes that shared global state across generators at runtime. [Modeling sessions](/docs/tutorials/realism/sessions) tracks one actor moving through a fixed sequence of stages with a finite state machine — one session id active in shared state at a time, one login before one logout. A correlated stream instead holds many threads open at once, tracked as a dict keyed by correlation id rather than a single current value, with each render picking one entry from that pool to advance instead of following one path from a fixed starting state. Reach for a state machine when exactly one sequence is in progress at a time; reach for a keyed pool when there could be several, interleaved, possibly spanning more than one host. Related [#related] * The [Realism](/docs/tutorials/realism) pillar for the other techniques that make synthetic data behave like production * The [Realistic timing](/docs/tutorials/realism/timing) lesson for shaping when these flows start, not just how they tie together * The [Modeling sessions](/docs/tutorials/realism/sessions) lesson for the single-actor, one-thread-at-a-time counterpart to the pool built here * The [Realistic values](/docs/tutorials/realism/values) lesson for shaping what each event contains, not just its place in a correlated thread * The [Sigma detection testing](/docs/tutorials/detection-testing) tutorial for attack telemetry correlated across several hosts * The [Web clickstream scenario](/docs/tutorials/web-clickstream) tutorial for a single-session FSM applied to a five-stage browsing funnel, instead of the concurrent pool built here * The [ECS fields](/docs/tutorials/formats/ecs) lesson for the complete field-naming conventions `event.sequence` and `related.*` belong to * The [state](/docs/plugins/event/template/state) reference for every method available on `shared` and `globals` * Ready-made generators in the [Eventum Hub](/hub) # Realistic test data: behaving like production A generator that fires events at a perfectly steady rate and picks every value uniformly at random is simple to build — and the easiest kind of synthetic data to spot. Real traffic rises and falls through the day, groups itself into sessions, carries a shared identifier across related events, and clusters around common values instead of spreading evenly. Data that ignores that still fills a schema, but it never behaves like the traffic it stands in for. What makes synthetic data realistic [#what-makes-synthetic-data-realistic] Several properties separate data that behaves like production from data that only fills a schema, and each is independent of the others: * **Timing** — real traffic is not a flat stream. It rises during business hours, bursts around specific triggers, and goes quiet at night or on weekends, following a rhythm instead of a constant rate. * **Sessions and state** — real activity is rarely a set of independent events. A login precedes a stretch of activity and ends with a logout, and each step depends on what happened earlier in the same sequence, not on a fresh random draw. * **Correlated events** — related events tie together across a stream instead of merely sharing a field by coincidence. A shared identifier carries unchanged through every event a flow or a request touches, and a per-host sequence number orders them exactly. * **Values** — real measurements are rarely spread evenly across a range. Most fall near a common case with a long tail of larger outliers, and categorical values such as status codes or protocols skew toward the common ones rather than splitting evenly. Getting one axis right without the others still produces a giveaway: perfectly even timing paired with skewed values reads as a load-test script. Sometimes the more direct move is to skip shaping any of this by hand and reuse a real capture that already has it built in — replaying an existing log with fresh timestamps rather than generating a new approximation of it. How Eventum generates it [#how-eventum-generates-it] Eventum exposes each axis as something you configure, not a fixed default: * **Timing** — the [time\_patterns](/docs/plugins/input/time-patterns) input plugin replaces a flat cron or timer schedule with a repeating time window, a baseline volume, and a distribution that places timestamps inside each window, so the timing itself carries a daily or weekly rhythm. * **Sessions and state** — the [template](/docs/plugins/event/template) event plugin's [mode: fsm](/docs/plugins/event/template/modes#fsm) turns a set of templates into states connected by transitions, paired with [state](/docs/plugins/event/template/state) that carries a session id or counter across the sequence. * **Correlated events** — a [shared](/docs/plugins/event/template/state) pool keyed by correlation id, rendered through [mode: all](/docs/plugins/event/template/modes#all) instead of a state machine, ties together several concurrent threads — a flow, a request-response pair — that all run at once. * **Values** — the same plugin fills in values with [module.rand](/docs/plugins/event/template/modules)'s skewed distributions — log-normal for byte sizes, exponential for durations, Gaussian for metrics — plus weighted choice for categories, Faker- and Mimesis-backed fields, and pre-built [sample](/docs/plugins/event/template/samples) datasets. * **Replay** — the [replay](/docs/plugins/event/replay) event plugin skips generation altogether and reads an existing log file back, rewriting only the timestamps it finds to the moment each line is actually replayed. Related [#related] * The [log and event formats](/docs/tutorials/formats) field guide for the record shapes these techniques bring to life. * Continue to [stream synthetic data to your stack](/docs/tutorials/delivery) — once the data behaves like production, deliver it to the backend you use. * Ready-made generators that already combine these techniques in the [Eventum Hub](/hub). # Replay logs with fresh timestamps Some test scenarios do not call for synthetic data at all — they call for a file that already exists, played back. A captured production sample from a system that's since been decommissioned, an incident capture pulled straight from a security ticket, a golden dataset a test suite has quietly depended on for months: reaching for a template that generates a plausible approximation throws away the one thing that made any of these worth keeping in the first place — their exact content, in their exact order. Generating from scratch reproduces the shape a source would produce; it cannot reproduce the one sequence of events that actually happened. Eventum's [replay](/docs/plugins/event/replay) event plugin replays historical logs without shipping their original, stale timestamps along with them. Every line in a captured file still carries whatever timestamp it had the moment the file was captured, and replay reads that file back line by line and, wherever a line carries a timestamp, rewrites it to the moment the event is actually replayed — the same real content, arriving with a fresh timestamp. When to replay instead of generate [#when-to-replay-instead-of-generate] Replay and the [template](/docs/plugins/event/template) plugin solve different problems, and the choice comes down to what a given test actually needs held constant. Replay preserves exact content and exact ordering — the file handed to it is the file that comes back out, line for line, with nothing invented and nothing left out. Generation synthesizes fresh data on every run instead: parameterized, varied from one event to the next, and able to keep producing for as long as the generator keeps running. Reach for replay when the value sits in one specific file: a captured incident that needs to run past a detection rule exactly as it happened, an exact reproduction of a known-bad or known-good case, a fixed golden dataset a test suite already depends on. Reach for generation instead when a test needs variety across many runs, parameters someone can adjust, or a stream that keeps going rather than stopping once a fixed file runs out. How log replay works [#how-log-replay-works] The replay event plugin reads a log file from `path` line by line and emits each line as one event, in file order — one line consumed per timestamp the input plugin schedules. Content by itself is not enough for most replay scenarios, though: a line's own timestamp is fixed at whatever moment the file was originally captured, and replaying it verbatim would ship stale timestamps into a system expecting current ones. Two fields fix that. `timestamp_pattern` is a regular expression with a named `timestamp` group that locates the original timestamp inside each line; `timestamp_format` is a strftime format string describing how the replacement should be rendered — left unset, replay falls back to ISO 8601. Together, they let the plugin swap out just that one substring for the event's actual current timestamp, leaving the rest of the line untouched. A line that doesn't match the pattern at all is emitted exactly as read, logged as a warning rather than treated as an error — which matters for a captured file that mixes timestamped entries with the odd header, separator, or footer line. `repeat` decides what happens once the file runs out: left at its default of `false`, the plugin stops producing once the last line has been replayed; set to `true`, it loops back to the start and keeps going instead, useful for pairing a small captured sample with a schedule that keeps ticking. Two more fields cover less common cases: `chunk_size` controls how much of the file is buffered in memory at a time — the default reads it in 1 MiB chunks rather than loading it whole, which keeps a large file safe to replay — and `encoding` overrides the default UTF-8 for a file captured with a different codec. Replay a log with fresh timestamps in Eventum [#replay-a-log-with-fresh-timestamps-in-eventum] Write the source log [#write-the-source-log] `generators/incident-replay/samples/source.log` is a small slice of a captured authentication log — three failed logins followed by the one that got through, the kind of snippet an analyst might pull straight from an incident ticket. Every line but the last carries a plain ISO-8601 timestamp; the last is a marker appended when the capture was saved, with no timestamp of its own: ```text title="generators/incident-replay/samples/source.log" 2026-03-14T02:11:03 sshd: Failed password for invalid user admin from 198.51.100.23 port 51422 ssh2 2026-03-14T02:11:05 sshd: Failed password for invalid user admin from 198.51.100.23 port 51423 ssh2 2026-03-14T02:11:08 sshd: Failed password for invalid user admin from 198.51.100.23 port 51424 ssh2 2026-03-14T02:11:12 sshd: Accepted password for root from 198.51.100.23 port 51425 ssh2 [end of capture] ``` Configure the replay generator [#configure-the-replay-generator] A per-second [cron](/docs/plugins/input/cron) tick supplies one timestamp per line, and replay's own fields do the rest: `path` to the file, `timestamp_pattern` to find the original timestamp, `timestamp_format` to render the replacement. The [file](/docs/plugins/output/file) output writes each line through the `plain` formatter — the events here are already-formatted text, not JSON, so `plain` is the only formatter that keeps them intact: ```yaml title="generators/incident-replay/generator.yml" input: - cron: expression: "* * * * * *" count: 1 event: replay: path: samples/source.log timestamp_pattern: '(?P\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2})' timestamp_format: '%Y-%m-%dT%H:%M:%S' output: - file: path: output/events.log write_mode: overwrite formatter: format: plain ``` * **path** — `samples/source.log`, resolved relative to `generator.yml`. * **timestamp\_pattern** — matches the plain `YYYY-MM-DDTHH:MM:SS` timestamp at the start of each line, naming the match `timestamp`. * **timestamp\_format** — renders the replacement in this shape, and the substituted timestamp lands in the exact position the original occupied (the two fields need not describe the same shape). * **formatter** — set to `plain`, the file output's own default, and the only correct choice here, since a replayed line is already-formatted text, not JSON. The output below was produced by running this generator in [sample mode](/docs/core/concepts/generator#sample-mode) (`--live-mode false`) with `--keep-order true`, so all five lines render at once, in file order, for inspection. `repeat` is left at its default of `false`, so the generator stops on its own once the fifth line has been replayed — no bound on the cron input was needed to make this a finite run. A generator left running normally drops both flags and lets cron tick indefinitely in [live mode](/docs/core/concepts/generator#live-mode-default) instead, replaying one line per second until the file is exhausted. The result [#the-result] Running this generator replays all five lines and then stops on its own — nothing forces it to continue once the last line has been replayed: ```text title="generators/incident-replay/output/events.log" 2026-07-17T14:53:19 sshd: Failed password for invalid user admin from 198.51.100.23 port 51422 ssh2 2026-07-17T14:53:20 sshd: Failed password for invalid user admin from 198.51.100.23 port 51423 ssh2 2026-07-17T14:53:21 sshd: Failed password for invalid user admin from 198.51.100.23 port 51424 ssh2 2026-07-17T14:53:22 sshd: Accepted password for root from 198.51.100.23 port 51425 ssh2 [end of capture] ``` Every one of the first four lines kept its exact original content — the same source IP, the same port, the same username, the same outcome — with only the leading timestamp moved from March 2026 to the moment this run actually happened, one second apart, matching the per-second cron schedule. The fifth line has nothing for `timestamp_pattern` to match, so it came back exactly as written in `source.log`, byte for byte — the only sign it was even considered is a warning logged at the time (`Failed to substitute timestamp into original message`, reason `No match found`), not an error, and not a dropped event. FAQ [#faq] Only the substring that `timestamp_pattern`'s named group matches changes — nothing else in a line is touched, and a line with no match at all comes back exactly as read, the way the `[end of capture]` marker line does in the example above. Leave `timestamp_pattern` unset entirely and replay changes nothing: every line is emitted exactly as it appears in the file. Yes — set `repeat: true` and replay resets to the beginning once it reaches the end of the file instead of stopping, which pairs a small captured sample with a schedule that keeps ticking indefinitely. Left at the default `false`, as in the example above, the plugin stops producing once the last line has been replayed, and the generator run ends with it. Either — replay itself doesn't decide which one fits, [streaming vs bulk](/docs/tutorials/foundations/streaming-vs-bulk) does. In [live mode](/docs/core/concepts/generator#live-mode-default), one line replays per scheduled timestamp, paced to the clock, so the five-line file above paired with a per-second schedule takes five real seconds. In [sample mode](/docs/core/concepts/generator#sample-mode) (`--live-mode false`), the whole file replays as fast as the pipeline can move, the way the run above did. Related [#related] * The [Realism](/docs/tutorials/realism) pillar for the other techniques that make synthetic data behave like production * The [Log and event formats](/docs/tutorials/formats) field guide for the record shapes a replayed file is likely to carry * The [Streaming vs bulk](/docs/tutorials/foundations/streaming-vs-bulk) lesson for how live mode paces a replay and sample mode replays a whole file at once * The [Sigma detection testing](/docs/tutorials/detection-testing) tutorial for replaying a captured incident straight into a detection rule instead of building attack telemetry from scratch * The [replay](/docs/plugins/event/replay) reference for every field, including `chunk_size` and `encoding` * The [template](/docs/plugins/event/template) reference for the alternative event plugin, when a test needs generated variety instead of one exact file * Ready-made generators in the [Eventum Hub](/hub) # Simulate user sessions: state machines A generator that fires isolated events — one random user logging an action here, an unrelated user somewhere else, each timestamp picked independently of the last — never produces the shape real activity actually takes. A person who uses a system logs in, does a handful of things, and eventually logs out, and every event in between belongs to that one visit: the same user, the same session id, in a fixed order, over a bounded stretch of time. Eventum models that shape with a state machine: each stage of a session — a login, an action, a logout — is a template, and the template's own logic decides when the machine is ready to move to the next stage. State shared across those templates carries whatever ties the sequence together, typically a session id and a running count of what has happened so far — the detail that turns a handful of otherwise disconnected-looking JSON lines into one traceable session. What a login-to-logout sequence looks like [#what-a-login-to-logout-sequence-looks-like] A session is any stretch of related events tied to one actor, bounded by a clear start and a clear end, with a variable amount of activity in between. A web visit starts with a login and ends with a logout or a timeout; a support ticket opens, gets updated a few times, and closes; a device connects, reports for a while, and disconnects. What makes these sequences — rather than just a pile of events that happen to share a user id — is that each step depends on where the sequence currently stands: there is no logout without a prior login, and the number of actions in between is not fixed in advance. Nothing stops a generator that treats every timestamp as a fresh, independent random pick from emitting a logout with no matching login, or a burst of unrelated actions under session ids that each appear exactly once. Modeling a session means giving the generator memory: an idea of what stage the current sequence is in, and what has already happened since it started. Modeling sessions with a state machine [#modeling-sessions-with-a-state-machine] Eventum's [template](/docs/plugins/event/template) event plugin has a picking mode built for exactly this: [mode: fsm](/docs/plugins/event/template/modes#fsm) turns a set of templates into states of a finite state machine, one template per stage of the sequence. A three-state session — login, action, logout — maps directly onto three templates connected by transitions: Each timestamp renders the machine's current state and then checks that state's transitions in order; the first one whose condition holds moves the machine, and if none do, it stays put and renders the same state again on the next timestamp. A transition's condition inspects state that the current template set while rendering — a flag, a counter, anything the template chose to record — so the sequence advances only when the template itself signals it is ready, not on a fixed schedule. That state is also what correlates the sequence. Every state in the machine is a separate template, and `locals` — the per-template scope — belongs to exactly one of them: a value set in `login`'s locals never reaches `action`'s. `shared` is the one scope every state in the same generator reads and writes, so a session id and a running step count belong there: `login` sets them once, `action` and `logout` read and update them on every later render, and those same shared values tie every event below back to one session. Simulate a login-to-logout session in Eventum [#simulate-a-login-to-logout-session-in-eventum] The templates [#the-templates] `login.jinja` marks the start of a session: it draws a new session id and a user, records both in shared state together with a step counter reset to 1, and clears any leftover `logout_ready` flag from whatever session came before it. ```jinja title="generators/user-sessions/templates/login.jinja" {%- set session_id = module.rand.crypto.uuid4() -%} {%- set user = module.rand.choice(["alice", "bob", "carol", "dave"]) -%} {%- do shared.set("session_id", session_id) -%} {%- do shared.set("user", user) -%} {%- do shared.set("step", 1) -%} {%- do shared.pop("logout_ready", None) -%} {"timestamp": "{{ timestamp.isoformat() }}", "session_id": "{{ session_id }}", "user": "{{ user }}", "event": "login", "step": 1} ``` `action.jinja` renders one step of activity: it reads the session id and user back out of shared state rather than generating its own, increments the step counter, and picks one of five actions at random. Once the counter reaches five, it sets `logout_ready` — the signal the state machine is waiting for to end the session: ```jinja title="generators/user-sessions/templates/action.jinja" {%- set step = shared.get("step", 1) + 1 -%} {%- do shared.set("step", step) -%} {%- set action = module.rand.choice(["view_dashboard", "update_profile", "export_report", "search_records", "change_settings"]) -%} {%- if step >= 5 -%} {%- do shared.set("logout_ready", true) -%} {%- endif -%} {"timestamp": "{{ timestamp.isoformat() }}", "session_id": "{{ shared.get('session_id') }}", "user": "{{ shared.get('user') }}", "event": "action", "action": "{{ action }}", "step": {{ step }}} ``` `logout.jinja` closes the session out: one more step increment, then the same session id and user one last time, with nothing left for the next session to inherit but the state machine's position: ```jinja title="generators/user-sessions/templates/logout.jinja" {%- set step = shared.get("step", 1) + 1 -%} {%- do shared.set("step", step) -%} {"timestamp": "{{ timestamp.isoformat() }}", "session_id": "{{ shared.get('session_id') }}", "user": "{{ shared.get('user') }}", "event": "logout", "step": {{ step }}} ``` The generator config [#the-generator-config] These three templates become the three states of an `fsm` machine, wired together by transitions: ```yaml title="generators/user-sessions/generator.yml" input: - cron: expression: "* * * * * *" count: 1 event: template: mode: fsm templates: - login: template: templates/login.jinja initial: true transitions: - to: action when: always: - action: template: templates/action.jinja transitions: - to: logout when: defined: shared.logout_ready - to: action when: always: - logout: template: templates/logout.jinja transitions: - to: login when: always: output: - file: path: output/events.jsonl write_mode: overwrite formatter: format: json ``` * **login** — marked `initial: true`, so the machine starts here; its only transition always fires, moving to `action` on the very next timestamp. * **action** — stays in `action` (the second, fallback transition) until its own template sets `shared.logout_ready`, at which point the first transition fires and the machine moves to `logout`. Transitions are checked in the order they are listed, so the `logout_ready` check has to come first. * **logout** — always returns to `login`, which starts the next session from scratch: a new session id, a new user, and the step counter reset to 1. The session below was produced by running this generator in [sample mode](/docs/core/concepts/generator#sample-mode) (`--live-mode false`) with a short bound added to the cron input, so a handful of sessions render at once for inspection. A generator left running normally drops that bound and lets cron tick indefinitely in [live mode](/docs/core/concepts/generator#live-mode-default), starting a fresh session every time the previous one logs out. The result [#the-result] Running this generator for 30 timestamps produced five complete sessions — 30 events in total, six per session: a login, four actions, and a logout, always in that order. One session in full, straight from the output file: ```json title="output/events.jsonl" {"timestamp": "2026-07-11T20:20:09+00:00", "session_id": "6eaa3aa6-b086-41e8-923d-4cff3de6f6b2", "user": "alice", "event": "login", "step": 1} {"timestamp": "2026-07-11T20:20:10+00:00", "session_id": "6eaa3aa6-b086-41e8-923d-4cff3de6f6b2", "user": "alice", "event": "action", "action": "update_profile", "step": 2} {"timestamp": "2026-07-11T20:20:11+00:00", "session_id": "6eaa3aa6-b086-41e8-923d-4cff3de6f6b2", "user": "alice", "event": "action", "action": "view_dashboard", "step": 3} {"timestamp": "2026-07-11T20:20:12+00:00", "session_id": "6eaa3aa6-b086-41e8-923d-4cff3de6f6b2", "user": "alice", "event": "action", "action": "search_records", "step": 4} {"timestamp": "2026-07-11T20:20:13+00:00", "session_id": "6eaa3aa6-b086-41e8-923d-4cff3de6f6b2", "user": "alice", "event": "action", "action": "change_settings", "step": 5} {"timestamp": "2026-07-11T20:20:14+00:00", "session_id": "6eaa3aa6-b086-41e8-923d-4cff3de6f6b2", "user": "alice", "event": "logout", "step": 6} ``` Every line carries the same `session_id`, and `step` climbs from 1 to 6 without a gap or a repeat — the only order the machine's transitions allow, since `action` cannot reach `logout` before its own template sets `logout_ready`, and nothing but `login` ever clears that flag for the next session. FAQ [#faq] Yes. The fixed four-action session above keeps the example easy to follow, but a session's length is decided by the template, not by `mode: fsm` itself. A common variation: on every render, `action` rolls a random chance of ending the session (`module.rand.chance(0.1)` for roughly one in ten) and sets the same flag the moment that roll succeeds, instead of or alongside the step-count check — so some sessions run for two actions and others for fifteen, the same way real visits vary. Not with the pattern above. One `fsm` machine tracks exactly one current state and one shared session id, so sessions here play out one after another — accurate for a single shared actor (one kiosk, one shared account, one audit trail per host), but not for many concurrent users. Modeling concurrent sessions needs a different design: a pool of in-flight session records tracked in shared state, with each render advancing one entry from that pool instead of a single session id. No. `shared` and `locals` live only in the running process's memory, so stopping and starting the generator again begins at the state marked `initial` with empty state — a fresh login for whatever timestamp comes next, not a continuation of whatever session was last active. Yes. `mode: fsm` supports any number of templates as states, each with its own transitions — a three-state login/action/logout cycle and a ten-state pipeline follow exactly the same mechanism. Real sessions often split a single "action" stage into several more specific ones — a product page, a cart, a checkout — once each stage needs to render different content or transition on a different condition. Related [#related] * The [Realism](/docs/tutorials/realism) pillar for the other axes that make synthetic data behave like production * The [Realistic timing](/docs/tutorials/realism/timing) lesson for shaping when these sessions start, not just what happens inside them * The [Correlated events](/docs/tutorials/realism/correlated-events) lesson for many concurrent threads tracked in a pool, instead of the one sequential session modeled here * The [Realistic values](/docs/tutorials/realism/values) lesson for shaping what each event contains, not just its place in a sequence * The [Web clickstream scenario](/docs/tutorials/web-clickstream) for the same technique applied to a five-state browsing session streamed into a real backend * The [FSM](/docs/plugins/event/template/fsm) and [state](/docs/plugins/event/template/state) references for the complete transition/condition catalog and every state method used above * Ready-made generators in the [Eventum Hub](/hub) # Simulate traffic patterns: peaks and bursts A generator wired to a [cron](/docs/plugins/input/cron) expression or a [timer](/docs/plugins/input/timer) interval produces exactly the same number of events every tick, for as long as it runs. That flat rate is enough to check that a pipeline moves data from one end to the other, but nothing about it resembles real traffic: a load test built on it never has to absorb a real peak, and a detection rule tuned against it never sees a baseline it can deviate from, because every window of the stream looks exactly like the last one. Eventum's [time\_patterns](/docs/plugins/input/time-patterns) input plugin replaces that flat tick with a distribution: a repeating window, a baseline volume for that window, and a statistical shape that decides where inside the window each timestamp lands. The same mechanism builds a rhythm that rises and falls through a cycle and, layered a second time, a short and sharp spike on top of it — two techniques for the same goal: simulating traffic patterns instead of a flat, mechanical tick. Why real traffic isn't flat [#why-real-traffic-isnt-flat] Production traffic follows the rhythm of whoever or whatever drives it. A consumer-facing API rises through business hours and falls back overnight; a batch pipeline sits quiet all day and then bursts the moment a scheduled job starts; a marketing email or an incident can push a spike through a system at a moment that has nothing to do with the time of day. None of this is noise to average away — it is the shape of the thing being tested. That shape matters for two different reasons, depending on what the generated data stands in for. A load or capacity test exists to find out whether infrastructure survives the worst few minutes of a day, not the average minute — autoscalers, connection pools, and queues are only meaningfully exercised by an actual swing from quiet to busy, never by a flat, unchanging rate. A detection rule or an anomaly score is calibrated against a baseline it expects to deviate from. A flat stream gives it none: either nothing ever looks anomalous, or every small fluctuation does. The same rhythm often repeats on a longer cycle too, since a weekday's traffic rarely matches a weekend's, though the techniques below focus on the daily cycle most systems feel first. Simulate a diurnal pattern [#simulate-a-diurnal-pattern] [time\_patterns](/docs/plugins/input/time-patterns) builds a distribution from four stages, each configured in a pattern file separate from the generator itself: * **Oscillator** — divides time into a repeating window. * **Multiplier** — sets a baseline number of timestamps per window. * **Randomizer** — adds variance so no two windows produce exactly the same count. * **Spreader** — places each window's timestamps inside it according to a statistical distribution. A beta spreader with equal shape parameters concentrates timestamps around the middle of the window and thins them out toward both edges — a symmetric bell centered on the window — the shape a business-hours peak traces across a day. The pattern file below models a day-long window this way: a baseline of 2,400 requests, up to 15% more or less from one day to the next, clustered toward the middle of each day and quiet at the edges. ```yaml title="generators/api-traffic/patterns/daily-traffic.yml" label: daily-traffic oscillator: start: "now" end: "+3d" period: 1 unit: days multiplier: ratio: 2400 randomizer: deviation: 0.15 direction: mixed spreader: distribution: beta parameters: a: 4 b: 4 ``` * **oscillator** — a one-day window, repeated for three days from the moment the generator starts. * **multiplier** — 2,400 requests as the baseline count for each day. * **randomizer** — up to ±15% variance, so no two days produce an identical total. * **spreader** — a beta distribution with equal shape parameters (`a: 4`, `b: 4`), a symmetric bell that peaks at the middle of each day and tapers off at both edges. That is how to simulate peak traffic without discarding the quiet hours around it — one distribution produces both from a single configuration. Layer a bursty traffic spike on top [#layer-a-bursty-traffic-spike-on-top] A single pattern file models one rhythm. `time_patterns` accepts more than one, merging every pattern's timestamps into the same chronological stream. That is what puts a short, high-volume trigger on top of the daily rhythm instead of replacing it. The pattern file below adds a ten-minute spike unrelated to the time of day, the kind a marketing blast or a scheduled batch job produces: ```yaml title="generators/api-traffic/patterns/traffic-spike.yml" label: traffic-spike oscillator: start: "+1d3h" end: "+10m" period: 10 unit: minutes multiplier: ratio: 900 randomizer: deviation: 0.0 direction: mixed spreader: distribution: uniform parameters: low: 0.0 high: 1.0 ``` `start` places the spike a day and three hours into the run, inside the quiet overnight stretch of the second day's cycle — one of the hours where the daily pattern alone produces its lowest counts. `end` is relative to this pattern's own `start`, not to the current time, so `+10m` is always a ten-minute window regardless of when the generator actually starts — see [date ranges and versatile datetime](/docs/core/concepts/scheduling#date-ranges-and-versatile-datetime) for how every relative expression resolves. A uniform spreader with no randomizer deviation packs exactly 900 requests evenly across those ten minutes, with no bell shape softening the edges — a spike is meant to look abrupt. The generator config [#the-generator-config] Both pattern files feed the same `time_patterns` input; a [template](/docs/plugins/event/template) renders each timestamp as one API request line using [module.rand](/docs/plugins/event/template/modules), and a [file](/docs/plugins/output/file) output writes the result through the `json` formatter: ```jinja title="generators/api-traffic/templates/request.jinja" {%- set path = module.rand.choice(["/api/v1/products", "/api/v1/cart", "/api/v1/checkout", "/api/v1/search"]) -%} {%- set status = module.rand.weighted_choice({200: 92, 404: 5, 500: 3}) -%} {"timestamp": "{{ timestamp.isoformat() }}", "path": "{{ path }}", "status": {{ status }}} ``` ```yaml title="generators/api-traffic/generator.yml" input: - time_patterns: patterns: - patterns/daily-traffic.yml - patterns/traffic-spike.yml event: template: mode: all templates: - request: template: templates/request.jinja output: - file: path: output/events.jsonl write_mode: overwrite formatter: format: json ``` Sample mode (`--live-mode false`) generates all three simulated days at once for inspection — the result below reflects that run. A generator left running normally uses [live mode](/docs/core/concepts/generator#live-mode-default) instead, pacing the same distribution against the real clock, with the oscillator's `end` set to `never` rather than `+3d` to keep repeating the daily cycle indefinitely. The result [#the-result] Running the generator above for three simulated days produced 7,346 requests in total — 2,138 on Day 1, 3,103 on Day 2, and 2,105 on Day 3. Counting how many land in each three-hour slice of each day shows the shape the configuration describes rather than just asserts it: | Hours into the day | Day 1 | Day 2 | Day 3 | | ------------------ | ----- | ----- | ----- | | 0-3 | 16 | 12 | 12 | | 3-6 | 142 | 1,045 | 128 | | 6-9 | 356 | 381 | 360 | | 9-12 | 521 | 513 | 533 | | 12-15 | 556 | 575 | 560 | | 15-18 | 389 | 402 | 385 | | 18-21 | 137 | 160 | 116 | | 21-24 | 21 | 15 | 11 | Day 1 and Day 3 carry the daily pattern alone: quiet in the first and last few hours of the cycle, rising through the following slices, peaking around the middle of the day, and falling back — the same shape twice, from two independent draws of the same configuration. Day 2 repeats that shape in every slice except one: the 3-to-6-hour slice jumps to 1,045, seven to eight times its counterpart on the other two days, because the ten-minute burst window falls inside exactly that slice. Zooming into that ten-minute window itself, 903 of the slice's 1,045 requests land inside it — 900 from the burst pattern, plus a handful the daily pattern would have produced there on its own. Three consecutive lines from the quiet edge of Day 1, minutes apart: ```json title="Quiet edge of Day 1 — output/events.jsonl" {"timestamp": "2026-07-11T20:58:01.041093+00:00", "path": "/api/v1/search", "status": 200} {"timestamp": "2026-07-11T21:04:36.927569+00:00", "path": "/api/v1/search", "status": 200} {"timestamp": "2026-07-11T21:10:03.832763+00:00", "path": "/api/v1/cart", "status": 200} ``` Three consecutive lines from inside the Day 2 burst, a second or a fraction of a second apart: ```json title="Inside the Day 2 burst — output/events.jsonl" {"timestamp": "2026-07-12T22:25:21.399336+00:00", "path": "/api/v1/products", "status": 200} {"timestamp": "2026-07-12T22:25:22.440915+00:00", "path": "/api/v1/products", "status": 200} {"timestamp": "2026-07-12T22:25:22.591498+00:00", "path": "/api/v1/cart", "status": 200} ``` FAQ [#faq] Yes. A generator accepts any number of input plugins, and their timestamps are merged into a single chronological stream before reaching the event stage — a steady [cron](/docs/plugins/input/cron) baseline and a `time_patterns` layer can run side by side exactly as the two pattern files above do. See [combining multiple inputs](/docs/core/concepts/scheduling#combining-multiple-inputs) for how the merge works. Yes, and [live mode](/docs/core/concepts/generator#live-mode-default) is the more common setting for `time_patterns` in practice: the generator paces each timestamp against the real clock instead of releasing all of them at once, turning the distribution above into an ongoing traffic shape. [Sample mode](/docs/core/concepts/generator#sample-mode), used to produce the result above, is the faster way to inspect a pattern's shape before leaving it running. Set the oscillator's `end` to `never` instead of a bounded value such as `+3d`. Reserve that setting for live mode: sample mode ignores wall-clock pacing entirely, so an unbounded pattern there tries to produce its whole infinite range at once instead of spreading it out over real time. A single pattern file applies the same oscillator, multiplier, and spreader to every window in its range, so one file cannot make Saturday quieter than Wednesday on its own. Layer a second pattern file scoped to the weekend dates with a lower `multiplier.ratio`, the same way the burst pattern above layers on top of the daily one — every pattern feeding the same `time_patterns` input merges into one stream regardless of how many there are. `uniform` spreads timestamps flat across the window, useful for a burst with no shape of its own, as above. `triangular` gives a single peak at a chosen point with straight, adjustable slopes on either side. `beta` is the most flexible: equal `a` and `b` values center a symmetric bell like the daily pattern above, while unequal values skew it earlier or later in the window. See the [time\_patterns reference](/docs/plugins/input/time-patterns#spreader) for every parameter. Related [#related] * The [Realism](/docs/tutorials/realism) pillar for the other axes that make synthetic data behave like production traffic * The [Modeling sessions](/docs/tutorials/realism/sessions) lesson for turning these timestamps into a connected sequence of actions instead of independent events * The [Realistic values](/docs/tutorials/realism/values) lesson for shaping what each event contains, not just when it happens * The [Stream synthetic data to your stack](/docs/tutorials/delivery) pillar for what a realistic arrival rate means for the backend receiving it * The [time\_patterns](/docs/plugins/input/time-patterns) reference for every oscillator, multiplier, randomizer, and spreader parameter * Ready-made generators in the [Eventum Hub](/hub) — every one currently ticks on a flat `cron` schedule, each a candidate for the same `time_patterns` treatment shown here # Generate realistic fake data: skewed values A template field set with `module.rand.number.integer(a, b)` or `module.rand.choice(...)` treats every outcome in its range as equally likely, and production data almost never behaves that way. A response body clusters around a common size and occasionally runs ten times larger; a request usually finishes in tens of milliseconds and only rarely drags into the hundreds; a status code reads `200` the overwhelming majority of the time and `500` only on the rare failure. Spread any of those fields evenly across its range instead, and it reads as generated the moment more than a handful of values sit side by side. Eventum's [template](/docs/plugins/event/template) event plugin fills a field from the same shape the real value would take, not a flat range: [module.rand.number](/docs/plugins/event/template/modules) draws a size or a duration from a skewed distribution, `module.rand.weighted_choice` picks a category with the odds real traffic actually shows, and `module.faker`, `module.mimesis`, or a pre-built [sample](/docs/plugins/event/template/samples) supply a name, an address, or a product realistic enough to survive a second look. Why uniform random values look synthetic [#why-uniform-random-values-look-synthetic] Three kinds of field give away a generator that only picks uniformly. A size or a duration — a payload in bytes, a response time in milliseconds, a transaction amount — clusters around a typical case in real systems, with a long thinning tail of larger outliers; spread the same field evenly across its range and a value becomes just as likely to be extreme as typical, which real measurements never are. A category — a status code, a protocol, a log level — carries real-world odds: success, the expected protocol, and a routine log level are each the common case, and picking uniformly gives the rare exception the same odds as the norm. A name, an address, or a product has to look real — a run of random letters or digits fills the field, but nobody mistakes it for an actual customer. The generator built later in this lesson makes the difference concrete. Its `bytes_sent` field is drawn with `module.rand.number.lognormal(6.9, 0.6)` and clamped to a realistic floor and ceiling; across 200 generated checkouts, half landed under 1,031 bytes, the busiest tenth reached 2,187, and only one touched the 3,326-byte ceiling of that run. A `module.rand.number.integer(250, 3326)` field over the same 200 checkouts would spread just as many values near 3,000 as near 500 — flat where the real shape clusters low with a thinning tail toward the top. Skewed distribution test data for realistic numbers [#skewed-distribution-test-data-for-realistic-numbers] Three distributions in [module.rand.number](/docs/plugins/event/template/modules) cover most skewed fields: | Function | Shape | Typical field | | ---------------------- | ---------------------------------------------------- | ------------------------------------------------------------ | | `lognormal(mu, sigma)` | Always positive, long right tail | Payload sizes, file sizes, transaction amounts | | `exponential(lambd)` | Many small values, a thinning tail toward large ones | Durations, wait times, time between events | | `gauss(mu, sigma)` | Symmetric bell centered on a mean | Scores, sensor readings, metrics that vary around a baseline | `lambd` is a rate, not a mean — pass `1 / ` to center the distribution where you want it. Every one of these can, in theory, return a value past any realistic bound, so pair it with `clamp(value, min, max)` to enforce a floor or a ceiling the field can't cross: a byte count can't go negative, a fraud score can't exceed 100. Two more distributions in the same namespace cover shapes these three don't: `pareto(alpha, xmin)` for a heavier tail than `lognormal` gives, and `triangular(low, high, mode)` for a bounded value with a known typical case. The [modules reference](/docs/plugins/event/template/modules) has the full signature for each. Weighted random data for realistic categories [#weighted-random-data-for-realistic-categories] `module.rand.weighted_choice(items, weights)` picks one item with the odds you assign it, taking either two parallel sequences or a single dict of `{item: weight}` pairs. A status code that succeeds 90% of the time, fails outright rarely, and gets rejected occasionally reads as `module.rand.weighted_choice({200: 90, 402: 7, 500: 3})` — the same function fits any skewed category: a protocol mix mostly `tcp` with a trickle of `udp`, an HTTP method mostly `GET` with occasional `POST` and rare `DELETE`, a log level mostly `INFO` with `WARN` and `ERROR` further down. `weighted_choices(items, weights, n)` returns several picks at once when a template needs more than one. Base the weights on whatever the real source actually shows — vendor documentation, a sample of production traffic, an existing dataset for the same kind of system — rather than a round number picked for convenience; the point of weighting is to match odds that exist somewhere, not just to move away from uniform. Domain values instead of synthetic noise [#domain-values-instead-of-synthetic-noise] [module.rand.string](/docs/plugins/event/template/modules) generates literal random characters — letters, digits, hex — which is exactly right for a token or an identifier and exactly wrong for a name or an address, where the field needs to look like a real one, not like noise. `module.faker` and `module.mimesis` fill that gap: `module.faker.locale['en_US'].name()` and `module.mimesis.locale['en'].address.city()` draw a fresh, realistic-looking value on every render, across hundreds of provider methods covering names, addresses, companies, products, and more. A [sample](/docs/plugins/event/template/samples) is the other option, for a smaller, fixed vocabulary instead of an unbounded stream of fresh values: a `csv` or `json` file loaded once at startup, picked from with `pick()` or, weighted the same way a category is, with `weighted_pick(weight)` — a services sample where `HTTP` outweighs `SSH` combines domain data and weighted choice in a single pick. Reach for a sample instead of `faker`/`mimesis` when a field should repeat a bounded, realistic set of values — the same fifty hostnames across a run — rather than a new one every time, or when picking from a pre-built list should cost less per event than generating a fresh value. None of these three techniques compute anything from data already produced — each draws a fresh, shaped value on its own, with nothing to aggregate and nothing to analyze first. Building a realistic fake data generator in Eventum [#building-a-realistic-fake-data-generator-in-eventum] A checkout API is a natural place to combine all three techniques at once: a payload size and a processing time that should cluster and thin out rather than spread evenly, a status that succeeds far more often than it fails, and a customer name that should read like a real one. The template [#the-template] ```jinja title="generators/checkout-api/templates/checkout.jinja" {%- set status = module.rand.weighted_choice({200: 90, 402: 7, 500: 3}) -%} {#- Payload size: right-skewed, most responses small, a rare large one -#} {%- set bytes_sent = module.rand.number.clamp(module.rand.number.lognormal(6.9, 0.6), 250, 20000) | round | int -%} {#- Processing time: most requests fast, a thinning tail toward slow -#} {%- set duration_ms = module.rand.number.clamp(module.rand.number.exponential(1 / 85), 5, 3000) | round | int -%} {#- Fraud score: clusters around a low baseline with roughly symmetric variance -#} {%- set risk_score = module.rand.number.clamp(module.rand.number.gauss(18, 12), 0, 100) | round(1) -%} {%- set customer = module.faker.locale['en_US'].name() -%} {"timestamp": "{{ timestamp.isoformat() }}", "customer": "{{ customer }}", "status": {{ status }}, "bytes_sent": {{ bytes_sent }}, "duration_ms": {{ duration_ms }}, "risk_score": {{ risk_score }}} ``` * **status** — `weighted_choice` over a dict: 90% succeed, 7% get rejected, 3% fail outright. * **bytes\_sent** — `lognormal(6.9, 0.6)` clamped to 250-20,000 bytes: a right-skewed payload size. * **duration\_ms** — `exponential(1 / 85)` clamped to 5-3,000 milliseconds: a rate built around an 85ms mean, with most requests well under it. * **risk\_score** — `gauss(18, 12)` clamped to 0-100: a fraud score centered low with roughly symmetric spread. * **customer** — `module.faker.locale['en_US'].name()`: a fresh, realistic name on every render. The generator config [#the-generator-config] ```yaml title="generators/checkout-api/generator.yml" input: - cron: expression: "* * * * * *" count: 1 event: template: mode: all templates: - checkout: template: templates/checkout.jinja output: - file: path: output/events.jsonl write_mode: overwrite formatter: format: json ``` The events below were produced by running this generator in [sample mode](/docs/core/concepts/generator#sample-mode) (`--live-mode false`) with a short bound added to the cron input, so two hundred checkouts render at once for inspection. A generator left running normally drops that bound and lets cron tick indefinitely in [live mode](/docs/core/concepts/generator#live-mode-default) instead. The result [#the-result] Running this generator for 200 simulated checkouts produced a realistic spread on every field instead of a flat one. `status` landed on `200` for 181 of them, `402` for 11, and `500` for 8 — close to, though not identical to, the 90/7/3 split the template configures, since `weighted_choice` picks with those odds rather than enforcing an exact count. `risk_score` ranged from the 0 floor up to 49.8, clustering low the way a Gaussian centered on 18 with a standard deviation of 12 is meant to. A representative run of nine consecutive checkouts: ```json title="output/events.jsonl" {"timestamp": "2026-07-11T21:00:14+00:00", "customer": "Jessica Mcbride", "status": 200, "bytes_sent": 663, "duration_ms": 38, "risk_score": 23.6} {"timestamp": "2026-07-11T21:00:15+00:00", "customer": "Samantha Bowen", "status": 200, "bytes_sent": 684, "duration_ms": 40, "risk_score": 29.0} {"timestamp": "2026-07-11T21:00:16+00:00", "customer": "Gregory Mcdonald", "status": 200, "bytes_sent": 2066, "duration_ms": 166, "risk_score": 13.5} {"timestamp": "2026-07-11T21:00:17+00:00", "customer": "Frank Brown", "status": 200, "bytes_sent": 1659, "duration_ms": 61, "risk_score": 13.5} {"timestamp": "2026-07-11T21:00:18+00:00", "customer": "Jonathan Ho PhD", "status": 402, "bytes_sent": 1238, "duration_ms": 57, "risk_score": 6.2} {"timestamp": "2026-07-11T21:00:19+00:00", "customer": "Sandra Mclean", "status": 200, "bytes_sent": 639, "duration_ms": 18, "risk_score": 29.3} {"timestamp": "2026-07-11T21:00:20+00:00", "customer": "Christine Flores", "status": 200, "bytes_sent": 788, "duration_ms": 16, "risk_score": 30.3} {"timestamp": "2026-07-11T21:00:21+00:00", "customer": "Jacob Allen", "status": 200, "bytes_sent": 1529, "duration_ms": 13, "risk_score": 15.0} {"timestamp": "2026-07-11T21:00:22+00:00", "customer": "Jerry Terry", "status": 200, "bytes_sent": 309, "duration_ms": 147, "risk_score": 39.3} ``` Eight of the nine landed on `200` and one on `402`, matching the odds the template sets rather than an even split; `bytes_sent` ranges from 309 to 2,066 in this slice alone, and `duration_ms` from 13 to 166 — a spread no `integer()` field produces on purpose, and exactly the one `lognormal` and `exponential` are built for. Elsewhere in the same run, the rarer `500` case shows up too: ```json title="output/events.jsonl" {"timestamp": "2026-07-11T21:00:34+00:00", "customer": "Cameron Marshall", "status": 500, "bytes_sent": 1855, "duration_ms": 177, "risk_score": 19.0} ``` FAQ [#faq] Match the shape to what the field measures: `lognormal` for a size or an amount that's always positive and occasionally spikes far above typical, `exponential` for a duration or a wait time where most values are short and a few run long, `gauss` for a score or a reading that varies roughly symmetrically around a baseline. `pareto` and `triangular` in the same [module.rand.number](/docs/plugins/event/template/modules) namespace cover a heavier tail and a bounded typical-case value, for fields the first three don't fit well. Wrap it in `module.rand.number.clamp(value, min, max)`, as `bytes_sent`, `duration_ms`, and `risk_score` all do in the template above. `lognormal`, `exponential`, and `gauss` are all theoretically unbounded on at least one side, so without a clamp a rare draw can land well past any value the field should realistically take. No. `weighted_choice` and `weighted_choices` treat weights as relative, not as percentages of a fixed total — `{200: 90, 402: 7, 500: 3}` and `{200: 900, 402: 70, 500: 30}` pick with identical odds, since only the ratio between the weights matters. Add up to 100 anyway if it makes the config easier to read; Eventum doesn't require it. When the field should repeat a small, fixed set of realistic values instead of drawing a fresh one on every render — the same pool of usernames or hostnames appearing more than once across a run, the way a real fleet of a few dozen devices would. A [sample](/docs/plugins/event/template/samples) also costs less per event once loaded, since picking a row is cheaper than generating a new `faker` value from scratch, which starts to matter at high event rates. No. As the intro to this section covers, each pick draws a fresh value from a distribution or a weighted set, not from a computation over prior events. That also keeps it cheap at scale: drawing event one million costs the same as drawing event one, since there's no growing history for a distribution or a weighted pick to scan. Related [#related] * The [Realism](/docs/tutorials/realism) pillar for the other axes that make synthetic data behave like production * The [Realistic timing](/docs/tutorials/realism/timing) lesson for shaping when events happen, not just what they contain * The [Modeling sessions](/docs/tutorials/realism/sessions) lesson for tying a sequence of these events to one actor * The [IoT test data](/docs/tutorials/iot-telemetry) tutorial for a Gaussian step applied to value drift in a live sensor stream * The [Log and event formats](/docs/tutorials/formats) field guide for fitting these values into the field names and structures a real format expects * The [modules](/docs/plugins/event/template/modules) reference for every `rand`, `faker`, and `mimesis` function * The [samples](/docs/plugins/event/template/samples) reference for loading and picking from pre-built datasets * Ready-made generators in the [Eventum Hub](/hub) # Synthetic test data: use cases Foundations, formats, realism, and delivery cover synthetic event and log data end to end — what it is, the shape it takes, how to make it realistic, and where it goes. A scenario puts them together into one complete project that solves a real job, from an empty directory to working output. This track covers the situations synthetic test data gets built for most often — proving a pipeline delivers data correctly, generating telemetry for SIEM and detection work, load-testing an API, seeding a database, tracking clickstream and IoT traffic, and firing scheduled alerts. Start with the end-to-end walkthrough below, then jump to whichever scenario matches your job. Related [#related] * The [log and event formats](/docs/tutorials/formats) field guide for the formats each scenario generates. * Ready-made generators for common data sources in the [Eventum Hub](/hub). # Login {/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */} Authenticate user via basic auth and create a server-side session with a HttpOnly cookie # Get Current User {/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */} Get current user # Logout {/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */} Clear user session # Get Info {/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */} Information about app and host # Delete Generator Config {/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */} Delete whole generator configuration directory with specified name. # Get Generator Config {/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */} Get generator configuration in the directory with specified name. # Create Generator Config {/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */} Create generator configuration in the directory with specified name. # Update Generator Config {/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */} Update generator configuration in the directory with specified name. # Restart {/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */} Restart instance # Delete Generator {/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */} Remove generator by its id. Stop it in case it is running. # Get Generator {/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */} Get generator parameters # Add Generator {/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */} Add generator. Note that `id` path parameter takes precedence over `id` field in the body. # Update Generator {/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */} Update generator with provided parameters. Note that `id` path parameter takes precedence over `id` field in the body. # Get Settings {/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */} Get settings # Update Settings {/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */} Update settings. Note that this only updates file. For changes to take effect u have to restart instance. # Stop {/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */} Stop instance # Discover Repositories {/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */} Search the repositories that publish generators in the open. A repository appears in the list by carrying the topic that defines it, and the content of a listed repository is not reviewed. # Remove Repository {/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */} Disconnect the repository with specified name. Generators installed from it are left in place. # Delete Secret Value {/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */} Delete secret with specified name to keyring # Get Secret Value {/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */} Get secret with specified name from keyring # Set Secret Value {/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */} Put secret with specified name to keyring. The name is what a configuration references as `${secrets.}`, so it must be words of lowercase letters, digits and `_`, separated by `.`. # Delete Scenario {/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */} Delete scenario (remove tag from all generators) # Get Scenario {/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */} Get scenario details # Delete Generator From Startup {/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */} Delete generator definition from list in the startup file # Get Generator From Startup {/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */} Get generator definition from list in the startup file # Add Generator To Startup {/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */} Add generator definition to list in the startup file # Update Generator In Startup {/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */} Update generator definition in list in the startup file # Dispatch Sometimes a template should not produce an event — for example, when a session pool is empty or a finite dataset is exhausted. The `dispatch` object provides three actions that let templates control this behavior at render time. dispatch.drop() [#dispatchdrop] Skips the current event entirely. No output is produced for this timestamp, and the `dropped` counter is incremented in the monitoring panel. Picker state (spin index, FSM state, chain position) is **not** rolled back — the pick already happened, only the output is discarded. ```jinja {%- set sessions = shared.get('sessions', []) -%} {%- if sessions | length == 0 -%} {%- do dispatch.drop() -%} {%- endif -%} {%- set sess = sessions.pop(0) -%} {%- do shared.set('sessions', sessions) -%} {"session": "{{ sess.id }}", "timestamp": "{{ timestamp.isoformat() }}"} ``` When there are no sessions to consume, the event is silently dropped instead of producing an incomplete record. In `all` mode, if any template calls `dispatch.drop()`, the output from all templates is discarded — including templates that already rendered successfully before the drop. *** dispatch.next() [#dispatchnext] Discards the current output and asks the generator to pick templates again for the same timestamp. This is useful when the selected template cannot produce a meaningful event but another one might. What happens on re-pick depends on the picking mode: | Mode | What happens | | ---------------- | --------------------------------------------------------- | | `all` | All templates are picked and rendered again from scratch. | | `any` / `chance` | A new random pick is made. | | `spin` / `chain` | The position advances to the next template. | | `fsm` | Transitions are re-evaluated with the updated state. | Parameters [#parameters] | Parameter | Type | Default | Description | | ------------- | ----- | ------- | ------------------------------------------------------------------------- | | `max_repicks` | `int` | `64` | Maximum number of re-pick attempts for one timestamp. Must be at least 1. | If the limit is reached, the event fails with an error. ```jinja {%- set pool = shared.get('pool', []) -%} {%- if pool | length == 0 -%} {%- do dispatch.next(max_repicks=10) -%} {%- endif -%} {%- set item = pool.pop(0) -%} {%- do shared.set('pool', pool) -%} {"item": "{{ item }}"} ``` If every possible template calls `dispatch.next()`, the re-pick limit will be reached and the event will fail. Make sure at least one reachable template can produce output. *** dispatch.exhaust() [#dispatchexhaust] Signals that the generator has finished producing events. The generator shuts down gracefully and no more timestamps are processed. This is useful for finite generation scenarios — for example, replaying a fixed dataset or generating an exact number of correlated sessions. ```jinja {%- set remaining = shared.get('remaining', 100) -%} {%- if remaining <= 0 -%} {%- do dispatch.exhaust() -%} {%- endif -%} {%- do shared.set('remaining', remaining - 1) -%} {"count": {{ remaining }}, "timestamp": "{{ timestamp.isoformat() }}"} ``` In `all` mode, if any template calls `dispatch.exhaust()`, remaining templates in the batch are not rendered. *** State and dispatch [#state-and-dispatch] Dispatch actions do **not** roll back state changes. Any values written to `locals`, `shared`, or `globals` before a dispatch call remain in effect. ```jinja {%- do shared.set('marker', 'written') -%} {%- do dispatch.drop() -%} ``` After this template runs, `shared.get('marker')` returns `'written'` even though no event was produced. **Best practice:** always check preconditions *before* mutating state. If you modify state first and then drop, the change is permanent even though the event was discarded. Recommended pattern [#recommended-pattern] ```jinja {%- set sessions = shared.get('sessions', []) -%} {# Check BEFORE mutating #} {%- if sessions | length == 0 -%} {%- do dispatch.drop() -%} {%- endif -%} {# Safe to mutate - we know sessions is non-empty #} {%- set sess = sessions.pop(0) -%} {%- do shared.set('sessions', sessions) -%} ``` # Examples Parameterized template reuse with vars [#parameterized-template-reuse-with-vars] Multiple template entries can point to the **same** `.jinja` file with different per-template variables. This eliminates duplication when events differ only by a few constants — such as protocol variants: ```yaml title="generator.yml" event: template: mode: chance params: hostname: fw-01 templates: - tcp_built: template: templates/connection-built.json.jinja chance: 220 vars: protocol: tcp iana_number: "6" - udp_built: template: templates/connection-built.json.jinja chance: 75 vars: protocol: udp iana_number: "17" - icmp_built: template: templates/connection-built.json.jinja chance: 15 vars: protocol: icmp iana_number: "1" ``` ```jinja title="templates/connection-built.json.jinja" {%- set src_ip = module.rand.network.ip_v4_private_c() -%} {%- set dst_ip = module.rand.network.ip_v4_public() -%} {%- set src_port = module.rand.number.integer(1024, 65535) -%} {%- set dst_port = module.rand.number.integer(1, 1023) -%} {"timestamp": "{{ timestamp.isoformat() }}", "hostname": "{{ params.hostname }}", "protocol": "{{ vars.protocol }}", "iana_number": {{ vars.iana_number }}, "src": "{{ src_ip }}:{{ src_port }}", "dst": "{{ dst_ip }}:{{ dst_port }}"} ``` One template file, three config entries — instead of three nearly identical template files. Access log with modules, params, and samples [#access-log-with-modules-params-and-samples] A generator that produces HTTP access log lines using random utilities, user-defined parameters, and sample data: ```csv title="samples/users.csv" name,email,role John,john@example.com,admin Jane,jane@example.com,user ``` ```yaml title="generator.yml" event: template: mode: chance params: app_name: my-service server_port: 8080 samples: users: type: csv source: samples/users.csv header: true paths: type: items source: ["/api/users", "/api/orders", "/api/products", "/health"] templates: - access: template: templates/access.jinja chance: 95 - error: template: templates/error.jinja chance: 5 ``` ```jinja title="templates/access.jinja" {%- set user = samples.users | random -%} {%- set path = module.rand.choice(samples.paths) -%} {%- set status = module.rand.weighted_choice([200, 301, 404], [85, 5, 10]) -%} {%- set bytes_sent = module.rand.number.integer(200, 15000) -%} {%- set request_id = module.rand.crypto.uuid4() -%} {{ timestamp.isoformat() }} [{{ params.app_name }}] {{ user.name }} ({{ user.email }}) GET {{ path }} {{ status }} {{ bytes_sent }} {{ request_id }} ``` Monotonic counter with shared state [#monotonic-counter-with-shared-state] All templates in a generator share a single incrementing counter via `shared`. The counter persists across renders regardless of which template is picked: ```yaml title="generator.yml" event: template: mode: all templates: - event_a: template: templates/event_a.jinja - event_b: template: templates/event_b.jinja ``` ```jinja title="templates/event_a.jinja" {%- set id = shared.get('seq', 1) -%} {%- do shared.set('seq', id + 1) -%} {"id": {{ id }}, "type": "A", "timestamp": "{{ timestamp.isoformat() }}"} ``` ```jinja title="templates/event_b.jinja" {%- set id = shared.get('seq', 1) -%} {%- do shared.set('seq', id + 1) -%} {"id": {{ id }}, "type": "B", "timestamp": "{{ timestamp.isoformat() }}"} ``` Since both templates use `shared`, the sequence is global to the generator. If `event_a` renders first and gets `id=1`, then `event_b` gets `id=2`. Cross-template correlation [#cross-template-correlation] Login events store session data in `shared` state that logout events consume — a common pattern for generating correlated event pairs: ```jinja title="templates/login.jinja" {%- set session_id = module.rand.crypto.uuid4() -%} {%- set user = samples.users | random -%} {%- set sessions = shared.get('sessions', []) -%} {%- do sessions.append({"id": session_id, "user": user.name}) -%} {%- do shared.set('sessions', sessions) -%} {{ timestamp.isoformat() }} LOGIN user={{ user.name }} session={{ session_id }} ``` ```jinja title="templates/logout.jinja" {%- set sessions = shared.get('sessions', []) -%} {%- if sessions -%} {%- set session = sessions.pop(0) -%} {%- do shared.set('sessions', sessions) -%} {{ timestamp.isoformat() }} LOGOUT user={{ session.user }} session={{ session.id }} {%- else -%} {{ timestamp.isoformat() }} LOGOUT user=anonymous session=none {%- endif -%} ``` Filtering sample data [#filtering-sample-data] Use `where(**conditions)` to pick rows that match one or more equality conditions in a single call: ```jinja {%- set admin = samples.users.where(role="admin").pick() -%} {{ timestamp.isoformat() }} ADMIN_ACTION user={{ admin.name }} action=config_change ``` Multiple conditions are AND-combined - useful for picking from a narrowed slice of the sample: ```jinja {%- set host = samples.internal_hosts.where(role="server", subnet="servers").pick() -%} ``` If no row matches, supply a `default` to avoid an error on an empty result: ```jinja {%- set maybe_admin = samples.users.where(role="admin").pick(default=None) -%} {%- if maybe_admin -%} {{ maybe_admin.name }} {%- endif -%} ``` For non-equality predicates, fall back to Jinja2's `selectattr` filter or inline `for ... if` loops: ```jinja {%- set category_urls = [] -%} {%- for u in samples.urls if u.category.startswith("api") -%} {%- do category_urls.append(u) -%} {%- endfor -%} {%- set url = category_urls | random -%} ``` # FSM Transitions [#transitions] Each transition defines a target state and a condition: | Parameter | Type | Description | | --------- | ------------------------ | ------------------------------------------------------------------ | | `to` | string | Target template alias. | | `when` | [condition](#conditions) | Condition that must evaluate to `true` for the transition to fire. | Transitions are evaluated **in order** — the first matching transition wins. If no transition matches, the machine stays in the current state. Use `always` as a fallback at the end of a transitions list. ```yaml transitions: # Evaluated first: go to "error" if too many failures - to: error when: gt: shared.fail_count: 10 # Evaluated second: go to "done" if counter hit zero - to: done when: eq: shared.remaining: 0 # Fallback: stay in "active" otherwise - to: active when: always: ``` Conditions [#conditions] Conditions inspect template state variables or event metadata. State is organized into three scopes, and field names follow the pattern `.`: | Scope | Pattern | Description | | --------- | -------------------- | ---------------------------------------------------------------------- | | `locals` | `locals.counter` | State local to the current template. Each template has its own locals. | | `shared` | `shared.status` | Shared across all templates within the same generator. | | `globals` | `globals.session_id` | Global state shared across all generators. | Templates set state values during rendering (e.g., `{% do shared.set("counter", shared.counter + 1) %}`), and transitions read those values to decide the next state. Comparison checks [#comparison-checks] Each takes a mapping of `: `: | Condition | Description | Example | | --------- | ------------------------------------ | -------------------------------------- | | `eq` | Field equals value | `eq: { shared.status: "ready" }` | | `gt` | Field greater than value | `gt: { locals.retries: 3 }` | | `ge` | Field greater than or equal to value | `ge: { shared.score: 100 }` | | `lt` | Field less than value | `lt: { locals.ttl: 0 }` | | `le` | Field less than or equal to value | `le: { shared.attempts: 5 }` | | `matches` | Field matches regex pattern | `matches: { shared.path: "^/api/.*" }` | ```yaml # Transition when a counter exceeds a threshold - to: overloaded when: gt: shared.request_count: 1000 # Transition when status matches a pattern - to: api_handler when: matches: shared.path: "^/api/v[0-9]+/" ``` Length checks [#length-checks] Check the length of a sequence field: | Condition | Description | Example | | --------- | ------------------------- | -------------------------------- | | `len_eq` | Length equals value | `len_eq: { shared.queue: 0 }` | | `len_gt` | Length greater than value | `len_gt: { shared.items: 10 }` | | `len_ge` | Length greater or equal | `len_ge: { shared.batch: 50 }` | | `len_lt` | Length less than value | `len_lt: { shared.buffer: 100 }` | | `len_le` | Length less or equal | `len_le: { shared.errors: 3 }` | ```yaml # Transition when a queue is full - to: flush when: len_ge: shared.event_buffer: 100 # Transition when all items are processed - to: complete when: len_eq: shared.pending_items: 0 ``` Membership checks [#membership-checks] | Condition | Description | Example | | ---------- | ----------------------------- | ------------------------------------------- | | `contains` | Sequence field contains value | `contains: { shared.seen_codes: 500 }` | | `in` | Value is in sequence field | `in: { shared.status: ["error", "fatal"] }` | ```yaml # Transition if a specific error code was encountered - to: handle_server_error when: contains: shared.error_codes: 500 # Transition if current status is one of several values - to: retry when: in: shared.status: ["timeout", "rate_limited", "unavailable"] ``` Timestamp checks [#timestamp-checks] Evaluate the current event timestamp against specified time components. Any subset of components can be provided — unspecified components are ignored. | Condition | Description | | --------- | ---------------------------------------------------- | | `before` | Current timestamp is before the specified time. | | `after` | Current timestamp is at or after the specified time. | Available components: `year`, `month`, `day`, `hour`, `minute`, `second`, `microsecond`. ```yaml # Business hours: different behavior before and after 9 AM - to: peak_traffic when: after: hour: 9 minute: 0 # Switch to end-of-year mode in December - to: year_end_processing when: after: month: 12 day: 1 ``` State and tag checks [#state-and-tag-checks] | Condition | Description | Example | | ---------- | --------------------------- | -------------------------------------------------- | | `defined` | State field exists | `defined: locals.user_id` | | `has_tags` | Event has tag(s) | `has_tags: critical` or `has_tags: [urgent, high]` | | `always` | Always true (unconditional) | `always:` | | `never` | Always false | `never:` | ```yaml # Transition only if a field has been set during rendering - to: authenticated when: defined: shared.auth_token # Transition based on event tags - to: alert when: has_tags: [critical, security] # Unconditional fallback (always goes to this state) - to: default_state when: always: ``` Logic operators [#logic-operators] Combine or negate conditions: | Operator | Description | | -------- | ---------------------------------------------------------- | | `or` | List of conditions (at least 2). True if **any** is true. | | `and` | List of conditions (at least 2). True if **all** are true. | | `not` | Single condition. Inverts the result. | ```yaml # AND: both conditions must be true - to: critical_alert when: and: - gt: shared.error_count: 5 - has_tags: critical # OR: either condition triggers the transition - to: throttle when: or: - gt: shared.request_count: 10000 - gt: shared.error_rate: 0.5 # NOT: transition when the field is NOT in the expected set - to: unknown_method when: not: in: shared.method: ["GET", "POST", "PUT", "DELETE"] # Nested: complex logic - to: escalate when: and: - gt: shared.fail_count: 3 - or: - has_tags: production - gt: shared.severity: 8 ``` *** Examples [#examples] User session flow [#user-session-flow] A user goes through login, browsing, and logout. The session tracks page views in shared state and transitions based on accumulated activity. ```yaml event: template: mode: fsm templates: - login: template: templates/login.jinja initial: true transitions: - to: browse when: always: - browse: template: templates/browse.jinja transitions: # After 10+ page views, user proceeds to checkout - to: checkout when: ge: shared.page_views: 10 # Small chance of early logout (handled in template via random) - to: logout when: eq: shared.should_leave: true # Otherwise keep browsing (stay in current state) - checkout: template: templates/checkout.jinja transitions: - to: logout when: always: - logout: template: templates/logout.jinja transitions: # Start a new session - to: login when: always: ``` In `templates/browse.jinja`, the template updates state: ``` {%- do shared.set("page_views", shared.get("page_views", 0) + 1) -%} {%- do shared.set("should_leave", random.random() < 0.05) -%} {{ timestamp }} GET /page/{{ random.randint(1, 100) }} 200 ``` HTTP request-response with errors [#http-request-response-with-errors] Model a client that sends requests and may get errors, retries, and eventually either succeeds or gives up. ```yaml event: template: mode: fsm templates: - send_request: template: templates/request.jinja initial: true transitions: - to: handle_error when: gt: shared.status_code: 399 - to: success when: le: shared.status_code: 399 - handle_error: template: templates/error.jinja transitions: # Give up after 3 retries - to: give_up when: ge: shared.retries: 3 # Retry - to: send_request when: always: - success: template: templates/success.jinja transitions: - to: send_request when: always: - give_up: template: templates/give_up.jinja transitions: # Reset and start fresh - to: send_request when: always: ``` Time-of-day traffic patterns [#time-of-day-traffic-patterns] Use timestamp conditions to switch between traffic patterns depending on the hour. ```yaml event: template: mode: fsm templates: - night_traffic: template: templates/night.jinja initial: true transitions: - to: morning_ramp when: after: hour: 6 - morning_ramp: template: templates/morning.jinja transitions: - to: peak_traffic when: after: hour: 9 - to: night_traffic when: before: hour: 6 - peak_traffic: template: templates/peak.jinja transitions: - to: evening_wind_down when: after: hour: 17 - evening_wind_down: template: templates/evening.jinja transitions: - to: night_traffic when: after: hour: 22 ``` Authentication flow with state checks [#authentication-flow-with-state-checks] Model a system where login creates a token, authenticated requests use it, and the token can expire. ```yaml event: template: mode: fsm templates: - unauthenticated: template: templates/login_attempt.jinja initial: true transitions: # Template sets shared.auth_token on successful login - to: authenticated when: defined: shared.auth_token # Stay unauthenticated if login fails - authenticated: template: templates/api_call.jinja transitions: # Token expired (template increments request_count) - to: unauthenticated when: gt: shared.request_count: 50 # Error threshold reached - to: locked_out when: and: - gt: shared.consecutive_errors: 5 - has_tags: auth_failure - locked_out: template: templates/locked.jinja transitions: # Cool down, then allow retry - to: unauthenticated when: gt: shared.cooldown_ticks: 10 ``` Multi-stage pipeline with queue tracking [#multi-stage-pipeline-with-queue-tracking] Track items flowing through processing stages using length checks. ```yaml event: template: mode: fsm templates: - ingesting: template: templates/ingest.jinja initial: true transitions: # When batch is full, move to processing - to: processing when: len_ge: shared.batch: 100 - processing: template: templates/process.jinja transitions: # Processing error: some items matched error pattern - to: error_handling when: contains: shared.failed_ids: "FATAL" # All processed - to: flushing when: len_eq: shared.batch: 0 - error_handling: template: templates/error_handler.jinja transitions: - to: processing when: not: contains: shared.failed_ids: "FATAL" - to: flushing when: always: - flushing: template: templates/flush.jinja transitions: - to: ingesting when: always: ``` # template Renders Jinja2 templates with a rich context — timestamps, data-generation modules, samples, and persistent state. This is the most commonly used event plugin and covers the vast majority of synthetic data generation scenarios. For a conceptual walkthrough of template features, see [Producing events](/docs/core/concepts/producing). Common fields [#common-fields] | Parameter | Type | Default | Description | | ----------- | -------------------------------------- | ------- | ----------------------------------------------------------------- | | `mode` | string | — | Required. One of: `all`, `any`, `chance`, `spin`, `chain`, `fsm`. | | `templates` | list of template configs | — | Required. At least one template. | | `params` | mapping | `{}` | Extra parameters accessible in templates via `params`. | | `samples` | mapping of [sample configs](./samples) | `{}` | Named datasets accessible in templates via `samples`. | The `mode` field determines how templates are selected for each incoming timestamp. *** Template context [#template-context] Inside a `.jinja` template, the following variables are available: | Variable | Type | Description | | ------------ | ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `timestamp` | `datetime` | Timezone-aware datetime of the current event. | | `tags` | `tuple[str, ...]` | Tags from the input plugin that produced this timestamp. | | `module` | module provider | Gateway to data-generation libraries and any Python module. See [Modules](./modules). | | `params` | `dict` | User-defined constant parameters from the `params` config field. | | `vars` | `dict` | Per-template variables from the `vars` config field. Each template entry can define its own `vars`, allowing the same `.jinja` file to be reused with different bindings. | | `samples` | sample reader | Named datasets from the `samples` config field. See [Samples](./samples). | | `locals` | state | Per-template state that persists across renders. See [State](./state). | | `shared` | state | State shared across all templates in the same generator. See [State](./state). | | `globals` | state | State shared across all generators, thread-safe. See [State](./state). | | `dispatch` | dispatch API | Control event flow: drop events, restart picking, or signal exhaustion. See [Dispatch API](./dispatch). | | `subprocess` | subprocess runner | Execute shell commands. See [Subprocess](./state#subprocess). | `timestamp` is a standard Python `datetime` object — all its methods work directly: `timestamp.isoformat()`, `timestamp.strftime('%Y-%m-%d')`, `timestamp.year`, `timestamp.hour`, etc. *** Jinja2 extensions [#jinja2-extensions] The template environment loads two Jinja2 extensions automatically: | Extension | What it enables | Example | | ------------------------- | ----------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------- | | `jinja2.ext.do` | Expression statements via `{% do ... %}` — call methods that return nothing without producing output. | `{% do shared.set('count', 0) %}` | | `jinja2.ext.loopcontrols` | `{% break %}` and `{% continue %}` inside `{% for %}` loops. | `{% for u in users %}{% if u.skip %}{% continue %}{% endif %}{{ u.name }}{% endfor %}` | The `do` extension is essential for state management — without it, calling `shared.set(...)` (which returns `None`) would require workarounds like `{% set _ = shared.set(...) %}`. *** Template entry [#template-entry] For `all`, `any`, `spin`, and `chain` modes: | Parameter | Type | Default | Constraints | Description | | ---------- | ------- | ------- | --------------------------------- | --------------------------------------------------------------------------------------------------------------------------------- | | `template` | path | — | Required. Must end with `.jinja`. | Path to the Jinja2 template file. | | `vars` | mapping | `{}` | — | Per-template variables accessible in the template via `vars`. Allows the same template file to be reused with different bindings. | For `chance` mode, add the `chance` field. For `fsm` mode, add `initial` and `transitions`. # Modes The `mode` field determines how templates are selected for each incoming timestamp. all [#all] Every template is rendered for every timestamp. Produces N events per timestamp, where N is the number of templates. ```yaml event: template: mode: all templates: - access_log: template: templates/access.jinja - error_log: template: templates/error.jinja ``` any [#any] A single template is chosen at random (uniform distribution) for each timestamp. ```yaml event: template: mode: any templates: - success: template: templates/success.jinja - error: template: templates/error.jinja ``` chance [#chance] A single template is chosen at random with **weighted probability**. Each template has a `chance` value — the higher the value relative to others, the more likely it is to be selected. | Extra parameter | Type | Constraints | Description | | --------------- | ----- | ------------- | ---------------------------- | | `chance` | float | Required. > 0 | Relative probability weight. | ```yaml event: template: mode: chance templates: - success: template: templates/success.jinja chance: 90 - error: template: templates/error.jinja chance: 10 ``` spin [#spin] Templates are rendered in round-robin order. The first timestamp uses the first template, the second uses the second, and so on, cycling back to the first after the last. ```yaml event: template: mode: spin templates: - request: template: templates/request.jinja - response: template: templates/response.jinja ``` chain [#chain] Templates are rendered in a fixed order defined by the `chain` list. All templates in the chain are rendered for every timestamp, in the specified order. | Extra parameter | Type | Constraints | Description | | --------------- | --------------- | ----------------------------- | ------------------------------------------- | | `chain` | list of strings | Required. At least one alias. | Ordered list of template aliases to render. | ```yaml event: template: mode: chain chain: [login, browse, checkout, logout] templates: - login: template: templates/login.jinja - browse: template: templates/browse.jinja - checkout: template: templates/checkout.jinja - logout: template: templates/logout.jinja ``` fsm [#fsm] Templates represent states in a **finite state machine**. After each timestamp, the machine evaluates transition conditions to decide the next state. This enables stateful event sequences like user sessions or protocol flows. **How it works:** 1. The machine starts at the template marked `initial: true`. 2. That template is rendered, producing an event. During rendering, the template can modify state variables (`locals`, `shared`, `globals`). 3. After rendering, transitions are evaluated **in order** — the first transition whose `when` condition is `true` fires. 4. The machine moves to the state named in the `to` field. 5. On the next timestamp, the new current template is rendered and the cycle repeats. 6. If no transition matches, the machine stays in the current state. | Extra parameter | Type | Default | Constraints | Description | | --------------- | ----------------------------------- | ------- | ------------------------------------ | ------------------------------------- | | `initial` | boolean | `false` | Exactly one template must be `true`. | Marks the starting state. | | `transitions` | list of [transition configs](./fsm) | `[]` | — | Possible transitions from this state. | ```yaml event: template: mode: fsm templates: - idle: template: templates/idle.jinja initial: true transitions: - to: active when: gt: shared.request_count: 0 - active: template: templates/active.jinja transitions: - to: idle when: eq: shared.request_count: 0 ``` For full details on FSM transitions, conditions, and examples, see [FSM](./fsm). # Modules The `module` object provides access to three built-in data-generation libraries and acts as a gateway to any Python package installed in the environment. Access a module by name: `module.`. `module.rand` [#modulerand] Built-in random utilities organized into namespaces. This is the most commonly used module for generating synthetic data. Top-level functions [#top-level-functions] | Function | Signature | Description | | ------------------ | ------------------------------- | --------------------------------------------------------------------------------------------- | | `choice` | `(items) → T` | Return a random item from a sequence. | | `choices` | `(items, n) → list[T]` | Return `n` random items with replacement. | | `weighted_choice` | `(items, weights) → T` | Return a random item with weighted probability. Also accepts a dict: `({item: weight, ...})`. | | `weighted_choices` | `(items, weights, n) → list[T]` | Return `n` items with weighted probability. Also accepts a dict: `({item: weight, ...}, n)`. | | `shuffle` | `(items) → list \| str` | Shuffle elements. Returns a string if the input is a string. | | `chance` | `(prob) → bool` | Return `True` with the given probability (0.0–1.0). | ```jinja {%- set method = module.rand.choice(["GET", "POST", "PUT", "DELETE"]) -%} {# Dict form — keys are items, values are weights #} {%- set status = module.rand.weighted_choice({200: 80, 301: 5, 404: 10, 500: 5}) -%} {# Two-sequence form — also supported #} {%- set status = module.rand.weighted_choice([200, 301, 404, 500], [80, 5, 10, 5]) -%} {% if module.rand.chance(0.05) %}RARE EVENT{% endif %} ``` `rand.number` [#randnumber] | Function | Signature | Description | | ------------- | --------------------------- | ---------------------------------------------------------------------------------- | | `integer` | `(a, b) → int` | Random integer in \[a, b] inclusive. | | `floating` | `(a, b) → float` | Random float in \[a, b]. | | `gauss` | `(mu, sigma) → float` | Random float from a Gaussian (normal) distribution. | | `lognormal` | `(mu, sigma) → float` | Random float from a log-normal distribution (always positive, right-skewed). | | `exponential` | `(lambd) → float` | Random float from an exponential distribution. `lambd` is the rate (1/mean). | | `pareto` | `(alpha, xmin=1.0) → float` | Random float from a Pareto distribution (heavy-tailed, values ≥ `xmin`). | | `triangular` | `(low, high, mode) → float` | Random float from a triangular distribution in \[`low`, `high`] peaking at `mode`. | | `clamp` | `(value, min, max) → float` | Clamp `value` to the range \[`min`, `max`]. | ```jinja {%- set port = module.rand.number.integer(1024, 65535) -%} {%- set latency = module.rand.number.floating(0.1, 2.5) -%} {%- set score = module.rand.number.gauss(50, 10) -%} {# Statistical distributions #} {%- set duration = module.rand.number.lognormal(3.0, 1.0) -%} {%- set wait_time = module.rand.number.exponential(0.5) -%} {%- set file_size = module.rand.number.pareto(1.5, xmin=100.0) -%} {%- set temperature = module.rand.number.triangular(15.0, 35.0, 22.0) -%} {# Combine a distribution with clamp to bound the result #} {%- set bytes_sent = module.rand.number.clamp(module.rand.number.gauss(5000, 2000), 0, 15000) -%} ``` `rand.string` [#randstring] | Function | Signature | Description | | ------------------- | ----------------------- | --------------------------------------------------------------------- | | `letters_lowercase` | `(size) → str` | Random lowercase ASCII letters. | | `letters_uppercase` | `(size) → str` | Random uppercase ASCII letters. | | `letters` | `(size) → str` | Random mixed-case ASCII letters. | | `digits` | `(size) → str` | Random digit characters. | | `punctuation` | `(size) → str` | Random ASCII punctuation characters. | | `hex` | `(size) → str` | Random hex characters (0–9, a–f). | | `pattern` | `(format_string) → str` | Random string built from a printf-like pattern. See specifiers below. | ```jinja {%- set token = module.rand.string.hex(32) -%} {%- set code = module.rand.string.letters_uppercase(6) -%} ``` Pattern specifiers [#pattern-specifiers] `pattern(format_string)` builds a random string by replacing format specifiers. Append `{N}` to a specifier to emit `N` random characters from its set instead of one. Everything else in the pattern is copied verbatim. | Specifier | Character set | | --------- | -------------------------- | | `%a` | lowercase letter (a–z) | | `%A` | uppercase letter (A–Z) | | `%l` | any letter (a–zA–Z) | | `%d` | digit (0–9) | | `%n` | non-zero digit (1–9) | | `%h` | lowercase hex (0–9, a–f) | | `%H` | uppercase hex (0–9, A–F) | | `%p` | ASCII punctuation | | `%w` | word character (a–zA–Z0–9) | | `%%` | literal `%` | ```jinja {%- set order_id = module.rand.string.pattern("ORD-%A{3}-%d{6}") -%} {%- set license_plate = module.rand.string.pattern("%A{2}%d{4}%A{2}") -%} {%- set ssn = module.rand.string.pattern("%n%d{2}-%d{2}-%d{4}") -%} ``` `rand.network` [#randnetwork] | Function | Signature | Description | | ------------------ | ------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `ip_v4` | `() → str` | Random IPv4 address (any range). | | `ip_v4_public` | `() → str` | Random public IPv4 (excludes private and reserved ranges). | | `ip_v4_private` | `() → str` | Random private IPv4 (RFC 1918, any class). | | `ip_v4_private_a` | `() → str` | Random Class A private IPv4 (10.0.0.0/8). | | `ip_v4_private_b` | `() → str` | Random Class B private IPv4 (172.16.0.0/12). | | `ip_v4_private_c` | `() → str` | Random Class C private IPv4 (192.168.0.0/16). | | `ip_v4_in_subnet` | `(cidr: str) → str` | Random IPv4 host address within the given CIDR subnet. | | `ip_v6` | `() → str` | Random IPv6 address (any range). | | `ip_v6_global` | `() → str` | Random global unicast IPv6 (`2000::/3`). | | `ip_v6_link_local` | `() → str` | Random link-local IPv6 (`fe80::/10`). | | `ip_v6_ula` | `() → str` | Random unique local IPv6 (`fc00::/7`). | | `mac` | `(*, oui: str = None, vendor: str = None) → str` | Random MAC address (colon-separated, e.g. `a4:3b:00:ff:12:9e`). With `oui` (`aa:bb:cc` or `aa-bb-cc`), the prefix stays fixed and only the last three bytes vary. With `vendor`, a prefix is picked at random from a built-in OUI table for that vendor (case-insensitive). The arguments are mutually exclusive. | Built-in vendor keys: `apple`, `aruba`, `broadcom`, `cisco`, `dell`, `fortinet`, `hp`, `huawei`, `ibm`, `intel`, `juniper`, `lenovo`, `microsoft`, `mikrotik`, `netgear`, `paloalto`, `samsung`, `tplink`, `ubiquiti`, `vmware`. ```jinja {%- set src_ip = module.rand.network.ip_v4_public() -%} {%- set dst_ip = module.rand.network.ip_v4_private_c() -%} {%- set client_ip = module.rand.network.ip_v4_in_subnet("10.0.1.0/24") -%} {%- set src_ip_v6 = module.rand.network.ip_v6_global() -%} {%- set mac_addr = module.rand.network.mac() -%} {%- set vmware_mac = module.rand.network.mac(oui="00:50:56") -%} {%- set dell_mac = module.rand.network.mac(vendor="dell") -%} ``` `rand.crypto` [#randcrypto] | Function | Signature | Description | | -------- | ---------- | ------------------------------------------------ | | `uuid4` | `() → str` | Random UUID v4. | | `md5` | `() → str` | Random 32-character hex string (MD5-length). | | `sha1` | `() → str` | Random 40-character hex string (SHA-1-length). | | `sha256` | `() → str` | Random 64-character hex string (SHA-256-length). | ```jinja {%- set request_id = module.rand.crypto.uuid4() -%} {%- set checksum = module.rand.crypto.sha256() -%} ``` `rand.datetime` [#randdatetime] | Function | Signature | Description | | ----------- | ------------------------- | ------------------------------------------------------------------------------- | | `timestamp` | `(start, end) → datetime` | Random timestamp in range \[start, end]. Both arguments are `datetime` objects. | Combined example [#combined-example] ```jinja {%- set src_ip = module.rand.network.ip_v4_public() -%} {%- set method = module.rand.weighted_choice(["GET", "POST", "PUT"], [70, 20, 10]) -%} {%- set status = module.rand.weighted_choice([200, 301, 404, 500], [80, 5, 10, 5]) -%} {%- set request_id = module.rand.crypto.uuid4() -%} {%- set bytes_sent = module.rand.number.integer(200, 15000) -%} {{ timestamp.isoformat() }} {{ src_ip }} {{ method }} /api/resource {{ status }} {{ bytes_sent }} {{ request_id }} ``` `module.faker` [#modulefaker] The [Faker](https://faker.readthedocs.io/) library for generating realistic fake data. Access a Faker instance by locale: ```jinja {{ module.faker.locale['en_US'].name() }} {# "John Smith" #} {{ module.faker.locale['en_US'].email() }} {# "john.smith@example.com" #} {{ module.faker.locale['en_US'].ipv4() }} {# "192.168.1.100" #} {{ module.faker.locale['en_US'].user_agent() }} {# "Mozilla/5.0 ..." #} {{ module.faker.locale['de_DE'].city() }} {# "Berlin" #} {{ module.faker.locale['ja_JP'].name() }} {# "山田 太郎" #} ``` Locale instances are created once and cached. Faker supports [hundreds of providers](https://faker.readthedocs.io/en/master/providers.html) — names, addresses, phone numbers, credit cards, companies, dates, user agents, and more. In Jinja2, both dot notation (`module.faker.locale.en_US`) and bracket notation (`module.faker.locale['en_US']`) work. Bracket notation is recommended for locale codes with special characters. `module.mimesis` [#modulemimesis] The [Mimesis](https://mimesis.name/) library for high-performance data generation. It provides three access paths: | Access path | Returns | Description | | ----------------------------- | --------- | ------------------------------------------------------ | | `module.mimesis.locale['en']` | `Generic` | Full Mimesis provider for a locale. | | `module.mimesis.enums` | module | Mimesis enums (`Gender`, `FileType`, `TLDType`, etc.). | | `module.mimesis.random` | module | Mimesis random utilities. | ```jinja {{ module.mimesis.locale['en'].person.full_name() }} {# "John Doe" #} {{ module.mimesis.locale['en'].internet.ip_v4() }} {# "10.0.1.42" #} {{ module.mimesis.locale['en'].address.city() }} {# "San Francisco" #} {{ module.mimesis.locale['ru'].person.full_name() }} {# "Иванов Иван" #} ``` Any Python module [#any-python-module] If the name isn't one of the three built-ins (`rand`, `faker`, `mimesis`), the `module` object imports it from the Python standard library or any package installed in the environment: ```jinja {# Standard library #} {{ module.json.dumps({"event": "test"}) }} {{ module.math.ceil(3.14) }} {{ module.hashlib.sha256(b"data").hexdigest() }} {{ module.base64.b64encode(b"hello").decode() }} {{ module.datetime.timedelta(seconds=30) }} ``` Modules are imported once and cached for subsequent access. If a module is not found, a `KeyError` is raised. # Samples Named datasets loaded once at startup and accessible in templates via `samples.`. Three sample types are available: items [#items] Inline list of values: | Parameter | Type | Constraints | Description | | --------- | ------ | ------------------ | -------------------------- | | `type` | string | Must be `"items"`. | Sample type discriminator. | | `source` | list | At least one item. | Inline list of values. | ```yaml samples: status_codes: type: items source: [200, 201, 301, 404, 500] ``` csv [#csv] Load from a CSV file: | Parameter | Type | Default | Constraints | Description | | ----------- | ------- | ------- | --------------------- | ------------------------------------------------------------------------------------------------------------------------- | | `type` | string | — | Must be `"csv"`. | Sample type discriminator. | | `source` | path | — | Must end with `.csv`. | Path to the CSV file. | | `header` | boolean | `false` | — | Whether the first row is a header. | | `delimiter` | string | `","` | Non-empty. | Column delimiter. | | `quotechar` | string | `'"'` | Single character. | Character used to quote fields containing the delimiter or newlines ([RFC 4180](https://www.rfc-editor.org/rfc/rfc4180)). | ```yaml samples: users: type: csv source: samples/users.csv header: true ``` json [#json] Load from a JSON file (array of objects with consistent keys): | Parameter | Type | Constraints | Description | | --------- | ------ | ---------------------- | -------------------------- | | `type` | string | Must be `"json"`. | Sample type discriminator. | | `source` | path | Must end with `.json`. | Path to the JSON file. | ```yaml samples: endpoints: type: json source: samples/endpoints.json ``` All objects in a JSON sample must have the **same set of keys**. If keys differ between objects, the sample will fail to load with an error. Accessing sample rows [#accessing-sample-rows] Each row in a sample is a tuple-like object. You can pick a random row and access its fields: ```jinja {%- set user = samples.users | random -%} ``` **Named access** — CSV samples with `header: true` and JSON samples expose fields by name, matching CSV column headers or JSON object keys: ```jinja {%- set user = samples.users | random -%} {{ user.name }} {# "John" #} {{ user.email }} {# "john@example.com" #} ``` **Index access** — all sample types (including `items` and CSV without headers) support positional index access: ```jinja {{ user[0] }} {# first field #} {{ user[1] }} {# second field #} ``` Both access styles work on the same row — named access is available whenever headers or keys are present, index access always works. Picking sample rows [#picking-sample-rows] Each sample provides methods for random row selection — both uniform and weighted. Uniform picking [#uniform-picking] Use `pick()` to select a single random row, or `pick_n(n)` to select multiple: ```jinja {%- set user = samples.users.pick() -%} {{ user.name }} {%- set five_users = samples.users.pick_n(5) -%} {%- for u in five_users -%} {{ u.name }} {%- endfor -%} ``` | Method | Signature | Description | | -------- | ---------------------------- | ------------------------------------------------ | | `pick` | `(default: Any = ...) → Row` | Pick a single random row (uniform distribution). | | `pick_n` | `(n: int) → list[Row]` | Pick `n` random rows with replacement. | `pick()` is equivalent to `| random` but is also available as a method for consistency with the weighted API below. Pass `default` to `pick()` to handle empty samples (for example after `where()` filtering): `samples.users.where(role="admin").pick(default=None)`. Without `default`, picking from an empty sample raises an error. `pick_n` returns an empty list on empty samples - no `default` needed. Weighted picking [#weighted-picking] If your sample includes a numeric column for weights, you can pick rows with probability proportional to those weights: ```csv title="samples/services.csv" name,port,protocol,weight HTTP,80,tcp,50 HTTPS,443,tcp,30 SSH,22,tcp,10 DNS,53,udp,10 ``` ```jinja {%- set service = samples.services.weighted_pick('weight') -%} {{ service.name }}:{{ service.port }} {%- set batch = samples.services.weighted_pick_n('weight', 10) -%} ``` | Method | Signature | Description | | ----------------- | ----------------------------------------- | ---------------------------------------------------------------------------------- | | `weighted_pick` | `(weight: str, default: Any = ...) → Row` | Pick a single random row with probability proportional to the named weight column. | | `weighted_pick_n` | `(weight: str, n: int) → list[Row]` | Pick `n` random rows with replacement, weighted by the named column. | The weight column must contain numeric values (integers or floats). Negative values and all-zero weights will produce an error. The weight column remains part of the returned row — you can simply ignore it in your template. `weighted_pick` accepts the same `default` parameter as `pick` to handle empty samples. `weighted_pick_n` returns an empty list on empty samples. Weights are extracted and cached on the first call. Repeated calls to `weighted_pick` with the same column are efficient even at high throughput. Filtering rows [#filtering-rows] Use `where(**conditions)` to filter a sample by one or more equality conditions, then pick from the result: ```jinja {%- set admin = samples.users.where(role="admin").pick() -%} {{ admin.name }} ``` Conditions are AND-combined. The example below picks a server in the `servers` subnet: ```jinja {%- set host = samples.internal_hosts.where(role="server", subnet="servers").pick() -%} ``` `where()` returns a new sample, so all picking methods - `pick`, `pick_n`, `weighted_pick`, `weighted_pick_n` - chain naturally. Calls can also be chained: ```jinja {%- set host = samples.internal_hosts.where(role="server").where(subnet="servers").pick() -%} ``` If no row matches, the filtered sample is empty. Use `pick(default=...)` to handle that case without an error: ```jinja {%- set maybe_host = samples.internal_hosts.where(role="server").pick(default=None) -%} {%- if maybe_host -%} {{ maybe_host.address }} {%- else -%} (no server) {%- endif -%} ``` | Method | Signature | Description | | ------- | ------------------------------ | ------------------------------------------------------------- | | `where` | `(**conditions: Any) → Sample` | Return a new sample of rows matching all equality conditions. | For non-equality predicates (substring match, ranges, etc.), use Jinja2's `selectattr` filter: `samples.users | selectattr("name", "search", "^a") | list`. `where()` covers the common equality case. # State Templates can store and retrieve values that persist across renders. State is organized into three scopes: | Scope | Variable | Visibility | Thread safety | Typical use | | ------ | --------- | --------------------------------- | ------------------- | --------------------------------------------- | | Local | `locals` | Current template only | Single-threaded | Per-template counters, accumulators | | Shared | `shared` | All templates in one generator | Single-threaded | Cross-template coordination, session tracking | | Global | `globals` | All generators in the application | Thread-safe (RLock) | Global counters, inter-generator data | `locals` and `shared` belong to templates. `globals` does not: it is the state of the whole event stage, and the [script](/docs/plugins/event/script) plugin receives the same object in its `produce` function, so a template and a script coordinate through the same keys. State API [#state-api] All three scopes provide the same methods: | Method | Signature | Description | | --------- | --------------------------- | ------------------------------------------------------------------------------ | | `get` | `(key, default=None) → Any` | Get a value. Returns `default` if the key doesn't exist. | | `set` | `(key, value) → None` | Set a value. | | `pop` | `(key, default=None) → Any` | Remove a key and return its value. Returns `default` if the key doesn't exist. | | `update` | `(mapping) → None` | Set multiple values at once from a dict. | | `clear` | `() → None` | Remove all values. | | `as_dict` | `() → dict` | Get a shallow copy of the entire state. | | `[key]` | bracket access | Same as `get(key)`. | The `globals` scope has three additional methods for manual locking: | Method | Description | | ------------------- | ----------------------------------------------------------------------------------------------- | | `acquire()` | Acquire the state lock. | | `release()` | Release the state lock. | | `release_if_held()` | Release every hold the calling thread has on the state lock, and return how many were released. | Individual `globals` operations (`get`, `set`, etc.) are already thread-safe. Use `acquire()` / `release()` only when you need multiple operations to execute atomically — for example, reading a counter and incrementing it without another generator modifying it in between. The lock is never held between events: a hold left behind — including one left by a render that failed before reaching `release()` — is released once the event is over, so it cannot block other generators. The same holds for every plugin that takes the lock, not only templates. Use [Scenarios](/docs/studio/scenarios) in Eventum Studio to visualize which generators read and write global state keys, and to manage global state values at runtime. Previewing events — in Studio, over the API or through MCP — runs against this same state, so a preview that writes a key changes what running generators read. Values written by a preview persist until they are overwritten or cleared. Examples [#examples] **Per-template counter** with `locals`: ```jinja {%- do locals.set('n', locals.get('n', 0) + 1) -%} Event #{{ locals.get('n') }} at {{ timestamp.isoformat() }} ``` **Monotonic record ID** with `shared` — a common pattern where all templates in a generator share a single incrementing counter: ```jinja {%- set record_id = shared.get('record_id', 1) -%} ... use record_id in the event body ... {%- do shared.set('record_id', record_id + 1) -%} ``` **Atomic compound operation** with `globals`: ```jinja {%- do globals.acquire() -%} {%- set total = globals.get('total', 0) -%} {%- do globals.set('total', total + 1) -%} {%- do globals.release() -%} ``` # Subprocess The `subprocess` variable lets you execute shell commands from templates. | Name | Type | Default | Constraints | Description | | --------- | ------- | ------- | ------------- | ------------------------------------------------------------------- | | `command` | string | — | — | Shell command to execute. | | `cwd` | string | `None` | — | Working directory. | | `env` | mapping | `None` | — | Environment variables. | | `timeout` | float | `30.0` | Up to `300.0` | Timeout in seconds. A value that is not positive counts as omitted. | The `run` method returns a result object with three fields: | Field | Type | Description | | ----------- | ------ | ----------------------------------- | | `stdout` | string | Standard output (decoded as UTF-8). | | `stderr` | string | Standard error (decoded as UTF-8). | | `exit_code` | int | Process exit code. | ```jinja {%- set result = subprocess.run('hostname', timeout=5.0) -%} {{ result.stdout | trim }} ``` Limits [#limits] Commands run under fixed ceilings, so one template cannot stall the whole application or exhaust its memory: * Every call is bounded by a timeout — the requested one, 30 seconds when none is given, and 300 seconds at the most. A command that runs longer is terminated together with the processes it started. * Each output stream is captured up to 8 MiB. A command that writes more is terminated as well. In both cases the event is not produced and the reason is written to the generator log. Subprocess calls run synchronously and block event production. Use short timeouts to avoid stalling the pipeline. # Bulk Delete Generators {/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */} Bulk delete several generators # Bulk Start Generators {/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */} Bulk start several generators # Bulk Stop Generators {/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */} Bulk stop several generators # Get Running Generators Stats {/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */} Get stats of all running generators # Export Generator Config {/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */} Export generator directory with specified name as a ZIP archive. Entries named in `exclude` are left out with everything under them. # Copy Generator File {/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */} Copy file from source to destination location inside generator directory with specified name. # Move Generator File {/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */} Move file from source to destination location inside generator directory with specified name. # Get Generator File Tree {/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */} Get file tree of the generator directory with specified name. # Import Generator Config {/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */} Import generator directory with specified name from a ZIP archive. The archive must hold a generator configuration, and the directory holding it becomes the root of the imported generator. # Get Generator Config Path {/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */} Get generator configuration path in the directory with specified name. # Rename Generator Config {/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */} Rename generator configuration directory. Instances that use the configuration are repointed at the new directory and must be stopped beforehand. # Rename Generator {/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */} Rename generator. The definition in the startup file is renamed along with the generator, and scenario membership is kept. # Start Generator {/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */} Start generator by its id # Get Generator Stats {/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */} Get stats of running generator # Get Generator Status {/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */} Get generator status # Stop Generator {/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */} Stop generator by its id # Release Event Plugin {/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */} Release event plugin with freeing acquired resource # Initialize Event Plugin {/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */} Initialize event plugin # Get Catalog {/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */} Get the catalog of generators published by the repository with specified name. The repository is fetched when it has not been fetched yet. # Check Repository {/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */} Check that the repository with specified name answers and publishes the branch or tag it names. # Refresh Catalog {/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */} Fetch the repository with specified name and read its catalog anew. # List Secret References {/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */} List what refers to the secret - the projects whose configuration reads it as `${secrets.}`, and the connected repositories authenticating with it # Rename Secret Value {/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */} Rename secret in keyring. The new name must be words of lowercase letters, digits and `_`, separated by `.`, since a configuration references it as `${secrets.}`. Everything referring to the secret follows: the token is rewritten in the configuration of every project reading it, and every connected repository authenticating with it is repointed at the new name. # Clear Scenario Global State {/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */} Clear global state shared across all event plugins # Get Scenario Global State {/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */} Get global state shared across all event plugins # Update Scenario Global State {/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */} Update global state shared across all event plugins # Rename Scenario {/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */} Rename scenario (rewrite tag in all generators that carry it) # Bulk Delete Generators From Startup {/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */} Bulk delete several generator definitions from list in the startup file # Delete Generator File {/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */} Delete file in specified path inside generator directory with specified name. # Get Generator File {/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */} Read file from specified path inside generator directory with specified name. Set `download` to receive the file as an attachment instead of inline content. # Upload Generator File {/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */} Upload file to specified path inside generator directory with specified name. # Put Generator File {/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */} Put file to specified path inside generator directory with specified name. # Create Generator Directory {/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */} Create directory inside generator directory in specified path. # Produce Events {/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */} Produce events using initialized event plugin # Format Events {/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */} Format events using specified formatter # Generate Timestamps {/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */} Generate timestamps using input plugins # Normalize Versatile Datetime {/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */} Normalize versatile date time expression # Remove Generator From Scenario {/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */} Remove generator from scenario # Add Generator To Scenario {/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */} Add generator to scenario # Delete Scenario Global State Key {/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */} Delete a key from global state # Get Scenario Global State Key {/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */} Get a specific global state key value # Clear Event Plugin Global State {/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */} Clear global state shared across all event plugins # Get Event Plugin Global State {/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */} Get global state shared across all event plugins # Update Event Plugin Global State {/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */} Patch global state shared across all event plugins # Install Generator {/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */} Install the published generator with specified name as a generator directory of the workspace. # Get globals usage for a generator in a scenario {/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */} Detect globals usage in Jinja2 templates and Python scripts via AST analysis. # Delete Event Plugin Global State Key {/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */} Delete a key from global state shared across all event plugins # Clear Template Event Plugin Shared State {/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */} Clear shared state of template event plugin # Get Template Event Plugin Shared State {/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */} Get shared state of template event plugin # Update Template Event Plugin Shared State {/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */} Patch shared state of template event plugin # Delete Template Event Plugin Shared State Key {/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */} Delete a key from shared state of template event plugin # Clear Template Event Plugin Local State {/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */} Clear local state of template event plugin for the specified template by its alias # Get Template Event Plugin Local State {/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */} Get local state of template event plugin for the specified template by its alias # Update Template Event Plugin Local State {/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */} Patch local state of template event plugin for the specified template by its alias # Delete Template Event Plugin Local State Key {/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */} Delete a key from local state of template event plugin for the specified template by its alias