Generator

The central building block of Eventum — a self-contained pipeline that turns timestamps into events and delivers them to one or more destinations.

A generator is the central building block of Eventum. It is a self-contained unit that knows when to produce events, what those events look like, and where to send them. You can run a single generator from the command line or dozens in parallel through the application server.

The three-stage pipeline

Every generator runs a pipeline with three stages connected in sequence:

Input → Event → Output

Each stage is handled by a plugin — a swappable component that you pick and configure in YAML. The stages are deliberately independent: changing when events happen doesn't affect what they contain, and adding a new destination doesn't require touching the schedule or the template.

Input — when events happen

An input plugin produces timestamps — the moments in time when events should occur. Different input plugins offer different scheduling strategies: cron expressions, fixed intervals, evenly spaced ranges, statistical distributions, and more.

A generator can have multiple input plugins working simultaneously. Their timestamps are merged into a single stream before reaching the event stage. Each input plugin can attach tags to its timestamps so the event stage knows which source produced them.

See Scheduling for the full reference.

Event — what events look like

An event plugin takes each timestamp and turns it into one or more event strings. Eventum provides three event plugins:

template — renders Jinja2 templates with access to Faker, Mimesis, random helpers, sample datasets, and persistent state. This is the primary way to generate events and covers most use cases.
script — executes a Python function, giving you full programmatic control when templates aren't enough.
replay — reads events from an existing log file and optionally replaces timestamps.

A generator has exactly one event plugin.

See Producing events for the full reference.

Output — where events go

An output plugin receives rendered events and writes them to a destination — the console, a local file, an HTTP endpoint, or a database.

A generator can have multiple output plugins. Every event is delivered to all configured outputs — you don't need to duplicate generators to write the same data to a file and an HTTP endpoint. Each output plugin can apply a formatter to transform events before writing.

See Outputting events for the full reference.

Generator configuration

A generator is defined by a single YAML file with three top-level keys matching the pipeline stages:

generator.yml

input:
  - cron:
      expression: "* * * * * *"
      count: 1

event:
  template:
    mode: all
    templates:
      - my_event:
          template: templates/event.jinja

output:
  - stdout: {}
  - file:
      path: output/events.log
      flush_interval: 1

input is a list — you can combine multiple input plugins.
event is a single object — one plugin with its settings.
output is a list — events go to every output in the list.

Each plugin is identified by its name (cron, template, stdout, etc.) and configured with a nested object of plugin-specific settings. See Configuration files for the full reference.

Variable substitution

Generator configs support ${params.name} and ${secrets.name} tokens. Parameters are passed at runtime (via CLI flags or the startup config), while secrets are resolved from the encrypted keyring. This lets you reuse the same config across environments without hard-coding values:

output:
  - opensearch:
      hosts:
        - ${params.opensearch_host}
      password: ${secrets.opensearch_password}

See Variables for details.

Execution modes

A generator can run in one of two modes:

Live mode (default)

Events are emitted at the actual moments defined by their timestamps, synchronized with the wall clock. If the input plugin says "event at 12:00:05", the generator waits until 12:00:05 to produce and deliver it.

This mode is designed for:

Simulating real-time traffic against a live system
Feeding a SIEM or monitoring tool with a continuous event stream
Stress-testing a pipeline at a controlled, realistic rate

By default, timestamps in the past are skipped (skip_past: true), so the generator starts producing from "now" forward.

Sample mode

All events are generated as fast as possible, regardless of what the timestamps say. A cron expression that would take an hour in live mode can produce its full output in seconds.

This mode is designed for:

Seeding a database with historical data
Creating test datasets
Backfilling a time range after an outage

Switch between modes with a single flag:

# Live mode (default)
eventum generate --id my-gen --path generator.yml ... --live-mode

# Sample mode
eventum generate --id my-gen --path generator.yml ... --live-mode false

How it works under the hood

Understanding the internal flow isn't required to use Eventum, but it helps when tuning performance or debugging.

A generator runs its pipeline in a dedicated thread. Inside that thread, an async executor manages three concurrent tasks connected by two queues:

The input task collects timestamps from all input plugins, merges them in chronological order, groups them into batches, and (in live mode) waits until the right moment before releasing each batch into the timestamps queue.
The event task reads timestamp batches, calls the event plugin's produce function for each timestamp, and puts the resulting event strings into the events queue.
The output task reads event batches and writes them to all output plugins concurrently, respecting a configurable concurrency limit.

The two queues provide natural backpressure: if the event plugin is slower than the input, the timestamps queue fills up and the input task pauses. If the output is slower than the event plugin, the events queue fills up and event production pauses. This prevents memory from growing unboundedly without dropping events.

How much memory a generator holds

The memory a generator occupies follows the size of a batch rather than the rate of events. A batch is in more than one place at once — one is produced while the previous one is written — and each destination formats its own copy of what it delivers, so the total comes to several times the size of a single batch.

For events of a few hundred bytes and the default batch of 10 000, a generator writing JSON to one destination settles around 20 MiB above what the process itself needs, and every further destination adds about 6 MiB. A batch of 1 000 events brings the whole figure under 2 MiB. Lower batch.size when a generator has to fit into a smaller footprint — queue.max_event_bytes bounds a different part, the batches that pile up in the queue when a destination stops keeping up.

The memory reported for a process is the largest amount it has needed so far, so the figure climbs in steps while the first batches pass through and then stays level. A generator that reaches that level and holds it is behaving as expected.

Batching

Timestamps travel through the pipeline in batches, and a batch is the unit of work for everything downstream: one pass of event rendering, one write per output plugin. Two parameters decide where a batch ends — batch.size, how many timestamps it holds, and batch.delay, the time span of the timestamps it covers.

size is the primary limit. What delay buys is a bound on the lag batching adds to delivery: a batch is released once its last timestamp comes due, so the first event in it waits at most delay seconds past its own moment. Where nothing is waited for, delay has no lag to bound and would only cut the run into batches the size of the event rate, so the batch is formed by size:

Configuration	Mode and situation	What ends the batch
`size` only	any mode	`size` is reached
`size` and `delay`	live, timestamps still ahead of real time	whichever of the two limits is reached first
`size` and `delay`	live, timestamps that have already passed — a generator started on a past range with `skip_past` disabled, or one running behind real time	`size` is reached
`size` and `delay`	sample mode	`size` is reached
`delay` only	any mode	the `delay` window — with no `size` set it is the only limit on how large a batch grows

Tuning parameters

Parameter	Default	What it controls
`batch.size`	`10000`	Max timestamps per batch — larger batches reduce overhead but increase latency
`batch.delay`	`1.0`	Max time span of the timestamps one batch covers — applies to timestamps still ahead of real time in live mode, everything else is grouped by size
`queue.max_timestamp_batches`	`10`	Timestamps queue depth — more buffering between input and event stages
`queue.max_event_batches`	`10`	Events queue depth — more buffering between event and output stages
`queue.max_event_bytes`	`268435456`	Memory the events queue may hold, 256 MiB by default — `null` leaves it unlimited
`max_concurrency`	`100`	Max parallel output write operations
`write_timeout`	`10`	Seconds before canceling a slow output write
`keep_order`	`false`	When `true`, output writes are serialized to maintain strict chronological order (slower but ordered)

These parameters are set either via CLI flags (eventum generate --batch.size 5000) or in the application config under generation.*. See eventum.yml for details.

Running multiple generators

When you need more than one generator — different event types, different schedules, different destinations — use eventum run to launch the full application. Each generator runs in its own thread with independent pipelines, queues, and plugins. They don't share state unless you explicitly use the globals state, available to templates and scripts alike.

The startup config lists all generators and lets you override parameters per generator:

startup.yml

- id: access-logs
  path: generators/access-logs/generator.yml
  live_mode: true

- id: error-logs
  path: generators/error-logs/generator.yml
  live_mode: true

- id: backfill
  path: generators/backfill/generator.yml
  autostart: false
  live_mode: false

Generators with autostart: false are loaded but not started — you can start them later through the REST API or Studio.