Alert Simulation: Scheduled Telegram Alerts

A critical alert that never reaches the on-call phone, a dashboard tile that miscounts what arrived, an escalation that never fires on an unacknowledged warning — each of these bugs waits for a real incident to surface, and surfaces at the cost of one. Firing realistic alerts on a schedule surfaces them on demand instead, exercising every piece downstream of the trigger: the chat, the paging rotation, the dashboard count, the escalation path.

The generator below fires exactly those alerts — randomized severity, service, and metric value, on a cron schedule — and delivers each over HTTP to a live Telegram chat through the Bot API. A miniature alerting pipeline in a few files, hitting a real notification channel instead of a mock.

Alert simulation and the pipeline it exercises

A detection rule that matches, a threshold that trips, or a health check that fails all produce the same next step: an alert, structured no differently from any other event Eventum generates — a severity, a service, a message, a timestamp. Testing that a rule fires correctly says nothing about what happens after — the notification routing, on-call paging, dashboard counting, and escalation the opening paragraph lists, everything downstream of the alert itself. A real incident exercises that whole chain eventually — at whatever severity and service it happens to produce.

Firing realistic alerts on a schedule exercises the same chain on demand, at whatever severity mix and service spread a test needs, instead of waiting on a rule to match or an incident to happen. Telegram is one concrete chat to land these in — nothing below depends on Telegram past the request body and the endpoint URL, and the same pipeline reaches Slack, Discord, or PagerDuty just as directly.

What you'll build

The generator uses:

• cron input — fires every 2 minutes.
• all picking mode — renders the one alert template on every tick, using conditional Jinja2 logic to vary message format and value range by severity.
• HTTP output — posts each rendered alert to the Telegram Bot API as its own request. See Send test data to an API endpoint for the delivery mechanics behind it.
• Secrets — bot token stored securely in the keyring.
• Parameters — chat ID passed through startup.yml.

Prerequisites

• Eventum installed
• A Telegram bot token (create one via @BotFather)
• The chat ID of the target chat (send a message to your bot, then query https://api.telegram.org/bot<TOKEN>/getUpdates to find it)

Project structure

Build it

mkdir -p eventum/generators/alerts/templates
cd eventum

{% set severity = module.rand.weighted_choice(["info", "warning", "critical", "resolved"], [0.4, 0.35, 0.2, 0.05]) %}
{% set service = module.rand.choice(["api-gateway", "auth-service", "db-primary", "cache-redis", "worker-queue", "nginx-lb"]) %}
{% set icons = {"info": "ℹ️", "warning": "⚠️", "critical": "🔴", "resolved": "✅"} %}
{% set metric = module.rand.choice(["CPU usage", "memory usage", "request latency", "error rate", "disk I/O", "connection pool"]) %}
{% if severity == "critical" %}
{% set value = module.rand.number.integer(90, 100) %}
{% elif severity == "warning" %}
{% set value = module.rand.number.integer(70, 89) %}
{% else %}
{% set value = module.rand.number.integer(10, 69) %}
{% endif %}
{
  "chat_id": {{ params.chat_id }},
  "parse_mode": "Markdown",
  "text": "{{ icons[severity] }} *{{ severity | upper }}* | `{{ service }}`\n\n{{ metric }}: *{{ value }}%*\nHost: `{{ module.faker.locale.en.hostname() }}`\nTime: {{ timestamp.strftime('%H:%M:%S UTC') }}"
}

input:
  - cron:
      expression: "*/2 * * * * 0"
      count: 1

event:
  template:
    mode: all
    params:
      chat_id: ${params.chat_id}
    templates:
      - alert:
          template: templates/alert.jinja

output:
  - stdout: {}
  - http:
      url: "https://api.telegram.org/bot${secrets.telegram_token}/sendMessage"
      method: POST
      success_code: 200
      headers:
        Content-Type: "application/json"
      formatter:
        format: plain

server:
  host: "0.0.0.0"
  port: 9474

path:
  startup: /home/user/eventum/startup.yml
  generators_dir: /home/user/eventum/generators
  logs: /home/user/eventum/logs
  keyring_cryptfile: /home/user/eventum/cryptfile.cfg

- id: alerts
  path: alerts/generator.yml
  params:
    chat_id: "123456789"

eventum-keyring set telegram_token

eventum run -c eventum.yml

{
  "chat_id": 123456789,
  "parse_mode": "Markdown",
  "text": "⚠️ *WARNING* | `api-gateway`\n\nconnection pool: *81%*\nHost: `srv-41.cole-williams.com`\nTime: 13:34:09 UTC"
}

Going further

• Any HTTP-reachable destination — as the intro notes, the request body and the URL are the only Telegram-specific parts. Point the same http output at a Slack Incoming Webhook, a Discord Webhook, or a PagerDuty Events API integration instead, reshaping the template's JSON to match the destination's expected payload. See Send test data to an API endpoint for the delivery mechanics this generalizes.
• Escalation schedule — use multiple inputs with different cron expressions: check every 2 minutes during business hours, every 10 minutes overnight.
• Incident sequences — switch to FSM mode to model alert → acknowledged → investigating → resolved workflows.
• Silence window — use the cron start/end fields to suppress alerts during maintenance windows.

What's next

FAQ

Related

• The Test Sigma rules with synthetic attack telemetry tutorial for generating the detection match an alert like this one would normally be triggered by
• The Logs vs metrics vs events lesson for treating an alert as a structured event like any other, not a special case
• The Send test data to an API endpoint lesson for the success_code, formatter, and batching mechanics behind the HTTP delivery this scenario relies on
• The Secrets reference for the keyring and ${secrets.*} substitution storing the bot token
• The Scenarios track for the wider picture of applied synthetic data
• Ready-made generators in the Eventum Hub