Running the Server

The API server exposes a REST API for managing workflows, runs, and steps. It also serves the web dashboard.

Example server

The repository includes a complete example server:

//! ironflow API server example.
//!
//! ```sh
//! cargo run -p ironflow-example-server
//! ```
//!
//! The dashboard is served automatically via the `dashboard` feature in `ironflow-api`.
//!
//! Environment:
//! - `IRONFLOW_ENV` (`production`, or `development` to boot without secrets;
//!   unset requires them)
//! - `IRONFLOW_INSECURE_COOKIES` (`1` or `true` drops the `Secure` cookie flag, ignored in production)
//! - `DATABASE_URL` (required in production)
//! - `JWT_SECRET` (required, >= 32 bytes; random per process in development)
//! - `WORKER_TOKEN` (required, >= 32 bytes; random per process in development,
//!   logged at startup for the worker)
//! - `PORT` (default: 3000)
//! - `DASHBOARD_DIR` (optional: overrides the embedded dashboard with a filesystem path)
//! - `ALLOWED_ORIGINS` (comma-separated list; omit to allow same-origin only)
//! - `TRUSTED_PROXIES` (optional: comma-separated IPs or CIDR ranges of the
//!   reverse proxies whose forwarding headers the rate limiters and the HTTPS
//!   redirect believe; unset, the TCP peer is the client and nothing is
//!   redirected to HTTPS)
//! - `TRUSTED_PROXY_HEADER` (default: `x-forwarded-for`; `x-real-ip` reads
//!   only `X-Real-IP` from a trusted proxy, with no fallback between the two)
//! - `WEBHOOK_URL` (optional: outbound webhook for run events)
//! - `ARTIFACTS_DIR` (optional: filesystem root for step artifacts; unset
//!   leaves artifacts disabled and the artifact routes answer `501`)
//! - `ARTIFACT_MAX_BYTES` (optional: per-artifact size limit, default 100 MiB)
//! - `IRONFLOW_DEFAULT_RUN_MAX_COST_USD` (optional: default per-run cost cap in
//!   USD, applied when neither the run creation request nor the workflow
//!   handler declares one; unset means no cap)
//! - `IRONFLOW_MONTHLY_COST_LIMIT_USD` (optional: global cost quota for the
//!   current calendar month in UTC; beyond it, creating a run returns
//!   `429 MONTHLY_BUDGET_EXCEEDED` while in-flight runs continue)
//! - `PURGE_MAX_AGE_DAYS`, `PURGE_MAX_RUNS_PER_WORKFLOW`, `PURGE_DRY_RUN`,
//!   `PURGE_INTERVAL_SECS`, `PROVIDER_ACCOUNT_USAGE_RETENTION_DAYS` and
//!   `SIGNAL_RETENTION_DAYS` (optional: retention of the purger, see
//!   `ServerConfig`)
//! - `IRONFLOW_SEED` (optional: when set to any value, seeds development data
//!   at startup -- users, runs, steps, API keys)
//!
//! The server runs with `ExecutionMode::Workers`: a run woken by a signal, an
//! approval or a human input goes back to `Pending`. Run `ironflow-example-worker`
//! (`examples/worker`) alongside, otherwise woken runs wait in `Pending`.
//!
//! `GET /api/v1/metrics` serves Prometheus metrics (the `prometheus` feature)
//! to an admin session or a `stats_read` API key owned by an admin.

use std::net::SocketAddr;
use std::process;
use std::sync::Arc;

use axum::http::header::{AUTHORIZATION, CONTENT_TYPE};
use axum::http::{HeaderValue, Method};
use tokio::net::TcpListener;
use tower_http::cors::CorsLayer;
use tracing::{info, warn};
use tracing_subscriber::EnvFilter;

use ironflow_api::config::ServerConfig;
use ironflow_api::purger::RunPurger;
use ironflow_api::routes::{RouterConfig, create_router};
use ironflow_api::sse::SseBroadcaster;
use ironflow_api::sse_limiter::SseLimits;
use ironflow_api::state::AppState;
use ironflow_artifacts::blob_store::BlobStore;
use ironflow_artifacts::local::LocalBlobStore;
use ironflow_auth::jwt::JwtConfig;
use ironflow_core::provider::AgentProvider;
use ironflow_core::providers::claude::ClaudeCodeProvider;
use ironflow_engine::artifact::DirectArtifactSink;
use ironflow_engine::budget::BudgetConfig;
use ironflow_engine::engine::{Engine, ExecutionMode};
use ironflow_engine::notify::{AuditLogSubscriber, Event, WebhookSubscriber, WorkflowEventBus};
use ironflow_store::crypto::{KeyRing, SECRET_KEYS_ENV};
use ironflow_store::memory::InMemoryStore;
use ironflow_store::store::Store;
use xtask::seed::{SeedOptions, seed_store};

#[tokio::main]
async fn main() {
    tracing_subscriber::fmt()
        .with_env_filter(
            EnvFilter::try_from_default_env()
                .unwrap_or_else(|_| "info,ironflow=debug".parse().expect("valid filter")),
        )
        .init();

    let config = ServerConfig::from_env().unwrap_or_else(|e| {
        eprintln!("{e}");
        process::exit(1);
    });

    let mut store = InMemoryStore::new();

    let key_ring = KeyRing::from_env().unwrap_or_else(|e| {
        eprintln!("invalid secret key configuration: {e}");
        process::exit(1);
    });

    let has_key_ring = key_ring.is_some();
    match key_ring {
        Some(ring) => {
            info!(
                active_version = ring.active_version(),
                configured_versions = ?ring.versions(),
                "secret store enabled"
            );
            store.set_key_ring(ring);
        }
        None => {
            info!("{SECRET_KEYS_ENV} not set, secret store disabled");
        }
    }

    let store: Arc<dyn Store> = Arc::new(store);

    // A secret encrypted with a key that is no longer configured is
    // unreadable. Fail here rather than at the first workflow that needs it.
    if has_key_ring {
        let status = store.secret_key_status().await.unwrap_or_else(|e| {
            eprintln!("cannot read secret key versions: {e}");
            process::exit(1);
        });

        if !status.is_consistent() {
            let missing: Vec<String> = status.missing.iter().map(|v| v.to_string()).collect();
            eprintln!(
                "secret key versions present in database but missing from configuration: {}\n\
                 set {SECRET_KEYS_ENV} to include them, or rotate before removing a key",
                missing.join(", ")
            );
            process::exit(1);
        }
    }
    if std::env::var("IRONFLOW_SEED").is_ok() {
        info!("IRONFLOW_SEED set, seeding development data...");
        let seed_opts = SeedOptions {
            force: false,
            artifacts_dir: config.artifacts_dir.clone(),
        };
        seed_store(&*store, &seed_opts).await.unwrap_or_else(|e| {
            warn!("seed skipped: {e}");
        });
    }

    let provider = Arc::new(ClaudeCodeProvider::new());

    if !config.cookie_secure {
        warn!("IRONFLOW_INSECURE_COOKIES is set: session cookies are sent without the Secure flag");
    }

    let jwt_config = Arc::new(JwtConfig {
        secret: config.jwt_secret.clone(),
        access_token_ttl_secs: 900,
        refresh_token_ttl_secs: 604800,
        cookie_domain: None,
        cookie_secure: config.cookie_secure,
    });

    let budget = BudgetConfig::from_env();
    info!(
        default_run_max_cost_usd = ?budget.default_run_max_cost_usd,
        monthly_cost_limit_usd = ?budget.monthly_cost_limit_usd,
        "cost guardrails loaded"
    );

    // Go through `build_engine` so the mode is the one the test pins.
    let mut engine = build_engine(store.clone(), provider, budget);
    ironflow_workflows::register_all(&mut engine).expect("failed to register workflows");

    // Artifacts stay off until a storage root is configured. The API and any
    // in-process run then share the same backend, so a file a step produces is
    // downloadable from the same server that stored it.
    let blob_store: Option<Arc<dyn BlobStore>> = config.artifacts_dir.as_ref().map(|dir| {
        info!(
            dir = %dir.display(),
            max_bytes = config.artifact_max_bytes,
            "artifact storage enabled"
        );
        Arc::new(LocalBlobStore::new(dir).max_bytes(config.artifact_max_bytes))
            as Arc<dyn BlobStore>
    });

    if let Some(ref blob) = blob_store {
        engine.set_artifact_sink(Arc::new(DirectArtifactSink::new(
            blob.clone(),
            store.clone(),
        )));
    }

    if let Some(ref webhook_url) = config.webhook_url {
        info!(url = %webhook_url, "registering webhook subscriber");
        engine.subscribe(
            WebhookSubscriber::new(webhook_url),
            &[Event::RUN_STATUS_CHANGED, Event::STEP_FAILED],
        );
    }

    // Every engine event is persisted for `GET /api/v1/audit-logs`: who approved,
    // rejected, paused or resumed what. Log lines and provider account events do
    // not go through the engine publisher, so they are not duplicated here.
    engine.subscribe(AuditLogSubscriber::new(store.clone()), Event::ALL);
    let sse_broadcaster = SseBroadcaster::new();
    let event_sender = sse_broadcaster.sender();
    engine.subscribe(sse_broadcaster, Event::ALL);

    let event_bus = WorkflowEventBus::new();
    engine.set_event_bus(event_bus.clone());

    let engine = Arc::new(engine);

    let cors = build_cors(&config);

    let mut state = AppState::new(
        store.clone(),
        engine.clone(),
        jwt_config,
        config.worker_token.clone(),
        event_sender,
    )
    .with_event_bus(event_bus)
    .with_sse_limits(SseLimits {
        per_principal: config.sse_max_connections_per_principal.map(|n| n as usize),
        global: config.sse_max_connections.map(|n| n as usize),
    });
    if let Some(blob) = blob_store {
        state = state.with_blob_store(blob);
    }

    let shutdown = state.spawn_background_tasks().await;
    tokio::spawn(
        RunPurger::from_config(store.clone(), &config)
            .with_blob_store(state.blob_store.clone())
            .run(shutdown.clone()),
    );
    let router_config = RouterConfig {
        dashboard_dir: config.dashboard_dir.clone(),
        rate_limit_auth: config.rate_limit_auth,
        rate_limit_general: config.rate_limit_general,
        trusted_proxies: config.trusted_proxies.clone(),
        enforce_https: config.is_production,
    };
    // The rate limiters key on the TCP peer address: connect info is required.
    let app = create_router(state, router_config)
        .layer(cors)
        .into_make_service_with_connect_info::<SocketAddr>();

    let addr = format!("0.0.0.0:{}", config.port);
    let listener = TcpListener::bind(&addr).await.expect("bind address");

    info!("==============================================");
    info!("  ironflow server on http://{addr}");
    info!(
        "  environment: {}",
        if config.is_production {
            "production"
        } else {
            "development"
        }
    );
    info!("==============================================");

    axum::serve(listener, app)
        .with_graceful_shutdown(async move {
            tokio::signal::ctrl_c().await.expect("ctrl+c handler");
            info!("shutting down...");
            shutdown.cancel();
        })
        .await
        .expect("serve");
}

/// Builds the server's engine. The server resumes no run itself: a run woken by a
/// signal, an approval, a human input or the Waker goes back to `Pending` and a
/// worker, which owns the agent and the tools, finishes it.
fn build_engine(
    store: Arc<dyn Store>,
    provider: Arc<dyn AgentProvider>,
    budget: BudgetConfig,
) -> Engine {
    Engine::new(store, provider)
        .with_budget_config(budget)
        .with_execution_mode(ExecutionMode::Workers)
}

/// Build CORS layer from config.
///
/// - If `allowed_origins` is set: only those origins are permitted (comma-separated).
/// - If unset: no extra origins are allowed (same-origin only).
///
/// Credentials (cookies) are always allowed so JWT cookies work cross-origin.
fn build_cors(config: &ServerConfig) -> CorsLayer {
    let methods = vec![Method::GET, Method::POST, Method::PUT, Method::DELETE];
    let headers = vec![AUTHORIZATION, CONTENT_TYPE];

    match config.allowed_origins {
        Some(ref raw) => {
            let origins: Vec<HeaderValue> = raw
                .split(',')
                .map(str::trim)
                .filter(|s| !s.is_empty())
                .filter_map(|s| match s.parse::<HeaderValue>() {
                    Ok(v) => Some(v),
                    Err(err) => {
                        warn!(origin = s, %err, "ignoring invalid CORS origin");
                        None
                    }
                })
                .collect();

            info!(?origins, "CORS: allowing configured origins");

            CorsLayer::new()
                .allow_origin(origins)
                .allow_methods(methods)
                .allow_headers(headers)
                .allow_credentials(true)
        }
        None => {
            info!("CORS: no ALLOWED_ORIGINS set, same-origin only");

            CorsLayer::new()
                .allow_methods(methods)
                .allow_headers(headers)
        }
    }
}

#[cfg(test)]
mod tests {
    use super::*;

    #[test]
    fn server_engine_resumes_woken_runs_on_workers() {
        let engine = build_engine(
            Arc::new(InMemoryStore::new()),
            Arc::new(ClaudeCodeProvider::new()),
            BudgetConfig::new(),
        );

        assert_eq!(
            engine.execution_mode(),
            ExecutionMode::Workers,
            "a Local server would resume woken runs itself, without the worker's agent or tools"
        );
    }
}

Environment variables

No secret is built into the binary. JWT_SECRET and WORKER_TOKEN must be at least 32 bytes and must not start with ironflow-dev-; generate them with openssl rand -hex 32. Only IRONFLOW_ENV=development boots without them. Production also requires DATABASE_URL. Any violation aborts at boot with every error listed.

Secret encryption keys

IRONFLOW_SECRET_KEYS holds one or more versioned AES-GCM keys, as version:hex pairs separated by commas. Each key is 64 hex characters (32 bytes):

IRONFLOW_SECRET_KEYS="1:0123...ef,2:fedc...10"
IRONFLOW_SECRET_ACTIVE_KEY_VERSION=2

Every key in the ring can decrypt; only the active one encrypts. Without any key the secret store stays off and workflows reading secrets fail.

IRONFLOW_SECRET_KEY (a single unversioned key) is still accepted and read as version 1, so existing deployments keep working. It is deprecated: when IRONFLOW_SECRET_KEYS is also set, it is ignored with a warning.

The server refuses to start if a stored secret uses a key version absent from the configuration, and names the missing versions. That is the safety net behind the rotation procedure below.

Rotating the encryption key

# 1. Add the new key without activating it, then restart.
#    New secrets stay on version 1; version 2 is merely available.
IRONFLOW_SECRET_KEYS="1:<hexA>,2:<hexB>"
IRONFLOW_SECRET_ACTIVE_KEY_VERSION=1

# 2. Activate version 2, then restart.
#    New secrets use version 2; older ones stay readable.
IRONFLOW_SECRET_ACTIVE_KEY_VERSION=2

# 3. Re-encrypt the existing secrets.
ironflow-cli secret rotate

# 4. Confirm version 1 is no longer used by any secret.
ironflow-cli secret key-status

# 5. Drop version 1, then restart.
IRONFLOW_SECRET_KEYS="2:<hexB>"

Step 1 is kept separate from step 2 so rolling back to the previous deployment stays possible for as long as nothing has been encrypted with the new key.

secret rotate works in batches and is safe to interrupt: secrets already re-encrypted are skipped on the next run, and every secret stays readable throughout. It re-encrypts in place: the ID, key and timestamps of a secret never change. A secret that cannot be decrypted is skipped and reported, and the command exits non-zero.

secret key-status reports which versions are configured, which are used by stored secrets, and which can be retired.

Transport security

Session cookies are HttpOnly, SameSite=Lax and Secure. Browsers treat http://localhost as a secure context, so Secure cookies work in local development. For a plain-HTTP setup on another origin, set IRONFLOW_INSECURE_COOKIES=1; the flag is ignored when IRONFLOW_ENV=production.

In production the example server also sets RouterConfig::enforce_https: TLS terminates at your reverse proxy, and any request that arrives with X-Forwarded-Proto: http is answered with a 308 Permanent Redirect to https://<host><path> (the host comes from X-Forwarded-Host, then Host). The method and body are preserved. Requests without X-Forwarded-Proto, such as in-cluster health probes and worker traffic, are not redirected. Every response carries Strict-Transport-Security.

The redirect reads X-Forwarded-Proto and X-Forwarded-Host only when the TCP peer is listed in TRUSTED_PROXIES. From any other peer these headers are client input: they are ignored and the request is served as is. Set TRUSTED_PROXIES in production, or nothing is redirected (the server logs a warning at startup).

The reverse proxy must therefore be listed in TRUSTED_PROXIES, set X-Forwarded-Proto (and X-Forwarded-Host if it rewrites Host), and overwrite any value sent by the client.

Rate limiting and client IP

An unauthenticated request is rate limited under its client IP: the address of the TCP peer. Serve the router with into_make_service_with_connect_info::<SocketAddr>(), as the example server does; without it the IP limit is skipped and a warning is logged once.

Forwarding headers are written by the client, so they are ignored unless the peer is listed in TRUSTED_PROXIES. Behind a reverse proxy, list it (TRUSTED_PROXIES=10.0.0.0/8). Only the header named by TRUSTED_PROXY_HEADER is read then. With the default, x-forwarded-for, X-Forwarded-For is read from the right, trusted hops are skipped and the first other address is the client. With x-real-ip, X-Real-IP is the client. The other header is never read, not even as a fallback. Without TRUSTED_PROXIES, every user shares the proxy’s bucket.

A request with an API key is counted under that key only once its secret is verified. A secret that does not verify counts on the IP bucket. Unverified secrets are checked at most 10 times per minute per client and key, then answered 429. Only an admin may set an API key’s rate_limit_override (a member gets 403), and it is honoured only for keys owned by an admin.

Sign-in and sign-up are also counted per targeted email, so attempts on one account spread over many addresses are limited too. The two routes have separate counters: a sign-up flood does not lock the account out of sign-in.

Password change (PATCH /api/v1/auth/password) sits behind the same limiter, counted per signed-in user: guesses at the current password beyond RATE_LIMIT_AUTH per minute answer 429 before the password is checked, and each wrong current password is logged with the user id.

Accounts and passwords

A password chosen at sign-up, password change or admin user creation must be 12 to 128 characters, not a common password (nor one padded with digits and symbols, like Password2024!), must not contain the email or username, and must not be repetitive. Otherwise the request answers 400 WEAK_PASSWORD with the rule it broke.

Sign-up answers 204 without a session, whether the email was free or already registered: the dashboard signs in next with the same credentials. A taken username still answers 409 DUPLICATE_USERNAME.

The first account created on an empty database becomes admin. The decision is atomic: concurrent sign-ups on a fresh install cannot each see an empty table, so exactly one of them is admin.

Sign-in on an unknown email runs one Argon2 verification against a throwaway hash, like a wrong password, so the answer time does not tell whether the email has an account. Both answer 401 INVALID_CREDENTIALS and are logged without the email.

Sign-up has no email verification, so it still tells whether an email has an account: a sign-up followed by a sign-in with the chosen password succeeds only if the email was free. When email enumeration matters, build the server without the sign-up feature in production and create accounts as an admin, or keep sign-up behind an invite flow.

API key scopes

A request authenticated with an API key (Authorization: Bearer irfl_...) must hold the scope of the route, otherwise it answers 403 INSUFFICIENT_SCOPE before any lookup, so a key without the scope cannot tell whether a run or a secret exists. The admin scope implies every other scope. Paths below are relative to /api/v1.

Routes that were admin-only stay so: POST /runs, cancel, retry, replay, pause and resume of a run, POST /signals, GET /metrics, and every admin route also need the key’s owner to be an admin, checked at request time, so demoting the owner takes effect at once (403 FORBIDDEN). Demoting a user also removes from each of their keys every scope a member may not hold (only workflows_read, runs_read and stats_read remain) and disables the keys left with no scope; promoting them again does not restore those scopes. Deleting a user deletes their keys, and a key whose owner no longer exists answers 401 TOKEN_REVOKED. Approving, rejecting and answering a gate keep their assignee and delegation rules on top of the scope.

On GET /events, a member session, or a key without the admin scope or owned by a member, only receives run events (run_*, step_*, approval_*, log_line, retry_forced, signal_*). User sign-in, sign-up and sign-out, Provider Account, schedule and workflow pause/resume events reach only an admin session or an admin-scope key owned by an admin. An open GET /events stream re-reads these rights every 30 seconds: it is closed when its key is revoked, disabled, expired or loses runs_read, or when its user or key owner is deleted, and it falls back to run events only when the user is demoted or the key loses its admin scope. Within those 30 seconds, events still go through.

Schedule, template and delegation routes are not admin-only: a key owned by a member reaches them exactly as that member’s dashboard session does. Revoking a delegation still needs to be its delegator or an admin, and listing still shows a non-admin only the delegations they are part of.

Signals and Provider Accounts have their own scopes (signals_send, accounts_read, accounts_manage). Dashboard sessions (JWT) are not scope-checked: only the account’s admin flag applies to them.

Metrics

Built with the prometheus feature, the server exposes GET /api/v1/metrics. It needs an admin session, or an API key with the stats_read (or admin) scope owned by an admin; an unauthenticated request gets 401. Create a key for the scraper (ironflow-cli api-key create prometheus --scope stats_read) and hand it to Prometheus as a bearer token:

scrape_configs:
  - job_name: ironflow
    metrics_path: /api/v1/metrics
    authorization:
      type: Bearer
      credentials_file: /etc/prometheus/ironflow-api-key
    static_configs:
      - targets: ["ironflow:3000"]

Audit log

The example server records every engine event in the audit log read by GET /audit-logs (and the Audit page of the dashboard) through AuditLogSubscriber. A custom server must subscribe it itself, otherwise the audit log stays empty:

engine.subscribe(AuditLogSubscriber::new(store.clone()), Event::ALL);

Log lines and Provider Account events do not go through the engine publisher, so they are neither recorded nor duplicated. Who acted is the entry’s user_id, also present in the payload of approval_granted, approval_rejected (a gate decision or a human input rejection, with its reason), workflow_paused and workflow_resumed. For an API key it is the key’s owner; the event keeps the key name as its label (approved_by, rejected_by). A decision taken by the SLA timer (system:timeout) has no user_id. A pause is recorded on every request, a resume only when the workflow was paused.

GET /events sends a lagged event, whatever its filters, to a client that read slower than events were published: data: {"missed": n} counts the events it lost. Its view is stale; reload it.

Retention

The example server starts a RunPurger built from these variables: it purges old runs, Provider Account usage history and received signals. A custom server must start it itself with RunPurger::from_config, otherwise these variables are ignored:

let shutdown = state.spawn_background_tasks().await;
tokio::spawn(
    RunPurger::from_config(store.clone(), &config)
        .with_blob_store(state.blob_store.clone())
        .run(shutdown.clone()),
);

Running

IRONFLOW_ENV=development cargo run -p ironflow-example-server

The server starts on http://localhost:3000. The dashboard is available at the root URL. In development it generates its secrets and logs the worker token (start workers with WORKER_TOKEN=...). Without IRONFLOW_ENV=development, set JWT_SECRET and WORKER_TOKEN (openssl rand -hex 32): no secret is built in.

Pausing runs and workflows

An administrator can hold work without cancelling it. Every route below needs an admin account (403 otherwise):

An unknown run or workflow answers 404. Pausing a finished, already paused or sub-workflow run, or resuming a run that is not paused, answers 400. The CLI offers the same actions as ironflow-cli run pause|resume <id> and ironflow-cli workflow pause|resume <name>. See Pausing runs and workflows for the lifecycle details.