MicroservicesAutomation Worker

Automation Worker

The Automation Worker is the background runtime for persisted automation runs. The backend creates each run and its immutable context, then places a small wake-up job containing the run ID and a stable claim ID on the automations BullMQ queue.

The worker does not connect to Postgres and does not expose HTTP endpoints. It uses the backend’s internal NestJS TCP receiver for persistence and backend-owned domain actions.

Queue Endpoint

QueueJob nameProducerConsumerPurpose
automationsexecute-runBackendAutomation WorkerWake the worker for a persisted run

The queue payload contains runId and claimId. The stable claim ID lets a BullMQ retry recover a backend claim when the original TCP response was lost.

Backend RPC Endpoints

The worker connects to BACKEND_MICROSERVICE_TARGET_HOST and BACKEND_MICROSERVICE_TARGET_PORT. The backend listens on port 3103 by default.

PatternFunction
automation.runtime.run.prepareNormalize due state and claim or recover a run
automation.runtime.plan.getLoad the action plan and check for resolved context
automation.runtime.action.executeExecute a backend-owned action or prepare an outbound request
automation.runtime.attempt.latestRead the latest persisted attempt for an action
automation.runtime.attempt.createCreate a durable action attempt
automation.runtime.attempt.updatePersist an action attempt result
automation.runtime.run.advanceAdvance the run cursor
automation.runtime.run.waitStore a durable WAIT checkpoint
automation.runtime.run.succeedMark a run successful
automation.runtime.run.failMark a run permanently failed
automation.runtime.run.retrySchedule the next retry timestamp
automation.runtime.run.dead-letterMark an exhausted run as dead-lettered
automation.runtime.run.notify-failedRequest the backend’s failure notification

Each RPC has a bounded deadline. AUTOMATIONS_BACKEND_RPC_TIMEOUT_MS defaults to 75000. A timeout is returned to BullMQ so the next job attempt can reconcile the persisted checkpoint before doing more work.

Functionality

  • Claims work with stable ownership and resumes persisted checkpoints.
  • Executes actions in their configured order.
  • Handles WAIT actions and delayed continuations.
  • Applies action retries and marks exhausted runs as dead-lettered.
  • Executes prepared FORWARD_REQUEST calls with per-target rate limiting, timeouts, response masking, and circuit breaking.
  • Delegates SEND_MAIL, TRIGGER_BROADCAST, and TOGGLE_ROOM_STATE actions to the backend.

Example: Automation Sends Mail

  1. The backend creates the automation run and enqueues execute-run.
  2. The worker claims the run over automation.runtime.run.prepare.
  3. The worker asks the backend to execute the SEND_MAIL action.
  4. The backend resolves variables, renders the template, prepares attachments and tracking, then enqueues a prepared message on the mail queue.
  5. The Mail Gateway sends the message and publishes its result through mail-results.
  6. The backend persists the result and returns the action outcome to the worker.
  7. The worker advances the cursor or schedules a retry.

Local Start

Start Valkey and the backend first, then run:

pnpm automation-worker:dev

Use packages/automation-worker/.env.example as the local environment template.

OverviewMail Gateway