AWS Step Functions for Workflow Orchestration

AWS Step Functions is the right orchestrator when a workflow has several AWS steps, needs retries and parallelism, and must survive failures halfway through. Use Standard workflows for anything long-running or audited, and Express for short, high-volume runs. Keep computation in Lambda, keep payloads under 256 KiB, and define the machine in CDK rather than raw JSON.

Typical fits are document ingestion, nightly batch reports, and agentic task runners. This is what works, what bites, and when to pick something else.

What it actually is

A state machine defined in Amazon States Language (ASL), a JSON format. Each state does one thing: invoke a Lambda, call an AWS API directly, wait, branch, fan out. AWS hosts the execution, applies retries, and keeps a per-step history. Think hosted, AWS-native Airflow-lite, billed per state transition.

You get a visual execution graph, declarative retries, direct service integrations without glue Lambdas, and no orchestration server to die mid-workflow. You pay with workflow logic in JSON, which gets hard to read past a dozen states.

Standard vs Express, with real limits

The arithmetic is usually reassuring. A pipeline with 6,000 executions a month at 12 transitions each is 72,000 transitions; minus the free 4,000, that is about $1.70 a month. The Lambdas, storage and model calls it orchestrates cost far more. Retries count as transitions, so a retry storm shows up on the bill.

Quotas worth knowing before you design (from the quotas page, October 2026): 256 KiB maximum input or output per state, task or execution; 1 MB maximum definition size; 1,000,000 open Standard executions per account per Region; Inline Map runs at most 40 concurrent iterations; and new accounts start with reduced state-transition quotas that AWS raises with usage.

The patterns that earn their place

Map for bounded parallelism. A nightly report job can fan out one branch per customer with MaxConcurrency set so downstream services are not hammered. Use ItemProcessor; the old Iterator field is deprecated. In Inline mode, one failed iteration fails the whole Map, so put a Catch inside the iteration if you want to collect failures instead. Past 40 concurrent iterations or 25,000 history events, switch to Distributed Map.

{
  "ProcessAllCustomers": {
    "Type": "Map",
    "ItemsPath": "$.customers",
    "MaxConcurrency": 10,
    "ItemProcessor": {
      "ProcessorConfig": { "Mode": "INLINE" },
      "StartAt": "ProcessOne",
      "States": {
        "ProcessOne": {
          "Type": "Task",
          "Resource": "arn:aws:states:::lambda:invoke",
          "Parameters": { "FunctionName": "report-render", "Payload.$": "$" },
          "Catch": [{ "ErrorEquals": ["States.ALL"], "ResultPath": "$.error", "Next": "MarkFailed" }],
          "End": true
        },
        "MarkFailed": { "Type": "Pass", "End": true }
      }
    },
    "End": true
  }
}

Callbacks for long waits. A .waitForTaskToken task pauses until an external system calls SendTaskSuccess with the token. An agent runner can use this for multi-minute tool calls without paying for an idle Lambda. Always set HeartbeatSeconds so a lost token fails with States.HeartbeatTimeout instead of waiting up to a year.

Retry plus Catch on every Task. A retry without a catch still fails the execution when attempts run out. Add "JitterStrategy": "FULL" when many executions retry the same dependency at once.

A complete machine you can deploy

Document ingestion with retries, a catch, a parallel enrichment step, and a failure notification:

{
  "Comment": "Ingest a document: extract, enrich in parallel, then persist",
  "StartAt": "Extract",
  "States": {
    "Extract": {
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke",
      "Parameters": { "FunctionName": "doc-extract", "Payload.$": "$" },
      "ResultSelector": { "text.$": "$.Payload.text" },
      "Retry": [{
        "ErrorEquals": ["Lambda.TooManyRequestsException", "Lambda.ServiceException", "States.TaskFailed"],
        "IntervalSeconds": 2, "MaxAttempts": 4, "BackoffRate": 2.0, "JitterStrategy": "FULL"
      }],
      "Catch": [{ "ErrorEquals": ["States.DataLimitExceeded"], "Next": "RecordFailure" },
                { "ErrorEquals": ["States.ALL"], "Next": "RecordFailure" }],
      "Next": "Enrich"
    },
    "Enrich": {
      "Type": "Parallel",
      "Branches": [
        { "StartAt": "Classify", "States": { "Classify": { "Type": "Task",
            "Resource": "arn:aws:states:::lambda:invoke",
            "Parameters": { "FunctionName": "doc-classify", "Payload.$": "$" }, "End": true } } },
        { "StartAt": "Embed", "States": { "Embed": { "Type": "Task",
            "Resource": "arn:aws:states:::lambda:invoke",
            "Parameters": { "FunctionName": "doc-embed", "Payload.$": "$" }, "End": true } } }
      ],
      "Catch": [{ "ErrorEquals": ["States.ALL"], "Next": "RecordFailure" }],
      "Next": "Persist"
    },
    "Persist": {
      "Type": "Task",
      "Resource": "arn:aws:states:::dynamodb:putItem",
      "Parameters": { "TableName": "documents", "Item": { "id": { "S.$": "$$.Execution.Name" } } },
      "End": true
    },
    "RecordFailure": {
      "Type": "Task",
      "Resource": "arn:aws:states:::sns:publish",
      "Parameters": { "TopicArn": "arn:aws:sns:eu-west-1:123456789012:ingest-failures", "Message.$": "$.Cause" },
      "Next": "Failed"
    },
    "Failed": { "Type": "Fail", "Error": "IngestFailed" }
  }
}

ResultSelector keeps the payload small. States.DataLimitExceeded is listed explicitly because the docs say States.ALL cannot catch it. A catch without ResultPath replaces the input with the error output, which is why RecordFailure can read $.Cause. Create it and publish a version:

aws stepfunctions create-state-machine --name ingestion-pdf \
    --definition file://ingest.asl.json \
    --role-arn arn:aws:iam::123456789012:role/sfn-ingest --publish
aws stepfunctions start-execution \
    --state-machine-arn arn:aws:states:eu-west-1:123456789012:stateMachine:ingestion-pdf:1 \
    --input '{"bucket":"docs","key":"in/report.pdf"}'

If the embedding branch is what brought you here, the storage side is covered in choosing a vector database.

Failure modes and the exact errors

• States.DataLimitExceeded: a state's input or output passed 256 KiB. Store the payload in S3 and pass the key; trim with ResultSelector. Remember States.ALL will not catch it.
• States.Runtime with "The JSONPath '$.x' specified for the field 'y.$' could not be found in the input": a path typo or a missing field. It is not retriable and no catch handles it. Use the TestState API to check one state against sample input before deploying.
• States.Permissions: the state machine role lacks the IAM action for that integration, for example lambda:InvokeFunction or dynamodb:PutItem.
• ExecutionLimitExceeded on StartExecution: you hit the 1,000,000 open-executions quota, usually from executions stuck on a callback with no heartbeat.
• ExecutionThrottled in CloudWatch: state transitions exceeded the token bucket (5,000 per second in us-east-1, us-west-2 and eu-west-1; 800 elsewhere for Standard). Request a quota increase or batch work into fewer transitions.
• History hits 25,000 events: the execution fails. Loops and large Inline Maps cause it; continue as a new execution or move to Distributed Map.

What's awkward, and what to do about it

JSON gets big fast. Writing state machines in the AWS CDK and synthesizing ASL makes reviews far easier. Local development is weak. AWS now labels Step Functions Local "unsupported" and points to the TestState API instead so test states with TestState and run integration tests in a dev account. Versioning used to be manual. It no longer is: publish immutable versions and point callers at an alias. Executions started on version 1 keep running on version 1 while new ones go to version 2, and an alias can split traffic between two versions for a canary.

Where this breaks

Skip Step Functions for a two-step "A then B" chain (an S3 event plus one Lambda is simpler), for very high-volume short work where Express request costs add up faster than an SQS-triggered Lambda chain, or for heavy computation, which belongs in a Lambda or container. Data platforms with lineage and backfill needs fit Airflow, Dagster or Prefect better. If your workflow spans clouds, an AWS-only orchestrator is the wrong layer. For the serverless building block these machines call, see deploying your first Lambda function.

Questions people ask

What is the difference between Standard and Express workflows?

Standard workflows run up to one year, bill $0.000025 per state transition after 4,000 free each month (us-east-1, October 2026), and keep execution history for 90 days. Express workflows run at most five minutes, bill per request plus duration, and only record history if you configure CloudWatch Logs. Choose Standard for long or audited work and Express for short, high-volume event handling.

What is the Step Functions payload size limit?

256 KiB of UTF-8 data for any task, state or execution input or output, and it is a hard quota. Exceeding it raises States.DataLimitExceeded, which a States.ALL catcher does not catch, so name it explicitly. The standard workaround is writing large payloads to S3 and passing only the bucket and key between states.

Can I run Step Functions locally?

Only partially. Step Functions Local still downloads as a JAR or Docker image, but AWS now marks it unsupported and without feature parity; it does not even support the ItemProcessor field. AWS recommends the TestState API, which runs a single state against your input in the cloud. A practical split is TestState for unit checks and a dev account for full runs.

The call we'd make

Use Step Functions for multi-step, async, occasionally failing AWS workflows: Standard by default, Express only for short high-volume paths. Define machines in CDK, put a retry and a catch on every task, name States.DataLimitExceeded explicitly, and publish versions behind an alias. If the workflow is two steps, pure computation, or a data platform with lineage needs, pick the simpler or more specialized tool.

Sources

• AWS Step Functions Developer Guide: Service quotas
• AWS Step Functions Developer Guide: Handling errors
• AWS Step Functions Developer Guide: Inline Map state
• AWS Step Functions pricing