# AWS Step Functions for Workflow Orchestration URL: https://www.devopsness.com/blog/aws-step-functions-workflow-orchestration Author: Kiril Urbonas Published: 2026-05-08 Updated: 2026-10-31 Categories: Cloud Tags: AWS, Step Functions, Workflows, Orchestration, Cloud Summary: Step Functions for batch processing, document ingestion, and agentic workflows: the patterns that work, the documented limits that bite, and where we'd reach for something else. --- # AWS Step Functions for Workflow Orchestration AWS Step Functions is the right orchestrator when a workflow has several AWS steps, needs retries and parallelism, and must survive failures halfway through. Use Standard workflows for anything long-running or audited, and Express for short, high-volume runs. Keep computation in Lambda, keep payloads under 256 KiB, and define the machine in CDK rather than raw JSON. > [!VERIFIED] > AWS Step Functions Standard and Express workflows, Amazon States Language (JSONPath), against the Step Functions Developer Guide (quotas, error handling, Inline Map, versions) and the AWS pricing page. Checked October 2026. Typical fits are document ingestion, nightly batch reports, and agentic task runners. This is what works, what bites, and when to pick something else. ## What it actually is A state machine defined in Amazon States Language (ASL), a JSON format. Each state does one thing: invoke a Lambda, call an AWS API directly, wait, branch, fan out. AWS hosts the execution, applies retries, and keeps a per-step history. Think hosted, AWS-native Airflow-lite, billed per state transition. You get a visual execution graph, declarative retries, direct service integrations without glue Lambdas, and no orchestration server to die mid-workflow. You pay with workflow logic in JSON, which gets hard to read past a dozen states. ## Standard vs Express, with real limits | Workflow | Max duration | Billing (us-east-1, Oct 2026) | History | Use it for | |---|---|---|---|---| | Standard | 1 year | $0.000025 per state transition, 4,000 free per month | Kept 90 days, 25,000 events per execution | Multi-minute jobs, human or callback waits, anything you debug after the fact | | Express | 5 minutes | $1.00 per million requests plus duration ($0.00001667 per GB-second at the first tier) | Only via CloudWatch Logs | High-volume, short event processing | The arithmetic is usually reassuring. A pipeline with 6,000 executions a month at 12 transitions each is 72,000 transitions; minus the free 4,000, that is about $1.70 a month. The Lambdas, storage and model calls it orchestrates cost far more. Retries count as transitions, so a retry storm shows up on the bill. Quotas worth knowing before you design (from the quotas page, October 2026): 256 KiB maximum input or output per state, task or execution; 1 MB maximum definition size; 1,000,000 open Standard executions per account per Region; Inline Map runs at most 40 concurrent iterations; and new accounts start with reduced state-transition quotas that AWS raises with usage. ## The patterns that earn their place **Map for bounded parallelism.** A nightly report job can fan out one branch per customer with `MaxConcurrency` set so downstream services are not hammered. Use `ItemProcessor`; the old `Iterator` field is deprecated. In Inline mode, one failed iteration fails the whole `Map`, so put a `Catch` inside the iteration if you want to collect failures instead. Past 40 concurrent iterations or 25,000 history events, switch to Distributed Map. ```json { "ProcessAllCustomers": { "Type": "Map", "ItemsPath": "$.customers", "MaxConcurrency": 10, "ItemProcessor": { "ProcessorConfig": { "Mode": "INLINE" }, "StartAt": "ProcessOne", "States": { "ProcessOne": { "Type": "Task", "Resource": "arn:aws:states:::lambda:invoke", "Parameters": { "FunctionName": "report-render", "Payload.$": "$" }, "Catch": [{ "ErrorEquals": ["States.ALL"], "ResultPath": "$.error", "Next": "MarkFailed" }], "End": true }, "MarkFailed": { "Type": "Pass", "End": true } } }, "End": true } } ``` **Callbacks for long waits.** A `.waitForTaskToken` task pauses until an external system calls `SendTaskSuccess` with the token. An agent runner can use this for multi-minute tool calls without paying for an idle Lambda. Always set `HeartbeatSeconds` so a lost token fails with `States.HeartbeatTimeout` instead of waiting up to a year. **Retry plus Catch on every Task.** A retry without a catch still fails the execution when attempts run out. Add `"JitterStrategy": "FULL"` when many executions retry the same dependency at once. ## A complete machine you can deploy Document ingestion with retries, a catch, a parallel enrichment step, and a failure notification: ```json { "Comment": "Ingest a document: extract, enrich in parallel, then persist", "StartAt": "Extract", "States": { "Extract": { "Type": "Task", "Resource": "arn:aws:states:::lambda:invoke", "Parameters": { "FunctionName": "doc-extract", "Payload.$": "$" }, "ResultSelector": { "text.$": "$.Payload.text" }, "Retry": [{ "ErrorEquals": ["Lambda.TooManyRequestsException", "Lambda.ServiceException", "States.TaskFailed"], "IntervalSeconds": 2, "MaxAttempts": 4, "BackoffRate": 2.0, "JitterStrategy": "FULL" }], "Catch": [{ "ErrorEquals": ["States.DataLimitExceeded"], "Next": "RecordFailure" }, { "ErrorEquals": ["States.ALL"], "Next": "RecordFailure" }], "Next": "Enrich" }, "Enrich": { "Type": "Parallel", "Branches": [ { "StartAt": "Classify", "States": { "Classify": { "Type": "Task", "Resource": "arn:aws:states:::lambda:invoke", "Parameters": { "FunctionName": "doc-classify", "Payload.$": "$" }, "End": true } } }, { "StartAt": "Embed", "States": { "Embed": { "Type": "Task", "Resource": "arn:aws:states:::lambda:invoke", "Parameters": { "FunctionName": "doc-embed", "Payload.$": "$" }, "End": true } } } ], "Catch": [{ "ErrorEquals": ["States.ALL"], "Next": "RecordFailure" }], "Next": "Persist" }, "Persist": { "Type": "Task", "Resource": "arn:aws:states:::dynamodb:putItem", "Parameters": { "TableName": "documents", "Item": { "id": { "S.$": "$$.Execution.Name" } } }, "End": true }, "RecordFailure": { "Type": "Task", "Resource": "arn:aws:states:::sns:publish", "Parameters": { "TopicArn": "arn:aws:sns:eu-west-1:123456789012:ingest-failures", "Message.$": "$.Cause" }, "Next": "Failed" }, "Failed": { "Type": "Fail", "Error": "IngestFailed" } } } ``` `ResultSelector` keeps the payload small. `States.DataLimitExceeded` is listed explicitly because the docs say `States.ALL` cannot catch it. A catch without `ResultPath` replaces the input with the error output, which is why `RecordFailure` can read `$.Cause`. Create it and publish a version: ``` $ aws stepfunctions create-state-machine --name ingestion-pdf \ --definition file://ingest.asl.json \ --role-arn arn:aws:iam::123456789012:role/sfn-ingest --publish $ aws stepfunctions start-execution \ --state-machine-arn arn:aws:states:eu-west-1:123456789012:stateMachine:ingestion-pdf:1 \ --input '{"bucket":"docs","key":"in/report.pdf"}' ``` If the embedding branch is what brought you here, the storage side is covered in [choosing a vector database](/blog/best-vector-databases). ## Failure modes and the exact errors - **`States.DataLimitExceeded`**: a state's input or output passed 256 KiB. Store the payload in S3 and pass the key; trim with `ResultSelector`. Remember `States.ALL` will not catch it. - **`States.Runtime`** with "The JSONPath '$.x' specified for the field 'y.$' could not be found in the input": a path typo or a missing field. It is not retriable and no catch handles it. Use the `TestState` API to check one state against sample input before deploying. - **`States.Permissions`**: the state machine role lacks the IAM action for that integration, for example `lambda:InvokeFunction` or `dynamodb:PutItem`. - **`ExecutionLimitExceeded`** on `StartExecution`: you hit the 1,000,000 open-executions quota, usually from executions stuck on a callback with no heartbeat. - **`ExecutionThrottled` in CloudWatch**: state transitions exceeded the token bucket (5,000 per second in us-east-1, us-west-2 and eu-west-1; 800 elsewhere for Standard). Request a quota increase or batch work into fewer transitions. - **History hits 25,000 events**: the execution fails. Loops and large Inline Maps cause it; continue as a new execution or move to Distributed Map. ## What's awkward, and what to do about it **JSON gets big fast.** Writing state machines in the AWS CDK and synthesizing ASL makes reviews far easier. **Local development is weak.** AWS now labels Step Functions Local "unsupported" and points to the `TestState` API instead so test states with `TestState` and run integration tests in a dev account. **Versioning used to be manual.** It no longer is: publish immutable versions and point callers at an alias. Executions started on version 1 keep running on version 1 while new ones go to version 2, and an alias can split traffic between two versions for a canary. ## Where this breaks Skip Step Functions for a two-step "A then B" chain (an S3 event plus one Lambda is simpler), for very high-volume short work where Express request costs add up faster than an SQS-triggered Lambda chain, or for heavy computation, which belongs in a Lambda or container. Data platforms with lineage and backfill needs fit Airflow, Dagster or Prefect better. If your workflow spans clouds, an AWS-only orchestrator is the wrong layer. For the serverless building block these machines call, see [deploying your first Lambda function](/blog/aws-lambda-deploy-first-serverless-function). ## Questions people ask ### What is the difference between Standard and Express workflows? Standard workflows run up to one year, bill $0.000025 per state transition after 4,000 free each month (us-east-1, October 2026), and keep execution history for 90 days. Express workflows run at most five minutes, bill per request plus duration, and only record history if you configure CloudWatch Logs. Choose Standard for long or audited work and Express for short, high-volume event handling. ### What is the Step Functions payload size limit? 256 KiB of UTF-8 data for any task, state or execution input or output, and it is a hard quota. Exceeding it raises `States.DataLimitExceeded`, which a `States.ALL` catcher does not catch, so name it explicitly. The standard workaround is writing large payloads to S3 and passing only the bucket and key between states. ### Can I run Step Functions locally? Only partially. Step Functions Local still downloads as a JAR or Docker image, but AWS now marks it unsupported and without feature parity; it does not even support the `ItemProcessor` field. AWS recommends the `TestState` API, which runs a single state against your input in the cloud. A practical split is `TestState` for unit checks and a dev account for full runs. ## The call we'd make Use Step Functions for multi-step, async, occasionally failing AWS workflows: Standard by default, Express only for short high-volume paths. Define machines in CDK, put a retry and a catch on every task, name `States.DataLimitExceeded` explicitly, and publish versions behind an alias. If the workflow is two steps, pure computation, or a data platform with lineage needs, pick the simpler or more specialized tool. ## Sources - [AWS Step Functions Developer Guide: Service quotas](https://docs.aws.amazon.com/step-functions/latest/dg/service-quotas.html) - [AWS Step Functions Developer Guide: Handling errors](https://docs.aws.amazon.com/step-functions/latest/dg/concepts-error-handling.html) - [AWS Step Functions Developer Guide: Inline Map state](https://docs.aws.amazon.com/step-functions/latest/dg/state-map-inline.html) - [AWS Step Functions pricing](https://aws.amazon.com/step-functions/pricing/)