I Built an Async Notification Flow That Called Back Into Itself

The problem I was solving

I was working on a notification flow where customers had to confirm that they had seen a message and followed its instructions. The confirmation came through a public API on our website.

Accepting the confirmation was quick, but the follow-up work was not. The backend needed to create snapshots, perform expensive computation, call another service, and write more records to DynamoDB. I did not want the customer waiting for all of that work behind a spinner.

Making the follow-up asynchronous was the right decision. The API could persist the confirmation, return 200 OK, and let the expensive processing happen later. Eventual consistency was acceptable for this feature.

The question was not whether the work should be asynchronous. The question was how to hand it off reliably without making the system unnecessarily difficult to operate.

What I built

The request first reached Service A. Service A wrote a confirmation record to DynamoDB and returned a successful response to the customer. Changes from the table were captured in Kinesis Data Streams, and Lambda consumed those records in batches.

For each confirmation, the Lambda called an internal endpoint on Service A. Service A then performed the snapshots, computation, downstream service call, and additional DynamoDB writes.

At first glance, this was reasonable:

• the customer received a fast response
• the confirmation was durable before processing began
• Kinesis buffered the asynchronous work
• Lambda supplied managed consumption and retries
• Service A retained the business logic for processing the confirmation

The Lambda was not performing the expensive work itself. It translated a Kinesis record into a request back to the service that had produced the record.

The runtime path returned to Service A. That was not automatically incorrect, but in this case the detour added more operational cost than isolation.

Why Kinesis was part of the design

Recovery time was an important requirement. Native DynamoDB Streams retain records for 24 hours. That was not enough time for an alarm to create an operational ticket, for the on-call engineer to investigate the failure, and for the team to deploy a safe fix.

We configured Kinesis Data Streams with its maximum 365-day retention period. When Lambda exhausted its processing attempts, its SQS on-failure destination received metadata for the failed Kinesis batch. That metadata included the stream, shard, and sequence-number range rather than the original records.

The Lambda had a separate recovery mode for messages redriven from SQS. It used the stored Kinesis metadata to retrieve the original records and process them again before retention expired.

That recovery path explains why Kinesis was not simply replaceable with the native DynamoDB stream in the original design. The longer retention gave the team a much larger recovery window.

It also created a deadline: the SQS message did not contain the full source records, so recovery still depended on those records remaining in Kinesis.

Where the design became painful

One failed confirmation retried the batch

Lambda received a batch from Kinesis and made multiple requests to Service A. If one request failed, the Lambda invocation failed. The event-source mapping could then retry records that had already been processed successfully.

As I remember it, the DynamoDB writes used versioning to tolerate reprocessing. That made persistence safer, but database idempotency does not automatically make every downstream call or other external side effect idempotent. A replay-safe workflow needs that guarantee at every boundary, not only at the final write.

The Lambda was mostly an adapter

The Lambda read Kinesis records, extracted confirmation data, and called Service A. It needed deployment configuration, IAM permissions, networking, authentication, alarms, and a recovery mode, even though the domain work still lived elsewhere.

An adapter is not inherently bad. In this case, however, it did not create a useful ownership boundary. It converted an event back into an internal request to the event's producer.

Debugging crossed too many systems

Answering “what happened to this confirmation?” could require correlating:

• the original Service A request
• the DynamoDB confirmation
• the Kinesis record and shard position
• the Lambda invocation and batch
• the internal callback to Service A
• the downstream Service B request
• the SQS failure metadata

The architecture had durable components, but the end-to-end operation was not represented as one durable execution that an engineer could inspect.

Background work competed with customer traffic

The Lambda callbacks returned to the same Service A deployment that handled customer requests. During a burst, background processing and public traffic competed for the same request-handling capacity and downstream connections.

The asynchronous boundary improved customer response time, but it did not isolate the expensive work from the customer-facing service.

What I would consider now

I did not implement a replacement, so I cannot honestly present one architecture as the proven answer. Looking back, I would evaluate two designs based on the shape of the work.

Option 1: a queue-backed worker

If the follow-up is fundamentally one unit of work owned by Service A, a dedicated worker is the simpler option.

The API and worker can share domain code while running with separate capacity. The worker consumes a durable command directly instead of receiving an event through Lambda and converting it into an HTTP callback.

This design still needs a reliable handoff. Writing a confirmation and sending an SQS message as two unrelated operations would introduce a failure gap. A transactional outbox or another durable publisher would be needed so that a successful confirmation cannot lose its background work.

I would favor this option when:

• the processing is one cohesive operation
• Service A should continue owning its business rules and data
• separate worker capacity provides enough isolation
• step-level execution history is not required

Option 2: Step Functions

If the follow-up is better understood as several independently retryable steps, Step Functions becomes more compelling.

The main benefit would not be eliminating Lambda at all costs. It would be representing the operation as a durable workflow with visible state, step-specific retries, and an execution history.

Step Functions could use AWS service integrations for workflow-owned DynamoDB state and small AWS API operations. That does not mean it should automatically write directly to Service A's private tables. Doing so could move coupling from an internal API contract to a database-schema contract and bypass business rules owned by the service.

Some steps might still require Lambda, a worker, or another compute service. That is reasonable when the step performs real domain work or adapts an external protocol.

I would favor this option when:

• the operation has several meaningful stages
• stages need different retry or timeout policies
• operators need to see exactly which stage failed
• partial completion and compensation must be modeled explicitly
• the workflow crosses service boundaries

How I would choose

The original architecture was defensible under time pressure, especially because recovery was a real requirement. My mistake was treating a collection of durable AWS components as if that automatically produced a simple, observable workflow.

The principles I took away

Async processing should create a useful boundary

Asynchronous processing decouples response time from processing time. Additional hops are worthwhile only when they improve durability, isolation, ownership, or operability.

A callback into the producer deserves scrutiny, not an automatic ban

An internal callback can preserve service ownership and business rules. It becomes questionable when the adapter adds no useful transformation or isolation and makes the operation harder to trace.

Batch retries make idempotency an end-to-end requirement

One failed record can cause successful work to run again. Versioned database writes help, but snapshots and downstream calls also need stable operation identifiers or their own replay protection.

Recovery design is part of the architecture

The 365-day Kinesis retention and sequence-number recovery mode were not incidental details. They existed because on-call engineers needed enough time to diagnose, fix, and redrive failures safely.

Choose orchestration according to the work

A worker is often enough for one durable background operation. A workflow engine becomes valuable when the process has meaningful stages, independent failure policies, or compensating actions.

References

• DynamoDB Streams retention
• Kinesis Data Streams retention
• Lambda failure handling for Kinesis event sources
• Using Kinesis Data Streams to capture DynamoDB changes
• EventBridge Pipes targets
• Step Functions DynamoDB integrations