How do I troubleshoot concurrency that's higher than the message group IDs or Maximum Concurrency value on an AWS Lambda SQS FIFO event source?

Short description

When you use Lambda with an Amazon SQS FIFO event source, the concurrency is equal to or less than the number of MessageGroupIds in the queue.

The following reasons can cause the function's concurrency to be higher than the number of unique and active MessageGroupIds in the queue:

• Multiple event sources are using the same function.
• The retention period is less than the function's runtime.
• Lambda concurrency metrics are higher than the number of MessageGroupIds.
• You use Lambda extensions.

Resolution

Multiple event sources use the same function

Make sure that the function has only one event source. When a function has multiple event sources, another event source might be using the function and cause a higher concurrency.

The retention period is less than the function's runtime

If the retention period that's configured for the queue is less than the function's runtime, then increase the retention period for the queue.

Also, concurrency increases because Lambda doesn't track messages from the same MessageGroupId and polls messages as soon as they're available. To make sure that messages don't expire during the Lambda function operation, increase the SQS FIFO queue retention period. Messages are processed in FIFO order for the MessageGroupId.

Lambda concurrency metrics are higher than the number of MessageGroupIds

Lambda concurrency metrics might be higher than the number of MessageGroupIds because of how metrics are produced. More messages aren't concurrently invoked. Instead, for every invoked message, Lambda increases the concurrency counter by one, and then decreases it when the invoke is completed with a returned response.

For event source mappings, concurrency is the number of Lambda handler operations in parallel. For the Lambda backend, functions that complete a handler code operation and are currently in the post-processing phase count toward the total concurrency.

Also, Lambda treats reserved concurrency similar to event source mappings. It's a best practice not to set reserved concurrency. If you must set reserved concurrency, then set it to a slightly higher value than the number of MessageGroupIds.

Note: New messages are invoked only after the previous message is complete. Messages are always processed in order.

Lambda extensions cause high concurrency

High concurrency can occur when you use Lambda extensions because extensions continue to run after a Lambda operation completes. For example, an extension is attached and didn't call /next, but the runtime called /next.

To check if your Lambda extensions cause high concurrency, take the following actions:

• Monitor the PostRuntimeExtensionsDuration performance metric to determine the cumulative time between the runtime Next API and the last extension Next API.
• Check if your Lambda extensions still run and hold the environment after a Lambda function completes.
• Check if your Lambda instances exceed the function concurrency. When two instances are in the same environment at the same time, throttling occurs. For more information, see Using the Lambda Extensions API to create extensions.
• Compare the ConcurrentExecutions metric against the PostRuntimeExtensionsDuration metric. Check if the maximum ConcurrentExecutions is more than the quota on the Amazon SQS event source. For more information, see Monitoring concurrency.

Related information

Using Lambda with Amazon SQS