How do I troubleshoot a FAILED lifecycle policy or a policy that completed, but images are still available in my EC2 Image Builder lifecycle policy?

Short description

The following are common issues that you might experience with your Image Builder lifecycle policy:

• The lifecycle policy has a FAILED status.
• The lifecycle policy completed, but the AMI, snapshots, or container images are still available.

Resolution

Note: If you receive errors when you run AWS Command Line Interface (AWS CLI) commands, then see Troubleshoot AWS CLI errors. Also, make sure that you're using the most recent AWS CLI version.

The lifecycle policy has a FAILED status

If a lifecycle policy has a FAILED status, then run the following command and check the reason description:
Note: Replace example-lifecycle-execution-id with the execution ID of the lifecycle policy and example-region with the AWS Region.

aws imagebuilder get-lifecycle-execution --example-lifecycle-execution-id --region example-region

Example output:

{
        "state": {
            "status": "FAILED",
            "reason": "Lifecycle execution failed. Reason: One or more resources failed."
        }

To troubleshoot the reason description, check the following:

Check the IAM role
To grant permissions for Image Builder to run lifecycle policies, use the default EC2ImageBuilderLifecycleExecutionRole or create a custom AWS Identity and Access Management (IAM) role. Make sure that your custom IAM role has the correct trust policy and includes the EC2ImageBuilderLifecycleExecutionPolicy managed policy.

Check the tags in the Amazon ECR repositories
For Amazon Elastic Container Registry (Amazon ECR) container images, check if the reason description is similar to the following:

"ImageBuilder is not authorized to perform: ecr:BatchGetImage on resource"

To resolve the preceding error message, add the LifecycleExecutionAccess: EC2 Image Builder tag to your Amazon ECR repositories. For more information, see Lifecycle management prerequisites for EC2 Image Builder images.

The lifecycle policy completed, but the AMI, snapshots, or container images are still available

To troubleshoot a lifecycle policy that's completed, but the AMI, snapshots, or container images are still available, check the following:

Check the lifecycle policy scope
To check the lifecycle policy scope, run the following command:
Note: Replace example-lifecycle-execution-id with the execution ID of the lifecycle policy and example-region with the Region.

aws imagebuilder get-lifecycle-execution --example-lifecycle-execution-id --region example-region

Example output:

{
    "lifecycleExecution": {
        "lifecycleExecutionId": "lce-#",
        "resourcesImpactedSummary": {
            "hasImpactedResources": false 
        }

If your lifecycle policy completed and you receive a false value for hasImpactedResources, then the runtime instance didn't identify any image resources. To resolve this, check the Scope tab of your lifecycle policy. Make sure that the recipes and tags that are selected match any images. Recipes and tags must match an image with new or active and available resources that the lifecycle policy hasn't applied yet.

Check the lifecycle policy rule conditions
Check the lifecycle policy rule conditions to identify the AMI, snapshots, and container images that are marked for deletion or retention. To view policy details, use either the AWS CLI or AWS Management Console.

To use the AWS CLI to view your lifecycle policy details, use the following command. Check the includeResources parameter for true or false. If includeResources shows "false", then the policy resources are retained after the lifecycle policy completes. 
Note: Replace example-policy-arn with the lifecycle policy ARN and example-region with the Region.

aws imagebuilder get-lifecycle-policy --lifecycle-policy-arn example-policy-arn --region example-region

Example output:

"policyDetails": [
            {
                "action": {
                    "type": "DELETE",
                    "includeResources": {
                        "amis": false,
                        "snapshots": false
                    }

To use the AWS Management Console to view your lifecycle policy details, complete the following steps:

1. Open the EC2 Image Builder console.
2. Select your policy, and then navigate to Rules > Rule conditions.
3. Check if the options Deregister associated AMIs and delete their snapshots or Delete associated ECR container images are selected. If those options aren't selected, then the policy resources are retained after the lifecycle policy completes.

Check that the images that belong to EC2 Image Builder are retained based on the lifecycle rules configuration
On the lifecycle policy Delete Rules, you can select By Age or By Count. If you select By Age, then you can select the Retain at least one image per recipe option. This option keeps only the latest available image for each recipe that's affected by this rule. If you select By Count, then you can select to Retain a specific number of images per recipe.

If you selected By Age, then use the following command:
Note: Replace example-policy-arn with the lifecycle policy ARN and example-region with the Region.

aws imagebuilder get-lifecycle-policy --lifecycle-policy-arn example-policy-arn --region example-region
"filter": {
    "type": "AGE",
    "value": 3,
    "unit": "YEARS",
    "retainAtLeast": 1
}

If you selected By Count, then use the following command:
Note: Replace example-policy-arn with the lifecycle policy ARN and example-region with the Region.

aws imagebuilder get-lifecycle-policy --lifecycle-policy-arn example-policy-arn --region example-region
"filter": {
    "type": "COUNT",
    "value": 4
}

For both the By Age and By Count options, you can also retain images based on exclusion rules or images that have a specific tag. For images that have a specific tag, select the option Retain images, AMIs, and snapshots with associated tags when you use the AWS Management Console.

To check for exclusion rules, use the following command:
Note: Replace example-policy-arn with the lifecycle policy ARN and example-tag with the required tag and example-region with the Region.

aws imagebuilder get-lifecycle-policy --lifecycle-policy-arn example-policy-arn --region example-region
"exclusionRules": {
    "tagMap": {
        "RetainImageTag": "example-tag"
    }
}

Make sure that the images, AMI, snapshots, or container images that weren't deleted don't match the Retained Images settings from the lifecycle rules configuration. For example, the Image Builder pipeline has only one output image resource, and the lifecycle policy is configured to Retain at least one image per recipe.

Check that your Image Builder images are tagged correctly
Check if your lifecycle policy uses the tags rule scope and the images, AMI, snapshots, and container images are still available on the account. If these images are available on the account after the policy completed, then check that the images are tagged correctly. Tags from AMIs, recipes, and pipelines aren't supported as a tags rule scope. Also, the lifecycle policy runs based only on the tags from the Image Builder image resources.

Note: Currently, you can't use the AWS Management Console or a pipeline configuration to tag Image Builder image resources.

To tag image resources, use the CreateImage API to create images or use the TagResource API to tag image resources that already exist.

To use the TagResource API to tag an image resource that exists, use the following command:
Note: Replace example-image-resource-arn with the image resource ARN and example-region with the Region.

aws imagebuilder tag-resource --resource-arn example-image-resource-arn --tags KeyName1=Value1,KeyName2=Value2 --region example-region

Check that your AMIs and snapshots that are distributed cross account aren't deleted
Image Builder requires permissions to perform lifecycle actions in destination accounts for associated resources. To grant permissions, create the IAM role that it uses to perform lifecycle actions in destination accounts. You must create this IAM role in the destination account.

For AMIs that fail to delete, make sure that the Ec2ImageBuilderCrossAccountLifecycleAccess role exists for Image Builder cross account lifecycle management on the target accounts. Make sure that this role uses the correct trust policy and has the Image Builder lifecycle managed policy EC2ImageBuilderLifecycleExecutionPolicy. Or, use a custom policy with the required permissions for lifecycle policy execution. For more information, see Create an IAM role for Image Builder cross-account lifecycle management.

For snapshots that fail to delete when the image has already been deleted, delete the snapshots manually.

Note: Lifecycle policies have a limit of 100 concurrent completions and 50 recipes and tags per resource.