cloudformation describe_stacks blocks stack deletion

[anatol-ju]

Describe the bug

I tried to find an answer for this, but it seems it's just a weird behaviour.

I'm running a Lambda to delete CloudFormation stacks using boto3. For this, the Lambda is triggered once a Glue job is finished. If successful, the Lambda should delete the stack containing the Glue job and the stack containing the Lambda afterwards.

When checking AWS Console, I can see that the deletion was triggered (DELETE_IN_PROGRESS), but the stack is returned to CREATE_COMPLETE status within 2 seconds.

Expected Behavior

When triggering deletion of a stack, the waiter (that is calling client.describe_stacks function) should not interfere with the deletion of the stack.

Current Behavior

The deletion is triggered, but the stack returned to it's previous state.

In the Events -> Status reason column in the console, you can see this next to CREATE_COMPLETE:

Export DataLakehouse-dev-Test-JobStack:ExportsOutputRefDataLakehousedevTestJobStack[...] cannot be deleted as it is in use by DataLakehouse-dev-Test-DestroyerStack

Reproduction Steps

This is the code I'm using:

client = boto3.client("cloudformation")
waiter = client.get_waiter("stack_delete_complete")

# os.environ["STACK_NAMES"] = "DataLakehouse-dev-Test-JobStack,DataLakehouse-dev-Test-DestroyerStack"
stack_names = set(os.environ["STACK_NAMES"].split(","))

for stack_name in stack_names:
    client.delete_stack(StackName=stack_name)
    try:
        waiter.wait(StackName=stack_name,
                    WaiterConfig={
                        'Delay': 30,
                        'MaxAttempts': 2
                    })
    except boto3.exceptions.Boto3Error:
        raise

Alternatively, I tried polling the status in a loop with:

stack_status = client.describe_stacks(StackName=active_stack)["Stacks"][0]["StackStatus"]

This had the same effect.

Possible Solution

No response

Additional Information/Context

No response

SDK version used

1.34.139

Environment details (OS name and version, etc.)

macOS, Sonoma 14.5, Python 3.12

[tim-finnigan]

Thanks for reaching out. I tried to reproduce this issue but was unable too. When running your code snippet it successfully deleted my stack. Do your stack events note any reason why this might be happening? (You can see those in the console or run describe_stack_events.)

Also in terms of error handling here, I recommend using except botocore.exceptions.ClientError as error:. If still seeing an issue could you provide the debug logs (with any sensitive info redacted) by adding boto3.set_stream_logger('') to your script?

[anatol-ju]

@tim-finnigan Thanks a lot for checking!
So, it seems the error only happens when you try to delete a stack from inside another stack.
Yesterday I was able to modify my infrastructure code so it deletes the same stack that executes the code. I was hoping it would be possible to do the same from a different stack (so delete a set of stacks as specified and afterwards delete the "destroyer stack").
Maybe there is an underlying limitation from AWS that I'm not aware of.

Regarding stack events, this is the only thing I can find:

In the Events -> Status reason column in the console, you can see this next to CREATE_COMPLETE:
Export DataLakehouse-dev-Test-JobStack:ExportsOutputRefDataLakehousedevTestJobStack[...] cannot be deleted as it is in use by DataLakehouse-dev-Test-DestroyerStack

[tim-finnigan]

Thanks for following up. You can delete stack instances from a stack set as documented here: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/stackinstances-delete.html. Does that address your use case? Otherwise please clarify and refer to this earlier comment:

Also in terms of error handling here, I recommend using except botocore.exceptions.ClientError as error:. If still seeing an issue could you provide the debug logs (with any sensitive info redacted) by adding boto3.set_stream_logger('') to your script?