Fargate Storage Encyrpted

I tried above policy and updated the cluster with the same key but still getting the same error. Then I checked in Event history then I see this(check the attached screenshot ). Then I also checked the Event Record -

{ "eventVersion": "1.09", "userIdentity": { "type": "AssumedRole", "principalId": "AROAYS4HWI4WFRQYPINK4:userID+development", "arn": "arn:aws:sts::accountID:assumed-role/developer-role/userID+development", "accountId": "accountID", "accessKeyId": "ASIAYS4HWI4WBCN3CDG6", "sessionContext": { "sessionIssuer": { "type": "Role", "principalId": "AROAYS4HWI4WFRQYPINK4", "arn": "arn:aws:iam:: accountID:role/aws/production/jobs/developer-role", "accountId": "accountID", "userName": "developer-role" }, "attributes": { "creationDate": "2024-07-07T17:56:59Z", "mfaAuthenticated": "false" }, "sourceIdentity": "userID" } }, "eventTime": "2024-07-07T18:06:10Z", "eventSource": "ecs.amazonaws.com", "eventName": "UpdateCluster", "awsRegion": "eu-west-1", "sourceIPAddress": "IP", "userAgent": "details", "errorCode": "InvalidParameterException", "errorMessage": "Insufficient key permissions provided to Fargate service principal for ephemeral storage encryption. Encountered KMS Error Code: AccessDeniedException, while running dry-run check for GenerateDataKeyWithoutPlaintext for key arn:aws:kms:eu-west-1: accountID:key/98a237ae-d89c-498b-b45e-c71587327101", "requestParameters": { "cluster": "my-cluster", "configuration": { "managedStorageConfiguration": { "fargateEphemeralStorageKmsKeyId": "arn:aws:kms:eu-west-1: accountID:key/98a237ae-d89c-498b-b45e-c71587327101" } } }, "responseElements": null, "requestID": "requestID", "eventID": "eventID", "readOnly": false, "eventType": "AwsApiCall", "managementEvent": true, "recipientAccountId": "accountID", "eventCategory": "Management", "tlsDetails": { "tlsVersion": "TLSv1.3", "cipherSuite": "TLS_AES_128_GCM_SHA256", "clientProvidedHostHeader": "ecs.eu-west-1.amazonaws.com" }, "sessionCredentialFromConsole": "true" }

On a general note, a quick way to troubleshoot KMS issues is to open the CloudTrail event history in the same region, filter by "event source" of "kms.amazonaws.com", and clicking the gear symbol in the upper right-hand corner to reveal the "error code" column.

Just find the error you just received (although lagging behind by a few minutes), and you'll see exact details of which principal tried to do what, along with plenty of other details, like the encryption context for the operation.

In this case, I expect you have one or more IAM roles that need the KMS permissions added either in their identity-based policies or in the KMS key policy with the IAM role explicitly identified in the Principal element, and you'll see which principal it is in the CloudTrail event.

The error you are encountering suggests that the Fargate service principal does not have the necessary permissions to use the KMS key for encrypting ephemeral storage. Let's go through your KMS policy and task execution role policy to ensure everything is set up correctly.

KMS Key Policy Here’s an overview of your KMS key policy. There are a few adjustments that we need to make:

https://docs.aws.amazon.com/kms/latest/developerguide/key-policies.html

Ensure the kms:GenerateDataKeyWithoutPlaintext permission is correctly granted. Correct the ARN formatting and ensure there are no extra spaces. Corrected KMS Key Policy

{
    "Version": "2012-10-17",
    "Id": "key-consolepolicy-3",
    "Statement": [
        {
            "Sid": "Enable IAM User Permissions",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::accountID:root"
            },
            "Action": "kms:*",
            "Resource": "*"
        },
        {
            "Sid": "Allow access for Key Administrators",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::accountID:role/aws-service-role/ecs.amazonaws.com/AWSServiceRoleForECS"
            },
            "Action": [
                "kms:Create*",
                "kms:Describe*",
                "kms:Enable*",
                "kms:List*",
                "kms:Put*",
                "kms:Update*",
                "kms:Revoke*",
                "kms:Disable*",
                "kms:Get*",
                "kms:Delete*",
                "kms:TagResource",
                "kms:UntagResource",
                "kms:ScheduleKeyDeletion",
                "kms:CancelKeyDeletion",
                "kms:RotateKeyOnDemand"
            ],
            "Resource": "*"
        },
        {
            "Sid": "Allow use of the key",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::accountID:role/aws-service-role/ecs.amazonaws.com/AWSServiceRoleForECS"
            },
            "Action": [
                "kms:Encrypt",
                "kms:Decrypt",
                "kms:ReEncrypt*",
                "kms:GenerateDataKey*",
                "kms:DescribeKey"
            ],
            "Resource": "*"
        },
        {
            "Sid": "Allow Fargate to use the key",
            "Effect": "Allow",
            "Principal": {
                "Service": "ecs-tasks.amazonaws.com"
            },
            "Action": [
                "kms:GenerateDataKey",
                "kms:GenerateDataKeyWithoutPlaintext",
                "kms:Decrypt"
            ],
            "Resource": "arn:aws:kms:eu-west-1:accountID:key/98a237ae-d89c-498b-b45e-c71587327101"
        },
        {
            "Sid": "Allow attachment of persistent resources",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::accountID:role/aws-service-role/ecs.amazonaws.com/AWSServiceRoleForECS"
            },
            "Action": [
                "kms:CreateGrant",
                "kms:ListGrants",
                "kms:RevokeGrant"
            ],
            "Resource": "*",
            "Condition": {
                "Bool": {
                    "kms:GrantIsForAWSResource": "true"
                }
            }
        }
    ]
}

Task Execution Role Policy Ensure that each Fargate service task execution role has the correct permissions to use the KMS key:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "kms:Decrypt",
                "kms:GenerateDataKeyWithoutPlaintext"
            ],
            "Resource": "arn:aws:kms:eu-west-1:accountID:key/98a237ae-d89c-498b-b45e-c71587327101"
        }
    ]
}

Additional Points to Check Account ID and Key ARN: Ensure that the account ID and key ARN are correctly specified without any extra spaces. Region Consistency: Make sure that the region specified in the ARN matches the region where the key and Fargate tasks are located. Policy Propagation: Sometimes, it takes a few minutes for policy changes to propagate. Wait a few minutes and try again if the changes were just made. Applying the Policies Update the KMS Key Policy:

Navigate to the AWS KMS console. Select the key. Edit the key policy to match the corrected policy above. Update the Task Execution Role:

Navigate to the IAM console. Find the task execution role associated with your Fargate service. Attach the corrected inline policy. Retry Cluster Update After ensuring the policies are correctly set up, try updating your cluster again to enable encryption with the customer-managed key.

If you continue to encounter issues, check the CloudTrail logs for any detailed errors related to KMS permissions and adjust the policies accordingly. If necessary, AWS Support can provide further assistance and insight into permissions issues.