How do I identify and resolve client connection issues when I use mTLS with the Application Load Balancer?

Short description

To investigate client connection issues, you must activate the connection logs for the Application Load Balancer. Use the connection logs to analyze request patterns and troubleshoot issues.

For more information on what information the connection logs provide, see Monitor your Application Load Balancers.

Resolution

The Application Load Balancer's connection logs entries identify the error you got when the client tried to connect to the Application Load Balancer.
Note: For information on the ways to process the errors, see Processing connection log files.

Filter connection logs to view specific errors

You can filter connection logs by the client's IP address to look for specific errors. For example, run the following command to look for failed requests for a specific client IP address (for example, a.a.a.a):

zcat *.log.gz | awk '($2 == "a.a.a.a" && $11 ~ "Failed")' | awk '{print $1, $5, $6, $8, $9, $10, $11, $12}' 

2024-04-05T18:31:29.268987Z TLSv1.3 ECDHE-RSA-AES256-SHA "CN=client-expired.alb-mtls-example.com,OU=IT,O=Company LLC,L=City,ST=State,C=US" NotBefore=2024-01-02T19:17:20Z;NotAfter=2024-04-02T19:17:20Z 02 "Failed:ClientCertExpired"
2024-04-05T19:29:35.128387Z TLSv1.3 ECDHE-RSA-AES256-SHA "CN=client-expired.alb-mtls-example.com,OU=IT,O=Company LLC,L=City,ST=State,C=US" NotBefore=2024-01-02T19:17:20Z;NotAfter=2024-04-02T19:17:20Z 02 "Failed:ClientCertExpired"
2024-04-05T20:16:44.387952Z TLSv1.3 ECDHE-RSA-AES256-SHA "CN=client-expired.alb-mtls-example.com,OU=IT,O=Company LLC,L=City,ST=State,C=US" NotBefore=2024-01-02T19:17:20Z;NotAfter=2024-04-02T19:17:20Z 02 "Failed:ClientCertExpired"

Note: Replace <a.a.a.a> with the client IP address that you are investigating for errors.

Identify the value present in the tls_verify_status field. Refer to error reason codes to identify the error. Then, use the following sections to address the specific error.

Error: "ClientCertMaxChainDepthExceeded"

This error occurs because the number of client certificates exceeded the maximum chain depth.

To resolve this error, follow these steps:

1. Confirm the number of certificates in the client certificate that's presented to the Application Load Balancer.

2. To examine the certificate chain depth, use any one of the following options:

   Windows: Choose the certificate, and select Certificate path. Then, count the certificates.
   -or-
   Run the following command in Microsoft PowerShell and count the certificates:

   certutil -dump <file.crt

   Linux or Unix: To count the certificates, run the following command:

   $ openssl crl2pkcs7 -nocrl -certfile <file.crt> | openssl pkcs7 -print_certs -noout | grep "subject" | wc -l

3. Reduce the number of certificates present within the client certificate's chain. For more information on the maximum supported certificate chain depth, see Quotas for your Application Load Balancer.

Error: "ClientCertMaxSizeExceeded"

This error occurs because the maximum client certificate size is exceeded.

To resolve this error, follow these steps:

1. Verify the size of the client certificate:
   Windows Choose the file. Then, select properties and view the file size.
   Linux Run du -h <file>.
   Macintosh Select the item in the Finder, and then press Command-I.
2. Reduce the size of the client certificate file presented to the Application Load Balancer. For more information on the maximum supported certificate size, see Quotas for your Application Load Balancer.

Error: "ClientCertCrlHit"

This error occurs when the client certificate exists within the certificate revocation list (CRL) that's used by the mTLS listener.

1. Review the connection log entry that shows the leaf_client_cert_subject field.
2. Download the CRL file from the Amazon Simple Storage Service (Amazon S3) bucket. The CRL file is referenced by the Amazon Elastic Compute Cloud (Amazon EC2) trust store.
3. Review the CRL for the client certificate that was identified in the log entry.
4. Renew the client certificate against the CA bundle used by the EC2 trust store. Retry the connection using the new non-revoked certificate.

Error: "ClientCertUntrusted"

This error occurs because the client certificate is untrusted.

To resolve this error, follow these steps:

1. Check that the certificate name in the leaf_client_cert_subject field was issued by the Certificate Authority (CA) present in the configured EC2 trust store.
   Note: If the client certificate's issuer isn't present within the CA certificate bundle used by the EC2 trust store, then mTLS won't work.

2. Run the following command against the client certificate:

   $ openssl crl2pkcs7 -nocrl -certfile <file.cert> | openssl pkcs7 -print_certs -noout | grep -e 'issuer' -e 'subject' | sed -e 's/subject._CN = /Subject: /; s/issuer._CN = /Issuer: /'"

3. To retrieve the subjects present within the bundle, run the following command against the CA certificate bundle used by the EC2 trust store:

   $ openssl crl2pkcs7 -nocrl -certfile <file.pem> | openssl pkcs7 -print_certs -noout | grep -e 'subject' | sed -e 's/subject.*CN = /Subject: /'

Error: "ClientCertNotYetValid"

This error occurs because the client certificate is not yet valid.

To resolve this error, follow these steps:

1. Check the certificate validity period. You can find this value in the leaf_client_cert_validity field of the connection log.

2. Run the following command against the certificate file:

   $ openssl x509 -in $i --text | grep -A 2 "Validity"

3. Check the Not Before date provided in step 2 against the timestamp field of the connection log.
   Note: Make sure that the client certificate is used only after the Not Before date.

4. To resolve this error, use the CA certificate bundle referenced by the EC2 trust store to re-issue the client certificate.

Error: "ClientCertExpired"

This error occurs when the client certificate is expired.

To resolve this error, follow these steps:

1. Check the certificate validity period. You can find this value in the leaf_client_cert_validity field of the connection log.

2. Run the following command against the certificate file:

   $ openssl x509 -in $i --text | grep -A 2 "Validity"

3. Check the Not After date provided in step 2 against the timestamp field of the connection log.
   Note: Make sure that the client certificate is used only before the Not After date.

4. To resolve this error, use the CA certificate bundle referenced by the EC2 trust store to re-issue the client certificate.

Error: "ClientCertTypeUnsupported"

This error occurs because the client certificate type is unsupported.

To resolve this error, follow these steps:

1. Check the certificate's version. You can find this value in the leaf_client_cert_subject field of the connection log. To find the version, run the following command against the client certificate:

   $ openssl x509 -in $i --text | grep -A 3 "Version:

2. Make sure that the certificate's version is X.509v3. This is the supported version.

Error: "ClientCertInvalid"

This error occurs because the client certificate is not valid.

To resolve this error, follow these steps:

1. Validate that the client certificate meets the requirements so that it can be used with mTLS and the Application Load Balancer.
2. Re-issue the certificate.

Error: "ClientCertRejected"

This error occurs because the client certificate is rejected by custom server validation.

To resolve this error, follow these steps:

1. Validate if the Application Load Balancer's listener uses mTLS in the Passthrough or Verify mode.
   In Passthrough mode, the load balancer uses HTTP headers to send the client certificate and chain to the target for verification. You get this error when the target rejects the client certificate.
2. Review the application logs on the target to determine the cause of the error.
3. As a best practice, use the Verify mode instead of the Passthrough mode on the mTLS listener.

Error: "ClientCertCrlProcessingError"

This error occurs when the certificate revocation list (CRL) experiences a processing error.

To resolve this error, follow these steps:

1. Document the timestamp associated with the error in the connection logs.
2. Confirm that the client certificate meets the defined requirements.
3. Make sure that the CRL configured isn't exceeding the defined quotas.

If you continue to get the error, contact AWS Support for further review. Provide the following information to the AWS Support engineer:

• Output of the following curl command:

   $ curl -vvv -X <HTTP action> --cert <client_cert.crt> --key <client_key.pem> https://alb.fqdn.com -A credstest

• Client certificate (don't provide the key file)

• Connection logs for the time period when the issue was experienced

Error: "UnmappedConnectionError"

This error occurs when the runtime connections fails to be mapped.

Contact AWS Support for further review. Provide the following information to the AWS Support engineer:

• Output of the following curl command:

   $ curl -vvv -X <HTTP action> --cert <client_cert.crt> --key <client_key.pem> https://alb.fqdn.com -A credstest

• Client certificate (don't provide the key file)

• Connection logs for the time period when the issue was experienced

Related information

How do I use Amazon Athena to query my Application Load Balancer connection logs?

Introducing mTLS for Application Load Balancer

How do I investigate issues related to the CA bundle for mTLS listeners on the Application Load Balancer?