Redshift Cluster node's disk usage is 100% most of the time
Hi Team,
We are using Redshift Cluster with the following configurations
- Node type: dc2.large
- Number of nodes: 4
- Storage Usage Details : 65.13% (416.84 of 640 GB used)
While performing some operations in the DB I'm getting the following error: There is a problem with the Redshift query. Please correct the query and try again. Original error message: ERROR: Insufficient system resources to support data size: consider increasing compute size. (Disk Full)
While analyzing the disk usage, i found that one of the compute nodes has 100% disk usage, while the others have usage of less than 60-70% also as mentioned above the total storage used is only 65%
So kindly assist me to rectify what is the root cause of this and how to resolve this.
Thanks. Pascal Rebello
- Newest
- Most votes
- Most comments
This could be related to Data skew. Data skew is caused when data in the table is unevenly distributed across the data slices on nodes. This uneven distribution can lead to performance degradation or disk space issues.
Please refer following link for how to identify tables which has high data skew and you can fix it with right distribution key.
Maintaining roughly 20% of your storage capacity as free space is essential for efficient system processing. If your current storage utilization is at 65%, you may soon exhaust your available storage. To accommodate your workload requirements and leverage the latest features, you might want to consider upgrading to either Ra3 nodes or Redshift Serverless, based on your usage. These options offer independent scaling for both compute and storage, providing you with greater flexibility and opportunities for performance optimization.
https://aws.amazon.com/redshift/features/ra3/
https://docs.aws.amazon.com/redshift/latest/mgmt/working-with-serverless.html
Relevant content
AWS OFFICIALUpdated 2 years ago- AWS OFFICIALUpdated a year ago
- AWS OFFICIALUpdated 2 years ago
- AWS OFFICIALUpdated 2 years ago
