Optimizing AWS Glue Costs: How to Reduce Expenses for Interactive Sessions?

1. Reduce Idle Timeout

The %idle_timeout setting in your script is currently set to 100 minutes. Reducing this value can help minimize the time your Glue resources are running without performing any tasks. For example, setting it to 10 or 20 minutes might be more cost-effective.

%idle_timeout 20

2. Optimize Worker Type and Number of Workers

The choice of worker type and the number of workers significantly impacts your costs. In your script, you are using G.1X worker type with 2 workers. Consider the following adjustments:

• Worker Type: If your job does not require the memory and CPU of G.1X, you can switch to a smaller worker type like Standard. However, ensure that this does not adversely affect your job's performance.
• Number of Workers: Start with 1 worker and scale up only if needed. This reduces the initial cost and helps you better understand the actual resource requirements of your job.

%worker_type Standard
%number_of_workers 1

3. Use AWS Glue Job Bookmarks

AWS Glue job bookmarks help keep track of the data that has already been processed. This can prevent reprocessing of the same data and save costs, especially useful if you are testing and re-running your script multiple times.

4. Efficient Data Handling

• Read Data in Batches: If you have a large dataset, read the data in batches instead of loading it all at once. This can help manage memory usage and reduce the overall processing time.
• Filter Data Early: Apply any filters or transformations early in the process to minimize the amount of data being processed.

5. Optimize Data Transformations

Ensure your data transformations are efficient. For example, avoid unnecessary operations and use optimized functions for data manipulation.

6. Optimize Output Format and Compression

• Output Format: Parquet is already a good choice for optimized storage and faster read/write operations.
• Compression: Use an efficient compression algorithm like SNAPPY to reduce the storage cost and improve performance.

Here is an adjusted version of your script with some of the recommendations applied:

%idle_timeout 20
%glue_version 4.0
%worker_type Standard
%number_of_workers 1

import sys
from awsglue.transforms import *
from awsglue.utils import getResolvedOptions
from pyspark.context import SparkContext
from awsglue.context import GlueContext
from awsglue.job import Job
  
sc = SparkContext.getOrCreate()
glueContext = GlueContext(sc)
spark = glueContext.spark_session
job = Job(glueContext)

# Create dynamicframe from JSON data from AWS S3, No glue catalog/crawler required for this method
connection_options = {
    "paths": ["s3://<s3-bucket-source>/"],
    "recurse": True
}

dynamic_frame = glueContext.create_dynamic_frame_from_options(
    connection_type="s3",
    connection_options=connection_options,
    format="json"
)

# Resolve choice to cast data types
json_dynamic_frame = dynamic_frame.resolveChoice(specs = [
    ('data.sensor.cpuCurrentLoad', 'cast:double'), 
    ('data.sensor.cpuLoadAverage_15', 'cast:double'), 
    ('data.sensor.cpuCurrentLoadSystem', 'cast:double'), 
    ('data.sensor.cpuCurrentLoadUser', 'cast:double'), 
    ('data.sensor.cpuLoadAverage_5', 'cast:double'),
    ('data.sensor.cpuCurrentLoadIdle', 'cast:double'),
    ('data.sensor.cpuLoadAverage_1', 'cast:double'),
    ('data.calculation.memoryFree_MB', 'cast:double'),
])

glueContext.write_dynamic_frame.from_options(
       frame = json_dynamic_frame,
       connection_type = "s3",
       connection_options = {"path": "s3://<S3-bucket-destination>/", "compression": "SNAPPY"},
       format = "parquet")

By following these recommendations, you should be able to significantly reduce the costs associated with your AWS Glue interactive sessions while maintaining the efficiency of your data processing tasks.