Pages

Topics

Tools

↑↓navigate↵openescclose
All guides

Python Boto3 CloudWatch: Sending Custom Metrics

Learn to publish custom CloudWatch metrics with Python Boto3: send metric data, create alarms, retrieve statistics, and avoid common production pitfalls.

~5 min readPython
#boto3#cloudwatch#aws#metrics
Python Boto3 sending a custom metric to CloudWatch dashboard with alarm threshold.

With CloudWatch custom metrics, you can monitor application-specific numbers that AWS does not track by default. The Python Boto3 library provides a straightforward interface to CloudWatch for this purpose. This article covers the core operations: sending metric data, creating alarms, and querying statistics, with code examples and operational details that matter in production.

Setting Up the CloudWatch Client

The first step is to create a Boto3 client for CloudWatch. The client requires valid AWS credentials, which can come from environment variables, IAM roles, or the shared credentials file. The region must match where your metrics and alarms should reside.

python
import boto3 cloudwatch = boto3.client('cloudwatch', region_name='us-east-1')

If you're running on EC2 or Lambda, the default credential chain picks up the instance role or execution role automatically. For local development, you might need to set AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY explicitly. The client is thread-safe, so you can reuse it across concurrent calls.

Publishing Custom Metrics with put_metric_data

The primary method for sending metrics is put_metric_data. It accepts a namespace at the top level and one or more metric data entries, each containing a metric name, value, timestamp, and optional dimensions and unit. Namespaces isolate metrics from different applications; choose a meaningful one like MyApp/Transactions.

python
from datetime import datetime, timezone response = cloudwatch.put_metric_data( Namespace='MyApp/Transactions', MetricData=[ { 'MetricName': 'ProcessingTime', 'Value': 125.4, 'Unit': 'Milliseconds', 'Timestamp': datetime.now(timezone.utc), 'Dimensions': [ {'Name': 'Environment', 'Value': 'production'}, {'Name': 'Service', 'Value': 'checkout'} ] } ] )

Dimensions are key-value pairs that let you filter and aggregate metrics later. For example, you can query the same metric across different services. The Timestamp is optional; if omitted, CloudWatch uses the time the request was received. For high-frequency metrics, you can send up to 20 values in a single request by adding more entries to MetricData.

Handling Metric Data Limits and Retries

Each put_metric_data call can contain up to 20 metric data entries. If you need to send more, split the data across multiple calls. CloudWatch also accepts a StorageResolution field on each metric data entry to specify whether the metric is standard (60-second) or high-resolution (1-second). High-resolution metrics cost more, so choose carefully.

Boto3 automatically retries on transient failures like throttling or network errors. The default retry mode is legacy, which retries up to 5 times with exponential backoff. For production workloads, consider configuring a more aggressive retry strategy via the Config object:

python
from botocore.config import Config config = Config( retries={'max_attempts': 10, 'mode': 'standard'} ) cloudwatch = boto3.client('cloudwatch', config=config)

The standard retry mode also handles throttling by inspecting the ThrottlingException response and waiting accordingly. Retries can cause duplicate metric data if the first request succeeded but the response was lost. Design your metric ingestion to tolerate duplicates where duplicates are not meaningful.

Creating CloudWatch Alarms with put_metric_alarm

Once you're publishing metrics, you often want to trigger actions when a threshold is breached. put_metric_alarm creates or updates an alarm based on a metric. You specify the metric name, namespace, dimensions, statistic, period, threshold, and comparison operator.

python
cloudwatch.put_metric_alarm( AlarmName='CheckoutProcessingTimeHigh', AlarmDescription='Alert when checkout processing time exceeds 500ms', MetricName='ProcessingTime', Namespace='MyApp/Transactions', Statistic='Average', Dimensions=[ {'Name': 'Environment', 'Value': 'production'}, {'Name': 'Service', 'Value': 'checkout'} ], Period=60, EvaluationPeriods=2, Threshold=500.0, ComparisonOperator='GreaterThanThreshold', AlarmActions=['arn:aws:sns:us-east-1:123456789012:my-topic'] )

Period is the length of time in seconds for each data point. EvaluationPeriods defines how many consecutive periods must breach the threshold before the alarm fires. The AlarmActions list can contain SNS topic ARNs, Auto Scaling policies, or EC2 actions. You can also set OKActions and InsufficientDataActions to notify on recovery or missing data.

When you update an existing alarm, the request is treated as the full desired state. If you need to change only one parameter, resend the other required fields as well; omitted parameters are reset to defaults. Alarm names must be unique within an AWS account and region.

Retrieving Metrics with get_metric_statistics

To inspect historical data, use get_metric_statistics. This method returns a list of data points for a given metric over a specified time range. You must provide the namespace, metric name, dimensions, start time, end time, period, and statistics.

python
from datetime import datetime, timedelta, timezone response = cloudwatch.get_metric_statistics( Namespace='MyApp/Transactions', MetricName='ProcessingTime', Dimensions=[ {'Name': 'Environment', 'Value': 'production'}, {'Name': 'Service', 'Value': 'checkout'} ], StartTime=datetime.now(timezone.utc) - timedelta(hours=1), EndTime=datetime.now(timezone.utc), Period=300, Statistics=['Average', 'Maximum', 'SampleCount'] ) datapoints = response['Datapoints']

The response contains a list of data points, each with a timestamp and the requested statistics. Note that CloudWatch aggregates raw data into the specified period. The period must be at least the metric's storage resolution: 60 seconds for standard metrics, or 1 second for high-resolution metrics. Requesting a shorter period can return empty or incomplete results.

Listing Metrics and Handling Pagination

The list_metrics operation returns all metrics that match a given namespace, metric name, or dimensions. This is useful for discovering what metrics exist or for building dashboards dynamically.

python
paginator = cloudwatch.get_paginator('list_metrics') for page in paginator.paginate(Namespace='MyApp/Transactions'): for metric in page['Metrics']: print(metric['MetricName'], metric.get('Dimensions'))

The response is paginated; using the paginator handles multiple pages automatically. Each metric object includes the namespace, metric name, and dimensions.

Error Handling and Boto3 Retry Configuration

Boto3 raises exceptions from the botocore.exceptions module. Common ones for CloudWatch include InvalidParameterValue, MissingParameter, LimitExceeded, and Throttling. Always catch these and handle them appropriately, especially in production scripts.

python
from botocore.exceptions import ClientError try: cloudwatch.put_metric_data(...) except ClientError as e: if e.response['Error']['Code'] == 'Throttling': # Apply backoff or queue for later pass else: raise

Throttling occurs when you exceed the API request rate. The account-level default for put_metric_data is 150 requests per second, but this quota can be increased. If you're sending many small batches, consider aggregating values into fewer requests to stay within the limit. Also, ensure your IAM role has the required permissions: cloudwatch:PutMetricData, cloudwatch:PutMetricAlarm, and cloudwatch:GetMetricStatistics. Using overly broad permissions like cloudwatch:* is convenient but not recommended for production.

Operational Considerations for Production

Custom metrics can incur charges, and high-resolution metrics cost more. Before publishing a high-cardinality metric, think about whether you really need per-instance or per-user dimensions. A metric with many unique dimension combinations can become expensive. Instead, aggregate data client-side or use a single metric with a Service dimension for a few known values.

Also consider retention. CloudWatch stores all metrics for 15 months, so you don't need to archive them yourself. However, if you need to query metrics frequently, using get_metric_statistics with a period of 60 seconds over a long range can be slow. For dashboards, use CloudWatch's built-in metric math instead of pulling raw data into your application.

When publishing metrics from a high-throughput service, batch values into a single put_metric_data call every few seconds rather than sending one request per event. This reduces API calls and throttling risk. Use the Timestamp field to preserve the actual event time; otherwise all points will appear at the request time and the metrics will look flat.

Finally, test your alarm thresholds with real data before deploying. A common mistake is setting EvaluationPeriods too high, which delays alerts, or too low, which causes flapping. Use CloudWatch's alarm history to verify that the alarm transitions correctly from INSUFFICIENT_DATA to OK to ALARM as you simulate load.

Try matching browser tools

All tools

Small utilities that fit Python work — they run entirely in this browser tab, so nothing you paste leaves your device.

All Python guides

More in Python

All Python guides

The guides filed next to this one in Python, in library order.