Module 8: Monitor S3 Replication and Requests
You will learn how to generate traffic to graph, and monitor S3 Replication metrics and requests.
Overview
In this module, you will learn how to monitor Amazon S3 Replication in the Multi-Region Access Point console, and S3 request metrics in Amazon CloudWatch.
Time to complete
10 minutes
Implementation
8.1 - Generating traffic to graph
- If you have only uploaded one or two objects to S3, there will not be very much to show on the graphs. If you wish to generate some traffic to graph, the following load generator bash commands in this step may be useful. You will need to set the environment variable $mrap_arn to the ARN of your Multi-Region Access Point before running it on the computer used in module 6.
- The commands create a 1 MiB file, and then write and read a new object using your access point every 20 seconds indefinitely.
Note: Remember to stop the script when you are done testing, in order to minimize your costs.
- Run the following commands:
#Create a 1 MiB file containing random data
dd if=/dev/urandom of=~/1MiB.file bs=1024 count=1024
#Write and read a new S3 object every 20 seconds
while true; do datetime=$(date "+%Y.%m.%d-%H.%M.%S") \
; aws s3 cp ~/1MiB.file s3://$mrap_arn/load_generator/1MiB_from_load_generator.$datetime \
; aws s3 cp s3://$mrap_arn/load_generator/1MiB_from_load_generator.$datetime ~/1MiB_from_S3 \
; sleep 20 ; done
8.2 - Replication metrics
- Navigate back to the browser tab with your Multi-Region Access Point open. If you need to re-open this, in the Amazon S3 console, choose Multi-Region Access Points. Then, select the name of your Multi-Region Access Point to configure additional settings.
- Choose the Replication and failover tab.
- Find the Replication Metrics section and choose 1h.
- The graph might display No data, until you select 1hr, because of the short period replication has been active.
- The graph might display No data, until you select 1hr, because of the short period replication has been active.
Notes:
- You can select the legend entries to hide or highlight individual lines, of which there is one per replication rule. Each one shows the Source bucket and the name of the rule.
- You can tell which bucket is the one the Multi-Region Access Point routed your writes to, as replication latency will only be above zero for that Source bucket.
- If you want to see the effect of failover, leave the load generator script running and change the failover configuration of your Multi-Region Access Point so that the bucket being written to is passive (see step 4). The change will take effect in about 2 minutes, and a few minutes later the graph will start to show some replication latency for the original bucket (as objects are now being replicated to it) and no latency for the bucket that is now being written to.
- On this page you can also review graphs for Bytes pending replication and Operations pending replication.
8.3 - Request metrics
- Once metrics have been logged (consider manually reading from and writing to each bucket directly to create a log for each request type and bucket), you can add them in CloudWatch under Metrics → All metrics → S3 → Request Metrics Per Filter. Graphing the sum of GetRequests and PutRequests over each 1-minute period provides a view of all activity (including replication activity) like this:
- The example image on the right shows the effect of setting the us-east-1 bucket to passive when all clients were closest to the us-east-1 region. Note: PutRequests to the us-east-1 bucket continued, as new objects were replicated into us-east-1. S3 Replication is independent of Multi-Region Access Point failover controls.
- The example image on the right shows the effect of setting the us-east-1 bucket to passive when all clients were closest to the us-east-1 region. Note: PutRequests to the us-east-1 bucket continued, as new objects were replicated into us-east-1. S3 Replication is independent of Multi-Region Access Point failover controls.
Conclusion
Congratulations! You have learned how to set up and use an Amazon S3 Multi-Region Access Point to access data in multiple regional S3 buckets via a single global endpoint, and tested the failover controls capability.
We recommend that you clean up your resources to stop the accrual of associated costs.