Rotating EBS Snapshots

September 14th, 2010 — 6:50pm

If you use Elastic Block Storage (EBS) for storing your files on your ec2 instances you more than likely backup those files using the ec2 snapshots. If you don’t already do this you should probably start, as EBS volumes are not 100% fault tolerant, and can (and do) degrade just like normal drives. A good script for taking snapshots of data can be found on the alestic.com website, called ec2-consistent-snapshot. You can find all the information for this script here:

http://alestic.com/2009/09/ec2-consistent-snapshot

How to Rotate EBS Snapshots

After using the ec2-consistent-snapshot script for a while I realized I would eventually need to find something to rotate these backups as they were growing out of control. Some of our volumes were having snapshots done every hour, and that was adding up quickly. Google provided me with no easy solution for rotating the snapshots, so I decided to write my own script.

Essentially what I wanted was to have a script that would rotate the snapshots in Grandfather-Father-Son type setup. I wanted to have hourly backups kept for 24 hours, daily kept for a week, weekly kept for a month, and monthly kept for a year. Anything older than that I don’t want, however the script can be tweaked to allow for older backups.

Basically what the script does is the following:

  • Gets a list of all snapshots and puts them into an array indexed by the volume and the date the snapshot was taken
  • For a given volume organize the snapshots so that there are only hourly snapshots for 1 day, daily snapshots for 1 week, weekly snapshots for 1 month, and monthly snapshots for 1 year and collect which snapshots require deleting.
  • Delete the snapshots that are set for delete.

I wrote the script in PHP, mainly because it is what I feel most comfortable using. I am also once again using the Amazon PHP library. Here is the script in it’s entirety.


<?php
/*
 * rotate-ebs-snapshots.php
 *
 * Author: Stefan Klopp
 * Website: http://www.kloppmagic.ca
 * Requires: Amazon ec2 PHP Library
 *
 */

ini_set("include_path", ".:../:./include:../include:/PATH/TO/THIS/SCRIPT");

// include the amazon php library
require_once("Amazon/EC2/Client.php");
require_once("Amazon/EC2/Model/DeleteSnapshotRequest.php");

// include our configuration file with out ACCESS KEY and SECRET
include_once ('.config.inc.php');

$service = new Amazon_EC2_Client(AWS_ACCESS_KEY_ID,
                                       AWS_SECRET_ACCESS_KEY);


// setup our array of snapshots
$snap_array = setup_snapshot_array();

// call to rotate (you can call this for every volume you want to rotate)
rotate_standard_volume('VOLUME_ID_YOU_WISH_TO_ROTATE');


/* 
 * Used to setup an array of all snapshots for a given aws account
 */
function setup_snapshot_array() {
    global $service;
    // Get a list of all EBS snapshots
    $response = $service->describeSnapshots($request);

    $snap_array = array();

    if ($response->isSetDescribeSnapshotsResult()) {
        $describeSnapshotsResult = $response->getDescribeSnapshotsResult();
        $snapshotList = $describeSnapshotsResult->getSnapshot();
        foreach ($snapshotList as $snapshot) {
            if ($snapshot->getStatus() == 'completed') {

                    // date is in the format of 2009-04-30T15:32:00.000Z
                    list($date, $time) = split("T", $snapshot->getStartTime());

                    list($year, $month, $day) = split("-", $date);
                    list($hour, $min, $second) = split(":", $time);

                    // convert the date to unix time
                    $time = mktime($hour, $min, 0, $month, $day, $year);

                    $new_row = array(
                            'snapshot_id'=>$snapshot->getSnapShotId(),
                            'volume_id'=>$snapshot->getVolumeId(),
                            'start_time'=>$time
                    );
                    // add to our array of snapshots indexed by the volume_id
                    $snap_array[$new_row['volume_id']][$new_row['start_time']] = $new_row;
            }
        }
    }

    // sort each volumes snapshots by the date it was created
    foreach ($snap_array as $vol=>$vol_snap_array) {
            krsort($vol_snap_array);
            $snap_array[$vol] = $vol_snap_array;
    }

    return($snap_array);
}

/*
 * Used to rotate the snapshots
 */
function rotate_standard_volume($vol_id) {
        global $snap_array, $service;

        // calculate the date ranges for snapshots
        $one_day = time() - 86400;
        $one_week = time() - 604800;
        $one_month = time() - 2629743;
        $one_year = time() - 31556926;

        $hourly_snaps = array();
        $daily_snaps = array();
        $weekly_snaps = array();
        $monthly_snaps = array();
        $delete_snaps = array();

        echo "Beginning rotation of volume: {$vol_id}\n";

        foreach($snap_array[$vol_id] as $time=>$snapshot) {

                echo "Testing snapshot {$snapshot['snapshot_id']} with a date of ".date("F d, Y @ G:i:s", $time)."... ";

                if ($time >=  $one_day) {
                        echo "Snapshot is within a day lets keep it.\n";
                        $hourly_snaps[$time] = $snapshot;
                }
                elseif ($time < $one_day &#038;&#038; $time >= $one_week) {
                        $ymd = date("Ymd", $time);
                        echo "Snapshot is daily {$ymd}.\n";

                        if (is_array($daily_snaps[$ymd])) {
                                echo "Already have a snapshot for {$ymd}, lets delete this snap.\n";
                                $delete_snaps[] = $snapshot;
                        }
                        else {
                                $daily_snaps[$ymd] = $snapshot;
                        }
                }
                elseif ($time < $one_week &#038;&#038; $time >= $one_month) {
                        $week = date("W", $time);
                        echo "Snapshot is weekly {$week}.\n";

                        if (is_array($weekly_snaps[$week])) {
                                echo "Already have a snapshot for week {$week}, lets delete this snap.\n";
                                $delete_snaps[] = $snapshot;
                        }
                        else {
                                $weekly_snaps[$week] = $snapshot;
                        }
                }
                elseif ($time < $one_month &#038;&#038; $time >= $one_year) {
                        $month = date("m", $time);
                        echo "Snapshot is monthly {$month}.\n";

                        if (is_array($monthly_snaps[$month])) {
                                echo "Already have a snapshot for month {$month}, lets delete this snap.\n";
                                $delete_snaps[] = $snapshot;
                        }
                        else {
                                $monthly_snaps[$month] = $snapshot;
                        }
                }
                else{
                        echo "Snapshot older than year old, lets delete it.\n";
                        $delete_snaps[] = $snapshot;
                }
        }

        foreach ($delete_snaps as $snapshot) {
                echo "Delete snapshot {$snapshot['snapshot_id']} with date ".date("F d, Y @ H:i", $snapshot['start_time'])." forever.\n";
                $request = new Amazon_EC2_Model_DeleteSnapshotRequest();
                $request->setSnapshotId($snapshot['snapshot_id']);
                $response = $service->deleteSnapshot($request);
        }
        echo "\n";
}

You can either run the script by editing the call to rotate_standard_volume. You can call this method for each volume you wish to rotate snapshots for. Also feel free to change the values of the date ranges to keep snapshots for a given date range for longer or shorter periods.

Finally to make this script effective you should have it run at least once a day via cron.

Conclusion

If you are like me and utilize EBS snapshots for backups of your data you will likely need to rotate those snapshots at some point. With the script above you should be able to quickly and easily rotate your snapshots. With a few tweaks you should be able to easily customize the rotation schedule to suit your needs.

Category: Amazon Web Services, Elastic Block Storage, Scripts
Tags: , , , , , 7 comments »

7 Responses to “Rotating EBS Snapshots”

  1. Stefan Klopp

    One thing to note is that this script can be run from any instance (or computer for that matter). It doesn’t have to reside on the instance that has the volume attached to it.

  2. Pablo

    Hello:

    I have a question. I can see the scripts only lists snapshots on region “US-east” (actually, it’s the default region). IS there a way to pick a different region ?

  3. Pablo

    Hey , i just found the solution :

    http://aws.amazon.com/code/PHP/1669

    $ec2Config = array (‘ServiceURL’ => ‘https://eu-west-1.ec2.amazonaws.com’);
    $ec2 = new Amazon_EC2_Client($accessKeyId, $secretAccessKey, $ec2Config);

    Pablo.

  4. Stefan Klopp

    Good find Pablo! Thanks for the comment.

  5. Rick

    Hi Stefan,

    1- Thank you for providing this. It is a wonderful tool!

    2- I was getting the following errors:
    PHP Deprecated: Function split() is deprecated in /root/custodian/rotate-snaps.php on line 50
    PHP Deprecated: Function split() is deprecated in /root/custodian/rotate-snaps.php on line 52
    PHP Deprecated: Function split() is deprecated in /root/custodian/rotate-snaps.php on line 53

    which I fixed by replacing deprecated split() with explode() in your original script.

    3- I was also getting errors of the type:
    PHP Notice: Undefined index: 20130405 in /root/custodian/rotate-snaps.php on line 111
    PHP Notice: Undefined index: 2013-02 in /root/custodian/rotate-snaps.php on line 123

    and after looking at the program flow, I tried replaced line 111:
    if (is_array($daily_snaps[$ymd])) {
    with: if (array_key_exists($ymd,$daily_snaps)) {

    similarly, I replaced line 123:
    if (array_key_exists($week,$weekly_snaps)) {
    with if (is_array($weekly_snaps[$week])) {

    it seems to be working fine, but I would appreciate your comments on these changes.

  6. Stefan Klopp

    Rick,

    Those changes look fine. You could have also used preg_split instead of split but explode works just as well.

    The switch to array_key_exists works as well.

    I should really push all these scripts up to github that way people can branch them.

  7. Rick

    Great. Thanks for your input and glad I can be of help!


Leave a Reply



Back to top