How to Setup a Webcam in Linux

At the beginning of this year I announced the launch of a webcam I had setup at my parents house in the Kootenays. In the announcement I talked briefly how the whole application worked, however wanted to get into more technical details here.

Hardware

  • Mini Compaq computer with Ubuntu Server installed
  • Canon A310 digital camera
  • USB Cable
  • Dedicated Server Hosted Remotely

Overview

The mini Compaq computer is located at my parents place sitting in their sun room. It is connected to the Internet, as well via a USB cable it is connected to the Canon A310 camera, which is setup on a small tripod pointing out my parents window.

On the server a cron is setup to run a script every 5 minutes. The script will determine if there is daylight enough outside to take a photo, and there is it will execute a gphoto2 command that will trigger the camera to take a photo and download the image to the computer. Once the image is on the computer it is then FTP’ed to my dedicated server for processing.

A cron is setup on my dedicated server for a script that checks an incoming folder for newly uploaded images. When a new image is found, it will process the image, creating several different file sizes, upload each file to Amazon S3 for storage and distribution through Amazon Cloudfront, and store a record of the image in a MySQL database.

Finally I have a web front end that reads from the database to display the latest photo, along with an archive of all the older photos that had been taken. Lets de-construct each step in the process to document how it all works.

Taking the Photo

The wonderful open source library gphoto2 does most of the work here. Gphoto2 allows you to control a camera in Linux via the command line. There is a large list of cameras on the gphoto2 website that are supported. Since I didn’t want to spend a lot of money on this project I bought an old Canon camera that was noted to work well with gphoto2.

On the mini computer at my parents a cron executes a PHP script every 5 minutes. The PHP script first checks to see if there are any other instances of gphoto2 running, if there are it exits. Next it determines at what time sunset and sunrise is, and if the time of day is between those two times as we don’t want to take photos of the darkness of night. Next the script checks to see if it is in the first 15 of sunrise, or last 15 minutes of sunset, if so we want to use a higher ISO on the camera to take the photo. Finally the gphoto2 script is called to take the photo.

Here is our cron entry:

*/5 * * * * /usr/bin/php /PATH/TO/BIN/bin/call-shoot-camera.php

Here is what my script looks like:


<?php
/**
 * Script to determine if it is light outside and whether to take a photo
 **/

// directory configuration
$base_dir = "/PATH/TO/HOME/FOLDER";
$archive_dir = $base_dir . "archive/".date("Y") . "/" . date("m") . "/" . date("d") . "/";

// check to see if gphoto2 is already running
$ps = `ps ua`;
if (strstr($ps, 'gphoto2')) {
    return 0; 
}

// fifteen minutes in seconds
$fifteen_min = 900;

// setup directory for todays date
if (!is_file($archive_dir)) {
    `/bin/mkdir -p $archive_dir`;
}

// sunrise/sunset config
$zenith=96.8;
$long = -118.00737;
$lat = 49.8672;

$now = time();

$sunrise = date_sunrise($now, SUNFUNCS_RET_TIMESTAMP, $lat, $long, $zenith, -8);
$sunset = date_sunset($now, SUNFUNCS_RET_TIMESTAMP, $lat, $long, $zenith, -8);

$first_fifteen = $sunrise + $fifteen_min;
$second_fifteen = $sunrise + $fifteen_min + $fifteen_min;

$last_fifteen = $sunset - $fifteen_min;
$second_last_fifteen = $sunset - $fifteen_min - $fifteen_min;

// set the ISO higher if we are in the first 15 minutes of sunrise or last 15 minutes of sunset
if (($now >= $sunrise && $now <= $first_fifteen) || ($now >= $last_fifteen && $now <= $sunset)) {
    $iso = 2;
}
else {
    $iso = 1;
}


// take a photo if the time of day is between sunrise and sunset
if ($now >= $sunrise && $now <= $sunset) {
    `/usr/bin/gphoto2 --set-config capturetarget=0 --set-config capture=1 
--set-config imagesize="medium 1" --set-config aperture=4 --set-config iso=$iso 
--set-config photoeffect=1 --capture-image-and-download 
--filename "/PATH/TO/CAPTURE/capture/archive/%Y-%m-%d_%H:%M:%S.jpg" 
--hook-script=/PATH/TO/BIN/bin/upload-and-set-new-photo.php`;
}

return 0;
?>

I will leave you to discover what all the flags do when calling the gphoto2 command, however would like to point out the –hook-script flag. What this flag does is call a script after the newly taken image has been downloaded from the camera to the computer. Thus we are able to call a second script that will upload our new photo to our server. This script is extremely basic and uses the standard linux FTP command.


<?php


if (getenv('ACTION') == 'download') {
    $file = getenv('ARGUMENT');
    $array = preg_split("/\//", $file);

    $final_file = array_pop($array);

    `/usr/bin/ftp -n -i -p HOSTNAME_OF_SERVER <<EOF
user FTPUSERNAME FTPPASSWORD
binary
put $file $final_file`;
}

As you can see, a really basic script that simply checks to see if the file was downloaded and if so to upload via ftp to our dedicated server.

I wanted to point out one final flag of gphoto2 before moving on to the dedicated server processing. An extremely handy flag when using gphoto2 with a new camera is the –list-config flag. This will tell you all the different config items of the camera you can set when using gphoto2.

Processing the Photo

On the dedicated server images are being uploaded to an incoming folder to be processed by another script. The processing script will scan the incoming folder looking for new images. Before processing an image, the script will check to make sure the image has not been written to within 20 seconds of the current time to make sure that the photo currently not being uploaded.

Once it is determined that the photo is not being uploaded we can process the image. The processing is done using the GD PHP extension, then uploaded to Amazon S3 using undesigned’s fantastic S3 PHP library. Finally the image information is stored in a MySQL database table.

Here is how the script looks:


<?php

// include our config file (contains db passwords and aws keys)
include_once("/PATH/TO/CONFIG/conf/config.inc");

// check if the incoming folder is already being processed
if (file_exists($cam_config['tmp_dir'] . 'process-incoming.pid')) {
    exit;
}
else {
    file_put_contents($cam_config['tmp_dir'] . 'process-incoming.pid', getmypid());
}

// libraries needed for s3 uploads and MySQL access
include_once($cam_config['base_dir'] . 'classes/s3-php5-curl/S3.php');
include_once($cam_config['base_dir'] . 'classes/db.cls');

$db = new db($cam_config['db_server'], $cam_config['db_user'], $cam_config['db_password'], $cam_config['db']);
$s3 = new S3($cam_config['aws_access_key'], $cam_config['aws_secret_key']);

// read all new files
$file_array = array();
if ($handle = opendir($cam_config['incoming_dir'])) {
    while (false !== ($file = readdir($handle))) {
        if ($file != "." &#038;&#038; $file != "..") {
            $file_array[] = $file;
        }
    }
    closedir($handle);
}

// get some exif info
foreach ($file_array as $file) {
    $file_full_path = $cam_config['incoming_dir'].$file;
    $file_base = basename($file, '.jpg');

    $filemtime = filemtime($file_full_path);

    $diff = time() - $filemtime;

    // skip any file that is still being written to
    if ($diff <= 20) {
        continue;
    }

    $exif = exif_read_data($file_full_path, 'File');

    $timestamp = $exif['FileDateTime'];

    $archive_dir = $cam_config['archive_dir'] . date("Y/m/d", $timestamp);

    if (!file_exists($archive_dir)) {
        `mkdir -p $archive_dir`;
    }

    $height = $exif['COMPUTED']['Height'];
    $width = $exif['COMPUTED']['Width'];
    // loop the different file sizes we want to create
    foreach ($cam_config['dir_array'] as $dir=>$new_width) {

        if ($dir == 'full') {
            // upload fullsize image to s3
            $s3->putObjectFile($file_full_path, $cam_config['bucket'], baseName($file_full_path), S3::ACL_PUBLIC_READ);
            `cp $file_full_path $archive_dir`;
        }
        else {
            // resize the image
            $new_height = ceil($new_width/$width * $height);
            $src = @imagecreatefromjpeg($file_full_path);
            $dst = @imagecreatetruecolor($new_width, $new_height);
            @imagecopyresampled($dst, $src, 0, 0, 0, 0, $new_width, $new_height, $width, $height);

            $new_file = $cam_config['tmp_dir'] . $file_base . '-' . $dir . '.jpg';
            @imagejpeg($dst, $new_file, 90);
            @imagedestroy($src);
            @imagedestroy($dst);

            // upload resized image to s3
            $s3->putObjectFile($new_file, $cam_config['bucket'], baseName($new_file), S3::ACL_PUBLIC_READ);
            `cp $new_file $archive_dir`;
        }

        // insert the photo into the database
        $picture_row = array(
            'cam_id'=>$cam_config['cam_id'],
            'timestamp'=>$timestamp,
            'filename'=>$file_base
        );

        $result = $db->createRow("picture", $picture_row);
    }

    `rm -f $file_full_path`;
    `rm -f {$cam_config['tmp_dir']}*.jpg`;
}

// remove the pid
unlink($cam_config['tmp_dir'] . 'process-incoming.pid');

With this script running on a cron we can now process any new file that gets uploaded to our server, upload it to Amazon S3, and log it in our database. With the photos now stored in the database we can easily setup a simple homepage to display the latest photo, and produce an archive of photos taken over time.

Summary

With the help of a few powerful open source programs and libraries and a small bit of hardware we are able to easily setup a camera that takes a photo every 5 minutes, and posts it to the Internet. You can view the webcam I setup at www.fauquiercam.com. In the future I will extend the processing further such that after every day, week, or month a time lapse is generated using mencoder.

Auto Scaling with HAProxy

In my last post I showed you how to setup Auto Scaling with Elastic Load Balancing. In this post I will show you how you can utilize Amazon’s Auto Scaling with an instance running HAProxy, and have HAProxy automatically update it’s configuration files when your setup scales up and down.

The Architecture

For the rest of this post I am going to assume that we have two ec2 images we are using. The first image is our load balancer running HAProxy, this image is setup to forward incoming traffic to our second image, which is the application image. The application image will be setup with Amazon’s Auto Scaling to build scale up and down depending on the load of the instances. Lets assume our load balancer image has a AMI of ami-10000000 and our application image has an AMI of ami-20000000.

Auto Scaling Setup

The setup for the auto scaling will be pretty similar to what we did in the previous post. First we will setup a launch config:

as-create-launch-config auto-scaling-test --image-id ami-20000000 --instance-type c1.medium

Then we will create our auto scaling group:

as-create-auto-scaling-group auto-scaling-test --availability-zones us-east-1a --launch-configuration auto-scaling-test --max-size 4 --min-size 2

You will notice we ran this command without setting a load balancer. Since we are using HAProxy we do not need to set this up. Finally we will create our trigger for the scaling:

as-create-or-update-trigger auto-scaling-test --auto-scaling-group auto-scaling-test --measure CPUUtilization --statistic Average --period 60 --breach-duration 120 --lower-threshold 30 --lower-breach-increment"=-1" --upper-threshold 60 --upper-breach-increment 2

Once you have executed these commands, you will have 2 new application instances launched. This presents us with a problem, in order to send traffic to these instances we need to update the HAProxy config file so that it knows where to direct the traffic.

Updating HAProxy

In the following example I am going to show you how with the power of a simple script we can monitor for instances being launched or removed, and update HAProxy accordingly. I will be using S3 storage, PHP and the Amazon PHP Library to do this. You can use any programming language you prefer if you care to rewrite this code, the key is understanding what is going on.

In order for us to identify the running application instances we will need to know the AMI of the instance. We know our application instance has an AMI of ami-20000000. We could store this information straight into our script, however if we ever rebuilt the application instance, we would then have to update the script, which would then force us to have to rebuild our HAProxy instance. Not a lot of fun. What I like to do, is to store the AMI in S3. That way if I ever update my application instance, I can just change a small file in S3 and have my load balancer pick up those changes. Lets assume I stored the AMI of the application image in a file called ami-application and uploaded it to an S3 bucket so that it can be found at http://stefans-test-ami-bucket.s3.amazonaws.com/ami-application.

The Script

Basically what our script is going to do is the following:

  • Get the AMI from S3 of our application image
  • Get a list of all running instances from Amazon, and log which ones match our application image
  • Get a default config file for HAProxy
  • Append the IP addresses of the running application instances to the config file
  • Compare the new config file to the old config file, if they are the same no action is needed, if they are different, replace the haproxy.cfg and restart the server

Our default config file for HAProxy will essentially be the full config without any server directives in the backend section. For example, our default config file could look something like:

global
        log 127.0.0.1   local0
        log 127.0.0.1   local1 notice
        maxconn 50000
        user haproxy
        group haproxy
        daemon
        chroot /var/chroot/haproxy

defaults
        log       global
        mode    tcp
        option   httplog
        option   forwardfor
        retries   2
        redispatch
        maxconn       50000
        timeout connect    10000
        timeout client        30000
        timeout server       60000
        stats uri /ha_stats
        stats realm Global\ statistics
        stats auth myusername:mypassword

frontend www *:80
        maxconn 40000
        mode http
        default_backend www_farm

backend www_farm
        mode http
        balance roundrobin

Now that we have our default config file, all we have to do to generate our HAProxy config is to add a server line to the end of it for each instance we have running. Lets look at the PHP code to do that:

<?php
// setup our include path, this is needed since the script is run from cron
ini_set("include_path", ".:../:./include:../include:/path/to/this/script/update-haproxy");

// including the Amazon EC2 PHP Library
require_once("Amazon/EC2/Client.php");

// include the config file containing your AWS Access Key and Secret
include_once ('.config.inc.php');

// location of AMI of the application image
$ami_location = 'http://stefans-test-ami-bucket.s3.amazonaws.com/ami-application';
$ami_id = chop(file_get_contents($ami_location));


// connect to Amazon and pull a list of all running instances
$service = new Amazon_EC2_Client(AWS_ACCESS_KEY_ID,
                                       AWS_SECRET_ACCESS_KEY);

$response = $service->describeInstances($request);

$describeInstancesResult = $response->getDescribeInstancesResult();
$reservationList = $describeInstancesResult->getReservation();


// loop the list of running instances and match those that have an AMI of the application image
$hosts = array();
foreach ($reservationList as $reservation) {
        $runningInstanceList = $reservation->getRunningInstance();
        foreach ($runningInstanceList as $runningInstance) {
                $ami = $runningInstance->getImageId();

                $state = $runningInstance->getInstanceState();

                if ($ami == $ami_id && $state->getName() == 'running') {

                        $dns_name = $runningInstance->getPublicDnsName();

                        $app_ip = gethostbyname($dns_name);

                        $hosts[] = $app_ip;
                }
        }
}

// get our default HAProxy configuration file
$haproxy_cfg = file_get_contents("/share/etc/.default-haproxy.cfg");

foreach ($hosts as $i=>$ip) {
        $haproxy_cfg .= '
        server server'.$i.' '.$ip.':80 maxconn 250 check';
}
// test if the configs differ
$current_cfg = file_get_contents("/path/to/haproxy.cfg");
if ($current_cfg == $haproxy_cfg) {
        echo "everything is good, configs are the same.\n";
}
else {
        echo "file out of date, updating.\n";
        file_put_contents('/path/to/this/script/.latest-haproxy.cfg', $haproxy_cfg);
        system("cp /path/to/this/script/.latest-haproxy.cfg /path/to/haproxy.cfg");
        system("/etc/init.d/haproxy reload");
}
?>

I think this script is pretty self explanatory, and does what we outlined as our goals for it above. If you ran the script from command line your HAProxy config file would get updated, and the server would be restarted.

Now that we have our working script, the last thing we need to do is setup this script to run on cron. I find updating every 2-5 minutes is sufficient to keep your config updated for auto scaling.

One of the nice benefits of having this script is it allows you the ability to easily pre-scale your solution if you know you are going to receive a big traffic spike. All you have to do is launch as many new instances of your application image and the script will manage the setup of HAProxy for you.

Conclusion

With under 100 lines of code and a few tools we were able to setup a script to keep the HAProxy config file up to date with your running application instances. This allows us the ability to use HAProxy instead of Amazon Load Balancing, but still get to have all the benefits of Auto Scaling. Lastly it is to note this script is just an example, and should be tailored to your own environment as you see fit. It is also best to store this script, and the default HAProxy config in a EBS volume if possible, as it will save you from having to rebuild your instance.