Amazon

Persistent storage for Amazon EC2 with EBS

by

Last week (20th August) Amazon Web Services released Amazon Elastic Block Store (EBS). Amazon EBS provides block level storage volumes for use with Amazon EC2 instances. Amazon EBS volumes are off-instance storage that persists independently from the life of an instance. Amazon Elastic Block Store provides highly available, highly reliable storage volumes that can be attached to a running Amazon EC2 instance and exposed as a device within the instance. Amazon EBS is particularly suited for applications that require a database, file system, or access to raw block level storage.

This is a follow up to my posting, http://blog.mohanjith.net/2008/02/amazon-ec2-with-rock-solid-persistent.html. The method described there is obsolete with the release of Amazon EBS. None the less you can place the jail file system inside the EBS block device and not worry about having to build AMIs from time to time.

Hope this information helps someone.

Duplicity chokes on OSError: [Errno 24] Too many open files

by

It was little bit too scary. Duplicity backup scripts were failing on the EC2 instances again, this time around it was not about not able to reach S3, but having too many files open. That was weird because it didn’t give such a error in the past. However the work around was to increase the maximum number of file descripters allowed for the user that was running the backup script.

How ever finding this solution was tought, actually it was a FreeBSD forum that had the solution. I though I would just write it down for Linux.

Step 1: Find out the current limit

To find out the current file descripter limit for a given use, log in as the particular user and run the following command.

 $ ulimit -n

By default on Debian it would be 1024.

Step 2: Increase the limit

You would have to edit /etc/security/limits.conf. You will find details on how to setup different limits in limits.conf itself. The record that you have to put in should look like the following. 

username hard nofile 2048

Step 3: Log out and Log back in

You would have to log out and log back in as the user that we updated the file descripter limit. Then run the following command. 

 $ ulimit -n

You should see the updated file descripter limit.

Hope this helps someone like me in desperation to get the backups in track. I would be doing more investigation as to why there are so many files open. If I find anything interesting I would definitely blog about it. Also for everyone’s reference there is a bug filed at the Savanah bug tracker by someone else who ran into the same issue

Amazon Web Services goes down

by

Amazon Web Services goes down, takes out some Web 2.0 sites, but not the sites that I was running on EC2. I got a shock when I got a Google alert that had news items about Amazon Web Services are down, I immediately went over to all of the sites I’m responsible for, but all of them were live and kicking. So the next stop was checking my mails, and sure there were mails of the cron job to do the backups failing.

I was using duplicity to backup complete file system of the EC2 instances, I have blogged about my approach in Amazon EC2 with rock solid persistent storage. I had the cron job failing during the S3 downtime, but I was serving all requests without a hitch.

I suspect the sites that went down were using PersistenceFS. Reading there documentation, they assume that S3 is going to be available at all times dispite the 99.99% uptime guarantee. That is a major design flaw. Also it is a utter waste of large storage provided in EC2 during the runtime.

I’m glad to say that despite the S3 downtime all my sites were running. I think the sites that went down reconsider their setup. Also I strongly recommend running redundant EC2 instances for any one planning on hosting sites.

Amazon EC2 with rock solid persistent storage

by

With the power of duplicity and chroot we can make a Amazon EC2 image that is as good as a harware node, i.e. with persistent storage. Let me explain how to do it your self as well. However I’ll be leaving out the minute details.

Step 1: Start an instance of a public AMI

I would recommend ami-76cb2e1f because i you are able to use the same image for x-large and large instance powering up. Also it has the ec2 ami tools installed and patched. Login to the instance as root using the certificate you provided when starting the instance. Also do not forget to give the following as User Data.

chroot_bucket=[your_bucket_name]

Step 2: Download and install duplicity and boto

You need to install duplicity 0.4.9 or later and boto 1.0 or later.

Step 3: Create a PGP key

Run the following and follow the instructions that will appear. 

 # gpg --gen-key

Please note down the key id because we need it later on. It should look something like 860BCFF6.

gpg: key 860BCFF6 marked as ultimately trusted

Step 4: Install libpam-chroot

You have to install libpam-chroot for it to be possible to push the user inside the chroot when the user logs in via ssh.

Step 5: Create the chroot

Create the chroot and install all the applications you need inside the chroot. Read about how to create a chroot in a debian system here. Create your users inside the chroot. It is important that you understand how chroot works as well.

Step 6: Push the users to chroot

You need to change /etc/security/chroot.cnf and add a line similar to bellow.

[username] /mnt/chroot

Step 6: Download the scripts

You need to download the scripts archive that contains the scripts necessary to do all the magic to ensure that data actually persist. Download it from http://www.mohanjith.net/downloads/amazon/ec2/ec2-chroot-persistence-1.0.tar.gz

Step 7: Extract and edit the scripts
Extract the scripts from out side the chroot, preferably in /.

 # cd /
 # tar -xzf [path_to_archive]/ec2-chroot-persistence-1.0.tar.gz

You need to edit /etc/init.d/ec2 and /etc/ec2/cron and change the lines that look like bellow.

export AWS_ACCESS_KEY_ID=[your_aws_access_key_id]
export AWS_SECRET_ACCESS_KEY=[your_aws_secret_access_key]
export PASSPHRASE=[your_gpg_passphrase]
export gpg_key=[your_gpg_key_id]

Step 8: Set up the scripts

You will also have to setup a cron job outside the chroot to backup the data to S3. The script to invoke is /etc/ec2/cron. I would recommend hourly backups, but anything more frequently will be bad because the time it takes to backup will increase drastically.

You will also have to make sure ec2 service (/etc/init.d/ec2) is run on power on, power off and restart. To do that you will have to create sym links to /etc/init.d/ec2 from /etc/rc0.d/K10ec2, /etc/rc3.d/S90ec2, /etc/rc4.d/S90ec2, and /etc/rc6.d/K10ec2.

Step 9: Where to persist data.

Run the bellow as root outside the chroot.

curl http://169.254.169.254/2007-08-29/user-data > /tmp/my-user-data

Step 8: Remaster the AMI

Step 10: Create your machine image
Read more about creating an machine image at Amazon EC2 Getting started guide here.

Step 11: Back up your chroot
Run /etc/ec2/cron to back up the chroot.

Step 12: Power off and power on

Power off the instance you are running with the public image and when it has properly shutdown, start the image we just created in step 10 with the chroot_bucket with the same bucket you provided when you power up the public image.

All the data in /mnt/chroot is backed up to S3 by /etc/ec2/cron and when the instance is started after a shutdown /mnt/chroot is restored from S3. The script is configured to backup on power down but it is always recommended to run /etc/ec2/cron just before a power down.

You might also want to set up dynamic DNS for your instance such that you don’t have to always try hard to remember the ugly public DNS provided by Amazon. You can use ddclient to update the dynamic DNS service with your new IP. You can install ddclient inside the chroot.

This method was tested for more than 1 month and everything worked smoothly for me, but depending on your configuration your experience may defer. It is always good to test before you use in production environment.

Why I was silent last week?

by

I was extremely busy, not a minute to blog last week. I was assigned the task of coming up with a solution to use Amazon EC2 with persistence such that it feels like a physical server to the users.

I successfully managed to set up such an AMI but, unfortunately I’m not in a position to make the image public, but definitely I’ll be bloggin about how you can set up a similar AMI.

Stand by for the all new discovery!