FUSE-based file system backed by Amazon S3

Amazon S3 Indroduction

Amazon s3 is a remote file storing system. It is totally different from Normal Servers like Apache, FTP server etc. If your site burns more bandwidth due to backups or multimedia contents such a video, image, audio etc, the maintenance cost will be very high. For a limited number of page views, local file saving mechanism is ok. Because the cost of bandwidth is comparatively low. But when the data transfer rate become higher, the bandwidth cost should be considered.

In such a case instead of saving files or images in local servers, You can use Amazon s3 server (S3 stands for Simple Storage Service). For this you have do two things

1) Register on http://www.amazon.com/gp/browse.html?node=16427261

2) Use api for uploading your files into s3 for web applications

3) Mount Amazon S3 bucket and use as a local file system for backups.

S3fs Indroduction

s3fs is a FUSE (Filesystem in Userspace) filesystem that allows you to mount an Amazon S3 bucket as a local filesystem. It stores files natively and transparently in S3 (i.e., you can use other programs to access the same files). Maximum file size=5G.Its quite useful and stable, e.g., can be used to easily copy daily backup tarballs to s3

To use it:

1. get an amazon s3 account
2. download the source, s3fs-r130-source.tar.gz (I’ve used Ubuntu 7.10 ) and slap the binary in, say, /usr/bin/s3fs
   1. you’ll need at least fuse-2.6
   2. for fedora probably need to do: yum install fuse-devel
   3. for ubuntu probably need to do something like: sudo apt get fuse-libs
   4. I got one dependency libcurl-dev, libxml2-dev libfuse-dev ( sudo apt-get install libcurl4-openssl-dev libxml2-dev libfuse-dev )
3. Installation
   1. $ sudo tar -xzvf s3fs-r130-source.tar.gz
   2. $ cd s3fs
   3. $ sudo make
   4. $ sudo make install
4. Mounting Amazon S3 Bucket

$ sudo /usr/bin/s3fs mybucket -o accessKeyId=myaccessKeyId -o secretAccessKey=mysecretAccessKey /mnt

Now you can read / write files to your bucket using Linux commands.

List the contents in the Bucket mounted to /mnt/ directory

$ sudo ls /mnt

s3fs will retry s3 transactions on certain error conditions. The default retry count is 2, i.e., s3fs will make 2 retries per s3 transaction (for a total of 3 attempts: 1st attempt + 2 retries) before giving up. You can set the retry count by using the “retries” option, e.g., “-oretries=2″

Options

• accessKeyId
• secretAccessKey
• defailt_acl ( default=”private”)
  • the default canned acl to apply to all written S3 Onjects
  • any created files will have this canned acl
  • any updated files will also have this canned acl applied
• prefix (default=”") (comming soon!)
  • a prefix to append to all S3 objects
• retries (default=”2″)
  • number of times to retry a failed S3 transaction
• use_cache (default=”" which means disabled)
  • local folder to use for local files cache