Where to put all that data? S3 resources

Lets face it, web apps are getting bigger. At least in terms of the amount of data they store. Take two big poster siblings, Flickr and del.icio.us. All those user accounts, all that interlinking between then, all that meta data. Technorati is another example – trying to keep up with all blogs everywhere in as near to real time as possible has got to be seriously data intensive. And at times this has shown through, all have had problems at some point in their short histories.

If you are looking to get into the world of big, community focused, web apps then you need space. And I know I cant afford a huge outlay on storage and bandwidth, even most small web shops would probably have something of a problem with the large outlay, limited short term money making problem.

An interesting solution to this would appear to exist in the shape of S3 from Amazon. It has been around a little time, and others have been intrigued enough to mention it, but I decided not to ramble on until I’d had a brief play. Well, it’s quite cool.

Amazon themselves provide a number of useful code examples in different languages, including Perl, Python, C\#, Java, Ruby and PHP. They also provide examples of the raw HTTP requests for thos that way inclined.

This openness (OK, so they dont really have a choicee, the whole product consists of an open API, it being closed would seem to limit it’s usage!), as always, has allowed the community to run with the idea, building up sample apps and some easy to use wrappers:

Nothing here feels complete as yet. Most are early hacks by people getting on board early and giving the rest of us a starting point.

Overall I like things like S3. If it works reliably and scales well then it could become something that just sits behind all sorts of realy cool ideas, preventing data loss and removing another potential headache for the small developer.

Any other resources, or ideas, more than welcome.