Industrial Hypertext

Perl Success Stories

How the Wayback Machine Works

"Almost everything we can, we do in Perl -- for ease of portability, maintability, flexibility. Because there's so much horsepower we don't really require a tight system. The crawlers record pages into 100MB files in a standard archive file format, and then store it on one of the storage machines. Those are just normal PCs with four IDE hard drives, and its just writes along until it's filled up and then it goes to the next one. It goes through a couple of these machines a day: hundreds of gigabytes a day. The total gathering speed when everything is moving is about 10 terabytes a month, or half a Library of Congress a month."

It appears that Perl is used to run the largest database in the world!

"This is 100 terabytes. At that size, this is the largest database ever built. It's larger than Walmart's, American Express', the IRS. It's the largest database ever built."