I started out building a tool to discover recently dropped domains and in the process domains search engine with a twist. You can check it out at R2G Tools . This was a extremely challenging project, with more than 120 million domains in the world and extremely tight budget. I finally pulled it off last week end. You would be surprised that whois.r2g.in is running off 3 commodity servers (mainly for redundancy, I could have squeezed it into 1 with only slight performance hit).
What’s behind
Database; MongoDB is the main database backend, with MySQL used to hold non domain records. MongoDB was chosen because of it’s ability to handle large amount of objects, schemaless and ability to index keys. Thanks to MongoDB I can search, count and even insert and update records in few seconds (most operations are few milliseconds).
Web application framework; whois.r2g.in is completely built with CodeIgniter. It’s my framework of choice. I like it’s clean and lean architecture.
Servers; I’m running Lighty behind Varnish HTTP accelerator. I also use eaccelerator to optimize PHP code. I also have a memcached instance on the web server to cache data from MySQL. Goal is to speed up whois.r2g.in as much as possible. There are 2 MongoDB instances sharing the load which the application connects. Everything is running off 3 Athlon X2 servers, 2 running MongoDB instances and one running the application.
Why use both MongoDB and MySQL?
That’s because I didn’t want to put all my eggs in one basket. Any data that doesn’t change much like WHOIS servers put into a MySQL table. Information such at WHOIS data were stored in a MongoDB collection.
What I gained?
Lot of experience about data mining, storing and analyzing. I learned a lot about how to optimize data mining, I managed to bring down the time it takes to analyze all the domains for drops to few hours from few days by just pre sorting the zone files. I also gained few rare domains and high page rank domains. I have become a domainer thanks to the project 😀
Please head over to R2G Tools and give it a try, you might discover a great domain while you are there and make a huge profit. Do not forget to send me some feed back 🙂