{"id":547,"date":"2010-06-25T14:47:12","date_gmt":"2010-06-25T14:47:12","guid":{"rendered":"https:\/\/www.mohanjith.net\/blog\/?p=547"},"modified":"2010-06-25T18:10:59","modified_gmt":"2010-06-25T18:10:59","slug":"building-r2g-tools","status":"publish","type":"post","link":"https:\/\/mohanjith.net\/blog\/2010\/06\/building-r2g-tools.html","title":{"rendered":"Building R2G Tools"},"content":{"rendered":"

I started out building a tool to discover recently dropped domains and in the process domains search engine with a twist. You can check it out at R2G Tools<\/a> . This was a extremely challenging project, with more than 120 million domains in the world and extremely tight budget. I finally pulled it off last week end. You would be surprised that whois.r2g.in is running off 3 commodity servers (mainly for redundancy, I could have squeezed it into 1 with only slight performance hit).<\/p>\n

What’s behind<\/strong><\/p>\n

Database; <\/strong>MongoDB<\/a> is the main database backend, with MySQL<\/a> used to hold non domain records. MongoDB was chosen because of it’s ability to handle large amount of objects, schemaless and ability to index keys. Thanks to MongoDB I can search, count and even insert and update records in few seconds (most operations are few milliseconds).<\/p>\n

Web application framework; <\/strong>whois.r2g.in is completely built with CodeIgniter<\/a>. It’s my framework of choice. I like it’s clean and lean architecture.<\/p>\n

Servers;<\/strong> I’m running Lighty<\/a> behind Varnish<\/a> HTTP accelerator. I also use eaccelerator<\/a> to optimize PHP code. I also have a memcached<\/a> instance on the web server to cache data from MySQL. Goal is to speed up whois.r2g.in as much as possible. There are 2 MongoDB instances sharing the load which the application connects. Everything is running off 3 Athlon X2 servers, 2 running MongoDB instances and one running the application.<\/p>\n

Why use both MongoDB and MySQL?<\/strong><\/p>\n

That’s because I didn’t want to put all my eggs in one basket. Any data that doesn’t change much like WHOIS servers put into a MySQL table. Information such at WHOIS data were stored in a MongoDB collection.<\/p>\n

What I gained?<\/strong><\/p>\n

Lot of experience about data mining, storing and analyzing. I learned a lot about how to optimize data mining, I managed to bring down the time it takes to analyze all the domains for drops to few hours from few days by just pre sorting the zone files. I also gained few rare domains and high page rank domains. I have become a domainer thanks to the project \ud83d\ude00<\/p>\n

Please head over to R2G Tools<\/a> and give it a try,\u00a0 you might discover a great domain while you are there and make a huge profit. Do not forget to send me some feed back \ud83d\ude42<\/p>\n