Group Actions
Engagement
  • Author Author: Former Member
  • Date Created: Date Created
  • Views 567 views
  • Likes 0 likes
  • Comments 2 comments
Related
Recommended

Use Raspberry server farm for crawling / parsing webites on a larger scale

Former Member

Hello forum,

 

being totally new to the Raspberry world, I hope to get a few comments and answers to my (maybe crazy) thoughts about using a Raspberry farm instead of Intel i7 systems to serve as a crawling and parsing system.

 

Application background: I have developed my own crawling  (internally using wget) and very specialized parsing software. The goal is to permanently crawl and parse a couple of 10 million websites websites for special purposes (very different from normal full text searches). It currently runs on OpenSuSe Linux on two Intel PC i7 3770 and 4770 nearing the end of beta tests. In the end I will need about 7-10 such PC to run all tasks permanently.

 

Since I must work on self-financed low-budget, I had the idea of employing a farm of Raspberry systems to handle these tasks. The software could be ported although this would mean some efforts, which must be justified.

 

However, this would only make sense if most of the following expectations could be fulfilled by a Raspberry server farm:

 

1) The initial hardware investment related to the same processing level must be less than 50% of what Intel based PCs would cost to justify the additional hassle, handling and software port.

2) The power consumption should be significantly lower.

3) Reliability should reach around 80% of the one of the Intel systems (i.e. somewhat less could be accepted).

4) Crawling / downloading could be separated from parsing. It would be sufficient if parsing was delegated to Raspberries. Crawling / downloading uses RAM disk as intermediate storage because the SSD drives quickly get to their limits and this might therefore better stay on the 32 GB RAM Intel PCs.

 

What do the Raspberry experts say?

 

a) Forget about it!

b) A viable concept worth considering.

c) Cost saving will not be significant enough.

d) Or what else?

 

Thank you very much in advance for your comments.

 

Best regards

FrankB

  • While the RasPi 2 is a lot faster than the RasPi 1 and is viable for some desktop applications, it does have limited processing power, memory, disk I/O, and networking performance.  Whether RasPi 2 has enough computing power to handle your application depends on what's limiting your performance.  If you're compute-bound, RasPi 2 will probably disappoint you.  If you're using a lot more than 1 GB RAM, RasPi 2 could be a disaster.  If your application needs SATA disks to get decent performance, RasPi 2 will disappoint: it only has SD cards or USB 2.0, both of which are slow compared to SATA.  If you need 100s of Mb/s of Ethernet (or GBE), RasPi's USB-based Ethernet will disappoint.

     

    OTOH, if your application is limited by your external Internet connection, then it may not matter that much how much processing power your CPUs have if they spend most of their time waiting for data from the Internet.

     

    If your application involves a lot of image processing, RasPi might perform quite well.  That GPU is very powerful if your can find or write the software you need to make use of it.

     

    So these are the tradeoffs I can think of.  I suspect RasPi 2 won't have the performance to do what you want and you'd be better off with PCs. Alternatively, you could look into much higher performance SBCs such as the US$100 ODROID-XU3 Lite which has eight ARM cores (four 1.8 GHz Cortex-A15, four Cortex A7), 2 GB DDR, USB 3.0, and MMC.

  • One thing I learned early about computers is that if you have a vision and persistence, you can do most anything.

     

    So I think that with some good system design, you should be able to make your server work.

     

    After you get it functional, then you can do performance testing to see where better solutions could make it work faster and more reliable.

     

    DAB