RE: Use Raspberry server farm for crawling / parsing webites on a larger scale

johnbeetem — Sun, 14 Jun 2015 19:58:02 GMT

While the RasPi 2 is a lot faster than the RasPi 1 and is viable for some desktop applications, it does have limited processing power, memory, disk I/O, and networking performance. Whether RasPi 2 has enough computing power to handle your application depends on what's limiting your performance. If you're compute-bound, RasPi 2 will probably disappoint you. If you're using a lot more than 1 GB RAM, RasPi 2 could be a disaster. If your application needs SATA disks to get decent performance, RasPi 2 will disappoint: it only has SD cards or USB 2.0, both of which are slow compared to SATA. If you need 100s of Mb/s of Ethernet (or GBE), RasPi's USB-based Ethernet will disappoint.

OTOH, if your application is limited by your external Internet connection, then it may not matter that much how much processing power your CPUs have if they spend most of their time waiting for data from the Internet.

If your application involves a lot of image processing, RasPi might perform quite well. That GPU is very powerful if your can find or write the software you need to make use of it.

So these are the tradeoffs I can think of. I suspect RasPi 2 won't have the performance to do what you want and you'd be better off with PCs. Alternatively, you could look into much higher performance SBCs such as the US$100 ODROID-XU3 Lite which has eight ARM cores (four 1.8 GHz Cortex-A15, four Cortex A7), 2 GB DDR, USB 3.0, and MMC.

RE: Use Raspberry server farm for crawling / parsing webites on a larger scale

DAB — Sun, 14 Jun 2015 18:48:27 GMT

One thing I learned early about computers is that if you have a vision and persistence, you can do most anything.

So I think that with some good system design, you should be able to make your server work.

After you get it functional, then you can do performance testing to see where better solutions could make it work faster and more reliable.

DAB