There’s no place like ::1

Map Crowd Reduce

| Comments

My latest open project is a “SETI-at-home-like infrastructure for massively distributed CPU-intensive jobs based on HTML5 WebWorkers and node.js for distributing tasks” (quote from Pedro Teixeira’s blog).

It’s called Map Crowd Reduce and it’s available on github.

This project made into the final SAPO Codebits 2010 programming contest, but unfortunately didn’t win :)

The project consists of a node.js server that starts a HTTP server on port 3000. To start a new computation, you must supply some javascript functions: a split/segmenter function that generates jobs, a map function, and a reduce function.

After that, the system gives you an URL that you can share with anyone wanting to contribute with CPU to the computation. The client side functions are run inside a WebWorker so it never blocks the main UI thread.

With the code there are 2 prewritten examples:

  • find the number of primes between 0 and 60000 (brute force)

this uses a brute-force method of testing if each number is a prime, from 0 to 60000. it runs very fast and can be used to debug.

  • find a MD5 collision with the MD5 on all words with 5 or less alfa words

this lasts 15 minutes on a single machine. when I tested it on the main stage, 50 people connected their browsers at it lasted like 20 seconds (#win).

We are currently trying to write a parallel Mandelbrot fractal using map reduce :) it will be awesome.

Finally, I couldn’t have built this project without the help of some friends.