mpimap 1.0.4
Parallel function mapping using mpi4py

# Mpimap

This package is a wrapper for `mpi4py` to allow for the easy running of functions in parallel on a single computer or on multiple nodes of a HPC cluster.

The code used to implement the `mpimap` methods will also function when no `mpi` environment is used, or only a single processor is specified.

## Setup

Once imported, create an instance of the `Mpimap` class:

```
mpi = mpimap.Mpimap()
```

To have each `mpi` process print its information, use:

```
mpi.info()
```

At this point, all `mpi` instance still continue to process all lines within the `script.py` being run or command sent to the interpreter. To put all worker nodes into a "listening" state where they only accept commands sent from the head process, use:

```
mpi.start()
```

From this point, command in the `script.py` running or command sent to the interpreter will only be processed by the head process. To determined the status of each worker process after it has been started, use:

```
mpi.status()
```

To kill all worker nodes once finished, use:

```
mpi.stop()
```

## Functions

To run code on each of the worker nodes once they have being "listening" for jobs, include the code in a function with no arguments and use:

```
mpi.run(func)
```

`Mpimap` include a `map()` function which behaves as the `builtin` version included with python:

```
output = mpi.map(func, args)
```

This will send a copy of the function to all worker nodes, and then queue the args list, sending values to each node not currently running a job. The input order is maintained by the output.

`Mpimap` also includes the function `gmap()`. This is a special instance of `map()` that is intended for running groups of jobs where an argument returning a "failed" state results in all jobs within that group being canceled:

```
output = mpi.gmap(func, args, groupind=0, failstate=None)
```

For this function, `args` is a list of lists. The argument `groupind` determines which entry in each list run by the function is used to determine that jobs group. The argument `failstate` is the value checked to determine if the job was a success or failure.

An addition static function is included called `gmatrix`. This can be used to generate a list of all possible combinations of two lists, include an input id number and append constants to all combinations:

```
x = [1, 2, 3]
y = [10, 20]
constants = ('a', 'b')
out = mpi.gmatrix(x, y, *constants)
```

This example would return:

```
>>> out
[('0', 1, 10, 'a', 'b'),
('1', 1, 20, 'a', 'b'),
('2', 2, 10, 'a', 'b'),
('3', 2, 20, 'a', 'b'),
('4', 3, 10, 'a', 'b'),
('5', 3, 20, 'a', 'b')]
```

## Example

To test the provided functions and check the difference in processing time between `builtin.map()` and `mpimap.map()`, run the example python script:

```
mpirun -n <Number of processors you want to use> python example.py
```

The full working example is given here:

```
import mpimap
import time

def func_cheap(*args):
	"""Do nothing"""
	return


def func_expensive(n):
	"""Basic factorising problem"""
	factors = set([])
	for i in xrange(n - 1):
		i = i + 1
		# Skip factors
		if i in factors:
			continue
		# Find factors
		if n % i == 0:
			factors.add(i)
			factors.add(n / i)

	return sorted(factors)


# Build mpi
mpi = mpimap.Mpimap(sleep=0, debug=False)
mpi.info()
mpi.start()

# Run function on all nodes
mpi.run(func_cheap)

# Set up function and arguments
args = range(5000, 10000)

# Not in parallel
t0 = time.time()
res = map(func_expensive, args)
dt = time.time() - t0
print '\nNon Parallel: {}'.format(dt)

# Parallel
t0 = time.time()
res = mpi.map(func_expensive, args)
dt = time.time() - t0
print '\nParallel: {}\n'.format(dt)

mpi.stop()
mpi.exit()
```