fftMPI documentation

Using the fftMPI library from your program

The source code for applications (apps) in the test dir are examples of how to use fftMPI from C++, C, Fortran, and Python. The apps that start with the word "simple" are about 150 lines each and are a complete illustration of how to setup a distributed grid and perform an FFT. The apps that start with "test3d" or "test2d" are more complex; they take many optional command-line arguments that exercise all the options and methods that fftMPI provides. An explantion of all the command-line options is on the runtest doc page.

Details for all 4 languages are given here of what any application needs to include to perform 3d FFTs or 3d Remaps. Just change "3" to "2" to perform 2d FFTs or 2d Remaps. You can perform both 2d and 3d FFTs or Remaps from the same app, by including both the 2d and 3d header files and linking to both the 2d and 3d fftMPI library files.

Calling fftMPI from your source code

Any file that makes a call to fftMPI needs to include a header file that defines the fftMPI API. These code lines also show how to allocate grids for single or double precision FFTs or Remaps.

IMPORTANT NOTE: As explained on the compile doc page, it is a compile-time choice to build fftMPI to perform either single or double precision FFTs. Your application must allocate its FFT grid data to match the library precision setting. You can look at the test apps to see how they do this in a flexible way so that the app can choose to perform its FFTs in either single or double precision.

In these code examples, fftsize is the number of grid points owned by the processor for an FFT. The "2" is for a pair of (real,imaginary) values stored for each grid point. If a remap is being performed "2" could be a different value, as explained on the remap API doc page. The value of fftsize to use in an app is returned by the setup() method and may include additional memory space for later stages of the FFT.


#include "fft3d.h"      // if performing FFTs
#include "remap3d.h"    // if performing Remaps
using namespace FFTMPI_NS; 
work = (float *) malloc(2*fftsize*sizeof(float));   // single precision
work = (double *) malloc(2*fftsize*sizeof(double)); // double precision 

The header files fft3d.h and remap3d.h define a typedef for FFT_SCALAR which is set to "float" or "double" depending on the precision you build fftMPI with. So you can define work vectors to be of type FFT_SCALAR if you wish.


#include "fft3d_wrap.h"     // if performing FFTs
#include "remap3d_wrap.h"   // if performing Remaps 
work = (float *) malloc(2*fftsize*sizeof(float));   // single precision
work = (double *) malloc(2*fftsize*sizeof(double)); // double precision 

As with C++, you can define work vectors to be of type FFT_SCALAR.


use iso_c_binding   ! use these lines in any subroutine/function that calls fftMPI
use fft3d_wrap      ! if performing FFTs
use remap3d_wrap      ! if performing Remaps 
real(4), allocatable, target :: work(:)   ! single precision 
real(8), allocatable, target :: work(:)   ! double precision


import numpy as np
from mpi4py import MPI
from fftmpi import FFT3dMPI     # if performing FFTs
from fftmpi import Remap3dMPI   # if performing Remaps 
work = np.zeros(2*fftsize,np.float32)    # single precision
work = np.zeros(2*fftsize,np.float)      # double precision 

To use fftMPI from Python, you must have Numpy and mpi4py installed in your Python. This is discussed further below.

Building your app with fftMPI

These header files listed above for each language are all in the fftMPI src directory. When you compile your app, it must be able to find the appropriate header file. The library files libfft3dmpi.so and libfft2dmpi.so (or *.a equilvalents) are also in the fftMPI src directory, after building the library:

For C++ and C, the compile and link commands can be something like this:

mpicxx -I/home/me/fftmpi/src -c test3d.cpp
mpicxx -L/home/me/fftmpi/src test3d.o -lfft3dmpi -o test3d 

where the -I and -L switches give the path to the fftMPI src dir.

For Fortran, the fft3d_wrap.f90 and/or remap3d_wrap.f90 files need to be in the directory with your app files. So you can copy it there; the fft*wrap.f90 files are already present in the test dir. The compile and link commands can then be something like this:

mpif90 -I/home/me/fftmpi/src -c fft3d_wrap.f90
mpif90 -I/home/me/fftmpi/src -c test3d_f90.f90
mpif90 -L/home/me/fftmpi/src test3d_f90.o fft3d_wrap.o -lfft3dmpi -lstdc++ -o test3d_f90 

where the -I and -L switches give the path to the fftMPI src dir.

For Python, there is no build step. However your Python script needs to be able to find the src/fftmpi.py and library files at run time; see the next section.

Note that if you built fftMPI as a static library, using an external 1d FFT library (FFTW or Intel MKL) then the link lines above also need to include the 1d FFT library. If you built fftMPI as a shared library this is not needed; the dependencies on the 1d FFT libraries were satistified when the fftMPI library was built. See the buildtest doc page for details on how to include these 1d FFT libraries in the link. To use the provided KISS FFT library (just a header file), no additional link arguments are needed.

Running your app with Python

To use fftMPI from Python, there are 3 things to consider. If any of them are not satisfied you will get a run-time error when you launch your Python script. Each topic is discussed in more detail below:

(1) Numpy and mpi4py must be installed in your Python

The allocation of the FFT grid in your app must be via Numpy vectors or arrays, which are then passed to fftMPI. Numpy is a numeric package already installed in most Pythons.

The mpi4py package is a wrapper on MPI and allows you to run Python scripts in parallel.

You can test for both of these packages as follows:

% python
>>> import numpy as np
>>> from mpi4py import MPI 

If an error results with either import command you will have to install the corresponding package.

For Numpy, go to http://www.numpy.org and follow the directions to download/install Numpy.

For mpi4py, go to http://www.mpi4py.org, unpack it, and install it via pip:

pip install mpi4py 

Or if you are using anaconda for your Python package management, you can type the following to download and install either package

conda install numpy
conda install mpi4py 

After mpi4py imports sucessfully, try running the following test.py script on a few processors, like this mpirun -np 4 python test.py

# test.py script
from mpi4py import MPI
me = world.rank
nprocs = world.size
print "Me %d Nprocs %d" % (me,nprocs) 

You should get 4 lines of output "Me N Nprocs 4", where N = 0,1,2,3.

IMPORTANT NOTE: When mpi4py is installed in your Python, it compiles an MPI library (e.g. inside conda) or uses a pre-existing MPI library it finds on your system. This MUST be the same MPI library that fftMPI and your app are built with. If they do not match, you will typically get run-time MPI errors when your app runs.

You can inspect the path to the MPI library that mpi4py uses like this:

% python
>>> import mpi4py
>>> mpi4py.get_config() 

(2) Your app must be able to find the src/fftmpi.py file

The src/fftmpi.py file is a Python wrapper on the fftMPI C interface. Python loads it when a statement like this is executed:

from fftmpi import FFT3dMPI 

If this fails because it cannot find fftmpi.py, you can do one of two things:

(a) Set the PYTHONPATH environment variable to include the fftMPI src dir, like this, either from the command line or in your shell start-up script:

setenv PYTHONPATH $PYTHONPATH:/home/sjplimp/fftmpi/src   # csh or tcsh
export PYTHONPATH=$PYTHONPATH:/home/sjplimp/fftmpi/src   # bash 

(b) From your Python app, you can augment the search path directly:

path_fftmpi = "/home/me/fftmpi/src"
from fftmpi import FFT3dMPI 

(3) Your app must be able to find the fftMPI library file

Python loads the fftMPI library when a statement like this is executed. It looks for the fft3dmpi.so file. Note that you must build fftMPI as a shared library to use it from Python.

fft = FFT3dMPI(world,precision) 

If this fails because if cannot find the library file (similar for 2d FFTs, or 3d/2d Remaps), you can do one of two things:

(a) You can add the fftMPI src dir to your LD_LIBRARY_PATH environment variable, e.g.

setenv LD_LIBRARY_PATH $LD_LIBRARY_PATH:/home/me/fftmpi/src   # csh or tcsh
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/home/me/fftmpi/src   # bash 

(b) You can "install" the fftMPI library file in a location your where your system can find it, such as /usr/local/lib. See the compile doc page for details on installing fftMPI after you build it. This typically requires super-user or sudo priveleges.