MPIHaloArrays

MPIHaloArrays is a high-level array type to help with halo, or ghost-cell exchanges commonly found in large-scale PDE problems. Very similar in goals and design to MPIArrays.jl and ImplicitGlobalGrid.jl. Domains can be decomposed into 1, 2, or 3D topology. Currently arrays are limited to 1, 2, or 3D.

Installation

The package is registered and can be installed with

pkg> add MPIHaloArrays

Basic Usage

This example shows how to set up the initial array, fill the halo/domain cells, and do a halo exchange

using MPI, MPIHaloArrays

MPI.Init()
rank = MPI.Comm_rank(comm)

# Create the MPI topology
topo = CartesianTopology([4,4], # use a 4x4 decomposition
                         [true, true]; # periodic in both dimensions   
                         do_corners=false) # exchange corner halo regions (significant speed advantage if you don't need it)

nhalo = 2 # Number of halo cells in each dimension (fixed for all dimensions)
N = 200

# create the array type; this pads the data on all sides with halo regions
x = MPIHaloArray(rand(N,N), topo, nhalo)

# fill all the halo regions with -1
fillhalo!(A, -1)

# fill the domain region with the current rank
filldomain!(A, rank)

# local (current rank) indexing works just like a normal array
x[1,1] .= 2.0

# Get the local/global indices of the _domain_ data (not including the halo cells)
ilo, ihi, jlo, jhi = local_domain_indices(x) # -> useful for looping without going into halo regions
ilo_g, ihi_g, jlo_g, jhi_g = global_domain_indices(x)

# Exchange data with neighbors
updatehalo!(x)

GC.gc()
MPI.Finalize()

Scatter and gather operations are also defined with scatterglobal and gatherglobal.

rank = 0 # MPI rank to scatter from / gather to

# start with a global Base.Array type to decompose and scatter to each rank
ni = 512; nj = 256
A_global = reshape(1:ni*nj, ni, nj);

# scatter - this internally converts A_global to multiple halo arrays. This is why
#           the nhalo and topology types are needed
A_local = scatterglobal(A_global, root, nhalo, topology) # -> returns a MPIHaloArray

# do some work...

# and now gather the decomposed domain and store on the root rank of choice
A_global_result = gatherglobal(A_local; root=root) # -> returns a Base.Array

At the moment, reductions are not implemented, but will be in the future...