Repository: AMReX-Codes/amrex — pull request #1842 (cpp)
Title: Feature/non local parallel copy

## Summary

This PR introduces "Multiblock" capabilities to AMReX.

## Additional background

Multiblock refers to having multiple distinct computational domains, known as blocks, that have some relation to each other.
This PR proposes a generic NonLocalBC::ParallelCopy that performs copies between two FabArrays that might live on different blocks. The ParallelCopy respects

 - An index mapping such as already present in NonLocalBC (DTOS)
 - A projection function that will be called on either sender or receiver side to perform actions like e.g. swapping two components or changing the sign of velocity components
 
The proposed usage is

```cpp
NonLocalBC::ParallelCopy(dest, destbox, src, scomp, dcomp, ncomp, ngrow, index_mapping, fab_projection);
```

## Notes

- In the current implementation, `DTOS` needs to be a map between index spaces where inverse images of boxes are boxes again.
- The name `NonLocalBC::ParallelCopy` is confusing if it does something else than `amrex::ParallelCopy`. Alternatives?
- Parts of the implementation might be shared with other places in AMReX, such as in communication procedures of FabArray. But this is out of scope.
- In addition to `SwapComponents`, a more generic `PermuteComponents` is useful.
- The test is not commented yet.

## Checklist

The proposed changes:
- [x] add new capabilities to AMReX

The following tasks need to be done:
- [x] Define CPOs for dispatching packing and unpacking of communication buffers
- [x] Implement the first version without interpolation
- [x] Provide a test case to prove correct index mapping, projection and communication behaviour 
- [ ] Write a tutorial with an example of using at least two blocks in a simple simulation
- [x] Improve type documentation and requirements

**The advection example produces the following result**

<p align="center">
<img src="https://media.giphy.com/media/W5AKaJHUQqLbdyHO6O/giphy.gif"/>
</p>
<p align="center">Figure 1. A test run of the multi block advection test case. It was initiated with 4 MPI Ranks. It shows a surface plot of a scalar component and arrow glyphs of an advected vector.</p>

Resolved issues:
title: Add a hook so that an AmrLevel can force a post-timestep regrid
body: The motivation for this change is that sometimes, during the course of a coarse
timestep, the timestep will found to be unstable due to a lack of resolution. In
this case the coarse level may want to signal that more resolution is required
immediately (as opposed to waiting for a future regrid), because even a single
unstable step may cause significant harm to the integrity of the simulation. An
example is a spurious numerical detonation caused by insufficient resolution in
a thermonuclear burning region. If this numerically-seeded detonation is detected
at the end of a timestep, more levels can be added immediately so that the fine
levels can improve the situation and quench the detonation (since they will
overwrite the coarse data at the end of the timestep).

Each AmrLevel now has a member variable post_step_regrid. To trigger a regrid
at the end of a step, but before any fine timesteps would be taken, they only need
to set this variable to one. At the beginning of every timestep, it will be reset to 0.

Hints:

Fix patch:
diff --git a/Src/Base/AMReX_Extension.H b/Src/Base/AMReX_Extension.H
index 40e2da188bc..96253c1437b 100644
--- a/Src/Base/AMReX_Extension.H
+++ b/Src/Base/AMReX_Extension.H
@@ -151,7 +151,8 @@
 #define AMREX_FALLTHROUGH ((void)0)
 #endif
 
-#if !defined(AMREX_NO_NODISCARD) && defined(__has_cpp_attribute) && __has_cpp_attribute(nodiscard)
+// CI uses -Werror -Wc++17-extentsion, thus we need to add the __cplusplus clause
+#if !defined(AMREX_NO_NODISCARD) && defined(__has_cpp_attribute) && __has_cpp_attribute(nodiscard) && __cplusplus >= 201603L
 #   define AMREX_NODISCARD [[nodiscard]]
 #elif !defined(AMREX_NO_NODISCARD) && (defined(__GNUC__) || defined(__clang__))
 #   define AMREX_NODISCARD __attribute__ ((warn_unused_result))
@@ -159,6 +160,18 @@
 #   define AMREX_NODISCARD
 #endif
 
+// Note: following compilers support [[no_unique_address]]
+//   - Clang >= 9.0
+//   - GCC >= 9.0
+//   - MSVC >= 19.26
+// Using no unique address makes empty base class optimization for multiple policies much easier
+#if !defined(AMREX_NO_NO_UNIQUE_ADDRESS) && defined(__has_cpp_attribute) && __has_cpp_attribute(no_unique_address) && __cpluspus >= 201803L
+#   define AMREX_NO_UNIQUE_ADDRESS [[no_unique_address]]
+#   define AMREX_HAS_NO_UNIQUE_ADDRESS 1
+#else
+#   define AMREX_NO_UNIQUE_ADDRESS
+#endif
+
 #endif /* !BL_LANG_FORT */
 
 #endif
diff --git a/Src/Base/AMReX_NonLocalBC.H b/Src/Base/AMReX_NonLocalBC.H
index e1c24bb7a72..b58dcb351ed 100644
--- a/Src/Base/AMReX_NonLocalBC.H
+++ b/Src/Base/AMReX_NonLocalBC.H
@@ -1,11 +1,979 @@
 #ifndef AMREX_NONLOCAL_BC_H_
 #define AMREX_NONLOCAL_BC_H_
 #include <AMReX_Config.H>
-
+#include <AMReX_TypeTraits.H>
 #include <AMReX_FabArray.H>
 
 namespace amrex { namespace NonLocalBC {
 
+////////////////////////////////////////////////////////////////////////////////////
+//                                                            [concept.IndexMapping]
+//
+
+//! \brief Return type of an InverseImage class member function.
+template <typename T, typename... Args>
+using Inverse_t = decltype(std::declval<T>().Inverse(std::declval<Args>()...));
+
+//! \brief Type trait that tests if T has an InverseImage class member function.
+template <typename T>
+struct HasInverseMemFn : IsDetectedExact<Dim3, Inverse_t, T, Dim3> {};
+
+//! \brief Tests if a given type IndexMap is usable as an index mapping between two index based
+//! coordinate systems.
+template <class IndexMap>
+struct IsIndexMapping
+    : Conjunction<IsCallableR<Dim3, IndexMap&, const Dim3&>, 
+                  HasInverseMemFn<const IndexMap&>> {};
+
+////////////////////////////////////////////////////////////////////////////////////
+//                                             [IndexMapping.MultiBlockIndexMapping]
+//
+
+//! \brief This struct describes an affine index transformation for two coordinate systems.
+//!
+//! This DTOS is used to map indices from one block to another. It respects a permutation of
+//! (x,y,z) coordinates, an offset and a change in orientation.
+struct MultiBlockIndexMapping {
+    //! \brief This vector needs to be a valid permutation
+    IntVect permutation{AMREX_D_DECL(0, 1, 2)};
+    //! \brief The offset in the source index space.
+    IntVect offset{AMREX_D_DECL(0, 0, 0)};
+    //! \brief A vector of 1 and -1 describing the orientation in each component.
+    IntVect sign{AMREX_D_DECL(1, 1, 1)};
+
+    //! \brief Applies this mapping on the index from destination space and returns an index in the
+    //!        source space.
+    //!
+    //! \param[in] i  The index that lives in the destination space.
+    //!
+    //! \return Returns an index in the source space that is given by this affine transformation.
+    AMREX_GPU_HOST_DEVICE Dim3 operator()(Dim3 i) const noexcept {
+        int iv[3]{i.x, i.y, i.z};
+        int iv_new[3]{};
+        for (int d = 0; d < AMREX_SPACEDIM; ++d) {
+            iv_new[d] = sign[d] * (iv[permutation[d]] - offset[d]);
+        }
+        return {iv_new[0], iv_new[1], iv_new[2]};
+    }
+
+    //! \brief The inverse function is given by rearringing all above terms.
+    //!
+    //! \param[in] i  The index that lives in the source space.
+    //!
+    //! \return Returns an index in the destination space that is given by this affine
+    //! transformation.
+    AMREX_GPU_HOST_DEVICE Dim3 Inverse(Dim3 i) const noexcept {
+        int iv_new[3]{i.x, i.y, i.z};
+        int iv[3]{};
+        for (int d = 0; d < AMREX_SPACEDIM; ++d) {
+            AMREX_ASSERT(sign[d] == 1 || sign[d] == -1);
+            iv[permutation[d]] = iv_new[d] * sign[d] + offset[d];
+        }
+        return {iv[0], iv[1], iv[2]};
+    }
+
+    IndexType operator()(IndexType it) const noexcept {
+        return IndexType{IntVect{AMREX_D_DECL(it[permutation[0]], it[permutation[1]], it[permutation[2]])}};
+    }
+
+    IndexType Inverse(IndexType it) const noexcept {
+        IntVect inverse_permutation;
+        for (int i = 0; i < AMREX_SPACEDIM; ++i) {
+            inverse_permutation[permutation[i]] = i;
+        }
+        return IndexType{IntVect{AMREX_D_DECL(it[inverse_permutation[0]], it[inverse_permutation[1]], it[inverse_permutation[2]])}};
+    }
+};
+
+//! \brief Applies the Dim3 to Dim3 mapping onto IntVects.
+//!
+//! This is used to map indices from the dest index space into the source index space.
+//!
+//! IntVect is being embedded in Dim3 by trailing zeros.
+//! Dim3 is being projected to IntVect by dopping z for AMREX_SPACEDIM = 2 or z and y components
+//! for AMREX_SPACEDIM = 1.
+//!
+//! \param[in] iv The IntVect that lives in the destination index space.
+//!
+//! \return Returns IntVect{dtos(Dim3{iv})}
+template <typename DTOS>
+EnableIf_t<IsCallableR<Dim3, DTOS, Dim3>::value, IntVect> 
+Apply(DTOS dtos, const IntVect& iv) 
+{
+    Dim3 i = dtos(iv.dim3());
+    return IntVect{AMREX_D_DECL(i.x, i.y, i.z)};
+}
+
+//! \brief Applies the Dim3 to Dim3 mapping onto Boxes but does not change the index type.
+//!
+//! This function assumes monotonicity of dtos in each component.
+//!
+//! \param[in] box The box that lives the destination index space
+//!
+//! \return Returns the smallest Box in the source index space that contains images of
+//! Apply(dtos, box.smallEnd()) and Apply(dtos, box.bigEnd()).
+template <typename DTOS>
+EnableIf_t<IsCallableR<Dim3, DTOS, Dim3>::value && !IsCallableR<IndexType, DTOS, IndexType>::value, Box> 
+Image(DTOS dtos, const Box& box) 
+{
+    IntVect mapped_smallEnd = Apply(dtos, box.smallEnd());
+    IntVect mapped_bigEnd = Apply(dtos, box.bigEnd());
+    IntVect smallEnd;
+    IntVect bigEnd;
+    for (int d = 0; d < AMREX_SPACEDIM; ++d) {
+        smallEnd[d] = std::min(mapped_smallEnd[d], mapped_bigEnd[d]);
+        bigEnd[d] = std::max(mapped_smallEnd[d], mapped_bigEnd[d]);
+    }
+    return Box{smallEnd, bigEnd, box.ixType()};
+}
+
+//! \brief Applies the Dim3 to Dim3 mapping onto Boxes and maps the index type.
+//!
+//! This function assumes monotonicity of dtos in each component.
+//!
+//! \param[in] box The box that lives the destination index space
+//!
+//! \return Returns the smallest Box in the source index space that contains images of
+//! Apply(dtos, box.smallEnd()) and Apply(dtos, box.bigEnd()).
+template <typename DTOS>
+EnableIf_t<IsCallableR<Dim3, DTOS, Dim3>::value && IsCallableR<IndexType, DTOS, IndexType>::value, Box> 
+Image(DTOS dtos, const Box& box) 
+{
+    // "Forget" the index type mapping and invoke Image without changing the index type.
+    Box srcbox = Image([&dtos](Dim3 i) { return dtos(i); }, box);
+    // Fix the index type of the resulting box
+    srcbox.setType(dtos(box.ixType()));
+    return srcbox;
+}
+
+//! \brief Applies the Dim3 to Dim3 invserse mapping onto IntVects.
+//!
+//! This is used to map indices from the src index space into the dest index space.
+//!
+//! IntVect is being embedded in Dim3 by trailing zeros.
+//! Dim3 is being projected to IntVect by dopping z for AMREX_SPACEDIM = 2 or z and y components
+//! for AMREX_SPACEDIM = 1.
+//!
+//! \param[in] iv The IntVect that lives in the src index space.
+//!
+//! \return Returns IntVect{dtos.Inverse(Dim3{iv})}
+template <typename DTOS>
+EnableIf_t<HasInverseMemFn<DTOS>::value, IntVect> 
+ApplyInverse(DTOS dtos, const IntVect& iv) 
+{
+    return Apply([&dtos](Dim3 i) { return dtos.Inverse(i); }, iv);
+}
+
+//! \brief Applies the inverse Dim3 to Dim3 mapping onto Boxes without changing the index type.
+//!
+//! This function assumes monotonicity in each component.
+//!
+//! \param[in] box The box that lives the source index space
+//!
+//! \return Returns the smallest Box in the destination index space that contains images of
+//! ApplyInverse(box.smallEnd()) and ApplyInverse(box.bigEnd()).
+template <typename DTOS>
+EnableIf_t<HasInverseMemFn<DTOS>::value && !IsCallableR<IndexType, DTOS, IndexType>::value, Box> 
+InverseImage(DTOS dtos, const Box& box) 
+{
+    return Image([&dtos](Dim3 i) { return dtos.Inverse(i); }, box);
+}
+
+//! \brief Applies the inverse Dim3 to Dim3 mapping onto Boxes
+//!
+//! This function assumes monotonicity in each component.
+//!
+//! \param[in] box The box that lives the source index space
+//!
+//! \return Returns the smallest Box in the destination index space that contains images of
+//! ApplyInverse(box.smallEnd()) and ApplyInverse(box.bigEnd()).
+template <typename DTOS>
+EnableIf_t<HasInverseMemFn<DTOS>::value && IsCallableR<IndexType, DTOS, IndexType>::value, Box> 
+InverseImage(DTOS dtos, const Box& box) 
+{
+    return Image([&dtos](auto&& i) { return dtos.Inverse(i); }, box);
+}
+
+//! \brief This is the index mapping based on the DTOS MultiBlockDestToSrc.
+static_assert(IsIndexMapping<MultiBlockIndexMapping>(),
+              "MultiBlockIndexMapping is expected to satisfy IndexMapping");
+
+////////////////////////////////////////////////////////////////////////////////////
+//                                                    [class.MultiBlockCommMetaData]
+//
+
+//! \brief This class stores data dependencies for an inter-block communication.
+//!
+//! In communication between two blocks one might need to do an index
+//! transformation from one block to antoher.
+struct MultiBlockCommMetaData : FabArrayBase::CommMetaData {
+    //! \name Constructors
+
+    //! \brief Build global meta data by calling the define() member function.
+    //!
+    //! \see MultiBlockCommMetaData::define
+    template <typename DTOS, 
+              typename = EnableIf_t<IsIndexMapping<DTOS>::value>>
+    MultiBlockCommMetaData(const FabArrayBase& dst, const Box& dstbox, const FabArrayBase& src,
+                           const IntVect& ngrow, DTOS dtos);
+
+    //! \brief Build global meta data by calling the define() member function.
+    //!
+    //! \see MultiBlockCommMetaData::define
+    template <typename DTOS, 
+              typename = EnableIf_t<IsIndexMapping<DTOS>::value>>
+    MultiBlockCommMetaData(const BoxArray& dstba, const DistributionMapping& dstdm,
+                           const Box& dstbox, const BoxArray& srcba,
+                           const DistributionMapping& srcdm, const IntVect& ngrow, DTOS dtos);
+
+    //! \name Manipulators
+
+    //! \brief Build global meta data that is being used to identify send and recv
+    //! dependencies in communication routines.
+    //!
+    //! This call is quadratic in the number of boxes, i.e. it is in
+    //! O(dstba.size() * srcba.size()). Therefore, it might be wise to cache the
+    //! construction of this object to minimize its computation.
+    //!
+    //! \tparam DTOS This parameter needs to satisfy IsIndexMapping and HasInverseImage.
+    //!
+    //! \param[in] dstba  The destination box array.
+    //! 
+    //! \param[in] dstdm  The destination distribution mapping.
+    //!
+    //! \param[in] dstbox The box that will be filled and that lives in the destination space.
+    //!
+    //! \param[in] srcba  The source box array.
+    //!
+    //! \param[in] srcdm  The source distribution mapping.
+    //!
+    //! \param[in] ngrow  The number of ghost cells that shall be considered when filling data.
+    //!
+    //! \param[in] dtos   The dest to source index mapping that has an inverse image.
+    //!
+    //! \return Nothing.
+    template <typename DTOS>
+    EnableIf_t<IsIndexMapping<DTOS>::value>
+    define(const BoxArray& dstba, const DistributionMapping& dstdm, const Box& dstbox,
+           const BoxArray& srcba, const DistributionMapping& srcdm, const IntVect& ngrow,
+           DTOS dtos);
+};
+
+////////////////////////////////////////////////////////////////////////////////////
+//                                                           [concept.FabProjection]
+//
+
+//! \brief This type trait tests if a type P is a projetion for FAB.
+template <typename P, typename FAB>
+struct IsFabProjection
+    : IsCallableR<typename FAB::value_type, P, Array4<const typename FAB::value_type>, Dim3, int> 
+{};
+
+////////////////////////////////////////////////////////////////////////////////////
+//                                                          [FabProjection.Identity]
+//                                                           [IndexMapping.Identity]
+
+//! \brief This class acts as a default no-op operator.
+//!
+//! This class satisfies IndexMapping and FabProjection.
+struct Identity  {
+    //! The identity function for Dim3
+    constexpr Dim3 operator()(Dim3 i) const noexcept { return i; }
+    //! The identity function for Dim3
+    constexpr Dim3 Inverse(Dim3 i) const noexcept { return i; }
+
+    //! \return Returns array(i.x,i.y,i.z,comp)
+    template <typename T>
+    constexpr T operator()(Array4<const T> array, Dim3 i, int comp = 0) const
+        noexcept(noexcept(array(i.x, i.y, i.z, comp))) {
+        return array(i.x, i.y, i.z, comp);
+    }
+
+    //! The identity for int
+    constexpr int operator()(int i) const noexcept { return i; }
+};
+static constexpr Identity identity{};
+
+static_assert(sizeof(Identity) == 1, "");
+static_assert(std::is_trivially_default_constructible<Identity>::value, "");
+static_assert(std::is_trivially_copy_assignable<Identity>::value, "");
+static_assert(std::is_trivially_copy_constructible<Identity>::value, "");
+static_assert(IsIndexMapping<Identity>(), "");
+static_assert(IsFabProjection<Identity, FArrayBox>(), "");
+
+////////////////////////////////////////////////////////////////////////////////////
+//                                                     [FabProjection.MapComponents]
+
+//! \brief This class takes a projection and a component map and combines them to form a new
+//! projection.
+//!
+//! We use this to apply a permutations on components FABs.
+//!
+//! \see SwapComponents
+template <typename Base, typename Map = Identity> struct MapComponents {
+    static_assert(IsCallable<Base, Array4<const Real>, Dim3, int>::value,
+                  "Base needs to be a callable function: (Array4<const T>, Dim3, i) -> auto.");
+
+    static_assert(IsCallableR<int, Map, int>::value,
+                  "Map needs to be a callable function: int -> int.");
+
+    AMREX_NO_UNIQUE_ADDRESS Base base;
+    AMREX_NO_UNIQUE_ADDRESS Map map;
+
+    template <typename T,
+              typename = EnableIf_t<IsCallable<Base, Array4<const T>, Dim3, int>::value>,
+              typename = EnableIf_t<IsCallableR<int, Map, int>::value>>
+    constexpr decltype(auto) operator()(Array4<const T> array, Dim3 i, int comp) const
+        noexcept(noexcept(base(array, i, map(comp)))) {
+        return base(array, i, map(comp));
+    }
+};
+
+#ifdef AMREX_HAS_NO_UNIQUE_ADDRESS
+static_assert(sizeof(MapComponents<Identity, Identity>) == 1, "");
+#endif
+static_assert(std::is_trivially_copy_assignable<MapComponents<Identity>>(), "");
+static_assert(std::is_trivially_copy_constructible<MapComponents<Identity>>(), "");
+static_assert(IsFabProjection<MapComponents<Identity>, FArrayBox>(), "");
+
+////////////////////////////////////////////////////////////////////////////////////
+//                                      [FabProjection.MapComponents.SwapComponents]
+
+//! \brief This is a permutation where only two components are swapped.
+//!
+//! This class is used with MapComponents to define fab projections that swap components on FABs,
+//! e.g. for velocity components. The template arguments I, J should be greater or equal to -1. The
+//! value -1 indicates run-time values and increase the object size of this function object.
+template <int I, int J> struct SwapComponents { 
+    static_assert(I >= 0 && J >= 0, "I >= 0 && J >= 0");
+
+    //! \brief Swaps indices I and J.
+    //!
+    //! \return If i == I then return J. If i == J then return I. Otherwise returns i.
+    constexpr int operator()(int i) const noexcept {
+        const int map[2] = {I, J};
+        return i == I || i == J ? map[std::size_t(i == I)] : i;
+    }
+};
+
+template <int I> struct SwapComponents<I, -1> {
+    static_assert(I >= 0, "I >= 0");
+
+    int J;
+    constexpr int operator()(int i) const noexcept {
+        const int map[2] = {I, J};
+        return i == I || i == J ? map[std::size_t(i == I)] : i;
+    }
+};
+
+template <int J> struct SwapComponents<-1, J> {
+    static_assert(J >= 0, "J >= 0");
+    
+    int I;
+    constexpr int operator()(int i) const noexcept {
+        const int map[2] = {I, J};
+        return i == I || i == J ? map[std::size_t(i == I)] : i;
+    }
+};
+
+template <> struct SwapComponents<-1, -1> {
+    int I;
+    int J;
+    constexpr int operator()(int i) const noexcept {
+        const int map[2] = {I, J};
+        return i == I || i == J ? map[std::size_t(i == I)] : i;
+    }
+};
+
+using DynamicSwapComponents = SwapComponents<-1, -1>;
+
+template <int I, int J> static constexpr SwapComponents<I, J> swap_indices{};
+
+static_assert(sizeof(SwapComponents<0, 1>) == 1, "");
+static_assert(sizeof(DynamicSwapComponents) == 2 * sizeof(int), "");
+static_assert(sizeof(SwapComponents<0, -1>) == sizeof(int), "");
+static_assert(sizeof(SwapComponents<-1, 1>) == sizeof(int), "");
+#ifdef AMREX_HAS_NO_UNIQUE_ADDRESS
+static_assert(sizeof(MapComponents<Identity, SwapComponents<0, 1>>) == sizeof(SwapComponents<0, 1>), "");
+#endif
+static_assert(std::is_trivially_default_constructible<MapComponents<Identity, SwapComponents<0, 1>>>(), "");
+static_assert(std::is_trivially_copy_assignable<MapComponents<Identity, SwapComponents<0, 1>>>(), "");
+static_assert(std::is_trivially_copy_constructible<MapComponents<Identity, SwapComponents<0, 1>>>(), "");
+static_assert(IsFabProjection<MapComponents<Identity, SwapComponents<0, 1>>, FArrayBox>(), "");
+
+static_assert(swap_indices<0, 1>(0) == 1, "");
+static_assert(swap_indices<0, 1>(1) == 0, "");
+static_assert(swap_indices<0, 1>(2) == 2, "");
+static_assert(DynamicSwapComponents{0, 1}(0) == 1, "");
+static_assert(DynamicSwapComponents{0, 1}(1) == 0, "");
+static_assert(DynamicSwapComponents{0, 1}(2) == 2, "");
+
+////////////////////////////////////////////////////////////////////////////////////
+//                                                                  [class.CommData]
+//                                                               [class.CommHandler]
+
+//! \brief This class holds data buffers for either immediate MPI send or recv calls.
+//!
+//! The data buffers need to be kept alive until all MPI_Request have been completed.
+//! All member variables are resized a corresponding call to PrepareCommBuffers and 
+//! the data transactions are initiated either by calls to PostRecvs or PostSends.
+struct CommData {
+#ifdef AMREX_USE_MPI
+    //! \brief Holds 'em all in one data pointer.
+    TheFaArenaPointer the_data = nullptr;
+    //! \brief Stores MPI ranks. For recvs it is the 'from' rank and for sends it is the 'to' rank.
+    Vector<int> rank{};
+    //! \brief Pointers to the_data that can be used for each single data transaction.
+    Vector<char*> data{};
+    //! \brief All offsets of data in the_data.
+    Vector<std::size_t> offset{};
+    //! \brief The size in bytes for each data transaction.
+    Vector<std::size_t> size{};
+    //! \brief The associated MPI_Request for each data transaction.
+    Vector<MPI_Request> request{};
+    //! \brief For each request the corresponding MPI_status, used for debugging.
+    Vector<MPI_Status> stats{};
+    //! \brief For each requst the copy comm tags for the corresponding data FABs. 
+    Vector<const FabArrayBase::CopyComTagsContainer*> cctc{};
+#endif
+};
+
+#ifdef AMREX_USE_MPI
+//! \brief Fill all class member variables of comm but the request and the stats vector.
+void PrepareCommBuffers(CommData& comm, const FabArrayBase::MapOfCopyComTagContainers& cctc,
+                        int n_components, std::size_t object_size, std::size_t align);
+
+//! \brief Initiate all recvieves with MPI_Irecv calls associated with tag mpi_tag. 
+void PostRecvs(CommData& comm, int mpi_tag);
+
+//! \brief Initiate all sends with MPI_Isend calls associated with tag mpi_tag. 
+void PostSends(CommData& comm, int mpi_tag);
+#endif
+
+
+//! \brief This class stores both recv and send buffers with an associated MPI tag.
+//!
+//! This handler stores data for a whole ParallelCopy MPI transaction.
+struct CommHandler {
+#ifdef AMREX_USE_MPI
+    int mpi_tag{};
+    CommData recv{};
+    CommData send{};
+#endif
+};
+
+////////////////////////////////////////////////////////////////////////////////////
+//                                                             [concept.DataPacking]
+//
+template <typename... Args>
+using PrepareSendBuffers_t = decltype(PrepareSendBuffers(std::declval<Args>()...));
+
+template <typename... Args>
+using PrepareRecvBuffers_t = decltype(PrepareRecvBuffers(std::declval<Args>()...));
+
+template <typename... Args>
+using PackSendBuffers_t = decltype(PackSendBuffers(std::declval<Args>()...));
+
+template <typename... Args>
+using UnpackRecvBuffers_t = decltype(UnpackRecvBuffers(std::declval<Args>()...));
+
+template <typename... Args>
+using LocalCopy_t = decltype(LocalCopy(std::declval<Args>()...));
+
+//! \brief This type trait tests if a given type DP satisfies the DataPacking concept for type FAB.
+template <typename DP, typename FAB>
+struct IsDataPacking : 
+    Conjunction<
+        IsDetected<LocalCopy_t, DP&, FabArray<FAB>&, const FabArray<FAB>&, const FabArrayBase::CopyComTagsContainer&>
+#ifdef AMREX_USE_MPI
+       ,IsDetected<PrepareSendBuffers_t, DP&, FabArray<FAB>&, const FabArray<FAB>&, CommData&, const FabArrayBase::MapOfCopyComTagContainers&>,
+        IsDetected<PrepareRecvBuffers_t, DP&, FabArray<FAB>&, const FabArray<FAB>&, CommData&, const FabArrayBase::MapOfCopyComTagContainers&>,
+        IsDetected<PackSendBuffers_t, DP&, const FabArray<FAB>&, CommData&>,
+        IsDetected<UnpackRecvBuffers_t, DP&, FabArray<FAB>&, CommData&>
+#endif
+    > {};
+
+////////////////////////////////////////////////////////////////////////////////////
+//                                                      [DataPacking.PackComponents]
+//
+// PackComponents is the simplest data packing policy.
+// This provides us with sane default behaviour that we can use when defining new 
+// data packing policies.
+
+//! \brief Contains information about which components take part of the data transaction.
+struct PackComponents {
+    int dest_component{0};
+    int src_component{0};
+    int n_components{0};
+};
+
+//! \brief Dispatch local copies to the default behaviour that knows no DTOS nor projection.
+template <typename FAB>
+amrex::EnableIf_t<IsBaseFab<FAB>::value>
+LocalCopy (const PackComponents& components, FabArray<FAB>& dest, const FabArray<FAB>& src,
+           const FabArrayBase::CopyComTagsContainer& local_tags) {
+#ifdef AMREX_USE_GPU
+    if (Gpu::inLaunchRegion()) {
+        local_copy_gpu(dest, src, components.dest_component, components.src_component,
+                       components.n_components, local_tags);
+    } else
+#endif
+    {
+        local_copy_cpu(dest, src, components.dest_component, components.src_component,
+                       components.n_components, local_tags);
+    }
+}
+
+#ifdef AMREX_USE_MPI
+//! \brief Calls PrepareComBuffers.
+template <typename FAB>
+amrex::EnableIf_t<IsBaseFab<FAB>::value>
+PrepareSendBuffers (const PackComponents& components, FabArray<FAB>& dest, const FabArray<FAB>& src,
+                    CommData& comm, const FabArrayBase::MapOfCopyComTagContainers& cctc) {
+    using T = typename FAB::value_type;
+    ignore_unused(dest, src);
+    PrepareCommBuffers(comm, cctc, components.n_components, sizeof(T), alignof(T));
+}
+
+//! \brief Calls PrepareComBuffers.
+template <typename FAB>
+amrex::EnableIf_t<IsBaseFab<FAB>::value>
+PrepareRecvBuffers (const PackComponents& components, FabArray<FAB>& dest, const FabArray<FAB>& src, 
+                    CommData& comm, const FabArrayBase::MapOfCopyComTagContainers& cctc) {
+    using T = typename FAB::value_type;
+    ignore_unused(dest, src);
+    PrepareCommBuffers(comm, cctc, components.n_components, sizeof(T), alignof(T));
+}
+
+//! \brief Serializes FAB data without any knowledge of a DTOS nor a projection.
+template <typename FAB>
+amrex::EnableIf_t<IsBaseFab<FAB>::value>
+PackSendBuffers (const PackComponents& components, const FabArray<FAB>& src, CommData& send) {
+#ifdef AMREX_USE_GPU
+    if (Gpu::inLaunchRegion()) {
+        FabArray<FAB>::pack_send_buffer_gpu(src, components.src_component, components.n_components,
+                                            send.data, send.size, send.cctc);
+    } else
+#endif // AMREX_USE_GPU
+    {
+        FabArray<FAB>::pack_send_buffer_cpu(src, components.src_component, components.n_components,
+                                            send.data, send.size, send.cctc);
+    }
+}
+
+//! \brief De-serializes FAB data without any knowledge of a DTOS nor a projection.
+template <typename FAB>
+amrex::EnableIf_t<IsBaseFab<FAB>::value>
+UnpackRecvBuffers (const PackComponents& components, FabArray<FAB>& dest, const CommData& recv) {
+#ifdef AMREX_USE_GPU
+    if (Gpu::inLaunchRegion()) {
+        unpack_recv_buffer_gpu(dest, components.dest_component, components.n_components, recv.data,
+                               recv.size, recv.cctc);
+    } else
+#endif // AMREX_USE_GPU
+    {
+        unpack_recv_buffer_cpu(dest, components.dest_component, components.n_components, recv.data,
+                               recv.size, recv.cctc);
+    }
+}
+#endif // AMREX_USE_MPI
+
+static_assert(IsDataPacking<PackComponents, FArrayBox>(),
+              "PackComponents is expected to satisfy the concept DataPacking.");
+
+////////////////////////////////////////////////////////////////////////////////////
+//                                    [DataPacking.ApplyDtosAndProjectionOnReciever]
+//
+//! \brief This class specializes behaviour on local copies and unpacking recieve buffers.
+//!
+//! It takes a DTOS and FabProjection to apply them on the reciever side. 
+template <typename DTOS = Identity, typename FabProj = Identity>
+struct ApplyDtosAndProjectionOnReciever : PackComponents {
+    constexpr ApplyDtosAndProjectionOnReciever() = default;
+    constexpr ApplyDtosAndProjectionOnReciever(const PackComponents& components, DTOS dtos_ = DTOS{}, FabProj proj_ = FabProj{})
+        : PackComponents(components), dtos(std::move(dtos_)), proj(std::move(proj_)) {}
+
+    AMREX_NO_UNIQUE_ADDRESS DTOS dtos;
+    AMREX_NO_UNIQUE_ADDRESS FabProj proj;
+
+    static_assert(IsCallableR<Dim3, DTOS, Dim3>(), "DTOS needs to be a callable: Dim3 -> Dim3");
+    static_assert(IsFabProjection<FabProj, FArrayBox>(), "FabProj needs to be at least a projection on FArrayBox.");
+};
+
+//! \brief Do local copies of FABs using DTOS and projection.
+template <typename FAB, typename DTOS, typename FabProj>
+amrex::EnableIf_t<IsBaseFab<FAB>::value>
+LocalCopy (const ApplyDtosAndProjectionOnReciever<DTOS, FabProj>& packing, FabArray<FAB>& dest,
+           const FabArray<FAB>& src, const FabArrayBase::CopyComTagsContainer& local_tags) {
+    static_assert(IsFabProjection<FabProj, FAB>(), "FabProj needs to be a projection for given FAB type.");
+#ifdef AMREX_USE_GPU
+    if (Gpu::inLaunchRegion()) {
+        local_copy_gpu(dest, src, packing.dest_component, packing.src_component,
+                       packing.n_components, local_tags, packing.dtos, packing.proj);
+    } else
+#endif
+    {
+        local_copy_cpu(dest, src, packing.dest_component, packing.src_component,
+                       packing.n_components, local_tags, packing.dtos, packing.proj);
+    }
+}
+
+#ifdef AMREX_USE_MPI
+//! \brief Copy from recieved data in the buffer to destination FABs using DTOS and projection.
+template <typename FAB, typename DTOS, typename FabProj>
+amrex::EnableIf_t<IsBaseFab<FAB>::value>
+UnpackRecvBuffers (const ApplyDtosAndProjectionOnReciever<DTOS, FabProj>& packing,
+                   FabArray<FAB>& dest, const CommData& recv) {
+    // If FAB is not FArrayBox we have not checked for the correct types yet.
+    static_assert(IsFabProjection<FabProj, FAB>(), "FabProj needs to be a projection for given FAB type.");
+#ifdef AMREX_USE_GPU
+    if (Gpu::inLaunchRegion()) {
+        unpack_recv_buffer_gpu(dest, packing.dest_component, packing.n_components, recv.data,
+                               recv.size, recv.cctc, packing.dtos, packing.proj);
+    } else
+#endif // AMREX_USE_GPU
+    {
+        unpack_recv_buffer_cpu(dest, packing.dest_component, packing.n_components, recv.data,
+                               recv.size, recv.cctc, packing.dtos, packing.proj);
+    }
+}
+#endif // AMREX_USE_MPI
+
+static_assert(IsDataPacking<ApplyDtosAndProjectionOnReciever<>, FArrayBox>(),
+              "ApplyDtosAndProjectionOnReciever<> is expected to satisfy the DataPacking concept.");
+
+////////////////////////////////////////////////////////////////////////////////////
+//                                                             [ParallelCopy_nowait]
+
+static constexpr struct NoLocalCopy {} no_local_copy{};
+static constexpr struct DoLocalCopy {} do_local_copy{};
+
+//! Initiate recv and send calls for MPI and immediately return without doing any work.
+//!
+//! DataPacking is a customization point object to control the behaviour of packing and unpacking
+//! send or recv data buffers. It is used to perform interpolation or data transformations on either
+//! sender or reciever side.
+//!
+//! This function performs a data packing on sender side and we expect a call to Parallel_finish
+//! that performs data unpacking on the reciever side.
+//!
+//! \param[out] dest     The Multifab that is going to be filled with recieved data.
+//!
+//! \param[in]  src      The Multifab that is used to fill the send buffers.
+//!
+//! \param[in]  cmd      The communication meta data object holds spatial information about FAB
+//!                      boxes that need to be filled and copied from.
+//!
+//! \param[in]  data_packing A CPO that controls behaviour of preparing buffers and packing the
+//!                          source data into the send buffers.
+//!
+//! \return Returns a CommHandler object that owns context and memory buffers for the whole life
+//!         time of the MPI transaction.
+template <typename FAB, typename DataPacking, 
+          typename = EnableIf_t<IsBaseFab<FAB>::value>,
+          typename = EnableIf_t<IsDataPacking<DataPacking, FAB>::value>>
+#ifdef AMREX_USE_MPI
+AMREX_NODISCARD CommHandler 
+ParallelCopy_nowait (NoLocalCopy, FabArray<FAB>& dest, const FabArray<FAB>& src,
+                     const FabArrayBase::CommMetaData& cmd, const DataPacking& data_packing) {
+    CommHandler handler{};
+    if (ParallelContext::NProcsSub() == 1) {
+        return handler;
+    }
+    //
+    // Do this before prematurely exiting if running in parallel.
+    // Otherwise sequence numbers will not match across MPI processes.
+    //
+    handler.mpi_tag = ParallelDescriptor::SeqNum();
+
+    const int N_rcvs = cmd.m_RcvTags ? cmd.m_RcvTags->size() : 0;
+    if (N_rcvs > 0) {
+        PrepareRecvBuffers(data_packing, dest, src, handler.recv, *cmd.m_RcvTags);
+        PostRecvs(handler.recv, handler.mpi_tag);
+    }
+
+    const int N_snds = cmd.m_SndTags ? cmd.m_SndTags->size() : 0;
+    if (N_snds > 0) {
+        PrepareSendBuffers(data_packing, dest, src, handler.send, *cmd.m_SndTags);
+        PackSendBuffers(data_packing, src, handler.send);
+        PostSends(handler.send, handler.mpi_tag);
+    }
+    return handler;
+}
+#else
+CommHandler ParallelCopy_nowait (NoLocalCopy, FabArray<FAB>&, const FabArray<FAB>&,
+                                 const FabArrayBase::CommMetaData&, const DataPacking&) {
+    return CommHandler{};
+}
+#endif
+
+//! Initiate recv and send calls for MPI and return after doing local work.
+//!
+//! DataPacking is a customization point object to control the behaviour of packing and unpacking
+//! send or recv data buffers. It is used to perform interpolation or data transformations on either
+//! sender or reciever side.
+//!
+//! This function performs a data packing on sender side and we expect a call to Parallel_finish
+//! that performs data unpacking on the reciever side.
+//!
+//! \param[out] dest     The Multifab that is going to be filled with recieved data.
+//!
+//! \param[in]  src      The Multifab that is used to fill the send buffers.
+//!
+//! \param[in]  cmd      The communication meta data object holds spatial information about FAB
+//!                      boxes that need to be filled and copied from.
+//!
+//! \param[in]  data_packing A CPO that controls behaviour of preparing buffers and packing the
+//!                          source data into the send buffers.
+//!
+//! \return Returns a CommHandler object that owns context and memory buffers for the whole life
+//!         time of the MPI transaction.
+template <typename FAB, typename DataPacking, 
+          typename = EnableIf_t<IsBaseFab<FAB>::value>,
+          typename = EnableIf_t<IsDataPacking<DataPacking, FAB>::value>>
+#ifdef AMREX_USE_MPI
+AMREX_NODISCARD 
+#endif
+CommHandler 
+ParallelCopy_nowait (FabArray<FAB>& dest, const FabArray<FAB>& src,
+                     const FabArrayBase::CommMetaData& cmd, const DataPacking& data_packing) {
+    CommHandler comm = ParallelCopy_nowait(no_local_copy, dest, src, cmd, data_packing);
+    // Eagerly do the local work and hope for some overlap with communication
+    if (cmd.m_LocTags && cmd.m_LocTags->size() > 0) {
+        LocalCopy(data_packing, dest, src, *cmd.m_LocTags);
+    }
+    return comm;
+}
+////////////////////////////////////////////////////////////////////////////////////
+//                                                             [ParallelCopy_finish]
+
+//! Blockingly wait for all communication to be done and fill the local FABs with recieved data.
+//!
+//! This function overload performs no local copies, i.e. from this MPI process to itself.
+//! It will block the current thread until all MPI recv and send requests are done and
+//! calls the DataPacking object to unpack the recieved buffers.
+//!
+//! \param[out] dest     The Multifab that is going to be filled with recieved data.
+//!
+//! \param[in]  handler  This object holds all data buffers that need to be kept alive as long as
+//!                      the data transaction is not done.
+//!
+//! \param[in]  cmd      The communication meta data object holds spatial information about FAB
+//!                      boxes that need to be filled.
+//!
+//! \param[in]  data_packing A CPO that controls behaviour of unpacking the recieved buffer to the
+//!                          destination FabArray.
+//!
+//! \returns Nothing.
+template <typename FAB, typename DataPacking>
+EnableIf_t<IsBaseFab<FAB>() && IsDataPacking<DataPacking, FAB>()>
+#ifdef AMREX_USE_MPI
+ParallelCopy_finish (FabArray<FAB>& dest, CommHandler handler,
+                     const FabArrayBase::CommMetaData& cmd, const DataPacking& data_packing) {
+    // If any FabArray is empty we have nothing to do.
+    if (dest.size() == 0) {
+        return;
+    }
+    // Return if nothing do
+    if (ParallelContext::NProcsSub() == 1) {
+        return;
+    }
+    // Unpack recieves
+    const int N_rcvs = cmd.m_RcvTags ? cmd.m_RcvTags->size() : 0;
+    if (N_rcvs > 0) {
+        ParallelDescriptor::Waitall(handler.recv.request, handler.recv.stats);
+#ifdef AMREX_D_DEBUG
+        if (!CheckRcvStats(stats, handler.recv_size, handler.mpi_tag)) {
+            amrex::Abort("NonLocalPC::ParallelCopy_finish failed with wrong message size");
+        }
+#endif
+        UnpackRecvBuffers(data_packing, dest, handler.recv);
+    }
+
+    // Wait for all sends to be done
+    const int N_snds = cmd.m_SndTags ? cmd.m_SndTags->size() : 0;
+    if (N_snds > 0) {
+        ParallelDescriptor::Waitall(handler.send.request, handler.send.stats);
+    }
+}
+#else
+ParallelCopy_finish (FabArray<FAB>&, CommHandler, const FabArrayBase::CommMetaData&, const DataPacking&) {}
+#endif
+
+//! Blockingly wait for all communication to be done and fill the local FABs with recieved data.
+//!
+//! This function overload performs local copies, i.e. from this MPI process to itself.
+//! It will block the current thread until all MPI recv and send requests are done and
+//! calls the DataPacking object to unpack the recieved buffers.
+//!
+//! \param[out] dest     The Multifab that is going to be filled with recieved data.
+//!
+//! \param[in]  src      The Multifab that is used to get the data for the local copies from.
+//!
+//! \param[in]  handler  This object holds all data buffers that need to be kept alive as long as
+//!                      the data transaction is not done.
+//!
+//! \param[in]  cmd      The communication meta data object holds spatial information about FAB
+//!                      boxes that need to be filled.
+//!
+//! \param[in]  data_packing A CPO that controls behaviour of unpacking the recieved data.
+//!
+//! \returns Nothing.
+template <typename FAB, typename DataPacking>
+EnableIf_t<IsBaseFab<FAB>() && IsDataPacking<DataPacking, FAB>()>
+ParallelCopy_finish (DoLocalCopy, FabArray<FAB>& dest, const FabArray<FAB>& src, CommHandler handler,
+                     const FabArrayBase::CommMetaData& cmd, const DataPacking& data_packing) {
+    // Eagerly do the local work and hope for some overlap with communication
+    if (cmd.m_LocTags && cmd.m_LocTags->size() > 0) {
+        LocalCopy(data_packing, dest, src, *cmd.m_LocTags);
+    }
+    ParallelCopy_finish(dest, std::move(handler), cmd, data_packing);
+}
+
+//! \brief Call ParallelCopy_nowait followed by ParallelCopy_finish, strong typed version.
+//!
+//! This function overload uses an already cached CommMetaData. This CommMetaData needs to be
+//! compatible with the specified DTOS and projection, otherwise undefined behaviour occurs.
+//!
+//! \param[out] dest The Multifab that is going to be filled with recieved data.
+//!
+//! \param[in]  src  The Multifab that is used to fill the send buffers.
+//!
+//! \param[in]  cmd  The communication meta data object holds spatial information about FAB
+//!                  boxes that need to be filled and copied from.
+//!
+//! \param[in]  srccomp  The first component in src that will be copied to dest.
+//!
+//! \param[in]  destcomp The first component in dest that will get written by src.
+//!
+//! \param[in]  ncomp    The number of successive components that will be copied.
+//!
+//! \param[in]  dtos An index mapping that maps indices from destination space to source space
+//!                  and from source space to destination space.
+//!
+//! \param[in]  proj A transformation function that might change the data when it is being copied.
+//!
+//! \return Nothing.
+template <typename FAB, typename DTOS = Identity, typename Proj = Identity>
+EnableIf_t<IsBaseFab<FAB>() && IsCallableR<Dim3, DTOS, Dim3>() && IsFabProjection<Proj, FAB>()>
+ParallelCopy (FabArray<FAB>& dest, const FabArray<FAB>& src, const FabArrayBase::CommMetaData& cmd,
+              SrcComp srccomp, DestComp destcomp, NumComps numcomp, DTOS dtos = DTOS{}, Proj proj = Proj{}) {
+    PackComponents components{};
+    components.dest_component = destcomp.i;
+    components.src_component = srccomp.i;
+    components.n_components = numcomp.n;
+    ApplyDtosAndProjectionOnReciever<DTOS, Proj> packing{components, std::move(dtos), std::move(proj)};
+    CommHandler handler = ParallelCopy_nowait(dest, src, cmd, packing);
+    ParallelCopy_finish(dest, std::move(handler), cmd, packing);
+}
+
+//! \brief Call ParallelCopy_nowait followed by ParallelCopy_finish.
+//!
+//! This function overload uses an already cached CommMetaData. This CommMetaData needs to be
+//! compatible with the specified DTOS and projection, otherwise undefined behaviour occurs.
+//!
+//! \param[out] dest The Multifab that is going to be filled with recieved data.
+//!
+//! \param[in]  src  The Multifab that is used to fill the send buffers.
+//!
+//! \param[in]  cmd  The communication meta data object holds spatial information about FAB
+//!                  boxes that need to be filled and copied from.
+//!
+//! \param[in]  srccomp  The first component in src that will be copied to dest.
+//!
+//! \param[in]  destcomp The first component in dest that will get written by src.
+//!
+//! \param[in]  ncomp    The number of successive components that will be copied.
+//!
+//! \param[in]  dtos An index mapping that maps indices from destination space to source space
+//!                  and from source space to destination space.
+//!
+//! \param[in]  proj A transformation function that might change the data when it is being copied.
+//!
+//! \return Nothing.
+template <typename FAB, typename DTOS = Identity, typename Proj = Identity>
+EnableIf_t<IsBaseFab<FAB>() && IsCallableR<Dim3, DTOS, Dim3>() && IsFabProjection<Proj, FAB>()>
+ParallelCopy (FabArray<FAB>& dest, const FabArray<FAB>& src, const FabArrayBase::CommMetaData& cmd,
+              int srccomp, int destcomp, int numcomp, DTOS dtos = DTOS{}, Proj proj = Proj{}) {
+    ParallelCopy(dest, src, cmd, SrcComp(srccomp), DestComp(destcomp), NumComps(numcomp), std::move(dtos), std::move(proj));
+}
+
+//! \brief Call ParallelCopy_nowait followed by ParallelCopy_finish, strong typed version.
+//!
+//! This function constructs a new MultiCommMetaData from the given DTOS, destbox and ngrow. 
+//!
+//! \param[out] dest The Multifab that is going to be filled with recieved data.
+//!
+//! \param[in]  destbox The index box in the destination space that will be filled by data from 
+//!                     src. The source box that describes the dependencies will be computed by 
+//!                     the specified DTOS.
+//!
+//! \param[in]  src  The Multifab that is used to fill the send buffers.
+//!
+//! \param[in]  srccomp  The first component in src that will be copied to dest.
+//!
+//! \param[in]  destcomp The first component in dest that will get written by src.
+//!
+//! \param[in]  ncomp    The number of successive components that will be copied.
+//!
+//! \param[in]  ngrow    The amount of ghost cells that will be taking into consideration.
+//!                      Note, even if destbox contains indices outside the domain we need to 
+//!                      specify an appropiate ngrow that covers the amound of ghost cells that
+//!                      we want to copy. 
+//!
+//! \param[in]  dtos An index mapping that maps indices from destination space to source space
+//!                  and from source space to destination space.
+//!
+//! \param[in]  proj A transformation function that might change the data when it is being copied.
+//!
+//! \return Returns the CommMetaData object that can be cached for future calls to ParallelCopy.
+template <typename FAB, typename DTOS = Identity, typename Proj = Identity>
+EnableIf_t<IsBaseFab<FAB>() && IsIndexMapping<DTOS>() && IsFabProjection<Proj, FAB>(),
+MultiBlockCommMetaData>
+ParallelCopy (FabArray<FAB>& dest, const Box& destbox, const FabArray<FAB>& src, SrcComp srccomp,
+              DestComp destcomp, NumComps numcomp, const IntVect& ngrow, DTOS dtos = DTOS{}, Proj proj = Proj{}) {
+    MultiBlockCommMetaData cmd(dest, destbox, src, ngrow, dtos);
+    ParallelCopy(dest, src, cmd, srccomp, destcomp, numcomp, std::move(dtos), std::move(proj));
+    return cmd;
+}
+
+//! \brief Call ParallelCopy_nowait followed by ParallelCopy_finish.
+//!
+//! This function constructs a new MultiCommMetaData from the given DTOS, destbox and ngrow. 
+//!
+//! \param[out] dest The Multifab that is going to be filled with recieved data.
+//!
+//! \param[in]  destbox The index box in the destination space that will be filled by data from 
+//!                     src. The source box that describes the dependencies will be computed by 
+//!                     the specified DTOS.
+//!
+//! \param[in]  src  The Multifab that is used to fill the send buffers.
+//!
+//! \param[in]  srccomp  The first component in src that will be copied to dest.
+//!
+//! \param[in]  destcomp The first component in dest that will get written by src.
+//!
+//! \param[in]  ncomp    The number of successive components that will be copied.
+//!
+//! \param[in]  ngrow    The amount of ghost cells that will be taking into consideration.
+//!                      Note, even if destbox contains indices outside the domain we need to 
+//!                      specify an appropiate ngrow that covers the amound of ghost cells that
+//!                      we want to copy. 
+//!
+//! \param[in]  dtos An index mapping that maps indices from destination space to source space
+//!                  and from source space to destination space.
+//!
+//! \param[in]  proj A transformation function that might change the data when it is being copied.
+//!
+//! \return Returns the CommMetaData object that can be cached for future calls to ParallelCopy.
+template <typename FAB, typename DTOS = Identity, typename Proj = Identity>
+EnableIf_t<IsBaseFab<FAB>() && IsIndexMapping<DTOS>() && IsFabProjection<Proj, FAB>(),
+MultiBlockCommMetaData>
+ParallelCopy (FabArray<FAB>& dest, const Box& destbox, const FabArray<FAB>& src, int srccomp,
+              int destcomp, int numcomp, const IntVect& ngrow, DTOS dtos = DTOS{}, Proj proj = Proj{}) {
+    return ParallelCopy(dest, destbox, src, SrcComp(srccomp), DestComp(destcomp), NumComps(numcomp), ngrow, std::move(dtos), std::move(proj));
+}
+
 // Rotate90 fills the lo-x and lo-y boundary regions by rotating the data
 // around (x=0,y=0) by 90 degrees in either direction.  It also fills the
 // corner of lo-x and lo-y boundary region by rotating the data by 180
diff --git a/Src/Base/AMReX_NonLocalBC.cpp b/Src/Base/AMReX_NonLocalBC.cpp
new file mode 100644
index 00000000000..78d97bca584
--- /dev/null
+++ b/Src/Base/AMReX_NonLocalBC.cpp
@@ -0,0 +1,110 @@
+#include "AMReX_NonLocalBC.H"
+
+namespace amrex {
+namespace NonLocalBC {
+#ifdef AMREX_USE_MPI
+// Note, this is copied and modified from PrepareSendBuffers and PostRcvs
+void PrepareCommBuffers(CommData& comm, 
+                        const FabArrayBase::MapOfCopyComTagContainers& cctc, int n_components,
+                        std::size_t object_size, std::size_t align)
+{
+    comm.data.clear();
+    comm.size.clear();
+    comm.rank.clear();
+    comm.request.clear();
+    comm.offset.clear();
+    comm.cctc.clear();
+    comm.stats.clear();
+
+    const int N_comms = cctc.size();
+    if (N_comms == 0) return;
+    // reserve for upcominf push_backs
+    comm.data.reserve(N_comms);
+    comm.size.reserve(N_comms);
+    comm.rank.reserve(N_comms);
+    comm.request.reserve(N_comms);
+    comm.offset.reserve(N_comms);
+    comm.cctc.reserve(N_comms);
+    // resize to provide buffer for later
+    comm.stats.resize(N_comms);
+
+    std::size_t total_volume = 0;
+    for (const auto& kv : cctc)
+    {
+        std::size_t nbytes = 0;
+        for (auto const& cct : kv.second)
+        {
+            // Note: Does this hold for all FAB types? 
+            // This nBytes() implementation is currently also assumed in unpack_recv_buffers
+            nbytes += cct.sbox.numPts() * object_size * n_components;
+        }
+
+        std::size_t acd = ParallelDescriptor::alignof_comm_data(nbytes);
+        nbytes = amrex::aligned_size(acd, nbytes);  // so that nbytes are aligned
+
+        // Also need to align the offset properly
+        total_volume = amrex::aligned_size(std::max(align, acd), total_volume);
+
+        comm.offset.push_back(total_volume);
+        comm.data.push_back(nullptr);
+        comm.size.push_back(nbytes);
+        comm.rank.push_back(kv.first);
+        comm.request.push_back(MPI_REQUEST_NULL);
+        comm.cctc.push_back(&kv.second);
+
+        total_volume += nbytes;
+    }
+
+    if (total_volume == 0)
+    {
+        comm.the_data.reset();
+    }
+    else
+    {
+        comm.the_data.reset(static_cast<char*>(amrex::The_FA_Arena()->alloc(total_volume)));
+        for (int i = 0; i < N_comms; ++i) {
+            comm.data[i] = comm.the_data.get() + comm.offset[i];
+        }
+    }
+}
+
+void PostRecvs(CommData& recv, int mpi_tag) {
+    AMREX_ASSERT(recv.data.size() == recv.offset.size());
+    AMREX_ASSERT(recv.data.size() == recv.size.size());
+    AMREX_ASSERT(recv.data.size() == recv.rank.size());
+    AMREX_ASSERT(recv.data.size() == recv.request.size());
+    MPI_Comm comm = ParallelContext::CommunicatorSub();
+    for (int i = 0; i < recv.data.size(); ++i) {
+        if (recv.size[i] > 0) {
+            const int rank = ParallelContext::global_to_local_rank(recv.rank[i]);
+            AMREX_ASSERT(recv.data[i] != nullptr);
+            recv.request[i] =
+                ParallelDescriptor::Arecv(recv.data[i], recv.size[i], rank, mpi_tag, comm).req();
+        }
+    }
+}
+
+void PostSends(CommData& send, int mpi_tag) {
+    const int n_sends = send.data.size();
+    AMREX_ASSERT(n_sends == send.size.size());
+    AMREX_ASSERT(n_sends == send.rank.size());
+    AMREX_ASSERT(n_sends == send.request.size());
+    MPI_Comm comm = ParallelContext::CommunicatorSub();
+    for (int j = 0; j < n_sends; ++j) {
+        if (send.size[j] > 0) {
+            const int rank = ParallelContext::global_to_local_rank(send.rank[j]);
+            AMREX_ASSERT(send.data[j] != nullptr);
+            send.request[j] =
+                ParallelDescriptor::Asend(send.data[j], send.size[j], rank, mpi_tag, comm).req();
+        }
+    }
+}
+#endif
+
+template MultiBlockCommMetaData ParallelCopy(FabArray<FArrayBox>& dest, const Box& destbox,
+                                             const FabArray<FArrayBox>& src, int destcomp,
+                                             int srccomp, int numcomp, const IntVect& ngrow,
+                                             MultiBlockIndexMapping, Identity);
+
+} // namespace NonLocalBC
+} // namespace amrex
diff --git a/Src/Base/AMReX_NonLocalBCImpl.H b/Src/Base/AMReX_NonLocalBCImpl.H
index 8fde837f27c..36869ceba58 100644
--- a/Src/Base/AMReX_NonLocalBCImpl.H
+++ b/Src/Base/AMReX_NonLocalBCImpl.H
@@ -1,9 +1,13 @@
+#ifndef AMREX_NONLOCAL_BC_H_
+#include "AMReX_NonLocalBC.H"
+#endif
+
 #ifndef AMREX_NONLOCAL_BC_IMPL_H_
 #define AMREX_NONLOCAL_BC_IMPL_H_
 #include <AMReX_Config.H>
+#include <AMReX_TypeTraits.H>
 
 namespace amrex { namespace NonLocalBC {
-
 struct Rotate90ClockWise {
     AMREX_GPU_HOST_DEVICE
     IntVect operator() (IntVect const& iv) const noexcept {
@@ -155,37 +159,63 @@ struct PolarFn2 { // for the x-y corners
     }
 };
 
-template <class FAB, class DTOS>
-void
-local_copy_cpu (FabArray<FAB>& mf, int scomp, int ncomp, FabArrayBase::CommMetaData const& cmd,
-                DTOS dtos)
-{
-    auto const& LocTags = *(cmd.m_LocTags);
-    int N_locs = LocTags.size();
+//! \brief Perform the local copies from src to dest without doing any MPI communication.
+//!
+//! This function assumes that all destination and source boxes stored in the local copy comm tags
+//! are related by the DTOS. If this is not the case the behaviour will at best be caught as an
+//! assertion.
+//!
+//! \param[out] dest  The destination FabArray that will be filled with data.
+//!
+//! \param[in]  src   The source FabArray where data will be taken from.
+//!
+//! \param[in]  dcomp The first component at the destination.
+//!
+//! \param[in]  scomp The first component at the srouce.
+//!
+//! \param[in]  ncomp The number of components that will be copied.
+//!
+//! \param[in]  local_tags The vector of copy com tags that describes each local copy
+//!                        transaction.
+//!
+//! \param[in]  dtos  The dest to source index mapping that will be used in the copy function.
+//!
+//! \param[in]  proj  A FAB projection that might transform the data.
+//!
+//! \return Nothing.
+template <class FAB, class DTOS = Identity, class Proj = Identity>
+EnableIf_t<IsBaseFab<FAB>() && IsCallableR<Dim3, DTOS, Dim3>() && IsFabProjection<Proj, FAB>()>
+local_copy_cpu (FabArray<FAB>& dest, const FabArray<FAB>& src, int dcomp, int scomp, int ncomp,
+                FabArrayBase::CopyComTagsContainer const& local_tags, DTOS dtos = DTOS{},
+                Proj proj = Proj{}) noexcept {
+    const int N_locs = local_tags.size();
     if (N_locs == 0) return;
 #ifdef AMREX_USE_OMP
 #pragma omp parallel for
 #endif
     for (int itag = 0; itag < N_locs; ++itag) {
-        const auto& tag = LocTags[itag];
-        auto const& sfab = mf.const_array(tag.srcIndex);
-        auto const& dfab = mf.array      (tag.dstIndex);
+        const auto& tag = local_tags[itag];
+        auto const& sfab =  src.const_array(tag.srcIndex);
+        auto const& dfab = dest.array      (tag.dstIndex);
         amrex::LoopConcurrentOnCpu(tag.dbox, ncomp, [=] (int i, int j, int k, int n) noexcept
         {
             auto const si = dtos(Dim3{i,j,k});
-            dfab(i,j,k,scomp+n) = sfab(si.x,si.y,si.z,scomp+n);
+            dfab(i,j,k,dcomp+n) = proj(sfab,si,scomp+n);
         });
     }
 }
 
-template <class FAB, class DTOS>
-void
-unpack_recv_buffer_cpu (FabArray<FAB>& mf, int scomp, int ncomp,
-                        Vector<char*> const& recv_data,
+//! \brief Unpack the recieved data into the local FABs.
+//!
+//! This function assumes that all destination and source boxes stored in the local copy comm tags
+//! are related by the DTOS. If this is not the case the behaviour will at best be caught as an
+//! assertion.
+template <class FAB, class DTOS = Identity, class Proj = Identity>
+EnableIf_t<IsBaseFab<FAB>() && IsCallableR<Dim3, DTOS, Dim3>() && IsFabProjection<Proj, FAB>()>
+unpack_recv_buffer_cpu (FabArray<FAB>& mf, int dcomp, int ncomp, Vector<char*> const& recv_data,
                         Vector<std::size_t> const& recv_size,
                         Vector<FabArrayBase::CopyComTagsContainer const*> const& recv_cctc,
-                        DTOS dtos)
-{
+                        DTOS dtos = DTOS{}, Proj proj = Proj{}) noexcept {
     amrex::ignore_unused(recv_size);
 
     const int N_rcvs = recv_cctc.size();
@@ -201,10 +231,9 @@ unpack_recv_buffer_cpu (FabArray<FAB>& mf, int scomp, int ncomp,
         for (auto const& tag : cctc) {
             auto const& dfab = mf.array(tag.dstIndex);
             auto const& sfab = amrex::makeArray4((T const*)(dptr), tag.sbox, ncomp);
-            amrex::LoopConcurrentOnCpu(tag.dbox, ncomp, [=] (int i, int j, int k, int n) noexcept
-            {
-                auto const si = dtos(Dim3{i,j,k});
-                dfab(i,j,k,scomp+n) = sfab(si.x,si.y,si.z,n);
+            amrex::LoopConcurrentOnCpu(tag.dbox, ncomp, [=](int i, int j, int k, int n) noexcept {
+                auto const si = dtos(Dim3{i, j, k});
+                dfab(i, j, k, dcomp + n) = proj(sfab, si, n);
             });
             dptr += tag.sbox.numPts() * ncomp * sizeof(T);
             AMREX_ASSERT(dptr <= recv_data[ircv] + recv_size[ircv]);
@@ -223,38 +252,36 @@ struct Array4Array4Box {
     Box const& box () const noexcept { return dbox; }
 };
 
-template <class FAB, class DTOS>
-void
-local_copy_gpu (FabArray<FAB>& mf, int scomp, int ncomp, FabArrayBase::CommMetaData const& cmd,
-                DTOS dtos)
-{
-    auto const& LocTags = *(cmd.m_LocTags);
-    int N_locs = LocTags.size();
+template <class FAB, class DTOS = Identity, class Proj = Identity>
+EnableIf_t<IsBaseFab<FAB>() && IsCallableR<Dim3, DTOS, Dim3>() && IsFabProjection<Proj, FAB>()>
+local_copy_gpu (FabArray<FAB>& dest, const FabArray<FAB>& src, int dcomp, int scomp, int ncomp,
+                FabArrayBase::CopyComTagsContainer const& local_tags, DTOS dtos = DTOS{},
+                Proj proj = Proj{}) noexcept {
+    int N_locs = local_tags.size();
     if (N_locs == 0) return;
 
     using T = typename FAB::value_type;
     Vector<Array4Array4Box<T> > loc_copy_tags;
     loc_copy_tags.reserve(N_locs);
-    for (int itag = 0; itag < N_locs; ++itag) {
-        const auto& tag = LocTags[itag];
-        loc_copy_tags.push_back({mf.array(tag.dstIndex), mf.const_array(tag.srcIndex), tag.dbox});
+    for (auto const& tag : local_tags) {
+        loc_copy_tags.push_back({dest.array(tag.dstIndex), src.const_array(tag.srcIndex), tag.dbox});
     }
 
     ParallelFor(loc_copy_tags, ncomp, [=] AMREX_GPU_DEVICE (int i, int j, int k, int n,
                                                             Array4Array4Box<T> const& tag) noexcept
     {
         auto const si = dtos(Dim3{i,j,k});
-        tag.dfab(i,j,k,scomp+n) = tag.sfab(si.x,si.y,si.z,scomp+n);
+        tag.dfab(i,j,k,dcomp+n) = proj(tag.sfab, si, scomp+n);
     });
 }
 
-template <class FAB, class DTOS>
-void
+template <class FAB, class DTOS = Identity, class Proj = Identity>
+EnableIf_t<IsBaseFab<FAB>() && IsCallableR<Dim3, DTOS, Dim3>() && IsFabProjection<Proj, FAB>()>
 unpack_recv_buffer_gpu (FabArray<FAB>& mf, int scomp, int ncomp,
                         Vector<char*> const& recv_data,
                         Vector<std::size_t> const& recv_size,
                         Vector<FabArrayBase::CopyComTagsContainer const*> const& recv_cctc,
-                        DTOS dtos)
+                        DTOS dtos = DTOS{}, Proj proj = Proj{})
 {
     amrex::ignore_unused(recv_size);
 
@@ -299,7 +326,7 @@ unpack_recv_buffer_gpu (FabArray<FAB>& mf, int scomp, int ncomp,
                                                    Array4Array4Box<T> const& tag) noexcept
     {
         auto const si = dtos(Dim3{i,j,k});
-        tag.dfab(i,j,k,scomp+n) = tag.sfab(si.x,si.y,si.z,n);
+        tag.dfab(i,j,k,scomp+n) = proj(tag.sfab, si ,n);
     });
 
     // There is Gpu::synchronize in ParalleFor above
@@ -310,23 +337,70 @@ unpack_recv_buffer_gpu (FabArray<FAB>& mf, int scomp, int ncomp,
 }
 #endif
 
-struct CommHandler
-{
-#ifdef AMREX_USE_MPI
-    int mpi_tag;
-    TheFaArenaPointer the_recv_data = nullptr;
-    TheFaArenaPointer the_send_data = nullptr;
-    //
-    Vector<int>         recv_from;
-    Vector<char*>       recv_data;
-    Vector<std::size_t> recv_size;
-    Vector<MPI_Request> recv_reqs;
-    //
-    Vector<MPI_Request> send_reqs;
-#endif
-};
+template <typename DTOS, typename>
+MultiBlockCommMetaData::MultiBlockCommMetaData (const FabArrayBase& dst, const Box& dstbox, const FabArrayBase& src,
+                                                const IntVect& ngrow, DTOS dtos)
+    : MultiBlockCommMetaData(dst.boxArray(), dst.DistributionMap(), dstbox, src.boxArray(),
+                                src.DistributionMap(), ngrow, dtos) {}
+
+template <typename DTOS, typename>
+MultiBlockCommMetaData::MultiBlockCommMetaData (const BoxArray& dstba, const DistributionMapping& dstdm,
+                                                const Box& dstbox, const BoxArray& srcba,
+                                                const DistributionMapping& srcdm, const IntVect& ngrow, DTOS dtos) {
+    define(dstba, dstdm, dstbox, srcba, srcdm, ngrow, dtos);
+}
+
+template <typename DTOS>
+EnableIf_t<IsIndexMapping<DTOS>::value>
+MultiBlockCommMetaData::define (const BoxArray& dstba, const DistributionMapping& dstdm, const Box& dstbox,
+                                const BoxArray& srcba, const DistributionMapping& srcdm, const IntVect& ngrow, 
+                                DTOS dtos) {
+    m_LocTags = std::make_unique<FabArrayBase::CopyComTagsContainer>();
+    m_SndTags = std::make_unique<FabArrayBase::MapOfCopyComTagContainers>();
+    m_RcvTags = std::make_unique<FabArrayBase::MapOfCopyComTagContainers>();
+    const int myproc = ParallelDescriptor::MyProc();
+    for (int i = 0, N = dstba.size(); i < N; ++i) {
+        const int dest_owner = dstdm[i];
+        const Box partial_dstbox = grow(dstba[i], ngrow) & dstbox;
+        if (partial_dstbox.isEmpty()) {
+            continue;
+        }
+        const Box partial_dstbox_mapped_in_src = Image(dtos, partial_dstbox).setType(srcba.ixType());
+        enum { not_first_only = 0, first_only = 1 };
+        std::vector<std::pair<int, Box>> boxes_from_src =
+            srcba.intersections(partial_dstbox_mapped_in_src, not_first_only, ngrow);
+        for (std::pair<int, Box> counted_box : boxes_from_src) {
+            const int k = counted_box.first;
+            const Box src_box = counted_box.second;
+            AMREX_ASSERT(k < srcba.size());
+            const int src_owner = srcdm[k];
+            if (dest_owner == myproc || src_owner == myproc) {
+                if (src_owner == dest_owner) {
+                    const BoxList tilelist(src_box, FabArrayBase::comm_tile_size);
+                    for (const Box& tilebox : tilelist) {
+                        const Box inverse_image = InverseImage(dtos, tilebox).setType(dstba.ixType());
+                        if ((inverse_image & partial_dstbox).ok()) {
+                            m_LocTags->emplace_back(inverse_image, tilebox, i, k);
+                        }
+                    }
+                } else {
+                    const Box inverse_image = InverseImage(dtos, src_box).setType(dstba.ixType());
+                    if ((inverse_image & partial_dstbox).ok()) {
+                        FabArrayBase::CopyComTagsContainer& copy_tags =
+                            (src_owner == myproc) ? (*m_SndTags)[dest_owner]
+                                                    : (*m_RcvTags)[src_owner];
+                        copy_tags.emplace_back(inverse_image, src_box, i, k);
+                    }
+                }
+            }
+        }
+    }
+}
 
 template <class FAB, class DTOS>
+#ifdef AMREX_USE_MPI
+AMREX_NODISCARD
+#endif
 CommHandler
 Comm_nowait (FabArray<FAB>& mf, int scomp, int ncomp, FabArrayBase::CommMetaData const& cmd,
              DTOS dtos)
@@ -339,11 +413,11 @@ Comm_nowait (FabArray<FAB>& mf, int scomp, int ncomp, FabArrayBase::CommMetaData
         if (N_locs == 0) return CommHandler{};
 #ifdef AMREX_USE_GPU
         if (Gpu::inLaunchRegion()) {
-            local_copy_gpu(mf, scomp, ncomp, cmd, dtos);
+            local_copy_gpu(mf, mf, scomp, scomp, ncomp, *cmd.m_LocTags, dtos);
         } else
 #endif
         {
-            local_copy_cpu(mf, scomp, ncomp, cmd, dtos);
+            local_copy_cpu(mf, mf, scomp, scomp, ncomp, *cmd.m_LocTags, dtos);
         }
         return CommHandler{};
     }
@@ -368,42 +442,38 @@ Comm_nowait (FabArray<FAB>& mf, int scomp, int ncomp, FabArrayBase::CommMetaData
     handler.mpi_tag = SeqNum;
 
     if (N_rcvs > 0) {
-        handler.the_recv_data = mf.PostRcvs(*cmd.m_RcvTags, handler.recv_data, handler.recv_size,
-                    handler.recv_from, handler.recv_reqs, ncomp, SeqNum);
+        handler.recv.the_data = mf.PostRcvs(*cmd.m_RcvTags, handler.recv.data, handler.recv.size,
+                                            handler.recv.rank, handler.recv.request, ncomp, SeqNum);
     }
 
-    if (N_snds > 0)
-    {
-        Vector<std::size_t> send_size;
-        Vector<int> send_rank;
-        Vector<char*> send_data;
-        Vector<const FabArrayBase::CopyComTagsContainer*> send_cctc;
-        handler.the_send_data = mf.PrepareSendBuffers(*cmd.m_SndTags, send_data,
-                              send_size, send_rank, handler.send_reqs, send_cctc, ncomp);
+    if (N_snds > 0) {
+        handler.send.the_data =
+            mf.PrepareSendBuffers(*cmd.m_SndTags, handler.send.data, handler.send.size,
+                                  handler.send.rank, handler.send.request, handler.send.cctc, ncomp);
 
 #ifdef AMREX_USE_GPU
         if (Gpu::inLaunchRegion()) {
-            FabArray<FAB>::pack_send_buffer_gpu(mf, scomp, ncomp, send_data,
-                                                send_size, send_cctc);
+            FabArray<FAB>::pack_send_buffer_gpu(mf, scomp, ncomp, handler.send.data,
+                                                handler.send.size, handler.send.cctc);
         } else
 #endif
         {
-            FabArray<FAB>::pack_send_buffer_cpu(mf, scomp, ncomp, send_data,
-                                                send_size, send_cctc);
+            FabArray<FAB>::pack_send_buffer_cpu(mf, scomp, ncomp, handler.send.data,
+                                                handler.send.size, handler.send.cctc);
         }
 
-        FabArray<FAB>::PostSnds(send_data, send_size, send_rank, handler.send_reqs, SeqNum);
+        FabArray<FAB>::PostSnds(handler.send.data, handler.send.size, handler.send.rank, handler.send.request, SeqNum);
     }
 
     if (N_locs > 0)
     {
 #ifdef AMREX_USE_GPU
         if (Gpu::inLaunchRegion()) {
-            local_copy_gpu(mf, scomp, ncomp, cmd, dtos);
+            local_copy_gpu(mf, mf, scomp, scomp, ncomp, *cmd.m_LocTags, dtos);
         } else
 #endif
         {
-            local_copy_cpu(mf, scomp, ncomp, cmd, dtos);
+            local_copy_cpu(mf, mf, scomp, scomp, ncomp, *cmd.m_LocTags, dtos);
         }
     }
 
@@ -422,16 +492,15 @@ Comm_finish (FabArray<FAB>& mf, int scomp, int ncomp, FabArrayBase::CommMetaData
     const int N_rcvs = cmd.m_RcvTags->size();
     if (N_rcvs > 0)
     {
-        Vector<const FabArrayBase::CopyComTagsContainer*> recv_cctc(N_rcvs,nullptr);
+        handler.recv.cctc.resize(N_rcvs, nullptr);
         for (int k = 0; k < N_rcvs; ++k) {
-            auto const& cctc = cmd.m_RcvTags->at(handler.recv_from[k]);
-            recv_cctc[k] = &cctc;
+            auto const& cctc = cmd.m_RcvTags->at(handler.recv.rank[k]);
+            handler.recv.cctc[k] = &cctc;
         }
-
-        Vector<MPI_Status> stats(N_rcvs);
-        ParallelDescriptor::Waitall(handler.recv_reqs, stats);
+        handler.recv.stats.resize(handler.recv.request.size());
+        ParallelDescriptor::Waitall(handler.recv.request, handler.recv.stats);
 #ifdef AMREX_D_DEBUG
-        if (!CheckRcvStats(stats, handler.recv_size, handler.mpi_tag)) {
+        if (!CheckRcvStats(stats, handler.recv.size, handler.mpi_tag)) {
             amrex::Abort("NonLocalBC::Comm_finish failed with wrong message size");
         }
 #endif
@@ -439,20 +508,20 @@ Comm_finish (FabArray<FAB>& mf, int scomp, int ncomp, FabArrayBase::CommMetaData
 #ifdef AMREX_USE_GPU
         if (Gpu::inLaunchRegion())
         {
-            unpack_recv_buffer_gpu(mf, scomp, ncomp, handler.recv_data,
-                                   handler.recv_size, recv_cctc, dtos);
+            unpack_recv_buffer_gpu(mf, scomp, ncomp, handler.recv.data,
+                                   handler.recv.size, handler.recv.cctc, dtos);
         } else
 #endif
         {
-            unpack_recv_buffer_cpu(mf, scomp, ncomp, handler.recv_data,
-                                   handler.recv_size, recv_cctc, dtos);
+            unpack_recv_buffer_cpu(mf, scomp, ncomp, handler.recv.data,
+                                   handler.recv.size, handler.recv.cctc, dtos);
         }
     }
 
     const int N_snds = cmd.m_SndTags->size();
     if (N_snds > 0) {
-        Vector<MPI_Status> stats(handler.send_reqs.size());
-        ParallelDescriptor::Waitall(handler.send_reqs, stats);
+        handler.send.stats.resize(handler.send.request.size());
+        ParallelDescriptor::Waitall(handler.send.request, handler.send.stats);
     }
 }
 #endif
@@ -573,6 +642,10 @@ FillPolar (FabArray<FAB>& mf, Box const& domain)
     FillPolar(mf, 0, mf.nComp(), mf.nGrowVect(), domain);
 }
 
+extern template MultiBlockCommMetaData ParallelCopy(FabArray<FArrayBox>& dest, const Box& destbox,
+                                                    const FabArray<FArrayBox>& src, int destcomp,
+                                                    int srccomp, int numcomp, const IntVect& ngrow,
+                                                    MultiBlockIndexMapping, Identity);
 }}
 
 #endif
diff --git a/Src/Base/AMReX_TypeTraits.H b/Src/Base/AMReX_TypeTraits.H
index 2fac1690ca4..1b1468bb503 100644
--- a/Src/Base/AMReX_TypeTraits.H
+++ b/Src/Base/AMReX_TypeTraits.H
@@ -39,16 +39,6 @@ namespace amrex
     template <bool B, class T = void>
     using EnableIf_t = typename std::enable_if<B,T>::type;
 
-
-    template <class T, class Enable = void>
-    struct IsStoreAtomic : std::false_type {};
-    //
-    template <class T>
-    struct IsStoreAtomic<T, typename std::enable_if <
-                                std::is_arithmetic<T>::value
-                                && sizeof(T) <= 8 >::type >
-        : std::true_type {};
-
     template <class T, class Enable = void>
     struct HasAtomicAdd : std::false_type {};
     template <> struct HasAtomicAdd<int> : std::true_type {};
@@ -124,6 +114,122 @@ namespace amrex
     {
         static constexpr bool value = std::is_same<T,U1>::value && Same<T,Us...>::value;
     };
+
+    ////////////////////////////////////////////////////////////////////////////////
+    //                                                           [traits.IsDetected]
+    //
+    // We use IsDetected as a SFINAE tool to test for valid expressions.
+    //
+    // is_detected was proposed to C++ but is surpassed by concepts.
+    //
+    // The implementation is taken from 
+    //
+    // https://en.cppreference.com/w/cpp/experimental/is_detected
+    
+    namespace detail {
+        template <class...> using Void_t = void;
+
+        struct Nonesuch {
+            Nonesuch() = delete;
+            ~Nonesuch() = delete;
+            Nonesuch(Nonesuch const&) = delete;
+            void operator=(Nonesuch const&) = delete;
+        };
+
+        template <class Default, class AlwaysVoid, template <class...> class Op,
+                class... Args>
+        struct Detector {
+            using value_t = std::false_type;
+            using type = Default;
+        };
+
+        template <class Default, template <class...> class Op, class... Args>
+        struct Detector<Default, Void_t<Op<Args...>>, Op, Args...> {
+            using value_t = std::true_type;
+            using type = Op<Args...>;
+        };
+    }
+
+    template <template <class...> class Op, class... Args>
+    using IsDetected = typename detail::Detector<detail::Nonesuch, void, Op, Args...>::value_t;
+
+    template <template <class...> class Op, class... Args>
+    using Detected_t = typename detail::Detector<detail::Nonesuch, void, Op, Args...>::type;
+
+    template <class Default, template <class...> class Op, class... Args>
+    using DetectedOr = typename detail::Detector<Default, void, Op, Args...>::type;
+
+    template <class Expected, template <typename...> class Op, class... Args>
+    using IsDetectedExact = std::is_same<Expected, Detected_t<Op, Args...>>;
+
+    ////////////////////////////////////////////////////////////////////////////////
+    //                                                           [traits.IsCallable]
+
+    namespace detail {
+        template <typename T, typename... Args>
+        using call_result_t = decltype(std::declval<T>()(std::declval<Args>()...));
+    }
+
+    //! \brief Test if a given type T is callable with arguments of type Args...
+    //!
+    //! This type trait is different from std::is_invocable since it only cares for the call syntax
+    //! f(args...) and disregards pointers to class methods and such.
+    //!
+    //! For consistency we use IsCallable over is_invocable since AMReX relies on call syntax
+    //! everywhere else. Using an equivalent function to std::invoke would increase call stacks in
+    //! debug mode, add more template instantiations and will obfuscate lots of generic code that
+    //! takes functions.
+    template <typename T, typename... Args>
+    struct IsCallable : IsDetected<detail::call_result_t, T, Args...> {};
+
+    //! \brief Test if a given type T is callable with arguments of type Args...
+    //!
+    //! \see IsCallable
+    template <typename R, typename T, typename... Args>
+    struct IsCallableR : IsDetectedExact<R, detail::call_result_t, T, Args...> {};
+
+    ////////////////////////////////////////////////////////////////////////////////
+    //                                                          [traits.Conjunction]
+    //                                                          [traits.Disjunction]
+    //                                                          [traits.Negation]
+
+
+    //! Logical traits let us combine multiple type requirements in one EnableIf_t clause. 
+    #if defined(__cpp_lib_logical_traits)
+    template <typename... Args> using Conjunction = std::conjunction<Args...>;
+    template <typename... Args> using Disjunction = std::disjunction<Args...>;
+    template <typename... Args> using Negation = std::negation<Args...>;
+    #elif defined(__cpp_lib_experimental_logical_traits)
+    template <typename... Args> using Conjunction = std::experimental::conjunction<Args...>;
+    template <typename... Args> using Disjunction = std::experimental::disjunction<Args...>;
+    template <typename... Args> using Negation = std::experimental::negation<Args...>;
+    #else
+    template <class...> struct Conjunction : std::true_type {};
+    template <class B1> struct Conjunction<B1> : B1 {};
+    template <class B1, class... Bn>
+    struct Conjunction<B1, Bn...>
+    : std::conditional_t<bool(B1::value), Conjunction<Bn...>, B1> {};
+
+    template <class...> struct Disjunction : std::false_type {};
+    template <class B1> struct Disjunction<B1> : B1 {};
+    template <class B1, class... Bn>
+    struct Disjunction<B1, Bn...>
+    : std::conditional_t<bool(B1::value), B1, Disjunction<Bn...>> {};
+
+    template <class Bool>
+    using Negation = std::integral_constant<bool, !bool(Bool::value)>;
+    #endif
+
+    // Move this down, because doxygen can not parse anything below IsStoreAtomic
+    template <class T, class Enable = void>
+    struct IsStoreAtomic : std::false_type {};
+    //
+    template <class T>
+    struct IsStoreAtomic<T, typename std::enable_if <
+                                std::is_arithmetic<T>::value
+                                && sizeof(T) <= 8 >::type >
+        : std::true_type {};
+
 }
 
 #endif
diff --git a/Src/Base/CMakeLists.txt b/Src/Base/CMakeLists.txt
index 70851dbc274..136902d0ec6 100644
--- a/Src/Base/CMakeLists.txt
+++ b/Src/Base/CMakeLists.txt
@@ -170,6 +170,7 @@ target_sources( amrex
    AMReX_FilND_C.cpp
    # Non-Local BC
    AMReX_NonLocalBC.H
+   AMReX_NonLocalBC.cpp
    AMReX_NonLocalBCImpl.H
    # Plotfile ----------------------------------------------------------------
    AMReX_PlotFileUtil.cpp
diff --git a/Src/Base/Make.package b/Src/Base/Make.package
index 5640cbc7a7f..ad73072213a 100644
--- a/Src/Base/Make.package
+++ b/Src/Base/Make.package
@@ -180,6 +180,7 @@ C$(AMREX_BASE)_headers += AMReX_FilND_C.H
 # Non-Local BC
 #
 C$(AMREX_BASE)_headers += AMReX_NonLocalBC.H AMReX_NonLocalBCImpl.H
+C$(AMREX_BASE)_sources += AMReX_NonLocalBC.cpp
 
 #
 # Plotfile


Test patch:
diff --git a/Tests/CMakeLists.txt b/Tests/CMakeLists.txt
index 2c515d96e6e..fee31c13603 100644
--- a/Tests/CMakeLists.txt
+++ b/Tests/CMakeLists.txt
@@ -1,7 +1,7 @@
 #
 # List of subdirectories to search for CMakeLists.
 #
-set( AMREX_TESTS_SUBDIRS AsyncOut )
+set( AMREX_TESTS_SUBDIRS AsyncOut MultiBlock )
 
 if (AMReX_PARTICLES)
    list(APPEND AMREX_TESTS_SUBDIRS Particles)
diff --git a/Tests/MultiBlock/Advection/CMakeLists.txt b/Tests/MultiBlock/Advection/CMakeLists.txt
new file mode 100644
index 00000000000..ac043eefd8d
--- /dev/null
+++ b/Tests/MultiBlock/Advection/CMakeLists.txt
@@ -0,0 +1,7 @@
+set(_sources     main.cpp)
+set(_input_files )
+
+setup_test(_sources _input_files)
+
+unset(_sources)
+unset(_input_files)
diff --git a/Tests/MultiBlock/Advection/GNUmakefile b/Tests/MultiBlock/Advection/GNUmakefile
new file mode 100644
index 00000000000..5417f2b6cd7
--- /dev/null
+++ b/Tests/MultiBlock/Advection/GNUmakefile
@@ -0,0 +1,22 @@
+AMREX_HOME = ../../../
+
+DEBUG	= FALSE
+#DEBUG	= TRUE
+
+DIM	= 3
+
+COMP    = gcc
+
+USE_MPI   = TRUE
+USE_OMP   = FALSE
+USE_CUDA  = FALSE
+USE_HIP   = FALSE
+
+include $(AMREX_HOME)/Tools/GNUMake/Make.defs
+
+include ./Make.package
+include $(AMREX_HOME)/Src/Base/Make.package
+include $(AMREX_HOME)/Src/Boundary/Make.package
+include $(AMREX_HOME)/Src/AmrCore/Make.package
+
+include $(AMREX_HOME)/Tools/GNUMake/Make.rules
diff --git a/Tests/MultiBlock/Advection/Make.package b/Tests/MultiBlock/Advection/Make.package
new file mode 100644
index 00000000000..6b4b865e8fc
--- /dev/null
+++ b/Tests/MultiBlock/Advection/Make.package
@@ -0,0 +1,1 @@
+CEXE_sources += main.cpp
diff --git a/Tests/MultiBlock/Advection/main.cpp b/Tests/MultiBlock/Advection/main.cpp
new file mode 100644
index 00000000000..13a31428fcf
--- /dev/null
+++ b/Tests/MultiBlock/Advection/main.cpp
@@ -0,0 +1,297 @@
+
+#include "AMReX_NonLocalBC.H"
+
+#include "AMReX.H"
+#include "AMReX_AmrCore.H"
+#include "AMReX_MultiFab.H"
+
+#include "AMReX_PlotFileUtil.H"
+
+void MyMain();
+
+int main(int argc, char** argv) {
+#ifdef AMREX_USE_MPI
+    MPI_Init(&argc, &argv);
+#endif
+    // Let me throw exceptions for triggering my debugger
+    amrex::Initialize(MPI_COMM_WORLD, std::cout, std::cerr, [](const char* msg) { throw std::runtime_error(msg); });
+    MyMain();
+    amrex::Finalize();
+#ifdef AMREX_USE_MPI
+    MPI_Finalize();
+#endif
+}
+
+using namespace amrex;
+
+enum idirs { ix, iy };
+enum num_components { three_components = 3 };
+
+static constexpr IntVect e_x = IntVect::TheDimensionVector(ix);
+static constexpr IntVect e_y = IntVect::TheDimensionVector(iy);
+class AdvectionAmrCore : public AmrCore {
+  
+  public:
+    AdvectionAmrCore(Direction vel, Geometry const& level_0_geom,
+                     AmrInfo const& amr_info = AmrInfo())
+        : AmrCore(level_0_geom, amr_info), velocity{vel} {
+        AmrCore::InitFromScratch(0.0);
+        InitData();  // CUDA does not allow extended lambdas in ctors.
+    }
+
+    void InitData() {
+        const auto problo = Geom(0).ProbLoArray();
+        const auto dx = Geom(0).CellSizeArray();
+#ifdef AMREX_USE_OMP
+#pragma omp parallel if (Gpu::notInLaunchRegion())
+#endif
+        for (MFIter mfi(mass,TilingIfNotGPU()); mfi.isValid(); ++mfi) {
+            Array4<Real> m = mass.array(mfi);
+            Array4<Real> vx = mass.array(mfi, 1);
+            Array4<Real> vy = mass.array(mfi, 2);
+            amrex::ParallelFor(mfi.tilebox(), [=] AMREX_GPU_DEVICE (int i, int j, int k) {
+                Real x[] = {AMREX_D_DECL(problo[0] + (0.5+i)*dx[0],
+                                         problo[1] + (0.5+j)*dx[1],
+                                         problo[2] + (0.5+k)*dx[2])};
+                const double r2 = AMREX_D_TERM(x[0] * x[0], + x[1] * x[1], + x[2] * x[2]);
+                constexpr double R = 0.1 * 0.1;
+                m(i, j, k) = r2 < R ? 1.0 : 0.0;
+                vx(i, j, k) = r2 < R ? 1.0 : 0.0;
+                vy(i, j, k) = 0.0;
+            });
+        }
+    }
+
+    void AdvanceInTime(double dt) {
+        // Do first order accurate godunov splitting
+        for (int d = 0; d < AMREX_SPACEDIM; ++d) {
+            DoOperatorSplitStep(dt, static_cast<Direction>(d));
+        }
+    }
+
+    void DoOperatorSplitStep(double dt, Direction dir) {
+        // Perform first order accurate upwinding with velocity 1 in the stored direction.
+        const double dx = Geom(0).CellSize(0);
+        const double a_dt_over_dx = dt / dx * (velocity == dir);
+        if (dir == Direction::x) {
+#ifdef AMREX_USE_OMP
+#pragma omp parallel if (Gpu::notInLaunchRegion())
+#endif
+            for (MFIter mfi(mass); mfi.isValid(); ++mfi) {
+                Array4<Real> m = mass.array(mfi);
+                Array4<Real> next = mass_next.array(mfi);
+                ParallelFor(mfi.growntilebox(e_y), int(three_components),
+                            [=] AMREX_GPU_DEVICE (int i, int j, int k, int n) {
+                                next(i, j, k, n) = m(i, j, k, n) - a_dt_over_dx * (m(i, j, k, n) - m(i - 1, j, k, n));
+                            });
+            }
+        } else if (dir == Direction::y) {
+#ifdef AMREX_USE_OMP
+#pragma omp parallel if (Gpu::notInLaunchRegion())
+#endif
+            for (MFIter mfi(mass); mfi.isValid(); ++mfi) {
+                Array4<Real> m = mass.array(mfi);
+                Array4<Real> next = mass_next.array(mfi);
+                ParallelFor(mfi.growntilebox(ix), int(three_components),
+                            [=] AMREX_GPU_DEVICE (int i, int j, int k, int n) {
+                                next(i, j, k, n) = m(i, j, k, n) - a_dt_over_dx * (m(i, j, k, n) - m(i, j - 1, k, n));
+                            });
+            }
+        }
+        std::swap(mass, mass_next);
+    }
+
+    MultiFab mass{};
+    MultiFab mass_next{};
+    Direction velocity{};
+
+  private:
+    void ErrorEst(int /*level*/, ::amrex::TagBoxArray& /*tags*/, double /*time_point*/,
+                  int /* ngrow */) override {
+        throw std::runtime_error("For simplicity, this example supports only one level.");
+    }
+
+    void
+    MakeNewLevelFromScratch(int level, double, const ::amrex::BoxArray& box_array,
+                            const ::amrex::DistributionMapping& distribution_mapping) override {
+        if (level > 0) {
+            throw std::runtime_error("For simplicity, this example supports only one level.");
+        }
+        const IntVect ngrow{AMREX_D_DECL(1, 1, 0)};
+        mass.define(box_array, distribution_mapping, three_components, ngrow);
+        mass_next.define(box_array, distribution_mapping, three_components, ngrow);
+    }
+
+    void MakeNewLevelFromCoarse(int /*level*/, double /*time_point*/, const ::amrex::BoxArray&,
+                                const ::amrex::DistributionMapping&) override {
+        throw std::runtime_error("For simplicity, this example supports only one level.");
+    }
+
+    void RemakeLevel(int /*level*/, double /*time_point*/, const ::amrex::BoxArray&,
+                     const ::amrex::DistributionMapping&) override {
+        throw std::runtime_error("For simplicity, this example supports only one level.");
+    }
+
+    void ClearLevel(int level) override {
+        if (level > 0) {
+            throw std::runtime_error("For simplicity, this example supports only one level.");
+        }
+        mass.clear();
+    }
+};
+
+using namespace NonLocalBC;
+struct OnesidedMultiBlockBoundaryFn {
+  AdvectionAmrCore* dest;
+  const AdvectionAmrCore* src;
+  MultiBlockIndexMapping dtos;
+  Box boundary_to_fill;
+  std::unique_ptr<MultiBlockCommMetaData> cmd{};
+  FabArrayBase::BDKey cached_dest_bd_key{};
+  FabArrayBase::BDKey cached_src_bd_key{};
+  ApplyDtosAndProjectionOnReciever<MultiBlockIndexMapping,
+                                   MapComponents<Identity, SwapComponents<1, 2>>>
+      packing{PackComponents{0, 0, three_components}, dtos};
+
+  AMREX_NODISCARD CommHandler FillBoundary_nowait() {
+    if (!cmd || cached_dest_bd_key != dest->mass.getBDKey() || cached_src_bd_key != src->mass.getBDKey()) {
+        cmd = std::make_unique<MultiBlockCommMetaData>(dest->mass, boundary_to_fill, src->mass, dest->mass.nGrowVect(), dtos);
+        cached_dest_bd_key = dest->mass.getBDKey();
+        cached_src_bd_key = src->mass.getBDKey();
+    }
+    
+    return ParallelCopy_nowait(no_local_copy, dest->mass, src->mass, *cmd, packing);
+  }
+
+  void FillBoundary_do_local_copy() const {
+    AMREX_ASSERT(cmd && cached_dest_bd_key == dest->mass.getBDKey() && cached_src_bd_key == src->mass.getBDKey());
+    if (cmd->m_LocTags && cmd->m_LocTags->size() > 0) {
+        LocalCopy(packing, dest->mass, src->mass, *cmd->m_LocTags);
+    }
+  }
+
+  void FillBoundary_finish(CommHandler handler) const {
+    ParallelCopy_finish(dest->mass, std::move(handler), *cmd, packing);
+  }
+};
+
+struct FillBoundaryFn {
+    std::vector<OnesidedMultiBlockBoundaryFn> boundaries;
+
+    void operator()(AdvectionAmrCore& core_x, AdvectionAmrCore& core_y) {
+        enum { coarsest_level = 0 };
+        core_x.mass.FillBoundary(core_x.Geom(coarsest_level).periodicity());
+        core_y.mass.FillBoundary(core_y.Geom(coarsest_level).periodicity());
+        std::vector<CommHandler> comms;
+        for (auto& boundary : boundaries) {
+            comms.push_back(boundary.FillBoundary_nowait());
+        }
+        for (auto& boundary : boundaries) {
+            boundary.FillBoundary_do_local_copy();
+        }
+        const std::size_t n_boundaries = boundaries.size();
+        for (std::size_t i = 0; i < n_boundaries; ++i) {
+            boundaries[i].FillBoundary_finish(std::move(comms[i]));
+        }
+        AMREX_ASSERT(!core_x.mass.contains_nan());
+    }
+};
+
+void WritePlotfiles(const AdvectionAmrCore& core_x, const AdvectionAmrCore& core_y, double time_point, int step)
+{
+    static const Vector<std::string> varnames{"Mass", "Vector_X", "Vector_Y"};
+    int nlevels = 1;
+    std::array<char, 256> x_pbuffer{};
+    std::array<char, 256> y_pbuffer{};
+    snprintf(x_pbuffer.data(), x_pbuffer.size(), "MultiBlock/core_x/plt%04d", step);
+    snprintf(y_pbuffer.data(), y_pbuffer.size(), "MultiBlock/core_y/plt%04d", step);
+
+    {
+        Vector<const MultiFab*> mf{&core_x.mass};
+        Vector<Geometry> geoms{core_x.Geom(0)};
+        Vector<int> level_steps{step};
+        Vector<IntVect> ref_ratio{};
+        std::string plotfilename{x_pbuffer.data()};
+        WriteMultiLevelPlotfile(plotfilename, nlevels, mf, varnames, geoms, time_point, level_steps, ref_ratio);
+    }
+    {
+        Vector<const MultiFab*> mf{&core_y.mass};
+        Vector<Geometry> geoms{core_y.Geom(0)};
+        Vector<int> level_steps{step};
+        Vector<IntVect> ref_ratio{};
+        std::string plotfilename{y_pbuffer.data()};
+        WriteMultiLevelPlotfile(plotfilename, nlevels, mf, varnames, geoms, time_point, level_steps, ref_ratio);
+    }
+}
+
+void MyMain() {
+    Box domain(IntVect{}, IntVect{AMREX_D_DECL(63, 63, 0)});
+    RealBox real_box1{{AMREX_D_DECL(-0.5, -0.3, 0.0)}, {AMREX_D_DECL(0.5, 0.7, 1.0)}};
+    RealBox real_box2{{AMREX_D_DECL(+0.55, -0.3, 0.0)}, {AMREX_D_DECL(1.55, 0.7, 1.0)}};
+
+    Array<int, AMREX_SPACEDIM> is_periodic1{0, 1};
+    Geometry geom1{domain, real_box1, CoordSys::cartesian, is_periodic1};
+
+    Array<int, AMREX_SPACEDIM> is_periodic2{1, 0};
+    Geometry geom2{domain, real_box2, CoordSys::cartesian, is_periodic2};
+
+    AmrInfo amr_info{};
+#if AMREX_SPACEDIM > 2
+    amr_info.blocking_factor[0][2] = 1;
+#endif
+
+    AdvectionAmrCore core_x(Direction::x, geom1, amr_info);
+    AdvectionAmrCore core_y(Direction::y, geom2, amr_info);
+
+    std::vector<OnesidedMultiBlockBoundaryFn> multi_block_boundaries{};
+    {   // Fill right boundary of core_x with lower mirror data of core_y
+        NonLocalBC::MultiBlockIndexMapping dtos{};
+        dtos.permutation = IntVect{AMREX_D_DECL(1, 0, 2)};
+        dtos.offset = (domain.bigEnd(iy) + 1) * e_y + domain.bigEnd(ix) * e_x;
+        dtos.sign = IntVect{AMREX_D_DECL(-1, 1, 1)};
+        Box right_boundary_to_fill_in_x = grow(shift(Box{domain.bigEnd(ix) * e_x, domain.bigEnd()}, e_x), e_y);
+        multi_block_boundaries.push_back({&core_x, &core_y, dtos, right_boundary_to_fill_in_x});
+    } { // Fill lower boundary of core_y with right mirror data of core_x
+        NonLocalBC::MultiBlockIndexMapping dtos{};
+        dtos.permutation = IntVect{AMREX_D_DECL(1, 0, 2)};
+        dtos.offset = domain.bigEnd(iy) * e_y - (domain.bigEnd(ix) + 1) * e_x;
+        dtos.sign = IntVect{AMREX_D_DECL(1, -1, 1)};
+        Box lower_boundary_to_fill_in_y = grow(shift(Box{domain.smallEnd(), domain.bigEnd() - domain.bigEnd(iy) * e_y}, -e_y), e_x);
+        multi_block_boundaries.push_back({&core_y, &core_x, dtos, lower_boundary_to_fill_in_y});
+    } { // Fill left boundary of core_x with upper mirror data of core_y
+        NonLocalBC::MultiBlockIndexMapping dtos{};
+        dtos.permutation = IntVect{AMREX_D_DECL(1, 0, 2)};
+        dtos.offset = -(domain.bigEnd(iy) + 1) * e_y;
+        Box left_boundary_to_fill_in_x = grow(shift(Box{domain.smallEnd(), domain.bigEnd() - domain.bigEnd(ix) * e_x}, -e_x), e_y);
+        multi_block_boundaries.push_back({&core_x, &core_y, dtos, left_boundary_to_fill_in_x});
+    } { // Fill upper boundary of core_y with left mirror data of core_x
+        NonLocalBC::MultiBlockIndexMapping dtos{};
+        dtos.permutation = IntVect{AMREX_D_DECL(1, 0, 2)};
+        dtos.offset = (domain.bigEnd(ix) + 1) * e_x;
+        Box upper_boundary_to_fill_in_y = grow(shift(Box{domain.bigEnd(iy) * e_y, domain.bigEnd()}, e_y), e_x);
+        multi_block_boundaries.push_back({&core_y, &core_x, dtos, upper_boundary_to_fill_in_y});
+    }
+
+    FillBoundaryFn FillBoundary{std::move(multi_block_boundaries)};
+
+    int step = 1;
+    const double min_dx1_dy2 = std::min(geom1.CellSize(0), geom2.CellSize(1));
+    const double cfl = 1.0;
+    const double dt = cfl * min_dx1_dy2;
+    const double final_time = 4.0;
+    double time_point = 0.0;
+
+    WritePlotfiles(core_x, core_y, time_point, step);
+    while (time_point < final_time) {
+        FillBoundary(core_x, core_y);
+
+        core_x.AdvanceInTime(dt);
+        core_y.AdvanceInTime(dt);
+
+        amrex::Print() << "Step #" << step << ", Time Point = " << time_point << '\n';
+
+        time_point += dt;
+        step += 1;  
+        WritePlotfiles(core_x, core_y, time_point, step);
+    }
+}
diff --git a/Tests/MultiBlock/IndexType/CMakeLists.txt b/Tests/MultiBlock/IndexType/CMakeLists.txt
new file mode 100644
index 00000000000..ac043eefd8d
--- /dev/null
+++ b/Tests/MultiBlock/IndexType/CMakeLists.txt
@@ -0,0 +1,7 @@
+set(_sources     main.cpp)
+set(_input_files )
+
+setup_test(_sources _input_files)
+
+unset(_sources)
+unset(_input_files)
diff --git a/Tests/MultiBlock/IndexType/GNUmakefile b/Tests/MultiBlock/IndexType/GNUmakefile
new file mode 100644
index 00000000000..a2198614ac9
--- /dev/null
+++ b/Tests/MultiBlock/IndexType/GNUmakefile
@@ -0,0 +1,20 @@
+AMREX_HOME = ../../../
+
+DEBUG	= FALSE
+#DEBUG	= TRUE
+
+DIM	= 3
+
+COMP    = gcc
+
+USE_MPI   = TRUE
+USE_OMP   = FALSE
+USE_CUDA  = FALSE
+USE_HIP   = FALSE
+
+include $(AMREX_HOME)/Tools/GNUMake/Make.defs
+
+include ./Make.package
+include $(AMREX_HOME)/Src/Base/Make.package
+
+include $(AMREX_HOME)/Tools/GNUMake/Make.rules
diff --git a/Tests/MultiBlock/IndexType/Make.package b/Tests/MultiBlock/IndexType/Make.package
new file mode 100644
index 00000000000..6b4b865e8fc
--- /dev/null
+++ b/Tests/MultiBlock/IndexType/Make.package
@@ -0,0 +1,1 @@
+CEXE_sources += main.cpp
diff --git a/Tests/MultiBlock/IndexType/main.cpp b/Tests/MultiBlock/IndexType/main.cpp
new file mode 100644
index 00000000000..3cfe48a4e41
--- /dev/null
+++ b/Tests/MultiBlock/IndexType/main.cpp
@@ -0,0 +1,157 @@
+#include "AMReX_NonLocalBC.H"
+
+#include "AMReX.H"
+#include "AMReX_iMultiFab.H"
+
+int MyMain();
+
+int main(int argc, char** argv) {
+#ifdef AMREX_USE_MPI
+    MPI_Init(&argc, &argv);
+#endif
+    // Let me throw exceptions for triggering my debugger
+    amrex::Initialize(MPI_COMM_WORLD, std::cout, std::cerr, [](const char* msg) { throw std::runtime_error(msg); });
+    int ret = MyMain();
+    amrex::Finalize();
+#ifdef AMREX_USE_MPI
+    MPI_Finalize();
+#endif
+    return ret;
+}
+
+amrex::iMultiFab InitializeMultiFab(const amrex::Box& domain)
+{
+    amrex::BoxArray ba(domain);
+    // do some work across MPI ranks.
+    ba.maxSize(8);
+    amrex::DistributionMapping dm(ba);
+    amrex::iMultiFab mf(ba, dm, 1, 0);
+    const int nx = domain.length(0);
+    const int ny = domain.length(1);
+    for (amrex::MFIter mfi(mf); mfi.isValid(); ++mfi) {
+        auto array = mf.array(mfi);
+        ParallelFor(mfi.tilebox(), [=](int i, int j, int k) 
+        {
+            array(i,j,k) = i + j*nx + k*nx*ny;
+        });
+    }
+    return mf;
+}
+
+bool ParallelCopyWithItselfIsCorrect(amrex::iMultiFab& mf, const amrex::Box& domain) {
+    const amrex::IntVect e_x = amrex::IntVect::TheDimensionVector(0);
+    const amrex::Box source_box{domain.smallEnd(), domain.bigEnd() - domain.bigEnd(0) * e_x, domain.ixType()};
+    const amrex::Box dest_box{domain.smallEnd() + domain.bigEnd(0) * e_x, domain.bigEnd(), domain.ixType()};
+    amrex::NonLocalBC::MultiBlockIndexMapping dtos;
+    dtos.offset = dest_box.smallEnd() - source_box.smallEnd();
+    amrex::NonLocalBC::ParallelCopy(mf, dest_box, mf, 0, 0, 1, amrex::IntVect{0}, dtos);
+    const int nx = domain.length(0);
+    const int ny = domain.length(1);
+    int fails = 0;
+    for (amrex::MFIter mfi(mf); mfi.isValid(); ++mfi) {
+        const amrex::Box section = dest_box & mfi.tilebox();
+        if (section.isEmpty()) continue; 
+        auto array = mf.const_array(mfi);
+        amrex::LoopOnCpu(section, [&](int i, int j, int k)
+        {
+            amrex::Dim3 si = dtos(amrex::Dim3{i,j,k});
+            int value = si.x + si.y*nx + si.z*nx*ny;
+            AMREX_ASSERT(array(i,j,k) == value);
+        });
+    }
+    return fails == 0;
+}
+
+int GetFaceDir(amrex::IndexType iv)
+{
+    static_assert(sizeof(amrex::IndexType) == sizeof(unsigned int), "IndexType is not punnable to unsigned int");
+    unsigned int value{0};
+    std::memcpy(&value, &iv, sizeof(unsigned int));
+    return ((value & 0b001) + (value & 0b010) + (value & 0b100)) >> 1;
+}
+
+amrex::Box GetFaceBoundary(const amrex::Box& domain, amrex::Orientation::Side side)
+{
+    int dir = GetFaceDir(domain.ixType());
+    const amrex::IntVect face_normal = amrex::IntVect::TheDimensionVector(dir);
+    amrex::Box box{};
+    AMREX_ASSERT(side == amrex::Orientation::Side::low || side == amrex::Orientation::Side::high);
+    if (side == amrex::Orientation::Side::low) {
+        box = amrex::Box{domain.smallEnd(), domain.bigEnd() - domain.bigEnd(dir) * face_normal, domain.ixType()};
+    } else {
+        box = amrex::Box{domain.smallEnd() + domain.bigEnd(dir) * face_normal, domain.bigEnd(), domain.ixType()};
+    }
+    return box;
+}
+
+bool ParallelCopyFaceToFace(amrex::iMultiFab& dest, const amrex::Box& domain_dest, amrex::Orientation::Side dest_side,
+                            const amrex::iMultiFab& src, const amrex::Box& domain_src, amrex::Orientation::Side src_side)
+{
+    int sdir = GetFaceDir(domain_src.ixType());
+    int ddir = GetFaceDir(domain_dest.ixType());
+    const amrex::Box src_box = GetFaceBoundary(domain_src, src_side);
+    const amrex::Box dest_box = GetFaceBoundary(domain_dest, dest_side);
+    
+    // Default construction is identity
+    amrex::NonLocalBC::MultiBlockIndexMapping dtos{};
+    // Change permutation to get a correct mapping between index types
+    std::swap(dtos.permutation[sdir], dtos.permutation[ddir]);
+    // Map smallest destination box index as an index in the source space
+    const amrex::IntVect dest_smallEnd_in_src = amrex::NonLocalBC::Apply(dtos, dest_box.smallEnd());
+    // Compute the offset to get the proper shift in the source space
+    dtos.offset =  dest_smallEnd_in_src - src_box.smallEnd();
+    // Sanity-Check that the correct box is being mapped
+    // Note, this checks index types, too!
+    AMREX_ASSERT(amrex::NonLocalBC::Image(dtos, dest_box) == src_box);
+
+    amrex::NonLocalBC::ParallelCopy(dest, dest_box, src, 0, 0, 1, amrex::IntVect{0}, dtos);
+    int fails = 0;
+    const int nx = domain_src.length(0);
+    const int ny = domain_src.length(1);
+    for (amrex::MFIter mfi(dest); mfi.isValid(); ++mfi) {
+        const amrex::Box section = dest_box & mfi.tilebox();
+        if (section.isEmpty()) continue; 
+        auto darray = dest.const_array(mfi);
+        auto sarray = src.const_array(mfi);
+        amrex::LoopOnCpu(section, [&](int i, int j, int k)
+        {
+            amrex::Dim3 si = dtos(amrex::Dim3{i,j,k});
+            AMREX_ASSERT(darray(i,j,k) == si.x + si.y*nx + si.z*nx*ny);
+        });
+    }
+    return fails == 0;
+}
+
+int MyMain()
+{
+    using namespace amrex;
+    Box domain{IntVect{}, IntVect{AMREX_D_DECL(31, 31, 31)}};
+    // Loop over all index types
+    for (int i = 0; i < AMREX_D_TERM(2,*2,*2); ++i) {
+        const auto ix = static_cast<IndexType::CellIndex>(static_cast<bool>(i & 0b001));
+        const auto iy = static_cast<IndexType::CellIndex>(static_cast<bool>(i & 0b010));
+        const auto iz = static_cast<IndexType::CellIndex>(static_cast<bool>(i & 0b100));
+        IndexType itype{AMREX_D_DECL(ix, iy, iz)};
+        Box converted_domain = convert(domain, itype);
+        iMultiFab mf = InitializeMultiFab(converted_domain);
+        ParallelCopyWithItselfIsCorrect(mf, converted_domain);
+    }
+    // Loop over all combinations of face orientations
+    const IntVect AMREX_D_DECL(e_x = IntVect::TheDimensionVector(0), e_y = IntVect::TheDimensionVector(1), e_z = IntVect::TheDimensionVector(2));
+    IndexType dirs[AMREX_SPACEDIM] = {AMREX_D_DECL(IndexType(e_x), IndexType(e_y), IndexType(e_z))};
+    Orientation::Side sides[2] = {Orientation::low, Orientation::high};
+    for (int ii = 0; ii < 2*AMREX_SPACEDIM; ++ii) {
+        int i = ii / 2;
+        Orientation::Side iside = sides[ii % 2];
+        for (int jj = 0; jj < 2*AMREX_SPACEDIM; ++jj) {
+            int j = jj / 2;
+            Orientation::Side jside = sides[j % 2];
+            Box converted_domain_x = convert(domain, dirs[i]);
+            iMultiFab dest = InitializeMultiFab(converted_domain_x);
+            Box converted_domain_y = convert(domain, dirs[j]);
+            const iMultiFab src = InitializeMultiFab(converted_domain_y);
+            ParallelCopyFaceToFace(dest, converted_domain_x, iside, src, converted_domain_y, jside);
+        }
+    }
+    return 0;
+}
\ No newline at end of file