Repository: AMReX-Codes/amrex — pull request #3142 (cpp)
Title: Fix nl test

This fixes a bug introduced in PR #3060 related to building neighbor lists between particles of different types. I also made it so that CI tests would have caught this bug. 

Closes #3127 

The proposed changes:
- [x] fix a bug or incorrect behavior in AMReX
- [ ] add new capabilities to AMReX
- [ ] changes answers in the test suite to more than roundoff level
- [ ] are likely to significantly affect the results of downstream AMReX users
- [ ] include documentation in the code and/or rst files, if appropriate


Resolved issues:
title: Polydisperse neighbor search algorithm
body: ## Summary
New functionality for building the particle neighbor list with multiple grids. The multi-grid approach can provide significant speed up with polydisperse particle-laden flows.

## Additional background
Algorithm based upon: 
Shire, T., Hanley, K.J. & Stratford, K. DEM simulations of polydisperse media: efficient contact detection applied to investigate the quasi-static limit. Comp. Part. Mech. 8, 653–663 (2021). https://doi.org/10.1007/s40571-020-00361-2

## Checklist
1. Verify correctness with standard single-grid (SG) neighbor search (MFiX regtests).
2. Test speed with SG neighbor search (Perlmutter w/ canonical sim)
3. Verify new multi-grid (MG) gives same results as SG (new polydisperse test)
4. Document speed up of MG vs SG for polydisperse test

The proposed changes:
- [ ] fix a bug or incorrect behavior in AMReX
- [x] add new capabilities to AMReX
- [ ] changes answers in the test suite to more than roundoff level
- [ ] are likely to significantly affect the results of downstream AMReX users
- [ ] include documentation in the code and/or rst files, if appropriate

title: Tests/Particles/NeighborList compilation error
body: ```
../../..//Src/Particle/AMReX_NeighborList.H(310): error: a value of type "amrex::Particle<0, 1> *" cannot be assigned to an entity of type "amrex::Particle<0, 0> *"
          detected during:
            instantiation of "void amrex::NeighborList<ParticleType>::build(SrcTile &, TargetTile &, CheckPair &&, const amrex::Gpu::DeviceVector<int> &, const amrex::Gpu::D
eviceVector<amrex::GpuArray<amrex::Real, 3U>> &, const amrex::Gpu::DeviceVector<amrex::GpuArray<amrex::Real, 3U>> &, const amrex::Gpu::DeviceVector<amrex::Dim3> &, const amr
ex::Gpu::DeviceVector<amrex::Dim3> &, int, int, int *) [with ParticleType=amrex::Particle<0, 0>, SrcTile=const amrex::ParticleTile<0, 0, 0, 0, amrex::DefaultAllocator>, Targ
etTile=amrex::ParticleTile<0, 1, 0, 0, amrex::DefaultAllocator>, CheckPair=CheckPair]" 
../../..//Src/Particle/AMReX_NeighborParticlesI.H(833): here
            instantiation of "void amrex::NeighborParticleContainer<NStructReal, NStructInt, NArrayReal, NArrayInt>::buildNeighborList(CheckPair &&, OtherPCType &, amrex::Ve
ctor<std::map<std::pair<int, int>, amrex::NeighborList<OtherPCType::ParticleType>, std::less<std::pair<int, int>>, std::allocator<std::pair<const std::pair<int, int>, amrex:
:NeighborList<OtherPCType::ParticleType>>>>, std::allocator<std::map<std::pair<int, int>, amrex::NeighborList<OtherPCType::ParticleType>, std::less<std::pair<int, int>>, std
::allocator<std::pair<const std::pair<int, int>, amrex::NeighborList<OtherPCType::ParticleType>>>>>> &, __nv_bool) [with NStructReal=0, NStructInt=0, NArrayReal=0, NArrayInt
=0, CheckPair=CheckPair, OtherPCType=PCType2]" 
main.cpp(131): here

../../..//Src/Particle/AMReX_NeighborList.H(331): error: no instance of overloaded function "amrex::DenseBins<T>::build [with T=amrex::Particle<0, 0>]" matches the argument 
list
            argument types are: (const size_t, amrex::Particle<0, 1> *const, int, amrex::BinMapper)
            object type is: amrex::DenseBins<amrex::Particle<0, 0>>
          detected during:
            instantiation of "void amrex::NeighborList<ParticleType>::build(SrcTile &, TargetTile &, CheckPair &&, const amrex::Gpu::DeviceVector<int> &, const amrex::Gpu::D
eviceVector<amrex::GpuArray<amrex::Real, 3U>> &, const amrex::Gpu::DeviceVector<amrex::GpuArray<amrex::Real, 3U>> &, const amrex::Gpu::DeviceVector<amrex::Dim3> &, const amr
ex::Gpu::DeviceVector<amrex::Dim3> &, int, int, int *) [with ParticleType=amrex::Particle<0, 0>, SrcTile=const amrex::ParticleTile<0, 0, 0, 0, amrex::DefaultAllocator>, Targ
etTile=amrex::ParticleTile<0, 1, 0, 0, amrex::DefaultAllocator>, CheckPair=CheckPair]" 
../../..//Src/Particle/AMReX_NeighborParticlesI.H(833): here
            instantiation of "void amrex::NeighborParticleContainer<NStructReal, NStructInt, NArrayReal, NArrayInt>::buildNeighborList(CheckPair &&, OtherPCType &, amrex::Ve
ctor<std::map<std::pair<int, int>, amrex::NeighborList<OtherPCType::ParticleType>, std::less<std::pair<int, int>>, std::allocator<std::pair<const std::pair<int, int>, amrex:
:NeighborList<OtherPCType::ParticleType>>>>, std::allocator<std::map<std::pair<int, int>, amrex::NeighborList<OtherPCType::ParticleType>, std::less<std::pair<int, int>>, std
::allocator<std::pair<const std::pair<int, int>, amrex::NeighborList<OtherPCType::ParticleType>>>>>> &, __nv_bool) [with NStructReal=0, NStructInt=0, NArrayReal=0, NArrayInt
=0, CheckPair=CheckPair, OtherPCType=PCType2]" 
main.cpp(131): here
```

Hints:

Fix patch:
diff --git a/Src/Particle/AMReX_NeighborParticlesI.H b/Src/Particle/AMReX_NeighborParticlesI.H
index 696553aaa47..ed2e1844725 100644
--- a/Src/Particle/AMReX_NeighborParticlesI.H
+++ b/Src/Particle/AMReX_NeighborParticlesI.H
@@ -828,9 +828,9 @@ buildNeighborList (CheckPair&& check_pair, OtherPCType& other,
             dxi_v.push_back(geom.InvCellSizeArray());
             plo_v.push_back(geom.ProbLoArray());
 
-            m_neighbor_list[lev][index].build(ptile, other_ptile,
-                                              std::forward<CheckPair>(check_pair),
-                                              off_bins_v, dxi_v, plo_v, lo_v, hi_v, ng);
+            neighbor_lists[lev][index].build(ptile, other_ptile,
+                                             std::forward<CheckPair>(check_pair),
+                                             off_bins_v, dxi_v, plo_v, lo_v, hi_v, ng);
         }
     }
 }


Test patch:
diff --git a/Tests/Particles/NeighborList/CMakeLists.txt b/Tests/Particles/NeighborList/CMakeLists.txt
new file mode 100644
index 00000000000..226771586b5
--- /dev/null
+++ b/Tests/Particles/NeighborList/CMakeLists.txt
@@ -0,0 +1,7 @@
+set(_sources     main.cpp)
+set(_input_files inputs)
+
+setup_test(_sources _input_files)
+
+unset(_sources)
+unset(_input_files)
diff --git a/Tests/Particles/NeighborList/main.cpp b/Tests/Particles/NeighborList/main.cpp
index 3b06653b9f9..b7860d7f2a3 100644
--- a/Tests/Particles/NeighborList/main.cpp
+++ b/Tests/Particles/NeighborList/main.cpp
@@ -65,9 +65,9 @@ void addParticles (PCType1& pc1, PCType2& pc2)
     auto& ptile1 = pc1.DefineAndReturnParticleTile(0, 0, 0);
     {
         PType1 p;
-        p.pos(0) = 12.0;
-        p.pos(1) = 12.0;
-        p.pos(2) = 12.0;
+        AMREX_D_TERM(p.pos(0) = 12.0;,
+                     p.pos(1) = 12.0;,
+                     p.pos(2) = 12.0;)
         p.id() = PType1::NextID();
         p.cpu() = ParallelDescriptor::MyProc();
         ptile1.push_back(p);
@@ -76,9 +76,9 @@ void addParticles (PCType1& pc1, PCType2& pc2)
     auto& ptile2 = pc2.DefineAndReturnParticleTile(0, 0, 0);
     {
         PType2 p;
-        p.pos(0) = 12.0;
-        p.pos(1) = 13.0;
-        p.pos(2) = 12.0;
+        AMREX_D_TERM(p.pos(0) = 12.0;,
+                     p.pos(1) = 13.0;,
+                     p.pos(2) = 12.0;)
         p.id() = PType2::NextID();
         p.cpu() = ParallelDescriptor::MyProc();
         p.idata(0) = 1;