Repository: AMReX-Codes/amrex — pull request #3581 (cpp)
Title: BCType::ext_dir_cc

Add a new BC type `BCType::ext_dir_cc` to address the issue in #3580. For cell conservation interpolation, the boundary location for `BCType::ext_dir` is on the domain face. Applications assuming boundary location at cell centers can now use `BCType::ext_dir_cc`.

## Checklist

The proposed changes:
- [x] fix a bug or incorrect behavior in AMReX
- [ ] add new capabilities to AMReX
- [ ] changes answers in the test suite to more than roundoff level
- [ ] are likely to significantly affect the results of downstream AMReX users
- [ ] include documentation in the code and/or rst files, if appropriate


Resolved issues:
title: Location of ext_dir
body: I just looked more carefully into this. When we need to fill a fine cell from spatial interpolation and it is near the physical boundary, we may need to use the physical boundary conditions stored in coarse level ghost cells to compute slopes. The BC type bug does not affect how ghost cells outside physical boundary are filled, because it calls a user provided function that does not use the BC type we set. For Flash-X, this function is `gr_fillPhysicalBC`. What the BC type bug affects is how slopes are computed. See https://github.com/AMReX-Codes/amrex/blob/87556458d867cb9cb2f148b4bcfc920c67bc918a/Src/AmrCore/AMReX_MFInterp_C.H#L13 for an example. You may wonder whether we treat `ext_dir` and `hoextrap` differently. This is because historically BoxLib codes assumed that the boundary location of `ext_dir` and `hoextrap` is exactly on the face, not half cell outside. So this is actually a known defect for codes that assume the location is at half cell outside. Maybe we should introduce a new type called `ext_dir_cc`. So I assume some Flash-X tests have `ext_dir` BC. If so, in the interpolation function, it was treated as being at cell centered before that bug fix and now as being on the domain face.

_Originally posted by @WeiqunZhang in https://github.com/AMReX-Codes/amrex/discussions/3578#discussioncomment-7190999_

Hints:

Fix patch:
diff --git a/Docs/sphinx_documentation/source/Basics.rst b/Docs/sphinx_documentation/source/Basics.rst
index fd2d12ee0b8..1227b5f7b71 100644
--- a/Docs/sphinx_documentation/source/Basics.rst
+++ b/Docs/sphinx_documentation/source/Basics.rst
@@ -2537,12 +2537,27 @@ The basic idea behind physical boundary conditions is as follows:
 
        ext_dir
            "External Dirichlet". It is the user's responsibility to write a routine
-           to fill ghost cells (more details below).
+           to fill ghost cells (more details below). The boundary location
+           is on the domain face even when the data inside the domain are
+           cell-centered.
+
+       ext_dir_cc
+           "External Dirichlet". It is the user's responsibility to write a routine
+           to fill ghost cells (more details below). The boundary location
+           is at the cell center of ghost cells outside the domain.
 
        foextrap
            "First Order Extrapolation"
            First order extrapolation from last cell in interior.
 
+       hoextrap
+           "High Order Extrapolation". The boundary location is on the domain
+           face even when the data inside the domain are cell-centered.
+
+       hoextrapcc
+           "High Order Extrapolation" The boundary location is at the cell
+           center of ghost cells outside the domain.
+
        reflect_even
            Reflection from interior cells with sign
            unchanged, :math:`q(-i) = q(i)`.
diff --git a/Src/Amr/AMReX_StateDescriptor.cpp b/Src/Amr/AMReX_StateDescriptor.cpp
index 161090d1a7f..4f1e3c3fe1b 100644
--- a/Src/Amr/AMReX_StateDescriptor.cpp
+++ b/Src/Amr/AMReX_StateDescriptor.cpp
@@ -19,7 +19,7 @@ StateDescriptor::bf_thread_safety (const int* /*lo*/,const int* /*hi*/,
     if (!bf_ext_dir_threadsafe) {
         bool has_ext_dir = false;
         for (int i=0; i<2*AMREX_SPACEDIM*ng && !has_ext_dir; ++i) {
-            has_ext_dir = bc[i]==BCType::ext_dir;
+            has_ext_dir = ((bc[i]==BCType::ext_dir) || (bc[i]==BCType::ext_dir_cc));
         }
         if (has_ext_dir) { thread_safe = false; }
     }
diff --git a/Src/Base/AMReX_BCUtil.H b/Src/Base/AMReX_BCUtil.H
index dd4e814610c..3ebc53a0576 100644
--- a/Src/Base/AMReX_BCUtil.H
+++ b/Src/Base/AMReX_BCUtil.H
@@ -9,11 +9,12 @@
 namespace amrex
 {
     // This is for filling cell-centered data outside physical domain
-    // (excluding periodic boundaries).  It only fills
-    // BCType::foextrap, BCType::hoextrap, BCType::reflect_even, and
-    // BCType::reflect::odd.  It does not fill BCType::ext_dir (i.e.,
-    // external Dirichlet).  If you have BCType::ext_dir, you can
-    // copy, paste and rename this function, and then modify it.
+    // (excluding periodic boundaries).  It only fills BCType::foextrap,
+    // BCType::hoextrap, BCType::hoextrapcc, BCType::reflect_even, and
+    // BCType::reflect::odd.  It does not fill BCType::ext_dir and
+    // BCType::ext_dir_cc (i.e., external Dirichlet).  If you have
+    // BCType::ext_dir or BCType::ext_dir_cc, you can copy, paste and rename
+    // this function, and then modify it.
     void FillDomainBoundary (MultiFab& phi, const Geometry& geom, const Vector<BCRec>& bc);
 }
 
diff --git a/Src/Base/AMReX_BCUtil.cpp b/Src/Base/AMReX_BCUtil.cpp
index c0a645ba05c..dc5b4f4f10c 100644
--- a/Src/Base/AMReX_BCUtil.cpp
+++ b/Src/Base/AMReX_BCUtil.cpp
@@ -15,7 +15,7 @@ void dummy_cpu_fill_extdir (Box const& /*bx*/, Array4<Real> const& /*dest*/,
                             const BCRec* /*bcr*/, const int /*bcomp*/,
                             const int /*orig_comp*/)
 {
-    // do something for external Dirichlet (BCType::ext_dir) if there are
+    // do something for external Dirichlet (BCType::ext_dir or BCType::ext_dir_cc) if there are
 }
 
 struct dummy_gpu_fill_extdir
@@ -27,7 +27,7 @@ struct dummy_gpu_fill_extdir
                      const BCRec* /*bcr*/, const int /*bcomp*/,
                      const int /*orig_comp*/) const
         {
-            // do something for external Dirichlet (BCType::ext_dir) if there are
+            // do something for external Dirichlet (BCType::ext_dir or BCType::ext_dir_cc) if there are
         }
 };
 
diff --git a/Src/Base/AMReX_BC_TYPES.H b/Src/Base/AMReX_BC_TYPES.H
index 872ac1eff5d..00fadd5e6d0 100644
--- a/Src/Base/AMReX_BC_TYPES.H
+++ b/Src/Base/AMReX_BC_TYPES.H
@@ -44,10 +44,8 @@ SYMMETRY       |  Un REFLECT_ODD |  REFLECT_EVEN    | REFLECT_EVEN   |
 
 
 INT_DIR      : data taken from other grids or interpolated
-EXT_DIR      : data specified on EDGE (FACE) of bndry for the linear solvers
-               for application codes, the location of the EXT_DIR data will
-               depend on how they do reconstruction, and may be edge or
-               cell-centered.
+EXT_DIR      : data specified on EDGE (FACE)
+EXT_DIR_CC   : data specified at cell center
 HOEXTRAP     : higher order extrapolation to EDGE of bndry
 HOEXTRAPCC   : linear extrapolation to CELL of bndry
 FOEXTRAP     : first order extrapolation from last cell in interior
@@ -74,6 +72,7 @@ enum mathematicalBndryTypes : int {
     ext_dir      =  3,
     hoextrap     =  4,
     hoextrapcc   =  5,
+    ext_dir_cc   =  6,
     user_1       = 1001,
     user_2       = 1002,
     user_3       = 1003
@@ -94,6 +93,7 @@ enum mathematicalBndryTypes : int {
 #define EXT_DIR 3
 #define HOEXTRAP 4
 #define HOEXTRAPCC 5
+#define EXT_DIR_CC 6
 
 #define Interior 0
 #define Inflow 1
diff --git a/Src/Base/AMReX_FILCC_1D.F90 b/Src/Base/AMReX_FILCC_1D.F90
index 873f67f5941..76e90a8d7a1 100644
--- a/Src/Base/AMReX_FILCC_1D.F90
+++ b/Src/Base/AMReX_FILCC_1D.F90
@@ -14,7 +14,7 @@
 !!                        corner of q array
 !! \param bc           => array of boundary flags bc(SPACEDIM,lo:hi)
 !!
-!! NOTE: all corner as well as edge data is filled if not EXT_DIR
+!! NOTE: all corner as well as edge data is filled if not EXT_DIR/EXT_DIR_CC
 ! -----------------------------------------------------------
 
 #ifndef AMREX_XSDK
diff --git a/Src/Base/AMReX_FILCC_2D.F90 b/Src/Base/AMReX_FILCC_2D.F90
index 89ef77d8384..c2ca8f7ddaf 100644
--- a/Src/Base/AMReX_FILCC_2D.F90
+++ b/Src/Base/AMReX_FILCC_2D.F90
@@ -16,7 +16,7 @@
 !!                        corner of q array
 !! \param bc           => array of boundary flags bc(SPACEDIM,lo:hi)
 !!
-!! NOTE: all corner as well as edge data is filled if not EXT_DIR
+!! NOTE: all corner as well as edge data is filled if not EXT_DIR/EXT_DIR_CC
 ! -----------------------------------------------------------
 
 subroutine filcc(q,q_l1,q_l2,q_h1,q_h2,domlo,domhi,dx,xlo,bc)
diff --git a/Src/Base/AMReX_FILCC_3D.F90 b/Src/Base/AMReX_FILCC_3D.F90
index aa3fec74ab0..59ce83d469d 100644
--- a/Src/Base/AMReX_FILCC_3D.F90
+++ b/Src/Base/AMReX_FILCC_3D.F90
@@ -6,7 +6,7 @@
 !> This routine is intended to be a generic fill function
 !! for cell centered data.  It knows how to exrapolate,
 !! and reflect data and can be used to supplement problem
-!! specific fill functions (ie. EXT_DIR).
+!! specific fill functions (ie. EXT_DIR/EXT_DIR_CC).
 !!
 !! \param q        <=  array to fill
 !! \param q_l1,q_l2,q_l3,q_h1,q_h2,q_h3   => index extent of q array
diff --git a/Src/Base/AMReX_FilCC_C.cpp b/Src/Base/AMReX_FilCC_C.cpp
index 7cdba486957..e2d8c6129fd 100644
--- a/Src/Base/AMReX_FilCC_C.cpp
+++ b/Src/Base/AMReX_FilCC_C.cpp
@@ -41,7 +41,7 @@ void fab_filcc (Box const& bx, Array4<Real> const& qn, int ncomp,
         if (lo.x < ilo) {
            const int imin = lo.x;
            const int imax = ilo-1;
-           if (bc.lo(0) == BCType::ext_dir) {
+           if (bc.lo(0) == BCType::ext_dir || bc.lo(0) == BCType::ext_dir_cc) {
                // Do nothing.
            } else if (bc.lo(0) == BCType::foextrap) {
                for (int k = lo.z; k <= hi.z; ++k) {
@@ -88,7 +88,7 @@ void fab_filcc (Box const& bx, Array4<Real> const& qn, int ncomp,
             const int imin = ihi+1;
             const int imax = hi.x;
 
-            if (bc.hi(0) == BCType::ext_dir) {
+            if (bc.hi(0) == BCType::ext_dir || bc.hi(0) == BCType::ext_dir_cc) {
                 // Do nothing.
             } else if (bc.hi(0) == BCType::foextrap) {
                 for (int k = lo.z; k <= hi.z; ++k) {
@@ -136,7 +136,7 @@ void fab_filcc (Box const& bx, Array4<Real> const& qn, int ncomp,
         if (lo.y < jlo) {
             const int jmin = lo.y;
             const int jmax = jlo-1;
-            if (bc.lo(1) == BCType::ext_dir) {
+            if (bc.lo(1) == BCType::ext_dir || bc.lo(1) == BCType::ext_dir_cc) {
                 // Do nothing.
             } else if (bc.lo(1) == BCType::foextrap) {
                 for (int k = lo.z; k <= hi.z; ++k) {
@@ -182,7 +182,7 @@ void fab_filcc (Box const& bx, Array4<Real> const& qn, int ncomp,
         if (hi.y > jhi) {
             const int jmin = jhi+1;
             const int jmax = hi.y;
-            if (bc.hi(1) == BCType::ext_dir) {
+            if (bc.hi(1) == BCType::ext_dir || bc.hi(1) == BCType::ext_dir_cc) {
                 // Do nothing.
             } else if (bc.hi(1) == BCType::foextrap) {
                 for (int k = lo.z; k <= hi.z; ++k) {
@@ -231,7 +231,7 @@ void fab_filcc (Box const& bx, Array4<Real> const& qn, int ncomp,
         if (lo.z < klo) {
             const int kmin = lo.z;
             const int kmax = klo-1;
-            if (bc.lo(2) == BCType::ext_dir) {
+            if (bc.lo(2) == BCType::ext_dir || bc.lo(2) == BCType::ext_dir_cc) {
                 // Do nothing.
             } else if (bc.lo(2) == BCType::foextrap) {
                 for (int k = kmin; k <= kmax; ++k) {
@@ -277,7 +277,7 @@ void fab_filcc (Box const& bx, Array4<Real> const& qn, int ncomp,
         if (hi.z > khi) {
             const int kmin = khi+1;
             const int kmax = hi.z;
-            if (bc.hi(2) == BCType::ext_dir) {
+            if (bc.hi(2) == BCType::ext_dir || bc.hi(2) == BCType::ext_dir_cc) {
                 // Do nothing.
             } else if (bc.hi(2) == BCType::foextrap) {
                 for (int k = kmin; k <= kmax; ++k) {
diff --git a/Src/Base/AMReX_bc_types.fi b/Src/Base/AMReX_bc_types.fi
index 2f3340ccfc4..d929b589b3d 100644
--- a/Src/Base/AMReX_bc_types.fi
+++ b/Src/Base/AMReX_bc_types.fi
@@ -17,3 +17,5 @@
       PARAMETER (HOEXTRAP=4)
       INTEGER HOEXTRAPCC
       PARAMETER (HOEXTRAPCC=5)
+      INTEGER EXT_DIR_CC
+      PARAMETER (EXT_DIR_CC=6)
diff --git a/Src/Base/AMReX_bc_types_mod.F90 b/Src/Base/AMReX_bc_types_mod.F90
index c1c6f237ba8..5b8c58c255a 100644
--- a/Src/Base/AMReX_bc_types_mod.F90
+++ b/Src/Base/AMReX_bc_types_mod.F90
@@ -15,6 +15,7 @@ module amrex_bc_types_module
   integer, parameter, public :: amrex_bc_ext_dir      =  3
   integer, parameter, public :: amrex_bc_hoextrap     =  4
   integer, parameter, public :: amrex_bc_hoextrapcc   =  5
+  integer, parameter, public :: amrex_bc_ext_dir_cc   =  6
   integer, parameter, public :: amrex_bc_user_1       =  1001
   integer, parameter, public :: amrex_bc_user_2       =  1002
   integer, parameter, public :: amrex_bc_user_3       =  1003
diff --git a/Src/Base/AMReX_filcc_mod.F90 b/Src/Base/AMReX_filcc_mod.F90
index c1a2b2951f4..1cca9b84090 100644
--- a/Src/Base/AMReX_filcc_mod.F90
+++ b/Src/Base/AMReX_filcc_mod.F90
@@ -165,7 +165,7 @@ subroutine amrex_filccn(lo, hi, q, q_lo, q_hi, ncomp, domlo, domhi, dx, xlo, bc)
           imin = lo(1)
           imax = ilo-1
 
-          if (bc(1,1,n) .eq. amrex_bc_ext_dir) then
+          if (bc(1,1,n) .eq. amrex_bc_ext_dir .or. bc(1,1,n) .eq. amrex_bc_ext_dir_cc) then
 
              ! Do nothing.
 
@@ -237,7 +237,7 @@ subroutine amrex_filccn(lo, hi, q, q_lo, q_hi, ncomp, domlo, domhi, dx, xlo, bc)
           imin = ihi+1
           imax = hi(1)
 
-          if (bc(1,2,n) .eq. amrex_bc_ext_dir) then
+          if (bc(1,2,n) .eq. amrex_bc_ext_dir .or. bc(1,2,n) .eq. amrex_bc_ext_dir_cc) then
 
              ! Do nothing.
 
@@ -311,7 +311,7 @@ subroutine amrex_filccn(lo, hi, q, q_lo, q_hi, ncomp, domlo, domhi, dx, xlo, bc)
           jmin = lo(2)
           jmax = jlo-1
 
-          if (bc(2,1,n) .eq. amrex_bc_ext_dir) then
+          if (bc(2,1,n) .eq. amrex_bc_ext_dir .or. bc(2,1,n) .eq. amrex_bc_ext_dir_cc) then
 
              ! Do nothing.
 
@@ -383,7 +383,7 @@ subroutine amrex_filccn(lo, hi, q, q_lo, q_hi, ncomp, domlo, domhi, dx, xlo, bc)
           jmin = jhi+1
           jmax = hi(2)
 
-          if (bc(2,2,n) .eq. amrex_bc_ext_dir) then
+          if (bc(2,2,n) .eq. amrex_bc_ext_dir .or. bc(2,2,n) .eq. amrex_bc_ext_dir_cc) then
 
              ! Do nothing.
 
@@ -461,7 +461,7 @@ subroutine amrex_filccn(lo, hi, q, q_lo, q_hi, ncomp, domlo, domhi, dx, xlo, bc)
           kmin = lo(3)
           kmax = klo-1
 
-          if (bc(3,1,n) .eq. amrex_bc_ext_dir) then
+          if (bc(3,1,n) .eq. amrex_bc_ext_dir .or. bc(3,1,n) .eq. amrex_bc_ext_dir_cc) then
 
              ! Do nothing.
 
@@ -533,7 +533,7 @@ subroutine amrex_filccn(lo, hi, q, q_lo, q_hi, ncomp, domlo, domhi, dx, xlo, bc)
           kmin = khi+1
           kmax = hi(3)
 
-          if (bc(3,2,n) .eq. amrex_bc_ext_dir) then
+          if (bc(3,2,n) .eq. amrex_bc_ext_dir .or. bc(3,2,n) .eq. amrex_bc_ext_dir_cc) then
 
              ! Do nothing.
 


Test patch:
diff --git a/Tests/FortranInterface/Advection_F/Source/fillpatch_mod.F90 b/Tests/FortranInterface/Advection_F/Source/fillpatch_mod.F90
index f337de6e7ff..c79f2aa7b9a 100644
--- a/Tests/FortranInterface/Advection_F/Source/fillpatch_mod.F90
+++ b/Tests/FortranInterface/Advection_F/Source/fillpatch_mod.F90
@@ -90,7 +90,7 @@ subroutine fill_physbc (pmf, scomp, ncomp, time, pgeom) bind(c)
                   geom%get_physical_location(plo), & ! physical location of lower left corner
                   lo_bc, hi_bc)                      ! bc types for each component
 
-             ! amrex_filcc doesn't fill EXT_DIR (see amrex_bc_types_module for a list of bc types
+             ! amrex_filcc doesn't fill EXT_DIR/EXT_DIR_CC (see amrex_bc_types_module for a list of bc types
              ! In that case, the user needs to fill it.
           end if
        end do
diff --git a/Tests/GPU/CNS/Source/CNS_bcfill.cpp b/Tests/GPU/CNS/Source/CNS_bcfill.cpp
index 1c787e29a96..a75c7ebce88 100644
--- a/Tests/GPU/CNS/Source/CNS_bcfill.cpp
+++ b/Tests/GPU/CNS/Source/CNS_bcfill.cpp
@@ -14,7 +14,7 @@ struct CnsFillExtDir
                      const BCRec* /*bcr*/, const int /*bcomp*/,
                      const int /*orig_comp*/) const
         {
-            // do something for external Dirichlet (BCType::ext_dir)
+            // do something for external Dirichlet (BCType::ext_dir/BCType::ext_dir_cc)
         }
 };