edit well with boundary conditions g will be more complicated but for example with neumann boundary conditions it is clear to see that the remaining surface integral should not be negative by comparing with the wikipedia page for helmholtz decomposition it should have the opposite sign the other part of the surface integral should also be the negative of what is written