Repository: go
Issue: runtime: increased memory usage in 1.23 with AzCopy
Labels: Performance,WaitingForInfo,NeedsInvestigation,BugReport

### Go version

go 1.23.1

### Output of `go env` in your module/workspace:

```shell
go env
set GO111MODULE=
set GOARCH=amd64
set GOBIN=
set GOCACHE=C:\Users\dphulkar\AppData\Local\go-build
set GOENV=C:\Users\dphulkar\AppData\Roaming\go\env
set GOEXE=.exe
set GOEXPERIMENT=
set GOFLAGS=
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GOINSECURE=
set GOMODCACHE=C:\Users\dphulkar\go\pkg\mod
set GONOPROXY=
set GONOSUMDB=
set GOOS=windows
set GOPATH=C:\Users\dphulkar\go
set GOPRIVATE=
set GOPROXY=https://proxy.golang.org,direct
set GOROOT=C:\Users\dphulkar\go\pkg\mod\golang.org\toolchain@v0.0.1-go1.23.1.windows-amd64
set GOSUMDB=sum.golang.org
set GOTMPDIR=
set GOTOOLCHAIN=auto
set GOTOOLDIR=C:\Users\dphulkar\go\pkg\mod\golang.org\toolchain@v0.0.1-go1.23.1.windows-amd64\pkg\tool\windows_amd64  
set GOVCS=
set GOVERSION=go1.23.1
set GODEBUG=
set GOTELEMETRY=local
set GOTELEMETRYDIR=C:\Users\dphulkar\AppData\Roaming\go\telemetry
set GCCGO=gccgo
set GOAMD64=v1
set AR=ar
set CC=gcc
set CXX=g++
set CGO_ENABLED=0
set GOMOD=C:\Users\dphulkar\azure-storage-azcopy\go.mod
set GOWORK=
set CGO_CFLAGS=-O2 -g
set CGO_CPPFLAGS=
set CGO_CXXFLAGS=-O2 -g
set CGO_FFLAGS=-O2 -g
set CGO_LDFLAGS=-O2 -g
set PKG_CONFIG=pkg-config
set GOGCCFLAGS=-m64 -fno-caret-diagnostics -Qunused-arguments -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=C:\Users\dphulkar\AppData\Local\Temp\go-build1895105759=/tmp/go-build -gno-record-gcc-switches
```

### What did you do?

- Used AzCopy version 10.27.0, built with Go version 1.23.1, to copy a dataset with the following characteristics:

    1. Total Data Size: 567 GB
    2. Total Files: 1,415,230 files

- Observed memory usage behavior during the operation.
- Customers also reported similar memory issues when using AzCopy versions built with **Go 1.23.1**: [Issue #2901](https://github.com/Azure/azure-storage-azcopy/issues/2901)
- To identify the root cause, multiple experiments were conducted:
  1. AzCopy 10.26.0 (built with Go 1.22.x) showed stable memory usage (~54% RAM).
  2. AzCopy 10.27.0 (built with Go 1.23.1) exhibited a gradual memory increase, peaking at 96%.
  3. Downgrading the Go runtime in AzCopy 10.27.x from 1.23.1 to 1.22.5/1.22.7 mitigated the issue, with memory usage stabilizing (~54%).
  4. Applying a workaround by setting runtime.MemoryProfileRate=0 in AzCopy 10.27.0 (built with Go 1.23.1) failed to resolve the problem, as memory usage still peaked at 99%.
  5. Tried setting the environment variable GODEBUG=profstackdepth=32 to limit profiling stack depth, but memory usage remained high, peaking at 97% RAM.
  6. Also experimented with lowering the profstackdepth value further, but it did not resolve the issue.

### What did you see happen?

- Memory Usage: Gradual increase, peaking at 96% RAM.
- CPU Usage: Fluctuating between 51% and 100%.
- Memory Profile: Did not stabilize, leading to high resource consumption over time.

**Observations:**

  1. Significant memory usage difference compared to prior versions (e.g., AzCopy 10.26).
  2. Downgrading the Go runtime from version 1.23.1 to 1.22.5 or 1.22.7 mitigated the issue, with memory usage stabilizing around 54%.
  3. The workaround of setting runtime.MemoryProfileRate=0 did not resolve the issue, as memory usage still reached 99% RAM.

### What did you expect to see?

- Consistent memory usage profile, similar to AzCopy 10.26 or when using Go 1.22.5/1.22.7.
- Stabilized memory usage without a gradual increase over time.