Repository: go
Issue: runtime: "fatal: systemstack called from unexpected goroutine" on armv6
Labels: NeedsInvestigation,compiler/runtime

<!--
Please answer these questions before submitting your issue. Thanks!
For questions please use one of our forums: https://github.com/golang/go/wiki/Questions
-->

Note: I am using karalabe's xgo for compiling to ARM6

### What version of Go are you using (`go version`)?

<pre>
$ go version
go version go1.12.17 linux/amd64
</pre>

### Does this issue reproduce with the latest release?

Yes, 1.12.17 and 1.14.0 both seem affected.

### What did you do?

Cross-compile application with github.com/mattn/go-sqlite3 as only import to armv6, with CGO_ENABLED=1

Code:
```golang
package main

import _ "github.com/mattn/go-sqlite3"

func main() {
	println("Hello")
}
```

Longer story is that my host OS is macOS, so I am using karalabe/go-1.12.12 by default, as target. However, I have also tested this on [go-1.12.17](https://github.com/chenrui333/xgo/blob/go-1.14/docker/go-1.12.17/Dockerfile) (`cd docker/go-1.12.17 && docker build -t karalabe/xgo-1.12.17`) and the latest, 1.14.0 (`cd docker/go-1.14.0 && docker build -t karalabe/xgo-1.14.0`), but no difference.

I am able to get Segfault on three occasions:
1. use code without gomod, output of built executable will be `Segmentation fault`
2. use code with gomod (only `go get github.com/mattn/go-sqlite3` as package), output of executable will be `Segmentation fault`
3. use code with gomod (only `go get github.com/mattn/go-sqlite3@v1.11.0` as package), output of executable will be `fatal: systemstack called from unexpected goroutineSegmentation fault`

## Building

### Test environment
Raspberry Pi Zero (not W). Building through cross-compilation because of no internet.
```
pi@raspberrypi:~ $ cat /proc/cpuinfo
processor	: 0
model name	: ARMv6-compatible processor rev 7 (v6l)
BogoMIPS	: 997.08
Features	: half thumb fastmult vfp edsp java tls
CPU implementer	: 0x41
CPU architecture: 7
CPU variant	: 0x0
CPU part	: 0xb76
CPU revision	: 7

Hardware	: BCM2835
Revision	: 900093
Serial		: 0000000088a33a5e

pi@raspberrypi:~ $ uname -a
Linux raspberrypi 4.14.98+ #1200 Tue Feb 12 20:11:02 GMT 2019 armv6l GNU/Linux
```


### Not using gomod

Build code:
```
$ mkdir diamondo25 && cd diamondo25
# write the main.go file from the code above
$ nano main.go
$ docker run -it -v $PWD:/code/ -w /code/ --entrypoint bash karalabe/xgo-1.12.17  
root@91db62d6dde7:/code# export CC=arm-linux-gnueabihf-gcc-5 GOOS=linux GOARCH=arm GOARM=6 CGO_ENABLED=1

root@91db62d6dde7:/code# CGO_CFLAGS='-march=armv6' CGO_CXXFLAGS='-march=armv6' go install std

root@91db62d6dde7:/code# go get github.com/mattn/go-sqlite3

root@91db62d6dde7:/code# go build -o hello .

root@91db62d6dde7:/code# exit
```
There's now a `hello` executable in the 'diamondo25' folder.

Copy the binary to a armv6l box. Running the executable SEGFAULTs right away:
```
pi@raspberrypi:~ $ ./hello
Segmentation fault

pi@raspberrypi:~ $ file hello
hello: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-armhf.so.3, for GNU/Linux 3.2.0, BuildID[sha1]=c60aae380bfbd3f2292bfa0a5718ecaece5cb328, not stripped
```

GDB doesn't help much:
```
pi@raspberrypi:~ $ gdb hello
GNU gdb (Raspbian 7.12-6) 7.12.0.20161007-git
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "arm-linux-gnueabihf".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from hello...done.
warning: Missing auto-load script at offset 0 in section .debug_gdb_scripts
of file /home/pi/hello.
Use `info auto-load python-scripts [REGEXP]' to list them.
(gdb) r
Starting program: /home/pi/hello
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/arm-linux-gnueabihf/libthread_db.so.1".

Program received signal SIGSEGV, Segmentation fault.
runtime.tracebackHexdump (stk=..., frame=0xbefffeb8, bad=3204447949) at /usr/local/go/src/runtime/traceback.go:992
992	/usr/local/go/src/runtime/traceback.go: No such file or directory.
(gdb) bt
#0  runtime.tracebackHexdump (stk=..., frame=0xbefffeb8, bad=3204447949) at /usr/local/go/src/runtime/traceback.go:992
#1  0x00000000 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
```

### Using gomod

```
$ mkdir diamondo25_gomod && cd diamondo25_gomod
# write the main.go file from the code above
$ nano main.go
$ docker run -it -v $PWD:/code/ -w /code/ --entrypoint bash karalabe/xgo-1.12.17
root@e35c28fde210:/code# go mod init hello
go: creating new go.mod: module hello

root@e35c28fde210:/code# go get github.com/mattn/go-sqlite3 
go: finding github.com/mattn/go-sqlite3 v2.0.3+incompatible
go: downloading github.com/mattn/go-sqlite3 v2.0.3+incompatible
go: extracting github.com/mattn/go-sqlite3 v2.0.3+incompatible

root@e35c28fde210:/code# export CC=arm-linux-gnueabihf-gcc-5 GOOS=linux GOARCH=arm GOARM=6 CGO_ENABLED=1

root@e35c28fde210:/code# CGO_CFLAGS='-march=armv6' CGO_CXXFLAGS='-march=armv6' go install std

root@e35c28fde210:/code# go build -o hello .
# Keep open for next variant
```
There's now a `hello` executable in the 'diamondo25_gomod' folder. The building of this executable took longer than the previous, no-gomod variant.

Running this also reports `Segmentation fault`

### Using gomod with go-sqlite3@v1.11.0

Using the docker container shell, change the go-sqlite3 module to v1.11.0
```
root@e35c28fde210:/code# go get github.com/mattn/go-sqlite3@v1.11.0
go: finding github.com/mattn/go-sqlite3 v1.11.0
go: downloading github.com/mattn/go-sqlite3 v1.11.0
go: extracting github.com/mattn/go-sqlite3 v1.11.0

root@c1619b650cc7:/code# go build -o hello .

```

Now, copying and running this executable will throw the `fatal: systemstack called from unexpected goroutineSegmentation fault` error:
```
pi@raspberrypi:~ $ ./hello
fatal: systemstack called from unexpected goroutineSegmentation fault
```

GDB reports different info this time:
```
pi@raspberrypi:~ $ gdb hello
GNU gdb (Raspbian 7.12-6) 7.12.0.20161007-git
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "arm-linux-gnueabihf".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from hello...
done.
warning: Missing auto-load script at offset 0 in section .debug_gdb_scripts
of file /home/pi/hello.
Use `info auto-load python-scripts [REGEXP]' to list them.
(gdb) r
Starting program: /home/pi/hello
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/arm-linux-gnueabihf/libthread_db.so.1".
fatal: systemstack called from unexpected goroutine
Program received signal SIGSEGV, Segmentation fault.
runtime.abort () at /usr/local/go/src/runtime/asm_arm.s:801
801	/usr/local/go/src/runtime/asm_arm.s: No such file or directory.
(gdb) bt
#0  runtime.abort () at /usr/local/go/src/runtime/asm_arm.s:801
#1  0x00063a48 in runtime.badsystemstack () at /usr/local/go/src/runtime/stubs.go:62
#2  0x00049748 in runtime.throw (s=...) at /usr/local/go/src/runtime/panic.go:610
#3  0x00000012 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
```

It is unclear where it really crashes, as it never reports the stack. I think this is causing the fatal exception.