x64 Assembly: Multithreading from Scratch Part 2: Threads
Tags: tutorial, multithreading, assembly
Published: 2023-03-16 23:36:02

In [Part 1](https://dev.to/ahmedwadod/x64-assembly-multithreading-from-scratch-part-1-hello-world-59p3) of this series we have looked into how to use `fork` system call to spawn a child process and branch out on the child process to do a different code path. In this part we will look into how we can implement a more functional way to do multithreading in assembly.

## Goal
We want to use any function/subroutine in our assembly code as a thread, meaning that we want a way to call a function and this function will run on a different thread, then we need a way to wait (on the parent process) for the threads we create.

## Implementation
We will create a file `lib/thread.asm` that will have two functions:
- `thread_run`: It will take a pointer to the code we want to execute (stored in `rax`) and then it will pass along the registers `rdi`, `rsi`, `rdx` and `r8` unchanged to the function now running on the child thread it will create. This function will call `fork` and then on the child thread it will jump to the code pointed at by `rax`, on the parent it will return the PID of the child process in `rax`.

- `thread_wait`: This function takes a single argument in `rax` which is a PID of a child process, it will wait for that child process to finish executing then it returns the exit code of that child process in `rax`.

### `thread_run`
```
# lib/thread.asm

.intel_syntax
.section .text

# thread_run
# A function to start running the given code on a child process
#
# Arguments:
# 	rax: Pointer to function(label)
# 	rdi, rsi, rdx, rcx, r8 : The arguments to the function
#
# Returns:
# 	rax: Child process id
#
.global thread_run
thread_run:
	push %rax # Push rax value to the stack

	# Fork this process
	mov %rax, 0x39
	syscall
	
	cmp %rax, 0x00 # Compare the returned PID
	mov %r10, %rax # Store the PID in r10 temporarily
	pop %rax # Get back rax value (Function pointer)
	je _invoke # Jump if this is the child process

	# If this is the parent process then return 
	mov %rax, %r10 # Restore the PID
	ret

_invoke:
	# Jump to the pointer of the function
	call %rax

	# Safe exit
	mov %rax, 0x3c
	mov %rdi, 0x00
	syscall

```

In the above code, the function starts by storing (pushing) the pointer that is in `rax` to the stack because we need the `rax` register to make a `fork` system call, then we compare the return of `fork` to zero and store the PID (the current value of `rax`) in `r10` register temporarly, and put back the function pointer in `rax` register and jump conditionally to the `_invoke` function/subroutine if the value of the earliy comparison is correct, other-wise we restore the PID in `rax` and return. `_invoke` subroutine will call the function in the address stored in `rax` and then exit safetly with the `exit` syscall.

### `thread_wait`
```
# lib/thread.asm

...

# thread_wait
# Wait for a process to exit
#
# Arguments: 
# 	rax: Process PID
#
# Returns:
# 	rax: Exit code
#
.global thread_wait
thread_wait:
	# Allocate space in the stack to store the exit code
	sub %rsp, 4
	
	mov %rdi, %rax # Move the PID to %rdi for wait4 syscall
	mov %rax, 0x3d # wait4 syscall
	mov %rsi, %rsp # Pointer to the exit code space
	mov %rdx, 0x00 # Flags
	mov %r10, 0x00 # NULL pointer
	syscall

	# Store the exit code in EAX (4 byte value of rax)
	mov %eax, [%rsp]
	add %rsp, 4

	ret

```
In the above code we start by allocating 4 bytes of memory on the stack, then we call the [wait4](https://man7.org/linux/man-pages/man2/wait4.2.html) system call, we basically give it the PID of the child process in `rdi` and a pointer to 4 bytes in memory to store the exit value of the child process that exited (We give it the 4 bytes we just allocated on the stack). After `wait4` syscall returns, we get the return value from the stack and deallocate that memory from the stack.

NOTE: The last step could be achived using `pop DWORD %rax` but for some reason using `gas` it didn't want to assemble, but I assume it will work fine on `MASM` or another assembler.

### Assembly
Now we will assemble this code the same way we did with `util.asm` using the following commad:
```
$ as lib/thread.asm -o thread.o
```
If eveything works fine we will now have a `thread.o` file that we will link our code with.


## Driver Code
To demonstrate how these two functions work, we will create a new directory `hello_threads` in the root of our project and put a `main.asm` file inside of it.

In this file we will write a program that will execute a thread to run the `print` function we wrote in `util.asm` last time and wait for the thread to finish executing, and then print from the main (parent) process.
Then the program will execute another thread with the `print` function and continue running the main (parent) process without waiting.
```
# hello_threads/main.asm

.global _start
.intel_syntax
.section .text

# The external functions we need
.extern thread_run
.extern thread_wait
.extern print

_start:
	# Create a thread
	call _log
	lea %rax, [%rip + print]
	lea %rdi, [%rip + thread_msg]
	mov %rsi, OFFSET thread_msg_len
	call thread_run

	# Wait for the process (PID is in rax) then print
	call thread_wait
	call _pmsg

	# Create a thread
	call _log
	lea %rax, [%rip + print]
	lea %rdi, [%rip + thread2_msg]
	mov %rsi, OFFSET thread2_msg_len
	call thread_run

	# Print without waiting
	call _pmsg
	
	# Exit
	jmp _exit


_pmsg:
	# Print 'parent_msg'
	lea %rdi, [%rip + parent_msg]
	mov %rsi, OFFSET parent_msg_len
	call print
	ret

_log:
	# Print 'log_msg'
	lea %rdi, [%rip + log_msg]
	mov %rsi, OFFSET log_msg_len
	call print
	ret

_exit:
	# Exit safetly with status code 0
	mov %rax, 0x3c
	mov %rdi, 0x00
	syscall

.section .data
	log_msg:
		.ascii "\nCreating a thread..\n" # 20
		log_msg_len = . - log_msg
	thread_msg:
		.ascii "Hello from thread 1!\n" #19
		thread_msg_len = . - thread_msg
	thread2_msg:
		.ascii "Hello from thread 2!\n" #19
		thread2_msg_len = . - thread2_msg
	parent_msg:
		.ascii "Hello from parent!!\n" #20
		parent_msg_len = . - parent_msg

```

Now we assemble our code and link the resulting object file with both `lib/util.o` and `lib/thread.o` with the following commads:
```
$ as hello_threads/main.asm -o hello_threads/hello_threads.o
$ ld hello_threads/hello_threads.o lib/util.o lib/thread.o -o hello_threads/hello_threads.elf -nostdlib
```

If everything goes well, we should now have a `hello_threads.elf` in `hello_threads` directory, if we execute the file we should see an output similar to this:
```
Creating a thread..
Hello from thread 1!
Hello from parent!!

Creating a thread..
Hello from parent!!
Hello from thread 2!
```

Notice how the first thread executes before the main (parent) thread due to us using the `thread_wait` function that we defined earlier. But in the second theard case it is different, the parent executes first -for most of the time- since it does not have to wait for the thread to finish executing.

Now that we have a functional way to spwan and wait for threads we will need a way for threads to communicate and share data with each other, so in the next part we will look into how we can allocate and share memory between threads. 

The code is available on the Github [repository](https://github.com/ahmedwadod/x64_asm_multithreading_from_scratch) if you want to check it out.