Linux Process Memory
Tags: linux, bash
Published: 2020-05-05 22:50:47

#Process Inspection
Last week I was doing (remote) computer security training with some of my colleagues from work. One of the lab workshops on Windows introduced us to a tool called [ProcessHacker](https://processhacker.sourceforge.io/). This uses the Windows API to give you finer grain control of Windows processes than is available with the tools provided by default.

One of the features we looked at was the ability to inspect the memory of another process. This uses the [ReadProcessMemory API function](https://docs.microsoft.com/en-us/windows/win32/api/memoryapi/nf-memoryapi-readprocessmemory) and is especially useful if you are looking at a process that you suspect should not be on your system.

However, the majority of our work is done using Linux systems, and so one of my colleagues wondered if it was possible to inspect memory from a Linux process in a similar way.

This reminded me of the information available in the [`/proc`](https://www.tldp.org/LDP/Linux-Filesystem-Hierarchy/html/proc.html) filesystem on Linux. I learned about this filesystem many years ago while learning to write kernel modules, and was really intrigued at how so much process instrumentation was made available in userspace using the standard file metaphor. This is why I like *nix style systems so much.

Given what is available under `/proc` I figured that I should be able to give my colleague what he was looking for. Once I was done, I thought it might be a useful exercise to document.

##/proc/${id}/maps
The first step in this was to pick the process that the admin on a system might want to inspect. If someone were trying to do something nefarious, then the first thing to look for might be the memory for a shell. Shells like `bash` will save their history to a file when they exit, but a user could always try something like:
`nohup after 5000 rm ~/.bash_sessions/*`
just before logging out. So as an admin, you may want to see what a suspicious user is doing right now, and not after they log out.

####Suspect User
So that I'd have something to look for, I used `su - testuser` to get into a user account, and typed the command: `sudo cp /bin/su /tmp/echo`
This gave me a bash session belonging to `testuser` with something I shouldn't see in the history.

Finding a user's process is done with the `ps` command. I first learned this command with BSD-style options, and because Linux supports these I never really looked at the others. But there are lots of options and lots of output format. Try a `man ps` and you'll see lots of details.

By default, the `ps` command shows the processes attached to the current terminal. This is usually just your current shell, and the `ps` command itself. You can ask to see all the commands for the current user, regardless of the terminal, by adding `u` to this command. You can also ask to see those commands that are not attached to any terminal by adding `x`. But the option we really want here is the ability to see commands that belong to all other users, which is `a`. Since I can usually find what I'm looking for with that information, I default to typing `ps aux`, and then I will look up a man page only if I need something else.

From the session belonging to `testuser` I typed `ps` and got:
```
  PID TTY          TIME CMD
10852 pts/0    00:00:00 bash
11874 pts/0    00:00:00 ps
```
So the process ID (PID) that I'm looking for is 10852. Back at a root prompt I went with my standard `ps aux` and got a very long listing, which included:
```
root     10851  0.0  0.5  50708  2944 pts/0    S    04:54   0:00 su - testuser
testuser 10852  0.0  1.0  23360  5364 pts/0    S+   04:54   0:00 -su
```
That's me executing the `su - testuser` to log in as that user, and the `-su` is the label I'm seeing for the bash shell that `su` started for me. There are various ways that a user can get to a shell (such as through ssh), but for now I've found the PID, and can work from there.

So what does the system have on that process?
```
$ cd /proc/10852
$ ls
attr             cpuset   limits      net            projid_map  stat
autogroup        cwd      loginuid    ns             root        statm
auxv             environ  map_files   numa_maps      sched       status
cgroup           exe      maps        oom_adj        schedstat   syscall
clear_refs       fd       mem         oom_score      sessionid   task
cmdline          fdinfo   mountinfo   oom_score_adj  setgroups   timers
comm             gid_map  mounts      pagemap        smaps       uid_map
coredump_filter  io       mountstats  personality    stack       wchan
```
There are lots of interesting things in here, and I encourage people to have a look at the [docs for /proc](https://www.tldp.org/LDP/Linux-Filesystem-Hierarchy/html/proc.html) for more details. But the data I'm looking for is in the process memory.

##mem
The `mem` file actually represents the process memory. If we look at the entry more carefully, we see:
```
$ ls -l mem
-rw------- 1 testuser testuser 0 May  1 15:25 mem
```
So it appears to be a file with no size that belongs to the user who owns the process. It is also only accessible to the process owner, but as root we can inspect it as well.

The file appears to have no size, but that is because it is virtual, and will access the memory as required. However, we can't just access this memory blindly, as it represents addresses that may not be used by the process. To find where data is, we need a map.

##maps
The `maps` virtual file is world readable:
```
-r--r--r-- 1 testuser testuser 0 May  1 15:25 maps
```
This file provides instrumentation on how memory is mapped into the process:
```
$ cat maps
...lots of data...
```
There is a lot of data here, but what does all this mean? It has a lot to do with how operating systems handle process memory.

###Memory Allocation
When a Linux process requires memory, it will use a library function like `malloc(3)`. This typically asks the operating system for more memory by using the `brk` system call, which it then manages locally. The library function gives the user code the memory it asked for, and then reclaims it for reuse when the memory is no longer needed.

However, memory obtained with `brk` call is permanently associated with the process, which may not be desired, such as when temporarily handling large buffers. When `malloc` is asked for very large blocks of data, it instead uses the `mmap(2)` function. To explain this call, we should look at virtual memory.

#### Virtual Memory
Back in the days of early personal computing, if a computer program referred to a memory address, such as 0x800 (2048), then this was a precise location in a particular memory chip of the computer. As computing became more powerful and applications demanded more, then it became possible for programs to require more memory than was available on the hardware, and for computers to run more than one program at a time. To manage this _virtual memory_ was created. This uses a combination of the hardware and the operating system to convert arbitrary memory addresses into a physical address that holds the required data.

There are several benefits to this. The first is that data can be moved from physical RAM and onto external media (usually a disk). Then, when the data is needed again, it can be re-loaded back into memory and the address translation will now refer to this new location. As a result, the computer can use some of the disk to behave as if it has more memory than is physically present (albeit, at the cost of speed). And because this translation is done on a process-by-process basis, it also means that processes can use the same addresses to refer to their data, without the risk of accessing the data from another process. This means that one process on a computer is unable to interfere with the operation of another process... at least, not without the intervention of the operating system.

Data gets moved around by an entire "page" at a time. This can be varied, but on Intel hardware the default is 4KiB. (4096 bytes).

Consequently, all memory addresses that a process sees are run through this translation process. If a process refers to an address, then one of 3 things can happen:
- The address is translated through tables to refer to a page located in physical memory. The data at the offset in the page is accessed.
- The address is translated through tables and the page is found to have been "swapped out". Everything pauses for a moment, while the OS reads that page from disk into an available page in physical memory, and an entry for that page is placed into the address translation tables. The data at the offset in the page is then accessed.
- The address is not found in the tables. This is an error.

With this mechanism, the process can act as if it can access any memory from 0 to 2<sup>32</sup> or 2<sup>64</sup> (depending on the hardware and operating system architecture). Of course, only those addresses that have been correctly set up (e.g. by allocating memory) will work.

Most of this is hidden from users, but it can be interacted with using [`mmap`](http://man7.org/linux/man-pages/man2/mmap.2.html) on Linux, or [`CreateFileMapping`](https://docs.microsoft.com/en-us/windows/win32/api/winbase/nf-winbase-createfilemappinga) or [`VirtualAlloc`](https://docs.microsoft.com/en-us/windows/win32/api/memoryapi/nf-memoryapi-virtualalloc) on Windows. Both `mmap` and `CreateFileMapping` allow you to specify a _file descriptor_ (or _file handle_ for Windows) which permits the OS to allocate virtual memory for you to use, where the locations on disk are provided by the file associated with the handle. Alternatively, on Linux `mmap` can specify that it is not using a file descriptor, which means that the allocated memory will be backed by the swap file of the operating system. Windows provides similar functionality by calling `CreateFileMapping` using a parameter of `INVALID_HANDLE_VALUE_`, or just by calling `VirtualAlloc`.

####System Pages
Using `mmap` or `CreateFileMapping` is how large blocks of memory can be set up by the process (possibly referencing files), but what about access to memory that contains library functions? From the perspective of the process, all library functions (including basic i/o functionality like printing text or file access) must appear within its address space.

This is handled in a very similar way to when the process calls `mmap` or `CreateFileMapping`. In this case, any libraries that the process needs will be mapped into memory and added to the virtual tables for the process. Most of the central libraries will already be in the system due to being used by other processes, so the only work that needs to be done here is to ensure that their addresses into the virtual memory tables for that process. For many libraries, this initialization is done at process launch. Indeed, the process itself gets launched by mem-mapping the executable file, and then the primary thread is pointed at the initialization address within those pages.

####Process Memory Map
Once a process is mapped in, all of its libraries are mapped in, and any additional memory requests have been fulfilled, then various address ranges will represent all of these different parts.

Let's look at the output again...
```
00400000-004f4000 r-xp 00000000 08:01 393218 /bin/bash
006f3000-006f4000 r--p 000f3000 08:01 393218 /bin/bash
006f4000-006fd000 rw-p 000f4000 08:01 393218 /bin/bash
006fd000-00703000 rw-p 00000000 00:00 0
01b48000-01d16000 rw-p 00000000 00:00 0 [heap]
7f40d6af8000-7f40d6b03000 r-xp 00000000 08:01 393362 /lib/x86_64-linux-gnu/libnss_files-2.21.so
7f40d6b03000-7f40d6d02000 ---p 0000b000 08:01 393362 /lib/x86_64-linux-gnu/libnss_files-2.21.so
7f40d6d02000-7f40d6d03000 r--p 0000a000 08:01 393362 /lib/x86_64-linux-gnu/libnss_files-2.21.so
7f40d6d03000-7f40d6d04000 rw-p 0000b000 08:01 393362 /lib/x86_64-linux-gnu/libnss_files-2.21.so
7f40d6d04000-7f40d6d0e000 r-xp 00000000 08:01 393369 /lib/x86_64-linux-gnu/libnss_nis-2.21.so
7f40d6d0e000-7f40d6f0e000 ---p 0000a000 08:01 393369 /lib/x86_64-linux-gnu/libnss_nis-2.21.so
7f40d6f0e000-7f40d6f0f000 r--p 0000a000 08:01 393369 /lib/x86_64-linux-gnu/libnss_nis-2.21.so
7f40d6f0f000-7f40d6f10000 rw-p 0000b000 08:01 393369 /lib/x86_64-linux-gnu/libnss_nis-2.21.so
7f40d6f10000-7f40d6f25000 r-xp 00000000 08:01 393345 /lib/x86_64-linux-gnu/libnsl-2.21.so
7f40d6f25000-7f40d7124000 ---p 00015000 08:01 393345 /lib/x86_64-linux-gnu/libnsl-2.21.so
7f40d7124000-7f40d7125000 r--p 00014000 08:01 393345 /lib/x86_64-linux-gnu/libnsl-2.21.so
7f40d7125000-7f40d7126000 rw-p 00015000 08:01 393345 /lib/x86_64-linux-gnu/libnsl-2.21.so
7f40d7126000-7f40d7128000 rw-p 00000000 00:00 0
7f40d7128000-7f40d712f000 r-xp 00000000 08:01 393347 /lib/x86_64-linux-gnu/libnss_compat-2.21.so
7f40d712f000-7f40d732e000 ---p 00007000 08:01 393347 /lib/x86_64-linux-gnu/libnss_compat-2.21.so
7f40d732e000-7f40d732f000 r--p 00006000 08:01 393347 /lib/x86_64-linux-gnu/libnss_compat-2.21.so
7f40d732f000-7f40d7330000 rw-p 00007000 08:01 393347 /lib/x86_64-linux-gnu/libnss_compat-2.21.so
7f40d7330000-7f40d74ca000 r-xp 00000000 08:01 393300 /lib/x86_64-linux-gnu/libc-2.21.so
7f40d74ca000-7f40d76ca000 ---p 0019a000 08:01 393300 /lib/x86_64-linux-gnu/libc-2.21.so
7f40d76ca000-7f40d76ce000 r--p 0019a000 08:01 393300 /lib/x86_64-linux-gnu/libc-2.21.so
7f40d76ce000-7f40d76d0000 rw-p 0019e000 08:01 393300 /lib/x86_64-linux-gnu/libc-2.21.so
7f40d76d0000-7f40d76d4000 rw-p 00000000 00:00 0
7f40d76d4000-7f40d76d6000 r-xp 00000000 08:01 393331 /lib/x86_64-linux-gnu/libdl-2.21.so
7f40d76d6000-7f40d78d6000 ---p 00002000 08:01 393331 /lib/x86_64-linux-gnu/libdl-2.21.so
7f40d78d6000-7f40d78d7000 r--p 00002000 08:01 393331 /lib/x86_64-linux-gnu/libdl-2.21.so
7f40d78d7000-7f40d78d8000 rw-p 00003000 08:01 393331 /lib/x86_64-linux-gnu/libdl-2.21.so
7f40d78d8000-7f40d78fe000 r-xp 00000000 08:01 395088 /lib/x86_64-linux-gnu/libtinfo.so.5.9
7f40d78fe000-7f40d7afd000 ---p 00026000 08:01 395088 /lib/x86_64-linux-gnu/libtinfo.so.5.9
7f40d7afd000-7f40d7b01000 r--p 00025000 08:01 395088 /lib/x86_64-linux-gnu/libtinfo.so.5.9
7f40d7b01000-7f40d7b02000 rw-p 00029000 08:01 395088 /lib/x86_64-linux-gnu/libtinfo.so.5.9
7f40d7b02000-7f40d7b24000 r-xp 00000000 08:01 395006 /lib/x86_64-linux-gnu/libncurses.so.5.9
7f40d7b24000-7f40d7d23000 ---p 00022000 08:01 395006 /lib/x86_64-linux-gnu/libncurses.so.5.9
7f40d7d23000-7f40d7d24000 r--p 00021000 08:01 395006 /lib/x86_64-linux-gnu/libncurses.so.5.9
7f40d7d24000-7f40d7d25000 rw-p 00022000 08:01 395006 /lib/x86_64-linux-gnu/libncurses.so.5.9
7f40d7d25000-7f40d7d47000 r-xp 00000000 08:01 393292 /lib/x86_64-linux-gnu/ld-2.21.so
7f40d7db5000-7f40d7df4000 r--p 00000000 08:01 662823 /usr/lib/locale/zu_ZA.utf8/LC_CTYPE
7f40d7df4000-7f40d7f24000 r--p 00000000 08:01 662822 /usr/lib/locale/zu_ZA.utf8/LC_COLLATE
7f40d7f24000-7f40d7f28000 rw-p 00000000 00:00 0
7f40d7f33000-7f40d7f34000 r--p 00000000 08:01 662821 /usr/lib/locale/zu_ZA.utf8/LC_NUMERIC
7f40d7f34000-7f40d7f35000 r--p 00000000 08:01 674657 /usr/lib/locale/en_US.utf8/LC_TIME
7f40d7f35000-7f40d7f36000 r--p 00000000 08:01 674656 /usr/lib/locale/en_US.utf8/LC_MONETARY
7f40d7f36000-7f40d7f37000 r--p 00000000 08:01 673013 /usr/lib/locale/ug_CN/LC_MESSAGES/SYS_LC_MESSAGES
7f40d7f37000-7f40d7f38000 r--p 00000000 08:01 672854 /usr/lib/locale/yi_US.utf8/LC_PAPER
7f40d7f38000-7f40d7f39000 r--p 00000000 08:01 672855 /usr/lib/locale/yi_US.utf8/LC_NAME
7f40d7f39000-7f40d7f3a000 r--p 00000000 08:01 674655 /usr/lib/locale/en_US.utf8/LC_ADDRESS
7f40d7f3a000-7f40d7f3b000 r--p 00000000 08:01 672853 usr/lib/locale/yi_US.utf8/LC_TELEPHONE
7f40d7f3b000-7f40d7f3c000 r--p 00000000 08:01 672852 /usr/lib/locale/yi_US.utf8/LC_MEASUREMENT
7f40d7f3c000-7f40d7f43000 r--s 00000000 08:01 288093 /usr/lib/x86_64-linux-gnu/gconv/gconv-modules.cache
7f40d7f43000-7f40d7f44000 r--p 00000000 08:01 674654 /usr/lib/locale/en_US.utf8/LC_IDENTIFICATION
7f40d7f44000-7f40d7f46000 rw-p 00000000 00:00 0
7f40d7f46000-7f40d7f47000 r--p 00021000 08:01 393292 /lib/x86_64-linux-gnu/ld-2.21.so
7f40d7f47000-7f40d7f48000 rw-p 00022000 08:01 393292  /lib/x86_64-linux-gnu/ld-2.21.so
7f40d7f48000-7f40d7f49000 rw-p 00000000 00:00 0
7ffcd6454000-7ffcd6475000 rw-p 00000000 00:00 0 [stack]
7ffcd654f000-7ffcd6551000 r--p 00000000 00:00 0 [vvar]
7ffcd6551000-7ffcd6553000 r-xp 00000000 00:00 0 [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall]
```
The first 3 lines all show address ranges written in hexadecimal, along with some other data:
```
00400000-004f4000 r-xp 00000000 08:01 393218 /bin/bash
006f3000-006f4000 r--p 000f3000 08:01 393218 /bin/bash
006f4000-006fd000 rw-p 000f4000 08:01 393218 /bin/bash
```
Hexadecimal is very useful for representing binary data, since each digit is exactly represented by 4 bits, whereas decimal needs a calculator to convert back and forth. The first of these addresses describes addresses in the range 4194304-5193728. This is a block of size `0xf4000`, or 999424 bytes.

These numbers seem arbitrary, so let's consider them in pages of 4k. 4096 is `0x1000` in hex, so the addresses range from page number `0x400` to page number `0x4f4`, for a range of `0xf4` pages. That makes the numbers look much smaller and more regular. 😊

The next block starts at page `0x6f3` and goes up to `0x6f4`, but is immediately followed by the next block which starts at that ending address, and goes up to `0x6fd`. So together, this makes 10 4k pages (or `0xa` pages).

Honestly, I don't really care about the size or ranges of these addresses. What I really notice is that the map explicitly connects these pages to the file `/bin/bash`. They also show an offset into the file where the mapping starts at. There is some overlap between these mapping, and the reason can be seen when the page properties are considered. The first mapping (covering the first `0xf4` pages of the file) is marked as _executable_. This means that it contains runnable code. The next mapping is marked as _read_ only, meaning that any access to those pages should be for reading data. I would expect to find static definitions in there, like message strings. The final block is marked as _read/write_. This might include statically initialized data that can be modified by the running processes.

Further down, we see lots of references to mappings of the shared libraries: the `.so` files. For instance, `libc-2.21.so` contains the standard libraries for the language C (which bash was written in), and `libncurses.so.5.9` contains the [_ncurses_ library](https://en.wikipedia.org/wiki/Ncurses) which is used for creating text interfaces.

Then we have 2 more types of page ranges. The first type includes the 6 ranges that have no file path information. These were presumably given to the process when code requested memory to be allocated, and will likely contain data structures. The second type includes labels rather than a file path. Of particular interest to this process are the _stack_ and the _heap_.

For those who are unfamiliar with these elements of memory layout, the _stack_ is a memory structure used by the computer to move from one function to another. Whenever a function is called, the computer adds data that represents arguments to the "top" of the stack, adds a pointer to the last "top" of the stack along with the current execution address all at the top of the stack, and then jumps to the new function. That function can now see its arguments near the top of the stack. Any variables it needs to create temporarily can also be added to the top of the stack. Once the function is done, the stack is read to find where the previous top-of-stack was along with the previous execution address, and these get restored.

This setup is how code can call into functions, and when the function is finished, it can "return" and the calling code continue where it left off. The data structure is often likened to a stack of plates, where plates can be added to the top, and then removed again after.

Now that we know what the stack is, we can see that it will be rapidly changing data that contains internal data structures representing the function calls.

The _heap_, on the other hand, is where a program gets memory that it requests with `malloc`. This is longer-term working memory. It will typically contain internal data structures, but if strings are being stored by a program, then this is where we would expect to find them. In this case, I'm hoping to find the history of the process, so I want to look in here.

##Pulling Out the Heap
The relevant line for the heap is:
```
00f5d000-0111d000 rw-p 00000000 00:00 0 [heap]
```
This means that we want to see memory starting at address `0xf5d000` and ending `0x1c0000` later at address `0x111d000`. Dividing by 4k pages, this starts at the `0xf5d` page, and extends for `0x1c0` pages.

We've already talked about how we can access memory through the `mem` virtual file, and it just so happens that we have the `dd` tool that can read precise portions of a file. So we just need to establish the arguments for it.

#####status
The `dd` command likes to describe how many blocks it read, how many it wrote, and things like that. We don't want to see that, so we can set the _status_ value to _none_.

#####file
The file to be read will be `mem`, but to access it from elsewhere, we want to refer to the file as `/proc/10668/mem`. This can generalized to read from an arbitrary `pid` variable with: `/proc/${pid}/mem`

#####ibs
It may be easier to ask `dd` to handle data a byte at a time, but we already know that the data comes in 4k pages, so it will be much more efficient to read the memory "file" in 4k blocks. This means that we want to set the "input block size" or _ibs_ to 4096.

#####skip
We also want to skip over all the invalid addresses right up to the start of the heap pages. That means skipping `0xf5d` pages. However, like _ibs_, we need to use decimal for the `dd` command. In this case, that's 3933 pages.

#####count
The number of blocks to read is the final address, minus the first address, divided by the block size. That's `0x111d000` - `0xf5d000`, which gives, `0x1c0000`. Dividing by `0x1000` (4096) gives `0x1c0`, which is 448 in decimal.

This gives us a final command line of:
```
dd status=none if=/proc/${pid}/mem ibs=4096 skip=3933 count=448
```
The data coming out can be saved or piped into another program for processing.

##Automating
This was fine for my system, but there was lots of calculating to get the values necessary to execute this command. I couldn't pass it along to my colleague in that form.

I decided that I could provide a function that takes an argument of the desired process ID to scan. As above, I'll refer to this as `$pid`. So that the PID can be placed in the middle of a string, we can wrap it with braces: `${pid}`

####Range
The address range for the heap can be found by using `grep` to search for the string "heap" in the `maps` file.
```
grep heap /proc/${pid}/maps
00f5d000-0111d000 rw-p 00000000 00:00 0 [heap]
```
That returns the entire line, but we just want the first part of the line. We can split the line up using the `cut` command, telling it to separate fields on the space character, and then saying that we just want the first field.
```
grep heap /proc/${pid}/maps | cut -d' ' -f1
00f5d000-0111d000
```
We can save this result to the variable `$range` by wrapping everything in backticks:
```bash
range=`grep heap /proc/${pid}/maps | cut -d' ' -f1`
```
####Arithmetic
Luckily for us, `bash` has several built in features that we need right now: text substitution, hex/decimal conversion, and arithmetic. 

To get the start of the range, we want everything in the range up to the first `-` character. We can do that by saying that we want to substitute a dash followed by any characters with an empty string. The syntax for this is: `${range/-*/}`

The second part of the range can be found by substituting "a series of characters followed by `-`" with an empty string: `${range/*-}`

Once we have string representations for the numbers, we can do arithmetic by wrapping expressions with `$((` `))`.

Arithmetic needs to be done in decimal, but once we are inside an arithmetic expression, we can print any hexadecimal (base 16) number as decimal by preceding the value with `16#`:
```bash
$((16#1c0))
448
```
So now we can build an expression for each of the `dd` parameters, by extracting the value, converting to decimal, and dividing by 4096:
```bash
# skip
$((16#${range/-*/}/4096))
# count
$(((16#${range/*-/}-16#${range/-*/})/4096))
```
The full command is then:
```bash
dd status=none if=/proc/${pid}/mem ibs=4096 skip=$((16#${range/-*/}/4096)) count=$(((16#${range/*-/}-16#${range/-*/})/4096))
```
####Processing
The dd command pulls out the heap, but this is binary data for an active process, and should not be printed. Data structures could be inspected using a tool like `hexdump` but in this case I'm looking for text in the user's history. The best tool for this is the [`strings`](http://man7.org/linux/man-pages/man1/strings.1.html) command, which filters out anything that is not printable data.

##Final Function
Here is the final function:
{% gist https://gist.github.com/quoll/62924ea8139a39a22a27e6c29818e37e %}

I can run it against the `testuser` process, but I'm looking for something suspicious done with `sudo`, so let's look for that:
```bash
$ string_heap 10852 | grep sudo
sudo
sudo cp /bin/su  /tmp/echo
sudo cp /bin/su  /tmp/echo
sudo cp /bin/su  /tmp/echo
/usr/share/bash-completion/completions/sudo
_sudo
sudo
_sudo
sudoedit
_sudo
_sudo
/usr/bin/sudo
sudo cp /bin/su /tmp/echo
*sudoedit
_sudo
/etc/init.d/sudo
gksudo
kdesudo
_sudo
*sudoedit
_sudo
_=/usr/bin/sudo
```
Sure enough... there is the command at the top of the output.

##Recap
This was a very short description of some of the instrumentation provided in the Linux `/proc` filesystem, in relation to process memory. We also discussed virtual memory, its allocation, and usage in Linux and Windows. Finally, I showed how to write a short `bash` script that can scan the memory of another process and look for suspicious activity.