Canaries and other techniques that leverage volatiles do not prevent an overflow; they just try to cope with the consequences of an overflow which has happened. (This is why they're generally lumped together as "mitigation".) The canary tries to detect the case of an overflow which overwrites the return address in a stack frame. Data Execution Prevention (DEP) takes this idea a step further, it assumes that the return address has been overwritten and followed, and it restricts the areas where execution could jump. ASLR is yet another step further: it "shuffles" the areas where execution is allowed.

More specifically, stack canaries work by modifying every function's prologue and epilogue regions to place and check a value on the stack respectively. As such, if a stack buffer is overwritten during a memory copy operation, the error is noticed before execution returns from the copy function. When this happens, an exception is raised, which is passed back up the exception handler hierarchy until it finally hits the OS's default exception handler. If you can overwrite an existing exception handler structure in the stack, you can make it point to your own code. This is a Structured Exception Handling (SEH) exploit, and it allows you to completely bypass the canary check.

DEP and NX (what OpenBSD calls W^X) mark important structures in memory as non-executable, and force hardware-level exceptions if you try to execute those memory regions. This makes normal stack buffer overflows where you set eip to esp+offset and immediately run your shellcode impossible, because the stack is non-executable. Bypassing DEP and NX requires a trick called Return-Oriented Programming (ROP).

ROP essentially involves finding existing snippets of code from the program (called gadgets) and jumping to them, such that you produce a desired outcome. Since the code is part of legitimate executable memory, DEP and NX don't matter. These gadgets are chained together via the stack, which contains the exploit payload. Each entry in the stack corresponds to the address of the next ROP gadget. Each gadget is in the form of instr1; instr2; instr3; ... instrN; ret, so that the ret will jump to the next address on the stack after executing the instructions, thus chaining the gadgets together. Often additional values have to be placed on the stack in order to successfully complete a chain, due to instructions that would otherwise get in the way.

The trick is to chain these ROPs together in order to call a memory protection function such as VirtualProtect, which is then used to make the stack executable, so your shellcode can run, via an jmp esp or equivalent gadget. Tools like mona (https://github.com/corelan/mona) can be used to generate ROP gadget chains, or to find ROP gadgets.

There are a few ways to bypass ASLR:

Direct RET overwrite - Often processes with ASLR will still load non-ASLR modules, allowing you to just run your shellcode via a jmp.

Partial EIP overwrite - Only overwrite part of EIP, or use a reliable information disclosure in the stack to find what the real EIP should be, then use it to calculate your target. We still need a non-ASLR module for this though.

NOP spray/sled - Create a big block of NOPs to increase chance of jump landing on legit memory. Difficult, but possible even when all modules are ASLR-enabled. This won't work if DEP is switched on.

Bruteforce - If you can try an exploit with a vulnerability that doesn't make the program crash, you can bruteforce 256 different target addresses until it works.

Again, the important theme here is that canaries, DEP and ASLR do not defeat overflows themselves, but target the generic overflow exploit methods which have traditionally been employed. The arms race between attackers and defenders in this space is becoming too specialized and increasingly, misses the point.

Additionally, PIE (required for ASLR) has a negative impact on performance: https://nebelwelt.net/publications/12TRpie/gccPIE-TR120614.p...

Additional reading:

https://www.corelan.be/index.php/2010/06/16/exploit-writing-...
https://www.corelan.be/index.php/2009/09/21/exploit-writing-...

Or if you're more academically oriented:
http://www.scs.stanford.edu/brop/
http://people.csail.mit.edu/rinard/paper/oakland15.pdf

As feld indirectly points out, Capsicum is a much (much) better technology, because it traps the exploit (in a sandbox).

Capsicum extends file descriptors to include the notion of what you are allowed to do with the file. They already have some limited support for this. If, for example, you specify O_RDONLY to the open() system call, then you will get an error if you try writing to the resulting file descriptor. This is largely advisory: There is nothing stopping you from using fstat() to get the original path, and then opening it in a new mode.

This is where Capsicum enters the picture. After a call to cap_enter(), the program is in capability mode and is not allowed to create any new file descriptors via most of the standard mechanisms.

In particular, system calls like open() and socket() will simply fail. This has the advantage that it's a very simple test to perform and therefore quite easy to get right: Just check one flag and give up if it's cleared.

Capability file descriptors behave just like normal ones. You can pass them to any system call that expects a file descriptor, but you may get an error if you don't have the correct rights. These include read and write permissions—and also a variety of other things.

Edit: Theo seems to be cautiously boarding the capabilities train with tame (https://marc.info/?l=openbsd-tech&m=143725996614627&w=2), introduced today.

But Capsicum, Linux's seccomp-bpf (which Theo describes as 'insane') and OS X's seatbelt are all similar. Gaol (https://github.com/pcwalton/gaol) uses either seccomp-bpf or seatbelt as a backend.

Windows 8 has an equivalent of this, using a "mitigation policy" called ProcessSystemCallDisablePolicy, which is set using SetProcessMitigationPolicy(). Chrome uses this for sandboxing on Windows (https://src.chromium.org/chrome/branches/1312/src/sandbox/wi...) and uses seccomp-bpf on linux.

See also: Solaris' Role-Based Access Control and Privileges models.
http://www.c0t0d0s0.org/archives/4075-Less-known-Solaris-fea...