Repository: rust
Issue: A codegen option to stub `default_hook` + safe optimizations to `lang_start_internal` shrink the minimum executable size by 60%
Labels: C-enhancement,A-codegen,T-libs-api,I-heavy

The smallest "Hello World" binary in Rust using the normal entry point is roughly:

```rust
extern crate libc;
use std::alloc::System;
#[global_allocator]
static ALLOCATOR: System = System;

fn main() {
    const BUF: &'static [u8] = b"Hello World!\n";
    unsafe {
        libc::write(libc::STDOUT_FILENO,
                    BUF.as_ptr() as *const libc::c_void,
                    BUF.len()
        )
    };
}
```

That's a good test of the minimum Rust binary size. Compiled with `opt-level = 's'`  `lto = true` on macOS 10.14 with the 17th's nightly, it is:

| panic                | symbols  | size |
|----------------------|----------|------|
| (default = 'unwind') |          | 160K |
| (default = 'unwind') | stripped | 121K |
| 'abort'              |          | 133K |
| 'abort'              | stripped | 101K |

However about 60% of that is code that's (usually, see below) not used anywhere else in a program, and could be eliminated by adding a codegen option to disable the default panic hook + making a few safe optimizations to `lang_start_internal`.

### 1. Codegen option to replace std::panicking::default_hook with an empty shim

This option would simply cause `std::panicking::default_hook` to be defined as `fn default_hook(_: &PanicInfo) {}`. With that definition, the above program shrinks from 120K -> 79K unstripped, 121K -> 55K striped (62K & 43K with `panic = 'abort'`). Decompilation indicates everything eliminated is backtrace/dsym/dwarf stuff.

Panicking silently by eliminating the default hook is desirable in some situations, especially in stripped executables. Note that the code size reduction can't be achieved with `std::panicking::set_hook` though, because panic calls before main prevent `default_hook` from being optimized away.

### 2. Optimize an inlined `sys_common::cleanup()`

`sys_common::cleanup` is inlined into `lang_start_internal` and guaranteed to only ever be called once.  However the compiler can't optimize away the tests in its `std::sync::Once::call_once`, resulting in many unnecessary panic checks.

Manually removing these further shrinks an unwinding executable to 69K unstripped, 47K stripped. (`panic = 'abort'` even further to 53K & 34K.) Symbols from `std::sync::Once` account for the reduction, and it would apply to any program not using that type.

### 3. Change `Thread::new` to `Thread::_new`

From:

```rust
let thread = Thread::new(Some("main".to_owned()));
```

to:

```rust
let thread = Thread::_new(::ffi::CString::_from_box_unchecked(Box::new(*b"main\0")));
```

In the former, `"main"` is guaranteed to result in a valid CString, and `Option` in guaranteed to be `Some`. Nonetheless 2 panic checks and a mutation are emitted.  In the latter, the resultant CString is still guaranteed to be valid. This cuts <1K, but skips the 2 unnecessary checks.

Note that `pub(crate) unsafe fn _from_box_unchecked` is added to `CString` to skip mutating `b"main"` to `b"main\0"` but is not strictly necessary to skip the checks. However `CString`'s Invariant 2 is still preserved with all public unsafe functions.

This doesn't currently happen, but `Thread::new(Some("main".to_owned()))` would be inlined into `lang_start_internal` and the checks and mutation would be optimized away if the compiler could be taught:

* that `"main".to_owned()` (`into()` a `Vec<u8>`) contains no `NulError`, and thus the call to `CString::new` could be rewritten to `CString::from_vec_unchecked`,
* that afterwards, `CString::from_vec_unchecked` could be short-circuited by appending `0` to the original `Vec<u8>`,
* and to apply these through calls to `Thread::new`.

### Comments

The final reductions are:

| panic                | symbols  | old  | new |
|----------------------|----------|------|-----|
| (default = 'unwind') |          | 160K | 69K |
| (default = 'unwind') | stripped | 121K | 47K |
| 'abort'              |          | 133K | 53K |
| 'abort'              | stripped | 101K | 34K |

`panic = 'unwind'` still correctly unwinds with an empty `default_hook` (albeit silently). AFAIK none of these changes should require an RFC, but let me know.