Repository: rust
Issue: include_str fails with large files
Labels: T-compiler,I-compilemem,C-bug

I just tried to `include_str!()` a 14g file.
It failed with:

<details>
<summary>failure with 16g file</summary>

```text
% cargo build
   Compiling foobar v0.1.0 (file:///home/benaryorg/.local/tmp/tmp.nlZAcHIVlh)
fatal runtime error: memory allocation failed
error: Could not compile `foobar`.
```
</details>

---

Same error with everything down to five gigabytes (which would fit in my ram thrice).
Then I thought, hey, 32-bit and ran the following:

<details>
<summary>failure with 2^32+1 file</summary>

```text
% truncate -s 4294967297 file # 2^32 + 1
% cargo build
   Compiling foobar v0.1.0 (file:///home/benaryorg/.local/tmp/tmp.nlZAcHIVlh)
fatal runtime error: memory allocation failed
error: Could not compile `foobar`.

To learn more, run the command again with --verbose.
```
</details>
<details>
<summary>failure with 2^32 file</summary>

```
% truncate -s 4294967296 file # 2 ^ 32
% cargo build
   Compiling foobar v0.1.0 (file:///home/benaryorg/.local/tmp/tmp.nlZAcHIVlh)
fatal runtime error: memory allocation failed
error: Could not compile `foobar`.

To learn more, run the command again with --verbose.
```
</details>
<details>
<summary>failure with 2^32-1 file</summary>

```
% truncate -s 4294967295 file # 2^32 - 1
% cargo build
   Compiling foobar v0.1.0 (file:///home/benaryorg/.local/tmp/tmp.nlZAcHIVlh)
thread 'main' panicked at 'assertion failed: line_len == 0 || ((*lines)[line_len - 1] < pos)', libsyntax_pos/lib.rs:969:9
note: Run with `RUST_BACKTRACE=1` for a backtrace.

error: internal compiler error: unexpected panic

note: the compiler unexpectedly panicked. this is a bug.

note: we would appreciate a bug report: https://github.com/rust-lang/rust/blob/master/CONTRIBUTING.md#bug-reports

note: rustc 1.27.1 (5f2b325f6 2018-07-07) running on x86_64-unknown-linux-gnu

note: compiler flags: -C debuginfo=2 -C incremental --crate-type bin

note: some of the compiler flags provided by cargo are hidden

error: Could not compile `foobar`.

To learn more, run the command again with --verbose.

```
</details>

<details>
<summary>failure with 2^32-1 file & RUST_BACKTRACE</summary>

```
% RUST_BACKTRACE=1 cargo build
   Compiling foobar v0.1.0 (file:///home/benaryorg/.local/tmp/tmp.nlZAcHIVlh)
thread 'main' panicked at 'assertion failed: line_len == 0 || ((*lines)[line_len - 1] < pos)', libsyntax_pos/lib.rs:969:9
stack backtrace:
   0: std::sys::unix::backtrace::tracing::imp::unwind_backtrace
             at libstd/sys/unix/backtrace/tracing/gcc_s.rs:49
   1: std::sys_common::backtrace::print
             at libstd/sys_common/backtrace.rs:71
             at libstd/sys_common/backtrace.rs:59
   2: std::panicking::default_hook::{{closure}}
             at libstd/panicking.rs:211
   3: std::panicking::default_hook
             at libstd/panicking.rs:227
   4: rustc::util::common::panic_hook
   5: std::panicking::rust_panic_with_hook
             at libstd/panicking.rs:467
   6: std::panicking::begin_panic
   7: syntax_pos::FileMap::next_line
   8: syntax::codemap::CodeMap::new_filemap_and_lines
   9: syntax::ext::source_util::expand_include_str
  10: <F as syntax::ext::base::TTMacroExpander>::expand
  11: syntax::ext::expand::MacroExpander::expand_invoc
  12: syntax::ext::expand::MacroExpander::expand
  13: syntax::ext::expand::MacroExpander::expand_crate
  14: rustc_driver::driver::phase_2_configure_and_expand_inner::{{closure}}
  15: rustc::util::common::time
  16: rustc_driver::driver::phase_2_configure_and_expand
  17: rustc_driver::driver::compile_input
  18: rustc_driver::run_compiler_impl
  19: <scoped_tls::ScopedKey<T>>::set
  20: syntax::with_globals
  21: <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
  22: __rust_maybe_catch_panic
             at libpanic_unwind/lib.rs:105
  23: rustc_driver::run
  24: rustc_driver::main
  25: std::rt::lang_start::{{closure}}
  26: std::panicking::try::do_call
             at libstd/rt.rs:59
             at libstd/panicking.rs:310
  27: __rust_maybe_catch_panic
             at libpanic_unwind/lib.rs:105
  28: std::rt::lang_start_internal
             at libstd/panicking.rs:289
             at libstd/panic.rs:374
             at libstd/rt.rs:58
  29: main
  30: __libc_start_main
  31: <unknown>
query stack during panic:
end of query stack

error: internal compiler error: unexpected panic

note: the compiler unexpectedly panicked. this is a bug.

note: we would appreciate a bug report: https://github.com/rust-lang/rust/blob/master/CONTRIBUTING.md#bug-reports

note: rustc 1.27.1 (5f2b325f6 2018-07-07) running on x86_64-unknown-linux-gnu

note: compiler flags: -C debuginfo=2 -C incremental --crate-type bin

note: some of the compiler flags provided by cargo are hidden

error: Could not compile `foobar`.

To learn more, run the command again with --verbose.
```
</details>

---

Content of the file was generated by `base64 /dev/urandom | pv -Ss 16g > file` (and subsequently `truncate`d).
A file created using `truncate` yields only the error of ≥2³² as described above though, so the error might have to do with `include_str!`.

---

This is happening on:

<details>
<summary>tested rustc stable</summary>

```
% rustc --version --verbose
rustc 1.27.1 (5f2b325f6 2018-07-07)
binary: rustc
commit-hash: 5f2b325f64ed6caa7179f3e04913db437656ec7e
commit-date: 2018-07-07
host: x86_64-unknown-linux-gnu
release: 1.27.1
LLVM version: 6.0
```
</details>
<details>
<summary>tested rustc nightly</summary>

```
% rustup run nightly rustc --version --verbose
rustc 1.29.0-nightly (254f8796b 2018-07-13)
binary: rustc
commit-hash: 254f8796b729810846e2b97620032ecaf103db33
commit-date: 2018-07-13
host: x86_64-unknown-linux-gnu
release: 1.29.0-nightly
LLVM version: 7.0
```
</details>

---

Used code:

```rust
fn main()
{
	let _res = include_str!("../file");
}
```

---

Nightly yields a litte more info:

<details>
<summary>nightly with backtrace and empty (truncated) file</summary>

```
% RUST_BACKTRACE=1 rustup run nightly cargo build --verbose
   Compiling foobar v0.1.0 (file:///home/benaryorg/.local/tmp/tmp.nlZAcHIVlh)
     Running `rustc --crate-name foobar src/main.rs --crate-type bin --emit=dep-info,link -C debuginfo=2 -C metadata=0edabc38d37db739 -C extra-filename=-0edabc38d37db739 --out-dir /home/benaryorg/.local/tmp/tmp.nlZAcHIVlh/target/debug/deps -C incremental=/home/benaryorg/.local/tmp/tmp.nlZAcHIVlh/target/debug/incremental -L dependency=/home/benaryorg/.local/tmp/tmp.nlZAcHIVlh/target/debug/deps`
memory allocation of 8589934592 bytes failed                                                                           error: Could not compile `foobar`.

Caused by:
  process didn't exit successfully: `rustc --crate-name foobar src/main.rs --crate-type bin --emit=dep-info,link -C debuginfo=2 -C metadata=0edabc38d37db739 -C extra-filename=-0edabc38d37db739 --out-dir /home/benaryorg/.local/tmp/tmp.nlZAcHIVlh/target/debug/deps -C incremental=/home/benaryorg/.local/tmp/tmp.nlZAcHIVlh/target/debug/incremental -L dependency=/home/benaryorg/.local/tmp/tmp.nlZAcHIVlh/target/debug/deps` (signal: 6, SIGABRT: process abort signal)
```
</details>

---

**Edit:** seems as if the problem is happening due to running out of RAM because of doubling allocation sizes. Reported `memory allocation of ${n} bytes failed` messages on nightly are always powers of two and htop shows those allocations adding up pretty well. However this also happens with a 1g file. Running out of 16g of RAM when `include_str!()`ing a 1g file shouldn't happen, yet I can't tell where the problematic allocation(s) happen(s).