Repository: yt-dlp
Issue: Downloading fragments when outputting to stdout can fail due to race condition when running multiple downloads in parallel
Labels: bug,core:downloader

### DO NOT REMOVE OR SKIP THE ISSUE TEMPLATE

- [X] I understand that I will be **blocked** if I *intentionally* remove or skip any mandatory\* field

### Checklist

- [X] I'm reporting a bug unrelated to a specific site
- [X] I've verified that I'm running yt-dlp version **2023.03.04** ([update instructions](https://github.com/yt-dlp/yt-dlp#update)) or later (specify commit)
- [X] I've checked that all provided URLs are playable in a browser with the same IP and same login details
- [X] I've checked that all URLs and arguments with special characters are [properly quoted or escaped](https://github.com/yt-dlp/yt-dlp/wiki/FAQ#video-url-contains-an-ampersand--and-im-getting-some-strange-output-1-2839-or-v-is-not-recognized-as-an-internal-or-external-command)
- [X] I've searched [known issues](https://github.com/yt-dlp/yt-dlp/issues/3766) and the [bugtracker](https://github.com/yt-dlp/yt-dlp/issues?q=) for similar issues **including closed ones**. DO NOT post duplicates
- [X] I've read the [guidelines for opening an issue](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#opening-an-issue)

### Provide a description that is worded well enough to be understood

When downloading a youtube video with "fragments" and sending the output to stdout, if I run two such downloads in parallel, there are race conditions writing and deleting the `--Frag1` temporary files.

```
$ yt-dlp 'https://www.youtube.com/watch?v=4aeETEoNfOg' --output - --newline --format 'bestvideo[vcodec^=avc1][height<=1080]' -vU --no-progress >/dev/null & yt-dlp 'https://www.youtube.com/watch?v=4aeETEoNfOg' --output - --newline --format 'bestvideo[vcodec^=avc1][height<=1080]' -vU --no-progress >/dev/null &
...
ERROR: Unable to rename file: [Errno 2] No such file or directory: '--Frag1.part' -> '--Frag1'
...
ERROR: unable to download video data: [Errno 2] No such file or directory: '--Frag1'
```

In my below verbose output, I've shown a command which will start two such downloads in parallel. You might have to run the test command a couple times to trigger the race condition - it's timing specific so sometimes it won't happen randomly.

I was wondering if I could change the temporary file location by making use of the `--paths temp:/tmp/foo` option, however I see that this is unsupported for my use case: `WARNING: --paths is ignored when an outputting to stdout`.

I did notice this in the output: `[debug] File locking is not supported. Proceeding without locking`. I guess this suggests that if I [install the `fcntl` module](https://github.com/yt-dlp/yt-dlp/blob/c459d45dd4d417fb80a52e1a04e607776a44baa4/yt_dlp/utils.py#L2173), that might solve this race condition by adding locking? 

Perhaps another workaround would be to change the working directory so that each invocation of `yt-dlp` has a unique working directory where its temp files will be stored.

Still, I am wondering if ideally we wouldn't have to even write temp files to disk at all. Especially on devices like raspberry pis, which use SD cards with a limited write lifespan (after a certain amount of writes, an SD card will wear out and die). Is there any way to not write these fragment files at all, especially when emitting to stdout?

Thanks for any insight you can provide.

### Provide verbose output that clearly demonstrates the problem

- [X] Run **your** yt-dlp command with **-vU** flag added (`yt-dlp -vU <your command line>`)
- [X] If using API, add `'verbose': True` to `YoutubeDL` params instead
- [X] Copy the WHOLE output (starting with `[debug] Command-line config`) and insert it below

### Complete Verbose Output

```shell
pi@piwall:~/development/piwall2 $ yt-dlp 'https://www.youtube.com/watch?v=4aeETEoNfOg' --output - --newline --format 'bestvideo[vcodec^=avc1][height<=1080]' -vU --no-progress >/dev/null & yt-dlp 'https://www.youtube.com/watch?v=4aeETEoNfOg' --output - --newline --format 'bestvideo[vcodec^=avc1][height<=1080]' -vU --no-progress >/dev/null &

[1] 32204
[2] 32205
pi@piwall:~/development/piwall2 $ [debug] Command-line config: ['https://www.youtube.com/watch?v=4aeETEoNfOg', '--output', '-', '--newline', '--format', 'bestvideo[vcodec^=avc1][height<=1080]', '-vU', '--no-progress']
[debug] Command-line config: ['https://www.youtube.com/watch?v=4aeETEoNfOg', '--output', '-', '--newline', '--format', 'bestvideo[vcodec^=avc1][height<=1080]', '-vU', '--no-progress']
[debug] Encodings: locale UTF-8, fs utf-8, pref UTF-8, out UTF-8, error UTF-8, screen UTF-8
[debug] Encodings: locale UTF-8, fs utf-8, pref UTF-8, out UTF-8, error UTF-8, screen UTF-8
[debug] yt-dlp version stable@2023.03.04 [392389b7d] (pip)
[debug] yt-dlp version stable@2023.03.04 [392389b7d] (pip)
[debug] Python 3.7.3 (CPython armv7l 32bit) - Linux-5.10.103-v7l+-armv7l-with-debian-10.13 (OpenSSL 1.1.1n  15 Mar 2022, glibc 2.28)
[debug] Python 3.7.3 (CPython armv7l 32bit) - Linux-5.10.103-v7l+-armv7l-with-debian-10.13 (OpenSSL 1.1.1n  15 Mar 2022, glibc 2.28)
[debug] exe versions: ffmpeg 4.1.9-0, ffprobe 4.1.9-0
[debug] Optional libraries: Cryptodome-3.11.0, brotli-1.0.9, certifi-2018.08.24, mutagen-1.45.1, secretstorage-2.3.1, sqlite3-2.6.0, websockets-10.0
[debug] exe versions: ffmpeg 4.1.9-0, ffprobe 4.1.9-0
[debug] Optional libraries: Cryptodome-3.11.0, brotli-1.0.9, certifi-2018.08.24, mutagen-1.45.1, secretstorage-2.3.1, sqlite3-2.6.0, websockets-10.0
[debug] Proxy map: {}
[debug] Proxy map: {}
[debug] Loaded 1786 extractors
[debug] Loaded 1786 extractors
[debug] Fetching release info: https://api.github.com/repos/yt-dlp/yt-dlp/releases/latest
[debug] Fetching release info: https://api.github.com/repos/yt-dlp/yt-dlp/releases/latest
Available version: stable@2023.03.04, Current version: stable@2023.03.04
yt-dlp is up to date (stable@2023.03.04)
Available version: stable@2023.03.04, Current version: stable@2023.03.04
yt-dlp is up to date (stable@2023.03.04)
[youtube] Extracting URL: https://www.youtube.com/watch?v=4aeETEoNfOg
[youtube] 4aeETEoNfOg: Downloading webpage
[youtube] Extracting URL: https://www.youtube.com/watch?v=4aeETEoNfOg
[youtube] 4aeETEoNfOg: Downloading webpage
[youtube] 4aeETEoNfOg: Downloading android player API JSON
[youtube] 4aeETEoNfOg: Downloading android player API JSON
[debug] Sort order given by extractor: quality, res, fps, hdr:12, source, vcodec:vp9.2, channels, acodec, lang, proto
[debug] Formats sorted by: hasvid, ie_pref, quality, res, fps, hdr:12(7), source, vcodec:vp9.2(10), channels, acodec, lang, proto, filesize, fs_approx, tbr, vbr, abr, asr, vext, aext, hasaud, id
[debug] Sort order given by extractor: quality, res, fps, hdr:12, source, vcodec:vp9.2, channels, acodec, lang, proto
[debug] Formats sorted by: hasvid, ie_pref, quality, res, fps, hdr:12(7), source, vcodec:vp9.2(10), channels, acodec, lang, proto, filesize, fs_approx, tbr, vbr, abr, asr, vext, aext, hasaud, id
[info] 4aeETEoNfOg: Downloading 1 format(s): 137
[info] 4aeETEoNfOg: Downloading 1 format(s): 137
[debug] Invoking dashsegments downloader on "https://rr7---sn-j5caxoxu-pn2z.googlevideo.com/videoplayback?expire=1678083235&ei=QzAFZPi4HM378gSe3b-oCA&ip=68.193.226.187&id=o-AEbreas7Q2s0q6pMcWFyb7YaptrnvYHEYUqRWHZ0BOCY&itag=137&source=youtube&requiressl=yes&mh=o0&mm=31%2C26&mn=sn-j5caxoxu-pn2z%2Csn-p5qlsnrl&ms=au%2Conr&mv=m&mvi=7&pl=22&gcr=us&initcwndbps=1833750&spc=H3gIhk33i1ImFjrSw-d4T3sQGzvb10g&vprv=1&svpuc=1&mime=video%2Fmp4&gir=yes&clen=83925256&dur=261.260&lmt=1628206110186514&mt=1678061097&fvip=2&keepalive=yes&fexp=24007246&c=ANDROID&txp=5535434&sparams=expire%2Cei%2Cip%2Cid%2Citag%2Csource%2Crequiressl%2Cgcr%2Cspc%2Cvprv%2Csvpuc%2Cmime%2Cgir%2Cclen%2Cdur%2Clmt&sig=AOq0QJ8wRAIgN1jXeizmNKr8V3OEcJyracfFYD2NrAzHvJNdSUnB8isCIGYIHK8Ua_VY0Jw5rSi0r3NK05NxgiRTBP3DyAdM_CyF&lsparams=mh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps&lsig=AG3C_xAwRgIhALm1H__T9xO8u3RUf45AN5tjF8BFrRAa2UqpgjXhRtvfAiEA4jcHqj8EFo_Ur7GMrUqNYGNkPbJk_E1w8x1jjOCLWKk%3D"
[dashsegments] Total fragments: 9
[download] Destination: -
[debug] File locking is not supported. Proceeding without locking
[debug] Invoking dashsegments downloader on "https://rr7---sn-j5caxoxu-pn2z.googlevideo.com/videoplayback?expire=1678083235&ei=QzAFZJOtH4v48gT76q_ADA&ip=68.193.226.187&id=o-AEfgcpR0KB_HRvGuYWfFkeZJEk6XzMamyaPN8o0jNyN0&itag=137&source=youtube&requiressl=yes&mh=o0&mm=31%2C26&mn=sn-j5caxoxu-pn2z%2Csn-p5qlsnrl&ms=au%2Conr&mv=m&mvi=7&pcm2cms=yes&pl=22&gcr=us&initcwndbps=1786250&spc=H3gIhhqTolNSiBaCAdQaBNggdsvvqdo&vprv=1&svpuc=1&mime=video%2Fmp4&gir=yes&clen=83925256&dur=261.260&lmt=1628206110186514&mt=1678061334&fvip=2&keepalive=yes&fexp=24007246&c=ANDROID&txp=5535434&sparams=expire%2Cei%2Cip%2Cid%2Citag%2Csource%2Crequiressl%2Cgcr%2Cspc%2Cvprv%2Csvpuc%2Cmime%2Cgir%2Cclen%2Cdur%2Clmt&sig=AOq0QJ8wRgIhAKOkGNW-Uedul-a6WGOyJcM47xOhU7SPu_JoIaZs29uPAiEAjc5OtsCrrz1wUvTg0mWUbHsj43FEC5iujsdn5JIzg8s%3D&lsparams=mh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpcm2cms%2Cpl%2Cinitcwndbps&lsig=AG3C_xAwRgIhAM1eMJc2aS9FzdsAqRknltrNT0k31LWKPjsUyO3LUorcAiEAmIDbvdHqvPLrpg-Ih9AQDVuhBZzsyoeKONOQ04yM48E%3D"
[dashsegments] Total fragments: 9
[download] Destination: -
[debug] File locking is not supported. Proceeding without locking
ERROR: Unable to rename file: [Errno 2] No such file or directory: '--Frag1.part' -> '--Frag1'
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/dist-packages/yt_dlp/downloader/common.py", line 230, in wrapper
    return func(self, *args, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/yt_dlp/downloader/common.py", line 254, in try_rename
    os.replace(old_filename, new_filename)
FileNotFoundError: [Errno 2] No such file or directory: '--Frag1.part' -> '--Frag1'

ERROR: unable to download video data: [Errno 2] No such file or directory: '--Frag1'
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/dist-packages/yt_dlp/utils.py", line 662, in sanitize_open
    stream = locked_file(filename, open_mode, block=False).__enter__()
  File "/usr/local/lib/python3.7/dist-packages/yt_dlp/utils.py", line 2221, in __init__
    self.f = os.fdopen(os.open(filename, flags, 0o666), mode, encoding=encoding)
FileNotFoundError: [Errno 2] No such file or directory: '--Frag1'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/dist-packages/yt_dlp/YoutubeDL.py", line 3247, in process_info
    success, real_download = self.dl(temp_filename, info_dict)
  File "/usr/local/lib/python3.7/dist-packages/yt_dlp/YoutubeDL.py", line 2970, in dl
    return fd.download(name, new_info, subtitle)
  File "/usr/local/lib/python3.7/dist-packages/yt_dlp/downloader/common.py", line 444, in download
    ret = self.real_download(filename, info_dict)
  File "/usr/local/lib/python3.7/dist-packages/yt_dlp/downloader/dash.py", line 60, in real_download
    return self.download_and_append_fragments_multiple(*args, is_fatal=lambda idx: idx == 0)
  File "/usr/local/lib/python3.7/dist-packages/yt_dlp/downloader/fragment.py", line 382, in download_and_append_fragments_multiple
    return self.download_and_append_fragments(*args[0], **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/yt_dlp/downloader/fragment.py", line 523, in download_and_append_fragments
    decrypt_fragment(fragment, self._read_fragment(ctx)), fragment['frag_index'], ctx)
  File "/usr/local/lib/python3.7/dist-packages/yt_dlp/downloader/fragment.py", line 136, in _read_fragment
    down, frag_sanitized = self.sanitize_open(ctx['fragment_filename_sanitized'], 'rb')
  File "/usr/local/lib/python3.7/dist-packages/yt_dlp/downloader/common.py", line 235, in wrapper
    retry.error_callback(err, 1, 0)
  File "/usr/local/lib/python3.7/dist-packages/yt_dlp/downloader/common.py", line 225, in error_callback
    sleep_func=fd.params.get('retry_sleep_functions', {}).get('file_access'))
  File "/usr/local/lib/python3.7/dist-packages/yt_dlp/utils.py", line 6149, in report_retry
    raise e
  File "/usr/local/lib/python3.7/dist-packages/yt_dlp/downloader/common.py", line 230, in wrapper
    return func(self, *args, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/yt_dlp/downloader/common.py", line 241, in sanitize_open
    f, filename = sanitize_open(filename, open_mode)
  File "/usr/local/lib/python3.7/dist-packages/yt_dlp/utils.py", line 664, in sanitize_open
    stream = open(filename, open_mode)
FileNotFoundError: [Errno 2] No such file or directory: '--Frag1'


[1]-  Exit 1                  yt-dlp 'https://www.youtube.com/watch?v=4aeETEoNfOg' --output - --newline --format 'bestvideo[vcodec^=avc1][height<=1080]' -vU --no-progress > /dev/null
```