Repository: yt-dlp
Issue: YouTube Shorts subscriptions feed is broken, a third of videos missing in the feed
Labels: account-needed,site-bug,triage,site:youtube

### DO NOT REMOVE OR SKIP THE ISSUE TEMPLATE

- [X] I understand that I will be **blocked** if I *intentionally* remove or skip any mandatory\* field

### Checklist

- [X] I'm reporting that yt-dlp is broken on a **supported** site
- [X] I've verified that I have **updated yt-dlp to nightly or master** ([update instructions](https://github.com/yt-dlp/yt-dlp#update-channels))
- [X] I've checked that all provided URLs are playable in a browser with the same IP and same login details
- [X] I've checked that all URLs and arguments with special characters are [properly quoted or escaped](https://github.com/yt-dlp/yt-dlp/wiki/FAQ#video-url-contains-an-ampersand--and-im-getting-some-strange-output-1-2839-or-v-is-not-recognized-as-an-internal-or-external-command)
- [X] I've searched [known issues](https://github.com/yt-dlp/yt-dlp/issues/3766) and the [bugtracker](https://github.com/yt-dlp/yt-dlp/issues?q=) for similar issues **including closed ones**. DO NOT post duplicates
- [X] I've read the [guidelines for opening an issue](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#opening-an-issue)
- [ ] I've read about [sharing account credentials](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#are-you-willing-to-share-account-details-if-needed) and I'm willing to share it if required

### Region

_No response_

### Provide a description that is worded well enough to be understood

First and foremost this is an issue with the YouTube shorts subscriptions feed itself, it comes up when just using <https://www.youtube.com/feed/subscriptions/shorts>.

I've noticed for a while now how sometimes when scrolling down to a part where I already was some time before, there are videos which I definitely haven't seen in the feed before. One would expect a subscriptions feed to be complete and chronological. This is not the case for the shorts subscriptions feed.

I have spent some time playing around now and I could see the behaviour based on new videos appearing (not based on day or time of day as far as I can see) and in that moment, the sliding window of the shorts that are visible changes slightly.

Here are some numbers: I kept downloading the first 2600 entries and saving each unique ID I came across. After a bunch of requests (whenever an actual new video was included in the feed) and approximately 9 actual new videos compared to the first request, the video that was in the first disappearing cycle I came across reappeared. At this point, counting the total number of entries, I was not at the expected 2600 encountered IDs, but at **3911**!

Here's an example diff of resulting video ids when querying the newest 35 videos of the feed. My example video IDs don't encode any meaning besides serving as an unique identifier.

```diff
+xxxxxxxxx90 -- new video compared to initial request
 xxxxxxxxx01 
 xxxxxxxxx02 
 xxxxxxxxx03 
 xxxxxxxxx04 
 xxxxxxxxx05 
 xxxxxxxxx06 
 xxxxxxxxx07 
 xxxxxxxxx08 
 xxxxxxxxx09 
 xxxxxxxxx10 
 xxxxxxxxx11 
 xxxxxxxxx12 
 xxxxxxxxx13 
 xxxxxxxxx14 
 xxxxxxxxx15 
+xxxxxxxxx80 -- a video that's not "new" but only now became visible
-xxxxxxxxx16 -- this one disappeared now
 xxxxxxxxx17 
 xxxxxxxxx18 
 xxxxxxxxx19 
 xxxxxxxxx20 
 xxxxxxxxx21 
 xxxxxxxxx22 
 xxxxxxxxx23 
 xxxxxxxxx24 
 xxxxxxxxx25 
 xxxxxxxxx26 
 xxxxxxxxx27 
 xxxxxxxxx28 
 xxxxxxxxx29 
 xxxxxxxxx30 
 xxxxxxxxx31 
+xxxxxxxxx81 -- another video that's not "new" but only now became visible
-xxxxxxxxx32 -- this one also disappeared
 xxxxxxxxx33 
 xxxxxxxxx34 
-xxxxxxxxx35 -- this one's just out of range now and not returned anymore, nothing special
```

If there's a second new video compared to the first time you checked, at each of these special points another one disappears and a "new" one appears, in this example it would be the video IDs xx..15 and xx..31. Rinse and repeat until after approximately 9 actual new videos appeared at the top of the feed compared to the first time, each video that disappeared after the first actual new video was in the feed (so in this example xx..16 and xx..32) has come back again. (However of course there are now like 8 different videos, the ones above these ones at each of the magic points, which have been initially visible and are now not visible anymore. The amount of videos that are not visible is approximately constant I guess. Don't pin me down on the specific number of 8 or 9 or something else, there may have been some off by one errors, especially when there's an extremely new video sometimes the sliding window moves even though the new video itself hasn't appeared at the top of the feed).

Now I feel like it's impossible to report a bug to YouTube, so I decided to collect my findings here. Since there might (maybe, I have no idea..) be other ways to query the shorts subscriptions feed, yt-dlp might change the way this specific feed data is retrieved to avoid being affected by this really weird bug.

As for the verbose output, I'm just copying the one I used for the other issue I opened, it's a command which can also be used to assess this bug, but the above example should be used to understand it, since it requires multiple requests and a changed feed to be noticeable, I think that would just not come across well using a bunch of console outputs (which I would like to censor anyways for personal reasons)

### Provide verbose output that clearly demonstrates the problem

- [X] Run **your** yt-dlp command with **-vU** flag added (`yt-dlp -vU <your command line>`)
- [ ] If using API, add `'verbose': True` to `YoutubeDL` params instead
- [X] Copy the WHOLE output (starting with `[debug] Command-line config`) and insert it below

### Complete Verbose Output

```shell
[debug] Command-line config: ['-vU', '--flat-playlist', '--skip-download', '--playlist-end', '50', 'https://youtube.com/feed/subscriptions/shorts', '--ignore-no-formats', '--print', '%(id)s %(title)s', '--cookies-from-browser', 'firefox']
[debug] Encodings: locale UTF-8, fs utf-8, pref UTF-8, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version nightly@2024.12.26.232815 from yt-dlp/yt-dlp-nightly-builds [0b6b7742c] (pip)
[debug] Python 3.13.1 (CPython x86_64 64bit) - Linux-6.12.6-200.fc41.x86_64-x86_64-with-glibc2.40 (OpenSSL 3.2.2 4 Jun 2024, glibc 2.40)
[debug] exe versions: ffmpeg 7.0.2 (fdk,setts), ffprobe 7.0.2
[debug] Optional libraries: Cryptodome-3.21.0, brotli-1.1.0, certifi-2024.12.14, mutagen-1.47.0, requests-2.32.3, sqlite3-3.46.1, urllib3-1.26.20, websockets-14.1
[debug] Proxy map: {}
Extracting cookies from firefox
[debug] Extracting cookies from: "/home/xxxxxxxxxxxxxxx/cookies.sqlite"
Extracted 1090 cookies from firefox
[debug] Request Handlers: urllib, requests, websockets
[debug] Loaded 1837 extractors
[debug] Fetching release info: https://api.github.com/repos/yt-dlp/yt-dlp-nightly-builds/releases/latest
Latest version: nightly@2024.12.26.232815 from yt-dlp/yt-dlp-nightly-builds
yt-dlp is up to date (nightly@2024.12.26.232815 from yt-dlp/yt-dlp-nightly-builds)
[youtube:tab] Extracting URL: https://youtube.com/feed/subscriptions/shorts
[youtube:tab] subscriptions: Downloading webpage
[download] Downloading playlist: subscriptions
[debug] [youtube:tab] Extracted SAPISID cookie
[youtube:tab] subscriptions page 1: Downloading API JSON
[youtube:tab] subscriptions page 2: Downloading API JSON
[youtube:tab] subscriptions page 3: Downloading API JSON
[youtube:tab] Playlist subscriptions: Downloading 50 items
[debug] The information of all playlist entries will be held in memory
[download] Downloading item 1 of 50
xxxxxxxxxxx NA
[download] Downloading item 2 of 50
...
[download] Finished downloading playlist: subscriptions
```