Repository: lycheeverse/lychee

Issue #1574: 3094bbc broke fragments checking: InvalidBaseJoin
Created: 2024-11-27T11:02:02Z
Before and after:
➜  lychee git:(87d5b56) cargo run -- ../network-documentation/services/zfs.md --include-fragments --base ../network-documentation/ --mode=emoji --timeout=5
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.87s
     Running `target/debug/lychee ../network-documentation/services/zfs.md --include-fragments --base ../network-documentation/ --mode=emoji --timeout=5`
  6/6 ━━━━━━━━━━━━━━━━━━━━ Finished extracting links                                                                                                                                                           
🔍 6 Total (in 0s) ✅ 6 OK 🚫 0 Errors
➜  lychee git:(87d5b56) git checkout -
Previous HEAD position was 87d5b56 Box Octocrab error as it is too large (#1543)
HEAD is now at 3094bbc Add support for relative links (#1489)
➜  lychee git:(3094bbc) cargo run -- ../network-documentation/services/zfs.md --include-fragments --base ../network-documentation/ --mode=emoji --timeout=5
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 16.22s
     Running `target/debug/lychee ../network-documentation/services/zfs.md --include-fragments --base ../network-documentation/ --mode=emoji --timeout=5`
[WARN] Error creating request: InvalidBaseJoin("#scrubs-wichtig")
[WARN] Error creating request: InvalidBaseJoin("#unterschiedliche-platten-in-einem-vdevpool")
  4/4 ━━━━━━━━━━━━━━━━━━━━ Finished extracting links                                                                                                                                                            
🔍 4 Total (in 0s) ✅ 4 OK 🚫 0 Errors

You can also reproduce that with the fragments folder in this repo:
Before: cargo run -- fixtures/fragments/  --include-fragments --mode=emoji --base .
  15/15 ━━━━━━━━━━━━━━━━━━━━ Finished extracting links                                                                                                                                                          Issues found in 2 inputs. Find details below.

[fixtures/fragments/file1.md]:
❌ file:///Users/leanderSchulten/git_projekte/lychee/fixtures/fragments/file2.md#missing-fragment
❌ file:///Users/leanderSchulten/git_projekte/lychee/fixtures/fragments/file1.md#missing-fragment

[fixtures/fragments/file.html]:
❌ file:///Users/leanderSchulten/git_projekte/lychee/fixtures/fragments/file.html#in-the-end

🔍 15 Total (in 0s) ✅ 12 OK 🚫 3 Errors

After:
⠁ 0/0 ━━━━━━━━━━━━━━━━━━━━ Extracting links                                                                                                                                                                     [WARN] Error creating request: InvalidBaseJoin("file1.md#lets-wear-a-hat-être")
[WARN] Error creating request: InvalidBaseJoin("#fragment-2")
[WARN] Error creating request: InvalidBaseJoin("file2.md#fragment-1")
[WARN] Error creating request: InvalidBaseJoin("#missing-fragment")
[WARN] Error creating request: InvalidBaseJoin("file2.md#missing-fragment")
[WARN] Error creating request: InvalidBaseJoin("#code-heading")
[WARN] Error creating request: InvalidBaseJoin("#explicit-fragment")
[WARN] Error creating request: InvalidBaseJoin("file.html#a-word")
[WARN] Error creating request: InvalidBaseJoin("file2.md#custom-id")
[WARN] Error creating request: InvalidBaseJoin("#kebab-case-fragment")
[WARN] Error creating request: InvalidBaseJoin("#kebab-case-fragment-1")
[WARN] Error creating request: InvalidBaseJoin("empty_file#fragment")
[WARN] Error creating request: InvalidBaseJoin("file1.md#fragment-1")
[WARN] Error creating request: InvalidBaseJoin("#in-the-beginning")
[WARN] Error creating request: InvalidBaseJoin("#in-the-end")
  0/0 ━━━━━━━━━━━━━━━━━━━━ Finished extracting links                                                                                                                                                            🔍 0 Total (in 0s) ✅ 0 OK 🚫 0 Errors

Duplicate issue #1657: Cannot extract relative reference links in Markdown
Created: 2025-03-19T08:44:29Z
Test case:
Inline [link1](target1.md)

Reference [link2][link2]

[link2]: target2.md

Collapsed [link3][]

[link3]: target3.md

Shortcut [link4]

[link4]: target4.md

Shortcut [link5] with full URL

[link5]: file:///path/to/target5.md
Save this as ~/junk/lychee/baz.md and process it with lychee baz.md --dump -vv, and it prints:
file:///home/wks/junk/lychee/target1.md (baz.md)
file:///path/to/target5.md (baz.md)

It successfully extracts the link to target1.md and resolved it as a relative URL starting with file:///....
But link2 to link4 failed to be extracted.  Link5 points to a full URL instead of a filename, and it is extracted, too.
I think the problem is in the handling of links in the markdown parser.
// excerpt from lychee-lib/src/extract/markdown.rs

pub(crate) fn extract_markdown(input: &str, include_verbatim: bool) -> Vec<RawUri> {
// ...
                match link_type {
                    LinkType::Inline => {
                        Some(vec![RawUri {
                            text: dest_url.to_string(),
                            element: Some("a".to_string()),
                            attribute: Some("href".to_string()),
                        }])
                    }
                    LinkType::Reference |
                    LinkType::ReferenceUnknown |
                    LinkType::Collapsed|
                    LinkType::CollapsedUnknown |
                    LinkType::Shortcut |
                    LinkType::ShortcutUnknown |
                    LinkType::Autolink |
                    LinkType::Email =>
                     Some(extract_raw_uri_from_plaintext(&dest_url)),
For inline links, it simply treats dest_url as the href.  But for all other kinds of links, it will invoke extract_raw_uri_from_plaintext which uses some kind of heuristics to detect URLs.  So anything that doesn't look like a URL in [label]: foo_bar_baz.md are ignored.

Closing comment (2025-03-20T20:56:23Z): Haven't looked too deeply into it, but could be a duplicate of #1574. There is #1624 if you want to give the current development branch a try.