Handling of symlinks on Windows (Perl, MSYS2, Cygwin)
Tags: perl, windows, msys2, cygwin
Published: 2021-07-04 16:14:53

After [reading about](https://dev.to/perldean/vscode-as-a-perl-ide-3cco) the Perl language server module for VS Code,
I was eager to test it since I had been missing a possibility to debug Perl code from inside VS Code. In the past I have been using emacs as my primary editor for a long time, but am gradually using VS Code more and more now. VS Code has built-in debugging support for the Node.js runtime and can debug JavaScript, TypeScript. For debugging other languages one can install Debugger extensions in the VS Code Marketplace. 

The [Language Server and Debugger for Perl](https://marketplace.visualstudio.com/items?itemName=richterger.perl) VS Code extension allows debugging of Perl scripts, see the link above for an extensive list of features. To use the extension you first need to install the Perl module [Perl::LanguageServer](https://metacpan.org/pod/Perl::LanguageServer). I first tried to install it on Ubuntu 21.04. The module installed fine here, but after installation the extension still did not work. I found that the reason was that I was using a custom `perl` installed with [perlbrew](https://perlbrew.pl/). Fortunately, it was easy to fix by going into the extension settings and setting an absolute path to the binary:

![VS Code extension settings page](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/otebjaxjdv9xxs8gm8go.png)
 
Next, I was curious to see if it also would install on Windows. Even if my preferred platform is Linux, I occasionally use Windows. For example when trying to answer questions on Stack Overflow that are related to Perl and Windows. I am using Windows 10 (run from Ubuntu through [KVM](https://linuxize.com/post/how-to-install-kvm-on-ubuntu-18-04)), Home edition, 21H1, and Strawberry perl version 5.32.1. As anticipated, I quickly ran into issues with installing dependent modules, in particular [IO::AIO](https://metacpan.org/pod/IO::AIO) was difficult.

![cpan Perl::LanguageServer log](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/gis6g1fd88xyhzn0wakv.PNG)

I downloaded [the source](https://metacpan.org/pod/IO::AIO) and tried to install `IO::AIO` manually (not using `cpanm`) first:

```
>perl Makefile.PL
[...]
*** It seems you are running perl version 5.032001, likely the "official" or
*** "standard" version. While there is nothing wrong with doing that,
*** standard perl versions 5.022 and up are not supported by IO::AIO.
*** While this might be fatal, it might also be all right - if you run into
*** problems, you might want to downgrade your perl or switch to the
*** stability branch.
***
*** If everything works fine, you can ignore this message.
***
***
*** Stability canary mini-FAQ:
***
*** Do I need to do anything?
***    With luck, no. While some distributions are known to fail
***    already, most should probably work. This message is here
***    to alert you that your perl is not supported by IO::AIO,
***    and if things go wrong, you either need to downgrade, or
***    sidegrade to the stability variant of your perl version,
***    or simply live with the consequences.
***
[...]

*** Your platform is not standards compliant. To get this module working, you need to
*** download and install win32 pthread (http://sourceware.org/pthreads-win32/).
***

Generating a gmake-style Makefile
Writing Makefile for IO::AIO
Writing MYMETA.yml and MYMETA.json

```

Apparently it wants me to install [POSIX Threads for Windows](https://sourceforge.net/projects/pthreads4w/). I also noticed that the [Makefile.PL](https://metacpan.org/release/MLEHMANN/IO-AIO-4.75/source/Makefile.PL#L67) is calling GNU Autotools [configure script](https://en.wikipedia.org/wiki/Configure_script) which, as far as I know, cannot be run without a POSIX subsystem like Cygwin or MSYS2. 

Since I already had Cygwin and MSYS2 installed, I decided to give [MSYS2](https://www.msys2.org/) a try. According to [wikipedia](https://en.wikipedia.org/wiki/Mingw-w64), MSYS2 ("minimal system 2") is a software distribution and a development platform for Windows, based on Mingw-w64 and Cygwin, that helps to deploy code from the Unix world on Windows. Instead of providing a full environment like Cygwin does, MSYS2 focuses on being a development and deployment platform.

From the MSYS2 terminal window, also using `perl` version 5.32.1 (but still a different binary than the `perl` used by the CMD prompt):
```
$ perl Makefile.PL
[...]
*** The stability canary says: (nothing, it was driven away by harsh weather)
***
*** It seems you are running perl version 5.032001, likely the "official" or
*** "standard" version. While there is nothing wrong with doing that,
*** standard perl versions 5.022 and up are not supported by IO::AIO.
*** While this might be fatal, it might also be all right - if you run into
*** problems, you might want to downgrade your perl or switch to the
*** stability branch.
***
*** If everything works fine, you can ignore this message.
***
[...]
Continue anyways?  [y]
configure: loading site script /etc/config.site
checking for gcc... gcc
checking whether the C compiler works... yes
checking for C compiler default output file name... a.exe
checking for suffix of executables... .exe
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether gcc accepts -g... yes
checking for gcc option to accept ISO C89... none needed
checking how to run the C preprocessor... gcc -E
[...]
checking for st_birthtimespec... yes
checking for st_gen... no
checking for statx... no
checking for accept4... yes
configure: creating ./config.status
config.status: creating config.h
Generating a Unix-style Makefile
Writing Makefile for IO::AIO
Writing MYMETA.yml and MYMETA.json
$ make
$ make test
[...]
t/03_errors.t ... 1/12 # Failed test 9 in t/03_errors.t at line 57
#  t/03_errors.t line 57 is:       ok (!$_[0]);
# Failed test 10 in t/03_errors.t at line 58
#  t/03_errors.t line 58 is:       ok ("\\test\\" eq readlink $some_link);
t/03_errors.t ... Failed 2/12 subtests
[...]
```
And more failed tests were to come. Most fails were due to an unexpected behavior of symlinks on Windows and MSYS2. Even if I was able to fix the issue with `IO::AIO`, there were other issues with other modules also needed by `Perl::LanguageServer`.

Despite this setback, I saw it as an opportunity to learn more about symlinks on Windows. Apparently there were similar issues with other modules that also needed to be fixed. As I was working with a pull request to [file-copy-recursive-reduced](https://github.com/jkeenan/file-copy-recursive-reduced) I was encouraged to write this blog post.

### Symlinks on Windows (MSYS2 and Cygwin)

A [symlink](https://en.wikipedia.org/wiki/Symbolic_link) is a file that contains a reference to another file or directory in the form of an absolute or relative path. The referenced text string is automatically followed by the operating system as a path to another file or directory. This other file or directory is called the "target". So the symlink is a second file that exists independently of its target. If a symlink is deleted, its target remains unaffected. If a symlink points to a target, and sometime later that target is moved, renamed or deleted, the symlink is not automatically updated or deleted, but continues to exist and still points to the now non-existing target. Symlinks pointing to non-existing targets are called broken symlinks.

On Unix-like operating systems, the target does not have to exist when a symlink is created, so broken symlinks can even be created initially. The ability to create broken symlinks is particularly useful when copying directories. Usually, the algorithm used to copy a directory just copies the files in the order returned by the `readdir()` system call and since it is legal to create broken symlinks it does not need to copy the target of a relative symlink before the symlink itself is copied. 

So on Unix-like operating systems symlinks and broken symlinks can always be created, whereas on Windows this is not always the case. Early versions of Windows did not have symlinks of any kind, Windows 95 introduced [file shortcuts](https://en.wikipedia.org/wiki/Shortcut_(computing)), Windows XP introduced native symlinks (only enabled by default for kernel mode programs), and [starting with](https://blogs.windows.com/windowsdeveloper/2016/12/02/symlinks-windows-10/) Windows 10 Insider build 14972, native symlinks could be created without needing to elevate the console as administrator (to enable this, go to the Windows settings app and choose _"Update & Security"_ -> _"For developers"_, and turn on _"Developer mode"_). 


Even if the Windows [shortcut](https://en.wikipedia.org/wiki/Shortcut_(computing)) file is just a metafile used by the Windows File Explorer, it has been used by Cygwin to emulate symlinks. However, the Cygwin shortcut file cannot be read properly by the File Explorer since it lacks many of the expected header fields, whereas a shortcut file created in the File Explorer can be read by Cygwin.
 
In addition to this, in Cygwin and MSYS2 there is a further complication to the creation of symlinks. In Cygwin, creation of symlinks depends on an environment variable called `CYGWIN`. Depending of the content of this environment variable, the creation of broken symlinks may fail, or the creation of non-broken symlinks may fail if developer mode (see discussion above) is not activated. The environment variable also regulates whether the symlink will be created as a shortcut file or as a native symlink. 

#### Behavior of `ln -s` on Cygwin

The behavior of the `ln --symbolic <target> <destination>` command in Cygwin [depends on](https://cygwin.com/cygwin-ug-net/using-cygwinenv.html) the environment variable `CYGWIN` which is used to configure many global settings for the Cygwin runtime system. It contains options separated by blank characters. The option that is important for the `ln -s` command is called `winsymlinks`. According to the [Cygwin documentation](https://cygwin.com/cygwin-ug-net/using-cygwinenv.html), there are four cases for the `winsymlinks` option to consider:

1. `winsymlinks` is not defined. (Note: this behavior differs from that of MSYS2, see below). This is called the default behavior for Cygwin.
a) If native symlinks are enabled (see discussion above), then this is equivalent to setting `winsymlinks` to `native` (e.g. `CYGWIN=winsymlinks:native`), see 3) below.
b) If native symlinks are not enabled, this is equivalent to setting `winsymlinks` to `lnk` see 2) below.

2. `winsymlinks` is empty (`CYGWIN=winsymlinks`) or `winsymlinks` is set to `lnk` (e.g. `CYGWIN=winsymlinks:lnk`)
Whether \<target> exists or not, `ln -s` creates <destination> as a Windows shortcut file.

3. `winsymlinks:native`
a) If native symlinks are enabled, and whether \<target> exists or not, creates \<destination> as a native Windows symlink. Note, this is most similar to the behavior of `ln -s` on *nix.
b) If native symlinks are not enabled, it is equivalent to setting `winsymlinks` to `lnk`, see 2) above.

4. `winsymlinks:nativestrict`
  a) If native symlinks are enabled and \<target> exists, creates \<destination> as a native Windows symlink,
  b) else if native symlinks are not enabled or if \<target> does not exist, `ln -s` fails.

#### Behavior of `ln -s` on MSYS2

Similiarly to the `CYGWIN` environment variable, the `MSYS` environment variable is used to configure global settings for the MSYS2 runtime system (since MSYS2 is based on Cygwin). The four cases for the `winsymlinks` option to consider is:

1. `winsymlinks` is not defined. The default behavior for MSYS2. (Note: this is not similar to Cygwin) 
   a) If \<target> exists, \<target> is (surprise!!) [copied to](https://github.com/msys2/MSYS2-packages/issues/249) \<destination>, so \<destination> does not become a symlink but simply a copy of \<target>, this happens whether \<target> is a file or a directory, or whether native symlinks are enabled or not. 
  b) If \<target> does not exist, `ln -s` fails.

2. `winsymlinks` or `winsymlinks:lnk` : (Similar to Cygwin, see above)

3. `winsymlinks:native` : (Similar to Cygwin, see above)

4. `winsymlinks:nativestrict` : (Similar to Cygwin, see avove)

#### Behavior of symlinks in Perl on Windows (MSYS2 and Cygwin)

In both MSYS2 and Cygwin,

         perl -MConfig -E'say $Config{d_symlink}'

prints `define` (meaning the `symlink` call is implemented), whereas in regular windows (CMD prompt and `$^O eq "MSWin32"`) with e.g. strawberry perl, `$Config{d_symlink}` is only defined for `perl` versions >= 5.33.5, see [perldelta](https://metacpan.org/release/CORION/perl-5.33.5/view/pod/perldelta.pod#Windows).

So the perl `symlink` function "works" on MSYS2 and Cygwin, and for newer versions of `MSWin32`. However, since the newest [Strawberry perl release](https://strawberryperl.com/releases.html) is currently at 5.32.1, I was not able to test how symlinks behave with `MSWin32`. I will therefore in the following focus on Cygwin and MSYS2.

It would be nice if one could easily check from within a Perl script if developer mode was on and thus native symlinks were enabled. However, the only way I found was to use XS (a C extension) to check the value of the registry key `SOFTWARE\Microsoft\Windows\CurrentVersion\AppModelUnlock`.

For the further discussion below, consider the Perl statement:

      symlink $target, $dest;

 If `$dest` exists, the [`symlink`](https://perldoc.perl.org/functions/symlink) command always fails (returning a value of 0 and setting `$!`). So consider the case where `$dest` does not exist: There are four cases for the `winsymlinks` option contained in the `MSYS` or `CYGWIN` environment variable to consider, as was done above for the `ln -s` command. It turns out that `symlink` behaves identically to the `ln -s` command, and when `ln -s` fails, `symlink` also fails and returns a value of 0 and sets `$!` (`ERRNO`).

Also note that the environment variables `MSYS` or `CYGWIN` cannot/should not be changed from within the Perl script itself. I am not sure why this does not work, but I tested it and it showed undefined behavior in my tests. So the variables should be set before `perl` is run, e.g. on the command line:

      MSYS=winsymlinks:native perl p.pl

What about the perl `-l` operator ? Tests show that it does not differentiate between a Windows shortcut file and a native symlink file. So

      say "symlink" if -l "foobar";

prints "symlink" for both file types. Further, there seems to be no tool available to determine which of the two file types a given symlink file is. This means that when copying a symlink file, it is difficult to determine if the destination should be a native symlink or a windows shortcut. Hence, copying a symlink can silently convert a native symlink to a shortcut file depending on the setting of the `CYGWIN` or `MSYS` environment variable and this is also how the `cp` command in MSYS2 (or Cygwin) 

     cp -a source destination

works.

### Summary

The behavior of symlinks is more complicated on Windows than on Linux. This is mainly a problem for programs that needs to copy directories. These programs may fail unexpectedly if they do not handle the different options in the `CYGWIN` or `MSYS2` environment variables. If the user does not set the environment variables, the default (meaning that the user did not set the `CYGWIN` variable) behavior on Cygwin is such that it will never fail if the target of the symlink does not exist or if native symlinks are not enabled. However, the default behavior on MSYS2 is different. MSYS2 will by default _not_ create a symlink when a symlink is apparently created by calling `ln -s`. Instead it creates a copy of the target file. If the target file does not exist, `ln -s` fails.

### Epilouge

I was in the end able to install `Perl::LanguageServer` on MSYS2, but when VS Code tried to use it, it crashed. I believe it was due to the module `IO::AIO`, but I have not looked further into the issue. I also discovered a new Perl module called [PLS](https://metacpan.org/pod/PLS). This module is currently under [active development](https://www.reddit.com/r/perl/comments/o9yfbp/pls_perl_language_server/) and implements [features like](https://marketplace.visualstudio.com/items?itemName=FractalBoy.pls) auto-completion. However, it currently does not implement the [Debug Adapter Protocol](https://microsoft.github.io/debug-adapter-protocol/) so debugging is not  available. This module also installed fine from my Windows CMD prompt (it uses [IO::Async](https://metacpan.org/pod/IO::Async) instead of `IO::AIO`), but it does not run from VS Code when I tested it. From the [discussion](https://www.reddit.com/r/perl/comments/o9yfbp/pls_perl_language_server/) on reddit I guess this problem will hopefully be fixed soon.