Extract Zip/7Zip Archives w/o Disk Write
Tags: zip, 7zip, powershell
Published: 2023-01-13 20:39:02

Sometimes when traveling, you want one thing _way_ at the bottom of your suitcase but you don't want to take everything else out. 

![Mary Poppins magic bag scene](//i.imgur.com/k5QbzKE.gif)

Devs often want to do the same with a remote archive: Read the archive byte-stream from an endpoint and get one or two file(s) you need. The same strategy for zip files can save time when processing a large number of archives (remember that [disk I/O is by far the latency bottleneck](https://cs61.seas.harvard.edu/site/2019/Storage/#:~:text=The%20storage%20hierarchy,a%20single%20unit.) of all types of memory\).

In particular, Magnar Myrtveit on SuperUser.com [asked](https://superuser.com/q/1506991/35758):

> Using PowerShell 5.1, how can I download a `tar.xz` archive and extract it without writing it to disk first?
> [&hellip;]

## Challenge: Download, extract `xz` archive in PowerShell

I describe two approaches to achieve the objective. First, without writing to disk; second, writing to temporary files. (Bonus ordinary `zip` instructions at the end.)

### Without writing anything to disk

![Kids from Mary Poppins looking at amazement to Ms Poppins](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/rzxaf113pqn35nh0hec7.png)

By example, suppose we want to download/extract files from the `mingw32-dev.tar.xz` XZ archive available at [MinGW - Minimalist GNU for Windows][1].

We can download without writing to disk as follows:

```powershell
$response = Invoke-WebRequest -Uri 'https://mirrors.gigenet.com/OSDN//mingw/70554/mingwrt-5.2.1-mingw32-dev.tar.xz' -ContentType 'application/octet-stream'
```

Our target archive is available to the shell as a byte array in `$r.Content`. How to extract this?

{% embed https://github.com/adoconnection/SevenZipExtractor  %}

Using [`SevenZipExtractor`][2] C# wrapper for 7Zip, we extract as follows:

```powershell
#Download and install from nuget.org
Find-Package -Source NuGet -Name SevenZipExtractor | Install-Package -Scope CurrentUser

#Add the SevenZip assembly to our current PowerShell session
(Get-Item (Join-Path (Split-Path (Get-Package SevenZipExtractor).Source) lib/netstandard*) |
  Sort-Object { [version] ($_.Name -replace '^netstandard') })[-1] |
    Get-ChildItem -Filter *.dll -Recurse |
      ForEach-Object { Write-Host "Adding ``$($_.Name)``"; Add-Type -LiteralPath $_.FullName }

```

The `SevenZipExtractor` class includes, inter alia, the following overloaded constructor signatures:

```csharp
public SevenZipExtractor(Stream archiveStream);
public SevenZipExtractor(Stream archiveStream, string password);
public SevenZipExtractor(Stream archiveStream, SevenZipFormat format);
public SevenZipExtractor(Stream archiveStream, string password, InArchiveFormat format);
```

Here `Stream` means data-type `System.IO.Stream` and `SevenZipFormat` means type `SevenZipExtractor.SevenZipFormat`.

So we can use the SevenZipExtractor class by

```powershell
# If $response.Content -is [byte[]] then it is sufficient to do 
$sevenZipStream = [System.IO.MemoryStream]::new(($r.Content))
# However, sometimes the file server does not respect the earlier ContentType specification
# In this case, we need to convert a string from encoding
if ($response.Content -isnot [byte[]])
{
    $encoding = $response.Encoding
    $bytes = $encoding.GetBytes($response.Content)
    $sevenZipStream = [System.IO.MemoryStream]::new($bytes)
}
else
{
    $sevenZipStream = [System.IO.MemoryStream]::new($response.Content)
}
$szExtractor = New-Object -TypeName SevenZipExtractor.ArchiveFile -ArgumentList @($sevenZipStream, )
$szExtractor.Extract("$env:TEMP",$False) # Instead of $env:TEMP, wherever you want the files to go
```

[Here][3] is a working example I wrote that implements the above approach to download and install `ffmpeg`.

## Writing to temp file to disk

If we were writing the file to disk, the objective is more simple to achieve using the [`7Zip4Powershell` module][4].

```powershell
Install-Module -Name 7Zip4Powershell
Import-Module -Name 7Zip4Powershell -Global
```

{% embed https://github.com/thoemmi/7Zip4Powershell %}

However, `7Zip4PowerShell` does not implement all the overloaded method signatures of `SevenZipExtractor`.

{% embed https://devblogs.microsoft.com/commandline/tar-and-curl-come-to-windows/   %}

[Since the Windows 10 Preview Build 17063][5], `bsdtar` is included with PowerShell. To extract a tar.xz file, invoke the `tar` command with the `-‍-‍extract` (`-‍x`) option and specify the archive file name after the `-‍f` option:

```powershell
tar -xf .\whatever.xz
```

`tar` auto-detects the compression type and extracts the archive. For more verbose output, use the `-‍v` option. This option tells `tar` to display the names of the files being extracted on the terminal.

### Ordinary zip file in memory

As a bonus, let's see how it works with ZIP. Here is an example in which I find `aes.h` inside the [ffmpeg source code][6] zip archive:

```powershell
[System.Reflection.Assembly]::LoadWithPartialName('System.IO.Compression')
    $IWRresult = Invoke-WebRequest -Uri "https://ffmpeg.zeranoe.com/builds/win64/dev/ffmpeg-latest-win64-dev.zip"  -SslProtocol Tls12 -Method Get
    $zipStream = New-Object System.IO.Memorystream
    $zipStream.Write($IWRresult.Content,0,$IWRresult.Content.Length)
    $zipFile = [System.IO.Compression.ZipArchive]::new($zipStream)
    #OK, what's in the archive I just downloaded?
    #Write the archive contents to the shell output
    $zipFile.Entries | Select-Object -ExcludeProperty @('Archive','ExternalAttributes') | Format-Table #I don't care about 'Archive' or 'ExternalAttributes', so I instruct suppress those
    #oh, there's my `aes.h` inside `ffmpeg-latest-win64-dev/include/libavutil/`
    $entry =  $zipFile.GetEntry('ffmpeg-latest-win64-dev/include/libavutil/aes.h')
    #now we have a streamreader, we can do all the things
    #for example, let's output the content to the screen
    $reader = [System.IO.StreamReader]::new($entry.Open())
    Write-Host $reader.ReadToEnd()

```

This is particularly useful if you want to store a small file or a long string as a compressed ZIP embedded as a base-64 string within a script (large files or text will flood the script even when compressed). Consider a large PowerShell object that we previous [saved as a Cli Xml file](https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.utility/export-clixml?view=powershell-7.3#examples). We may want to make this object available without also passing the XML. Imagine executing the following steps in a PowerShell session:

```powershell
#Requires -Version 6.0
# Earlier, we saved the object to a file. Now we can read it back in.
[pscustomobject]@{ Greeting = "Hello World" } | Export-Clixml -Path $env:TEMP\helloworld.xml  # Imagine we did this earlier

# We can save this XML file to a zip file using Write-Zip and then encode the bytes as a base64 string
Write-Zip -Path "$env:TEMP\helloworld.xml" -OutputPath $env:TEMP\helloworld.zip -Level 9
$helloZipStream = [System.IO.FileStream]::new("$env:TEMP\helloworld.zip",[System.IO.FileMode]::Open)
$helloBytes = [byte[]]::new(($helloZipStream.Length))
$helloZipStream.Read($helloBytes, 0, $helloZipStream.Length)  # Will return the number of bytes read; note this
$helloZipStream.Close()
$helloZipStream.Dispose()  # Make the file available to the filesystem again
($helloBase64str = [System.Convert]::ToBase64String($helloBytes)) | Set-Clipboard  # We will paste this in the next script
```

This produces the base64 encoded string `UEsDBC0AAAAIAJurslaXhee3////`&hellip;`AAAABAAEAUAAAAP0AAAAAAA==`. This string is the 355-byte long content of our zip file. Now, in another script, we can load this zip file in memory like this:

```powershell
# Now we want to read the object back in
if (Test-Path -Path "$env:TEMP\helloworld.xml")
{
    $helloWorld = Import-Clixml -Path "$env:TEMP\helloworld.xml"
}
else
{
    # For those who don't have the cli xml file on disk, provide them the base64 encoded zip file with the xml as content (increaces code complexity, but reduces the length of the string)
    # This is a base64 encoded zip file containing the helloworld.xml file
    $helloZip = [System.IO.Compression.ZipArchive]::new([System.IO.MemoryStream]::new([System.Convert]::FromBase64String('UEsDBC0AAAAIAJurslaXhee3//////////8OABQAaGVsbG93b3JsZC54bWwBABAAJwEAAAAAAAC9AAAAAAAAAG2OywrCMBBF94L/EPIBSSqupC2IC3VhFSO6rnW0lSaRzoj6945vUFdz5wznMvF0vUexhAar4BMZqUgZFUlxdrXHRJZEh57WWJTgclSuKpqAYUuqCE4fwom9Eupad4zpatOVabslxK1TzGE73iTSPBDDRfbDbjS1FyRwapL7fAcOPKn+kYLLiR9SMzs4Im/cCAXFevHH/L5xyl5xYj+CFVkihw0AVX4n0xH/HcQqNPUm1vbtPo1Ycy2n+8T0ClBLAQIzAC0AAAAIAJurslaXhee3//////////8OABQAAAAAAAAAAAAAAAAAAABoZWxsb3dvcmxkLnhtbAEAEAAnAQAAAAAAAL0AAAAAAAAAUEsFBgAAAAABAAEAUAAAAP0AAAAAAA=='), 0, 355))
    $helloWorld = [System.Management.Automation.PSSerializer]::Deserialize([System.IO.StreamReader]::new($dbZip.GetEntry('helloworld.xml').Open()).ReadToEnd())
}

# Now we can use the object
$helloWorld | Write-Host -ForegroundColor Cyan
```

Another example that aims to download archive, run EXE from that archive all in memory:

```powershell
[System.Reflection.Assembly]::LoadWithPartialName('System.IO.Compression')
$IWRresult=Invoke-WebRequest -Uri 'https://ffmpeg.zeranoe.com/builds/win64/static/ffmpeg-20200424-a501947-win64-static.zip' -Method Get -SslProtocol Tls12
$zipStream = New-Object System.IO.Memorystream
$zipStream.Write($IWRresult.Content,0,$IWRresult.Content.Length)
$zipFile = [System.IO.Compression.ZipArchive]::new($zipStream)
#OK, what did I just download?
#Write the contents to the shell output
$zipFile.Entries | Select-Object -ExcludeProperty @('Archive','ExternalAttributes') | Format-Table #I don't care about 'Archive' or 'ExternalAttributes', so I instruct suppress those
#I see there is 'ffmpeg-20200424-a501947-win64-static/bin/ffmpeg.exe' entry
$zipEntry = $zipFile.GetEntry('ffmpeg-20200424-a501947-win64-static/bin/ffmpeg.exe')
$binReader = [System.IO.BinaryReader]::new($zipEntry.Open())
#need external modules `PowerShellMafia/PowerSploit` to be able to run exe from memory (without writing to disk); see comments below this code block
Invoke-ReflectivePEInjection -PEBytes $binReader.ReadBytes() -ExeArgs "Arg1 Arg2 Arg3 Arg4"
```

{% embed https://github.com/PowerShellMafia/PowerSploit %}

Please see [PowerShellMafia/PowerSploit][7] for more information about how to run an EXE from the memory. Find examples here: [GitHub Examples][8]


  [1]: https://sourceforge.net/projects/mingw/files/MinGW/Base/mingwrt/mingwrt-5.0.2/
  [2]: https://www.nuget.org/packages/SevenZipExtractor
  [3]: https://gist.github.com/mavaddat/1e75ab2df980aceac75d7d065f3210a1
  [4]: https://www.powershellgallery.com/packages/7Zip4PowerShell/
  [5]: https://devblogs.microsoft.com/commandline/tar-and-curl-come-to-windows/
  [6]: https://git.ffmpeg.org/ffmpeg.git
  [7]: https://github.com/PowerShellMafia/PowerSploit
  [8]: https://github.com/search?l=PowerShell&q=Invoke-ReflectivePEInjection&type=Code