Repository: containers/buildah

Issue #1709: Rootless overlay whiteouts throw error when stored
Created: 2019-07-07T02:10:18Z
Description
Deleting existing files in lower layers causes errors when running buildah/podman in rootless mode and using the overlay storage driver. Deleting the files works fine, but it seems to have issues when it later tries to use the whiteout files and fails with operation not permitted. It also has the same issue when pulling a container where a file from a lower layer is deleted. All of these have no issues when using the vfs driver instead.
Steps to reproduce the issue:
$ ctr=$(buildah from alpine)
$ buildah run $ctr rm /etc/alpine-release
$ buildah commit --rm $ctr
Getting image source signatures
Copying blob 256a7af3acb1 skipped: already exists
Copying blob d722205207d3 done
Copying config eb5739d8d7 done
Writing manifest to image destination
Storing signatures
error committing container "alpine-working-container" to "": error copying layers and metadata for container "5c77e22cdb9772b46833bf727d0c6f0139a5e4fad137286ac432fb6f72a1a520": Error committing the finished image: error adding layer with blob "sha256:d722205207d33a3bdaee8e1ab92a2bb66d31ac78dc06fb836b9d46e490dd8faa": Error processing tar file(exit status 1): operation not permitted
ERRO[0002] exit status 1                                
$ 

You can see some relevant strace output here from pulling/importing another container with the same issue. You can see it seems to have issues when it tries to create the whiteout node:
read(0, "var/cache/apt/.wh.pkgcache.bin\0\0"..., 512) = 512
lstat("/var/cache/apt", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
lstat("/var/cache/apt/.wh.pkgcache.bin", 0xc420bf52e8) = -1 ENOENT (No such file or directory)
mknodat(AT_FDCWD, "/var/cache/apt/pkgcache.bin", S_IFCHR|000, makedev(0, 0)) = -1 EPERM (Operation not permitted)
write(2, "operation not permitted", 23) = 23

buildah info:
$ buildah info
{
    "host": {
        "Distribution": {
            "distribution": "ubuntu",
            "version": "18.04"
        },
        "MemTotal": 16775352320,
        "MenFree": 6280101888,
        "SwapFree": 2147479552,
        "SwapTotal": 2147479552,
        "arch": "amd64",
        "cpus": 8,
        "hostname": "computator",
        "kernel": "4.15.0-54-generic",
        "os": "linux",
        "rootless": true,
        "uptime": "4h 40m 4.82s (Approximately 0.17 days)"
    },
    "store": {
        "ContainerStore": {
            "number": 0
        },
        "GraphDriverName": "overlay",
        "GraphOptions": null,
        "GraphRoot": "/home/computator/.local/share/containers/storage",
        "GraphStatus": {
            "Backing Filesystem": "extfs",
            "Native Overlay Diff": "false",
            "Supports d_type": "true",
            "Using metacopy": "false"
        },
        "ImageStore": {
            "number": 1
        },
        "RunRoot": "/run/user/1000"
    }
}

Duplicate issue #2326: ubuntu:18.04 container commit fails with error processing tar file: operation not permitted
Created: 2020-04-27T03:22:40Z
Description


rootless buildah and podman fails to commit ubuntu:18.04 images at applying tar. I can reproduce this only when apt-get install is run in container. Also buildah commit --squash succeeds.


ubuntu:16.04 pull fails at applying tar.


Debian, fedora works fine.


Describe the results you received:
1. fails to commit ubuntu:18.04 images at applying tar
buildah bud --format=docker -f ./Dockerfile -t ub .
DEBU Docker v2s2 manifest = {"schemaVersion":2,"mediaType":"application/vnd.docker.distribution.manifest.v2+json","config":{"mediaType":"application/vnd.docker.container.image.v1+json","size":3717,"digest":"sha256:3b5788b6bdee7cdc709873290806de0baad684ecc9bc7740b17e9c1683844180"},"layers":[{"mediaType":"application/vnd.docker.image.rootfs.diff.tar","size":65582080,"digest":"sha256:b7f7d2967507ba709dbd1dd0426a5b0cdbe1ff936c131f8958c8d0f910eea19e"},{"mediaType":"application/vnd.docker.image.rootfs.diff.tar","size":991232,"digest":"sha256:a6ebef4a95c345c844c2bf43ffda8e36dd6e053887dd6e283ad616dcc2376be6"},{"mediaType":"application/vnd.docker.image.rootfs.diff.tar","size":15872,"digest":"sha256:838a37a24627f72df512926fc846dd97c93781cf145690516e23335cc0c27794"},{"mediaType":"application/vnd.docker.image.rootfs.diff.tar","size":3072,"digest":"sha256:28ba7458d04b8551ff45d2e17dc2abb768bf6ed1a46bb262f26a24d21d8d7233"},{"mediaType":"application/vnd.docker.image.rootfs.diff.tar","size":5832704,"digest":"sha256:ce7e5a3f3cfb31126349b969076c496727db00749268346c66c4873ad5a9ea9f"}]}
DEBU Using blob info cache at /b/workspace/containers-43346/storage/cache/blob-info-cache-v1.boltdb
DEBU IsRunningImageAllowed for image containers-storage:
DEBU  Using transport "containers-storage" policy section
DEBU  Requirement 0: allowed
DEBU Overall: allowed
DEBU start reading config
DEBU finished reading config
DEBU Manifest has MIME type application/vnd.docker.distribution.manifest.v2+json, ordered candidate list [application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.v1+prettyjws, application/vnd.oci.image.manifest.v1+json, application/vnd.docker.distribution.manifest.v1+json]
DEBU ... will first try using the original manifest unmodified
DEBU Skipping blob sha256:b7f7d2967507ba709dbd1dd0426a5b0cdbe1ff936c131f8958c8d0f910eea19e (already present):
DEBU Skipping blob sha256:a6ebef4a95c345c844c2bf43ffda8e36dd6e053887dd6e283ad616dcc2376be6 (already present):
DEBU Skipping blob sha256:838a37a24627f72df512926fc846dd97c93781cf145690516e23335cc0c27794 (already present):
DEBU Skipping blob sha256:28ba7458d04b8551ff45d2e17dc2abb768bf6ed1a46bb262f26a24d21d8d7233 (already present):
DEBU reading layer "sha256:ce7e5a3f3cfb31126349b969076c496727db00749268346c66c4873ad5a9ea9f"
DEBU No compression detected
DEBU Using original blob without modification
DEBU finished reading layer "sha256:ce7e5a3f3cfb31126349b969076c496727db00749268346c66c4873ad5a9ea9f"
DEBU No compression detected
DEBU Using original blob without modification
**DEBU Applying tar in /b/workspace/containers-43346/storage/overlay/78d26bcbe7cbdb00449b6780728230b5f27d8daf4d34d397fb3bb1c5e852edf8/diff
error committing container for step {Env:[PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin] Command:run Args:[set -eux;       apt-get update       && apt-get install -y --no-install-recommends wget       && rm -rf /var/lib/apt/lists/*] Flags:[] Attrs:map[] Message:RUN set -eux;       apt-get update       && apt-get install -y --no-install-recommends wget       && rm -rf /var/lib/apt/lists/* Original:RUN  set -eux;       apt-get update       && apt-get install -y --no-install-recommends wget       && rm -rf /var/lib/apt/lists/*}: error copying layers and metadata for container "e994ee14149a1e38bde50ffa18fb8c25883009dd16c81710491106a0786ad0fe": Error committing the finished image: error adding layer with blob "sha256:ce7e5a3f3cfb31126349b969076c496727db00749268346c66c4873ad5a9ea9f": Error processing tar file(exit status 1): operation not permitted**

Same is reproducible from buildah commandline:
buildah unshare
# cat /proc/self/uid_map
         0      43346          1
         1    1000000      65536
# buildah run ubuntu-working-container-1 bash
root@69dd594c64ca:/# apt-get update && apt-get install wget -y --no-install-recommends && rm -rf /var/lib/apt/lists/*
root@69dd594c64ca:/# exit
exit
subprocess exited with status 130
subprocess exited with status 130
# buildah commit ubuntu-working-container-1 ub1
Getting image source signatures
Copying blob b7f7d2967507 skipped: already exists
Copying blob a6ebef4a95c3 skipped: already exists
Copying blob 838a37a24627 skipped: already exists
Copying blob 28ba7458d04b skipped: already exists
Copying blob ae2019e84a58 done
Copying config 799cd21096 done
Writing manifest to image destination
Storing signatures
**error committing container "ubuntu-working-container-1" to "ub1": error copying layers and metadata for container "78eac0ab7e22a20ca7b6ed5c4f5549f29f4d9a6de58ece755fcf280fca6c40b3": Error committing the finished image: error adding layer with blob "sha256:ae2019e84a588eb817fdf58f7dcf6cd592eadb21f03574e2777181ab392e9783": Error processing tar file(exit status 1): operation not permitted**

2. ubuntu:16.04 pull fails at applying tar.
buildah --log-level=debug pull ubuntu:16.04
DEBU GET https://auth.docker.io/token?scope=repository%3Alibrary%2Fubuntu%3Apull&service=registry.docker.io
DEBU GET https://registry-1.docker.io/v2/library/ubuntu/manifests/16.04
DEBU Using blob info cache at /b/workspace/containers-43346/storage/cache/blob-info-cache-v1.boltdb
DEBU Source is a manifest list; copying (only) instance sha256:a4fc0c40360ff2224db3a483e5d80e9164fe3fdce2a8439d2686270643974632 for current system
DEBU GET https://registry-1.docker.io/v2/library/ubuntu/manifests/sha256:a4fc0c40360ff2224db3a483e5d80e9164fe3fdce2a8439d2686270643974632
DEBU IsRunningImageAllowed for image docker:docker.io/library/ubuntu:16.04
DEBU  Using default policy section
DEBU  Requirement 0: allowed
DEBU Overall: allowed
DEBU Downloading /v2/library/ubuntu/blobs/sha256:005d2078bdfab5066ae941cea93f644f5fd25521849c870f4e1496f4526d1d5b
DEBU GET https://registry-1.docker.io/v2/library/ubuntu/blobs/sha256:005d2078bdfab5066ae941cea93f644f5fd25521849c870f4e1496f4526d1d5b
Getting image source signatures
DEBU Manifest has MIME type application/vnd.docker.distribution.manifest.v2+json, ordered candidate list [application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.v1+prettyjws, application/vnd.oci.image.manifest.v1+json, application/vnd.docker.distribution.manifest.v1+json]
DEBU ... will first try using the original manifest unmodified
DEBU Downloading /v2/library/ubuntu/blobs/sha256:53e3366ec435596bed2563cc882ba47ec25df6be2b1027e3243e83589c667c1e
DEBU GET https://registry-1.docker.io/v2/library/ubuntu/blobs/sha256:53e3366ec435596bed2563cc882ba47ec25df6be2b1027e3243e83589c667c1e
DEBU Skipping blob sha256:e92ed755c008afc1863a616a5ba743b670c09c1698f7328f05591932452a425f (already present):
Copying blob e92ed755c008 skipped: already exists
DEBU Downloading /v2/library/ubuntu/blobs/sha256:ee690f2d57a128744cf4c5b52646ad0ba7a5af113d9d7e0e02b62c06d35fd14c
DEBU GET https://registry-1.docker.io/v2/library/ubuntu/blobs/sha256:ee690f2d57a128744cf4c5b52646ad0ba7a5af113d9d7e0e02b62c06d35fd14c
DEBU Downloading /v2/library/ubuntu/blobs/sha256:b9fd7cb1ff8f489cf082781b0e1fe0c13b840e20147e8fc8204b4592da7c2f70
Copying blob e92ed755c008 skipped: already exists
DEBU Detected compression format gzip
Copying blob e92ed755c008 skipped: already exists
Copying blob 53e3366ec435 done
Copying blob e92ed755c008 skipped: already exists
Copying blob 53e3366ec435 done
Copying blob e92ed755c008 skipped: already exists
Copying blob 53e3366ec435 done
Copying blob b9fd7cb1ff8f done
Copying blob ee690f2d57a1 done
DEBU No compression detected
DEBU Using original blob without modification
Copying config 005d2078bd done
Writing manifest to image destination
Storing signatures
**DEBU Applying tar in /b/workspace/containers-43346/storage/overlay/df4476db7aa7e8a617e5c14ac2dd6a47876b2d46269f8155c3c633996b011b77/diff
INFO Warning: pull failed, retrying in 2s ... (1/3)**

Output of rpm -q buildah or apt list buildah:
built from source

Output of buildah version:
buildah --version
buildah version 1.14.8 (image-spec 1.0.1-dev, runtime-spec 1.0.1-dev)

Output of cat /etc/*release:
NAME="Ubuntu"
VERSION="16.04.6 LTS (Xenial Xerus)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 16.04.6 LTS"
VERSION_ID="16.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
VERSION_CODENAME=xenial
UBUNTU_CODENAME=xenial

Output of uname -a:
Linux 4.4.98+ #1 SMP Wed Mar 7 15:28:11 PST 2018 x86_64 x86_64 x86_64 GNU/Linux

Output of cat /etc/containers/storage.conf:
# ~/.config/containers/storage.conf
[storage]
driver = "overlay"
rootless_storage_path = "/b/workspace/containers-$UID/storage"

[storage.options]

[storage.options.overlay]
ignore_chown_errors = "true"
override_kernel_check = "true"
mountopt = "nodev"

# cat ~/.config/containers/containers.conf
[containers]
netns = "host"
userns = "host"
ipcns = "host"
utsns = "host"
cgroupns = "host"
cgroups = "disabled"

[engine]
cgroup_manager = "cgroupfs"
events_logger = "file"

#/etc/containers/storage.conf
[storage]
driver = "vfs"
runroot = "/var/run/containers/storage"
graphroot = "/var/lib/containers/storage"

[storage.options]

[storage.options.vfs]
ignore_chown_errors = "true"

Output of buildah --log-level=debug info:
DEBU [graphdriver] trying provided driver "overlay"
DEBU overlay: ignore_chown_errors=true
DEBU cached value indicated that overlay is supported
DEBU cached value indicated that metacopy is not being used
DEBU cached value indicated that native-diff is not being used
WARN Not using native diff for overlay, this may cause degraded performance for building images: failed to set opaque flag on middle layer: operation not permitted
DEBU backingFs=extfs, projectQuotaSupported=false, useNativeDiff=false, usingMetacopy=false
{
    "host": {
        "CgroupVersion": "v1",
        "Distribution": {
            "distribution": "ubuntu",
            "version": "16.04"
        },
        "MemTotal": 270376108032,
        "MenFree": 116001415168,
        "OCIRuntime": "crun",
        "SwapFree": 0,
        "SwapTotal": 0,
        "arch": "amd64",
        "cpus": 28,
        "hostname": "buildcontainer",
        "kernel": "4.4.98+",
        "os": "linux",
        "rootless": true,
        "uptime": "1424h 13m 23.61s (Approximately 59.33 days)"
    },
    "store": {
        "ContainerStore": {
            "number": 4
        },
        "GraphDriverName": "overlay",
        "GraphOptions": [
            "overlay.ignore_chown_errors=true",
            "overlay.mountopt=nodev"
        ],
        "GraphRoot": "/b/workspace/containers-43346/storage",
        "GraphStatus": {
            "Backing Filesystem": "extfs",
            "Native Overlay Diff": "false",
            "Supports d_type": "true",
            "Using metacopy": "false"
        },
        "ImageStore": {
            "number": 6
        },
        "RunRoot": "/run/user/43346/containers"
    }
}

Closing comment (2020-04-28T20:46:52Z): @rhatdan This seems to be related to #1709 - rootless overlay whiteouts throw error when stored. https://github.com/containers/buildah/blob/master/troubleshooting.md#6-rootless-buildah-bud-fails-when-using-overlayfs :) I am wondering how docker rootless does this?