Repository: kubernetes
Issue: Kubernetes should configure the ambient capability set
Labels: sig/node,kind/feature,lifecycle/frozen,sig/security

> /kind bug

**What happened**:

The following takes place on a k8s 1.8.2 cluster.

I have a Docker container image that wants to listen on :80, and specifies a non-root USER. To get this running, in my pod spec the container has the following security context:

```yaml
securityContext:
    capabilities:
        drop:
        - all
        add:
        - NET_BIND_SERVICE
    allowPrivilegeEscalation: false
```

When I schedule this pod on the cluster, the container fails to bind to :80 (permission denied), and goes into a crashloop. Note that Kubernetes did not complain that this configuration is in any way infeasible.

The reason for this is that Linux capabilities interact in surprising ways with other security mechanisms. In this case, the problem is that I'm also running the container as a non-root user, and Kubernetes/Docker are only setting the inherited, permitted, effective and bounding capability sets. The catch is: the effective and permitted sets get _cleared_ when you transition from UID 0 to UID !0, so my container ends up with:

```
CapInh:	0000000000000400
CapPrm:	0000000000000000
CapEff:	0000000000000000
CapBnd:	0000000000000400
CapAmb:	0000000000000000
NoNewPrivs:	1
```

0x400 is CAP_NET_BIND_SERVICE, and as you can see my effective capabilities do not have this bit set.

The linux kernel corrected this very confusing behavior by introducing the ambient capability set, which does not have surprising behaviors when you transition from UID 0 to !0. If you've set a capability as ambient, you keep it unless you explicitly revoke it.

**What you expected to happen**:

I expect the capabilities I assign in my podspec to still exist when my main binary `exec`s, regardless of other security context configuration (assuming k8s accepted my manifest as valid). To me, that translates to: k8s should be writing the caps described by `securityContext.capabilities` into the ambient capability set, as well as the other capability sets.

Alternatively, if you believe the current behavior of `securityContext.capabilities` is working as intended, there should be another knob somewhere that I can use to populate the ambient capability set. However, I would strongly encourage you to instead consider the current behavior of `securityContext.capabilities` combined with non-root users as a bug, because it will likely trip up ~everyone using it unless they know a lot about the linux capability implementation.

**How to reproduce it (as minimally and precisely as possible)**:

Deploy this pod to a cluster using the default container runtime. You should see it crashlooping, with `kubectl logs bug-demo` showing that netcat is not allowed to bind to :80. If you comment out `runAsUser` and let the container binary run as root, it'll work fine. Similarly, if you modify the container to have a binary that has been altered with `setcap net_bind_service=+ep`, the contianer will run correctly as !root, because the setcap'd binary allows the container to regain the privileges it lost when transitioning out of UID 0.

```yaml
apiVersion: v1
kind: Pod
metadata:
  name: bug-demo
spec:
  containers:
  - name: netcat
    image: danderson/bug-demo:latest
    args:
    - /bin/sh
    - -c
    - "netcat -l -p 80"
    securityContext:
      runAsUser: 65534
      capabilities:
        drop:
        - all
        add:
        - NET_BIND_SERVICE
      allowPrivilegeEscalation: false
```

**Anything else we need to know?**:

**Environment**:
- Kubernetes version (use `kubectl version`): 1.8.2 server, 1.8.1 kubectl
- Cloud provider or hardware configuration: bare metal cluster, single node (master taint removed), set up with `kubeadm`.
- OS (e.g. from /etc/os-release): Debian Testing
- Kernel (e.g. `uname -a`): Linux pandora 4.12.0-2-amd64 #1 SMP Debian 4.12.13-1 (2017-09-19) x86_64 GNU/Linux
- Install tools: `kubeadm`
- Others: