Repository: kubernetes
Issue: Unable to use k8s service cluster IP with ipencap/IPIP
Labels: kind/bug,sig/network,sig/node,needs-triage

### What happened?

Hello, this is likely not a bug as I am trying to do something mildly unholy and likely need to use some other k8s knob but I wasn't able to make an account on the discuss forum for some reason so here I am. I assume the issue is ipencap technically doesn't have a port as part of the protocol while the inner packet does so the service never gets hit but I'll explain my case as someone might have a better idea as I could easily be missing something much simpler/easier to do since I do not have a ton of kubernetes knowledge. I am also using k3s which I wouldn't expect to matter for services but if that's an auto rejection then fair enough.

In my case I have two pods: Pod A and Pod B where each has a container with IPs setup on an iface named eth0 at 100.70.0.X/24 (so let's just say 100.70.0.1 and 100.70.0.2). They also have cluster IP services setup at 200.70.0.X/24 (so again let's say 200.70.0.1 and 200.70.0.2). Traffic going to a different pod's cluster IP goes through the eth0 IPs so packets going to an open service port of 5000 would end up looking something like 
`<IP src=100.70.0.1 dst=200.70.0.2 dport=5000>` which works.

I am attempting to use ipencap to get traffic from a container in pod A to go to a container in pod B to then route that packet out to the wider world as I require a specific service and interface setup in the container in pod B. Trying to use the static cluster IP fails while using the eth0 IPs directly works with the problem being the eth0 IPs are set by an outside source and change on container restart while I need something static while keeping the packet relatively simple protocol wise. It's where what I'm doing is vaguely router/VPN-ish but not quite and it's not all the traffic, just very specific traffic.

There are good/dumb reasons for this where it's complicated to wrap this packet in something more complicated due to what the receiving service is expecting although if someone has a suggestion I'm happy to hear it. I explored trying to use GRE or vxlan with similar results where the cluster IP doesn't seem to work (along with further issues in my case as setting things up across the outside host to have a shared vxlan or something is complicated).

If I do direct eth0 IPs then this works as I expect where if port 5000 is defined I can do things like `curl http://200.75.0.2:5000`. But setting up an ipencap packet like below where I'm wanting to go to an outside IP of like 10.20.30.40 doesn't:

`<Outer IP src=100.70.0.1 dst=200.75.0.2 | Inner IP src=100.70.0.1 dst=10.20.30.40 dport=5000>`

I setup something using scapy to mess with this but I haven't been able to get past what I think is how this internally handles the vip and ports.


### What did you expect to happen?

I create an IPIP packet using a cluster IP address with an exposed port in the inner packet and can receive this packet.

### How can we reproduce it (as minimally and precisely as possible)?

scapy makes this easy with something like below for a UDP packet.

```python3
>>> from scapy.all import *
>>> inner_ip_udp = IP(src="100.70.0.1", dst="10.20.30.40")/UDP(sport=2134, dport=5000)/"Hello World!"
>>> outer_ip_udp = IP(src="100.70.0.1", dst="200.70.0.2")/inner_ip_udp
```

Otherwise can mess with ip link but where again using the eth0 IPs works but the cluster IPs does not since I am 99% sure the port is the requirement to work.

```bash
ip link add mytun type ipip local 100.70.0.1 remote 200.70.0.2
ip link set mytun up
```

Then just force something like `curl http://10.20.30.40:5000 --interface mytun` and using the eth0 IPs I see the packet but with cluster IP I do not.

### Anything else we need to know?

As mentioned this is running with k3s and it's too complicated to change my setup for the service I'm trying to reach to switch away from k3s to see if this would work in k8s or something else for whatever reason. 

And thanks to whoever reads all this!

### Kubernetes version

<details>

```console
$ kubectl version
Client Version: v1.30.3+k3s1
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.30.3+k3s1
```

</details>


### Cloud provider

<details>
This is running on a local machine so no cloud provider
</details>


### OS version

<details>

```console
PRETTY_NAME="Ubuntu 24.04.1 LTS"
NAME="Ubuntu"
VERSION_ID="24.04"
VERSION="24.04.1 LTS (Noble Numbat)"
VERSION_CODENAME=noble
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=noble
LOGO=ubuntu-logo
```

</details>


### Install tools

<details>

</details>


### Container runtime (CRI) and version (if applicable)

<details>

</details>


### Related plugins (CNI, CSI, ...) and versions (if applicable)

<details>

</details>