Repository: kubernetes
Issue: Cluster internal PTR requests getting forwarded to upstream DNS servers #6315
Labels: kind/bug,sig/network,kind/support,triage/accepted

### What happened?

Issue: We are seeing lot of PTR requests getting forwarded to upstream DNS servers. All these PTR requests are trying to resolve cluster internal IPs'. These should get resolved internally to cluster and not get forwarded to upstream DNS servers. This is causing a huge amount of spamming on out upstream DNS servers. Interesting thing is, these requests are not failing and getting resolved fine.

### What did you expect to happen?

All cluster internal IPs' should get not leave cluster and get resolved in the cluster.

### How can we reproduce it (as minimally and precisely as possible)?

This is the setup we have:
Setup:
Kubernetes Cluster version: v1.22.8
coredns version - k8s.gcr.io/coredns/coredns:v1.8.0
nodelocaldns - k8s.gcr.io/dns/k8s-dns-node-cache:1.21.1

coredns configmap:

```

apiVersion: v1
data:
  Corefile: |
    .:53 {
        errors
        health {
            lameduck 5s
        }
        ready
        kubernetes cluster.local in-addr.arpa ip6.arpa {
          pods insecure
          fallthrough in-addr.arpa ip6.arpa
        }
        prometheus :9153
        forward . /etc/resolv.conf {
          prefer_udp
        }
        cache 30

        loop
        reload
        loadbalance
    }
kind: ConfigMap
``` 

### Anything else we need to know?

_No response_

### Kubernetes version

<details>

```console
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"26", GitVersion:"v1.26.2", GitCommit:"fc04e732bb3e7198d2fa44efa5457c7c6f8c0f5b", GitTreeState:"clean", BuildDate:"2023-02-22T13:39:03Z", GoVersion:"go1.19.6", Compiler:"gc", Platform:"darwin/amd64"}
Kustomize Version: v4.5.7
Server Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.8", GitCommit:"7061dbbf75f9f82e8ab21f9be7e8ffcaae8e0d44", GitTreeState:"clean", BuildDate:"2022-03-16T14:04:34Z", GoVersion:"go1.16.15", Compiler:"gc", Platform:"linux/amd64"}
```

</details>


### Cloud provider

<details>
None, these are on-prem machines
</details>


### OS version

<details>

```console
# On Linux:
$ cat /etc/os-release
NAME="Ubuntu"
VERSION="20.04.6 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.6 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal
$ uname -a
Linux ipp1-0783.ipp1u1.colossus 5.4.0-148-generic #165-Ubuntu SMP Tue Apr 18 08:53:12 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
```

</details>


### Install tools

<details>
kubespray
</details>


### Container runtime (CRI) and version (if applicable)

<details>
containerd version 1.6.4
</details>


### Related plugins (CNI, CSI, ...) and versions (if applicable)

<details>
calico - quay.io/calico/typha:v3.22.3
coredns version - k8s.gcr.io/coredns/coredns:v1.8.0
nodelocaldns - k8s.gcr.io/dns/k8s-dns-node-cache:1.21.1
</details>