Repository: kubernetes-sigs/gateway-api

Issue #2513: ExemptFeatures knob doesnt work for Experimental Conformance Profile
Created: 2023-10-24T18:11:53Z
What happened:
Trying to run all the conformance tests related to Ingress by setting SupportedFeatures: suite.AllFeatures to run all tests and also setting ExemptFeatures: suite.MeshCoreFeatures to disable tests tied to mesh
Sample config
		suite.ExperimentalConformanceOptions{
			Options: suite.Options{
				Client:     mgrClient,
				RestConfig: cfg,
				// This clientset is needed in addition to the client only because
				// controller-runtime client doesn't support non CRUD sub-resources yet (https://github.com/kubernetes-sigs/controller-runtime/issues/452).
				Clientset:            k8sClientset,
				GatewayClassName:     *flags.GatewayClassName,
				Debug:                *flags.ShowDebug,
				CleanupBaseResources: *flags.CleanupBaseResources,
				SupportedFeatures:    suite.AllFeatures,
				ExemptFeatures: suite.MeshCoreFeatures,
			},
			Implementation:      *implementation,
			ConformanceProfiles: conformanceProfiles,
		})

Running with this config causes the mesh tests to run, which is not respecting the ExemptFeatures knob
2023-10-24T00:29:27.1815127Z     --- PASS: TestExperimentalConformance/MeshBasic (0.28s)
2023-10-24T00:29:27.1816429Z         --- PASS: TestExperimentalConformance/MeshBasic/0_request_to_'echo'_should_receive_a_200 (0.27s)
2023-10-24T00:29:27.1817657Z     --- FAIL: TestExperimentalConformance/MeshConsumerRoute (30.13s)
2023-10-24T00:29:27.1819337Z         --- FAIL: TestExperimentalConformance/MeshConsumerRoute/request_from_consumer_route's_namespace_modified_by_HTTPRoute (30.00s)
2023-10-24T00:29:27.1821484Z         --- PASS: TestExperimentalConformance/MeshConsumerRoute/request_not_from_consumer_route's_namespace_not_modified_by_HTTPRoute (0.10s)
2023-10-24T00:29:27.1896361Z     --- FAIL: TestExperimentalConformance/MeshFrontendHostname (30.13s)
2023-10-24T00:29:27.1899950Z         --- FAIL: TestExperimentalConformance/MeshFrontendHostname/Send_to_service_with_wrong_hostname (30.00s)
2023-10-24T00:29:27.1903928Z         --- PASS: TestExperimentalConformance/MeshFrontendHostname/Send_to_other_service_with_matching_hostname (0.10s)
2023-10-24T00:29:27.1905237Z     --- FAIL: TestExperimentalConformance/MeshFrontend (30.15s)
2023-10-24T00:29:27.1940433Z         --- FAIL: TestExperimentalConformance/MeshFrontend/Send_to_service (30.00s)
2023-10-24T00:29:27.1942228Z         --- PASS: TestExperimentalConformance/MeshFrontend/Send_to_pod_IP (0.12s)
2023-10-24T00:29:27.1943985Z     --- FAIL: TestExperimentalConformance/MeshPorts (90.15s)
2023-10-24T00:29:27.1945331Z         --- FAIL: TestExperimentalConformance/MeshPorts/Explicit_port_set,_send_to_that_port (30.00s)
2023-10-24T00:29:27.1946893Z         --- PASS: TestExperimentalConformance/MeshPorts/Explicit_port,_send_to_an_excluded_port (0.10s)
2023-10-24T00:29:27.1948228Z         --- FAIL: TestExperimentalConformance/MeshPorts/No_port_set (30.00s)
2023-10-24T00:29:27.1949408Z         --- FAIL: TestExperimentalConformance/MeshPorts/No_port_set#01 (30.00s)

What you expected to happen:
The exemptFeatures knob should work
How to reproduce it (as minimally and precisely as possible):
Here's a PR that is experiencing the issue envoyproxy/gateway#2056
Anything else we need to know?:

Duplicate issue #2347: ExemptFeatures did not skip features in experimental suite 
Created: 2023-08-25T03:17:24Z
Flaws in conformance tests

What happened:
ExemptFeatures did not skip features in experimental suite
What you expected to happen:
Features in ExemptFeatures should skip in experimental suite
How to reproduce it (as minimally and precisely as possible):
Anything else we need to know?:

Closing comment (2023-10-25T23:55:43Z): /close duplicate of #2513