CVE-2022-23606 (published 2022-02-22T23:15Z)

Envoy is an open source edge and service proxy, designed for cloud-native applications. When a cluster is deleted via Cluster Discovery Service (CDS) all idle connections established to endpoints in that cluster are disconnected. A recursion was introduced in the procedure of disconnecting idle connections that can lead to stack exhaustion and abnormal process termination when a cluster has a large number of idle connections. This infinite recursion causes Envoy to crash. Users are advised to upgrade.

Severity: MEDIUM
CVSS v2 base score: 4.0 (AV:N/AC:L/Au:S/C:N/I:N/A:P)
CVSS v3 base score: 6.5, MEDIUM (CVSS:3.1/AV:N/AC:L/PR:L/UI:N/S:U/C:N/I:N/A:H)

Weakness information:
cwe_id: CWE-674
cwe_name: Uncontrolled Recursion
description: The product does not properly control the amount of recursion that takes place,  consuming excessive resources, such as allocated memory or the program stack.
extended_description: 
        
      
      

Fix information:
repo_url: https://github.com/envoyproxy/envoy
commit_details: repo_url: https://github.com/envoyproxy/envoy
author: Yan Avlasov
author_date: 2022-02-01 19:33:02+00:00
msg: CVE-2022-23606

Avoid closing other connections to prevent deep recursion when a large number of idle connections are closed at the start of a pool drain, when a connection is closed.

Signed-off-by: Yan Avlasov <yavlasov@google.com>
repository: repo_url: https://github.com/envoyproxy/envoy
repo_name: envoyproxy/envoy
description: Cloud-native high-performance edge/middle/service proxy
repo_language: C++
owner: envoyproxy
file_changes: filename: current.rst
change_type: ModificationType.MODIFY
diff: @@ -28,6 +28,7 @@ Bug Fixes
 * data plane: fixing error handling where writing to a socket failed while under the stack of processing. This should genreally affect HTTP/3. This behavioral change can be reverted by setting ``envoy.reloadable_features.allow_upstream_inline_write`` to false.
 * eds: fix the eds cluster update by allowing update on the locality of the cluster endpoints. This behavioral change can be temporarily reverted by setting runtime guard ``envoy.reloadable_features.support_locality_update_on_eds_cluster_endpoints`` to false.
 * tls: fix a bug while matching a certificate SAN with an exact value in ``match_typed_subject_alt_names`` of a listener where wildcard ``*`` character is not the only character of the dns label. Example, ``baz*.example.net`` and ``*baz.example.net`` and ``b*z.example.net`` will match ``baz1.example.net`` and ``foobaz.example.net`` and ``buzz.example.net``, respectively.
+* upstream: fix stack overflow when a cluster with large number of idle connections is removed.
 * xray: fix the AWS X-Ray tracer extension to not sample the trace if ``sampled=`` keyword is not present in the header ``x-amzn-trace-id``.
 
 Removed Config or Runtime

programming_language: Markdown
filename: BUILD
change_type: ModificationType.MODIFY
diff: @@ -14,6 +14,7 @@ envoy_cc_library(
     hdrs = ["conn_pool_base.h"],
     deps = [
         "//envoy/stats:timespan_interface",
+        "//source/common/common:debug_recursion_checker_lib",
         "//source/common/common:linked_object",
         "//source/common/stats:timespan_lib",
         "//source/common/upstream:upstream_lib",

programming_language: Ruby
filename: conn_pool_base.cc
change_type: ModificationType.MODIFY
diff: @@ -1,6 +1,7 @@
 #include "source/common/conn_pool/conn_pool_base.h"
 
 #include "source/common/common/assert.h"
+#include "source/common/common/debug_recursion_checker.h"
 #include "source/common/network/transport_socket_options_impl.h"
 #include "source/common/runtime/runtime_features.h"
 #include "source/common/stats/timespan_impl.h"
@@ -349,6 +350,8 @@ void ConnPoolImplBase::transitionActiveClientState(ActiveClient& client,
 void ConnPoolImplBase::addIdleCallbackImpl(Instance::IdleCb cb) { idle_callbacks_.push_back(cb); }
 
 void ConnPoolImplBase::closeIdleConnectionsForDrainingPool() {
+  Common::AutoDebugRecursionChecker assert_not_in(recursion_checker_);
+
   // Create a separate list of elements to close to avoid mutate-while-iterating problems.
   std::list<ActiveClient*> to_close;
 
@@ -403,11 +406,7 @@ bool ConnPoolImplBase::isIdleImpl() const {
          connecting_clients_.empty();
 }
 
-void ConnPoolImplBase::checkForIdleAndCloseIdleConnsIfDraining() {
-  if (is_draining_for_deletion_) {
-    closeIdleConnectionsForDrainingPool();
-  }
-
+void ConnPoolImplBase::checkForIdleAndNotify() {
   if (isIdleImpl()) {
     ENVOY_LOG(debug, "invoking idle callbacks - is_draining_for_deletion_={}",
               is_draining_for_deletion_);
@@ -417,6 +416,14 @@ void ConnPoolImplBase::checkForIdleAndCloseIdleConnsIfDraining() {
   }
 }
 
+void ConnPoolImplBase::checkForIdleAndCloseIdleConnsIfDraining() {
+  if (is_draining_for_deletion_) {
+    closeIdleConnectionsForDrainingPool();
+  }
+
+  checkForIdleAndNotify();
+}
+
 void ConnPoolImplBase::onConnectionEvent(ActiveClient& client, absl::string_view failure_reason,
                                          Network::ConnectionEvent event) {
   if (client.state() == ActiveClient::State::CONNECTING) {
@@ -487,7 +494,15 @@ void ConnPoolImplBase::onConnectionEvent(ActiveClient& client, absl::string_view
 
     dispatcher_.deferredDelete(client.removeFromList(owningList(client.state())));
 
-    checkForIdleAndCloseIdleConnsIfDraining();
+    // Check if the pool transitioned to idle state after removing closed client
+    // from one of the client tracking lists.
+    // There is no need to check if other connections are idle in a draining pool
+    // because the pool will close all idle connection when it is starting to
+    // drain.
+    // Trying to close other connections here can lead to deep recursion when
+    // a large number idle connections are closed at the start of pool drain.
+    // See CdsIntegrationTest.CdsClusterDownWithLotsOfIdleConnections for an example.
+    checkForIdleAndNotify();
 
     client.setState(ActiveClient::State::CLOSED);
 

programming_language: C++
filename: conn_pool_base.h
change_type: ModificationType.MODIFY
diff: @@ -6,6 +6,7 @@
 #include "envoy/stats/timespan.h"
 #include "envoy/upstream/cluster_manager.h"
 
+#include "source/common/common/debug_recursion_checker.h"
 #include "source/common/common/dump_state_utils.h"
 #include "source/common/common/linked_object.h"
 
@@ -222,6 +223,9 @@ public:
   void onConnectionEvent(ActiveClient& client, absl::string_view failure_reason,
                          Network::ConnectionEvent event);
 
+  // Check if the pool has gone idle and invoke idle notification callbacks.
+  void checkForIdleAndNotify();
+
   // See if the pool has gone idle. If we're draining, this will also close idle connections.
   void checkForIdleAndCloseIdleConnsIfDraining();
 
@@ -371,6 +375,7 @@ private:
   bool deferred_deleting_{false};
 
   Event::SchedulableCallbackPtr upstream_ready_cb_;
+  Common::DebugRecursionChecker recursion_checker_;
 };
 
 } // namespace ConnectionPool

programming_language: C++
filename: utility.cc
change_type: ModificationType.MODIFY
diff: @@ -442,6 +442,34 @@ envoy::config::cluster::v3::Cluster ConfigHelper::buildStaticCluster(const std::
                   name, name, address, port, lb_policy));
 }
 
+envoy::config::cluster::v3::Cluster ConfigHelper::buildH1ClusterWithHighCircuitBreakersLimits(
+    const std::string& name, int port, const std::string& address, const std::string& lb_policy) {
+  return TestUtility::parseYaml<envoy::config::cluster::v3::Cluster>(
+      fmt::format(R"EOF(
+      name: {}
+      connect_timeout: 50s
+      type: STATIC
+      circuit_breakers:
+        thresholds:
+        - priority: DEFAULT
+          max_connections: 10000
+          max_pending_requests: 10000
+          max_requests: 10000
+          max_retries: 10000
+      load_assignment:
+        cluster_name: {}
+        endpoints:
+        - lb_endpoints:
+          - endpoint:
+              address:
+                socket_address:
+                  address: {}
+                  port_value: {}
+      lb_policy: {}
+    )EOF",
+                  name, name, address, port, lb_policy));
+}
+
 envoy::config::cluster::v3::Cluster ConfigHelper::buildCluster(const std::string& name,
                                                                const std::string& lb_policy) {
   API_NO_BOOST(envoy::config::cluster::v3::Cluster) cluster;

programming_language: C++
filename: utility.h
change_type: ModificationType.MODIFY
diff: @@ -153,6 +153,11 @@ public:
   buildStaticCluster(const std::string& name, int port, const std::string& address,
                      const std::string& lb_policy = "ROUND_ROBIN");
 
+  static envoy::config::cluster::v3::Cluster
+  buildH1ClusterWithHighCircuitBreakersLimits(const std::string& name, int port,
+                                              const std::string& address,
+                                              const std::string& lb_policy = "ROUND_ROBIN");
+
   // ADS configurations
   static envoy::config::cluster::v3::Cluster
   buildCluster(const std::string& name, const std::string& lb_policy = "ROUND_ROBIN");

programming_language: C++
filename: BUILD
change_type: ModificationType.MODIFY
diff: @@ -110,10 +110,12 @@ envoy_proto_library(
 
 envoy_cc_test(
     name = "cds_integration_test",
+    size = "large",
     srcs = ["cds_integration_test.cc"],
     data = [
         "//test/config/integration/certs",
     ],
+    shard_count = 4,
     deps = [
         ":http_integration_lib",
         "//source/common/config:protobuf_link_hacks",

programming_language: Ruby
filename: cds_integration_test.cc
change_type: ModificationType.MODIFY
diff: @@ -37,7 +37,8 @@ public:
                                 sotwOrDelta() == Grpc::SotwOrDelta::Sotw ||
                                         sotwOrDelta() == Grpc::SotwOrDelta::UnifiedSotw
                                     ? "GRPC"
-                                    : "DELTA_GRPC")) {
+                                    : "DELTA_GRPC")),
+        cluster_creator_(&ConfigHelper::buildStaticCluster) {
     if (sotwOrDelta() == Grpc::SotwOrDelta::UnifiedSotw ||
         sotwOrDelta() == Grpc::SotwOrDelta::UnifiedDelta) {
       config_helper_.addRuntimeOverride("envoy.reloadable_features.unified_mux", "true");
@@ -79,14 +80,14 @@ public:
     // Create the regular (i.e. not an xDS server) upstreams. We create them manually here after
     // initialize() because finalize() expects all fake_upstreams_ to correspond to a static
     // cluster in the bootstrap config - which we don't want since we're testing dynamic CDS!
-    addFakeUpstream(Http::CodecType::HTTP2);
-    addFakeUpstream(Http::CodecType::HTTP2);
-    cluster1_ = ConfigHelper::buildStaticCluster(
+    addFakeUpstream(upstream_codec_type_);
+    addFakeUpstream(upstream_codec_type_);
+    cluster1_ = cluster_creator_(
         ClusterName1, fake_upstreams_[UpstreamIndex1]->localAddress()->ip()->port(),
-        Network::Test::getLoopbackAddressString(ipVersion()));
-    cluster2_ = ConfigHelper::buildStaticCluster(
+        Network::Test::getLoopbackAddressString(ipVersion()), "ROUND_ROBIN");
+    cluster2_ = cluster_creator_(
         ClusterName2, fake_upstreams_[UpstreamIndex2]->localAddress()->ip()->port(),
-        Network::Test::getLoopbackAddressString(ipVersion()));
+        Network::Test::getLoopbackAddressString(ipVersion()), "ROUND_ROBIN");
 
     // Let Envoy establish its connection to the CDS server.
     acceptXdsConnection();
@@ -133,6 +134,10 @@ public:
   envoy::config::cluster::v3::Cluster cluster2_;
   // True if we decided not to run the test after all.
   bool test_skipped_{true};
+  Http::CodecType upstream_codec_type_{Http::CodecType::HTTP2};
+  std::function<envoy::config::cluster::v3::Cluster(const std::string&, int, const std::string&,
+                                                    const std::string&)>
+      cluster_creator_;
 };
 
 INSTANTIATE_TEST_SUITE_P(IpVersionsClientTypeDelta, CdsIntegrationTest,
@@ -343,5 +348,160 @@ TEST_P(CdsIntegrationTest, VersionsRememberedAfterReconnect) {
   ASSERT_TRUE(codec_client_->waitForDisconnect());
 }
 
+// This test verifies that Envoy can delete a cluster with a lot of idle connections.
+// The original problem was recursive closure of idle connections that can run out
+// of stack when there are a lot of idle connections.
+TEST_P(CdsIntegrationTest, CdsClusterDownWithLotsOfIdleConnections) {
+  constexpr int num_requests = 2000;
+  // Make upstream H/1 so it creates connection for each request
+  upstream_codec_type_ = Http::CodecType::HTTP1;
+  // Relax default circuit breaker limits and timeouts so Envoy can accumulate a lot of idle
+  // connections
+  cluster_creator_ = &ConfigHelper::buildH1ClusterWithHighCircuitBreakersLimits;
+  config_helper_.addConfigModifier(
+      [&](envoy::extensions::filters::network::http_connection_manager::v3::HttpConnectionManager&
+              hcm) -> void {
+        hcm.mutable_route_config()
+            ->mutable_virtual_hosts(0)
+            ->mutable_routes(0)
+            ->mutable_route()
+            ->mutable_timeout()
+            ->set_seconds(600);
+        hcm.mutable_route_config()
+            ->mutable_virtual_hosts(0)
+            ->mutable_routes(0)
+            ->mutable_route()
+            ->mutable_idle_timeout()
+            ->set_seconds(600);
+      });
+  initialize();
+  std::vector<IntegrationStreamDecoderPtr> responses;
+  std::vector<FakeHttpConnectionPtr> upstream_connections;
+  std::vector<FakeStreamPtr> upstream_requests;
+  codec_client_ = makeHttpConnection(makeClientConnection((lookupPort("http"))));
+  // The first loop establishes a lot of open connections with active requests to upstream
+  for (int i = 0; i < num_requests; ++i) {
+    Http::TestRequestHeaderMapImpl request_headers{{":method", "GET"},
+                                                   {":path", "/cluster1"},
+                                                   {":scheme", "http"},
+                                                   {":authority", "host"},
+                                                   {"x-lyft-user-id", absl::StrCat(i)}};
+
+    auto response = codec_client_->makeHeaderOnlyRequest(request_headers);
+    responses.push_back(std::move(response));
+
+    FakeHttpConnectionPtr fake_upstream_connection;
+    waitForNextUpstreamConnection({UpstreamIndex1}, TestUtility::DefaultTimeout,
+                                  fake_upstream_connection);
+    // Wait for the next stream on the upstream connection.
+    FakeStreamPtr upstream_request;
+    AssertionResult result =
+        fake_upstream_connection->waitForNewStream(*dispatcher_, upstream_request);
+    RELEASE_ASSERT(result, result.message());
+    // Wait for the stream to be completely received.
+    result = upstream_request->waitForEndStream(*dispatcher_);
+    RELEASE_ASSERT(result, result.message());
+    upstream_connections.push_back(std::move(fake_upstream_connection));
+    upstream_requests.push_back(std::move(upstream_request));
+  }
+
+  // This loop completes all requests making the all upstream connections idle
+  for (int i = 0; i < num_requests; ++i) {
+    // Send response headers, and end_stream if there is no response body.
+    upstream_requests[i]->encodeHeaders(default_response_headers_, true);
+    // Wait for the response to be read by the codec client.
+    RELEASE_ASSERT(responses[i]->waitForEndStream(), "unexpected timeout");
+    ASSERT_TRUE(responses[i]->complete());
+    EXPECT_EQ("200", responses[i]->headers().getStatusValue());
+  }
+
+  test_server_->waitForCounterGe("cluster_manager.cluster_added", 1);
+
+  // Tell Envoy that cluster_1 is gone. Envoy will try to close all idle connections
+  EXPECT_TRUE(compareDiscoveryRequest(Config::TypeUrl::get().Cluster, "55", {}, {}, {}));
+  sendDiscoveryResponse<envoy::config::cluster::v3::Cluster>(Config::TypeUrl::get().Cluster, {}, {},
+                                                             {ClusterName1}, "42");
+  // We can continue the test once we're sure that Envoy's ClusterManager has made use of
+  // the DiscoveryResponse that says cluster_1 is gone.
+  test_server_->waitForCounterGe("cluster_manager.cluster_removed", 1);
+
+  // If we made it this far then everything is ok.
+  for (int i = 0; i < num_requests; ++i) {
+    AssertionResult result = upstream_connections[i]->close();
+    RELEASE_ASSERT(result, result.message());
+    result = upstream_connections[i]->waitForDisconnect();
+    RELEASE_ASSERT(result, result.message());
+  }
+  upstream_connections.clear();
+  cleanupUpstreamAndDownstream();
+  ASSERT_TRUE(codec_client_->waitForDisconnect());
+}
+
+// This test verifies that Envoy can delete a cluster with a lot of connections in the connecting
+// state and associated pending requests. The recursion guard in the
+// ConnPoolImplBase::closeIdleConnectionsForDrainingPool() would fire if it was called recursively.
+//
+// Test is currently disabled as there is presently no reliable way of making upstream connections
+// hang in connecting state.
+TEST_P(CdsIntegrationTest, DISABLED_CdsClusterDownWithLotsOfConnectingConnections) {
+  // Use low number of pending connections to prevent bumping into the default
+  // limit of 128, since the upstream will be prevented below from
+  // accepting connections.
+  constexpr int num_requests = 64;
+  // Make upstream H/1 so it creates connection for each request
+  upstream_codec_type_ = Http::CodecType::HTTP1;
+  cluster_creator_ = &ConfigHelper::buildH1ClusterWithHighCircuitBreakersLimits;
+  config_helper_.addConfigModifier(
+      [&](envoy::extensions::filters::network::http_connection_manager::v3::HttpConnectionManager&
+              hcm) -> void {
+        hcm.mutable_route_config()
+            ->mutable_virtual_hosts(0)
+            ->mutable_routes(0)
+            ->mutable_route()
+            ->mutable_timeout()
+            ->set_seconds(600);
+        hcm.mutable_route_config()
+            ->mutable_virtual_hosts(0)
+            ->mutable_routes(0)
+            ->mutable_route()
+            ->mutable_idle_timeout()
+            ->set_seconds(600);
+      });
+  initialize();
+  test_server_->waitForCounterGe("cluster_manager.cluster_added", 1);
+  std::vector<IntegrationStreamDecoderPtr> responses;
+  codec_client_ = makeHttpConnection(makeClientConnection((lookupPort("http"))));
+  // Stop upstream at UpstreamIndex1 dispatcher, to prevent it from accepting TCP connections.
+  // This will cause Envoy's connections to that upstream hang in the connecting state.
+  fake_upstreams_[UpstreamIndex1]->dispatcher()->exit();
+  for (int i = 0; i < num_requests; ++i) {
+    Http::TestRequestHeaderMapImpl request_headers{{":method", "GET"},
+                                                   {":path", "/cluster1"},
+                                                   {":scheme", "http"},
+                                                   {":authority", "host"},
+                                                   {"x-lyft-user-id", absl::StrCat(i)}};
+
+    auto response = codec_client_->makeHeaderOnlyRequest(request_headers);
+    responses.push_back(std::move(response));
+  }
+
+  // Wait for Envoy to try to establish all expected connections
+  test_server_->waitForCounterEq("cluster.cluster_1.upstream_cx_total", num_requests);
+
+  // Tell Envoy that cluster_1 is gone. Envoy will try to close all pending connections
+  EXPECT_TRUE(compareDiscoveryRequest(Config::TypeUrl::get().Cluster, "55", {}, {}, {}));
+  sendDiscoveryResponse<envoy::config::cluster::v3::Cluster>(Config::TypeUrl::get().Cluster, {}, {},
+                                                             {ClusterName1}, "42");
+  // We can continue the test once we're sure that Envoy's ClusterManager has made use of
+  // the DiscoveryResponse that says cluster_1 is gone.
+  test_server_->waitForCounterGe("cluster_manager.cluster_removed", 1);
+
+  cleanupUpstreamAndDownstream();
+  ASSERT_TRUE(codec_client_->waitForDisconnect());
+  // If we got here it means that the recursion guard in the
+  // ConnPoolImplBase::closeIdleConnectionsForDrainingPool() did not fire, which is what this test
+  // validates.
+}
+
 } // namespace
 } // namespace Envoy

programming_language: C++