Repository: vespa-engine/vespa
File: config-model/src/main/java/com/yahoo/vespa/model/admin/clustercontroller/ClusterControllerContainerCluster.java (Java)

Code before:
    protected Set<Path> unnecessaryPlatformBundles() { return UNNECESSARY_BUNDLES; }

    @Override protected boolean messageBusEnabled() { return false; }

    @Override
    public void getConfig(QrStartConfig.Builder builder) {
        super.getConfig(builder);

        var adjustment = isHostedVespa ? calculateJvmHeapAdjustment() : 0;
        // TODO: Go back to 128 MiB as base value for hosted as well after this has rolled out?
        int baseValue = isHostedVespa ? 192 : 128;
        builder.jvm.heapsize(baseValue + adjustment);
    }

    public void updateNodeCount(int nodes) {
        totalNumberOfContentNodes += nodes;
    }

    private int calculateJvmHeapAdjustment() {
        if (!adjustCCMaxHeap) return 0;

        // Heuristic to set JVM heap size for cluster controller
        // 300 nodes => need heap size of about 400 MiB, minimum heap should be 128 MiB
        // Increase in steps to avoid changes to heap size with small changes in node count.
        int adjustmentFactor = 52; // 52 MiB increase per step
        var step = Math.min(4, totalNumberOfContentNodes / 50); // max 4 steps (200+ nodes), 0 steps when < 50 nodes
        int adjustment = (step * adjustmentFactor);

        if (adjustment > 0.0) {
            log.log(INFO, "Increased cluster controller max heap size memory with " + adjustment +
                    " MiB due to having many content nodes(" + totalNumberOfContentNodes + ")");
        }
        return adjustment;
    }

    public ReindexingContext reindexingContext() { return reindexingContext; }

    private static ReindexingContext createReindexingContext(DeployState deployState) {
        return new ReindexingContext(deployState.reindexing().orElse(Reindexing.DISABLED_INSTANCE));
    }

}


Review comment (suggestion):
Consider an ideal heap size based on a linear model `f(n) = a n + b`, where `n` is the number of nodes, and `b` is the overhead at 0 nodes.

I have asked Perplexity, Gemini, and ChatGPT for the recommended comfortable heap size based on the output of `jcmd GC.heap_info` and `jstat -gc` from the incident, and a sample app with 2 content nodes.  The results are summarized as:

1. 300 a + b = 350 MiB
2. 2 a + b = 128 MiB

I'm using 350 instead of 400, because they recommended a heap size of ~350 just from the data, but since that was close enough to our current setting of (400 MiB), they recommended keeping 400.

This gives `a=0.75, b=128`.

The model naturally confirms the open source setting for small clusters of 128 MiB.  To make the transition easier, we should do `max(f(n), 192 MiB)` for now.  And because the Podman container has a max 400 MiB headroom, we should do `min(400 MiB, max(f(n), 192 MiB))`.  And then, we should do this in steps of 50 nodes, to avoid lots of restarts as in this PR (n -> `50 * (n/50)`).

The PR's heap size versus the above model:
* 0-49: 192 vs 128 (192)
* 50-99: 244 vs 166 (192)
* 100-149: 296 vs 203
* 150-199: 348, 240
* 200-249: 400, 278
* 250-299: 400, 316
* 300-349: 400, 353

Code after:
    protected Set<Path> unnecessaryPlatformBundles() { return UNNECESSARY_BUNDLES; }

    @Override protected boolean messageBusEnabled() { return false; }

    @Override
    public void getConfig(QrStartConfig.Builder builder) {
        super.getConfig(builder);

        builder.jvm.heapsize(calculateHeapSize());
    }

    public void updateNodeCount(int nodes) {
        totalNumberOfContentNodes += nodes;
    }

    private int calculateHeapSize() {
        int baseValue = 128;
        if (!isHostedVespa) return baseValue;

        return Math.min(baseValue + calculateJvmHeapAdjustment(), 400);
    }

    private int calculateJvmHeapAdjustment() {
        if (!adjustCCMaxHeap) return 0;

        // Heuristic to set JVM heap size for cluster controller
        // 300 nodes => need heap size of about 400 MiB, minimum heap should be 128 MiB
        // Increase in steps to avoid changes to heap size with small changes in node count.
        double adjustmentFactor = 0.75; // 0.75 MiB per node
        var step = totalNumberOfContentNodes / 50;
        int adjustment = (int) (step * adjustmentFactor * 50);

        if (adjustment > 0.0) {
            log.log(INFO, "Increased cluster controller max heap size memory with " + adjustment +
                    " MiB due to having many content nodes(" + totalNumberOfContentNodes + ")");
        }
        return adjustment;
    }

    public ReindexingContext reindexingContext() { return reindexingContext; }

    private static ReindexingContext createReindexingContext(DeployState deployState) {
        return new ReindexingContext(deployState.reindexing().orElse(Reindexing.DISABLED_INSTANCE));
    }

}