Paper: Computing Low-Entropy Couplings for Large-Support Distributions (ICLR-2024)

Reviewer comment (Experiments):
The experiments seem to be heavily centered around specific domains like Markov coding games and steganography. While these are valuable explorations, they might not give a complete picture of ARIMEC's versatility across other potential domains or applications. I'd like a bit more discussion on the potential applications of ARIMEC.

Author response:
We agree that such a discussion is warranted. To address this concern, we added the following paragraph about additional potential applications for ARIMEC to the conclusion: 

"For future work, there are a few application directions in which it would be interesting to push further with ARIMEC. First is unencrypted steganography. This direction is exciting because ARIMEC can achieve high throughput rates, as we observed in Figure 5, and because minimum-entropy coupling's usage for steganography was only recognized recently Schroeder de Witt et al. (2023). Thus, there may be real-world settings in which it is applicable, especially since unencrypted steganography requires no key exchange. Second, because ARIMEC is the first IMEC algorithm capable of handling arbitrary discrete distributions, it opens the door to using large support distributions for classical minimum-entropy coupling applications in which the distributions may be non-factorable, such as entropic causal inference, random number generation, functional representations, and dimensionality reduction."

Reviewer comment (Evaluation):
Figure 3, why do you compare Token-wise Error Rate?

Author response:
We are trying to measure the error rate at which the language model text is decoded properly, as this is the metric of interest in the steganography use case of minimum-entropy coupling. This error rate is related to joint entropy, but not exactly the same, so it requires separate treatment.

Reviewer comment (Theory):
**Notion of efficiency is never formally defined.** The main reason for using heuristic MEC algorithms instead of the provable approximation algorithms is to avoid the log-linear-in-N run time. Computational efficiency, at least in CS, is always defined relative to some problem instance size (i.e., it is an asymptotic notion). If N is “too big”, then what is the right index for the instance size for MEC? This would make sense if one explicitly assumed that X is supported on {0,1}^n and Y is supported on {0,1}^m, and we call any quantity “too large” or “intractably large” if it grows superpolynomially in n or m.

Author response:
We thank the reviewer for this feedback. We agree that the way we were presenting the material previously was lacking formality. We attempted to address the reviewer's concerns in the revised version of the submission. To summarize:
- We now describe algorithm runtime using big-O analysis.
- We removed all of the "conditions" from our presentation of the material (including but not limited to those mentioned as of concern by the reviewer).
- Regarding Algorithm 3 in its generic form, it is *not* the case that optimization over the collection of partitions U can be implemented efficiently. To make this more clear, we have added the text: "Note that whether Algorithm 3 can be implemented efficiently depends on the distribution $\nu$ and the set of partitions $\mathfrak{U}$."
- Following up on the previous point:
    1. For TIMEC, optimization over the collection of partitions can be implemented efficiently (the finest partition will always have maximum entropy). 
    2. For FIMEC, optimization over the collection partitions can be implemented efficiently under the assumption that the distribution is factorable.
    3. For ARIMEC, it is less obvious that optimization over the collection of partitions can be implemented efficiently. To address this issue, we introduced a pruning technique (which was in the appendix of the original submission) that can prove large parts of the prefix partition tree cannot induce a maximum-entropy partition. While we do not show a polynomial time guarantee for this pruning procedure, we found that, for all of the distributions that we have tried in the experiments, it only required checking a small number of partitions (on the order of 1) on average in order to find a provably maximum-entropy partition.

Reviewer comment (Theory):
**Missing run-time analysis of subroutines.** In Algorithm 3 (IMEC), is it clear that the optimization over the collection of partitions U can be implemented efficiently?

Author response:
We thank the reviewer for this feedback. We agree that the way we were presenting the material previously was lacking formality. We attempted to address the reviewer's concerns in the revised version of the submission. To summarize:
- We now describe algorithm runtime using big-O analysis.
- We removed all of the "conditions" from our presentation of the material (including but not limited to those mentioned as of concern by the reviewer).
- Regarding Algorithm 3 in its generic form, it is *not* the case that optimization over the collection of partitions U can be implemented efficiently. To make this more clear, we have added the text: "Note that whether Algorithm 3 can be implemented efficiently depends on the distribution $\nu$ and the set of partitions $\mathfrak{U}$."
- Following up on the previous point:
    1. For TIMEC, optimization over the collection of partitions can be implemented efficiently (the finest partition will always have maximum entropy). 
    2. For FIMEC, optimization over the collection partitions can be implemented efficiently under the assumption that the distribution is factorable.
    3. For ARIMEC, it is less obvious that optimization over the collection of partitions can be implemented efficiently. To address this issue, we introduced a pruning technique (which was in the appendix of the original submission) that can prove large parts of the prefix partition tree cannot induce a maximum-entropy partition. While we do not show a polynomial time guarantee for this pruning procedure, we found that, for all of the distributions that we have tried in the experiments, it only required checking a small number of partitions (on the order of 1) on average in order to find a provably maximum-entropy partition.

Reviewer comment (Novelty):
**Motivation for ARIMEC.** In what sense is ARIMEC better than previous approaches? In the regimes where TIMEC and FIMEC performs well, is it expected that ARIMEC performs not worse than these two approaches? Also, what is the motivation behind using partitions defined using the prefix tree?

Author response:
We attempt to make the motivation clear in the abstract: "We derive a new IMEC instance from this formalism, which we call ARIMEC, that, unlike existing IMEC algorithms, can be applied in practice to arbitrary discrete distributions, and introduce associated operations that make ARIMEC efficient in practice." To state in different words, ARIMEC is the first algorithm that can produce couplings of large support distributions of arbitrary structure in practice. This contrasts with FIMEC, which requires one of the two distributions to be factorable. Please let us know if there is a better way we can characterize this motivation.

Reviewer comment (Experiments):
**Missing details on the steganography task.** Details of the steganography experiment are missing (even including Appendix D), which makes it hard to understand what experiment is exactly being conducted here. Could the authors please expand on the last paragraph of Section 5?

Author response:
We apologize that the details here were unclear. To clarify:
1. For the information-theoretic steganography experiments, the task is to perform a coupling between uniformly randomly distributed byte strings and the distribution over 100 tokens induced by GPT-2. The error rate is the proportion of the time that the MAP byte string is not equal to the true byte string. Symbolically $E_{X} E_{Y \sim \gamma(Y \mid X)} I[X \neq \arg \max_{x} \gamma(x \mid Y)]$. We have added this clarification about error rate to the main text.
2. For unencrypted steganography, the task is to perform a coupling between the distribution over 100 tokens induced by GPT-2 conditioned on two different prompts. Error rate is defined the same way.

Please let us know if there is any additional detail that would be helpful to add.

Reviewer comment (Theory):
If the encoding is perfectly secure (i.e., encoding of ciphertext X into stegotext S is deterministic) and the distribution of stegotext S and covertext C is the same, doesn’t this mean that one can “hallucinate” secret messages from innocuous text?

Author response:
Naively, yes, if the receiver is expecting a message when no message is actually being sent, this is exactly what will occur. However, in practice, one can mitigate this possibility using standard error detection techniques. For this reason, the information-theoretic steganography model introduced by Cachin (1998), which we used in the submission, is common in steganography literature.