Paper: Vocabulary for Universal Approximation: A Linguistic Perspective of Mapping Compositions (ICLR-2024)

Reviewer comment (Writing):
The paper could use a "Preliminaries" section to introduce the non-expert reader to some of the relevant concepts, such as ODEs and orientation-preserving diffeomorphisms. Instead, the paper refers to a textbook introduction to dynamical systems, which is not practical. Only slightly more than 8 / 9 pages were used, so there is almost 1 page of space left for these kinds of things.

Author response:
We thank the reviewer for the careful reading and the constructive comments and suggestions. Especially bringing the excellent paper on CMOW to our attention.

1. Preliminaries. We have revised the paper to include basic knowledge of the OP diffeomorphism, ODE, and flow map. This is beneficial to increase the readability of this paper for people who are not in the field of mathematics. The citations of references have also been revised accordingly.

Reviewer comment (Theory):
There is a disconnect between compositional approximation of functions over compact continuous domains (on which the paper's theorems rely) and functions over discrete sequences, which are more relevant to linguistics. This limitation has not been addressed or even mentioned.

Author response:
- **Discrete vs. Continuous and Compactness**. We revised the paper to address the compactness in the conclusion section. There are differences between discrete and continuous systems, such as the difference between ResNet and neural ODE. We thank the reviewer for providing the case in linguistics. For the approximation, it is possible to extend the compact domain to unbounded domains (see [this paper](https://doi.org/10.1016/j.neunet.2021.10.001) for example). However, functions on unbounded domains should satisfy certain conditions, such as having finite integrals. We are not sure if this development is related to problems in linguistics.

Reviewer comment (Theory):
In linguistics, functions are often over discrete domains such as string recognition problems or string transductions, making the notion of compactness from continuous domains not directly applicable. Embedding discrete domains into continuous ones does not fully resolve this because unbounded input sequences lead to non-compactness or require increasing precision, meaning that universal approximation results relying on compactness will fail for long sequences.

Author response:
- **Discrete vs. Continuous and Compactness**. We revised the paper to address the compactness in the conclusion section. There are differences between discrete and continuous systems, such as the difference between ResNet and neural ODE. We thank the reviewer for providing the case in linguistics. For the approximation, it is possible to extend the compact domain to unbounded domains (see [this paper](https://doi.org/10.1016/j.neunet.2021.10.001) for example). However, functions on unbounded domains should satisfy certain conditions, such as having finite integrals. We are not sure if this development is related to problems in linguistics.

Reviewer comment (Theory):
Given that the explicit goal is to connect approximation theory and linguistics (where discrete sequences matter), it is necessary to mention the caveat that the “compact domain” condition implies results only apply to bounded-length sequences. Otherwise, readers might misinterpret the scope.

Author response:
- **Discrete vs. Continuous and Compactness**. We revised the paper to address the compactness in the conclusion section. There are differences between discrete and continuous systems, such as the difference between ResNet and neural ODE. We thank the reviewer for providing the case in linguistics. For the approximation, it is possible to extend the compact domain to unbounded domains (see [this paper](https://doi.org/10.1016/j.neunet.2021.10.001) for example). However, functions on unbounded domains should satisfy certain conditions, such as having finite integrals. We are not sure if this development is related to problems in linguistics.

Reviewer comment (Writing):
Some parts of the notation are verbose or unclear. For example:  
- In line 88, \( V \) and \( V_F \) are the same, and \( F \) and \( V \) are distinguished only to allow an index \( i \) over \( F \). The set could simply be defined once and enumerated, and \(\Sigma\) might conventionally be used instead of \( V \) for the finite vocabulary.  
- In Equation 10, the notation and indexing are unclear; alternative forms could improve readability (e.g., \( h_1 * g_1 * h_2 * g_2 \dots \)).  
- In Equation 13, the meaning of the upside-down plus/minus is unexplained.

Author response:
- **Notations and structures**. We revised the notations to reduce misunderstanding, and we added a subsection to introduce basic knowledge for involved concepts. In addition, we exchanged the order of Part 1/2, and the title of Sec 3/4, which make the correspondence more clear.

Reviewer comment (Presentation):
While the authors have usefully presented the proof structure multiple times (Sections 2, 3, 4), this could be made more explicit with back references and more thorough textual explanations to improve flow between sections. The Part 1/Part 2 structure is reflected in Section 3 but not carried through to Part 4, which is potentially confusing.

Author response:
- **Notations and structures**. We revised the notations to reduce misunderstanding, and we added a subsection to introduce basic knowledge for involved concepts. In addition, we exchanged the order of Part 1/2, and the title of Sec 3/4, which make the correspondence more clear.