Paper: A Joint Spectro-Temporal Relational Thinking Based Acoustic Modeling Framework (ICLR-2024)

Reviewer comment (Reproducibility):
Are the baselines trained with cross-entropy objective?

Author response:
**AR:** No, the baselines used in our experiments were not trained with the cross-entropy objective. Instead, when we fine-tuned the baselines, we followed Baevski et al. (2020) and optimized the standard connectionist temporal classification (CTC) loss function (Graves et al., 2006). CTC is a commonly employed method for end-to-end sequence modeling tasks in the field of speech processing.

Reviewer comment (Theory):
Would it make sense to impose left to right constraint between the time steps for causality?

Author response:
**AR:** We agree that it is essential to consider causality when modeling sequences. In our proposed relational thinking modeling, we have already taken causality into account. As stated at the beginning of Section 3.1, for each time step $t$, we constructed the feature map $\\mathbf{C}\_t$ by incorporating the acoustic feature vectors from the current time step and the previous $w - 1$ time steps as
$$
\\mathbf{C}\_t = [\\mathbf{c}\_{t - w + 1}, \\ldots, \\mathbf{c}\_t]. \\quad \\quad \\quad \text{(e)}
$$
This formulation (e) explicitly excluded acoustic feature vectors from future time steps. Essentially, this aligned with the concept of a left-to-right constraint by considering only past and current information and not incorporating information from future time steps.

Reviewer comment (Evaluation):
PER analysis at the speaker level may give further intuition on how the model performs as compared to human perceptions.

Author response:
**AR:** We appreciate the reviewer's suggestion regarding PER analysis at the speaker level. It is indeed a valuable perspective for understanding how the model's performance compares to human perceptions. In this particular study, we adhered to the established protocol of reporting PER over the entire split to maintain consistency with previous research in phoneme recognition, ensuring a fair comparison. However, we acknowledge the significance of speaker-level analysis and plan to explore this approach in our future work.

Reviewer comment (Experiments):
It would be good to see Fig.5 repeated with t1f8 and t8f1 models.

Author response:
**AR:** As indicated in Table 1, the two joint spectro-temporal models, w20-t4f2 and w20-t2f4, demonstrated superior performance compared to the purely temporal and spectral models, w20-t8f1 and w20-t1f8. This suggested that the spectro-temporal models had a better grasp of the inherent relational information involved in speech. Therefore, we believed it would be more beneficial to conduct further analyses on these joint spectro-temporal models rather than the purely temporal or spectral model to gain more accurate and valuable insights.

Reviewer comment (Theory):
Does feature mapping (Eq. 1) involve mixing of features within frequency bins? Is $ \Lambda $ diagonal or not?

Author response:
**AR:** As mentioned in Section 3.1, in our proposed framework, we utilized a temporal convolution to implement the filtering operator $\\Xi$ in (1). Therefore, it is possible that this operation may involve linear combinations of features within frequency bins. However, we can also choose to implement $\\Xi$ with various other methods (e.g., pooling), which may not mix features within frequency bins.

$\\mathbf{\\Lambda}\_{t, d^{(f)}, d^{(t)}} \\in \\mathbb{R}^{D\_s \\times \\check{w}\_s}, \forall d^{(f)}, d^{(t)}$ are sub feature maps of $\\check{\\mathbf{C}}\_t$, which consist of the spectro and temporal domain feature information, and they are not necessarily diagonal.

Reviewer comment (Experiments):
Have you considered other types of spectra-temporal features for comparison? One example could be from, https://engineering.jhu.edu/lcap/data/uploads/pdfs/interspeech2012_carlin.pdf

Author response:
**AR:**  We thank the reviewer for directing us to the reference. Given that our paper primarily aimed to explore relational thinking modeling in recognition tasks, we anticipate that such modeling should be agnostic to acoustic feature types. Consequently, we utilized the state-of-the-art acoustic features extracted by wav2vec2. However, we acknowledge and appreciate the significance of considering various types of acoustic features, which can greatly influence system performance. To this end, we have performed additional experiments using MFCCs as an alternate acoustic feature type (as detailed in **A5: Generalization to Other Acoustic Features** in Section 5.1). As highlighted by the reviewer's reference, it is clear that their proposed spectro-temporal features exhibited similar performance to MFCCs in clean speech scenarios, which aligned with our study. These experiments with MFCCs yielded consistent results and further validated the generalizability of our proposed framework.

Reviewer comment (Experiments):
Have you considered a comparison between spectro-temporal HMM based recognition and the proposed approach?

Author response:
**AR:** We agree with the reviewer that noting the structural similarities, such a comparison would be interesting. However, given the substantial advancements in deep learning based models, especially those incorporating pre-trained encoders like wav2vec2, which have consistently outperformed HMM based models in phoneme recognition and speech recognition tasks, we did not prioritize a comparison between our proposed approach and HMM-based recognition. Furthermore, the primary focus of this study was to explore the effectiveness of relational thinking modeling, and we aimed to build upon the success of deep learning models in this context.

Reviewer comment (Novelty):
The innovation is not clear. The paper claims their innovation of using relational thinking based modeling method on spectro-temporal domain for acoustic modeling. However, from the description in the paper, there is no distinction between relational thinking based modeling and self-attention based modeling. For example, with several self-attention modeling layers stacked, it's equivalent to the so called relational thinking based model that pair-wised relation will be learnt among the transformed forms of each node (each time step), rather than the single step embedding. Therefore, theoretically, there is no difference between self-attention and relational thinking based method.

Author response:
**AR:** Relational thinking and self-attention are two distinct mechanisms designed to capture different types of information. To illustrate the distinction, let us consider a simplified 2\-layer self-attention network w.l.o.g., where each layer $l$ comprises two nodes $\\mathbf{h}\_1^{(l)}$ and $\\mathbf{h}\_2^{(l)}$. According to (13), the calculation of nodes in the subsequent layer $l + 1$ is as follows:
$$
\\left [ \\mathbf{h}\_1^{(l + 1)}, \\mathbf{h}\_2^{(l + 1)} \\right ] = \\mathbf{W}\_v^{(l)} \\left [ \\mathbf{h}\_1^{(l)}, \\mathbf{h}\_2^{(l)} \\right ] \\left [ 
\\begin{array}{cc}
\\alpha\_{1, 1}^{(l)} & \\alpha\_{2, 1}^{(l)} \\\\
\\alpha\_{1, 2}^{(l)} & \\alpha\_{2, 2}^{(l)} 
\\end{array}
\\right ]. \\quad \\quad \\quad \\text{(b)}
$$
As a result, the state of a node $\\mathbf{h}\_2^{(3)}$ after undergoing two layers of self-attention calculations is 
$$
\\begin{align}
\\mathbf{h}\_2^{(3)} & = \\alpha\_{2, 1}^{(2)} \\mathbf{W}\_v^{(2)} \\left ( \\alpha\_{1, 1}^{(1)} \\mathbf{W}\_v^{(1)} \\mathbf{h}\_1^{(1)} + \\alpha\_{1, 2}^{(1)} \\mathbf{W}\_v^{(1)} \\mathbf{h}\_2^{(1)} \\right ) + \\alpha\_{2, 2}^{(2)} \\mathbf{W}\_v^{(2)} \\left ( \\alpha\_{2, 1}^{(1)} \\mathbf{W}\_v^{(1)} \\mathbf{h}\_1^{(1)} + \\alpha\_{2, 2}^{(1)} \\mathbf{W}\_v^{(1)} \\mathbf{h}\_2^{(1)} \right ) \\quad \\quad \\quad \\text{(c-1)} \\\\
& = \\left ( \\alpha\_{1, 1}^{(1)} \\alpha\_{2, 1}^{(2)} + \\alpha\_{2, 1}^{(1)} \\alpha\_{2, 2}^{(2)} \\right ) \\mathbf{W}\_v^{(2)} \\mathbf{W}\_v^{(1)} \\mathbf{h}\_1^{(1)} + \\left ( \\alpha\_{1, 2}^{(1)} \\alpha\_{2, 1}^{(2)} + \\alpha\_{2, 2}^{(1)} \\alpha\_{2, 2}^{(2)} \\right ) \\mathbf{W}\_v^{(2)} \\mathbf{W}\_v^{(1)} \\mathbf{h}\_2^{(1)}. \\quad \\quad \\quad \\text{(c-2)}
\end{align}
$$
Even though (c\-1) may exhibit a similar form to (12) (i.e., the graph embedding obtained by relational thinking, also as shown below),
$$
\\mathbf{r} = \\sum\_{(i, j) \\in \\{ (i, j) | i < j, (i, j) \\in \\bar{\\mathcal{E}} \\}} \\bar{\\alpha}\_{i, j} \\bar{f}\_{\\theta}(\\mathbf{h}\_{i}, \mathbf{h}\_{j}), \\quad \\quad \\quad \\text{(d)}
$$
particularly when we view $\\mathbf{W}\_v^{(2)} \\left ( \\alpha\_{1, 1}^{(1)} \\mathbf{W}\_v^{(1)} \\mathbf{h}\_1^{(1)} + \\alpha\_{1, 2}^{(1)} \\mathbf{W}\_v^{(1)} \\mathbf{h}\_2^{(1)} \\right )$ and $\\mathbf{W}\_v^{(2)} \\left ( \\alpha\_{2, 1}^{(1)} \\mathbf{W}\_v^{(1)} \\mathbf{h}\_1^{(1)} + \\alpha\_{2, 2}^{(1)} \\mathbf{W}\_v^{(1)} \\mathbf{h}\_2^{(1)} \\right )$ as node pair embedding functions from a linear family $\\mathcal{F} \\left ( \\mathbf{h}\_1^{(1)}, \\mathbf{h}\_2^{(1)} \\right )$, it is crucial to note that $\\mathbf{h}\_2^{(3)}$ is fundamentally still a weighted sum of node embeddings (as revealed by (c-2)) rather than a weighted sum of node pair embeddings as obtained by relational thinking (d), where $\\bar{f}\_{\\theta}(\\mathbf{h}\_{i}, \\mathbf{h}\_{j})$ is an arbitrary node pair embedding function. Therefore, the stacked self-attention mechanism and relational thinking are not entirely equivalent, and models cannot solely rely on the self-attention mechanism to effectively assess the importance of a pair of nodes, as discussed in Appendix D.

Furthermore, it is also important to highlight that our proposed relational thinking modeling is a variational approach, whereas the self-attention mechanism is a deterministic approach. As elucidated in Nan et al. (2023), deterministic models can produce incorrect predictions when confronted with unseen data, particularly due to the discontinuous and sparse nature of the latent space. In contrast, variational models address this challenge by mapping the input data to collections of distributions in the latent space rather than singular points. By capturing uncertainty in the latent space, variational modeling enhances model generalization, rendering the model more adaptable to diverse and unseen data. Also, we have further clarified our novelty; please refer to our responses to Reviewer Q5NM for more details.

Reviewer comment (Experiments):
The experiment part is not complete. In order to demonstrate the superiority of the proposed relational thinking based method compared to the self-attention based method, the results of the self-attention should be included as one of the baseline results. However, the results of the paper only includes these feature extraction based method without extra modeling. In addition, the paper should also list the model size of each compared method to have a more fair comparision.

Author response:
**AR:** Our experiment **A4: Comparison with SOTA** in Section 5.1 indeed compared our proposed model with the self\-attention mechanism based baseline. The majority of state\-of\-the\-art systems in speech processing have been developed using transformer architectures (Vaswani et al., 2017), which fundamentally rely on self-attention mechanisms. Our baseline, wav2vec2 (Baevski et al., 2020), is just one example of these transformer\-based systems. By comparing these two models, we were in fact allowed to evaluate the additional benefits provided by relational thinking modeling in enhancing speech representation.

Additionally, in our revised version of the paper, we have listed the model size of each compared method, as can be found in Section 5.1. We also provide the total numbers of parameters for the models used in our experiments below:

a) wav2vec2 baseline: 94.4M;

b) proposed relational thinking based models: 100.8M.

Reviewer comment (Evaluation):
The tradeoff study between temporal context and spectral context is not able to lead such conclusion that higher frequency domain resolution provides more benefits compared to higher time domain resolution, as the results of these two setting are very close in the test set (20.80 vs. 20.66).

Author response:
**AR:** Our concluding remark regarding the trade-off between temporal context and spectral context was based on a comparison not only between w20\-t4f2 and w20\-t2f4 but also between w20\-t8f1 and w20\-t1f8. It is true that the performance differences between w20\-t4f2 and w20\-t2f4 were not sufficiently significant. We appreciate the reviewer's feedback and acknowledge this limitation.

However, our intention was not to make a definitive conclusion based on the current results. Instead, we aimed to highlight the potential benefits of various strategies for resolution setting in both domains. To draw more robust and convincing conclusions, we recognize the need for a more comprehensive investigation of the impact of resolution settings on system performance, which we plan to explore in our future work.

In the revised version of the paper, we have weakened our concluding remark to improve the clarity and accuracy of the description and better convey the experimental results.

Reviewer comment (Experiments):
Have you done such experiment that replace the relational thinking based model with Transformer/Conformer type of module? If so, what is the performance?

Author response:
**AR:** We did not conduct any experiments in which the relational thinking module was replaced with a transformer/conformer type module. In fact, such experiments are not feasible due to the prevalent use of transformer/conformer based architectures in state\-of\-the\-art acoustic feature extraction methods, including the wav2vec2 that we employed in our proposed framework. These architectures are deeply integrated into the foundation of modern acoustic feature extraction, making it impractical to substitute them in our experiments.

Instead, to assess the additional benefits offered by relational thinking beyond what the attention mechanism has achieved, we compared the performance of two models: 1) the wav2vec2 baseline, which incorporated a transformer\-based feature extraction module; 2) the proposed model, which utilized a transformer-based feature extraction module in conjunction with a relational thinking module. Obviously, introducing a model that replaces the relational thinking module with a transformer\-based module would be redundant. Such a model would essentially replicate the structure of the wav2vec2 baseline, defeating the purpose of our investigation into the benefits of relational thinking.

Reviewer comment (Evaluation):
Could you please explain why the proposed method cannot perform well in non-vowel recognition?

Author response:
**AR:** As we stated at the beginning of Section 5.3, one potential reason why the proposed relational thinking based model cannot show significant advantages over the baseline in recognizing non\-vowels is that non\-vowel phonemes tend to have shorter durations compared to vowels. This shorter duration for non-vowels does not allow the relational thinking module to capture as much relational information, which could benefit the downstream recognition task.  The subsequent analyses conducted in this subsection partially supported our hypothesis.

Reviewer comment (Novelty):
Incremental Technical Contribution: The technical developments in this work appear to be an incremental advancement from Huang et al. (2020). The main modification is the extension of the input from one dimension to two dimensions, followed by a direct application of the relational thinking network proposed by Huang et al.

Author response:
**AR:** In addition to extending the input from one dimension to two dimensions, our work differs from that of Huang et al. (2020) in three perspectives:

a) Our focus differs from Huang et al. (2020) as we examined a distinct application, one that readily extends to scenarios where the input and output sequences are of different lengths. This is a common scenario in sequence modeling tasks like speech recognition.  To accommodate these real-world applications, we developed a novel loss function that can handle such cases. This stands in contrast to the loss function proposed by Huang et al. (2020), which is applicable only to cases where input and output sequences have equal lengths. This paper addressed the challenges that arise when dealing with input and output sequences of different lengths, a common scenario in sequence modeling tasks like speech recognition. To accommodate real-world applications, we developed a novel loss function that can handle such cases. This stands in contrast to the loss function proposed by Huang et al. (2020), which is applicable only to cases where input and output sequences have equal lengths. 

b) Our paper included a comprehensive series of analyses toward the recognition results and the learned relational information, providing valuable insights into human speech perception and comprehension. We uncovered the patterns involved in the captured relations (i.e., the edges $\bar{\alpha}^{(t)}_{i, j}$ of the task-specific graph), which exhibited more similarities for phoneme classes within the same sub-group, while showed significant variations between phoneme classes from different sub-groups. We also conveyed that relational thinking modeling primarily enhanced the system by improving its performance in recognizing vowels. These insights represent a novel and previously unexplored aspect of research.

c) We conducted a theoretical analysis that explained the distinctions between relational thinking and the self-attention mechanism (in Appendix~D). Our analysis underscored that relational thinking captured a new type of (co-occurrence or pair-wise) information that could offer additional advantages beyond what the attention mechanism has achieved.

Reviewer comment (Experiments):
Dataset Limitations: The experiments rely heavily on the TIMIT dataset, which is relatively small in size. To firmly establish the proposed method's efficacy and robustness, it is imperative to test it on larger, more diverse datasets and under complex conditions.

Author response:
**AR:** As clarified in Section 4, we used the TIMIT dataset because this study primarily aimed to delve deeper into the concepts of relational thinking with speech understanding and recognition tasks at the fine-grained level, serving as an initial step toward a comprehensive understanding of how this can be applied to speech. Such an analysis is feasible exclusively with the TIMIT dataset, which stands out as one of the very few datasets that provide precise information about the start and end instants of each phoneme. This unique attribute allowed us to conduct more in-depth analyses of the recognition results. For instance, it allowed us to uncover the patterns involved in the captured relations (i.e., the edges $\bar{\alpha}^{(t)}_{i, j}$ of the task-specific graph). These patterns exhibited more similarities for phoneme classes within the same sub-group, while they showed significant variations between phoneme classes from different sub-groups (as detailed in Section 5.2 and Appendix G.2). It also facilitated our ability to convey that relational thinking modeling primarily enhanced the system by improving its performance in recognizing vowels (as detailed in Section 5.3 and Appendix G.1). Without the manually annotated labels in the TIMIT dataset, we would not have been able to attain these insights. With the refined knowledge and deep understanding, we would like to extend and corroborate our findings on larger datasets beyond phoneme recognition in the future.