From: ICLR.cc/2023/Conference/Program_Chairs
title: Paper Decision
decision: Reject
metareview:_summary,_strengths_and_weaknesses: The paper introduces a method to infer meaningful groupings of characters in a text. The authors adapt slot attention to learn text representations, hoping that those slots would extract meaningful units from sequences of characters. The authors made some changes to the original design, including a different parameter initialization and an L0-drop layer. Experiments show that the models with slot attention induce subword units better predict human-labeled morphemes, segmentations from Morfessor, and segmentations from BPE, than stride-attention baselines.

The reviewers consider that experimental evidence needs to be stronger to show that slot attention learns something meaningful and benefits text representation. The reviewers suggest that some downstream tasks are required to evaluate the proposed method to demonstrate the effectiveness of the induced slot representation. The authors argue that due to large-scale engineering, the evaluation on  downstream tasks is beyond the scope of this work. However, machine learning-based probing can be misleading, so using probing results to draw conclusions is insufficient. Therefore, we still encourage the authors to evaluate the induced slot representation on some downstream tasks.

Some reviewers had questions about probing, and I believe the authors' feedback answers the questions. 
justification_for_why_not_higher_score: Overall, the proposed method is novel and has potential. But the current version needs more support to show that the learned units are meaningful. So the current version is not acceptable. 
justification_for_why_not_lower_score: N/A

From: ICLR.cc/2023/Conference/Paper4442/Reviewer_MN1D
title: Do slots have meaning?
comment: I thank the authors for their response, below I will clarify some of the observations in the original review and highlight some aspects that the work still needs to improve.

Regarding the importance of the task.  I am *not* saying that finding some small individual units in an unsupervised way is not important. On the contrary, I think it is of essential importance for generalization and compositionality. But, even though it is hard to evaluate the results of such algorithms, we need to have some form of validation of the relevance of the representations. It is crucial to prove that the obtained groupings / units are meaningful in some way. Here lies the importance of testing on downstream tasks. 

As the authors say, “Characters don’t have meaning, but words do”. Now the most important question is: do the obtained unit have meaning? Do they convey something more than using words as units? Do they have some advantages? Or do we find something interesting about the inferred units that makes them relevant even without necessarily improving some downstream task? At the moment there is little evidence to answer these questions.

Regarding the weak baseline. The authors highlight that a random representation constitutes a good baseline. In general, this is the case. But not for the *specific* proposed method. As said in the review, each slot constitutes an average of input representation. The input-slot assignments are random, thus each slot practically receives uniform information from all inputs. It is probably impossible to distinguish one slot from another for this reason. This is why this baseline is weak, not only because it starts with random weights. 

As for the downstream evaluations, I agree that they require a lot of engineering work, but I think they are necessary. The probing experiments are still too weak to clearly show that the method has learned something meaningful. I think usefulness does not constitute only an “engineering criteria” but evaluating it constitutes a key scientific investigation.

Overall, I agree that the proposed method has good potential for an important problem. But in the current form, we don’t understand if the units are meaningful or not. I think more investigations are needed.

From: ICLR.cc/2023/Conference/Paper4442/Authors
title: Response to Reviewer RoTY: Thank you for the helpful feedback
comment: Thank you for your helpful feedback. We are glad that you found our approach reasonable and our proposed models effective. Below we addressed some of your questions and concerns. 

> The current work doesn't show the approach's usefulness over downstream tasks.

Please see our general response.

> A deeper analysis that focuses on the discovery of sublexical units will strengthen the paper.

It is difficult to precisely characterize what is linguistically meaningful, other than to use labeled data. Do you have something specific in mind? For the languages where we have real morpheme data available (see Table 1), we do see better results for real morphemes than for BPE, both in absolute terms and relative to the stride model.  This improvement is substantial in recall, meaning that more of the true morphemes can be accurately predicted than the BPE tokens. 

> Reverse Probing is confusing. How were the target subword units incorporated into the input of the classifier?

We have updated section B.4 to include a more detailed explanation of the reverse probing task. To answer your question, we do not use the learned embeddings from our encoder. We learn an embedding layer for each of the reverse probes. More specifically, we have a layer that maps every token to a continuous vector as it is usually done in an NLP pipeline. What we do reuse from the forward probing task, is the matching between the tokens and slots, i.e., we do not run the matching algorithm. Please also see the general response for a discussion on our probing experiments.

From: ICLR.cc/2023/Conference/Paper4442/Authors
title: General Response to Reviewers
comment: We thank all the reviewers for their valuable feedback. Below we address some of the common questions and concerns raised by reviewers. 

**Reconstruction loss report** 

Following Reviewer aQFF’s suggestion on reporting the reconstruction quality of our proposed models, here, we report the reconstruction quality of our slot attention and stride-based models on different languages. The two models have the same average number of units for fair comparison. In all languages, the slot-attention based model better reconstructs the sequence than the stride-based model. The reconstruction loss is a good indicator of how well the representation vectors carry the information about the structure and meaning of the encoded input to the decoder. We also include these results in Appendix A.2.

|          | Slot attention | (r=6)           |   | stride=6 |        |
|----------|---------------------|------------|---|----------|--------|
| language | train               | test       |   | train    | test   |
| de       | **0.0022**          | **0.0039** |   | 0.0174   | 0.0213 |
| en       | **0.0014**          | **0.0033** |   | 0.0120   | 0.0154 |
| cs       | **0.0021**          | **0.0035** |   | 0.0089   | 0.0137 |
| fi       | **0.0028**          | **0.0045** |   | 0.0097   | 0.0135 |
| fr       | **0.0043**          | **0.0075** |   | 0.0182   | 0.0229 |
| es       | **0.0014**          | **0.0036** |   | 0.010    | 0.0149 |

**More details and illustrations of probing experiments**

Three reviewers (aQFF, RoTY, nzdu) had questions about our probing experiments.  We have improved the explanation of these evaluations in the paper.  In particular, as mentioned by Reviewers RoTY and nzdu, we updated section B.4 in the Appendix to contain a more detailed explanation of the reverse probing setup and an illustration of how the two probes are related.  Here we provide an explanation which targets the reviewers’ specific comments and misunderstandings.

Probing (here called forward probing) is a standard method in the literature on interpretability to evaluate whether a given representation contains the information being predicted by the probe.  In our experiments, we add the constraint that each target token needs to be predicted from an individual slot vector, rather than the information being distributed across slots.  To measure this, we add the novel reverse probing evaluation, which measures whether these same slot vectors can in turn be predicted from their associated target token.  Together, these evaluations measure to what extent the learned slot vectors contain all and only the information in the target tokens, with this information partitioned across slots/tokens in the same way.  Thus, they evaluate whether our slot attention method is learning the same level of abstraction and modularity as the target representation.

**Evaluation on Downstream tasks**

The large-scale engineering needed to evaluate the usefulness on downstream tasks is outside the scope of this work.  Our contributions are to demonstrate how slot attention can be extended to make it effective in the unsupervised learning of meaningful units in text.  Research on this topic is still at an early stage, so it would be counterproductive to prevent the publication of pioneering work like ours on the basis of engineering criteria like usefulness.

Our model could be integrated into any possible downstream task in an end-to-end fashion to obtain the required representations for a task by skipping the tokenization step. We anticipate that our method would be especially helpful in languages that are rich in morphologically. It is in these languages that BPE struggles to find good tokenization of rare words and usually split them into meaningless tokens.

From: ICLR.cc/2023/Conference/Paper4442/Authors
title: Response to Reviewer nzdu : Probing individual slot vectors 
comment: Thank you for your helpful comments. We are glad that you find the direction of our work interesting. We have included more details on the probing experiments in section B.4 as you recommended. Below we answer some of your questions and concerns.

> I still have some doubts on how meaningful the “induced slots” are and I think the two evaluations are not quite enough to show that these slots indeed represent meaningful units. 

We have included a general explanation about our probing experiments in the general response, and we will improve the explanation in the paper.  Regarding the reviewer's specific points, note that each probing classifier only sees or predicts a single slot vector.  For both the attention maps and the forward probing measure, the results show that (in most cases) all the information needed to predict a given character or token, respectively, is in a single slot, not distributed across slots.  Also, the reverse probing results show that (to a large extent) all the information in a given slot can be predicted from its associated token.  Therefore, forward probing plus reverse probing taken together show that all and only the information for a given token is in its associated individual slot.  This is a much stronger methodology for identifying precisely whether slots are discovering tokens than any proposed previously.

> To show the effectiveness of the induced slot representations, applications on real text tasks are lacking.

Please see our general response.

From: ICLR.cc/2023/Conference/Paper4442/Authors
title: Response to Reviewer MN1D: Characters don’t have meaning, but words do
comment: We thank you for your thoughtful comments. We are happy that you find our method sound and capable of producing meaningful units. Below we answer some of your questions and concerns.

> Do we have any reason to believe that they would be beneficial for reasoning over text? Does discovering these units have a meaning on its own?

We find this question rather surprising. Individual characters do not have meaning, but when they are grouped into words they become the most meaningful perceptual units in any modality.  It is hard to imagine how any model could reason about text without first identifying words. Linguistics has shown that in general words can often be decomposed into smaller units that are still meaningful, called morphemes.  Unsupervised morphology induction has been a long-lasting line of work in NLP as described in section 3 (unsupervised morphology learning discussion). Therefore, it is of course important on its own to find these smallest building blocks of language without any supervision and assumptions about that language. 

> Good performance for a downstream task would be ideal.

Please see the general response.

> The untrained baseline is too weak.

The untrained baseline is only intended to be a baseline, to show the effectiveness of our training paradigm. This is a common baseline in probing tasks as in Conneau et al., 2018; Oord et al., 2018, which essentially shows the probing classifier’s capacity to memorize the task, even with random input representations. We have also included the stride-based models in our experiments which could be viewed as very strong baselines.

From: ICLR.cc/2023/Conference/Paper4442/Authors
title: Response to Reviewer aQFF: Several misunderstandings (II) 
comment: > No clear or consistent trend found in Fig 3 between the performance of slot-attention and the performance of stride-attention.

In Figure 3, subfigures (a), (b), (c ), (e) and (f) you can see that the blue points are nearer to the right bottom corner of the plot showing the better trade-off handling of the slot attention based models. In subfigures (g) and (h) the slot attention based models are lying towards the right while having similar values on the y-axis meaning that they are also superior to the stride-based models. The only subfigure where we cannot draw the same conclusion is subfigure (d) where in some points the stride-based model is working better. We anticipate that the reason for this is that Finnish is a morphologically complex language and BPE is not a good approximation of its morphemes. 

> Preliminary results (Sec Limitations) showed slot-attention is slightly worse than stride-attention

Here, our main argument is that among character-based models our proposed models work better. In addition, the stride-based models have not been used previously and could be considered as strong models on their own. This is also part of our contributions. We left work on downstream tasks for future work.

**Localized Points:**

2. Thank you for your suggestion. 

3. Thank you for your great suggestion. We reported the numbers in the general response. 

4, 5. Please see above.

6. Arabic morphology is notoriously difficult. We will address this in future work.

7. The citation is for the general idea of probing, not for the specific use of it. To our knowledge, we are the first to propose learning abstract meaning-bearing units without any assumptions on the language, so of course, there is no prior work on evaluating such representations. This probing framework is our proposition for this evaluation. By that citation, we meant to show that probing for linguistic information is a common task in NLP. And since we are not able to directly compare the segmentations, and that morphemes are not always segments, we need to do some sort of probing. 

8. This is incorrect. We are predicting a probability density function, not a discrete probability, and it is possible for a probability density function to get values higher than 1 in a very small interval. In this setup, if we take the integral over that small interval, $dx$ would scale this density down and turn it into a probability, but we do not compute this.  Specifically, when the model converges, it is able to predict the slot vector so precisely (small sigma and very precise mu), therefore leading to a density higher than 1, for which the negative-logarithm would be negative. 

9, 10. Please see above.

From: ICLR.cc/2023/Conference/Paper4442/Authors
title: Response to Reviewer aQFF: Several misunderstandings (I)
comment: Thank you for your detailed feedback on our work. We are glad that you found our idea interesting, and our paper clear. But we still think that some of the criticisms are rooted in misunderstandings. Here we address some of your questions and concerns. 

**Weaknesses:**

>Limited technical contribution.

We disagree.  Two of the classic characteristics of language are its large vocabulary and its varying input length, both of which are not needed for the experiments in previous work on slot attention.  Our extensions to slot attention address exactly these characteristics, namely by learning multiple slot distributions and by adding an L0-Drop layer to vary the number of slots needed for different inputs, respectively.  

>Reporting metrics on the training and validation set and comparing them against the baselines will make a point.

Thank you for this suggestion. Please see the general response.

>No results from downstream tasks, such as POS tagging and segmentation.

Please refer to the general response on a discussion about downstream task evaluation. We are learning abstract continuous representations of text, therefore, it is not straightforward to evaluate our model on word-level tasks such as POS tagging or character-level tasks such as segmentations. What is your suggestion for this setup? 

>Actual realization does not seem to enforce any meaningful bottleneck.

There is a bottleneck, both in theory and in practice.  Any continuous vector can in theory capture an arbitrary amount of information; a bottleneck is introduced by adding noise.  We add a fixed amount of noise to the slot vectors at initialization, which then carries through to the computed slot vector via the skip connections.  In practice, we find that this is an effective way to introduce a bottleneck and thereby get meaningful learning and avoid memorization.  When we do not add this noise, learning fails in the way the reviewer predicts.

>The observed relation may be a bias from the encoder-decoder bottleneck and has nothing to do with slot attention.

The difficulty is the opposite; each encoder vector carries information from multiple input characters as a result of its self-attention layers, but the decoder predicts exactly one character at a time.  If there were information about the predicted character in more than one slot, then the decoder would pay attention to more than one slot, which in general it does not.  As expected, we observe the same attention patterns in the attention of slots over the input embedding vectors, but the pattern is less sharp than on the decoder side.  We have added some examples of these attention patterns in Appendix A.3 for illustration.

>Unsupported claims that attribute the emergence of continuous span to the effectiveness of their model.

We have provided enough evidence to justify this claim, and in the absence of any alternative explanation, we see no reason to change it. The phrasing “We believe” is precisely there to signal that this is not a claim. 

>The probing task seems unnecessarily convoluted.

Please see our general response for an explanation of forward and backward probing. Forward probing, or just “probing” in the literature, is a standard method from work on interpretability of deep learning models.  It tests whether the information in the target representation is included in the learned representation.  We add reverse probing to see whether the information in the learned representation is included in the target representation.  And we add an alignment between slots and target tokens to ensure that the information is partitioned in the same way.

> In the probing task (Sec 4.4), slot-attention has a possibly unfair advantage that its number of slots are set dynamically.

This is exactly where our contributions are.  Why is it unfair to propose a model which is better?  The stride-based models are a hand-coded solution to selecting the number of embedding vectors, whereas our model decides dynamically on how many slot vectors to use for each input. Our control is only over the average number of open gates in a batch not per input. We do compare the slot-attention based model versus stride-based models which have the same average number of vectors. We can also vary the number of vectors in a stride-based model by varying the stride length which we do in the informativeness vs abstractness experiments.

From: ICLR.cc/2023/Conference/Paper4442/Reviewer_aQFF
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: Slot attention (Locatello et al., 2020) was originally designed to learn object-centric representations of images. The authors adapt slot attention to learn text representations, in the hope that those slots would similarly extract meaningful units from sequences of characters. The authors made minor changes to the original design, including a different parameter initialization, an L0-drop layer from Louizos et al. (2018). Experiments show that the models with slot attention 1) produce non-overlapping continuous attention patterns when decoding, and 2) induce subword units that better predict human-labeled morphemes, segmentations from Morfessor, and segmentations from BPE, than stride-attention baselines.
strength_and_weaknesses: ### Strengths
1. Interesting idea to adapt unsupervised (implicit) segmentation from image to text.
2. The paper is easy to follow, with major motivation, design, and results are clearly stated. There are few under-supported claims or relatively minor points worth clarifying, e.g. LPs P2, Sec 2.1; P6, Sec 4.2; P7, Sec 4.3; P8, Fig 3, but themselves are unlikely to refute the key contributions.

### Weaknesses
1. Limited technical contribution. No design specific for the language data. More or less a straightforward application of slot attention. This itself won't be a reason to reject if the experimental results are strong. However,
2. Experimental evidence not strong enough to show that slot attention is beneficial in learning text representation. The weakness is multifold. Roughly categorized into insufficient experiments (a, b), questionable experiment setting (c, d, f, g), weak evidence (e, h, i).
    - a. The authors trained their model using reconstructive objectives. The performance of this task can show how good a model is at capturing the key meaning and the structure of a sequence. Reporting metrics on the training and validation set and comparing them against the baselines will make a point. See also LP Sec 4.
    - b. The authors showed no results from downstream tasks. This will be the most direct evidence that slot attention is at least helpful to certain NLP tasks. For example, pos-tagging can show if it captures word meanings or roles; segmentation can show if it learns morphology. Compare against character-level and stride-attention models would provide evidence.
    - c. Although the authors claim the encoder-decoder information bottleneck being an important factor to the effectiveness of the method, the actual realization does not seem to enforce any meaningful bottleneck and the input sequence can in theory be memorized verbatim. This raises concern on if the setup really facilitates meaningful learning and if the observed behaviors of the learned slots are legit signs of learning. See LP P6, Sec 4.1.
    - d. In Sec 4.2, the authors visualize the attention of decoder, i.e. output sequence, over slots. Why not inspect the relation between the slot and the input sequence? The observed relation may be a bias from the encoder-decoder bottleneck and has nothing to do with slot attention. Comparison against other bottleneck baselines can improve the support. See also LP Sec 4.2, observations.
    - e. Unsupported claims that attribute the emergence of continuous span to effectiveness of their model. Need further evidence to determine if this is an inherent bias or actually aligns with the language data. See also P6, Sec 4.2, "We believe ..."
    - f. The probing task of "predicting reference subword segmentation from the subwords induced from the current model" (Sec 4.3) seems unnecessarily convoluted. I am not aware of anything like this in the existing work. The authors cited a survey but I was not able to find any discussion of this specific probing or even similar probing. See LP P7, Sec 4.3, top. The forward probing task, the reverse probing task, the design of how to induce units, all seems convoluted and artificial. If the authors are able to induce subword units from the slots, can they go one step further and induce subword segmentations and directly evaluate on the subword segmentation task? Subword segmentation or morphological segmentation is an established task. Many benchmarks available. The authors can also brand it as a downstream task evaluation.
    - g. In the probing task (Sec 4.4), slot-attention has a possibly unfair advantage that its number of slots are set dynamically. See LP P8, top.
    - h. No clear or consistent trend found in Fig 3 between the performance of slot-attention and the performance of stride-attention.
    - i. Preliminary results (Sec Limitations) showed slot-attention is slightly worse than stride-attention (baseline) on the downstream task of text classification, measured by F-1 on ArXiv-L. See LP P9, Limitations, bottom.


### Localized points (LPs)
1. P2, Sec 2.1, "... where we expect our slots to represent something like the set of morphemes ...". Can you be specific what are the exact properties you want from this slot formation?
2. P3, after Eq (1), "In other words, the initial value of slots are independent samples of a single Normal distribution with learnable parameters." This sentence repeats the sentence directly before it.
3. Sec 4. What are the numbers from reconstructive training?
4. P6, Sec 4.1, "We feed in the sentences with less than 128 characters to our model and consider the number of slots as 64 (half of the maximum input length). In addition, we take the dimension of slots as 128." Assuming the characters are ASCII, each character can be represented in 8 bits. 128x8 = 64x16 bit. A 128-d real vector is more than enough to fully store 16-bit information and thus the entire sequence can be stored verbatim. I see no bottleneck here.
5. P6, Sec 4.2, observations on the slot attention pattern. Why do the authors focus on the attention pattern with respect to the output sequence not the input sequence? It could be the case that an arbitrary bottleneck without any of the slot-attention design gives a similar pattern. "An arbitrary bottleneck" could be taking stride attention or simply reduce the number of vectors (e.g. the first k vector, or the vector of some special token) or the dimension of the vectors (e.g. reducing from 128d to 16d) being passed from the encoder to decoder. I will be more convinced that the slot-attention really did something non-trivial if a bottleneck itself is unable to give such a pattern.
6. P6, Sec 4.2, "We believe that the emergence of contiguous spans is a result of the bottleneck ..." You can test whether the emergence of spans is learned from the text or merely an inductive bias of the model. You can train the model over languages with nonconcatenative morphology, e.g. Arabic, and see if the learned slot is still contiguous or aligns with the morphology.
7. P7, Sec 4.3, top, "Predicting the previously proposed unit from our induced unit is a probing task, as used in previous work (Belinkov & Glass, 2019), ..." Can the authors point us to which previous work used "predicting reference subword segmentation from the subwords induced from the current model" as a probe task? The cited survey does not particularly discuss any probing where one predicts subword units from latent presentation or induced units.
8. P8, Fig 3. Why are all the NLL losses negative? Negative log-likelihood should take value from [0, +\infty).
9. P8, top, "the number of open gates in the L0Drop layer is approximately the same as the target number." The number of tokens generated from stride-attention (baseline) is fixed but the number of tokens (determined by the number of gates) is set dynamically. This is an unfair advantage.
10. P9, Limitations, bottom, "For the ArXiv-L, on average, we get 0.396, 0.409,
0.2732 F1 for slot-attn, stride=6, and character-based models, respectively, on Dev set, and 0.393, 0.394, 0.271 on Test set." It seems slot-attn <= stride-attn from these preliminary results.

clarity,_quality,_novelty_and_reproducibility: Clarity: Good. The authors successfully conveyed major points.

Quality: Fair. The experiments show some promising results for using slot attention in learning text, but the support is weak due to insufficient experiments (2a, 2b), questionable experiment setting (2c, 2e, 2f), weak evidence (2d, 2g, 2h).

Novelty: Mixed. The authors claim to be the first to apply slot attention to text. However, it seems to be a straightforward application with little modifications and nothing specific to text.

Reproducibility: Good, conditioning on code and data release. The authors provided detailed experimental settings in the appendix.

summary_of_the_review: The work has novelty as the first to apply slot attention to text. However, the adaptation seems straightforward and no text-specific design was added. It itself won't be a reason to reject but the experimental designs are flawed and that the results do not answer if slot attention is beneficial to text learning. Stronger, more direct experimental support would improve the strength of the paper. 
correctness: 2: Several of the paper’s claims are incorrect or not well-supported.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 3: reject, not good enough

From: ICLR.cc/2023/Conference/Paper4442/Reviewer_RoTY
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: This method proposes dynamic slot attention based approach to discover subword units in a completely unsupervised manner. The character based encoder representations are passed through a bottleneck of K slot noisy vectors (K < length of input) and these slot vectors are then used for reconstruction with a lightweight decoder with attention over the slots. In addition to the reconstruction loss, an L0 drop term (relaxed via Hard-Concrete reparametrization) is also added to the training loss and inference to encourage sparseness of slots that are being used. This approach is empiricqally compared to a random baseline and a reasonable stride based approach is also proposed for comparison with a better baseline. Intrinsic evaluation testing correspondence of slots to sub-lexical units like morphemes and BPE validate the effectiveness of this approach against the proposed baseline.
strength_and_weaknesses: Strengths:

-- The proposed approach is reasonable with sensible design choices and loss terms.

-- The proposed approach effectively learns contiguous spans that correspond well with human-interpretable morphemes, and automatic subword extraction methods like BPE.

Weaknesses:

-- Although, it is mentioned in the "limitations" section, I believe it is a fundamental drawback of the current work that it doesn't show the approach's usefulness over downstream tasks and the evaluation is mostly intrinsic. I find it difficult to articulate the importance of learning subword units with such an expensive procedure. For example, if the proposed use is better tokenization, then experiments that support this should be performed to justify using this expensive approach for tokenization.

-- Related to above, a deeper analysis that focuses on the discovery of sublexical units will strengthen the paper. While this approach is shown to be correlated with morphemes and BPE tokens, these two subword units are intrinsically very different. Hence, analysis of the proposed approach in context of linguistically meaningful units in contrast with engineering-oriented units like BPE should be performed in greater detail.

-- "Reverse Probing" is confusing. I don't understand how these experiments were performed. How were the target subword units incorporated into the input of the classifier? Was the learned encoder over characters used for reverse probing. If yes, then isn't there a concern of using an encoder that was learned jointly with the slots for probing over the slots?
clarity,_quality,_novelty_and_reproducibility: Please see my review above. Some aspects like reverse probing can be clarifies in greater detail and more analysis will shed light on aspects of the proposed approach.
summary_of_the_review: Overall, the proposed approach is reasonable and tackles an interesting problem of discovering subword units in a completely unsupervised manner. However, I have slight concern over the lack of extrinsic evaluation and a deeper motivation for the intrinsic properties.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 6: marginally above the acceptance threshold

From: ICLR.cc/2023/Conference/Paper4442/Reviewer_MN1D
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: The paper introduces a method to infer meaningful groupings of characters in a text. It is based on Slot attention method developed in the visual domain to obtain slots (vector representations), corresponding to meaningful parts of an image (objects). The paper adapts this method to work on a sequence of characters. The Slot attention method is used as a bottleneck module inside an encode-decoder Transformer method that takes as input a sequence of characters and is trained to reproduce it. The proposed Dynamic Capacity Slot Attention method can use a different number of slots for each sequence by using a sparsification layer ($L_0$Drop) and uses different initialization for each slot for better optimization. The method seems to produce a meaningful grouping of characters, that is similar to previous methods.

strength_and_weaknesses: **Strong Points**:

The proposed method is sound and is capable of producing meaningful groupings of characters.

The usage of sparsification layer $L_0$Drop (Zhang et al., 2021) to control the number of slots is a good idea for using a different number of slots for each sample.


**Weak Points**:

It is not clear what is the purpose of the grouping of characters. In vision, objects are central for reasoning over real-world scenes, so it makes more sense that obtaining unsupervised entities that are similar to objects is desirable. Do we have any reason to believe that they would be beneficial for reasoning over text? Does discovering these units have a meaning on its own?

Although the authors note that evaluating the obtained representations is outside the scope of this work, it should not be the case. We need a reason to do this kind of grouping, and good performance for a downstream task would be ideal. 

Regarding the probing: The untrained baseline is too weak, because before training each slot constraints an average of all input tokens, which are also random at first, thus it is extremely unlikely that these representations could be used in a learning task. Maybe a model that averages all the tokens corresponding to the same word be a better baseline. 

clarity,_quality,_novelty_and_reproducibility: The paper is mainly clear and easy to reproduce. The idea is original, but the purpose is not clear yet.
summary_of_the_review:  The paper proposes a good and sound method for grouping elements in a sequence. It is technically sound and seems to produce meaningful grouping but at the moment the purpose of these groupings is unclear. 
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold

From: ICLR.cc/2023/Conference/Paper4442/Reviewer_nzdu
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: This work proposes a method to induce meaningful text units from character sequences. The method is based on Slot Attention for unsupervised object discovery in computer vision. To adapt it to texts, a modification is introduced by using separate trainable parameters for each slot rather than shared. The full model consists of a character-encoder, the modified slot attention module, an L0-drop module to enforce sparsity and a shallow decoder for recovering input sequences, and the full model is trained with an auto-encoding objective. The evaluation of the proposed model is performed by qualitatively visualizing attention maps as well as quantitative probing experiments to measure the correspondence between the induced slots and linguistic or statistical units.
strength_and_weaknesses: Strength:

This work explores an interesting direction of letting the neural models induce meaningful linguistic units in an unsupervised way. This might lead to better segmentation/character-level representation methods, and provide more insights on the inner mechanisms of neural models.

Weakness:

I still have some doubts on how meanful the “induced slots” are and I think the two evaluations are not quite enough to show that these slots indeed represent meaningful units. Though the patterns in the attention maps and the obvious improvements in the probing experiments look interesting, I still don’t feel these are enough to arrive at the conclusion that the induced slots are very meaningful, unless there is a more clear way to show what they actually represent. Attention only shows the information may exist but not necessarily that the slots individually represent meaningful units, so does probing (especially with the matching-based probe training).

To show the effectiveness of the induced slot representations, applications on real text tasks are lacking (except for an initial experiment where there seems to be no obvious performance difference between slot-attn and simpler stride-based representations).

clarity,_quality,_novelty_and_reproducibility: Figure 2 might be readjusted for easier reading. And it would be better if there could be more illustrations for the probing experiments for easier understanding.
summary_of_the_review: This paper explores an interesting question on the unsupervised induction of meaningful text units and proposes a modified Slot-Attention based model for this purpose. Though some of the results and patterns look interesting, it still remains unclear to me how meaningful the slot representations are and there is a lack of further explorations on the effectiveness of them in actual tasks.
correctness: 2: Several of the paper’s claims are incorrect or not well-supported.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold