Paper: Encoding Expert Knowledge into Federated Learning using Weak Supervision (ICLR-2024)

Reviewer comment (Reproducibility):
If possible, an open-source code base for federated learning with WeaSEL or other non end-to-end programmatic weak supervision would be very beneficial to the community.

Author response:
Regarding the suggestion to include an **open-source code base**, we plan to open-source our code when we release our paper. In the meantime, we have made the code anonymously available at https://anonymous.4open.science/r/wshfl_pipeline-A13C/README.md. We have included a link to this resource in the manuscript.

Reviewer comment (Theory):
From the automatic labeling functions generation scheme, for example, the unigram proposal mechanism still can pose private data leakage risks. I would very much appreciate it if the authors can offer some ideas about initial solutions around this issue.

Author response:
We wanted to draw your attention to the numerous changes that we have made to the manuscript in response to your valuable comments and feedback. A summary of these changes is provided in the general comment we posted.  Based on your suggestions, we have made our code open source and provided a thorough discussion on the risk of privacy leakage from LFs. We have also outlined some initial solutions around the issues in this rebuttal and in the manuscript.

Reviewer comment (Presentation):
Figure 1 aims to visualize the strategy for generating LFs in the WSHFL method. However, the figure appears to be a high-level representation without detailed annotations or explanations. The flow between the different stages (a to d) and the significance of the values associated with the "IF nice" statements are not immediately clear. A more detailed caption or accompanying text might help in understanding the figure's content.

Author response:
Dear Reviewer W4Yv,  
Thank you for your time and effort in improving our paper.  

We wanted to draw your attention to the changes that we have made to the manuscript in response to your valuable comments and feedback. A summary of these changes is outlined in the general comment we posted.  **We would specifically like to draw your attention to our new experiments on federated semi-supervised learning that you emphasized (Appendix A.7).** In addition, we've incorporated changes to Figure 1 as per your suggestions, and introduced Appendix A.1, which complements Figure 1 with a detailed caption and explanation.

*We are eagerly looking forward to your response. If our revisions have effectively addressed your concerns, we kindly request you to consider adjusting your scores accordingly.*

Thank you once again for your thoughtful review.

Reviewer comment (Experiments):
The paper mentions comparisons with fully supervised baselines, but there seems to be a lack of comprehensive comparison with state-of-the-art methods in FSSL. Such a comparison would be crucial to establish the proposed method's effectiveness and relevance in the current research landscape.

Author response:
Dear Reviewer W4Yv,  

We thank you for your time and effort in reviewing our paper. We are happy that you like our motivating example. As you point out, on-device data annotation is a timely and relevant challenge. 

We have commented on the concerns on presentation and novelty raised by the reviewers in our general response. Regarding your concern on the *lack of comprehensive comparison with state-of-the-art methods in FSSL*, we want to note that most of these techniques have been proposed for image data and rely on some form of weak or strong data augmentation [1, 3, 5, 7, 8, 9, 10, 14, 15]. Thus, it is not trivial to naively extend them to text and time-series data. As such FSSL for new data modalities is an exciting research direction in itself. Having said that, we are **actively working on modifying some of these techniques with the aim of addressing your concern in time**. 

Please let us know if there are any further areas we can improve on.

Reviewer comment (Writing):
I find the method section 3 dense and confusing. I have following confusions/questions. How are the candidate LFs generated on client? What is exactly happening at the server, in particular in ExpertQuery function? This function is providing $u_t$ and the TrainClient function is giving estimates $\hat{u}_t$. What is the neural network trained on and for what purpose? $u_j$ is a variable denoting whether $\lambda_j$ is selected or not, but statements like “where we sequentially inspect the candidates in order to discover members of our desired class ($u_j=1)$ make it look like it is one of the classes in $\mathcal{Y}$. The presentation of section 3 can be improved greatly to make it more lucid. There is very little horizontal space between the algorithm block.

Author response:
Dear Reviewer mKax, 

We thank you for your time and effort in reviewing our paper. 

We are glad that you found our work to be well motivated, feasible and effective. We are actively working on improving the readability of Section 3, and have commented on the privacy concerns raised by the reviewers in our general response. Below are answers to your specific questions:

**How are the candidate LFs generated on the client?** 

Section 4 describes how LFs are generated in each client. Figures 2 and 3 show examples of text and time-series LFs respectively.

1. To generate text LFs, each client first identifies a set of unigrams (i.e. words) within a certain document frequency range. For example, say the client identifies 2 unigrams in their vocabulary: [nice, bad]. To create labeling functions, each client takes the cross product of these unigrams and the set of possible labels. If the set of possible labels is [negative sentiment, positive sentiment], then the client generates 4 LFs: [nice → positive sentiment, nice → negative sentiment, bad → positive sentiment, bad → negative sentiment]. To find keywords, we use the CountVectorizer implementation in scikit-learn for the same. 
2. To generate time-series labeling functions, we first cluster time-series into $k$ clusters. We refer to the cluster representatives (the mean time-series) of these clusters as representative templates. To construct labeling functions, we take the cross product of these representative templates and the set of possible labels. The time-series labeling function is shown in Fig. 3. Intuitively, the closer a time-series is to a representative template, the more likely it is to be assigned to the class corresponding to the representative template.  

We have now added these toy examples to the appendix.

**What is exactly happening at the server, in particular in `ExpertQuery` function?** 

`ExpertQuery` happens at the server, and is the process by which a human domain expert assigns $u_j=1$ (LF is useful/accurate) or $u_j=0$ (LF is not useful/accurate) to the given LF. In our work, the expert is positioned at the server.

**What is the neural network trained on and for what purpose?** 

From the context of your question, we assume you are referring to the neural network $h_k$. This network is trained on the LFs that have already been inspected by the expert $h_k: \lambda_j \to u_j$ and, intuitively, predicts the probability that a new LF will be considered useful by the expert. We use this network to later on to select which LF we will actually show to the expert for inspection.
How many times are each method run in experiments? Does running more times reduce variance? We mention in the "Methods and Models" section that "Unless mentioned otherwise, we repeat each experiment five times with different random seeds and report the mean and standard deviation." In preliminary analyses, we ran some experiments with 10 repetitions, but the behavior of the error bars did not qualitatively change. 

Please let us know if there are any further areas we can improve on. Based on your questions, we have already made some changes to improve the readability of our work.

Reviewer comment (Evaluation):
The solution is aimed at federated learning setting, so privacy should be ensured. It is not clear to me whether LFs can leak personal information and does the overall solution ensures privacy.

Author response:
Dear Reviewer mKax, 

Thank you for your time and effort in improving our paper. 

We wanted to draw your attention to the changes that we have made to the manuscript in response to your valuable comments and feedback. A summary of these changes is available in the general comment we have posted. We focused on enhancing the readability of our manuscript by updating Section 3, incorporating additional details and examples in the appendix, and engaging in a more comprehensive discussion on the potential privacy risks associated with labeling functions (LFs).

*We are eagerly looking forward to your response. If our revisions have effectively addressed your concerns, we kindly request you to consider adjusting your scores accordingly.*

Thank you once again for your thoughtful review.

Reviewer comment (Theory):
How does the method ensure privacy? The data is not shared with the experts/server but I think there is a potential risk of information leak via labeling functions.

Author response:
Dear Reviewer mKax, 

Thank you for your time and effort in improving our paper. 

We wanted to draw your attention to the changes that we have made to the manuscript in response to your valuable comments and feedback. A summary of these changes is available in the general comment we have posted. We focused on enhancing the readability of our manuscript by updating Section 3, incorporating additional details and examples in the appendix, and engaging in a more comprehensive discussion on the potential privacy risks associated with labeling functions (LFs).

*We are eagerly looking forward to your response. If our revisions have effectively addressed your concerns, we kindly request you to consider adjusting your scores accordingly.*

Thank you once again for your thoughtful review.

Reviewer comment (Experiments):
How many times are each method run in experiments? Does running more times reduce variance?

Author response:
Dear Reviewer mKax, 

We thank you for your time and effort in reviewing our paper. 

We are glad that you found our work to be well motivated, feasible and effective. We are actively working on improving the readability of Section 3, and have commented on the privacy concerns raised by the reviewers in our general response. Below are answers to your specific questions:

**How are the candidate LFs generated on the client?** 

Section 4 describes how LFs are generated in each client. Figures 2 and 3 show examples of text and time-series LFs respectively.

1. To generate text LFs, each client first identifies a set of unigrams (i.e. words) within a certain document frequency range. For example, say the client identifies 2 unigrams in their vocabulary: [nice, bad]. To create labeling functions, each client takes the cross product of these unigrams and the set of possible labels. If the set of possible labels is [negative sentiment, positive sentiment], then the client generates 4 LFs: [nice → positive sentiment, nice → negative sentiment, bad → positive sentiment, bad → negative sentiment]. To find keywords, we use the CountVectorizer implementation in scikit-learn for the same. 
2. To generate time-series labeling functions, we first cluster time-series into $k$ clusters. We refer to the cluster representatives (the mean time-series) of these clusters as representative templates. To construct labeling functions, we take the cross product of these representative templates and the set of possible labels. The time-series labeling function is shown in Fig. 3. Intuitively, the closer a time-series is to a representative template, the more likely it is to be assigned to the class corresponding to the representative template.  

We have now added these toy examples to the appendix.

**What is exactly happening at the server, in particular in `ExpertQuery` function?** 

`ExpertQuery` happens at the server, and is the process by which a human domain expert assigns $u_j=1$ (LF is useful/accurate) or $u_j=0$ (LF is not useful/accurate) to the given LF. In our work, the expert is positioned at the server.

**What is the neural network trained on and for what purpose?** 

From the context of your question, we assume you are referring to the neural network $h_k$. This network is trained on the LFs that have already been inspected by the expert $h_k: \lambda_j \to u_j$ and, intuitively, predicts the probability that a new LF will be considered useful by the expert. We use this network to later on to select which LF we will actually show to the expert for inspection.
How many times are each method run in experiments? Does running more times reduce variance? We mention in the "Methods and Models" section that "Unless mentioned otherwise, we repeat each experiment five times with different random seeds and report the mean and standard deviation." In preliminary analyses, we ran some experiments with 10 repetitions, but the behavior of the error bars did not qualitatively change. 

Please let us know if there are any further areas we can improve on. Based on your questions, we have already made some changes to improve the readability of our work.