From: NeurIPS.cc/2022/Conference/Program_Chairs
title: Paper Decision
decision: Accept

From: NeurIPS.cc/2022/Conference/Paper685/Area_Chair_N1HA
metareview: This paper proposes a VFL technique that is effective in practice (for some datasets) but intuitively may not be general enough for a significant portion of common settings, such as when the identifiers are names. While we recommend to accept this work, we hope the authors can seriously revise this paper in the final version on:
1. Ensuring that overclaiming statements are removed.
2. Adding evidence showing that when the identifiers are names, it can also be effective, in the final version.
3. Adding discussion on how the idea of "similar but not exactly the same samples can be beneficial" used in other topics, e.g., kNN, graph neural networks, semi-supervised learning, personalized federated learning, etc. is related to the one used in VFL.
recommendation: Accept
confidence: Less certain
award: No

From: NeurIPS.cc/2022/Conference/Paper685/Authors
title: Additonal Experiments on a Real Dataset Linked with "Names"
comment: 


We thank the reviewer for raising the score. To address the reviewer's last concern regarding the effectiveness of FedSim on identifiers like "names", we conduct experiments on an **additional real-world dataset "_company_"**, the identifiers in which are "**company names**". Our experimental results demonstrate that **FedSim has significant (5+%) improvement on some datasets linked on 'names'**.

## Task and Dataset Description

### Dataset A (Primary Party)
**Public Link**: https://www.kaggle.com/datasets/mirbektoktogaraev/should-this-loan-be-approved-or-denied

**Description**: A dataset of loan transactions between companies and banks from the U.S. Small Business Administration (SBA).

**Size**: 77225 x 91 (We use a subset of the original dataset due to time constraint)

**Labels**: SBA’s guaranteed amount of approved loan **ranging from 500 to 3,675,000**.

### Dataset B (Secondary Party)
**Public Link**: https://www.kaggle.com/datasets/peopledatalabssf/free-7-million-company-dataset

**Description**: A dataset of companies across 237 countries. The features include company size from 1-10,000+, company location, number of employees, etc.

**Size**: 220583 x 157 (We use a subset of the original dataset due to time constraint)

### Task
The task is to predict SBA’s guaranteed amount of approved loan in each transaction (**regression**) according to the historical transactions and company information. Two datasets are linked on the **company names**. We list some examples of such identifiers below.
> (Examples from dataset A)
> 
>ANASTASIA CONFECTIONS, INC.
>
>TRIANGLE MACHINE & MFG., INC.
>
>STEPHANIE DEVELOPMENT, L.L.C.
>
> (Examples from dataset B)
> 
>bright futures, a college and career management company
>
>new york life insurance company
>
>intercontinental hotels group (ihg)


## Experimental Setting
### Preprocessing
We preprocess the original dataset for training by 

- Converting all characters to lower case
- Removing all rows with NaN
- Converting categorical features to one-hot
- Removing all rows with duplicate company names

For Exact, we directly compare the **lower-case string identifiers** and get **77k exactly matched** pairs. For all other models (Top1Sim, AvgSim, FeatureSim, FedSim), we generate Bloom filters from strings and compare the Hamming similarity between Bloom filters. The generating of Bloom filters follows FEDERAL [28] (see Appendix H in the revision).

### Results
The RMSE scores of each baseline and FedSim are presented in the table below. We seek the reviewer's understanding that we are unable to present the variance due to the time constraint of the author-reviewer discussion. 

| Models     | RMSE  |
|------------|-------|
| Top1Sim    | 42210 |
| Exact      | 41875 |
| AvgSim     | 37598 |
| FeatureSim | 37618 |
| **FedSim**     | **35737** |

We observe from the table that **FedSim improves on all the baselines by at least 5%**, which demonstrates the efficacy of FedSim on datasets linked by 'names'. Notably, these large RMSE scores are reasonable because of the large range of labels (500 to 3,675,000). We validate this fact by observing that the **R2 scores of all the models are above 0.9**, which indicates the prediction is **highly related** to real labels.

Another observation is that **similarity-based models** (FedSim, FeatureSim, AvgSim) **outperforms Exact despite the noise introduced by generating Bloom filters**. This result further implies that exact linkage is not suitable for many real-world applications. Moreover, in the applications that require Bloom filters to preserve privacy, Exact is also ineligible to compare these Bloom filters.

Unfortunately, we do not have enough time to update another version of the revision or conduct comprehensive experiments on this dataset before the due of the author-reviewer discussion. These contents will be added in future revisions. Nonetheless, **we believe that these preliminary results have already demonstrated the effectiveness of FedSim on datasets linked by "names"**. We sincerely hope to have the opportunity to update these experimental results in the final revision.

From: NeurIPS.cc/2022/Conference/Paper685/Reviewer_W2Lk
title: Response acknowledged.
comment: Although I still have doubts on the general effectiveness of the proposed methods (especially when the identifiers are names), I am still grateful that the authors justify part of their assumptions (general availability of common attributes, and common lack of unique keys). I am willing to raise from 3 to 4.

From: NeurIPS.cc/2022/Conference/Paper685/Authors
title: Author-Reviewer Discussion Due Approaching
comment: Dear reviewers,

Thank you for your efforts in reviewing our paper. We specially thank Reviewer W2LK and 8KCf for their prompt response and meaningful discussion. We will appreciate it if we can receive more feedback from you before the Author-Reviewer discussion due. Any comments will be welcomed. We also hope you could reconsider your ratings if appropriate.

Best Wishes,

Authors of Submission 685

From: NeurIPS.cc/2022/Conference/Paper685/Authors
title: Further Response to Reviewer 8KCf
comment: We thank the feedback from the reviewer.

1. We appreciate the reviewer's advice. We had mentioned the training time in lines 206-207 in our draft. We have added the dimension in line 285 in our revision.

3. We apologize for the mistake of using the traditional Gaussian mechanism that requires $\varepsilon\in (0,1)$. This result can be extended to $\varepsilon >0$ in [1] by adding more noise (larger $\sigma$). **We have revised the theorem, proof, and figure in the updated revision**. This mistake does not affect our previous conclusion because the larger $\sigma$ makes differential privacy **even more impractical**. In summary, differential privacy that protects against all possible attacks is impractical in this scenario; as an alternative, our analysis suggests that the privacy risk against certain attacks can be significantly reduced by adding Gaussian noise.

[1] Balle, Borja, and Yu-Xiang Wang. "Improving the gaussian mechanism for differential privacy: Analytical calibration and optimal denoising." International Conference on Machine Learning. PMLR, 2018.

From: NeurIPS.cc/2022/Conference/Paper685/Reviewer_8KCf
title: Author response acknowledged
comment: I thank the authors for their response.

1. (for Q2, Q3) I appreciate the authors providing experiment details in the Appendix. By only reading the main Sections before seeing the response, I had no idea that this information was provided in the Appendix. I understand that due to the page limitation, it would be difficult to provide all this information in the experiment section. I recommend having a brief introduction in Section 6.1 (for example, see Appendix for training time and dimensions).

2. (for W1, Q1) I appreciate the authors' introducing more backgrounds about PPRL and FEDERAL [19] in the revised version of the paper.

3. (for W2) Similarity-based training paradigm design seems a challenging problem under the federated learning setting. However, in its current form, the solution to this problem seems somewhat straightforward (without concrete privacy analysis). The privacy analysis can also be an important contribution. I read Appendix E.2 and find it hard to follow. For example, Theorem 2 shows that Procedure G is differential privacy when ε, δ ∈ (0,1), while line 782 of page 22 shows that the value of ε is very large and not in range (0,1), thus does not follow Theorem 2. I think a more concrete discussion on privacy risks related to differential privacy would be necessary.

From: NeurIPS.cc/2022/Conference/Paper685/Authors
title: Response to the Clarification
comment: We thank the reviewer's prompt clarification and respond as below.

The assumption "similar shared attributes leading to similar records" is **generally true** according to our application study in GRLC. We demonstrate this point as follows. First, this assumption holds if the identifier **contains only "address"** as shown in our experiments, leading to a clear improvement. Therefore, this assumption also holds if the identifier **contains "address"** because the similarity of one attribute affects the similarity of the quasi-identifier containing this attribute. Second, this assumption potentially holds if the identifier **contains other attributes excluding "name" and "address"** because our experiments indicate that FedSim has improvement (though maybe marginal) when identifiers are "game title" or "time". Third, FedSim has no improvement only when the assumption does not hold on **all the shared attributes**. We summarize the eight applications with shared attributes in the table below. Please refer to Table 11 in Appendix I of the revision for the references.

| Identifier category                                              | #projects in the category | References       | Percentage | Improvement of FedSim  |
|------------------------------------------------------------------|---------------------------|------------------|------------|-----------------------|
| Identifier contains "address"                                    | 5                         | [15,29,39,44,45] | 62.5%      | Clear Improvement     |
| Identifier contains "name" and other attributes except "address" | 1                         | [19]             | 12.5%      | Potential Improvement |
| Identifier contains neither "name" nor "address"                  | 1                         | [29]             | 12.5%      | Potential Improvement |
| Identifier contains "name" only                                  | 1                         | [3]              | 12.5%      | No Improvement        |

From the table, it can be observed that **FedSim can have a clear improvement on the majority (62.5%) of applications with shared features**. On the contrary, the applications whose identifiers contain only "name" are relatively rare (12.5%).

We also highlight that FedSim produces close performance to baselines even when the identifier is "name" or unique ID, i.e., **FedSim $\ge$ baselines under all the identifiers** with a modest $K$. This is demonstrated by an experiment in Appendix D.6. This lack of improvement is because of the nature of the identifier instead of the technical design of FedSim. We present these marginal-improvement cases to demonstrate the limitation of FedSim, which is also requested by _Reviewer ddh8_.

Although our experiments have covered geolocations, time, and titles, we appreciate the reviewer's advice to conduct experiments on more real-world applications. Since the data of most linkage applications in GRLC are not publically available, we are trying to find real-world data linked on quasi-identifiers. If we can complete experiments before the revision due, we will add them to the revision.

### Comment on the example of dblp
The example of dblp is not wholely against our assumption. Considering a linkage between dblp and other websites on the "name" attribute only, although there are indeed numerous "Wei Wang" in different fields, "Wei Wang" and "Wei Wang" are **possibly** to be in the same field, whereas "Wei Wang" and "Chermaine Deepa Antony" are **unlikely** to be in the same field because no "Wei Wang" works in skeletal age assessment (the field of Chermaine). In addition, we may also need to consider similar names in the linkage since "Wei Wang" can be referred to as "Wang Wei" on other websites. In summary, this example implies that **the similarity of names indicates the possibility of being the same user**, thus indicating the similarity of data records. Nonetheless, this indication can be weak (i.e., $\Delta$ is small in our metric), thus the improvement of FedSim could be small in this case. The improvement on "game" and "song" also illustrate this weak indication. The indication is more evident when considering not only "name" but a quasi-identifier with other attributes, especially the fuzzy ones like "address".

From: NeurIPS.cc/2022/Conference/Paper685/Reviewer_W2Lk
title: Clarification of my remark
comment: I would like to clarify my exact meaning regarding the "lack of general effectiveness". 

From my perspective, the intuition or the foundation of your work is that, "samples with similar shared attributes are similar in general and may help VFL modeling". To show the validity of your foundation, you have to demonstrate two facts. 

1. Shared attributes widely exist for non-exact record matching. The authors have made a good job in demonstrating this point using examples of German Record Linkage Center. 
2. The shared attributes are generally helpful for better VFL modeling, or similarly, similar shared attributes lead to similar records in general. My doubt primarily lies in this point. In the experiments, the authors use two types of shared attributes, the first being geospatial information (house, bike, hdb), and the second being names (song, game). It is intuitive that geospatial information will be helpful. However, I find it hard to believe that similar names (song and game titles) may be helpful. In fact, the experiments also show this point, as the improvements on song and game are very marginal. In practice, when we talk about users, it is also not true that people with similar names have similar properties (e.g. there are numerous 'Wei Wang' on dblp (https://dblp.org/pid/35/7092.html), many of whom have wildly different research topics.) 
3. Finally, I would like to make it clear that, what I am doubtful is the general effectiveness of similar 'names' rather than similar 'strings'. Strings may be helpful, but names may not. Sorry for the confusion.

To summarize, it seems that, as far as the experiments show, the effectiveness of FedSim is prominent only when the shared attributes are geospatial locations. More justification is needed to verify its general effectiveness.

From: NeurIPS.cc/2022/Conference/Paper685/Authors
title: Further Response to Reviewer W2LK
comment: 
In the beginning, we would like to correct that we **did not** claim that "_in most cases, quasi-identifiers of users (address, name, birth date, postal code) are strings_". Instead, we claimed that "_the similarity of some fields can reflect the similarity of the property, such as **address** (**5/11** projects) in **GPS or string** format_". This implies that **the address existing in almost half of projects is a common identifier in practice**. Our experiments show **significant improvement** of FedSim on 'house' and 'hdb' dataset with **geo-encoded address** as identifiers.

**Using geo-encoded address is a common practice in record linkage**.  Among the five applications using addresses, one explicitly claims that they use geocoded addresses instead of strings.

> "_Second, individual addresses were geocoded and geographically clustered ..._" - [4]

Two have geolocations in the dataset but do not claim the usage in the linkage.

> "_It may be possible to assign establishments to these companies, in principle, by including geo-information on companies’ and establishments’ addresses._" - [2]
>   "_Microm provides geomarketing data at the street level and was added to the PIAAC data based on respondents’ addresses._" - [5]

**The geolocation and string can be transferred to each other given a map.**  Even if the original addresses are strings, we can easily transfer these strings to the geolocations if it brings better accuracy. Intuitively, geolocation contains more spatial information than strings. Thus, geolocation is more suitable for the VFL applications that consider spatial information.

In addition, **the data type is not a limitation by design in FedSim**. FedSim focuses on _learning useful information from similarities_, while _designing linkage functions to calculate similarities_ is an orthogonal topic studied in PPRL. No matter what data type the identifier is, given a linkage function that can obtain useful similarities, FedSim can improve based on the similarities. We highlight that the reason why "game" and "song" has smaller improvement is **their similarity distribution benefiting Top1Sim** (i.e., $\Delta(Top1Sim)$ is small) instead of their identifiers being strings.

From: NeurIPS.cc/2022/Conference/Paper685/Reviewer_W2Lk
title: Author response acknowledged
comment: I acknowledge the author response. 

1. I appreciate the authors' case study on real world record linkage. The results provide justification on real-world data linkage processes and offer insights about when FedSim can be applied. This partially addresses my concerns. 

2. Given the results, I still consider the assumption "shared features are useful" strong. As you have made clear in the German Record Linkage Center case, in most cases, quasi-identifiers of users (address, name, birth date, postal code) are strings. However, in your evaluation, on 'game' and 'song' data whose shared attributes are strings, the improvements are quite marginal (0.17% on game, 0.01 on song). Therefore, in the existing form, FedSim does not seem to be widely effective. 

3. I appreciate the authors clarifying the inference process and removing the overclaiming. 

However, in its current form, the paper still suffers from problems such as limited insights and lack of general effectiveness. I thus cannot recommend this paper or change my rating to a more positive side.

From: NeurIPS.cc/2022/Conference/Paper685/Authors
title: Response to Reviewer W2LK (Part 2)
comment: ## Response to Weakness

3. ___"Biased evaluation"___

  

As we have explained in "the response to Weakness 2", our experiments are based on two assumptions that **commonly exist in real-world applications**. For example, address (in house and hdb dataset) is used for linkage in five out of 13 applications.

  

Besides those datasets where FedSim outperforms baselines, **we have also conducted a study on when FedSim fails in Appendix D.6**. We generate a synthetic dataset "sklean-random" with unique pure random identifiers. Our results in Table 9 show that FedSim has a close performance to Exact/Top1Sim only when $K=30$. If we increase $K$ to 50, the accuracy of FedSim significantly drops below Exact/Top1Sim due to overfitting. In addition, song dataset linked with song titles is another good example. FedSim has little improvement on song because the similarity of titles is little related to the similarity of content.

  

In summary, FedSim has a close performance to Exact on meaningless shared features, which has already been shown in our experiments.

  

  

4. ___"Inference procedure not discussed"___

  

During the inference, for each data record in the primary party, $K$ most similar data records in secondary parties are linked. These linked data records are fed into the FedSim model to derive the final prediction. We have clarified this point in paragraph 5 of Section 4.2.

  

5. ___"Minor issues: coloring of figures"___

  

We appreciate the reviewer's advice and have set a different marker for each line in our figures.

  

## Response to Questions

  

1. Please refer to our response to "Weakness 4" for details.

  

2. We will add experiments on both datasets once completed. As we have elaborated in "Response to Weakness 3", **we would expect FedSim to have a similar performance to Exact on the datasets with meaningless shared features**.

  

3. We appreciate the reviewer's advice and have removed the initial claim in our revision.

From: NeurIPS.cc/2022/Conference/Paper685/Authors
title: Response to Reviewer W2Lk (Part 1)
comment: ## Response to Weakness

1. ___"The insight seems to be overclaiming."___

We appreciate the reviewer's advice and have removed this claim in our revision.

2. ___"Strong and not realistic assumptions on the 'common features'."___

Since VFL is a rather recent paradigm with fewer real applications, we investigate the applications of a traditional area "record linkage" which has many existing applications. The application scope of record linkage reflects the application scope of VFL, because all the VFL algorithms require the data to be linked (either on ID or other features) before training.

German Record Linkage Center (GRLC) published an article [1] to summarize its completed linkage projects since 2011. We list the information of identifiers in these eight applications as follows.

- "_Since there was no common identifier available to make such an assignment, the record linkage had to be conducted using alternative identifiers, such as establishment names and addresses._" - [2]

- "_We use the Merge ToolBox (MTB) software (Schnell et al., 2004) to perform distancebased record linkage of the two address data files. The fields street and city name were compared with N-grams throughout._" - [3]

- "_Data is stored in spells that are linked to persons. The spells come along with a variety of socio-demographic variables such as gender, age, or nationality as well as spatial information such as addresses, municipality, and region type._" - [4]

- "_Microm provides geomarketing data at the street level and was added to the PIAAC data based on respondents’ addresses._" - [5]

- "_Regarding their partners, given the lack of a unique identifier key, it was necessary to link the individuals by using non-unique and error-prone identifiers, such as names and addresses._" - [6]

- "_In the first exact linkage step, the complete agreement in all other linkage-relevant fields was decisive, i.e. first and last name, date of birth, gender, street, house number, postal code and city._" (Translated from German) - [7]

- "_to link the data rows in which there is exact match of characteristics (e.g. spins in the name initials or in the date of birth, deviations by one or a few days in different dates)_" (Translated from German) - [8]

- "_We link establishments to transactions based on company names, because there are no common company identifiers that would easily link our PE buyout sample to the Establishment History Panel (BHP)._" - [9]

Two observations can be made from these projects. First, among the 11 completed projects, eight projects require linking datasets without user ID. **This implies that, in the majority (around 8/11) of real applications, there does not exist a shared user ID, which supports our first assumption**. Second, the similarity of some fields can reflect the similarity of the property, such as address (5/11 projects) in GPS or string format. Even for the field whose own similarity does not reflect record property, such as name, birth date, and postal code, the similarity of the quasi-identifier containing these fields (2/11 projects) can reflect the property. This is because records with more matched fields are more likely to belong to the same user, especially when considering typos that widely exist in practice [2-9]. Therefore, **the similarity of shared features is related to the property of records in many (around 7/11) real-world cases, which supports our second assumption.** We have included the evidence in the introduction and Appendix I of our revision.

[1] Antoni, Manfred, and Rainer Schnell. "The past, present and future of the German Record Linkage Center (GRLC)." Jahrbücher für Nationalökonomie und Statistik (2019)

[2] Schild, Christopher-Johannes. "Linking'Orbis' Company Data with Establishment Data from the German Federal Employment Agency." (2016).

[3] Eberle, Johanna, and Michael Weinhardt. "Record linkage of the linked employer-employee survey of the socio-economic panel study (SOEP-LEE) and the establishment history panel (BHP)." (2016).

[4] Kroh, Martin, et al. The 2013 IAB-SOEP Migration Sample (M1): Sampling design and weighting adjustment. No. 271. SOEP Survey Papers, 2015.

[5] Perry, Anja, and Beatrice Rammstedt. "The research data center PIAAC at GESIS." Jahrbücher für Nationalökonomie und Statistik 236.5 (2016).

[6] Schild, Christopher-Johannes, and Manfred Antoni. "Linking Survey Data with Administrative Social Security Data-the Project'Interactions Between Capabilities in Work and Private Life'." German Record Linkage Center, Working Paper Series (2014).

[7] Antoni, Manfred, and Arne Bethmann. "PASS-Befragungsdaten verknüpft mit administrativen Daten des IAB." (2014).

[8] Gramlich, Tobias. "’STROKES’–Record Linkage der Schlaganfälle in Hessen 2007-2010 (Strokes-Record Linkage of Stroke Cases in Hesse 2007-2010)." (2014).

[9] Antoni, Manfred, Ernst Maug, and Stefan Obernberger. "Private equity and human capital risk." Journal of Financial Economics 133.3 (2019).

From: NeurIPS.cc/2022/Conference/Paper685/Authors
title: Response to Reviewer ddh8
comment: 
## Response to Weakness

1. We appreciate the reviewer's advice and have removed this claim in the revision.

  

2. **Only on the five real-world datasets without privacy protection, does FedSim consistently outperform baselines.** There are some cases in that FedSim has similar or lower accuracy in our experiments. These cases include

  

- In Figure 5(a), on boone (synthetic) dataset, **FedSim always has almost the same accuracy as Top1Sim** under different $\sigma_{cf}$.

- In Figure 5(a), on frog (synthetic) dataset, **FedSim is outperformed by AvgSim under large scale of noise ($\sigma_{cf}=0.2$)**, because the similarities is no longer meaningful.

- In Figure 6, on house, taxi dataset, **FedSim is outperformed by AvgSim** under small $\tau$ (large noise on similarities).

- In Figure 6, on game dataset, **FedSim is outperformed by Top1Sim** under large $\tau$ (small noise on similarities).

- In Figure 8, on song dataset, **FedSim has a close performance to AvgSim and FeatureSim**.

  

Furthermore, **we have conducted a study on when FedSim fails in Appendix D.6**. We generate a synthetic dataset "sklean-random" with unique pure random identifiers. Our results in Table 9 show that FedSim has **close performance to Exact/Top1Sim** only when $K=30$. If we increase $K$ to 50, **the accuracy of FedSim significantly drops below Exact/Top1Sim** due to overfitting.

  

In summary, **our experimental results have already shown that FedSim fails in certain cases as we elaborate above**. We add more discussions on the assumption and limitations of FedSim in Appendix I of the revision.

  

3. The framework of FedSim cannot be directly applied to other VFL algorithms, e.g., logistic regression and tree-based algorithms. This is because FedSim requires the similarity model and merge model to be trained together with the main VFL model, which requires the VFL algorithm to be neural-network-based. To adapt FedSim to other VFL algorithms, a possible direction is to exchange the intermediate information in these algorithms like exchanging gradients in SplitNN. We leave this topic as our future work. We discuss these future directions in Appendix F of the revision.

  

## Response to Questions

 We add a detailed discussion of the limitation (assumption) and application scope of FedSim in Appendix I of the revision. In brief, we assume that the similarity between identifiers is related to the similarity between data records. We also demonstrate that this assumption commonly holds in practice.

In addition, in Section 4.3, we have proposed a data-linkage-based metric to estimate the improvement of FedSim over baselines (i.e., AvgSim, Top1Sim, Exact) without training. In brief, we calculate a score $\Delta$ based on the similarity distribution to estimate the potential improvement of FedSim. This metric has also been validated in our experiments (Figure 5(b)). From the figure, we observe that **$\Delta(Top1Sim)$ is positively correlated with real improvement on Top1Sim**, indicating that the metric can be effectively used to estimate the improvement of FedSim without training.

From: NeurIPS.cc/2022/Conference/Paper685/Authors
title: Response to Reviewer 8KCf
comment: 
## Response to Weakness

W1: We include a brief introduction of FEDERAL [19] in **Appendix H of the revision** due to the page constraint.

  

W2: Besides considering PPRL in VFL, the main challenges are similarity-based training paradigm design and privacy risk analysis.

  

W3: We formulate the perturbation under differential privacy in **Appendix E.2 of the revision**. Specifically, we demonstrate that __differential privacy that protects against all possible attacks is impractical in this scenario__; as an alternative, our analysis suggests that the privacy risk against certain attacks can be significantly reduced by adding Gaussian noise. Taking house dataset as an example, $\mu_0=-46237.78, \sigma_0=21178.86$. Letting $\delta=10^{-5}$, we can derive the values of $\tau$ and $\varepsilon$ under different noise scales $\sigma$. When setting $\sigma=1$, the differential privacy parameter $\varepsilon=1.5\times 10^6$ implies that there is almost no privacy guarantee at all. Nonetheless, the attacking success rate $\tau=2.7\times 10^{-5}$ suggests that the privacy risk from certain attacks can be quite low.

  

## Response to Questions

Q1: We follow the reviewer's advice by highlighting our contributions to training paradigm design and privacy analysis in the introduction of our revision. Meanwhile, we **analyze the noise under differential privacy and compare differential privacy with our metric in Appendix E.2 of the revision**. Please refer to the "Response to W3" for our main findings.

  

Q2: For the training process, **we have compared the training time of FedSim with baselines in Appendix D.4**. The results indicate that FedSim requires **longer but acceptable training time** compared to baselines. In addition, we also present the **number of parameters of each model in Appendix D.4 in the revision**. For the linkage process, the computational complexity has been analyzed in FEDERAL [19]. We also empirically **present the linkage time in Appendix D.4 in the revision**. The results indicate that linking Bloom filters for privacy is more time-consuming than linking raw float/string features. This indicates that a more efficient PPRL approach is desired.

  

Q3: The dimension of identifiers in each dataset is presented in Table 3 in Appendix C. **Most datasets contain multi-dimensional identifiers with dimensions ranging from 2 to 30.** For the datasets with multi-dimensional identifiers, we simply **calculate a similarity value between two identifier vectors** to improve efficiency and avoid overfitting. This might cause some information loss, but we find **the accuracy loss is negligible in our experiments**. In addition, we highlight that $\sigma_{cf}$ is only for generating synthetic datasets. **The noise $\sigma$ for privacy is added to a similarity value instead of each dimension.**

  

Q4: We have introduced more backgrounds of PPRL in Appendix H in the revision. For both numeric and nominal identifiers, we generate a Bloom filter for each identifier by following [19]. The Hamming distances ($\text{dist}(k^P_i,k^S_j)$) between these Bloom filters are calculated. Then, we perform normalization (Equation 1) and noise addition (Equation 2) to obtain $s_{ij}$.

From: NeurIPS.cc/2022/Conference/Paper685/Authors
title: Response to Reviewer gik3
comment: 
## Response to Questions

Q1: "___Can you describe how the presented work can be further improved?___"

  

Response to Q1: Regarding the performance, the CNN merge gate still introduces many additional parameters. The sort gate and merge gate can be potentially replaced by attention; thus, further performance improvement is possible. Regarding privacy, more advanced attacks and stronger privacy metrics are desired.

  

Q2: "___What are the potential privacy implications when choosing a large K value for the number of soft linkages during training?___"

  

Response to Q2: Increasing $K$ enlarges the possibility of different $k_i^A$ matching the same $k_j^B$, thus increasing $|Q|$ in Equation 12 in Appendix A. As explained in the last paragraph of Appendix A, increasing |Q| leads to a lower success rate of the intuitive attack due to the noise added to similarities. In summary, **increasing $K$ potentially leads to higher privacy risk, but the success rate of the intuitive attack remains unchanged.**

From: NeurIPS.cc/2022/Conference/Paper685/Authors
title: Response to Reviewer S97P
comment: 
## Response to Weakness

1. Our contribution includes two parts: the framework of FedSim and the privacy concerns of FedSim. For the first part, FedSim only modifies the model structure of SplitNN, while following the training process of SplitNN. Therefore, the analysis of convergence in FedSim also follows the analysis on SplitNN, which is orthogonal to this paper. For the second part, we theoretically prove the success rate of our proposed attack, and theoretically demonstrate the impracticability of differential privacy in this scenario.

  

2. In **Appendix E.2 of our revision**, we demonstrate that **differential privacy that protects against all possible attacks is impractical in this scenario**; as an alternative, our analysis suggests that the privacy risk against certain attacks can be significantly reduced by adding Gaussian noise. Taking house dataset as an example, $\mu_0=-46237.78, \sigma_0=21178.86$. Letting $\delta=10^{-5}$, we can derive the values of $\tau$ and $\varepsilon$ under different noise scales $\sigma$. When setting $\sigma=1$, the differential privacy parameter $\varepsilon=1.5\times 10^6$ implies that there is almost no privacy guarantee at all. Nonetheless, the attacking success rate $\tau=2.7\times 10^{-5}$ suggests that the privacy risk from certain attacks can be quite low.

  

3. The main distinction in the soft linkage is the similarity metric. As explained in the second paragraph of Appendix C, our experiments already covered different similarity metrics including Euclidean similarity, edit similarity, and Hamming similarity. We further add **an ablation study of similarity metrics in Appendix D.7 in the revision**. The results indicate that the similarity metric does not significantly affect the performance.

  

4. For the training process, **we have compared the training time of FedSim with baselines in Appendix D.4**. The results indicate that FedSim requires **longer but acceptable training time** compared to baselines. In addition, we also present the **number of parameters of each model in Appendix D.4 in the revision**. For the linkage process, the computational complexity has been analyzed in FEDERAL [19]. We also empirically **present the linkage time in Appendix D.4 in the revision**. The results indicate that linking Bloom filters for privacy is more time-consuming than linking raw float/string features. This indicates that a more efficient PPRL approach is desired.

  

## Response to Questions

Q1: According to a survey [40], most linkage approaches contain three steps: blocking, comparison, and classification. Our soft linkage approach, following traditional PPRL in "blocking" and "comparison" steps, **directly outputs the similarities with normalization and noise addition** without performing the "classification" step. We have highlighted this point in the first paragraph of Section 4.1 in our revision.

  

Q2: The output of sort gate $\mathbf{o}_i'$, containing $K$ output vectors with $l_m$ dimensions, is regarded as a 2D input of size $K\times l_m$ for the CNN merge gate. Then, the merge gate effectively merges these output vectors with close similarities and outputs the final prediction. We have clarified this point the the "merge gate" paragraph in Section 4.2 in the revision.

  

Q3: Please refer to "Response to Weakness 2"

  

Q4: Yes, FedSim requires **longer but acceptable training time** compared to baselines. Please refer to "Response to Weakness (4)" for details.

From: NeurIPS.cc/2022/Conference/Paper685/Reviewer_S97P
rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary: This paper introduces a novel FedSim approach which leverages the record similarity for soft linkage on top of the SplitNN. It is an improvement over the existing Vertical Federated Learning, which is mainly based on the Exact and Top1Sim. The paper also consider the privacy risk caused by similarity. The experimental results show the performance improvements of FedSim against some existing benchmarks. 
strengths_and_weaknesses: Pros: 
1. The problem is interesting and timely
2. The idea is simple and the results demonstrate its power
3. The paper is easy to read

Cons: 
1. Lack of theoretical analysis of the proposed method
2. The privacy analysis seems to be weak; can it be linked to differential privacy? 
3. The experimental results are not convincing; lack of detailed ablation study. For instance, it will be interesting to know how different soft linkage approaches can help with the task. 
4. The computational complexity is not discussed (and comparing with other method) 
questions: Q1: There are many soft linkage research, i.e., matching pairs between different parties. Can you briefly explain what is the significance/novelty/difference between your approach and other approaches on record linkage?
 
Q2: The merge gate is not clear; can you provide a formal description?

Q3: How the privacy guarantee relates to differential privacy?

Q4: Computationally, will the approach introduce additional cost?

limitations: No 
ethics_flag: No
soundness: 3 good
presentation: 3 good
contribution: 3 good

From: NeurIPS.cc/2022/Conference/Paper685/Reviewer_gik3
rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
confidence: 2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary: This paper describes a framework for enabling VFL while addressing records linkage as a part of the training process. 
The authors motivated their work by applications where existing VFL record linkage solutions were insufficient. For instance, if the linking data include GPS coordinates or names it would make sense to perform fuzzy matching. Additionally, the authors buckled the trend when instead of dividing record linkage and training into separate phases they combined them into one coupled phase where the weight or top K similarities would also be discovered during training. 
Even through the paper is primary focuses on 2 parties VFL the authors also describe how to extend their framework to multiple parties.

strengths_and_weaknesses: This paper is well-written and addresses an important issue. It is well motivated and supported by strong results on multiple synthetic and real-world datasets. The authors attached source code and instructions on how to build the framework and how to run tests. So it appears that their work can be replicated.

questions: Can you describe how the presented work can be further improved? 
What are the potential privacy implications when choosing large K value for the number of soft linkages during training?
limitations: I believe the authors have sufficiently addressed the limitations of their work related to scalability and privacy.
ethics_flag: No
soundness: 3 good
presentation: 4 excellent
contribution: 3 good

From: NeurIPS.cc/2022/Conference/Paper685/Reviewer_8KCf
rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary: Federated Learning (FL) is a collaborative learning framework for training a model from datasets owned by different parties. A commonly used paradigm for FL is Vertical Federated Learning (VFL), where parities share different features of the same set of samples. Before actually doing the privacy-preserving training process, parties need to first link features with the same or similar identifiers. Existing VFL solutions focus more on exact identifier linkage (one-to-one linkage) by leveraging hashing methods or cryptographic protocols like Private Set Intersection (PSI). However, in some scenarios, linking identifiers with high (but not top) similarity benefits the training. 

This paper designed a coupled training paradigm, FedSim, that introduced Privacy-Preserving Record Linkage (PPRL) that supports high similarity linkage into VFL. Under the SplitNN framework, this paper introduced a new model structure that can take the advantage of the additional similarity information to train a better SplitNN model. The experiments show that the proposed framework can train a better model in a vertical federated manner with little additional cost.
strengths_and_weaknesses: Strengths:

S1: Identify a good topic that needs to be addressed in VFL. 

S2: Give a reasonable and lightweight solution for the problem. The solution can provide some privacy guarantees under the weak security model (without formally provable security).

S3: Provide sufficient experiments.

Weaknesses:

W1: Lack of brief introduction for the PPRL method, making the proposal somewhat hard to understand without seeing [19].

W2: Limited contribution to introducing PPRL into VFL. 

W3: The privacy guarantees provided by perturbing identifiers using Gaussian noises need to be further discussed.
questions: Q1: Introducing PPRL into VFL seems a limited contribution.
Although this paper identified a good topic in VFL (Similarity linkage), directly introducing PPRL into VFL seems a straightforward way of solving the problem. I found the main contribution of the paper is not introducing PPRL into VFL, but a new framework (shown in Section 4.2) that leverages the additional similarity information for benefiting VFL. Also, this paper in fact introduced a new way of securing PPRL, i.e., adding Gaussian noises during the PPRL step. To the best of my knowledge, this method can be formalized using Gaussian Differential Privacy or related techniques. I strongly recommend the authors have further analysis and discussions on these two parts, instead of only highlighting the (somewhat limited) contribution of introducing PPRL into VFL.

Q2: About performance comparisons.
Although there are some methods that support similarity linkage, most of which introduce cryptographic tools (e.g., homomorphic encryption, public-key encryption). Such cryptographic tools would introduce high computation costs, making the VFL inefficient and hard to be deployed in practice. This paper introduced a lightweight solution to this problem, namely, the Bloom Filter method proposed in [19]. Even though the Bloom Filter method is lightweight compared with heavy cryptographic operations, it would still introduce some additional symmetric cryptographic operations (like hashes) and some additional communications (like sending Bloom Filters to the server and sharing the similarities). I am wondering if the authors can provide time consumption and communication costs for each phase of FedSim. I personally think these experimental results would further show the efficiency of FedSim.

Q3: About the high-dimensional identifier case
In Section 6.1, the authors stated that the experiments used a perturbed identifier for linkage, which is generated by adding Gaussian noises with the same variance (\sigma_{cf}) to each dimension (See P7, line 288). However, it seems that all experiments use only one dimension as the identifier. It would be better if the authors provide additional discussion on the high-dimensional identifier case.

Q4: About the background of the Bloom Filter PPRL method
Bloom Filter is a well-known technique that has been used in many situations. The Bloom Filter PPRL method used in this paper is a modified one introduced in [19], which does not only support similarity computation between nominal inputs but also numeric inputs. Without the necessary background introduction about the Bloom Filter method proposed in [19], it seems hard for the reader to understand how to calculate the similarity for numeric identifier in FedSim. 
limitations: The authors provided limitations adequately in the Appendix. 
ethics_flag: No
soundness: 2 fair
presentation: 3 good
contribution: 2 fair

From: NeurIPS.cc/2022/Conference/Paper685/Reviewer_ddh8
rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary: The paper proposes FedSim, a new VFL training paradigm. FedSim explicitly explores the similarity information between the non-matched pairs and couples the correspond module into SplitNN (an existing VFL training model). The authors also provide privacy guarantee to a greedy attack algorithm. Finally, they verify the effectiveness of their algorithm on synthetic and real datasets.
strengths_and_weaknesses: Strengths: 
1. The paper is easy to follow: the motivation is sound, the flow is smooth and the presentation is clear.
2. The paper makes solid contribution by witnessing the potential information loss in the process of exact-match record linkage, and coupling the similarity information with the existing SplitNN model to improve the performance of VFL. Overall this feels like a simple-but-effective idea to me and is the major factor that reflects my rating.

Weaknesses:
1. The author emphasize that 'non-matched pairs can also benefit VFL with a properly designed training method', and they see it as a 'counterintuitive' conclusion. While I agree that this is indeed a nice and novel observation, I would like to point out that in contrast to being 'counterintuitive', this feels most natural to me. Placing it in a broader context of machine learning, an analogy would be 'data-augmentation will benefit supervised learning'. I think the tone needs to be revised here --- it is ok to acknowledge that your approach is 'simple' --- this will actually serve as a plus if it is also 'effective'. 
2. Following the point above, while I believe that a specific type of data augmentation will be effective in **some** learning tasks, there **must** exist other scenarios where it won't work. Similarly, I don't think it is the case that FedSim **always** outperforms its counterparts. Sometimes exact-match is more favorable, for example, top-K record linkage based on the similarity of phone numbers does not make too much sense to me. The authors do **not** discuss the limitations of their approach at all, and specifically, the experiment results demonstrate that FedSim consistently beats other baselines in all **selected** tasks, which actually downgrades my overall rating of this paper --- I tend to believe that the experimental evaluations are 'crafted'.
3. I also do not see enough discussions on the reliance of the training framework. Can the similarity-based module be easily incorporated into other VFL protocols (e.g., logistic regression based on homomorphic encryption)?
questions: I would like to see a **detailed** discussion of the limitations, application scope (beyond listing two positive examples) and comparison to existing approaches of FedSim. This will greatly benefit the practitioners. Say if two financial institutions want to vertically train a machine learning model, should they consider your algorithm or not and why?
limitations: The authors only point out a few extensions (e.g., generalization to multiple parties, other attack methods). 
ethics_flag: No
soundness: 3 good
presentation: 3 good
contribution: 3 good

From: NeurIPS.cc/2022/Conference/Paper685/Reviewer_W2Lk
rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary: This paper proposes a framework for vertical federated learning (VFL), called FedSim. 

In general, VFL contains two steps, the first being the record linkage, the second being the model learning. FedSim is based on the observation that existing methods separate these two steps and focus solely on the model learning step. For the record linkage step, existing works use simple exact/one-to-one matching, and thus fail to exploit information from non-matched entities. 

FedSim proposes a method to exploit similar records in VFL. It matches samples across parties according to similarity, and further uses the similarity information to adaptively fuse features from the other party. Further, a simple way to estimate the performance improvement is proposed, and a theoretically grounded method to ensure privacy is proposed. Experiments on several datasets show the effectiveness of FedSim. 
strengths_and_weaknesses: ## Originality
The main idea of the paper is to use similar but not exactly matched samples to assist vertical federated learning. 

+ **Pros** The idea is somewhat new in VFL. The common practice of VFL is that training is only performed on exactly matched samples. 

- **Cons** The authors seem to be overclaiming on the originality and insights of the idea. The authors highlight the insight that "non-matched pairs can also benefit VFL" in line 68, Introduction, and regard it as 'counter-intuitive'. However, although the insight is indeed not explored in VFL, I would not consider the insight highly novel or counter-intuitive. Specifically, in the sense of general machine learning, the insight that "similar but not the same samples can boost machine learning performance" has been well recognized and exploited. For example, Graph Neural Networks (GNN) [1, 2] are perfect illustrations of this insight. The primary idea of graph neural networks is to aggregate features from neighboring nodes of the central node, which is basically using similar samples to improve feature learning for the central node. Furthermore, even though there may not be explicit links (e.g. friends in social networks, citations in academic networks), using general feature similarity to build implicit graphs is also a common idea. For example, [3] builds a nearest neighbor graph on latent features to facilitate deep semi-supervised learning. Therefore, the insight the authors claim has been known and used in a larger area and may not be "counter-intuitive" or "new". The authors indeed made a valid observation, but they should not over-claim. 

## Quality
The designed method to exploit record similarity contains a merge gate, a weight gate, and a sort gate. Similar but non-exactly matched samples are weighted by the sort gate and the weight gate to assist VFL. The authors also propose a method based on histogram to estimate performance gains. Further, the authors discuss the potential privacy leakage caused by the record similarity, and propose a method to protect data privacy with theoretical analysis. Experimental results show that FedSim outperforms simple baselines that utilize record similarity (Top1Sim, AvgSim). Also, the impact of privacy protection on accuracy is studied, and the improvement estimation method is shown to have positive correlation with the actual performance improvement. 
 
- **Pro**. I appreciate the authors' discussion and method about privacy leakage caused by the record similarity. As FL puts emphasis on data privacy, such discussion is necessary. Also, it is good to see the experiments on the tradeoff between privacy and accuracy. I also consider the improvement estimation method interesting. Experiments also validate the effectiveness of the estimation. 

- **Con**. The authors made two important assumptions in this paper, that some features are shared between the primary and the secondary parties, and that similarity of the shared features are indicative of the common information (Section 4.3). However, in practice, I would like to raise my doubts on both assumptions. 
    1. **Shared features**. In practical VFL applications where each sample corresponds to a user, the common features held by different parties are often limited [4]. Specifically, in most cases, the only shared feature may be the user ID, and practitioners apply private set intersection (PSI). In this case, FedSim cannot apply. 
    2. **Usefulness of shared features**. Even if such features do exist, it is non-trivial to determine whether the similarities between them are useful. For example, suppose there exist several quasi-identifiers regarding a user, e.g. name, phone number, email, and so on. It is generally hard to say that people with similar names/phone numbers/email service providers have similar properties. 
    3. **Experiments are biased towards the two assumptions**. Continuing from the unrealistic assumptions, I would like to point out that the authors have manually chosen datasets with indicative common features. For example, in the bike dataset, the shared features are longitudes and latitudes of start/end points, and the task is to predict the travel time. It is clear that routes with similar start/end points will have similar travel time, which leads to a bias that favors similarity-based methods against exact-matching-based ones. It is not clear how FedSim compares against exact-matching-based methods when no such indicative common features are present, e.g. the only common features are quasi-identifiers such as names. 

## Clarity
The presentation of this paper is well-organized. I find it easy to follow the paper. 

## Significance. 
I find the paper to be marginally significant. As said in the "originality" part, the paper builds on a known idea to improve the performance of VFL. Thus, while the idea and the techniques are sound, it may not bring much insight to the community. 

## Overall Strengths: 
- **Good Presentation**. The paper is well-presented and generally easy to follow. 
- **Discussion on privacy**. See 'quality'. 
- **Improvement estimation method is interesting**. See 'quality'. 

## Overall Weaknesses:
- **The insight seems to be overclaiming.**. See "originality". 
- **Strong and not realistic assumptions on the "common features"**. See 'quality'. 
- **Biased evaluation**. See "quality"
- **Inference procedure not discussed** The authors fail to discuss how the primary party does inference for new samples. Does it need to query similar samples from the secondary party? I would like to see a discussion on the extra procedures and extra costs of the inference phase. 
- **Minor issues**: Please use a better coloring to plot the figures. The current coloring is not friendly to black-and-white printers and color-blind readers.  

[1] Hamilton et al. Inductive Representation Learning in Large Graphs. NIPS 2017

[2] Kipf et al. Semi-supervised Classification using Graph Convolutional Networks. ICLR 2017

[3] Iscen et al. Label Propagation for Deep Semi-supervised Learning. CVPR 2019

[4] Sun et al. Vertical Federated Learning without Revealing Intersection Membership. Arxiv 2021.

questions: 1. How does a model learned by FedSim perform inference? Any additional procedures/overhead?

2. I would like to see how FedSim performs on other datasets commonly used in VFL, e.g. Criteo and Avazu [4]. In these cases, no strong priors about indicative common features (e.g. longitude/latitude) exist. 

3. Please justify how the insights learned from FedSim differs from the insights learned from existing works on GNN and graph-based semi-supervised learning ([1-3]). Otherwise, you should tone down the claims in the Introduction. 
limitations: I don't see anything that raises ethical concerns. 
ethics_flag: No
ethics_review_area: I don’t know
soundness: 2 fair
presentation: 3 good
contribution: 2 fair