Paper: Rotative Factorization Machines (ICLR-2024)

Reviewer comment (Novelty):
The proposed method is similar to EulerNet with a stack of attention module.

Author response:
In Section 3.3, we highlight the novelty and differences between our approach and other feature interaction methods.	
Our proposed method and EulerNet [1] both model the feature interactions in the complex vector space, but there are still some noticeable differences:

- First, EulerNet [1] uses simple **linear layers** to model arbitrary-order  feature interactions, with interaction orders determined by predefined linear combination coefficients.
In contrast, our approach develops a **self-attentive rotation mechanism** to model feature interactions, with interaction orders learned from attention scores derived from the context.
This enables our approach to effectively capture feature dependencies in varying contexts.

- Second, EulerNet [1] simultaneously optimizes the modulus and phase of the complex feature interactions in a coupled way, leading to the the **exponential explosion** issue when dealing with a large order, due to the exponential growth in the modulus of complex features.
In contrast, we decouple the learning processes of the modulus and phase of complex features. Specifically, we use complex rotations to model complicated interactions and incorporate a modulus amplification network to enhance representations.	
This approach effectively avoids the exponential explosion issue, allowing for the learning of **arbitrarily large order**.

[1] EulerNet: Adaptive Feature Interaction Learning via Euler’s Formula for CTR Prediction, SIGIR 2023.

Reviewer comment (Theory):
The Theorem is not properly defined, used and explained.

Author response:
**(1) There is no reason for $\Delta_{R}$ to be better than $\Delta_{RFM}$, why we approximate it?**

- We approximate $\Delta_{R}$ to demonstrate that our method (**rotation-based** interaction) can be instantiated as any traditional **inner-product-based** feature interaction model (e.g. FM [1], EulerNet [2], AFN [3]), since the interaction learning function of arbitrary traditional models [1][2][3] can be expressed using $\Delta_{R}$. Therefore, any traditional model can be viewed as a special case of our proposed method, demonstrating the effectiveness of our approach.

**(2) If $\Delta_{R}$
 is indeed better, why not directly use $\Delta_{R}$
 to capture feature interaction?**
 - Previous studies [1][2][3] have demonstrated the effectiveness of using $\Delta_{R}$ to model feature interactions. However, as mentioned earlier, **$\Delta_{R}$ cannot effectively capture arbitrary high-order feature interactions** (due to the exponential explosion issue). 
 In contrast, our approach ($\Delta_{RFM}$) not only has approximate results to $\Delta_{R}$ when the order is low, but also can scale to **arbitrarily high orders** of feature interactions, thereby illustrating the effectiveness and superiority of our proposed approach.

 **(3) Lemma A.1 holds when you assume the embeddings are random vectors. They however are not.**

- As a recent study [4] shows, the feature embedding space can be regarded as an unobservable random variable sampled from a certain distribution. 
Following the study [4], on the strength of Mean-field Theory [5], we suppose different fields are mutually independent.
Thus, under the constraint of the L2 norm, for ease of theoretical analysis, we approximate it by sampling within a unit hypersphere.
Because for an unknown machine learning task, we cannot specify a particular and accurate distribution; it may be distributed in any region of the unit sphere.

Reviewer comment (Novelty):
"To our knowledge, it is the first work that is capable of learning the interactions with arbitrarily large order adaptively from the corresponding interaction contexts." Isn't this also done by [1]?

Author response:
Although AFN [1] can adaptively learn the feature interaction orders to some extent, but:
- First, the order of feature interactions in AFN [1] cannot take arbitrarily large values, due to the exponential explosion issue, e.g., considering the interaction term $\boldsymbol e_j^{\alpha_j}$ with embedding $\boldsymbol e_j = [0.5, 2]$, its order $\alpha_j$ cannot be 100, since $2^{100}$ is exceptionally large, which makes them suffer from the gradient explosion issue.
We have also provided the detailed analysis in the Section 4.3 (**Arbitrary-Order Learning Analysis**).
Such approaches  cannot scale to high-order cases in industrial scenarios.
As our solution, we project features to the complex plane (polar angles) that converts the interactions into complex rotations.
Such transformation can effectively avoid the exponential explosion issue and allows for the learning of **arbitrary large order**. (The detailed proofs are also provided in Section 3.1.)

- Second, the orders learned in AFN [1] cannot capture the dependencies between different features, i.e., the interaction order of each feature is independently learned.
However, as prior work [2] shows, in real-world applications, the importance  of a certain feature (controlled by the order) is often influenced by other features.
It is challenging for AFN [1] to effectively capture the varied feature importance in different interaction contexts.
Unlike AFN, the interaction orders learned in our approach can effectively capture the dependencies between different features through the proposed self-attentive rotations, enabling it to learn the varied interaction orders from **the corresponding interaction contexts**.

[1] Adaptive Factorization Network: Learning Adaptive-Order Feature Interactions, AAAI 2020.

[2] Enhancing CTR Prediction with Context-Aware Feature
Representation Learning, SIGIR 2022.

Reviewer comment (Evaluation):
What is the training time for the proposed model? How does it compare to other models?

Author response:
We have discussed the inference efficiency of the model in Table 3 and analyzed the model complexity in Appendix E.
Our model's overall complexity aligns with that of the transformer. To gain a more comprehensive understanding of the complexity information of the model, we present the training times for the experiments in the table below.
Please note that, due to the one-epoch phenomenon [1], we provide the training time (in seconds) for one epoch to facilitate a fair comparison.

| Model | Criteo | Avazu | ML-1M | ML-Tag | Frappe |
|----------|----------|----------|----------|----------|----------|
|   LR  |   355.51  |   252.98  |   2.42  |   4.30  |   0.76  |
|   FwFM  |   375.87  |   301.37  |   2.57  |   4.71  |   1.03  |
|   FmFM  |   610.29  |  530.49  |   2.99  |   6.94  |   1.26  |
|   NFM  |   565.35  |   464.63  |   5.07 |   10.30  |   2.76  |
|   CIN  |   803.92  |   713.59  |   9.93  |   15.51  |   3.51  |
|   CrossNet  |   452.21  |   403.73  |   3.39  |   11.47  |   1.37  |
|   PNN  |   654.77  |   332.21  |  4.84  |   10.65  |   1.60  |
|   Transformer  |  745.99   |   563.52  |   8.08  |   21.19  |   3.74  |
|   DeepFM  |   445.21  |  384.64  |   4.50  |   10.49  |  1.63  |
|  xDeepFM  |  857.64  |  724.78  |  10.94  |  17.77  |  3.84  |
|  DCNV2  |  474.63  |  410.39  | 3.70  |  12.57  |  1.45  |
|  AFN+  |  930.74  |  745.45  |  10.03  |  16.73  |  21.72 |
|  ARM-Net+  |  1754.36  |  1140.30  |  14.89 |  30.69  |  27.94  |
|  EulerNet  |  597.96  |  510.22  |  5.29  |  11.09  |  1.82  |
|  RFM  |  783.43  |  681.27  |  6.11  |  17.59  |  1.92  |

[1] Towards Understanding the Overfitting Phenomenon of Deep Click-Through Rate Prediction Models, CIKM 2022.

Reviewer comment (Theory):
Can the proposed method be interpreted as using dual embedding vectors for one feature, with the constraint that they have unit norm in one dimension?

Author response:
**(4) Can the proposed method be interpreted as using dual embedding vectors for one feature, with the constraint that they have unit norm in one dimension?**

- Yes, this can be understood from a certain perspective. 
As we represent a feature using a vector located on the unit circle in the complex plane, this complex vector can be expressed using two real vectors, i.e., $\mathbf r + i \mathbf p$, s.t., $\mathbf r^2 +  \mathbf p^2 = \mathbf 1$.
Specifically, we use the arctangent function ($atan2$) of these dual embedding vectors to obtain the polar angle vectors in the complex plane, and model the feature interactions through the rotation transformations based on these polar vectors.

Reviewer comment (Novelty):
The model still is too similar to EulerNet, despite it replaces the linear block with an attention block. The claims regarding arbitrarily large order and exponential explosion issue are not supported. It could be redundant to separately learn modulus and phase. If the exponential explosion issue occurs, I cannot see how such a scheme could help. The modulus could still be too large.

Author response:
In Section 3.3, we highlight the novelty and differences between our approach and other feature interaction methods.	
Our proposed method and EulerNet [1] both model the feature interactions in the complex vector space, but there are still some noticeable differences:

- First, EulerNet [1] uses simple **linear layers** to model arbitrary-order  feature interactions, with interaction orders determined by predefined linear combination coefficients.
In contrast, our approach develops a **self-attentive rotation mechanism** to model feature interactions, with interaction orders learned from attention scores derived from the context.
This enables our approach to effectively capture feature dependencies in varying contexts.

- Second, EulerNet [1] simultaneously optimizes the modulus and phase of the complex feature interactions in a coupled way, leading to the the **exponential explosion** issue when dealing with a large order, due to the exponential growth in the modulus of complex features.
In contrast, we decouple the learning processes of the modulus and phase of complex features. Specifically, we use complex rotations to model complicated interactions and incorporate a modulus amplification network to enhance representations.	
This approach effectively avoids the exponential explosion issue, allowing for the learning of **arbitrarily large order**.

[1] EulerNet: Adaptive Feature Interaction Learning via Euler’s Formula for CTR Prediction, SIGIR 2023.

Reviewer comment (Theory):
The Theorem does not support the model well. A simple MLP would do the same as a universal approximator, without all these loose/inappropriate bounds, under which I think related work could be as powerful as the proposed method.

Author response:
**(1) There is no reason for $\Delta_{R}$ to be better than $\Delta_{RFM}$, why we approximate it?**

- We approximate $\Delta_{R}$ to demonstrate that our method (**rotation-based** interaction) can be instantiated as any traditional **inner-product-based** feature interaction model (e.g. FM [1], EulerNet [2], AFN [3]), since the interaction learning function of arbitrary traditional models [1][2][3] can be expressed using $\Delta_{R}$. Therefore, any traditional model can be viewed as a special case of our proposed method, demonstrating the effectiveness of our approach.

**(2) If $\Delta_{R}$
 is indeed better, why not directly use $\Delta_{R}$
 to capture feature interaction?**
 - Previous studies [1][2][3] have demonstrated the effectiveness of using $\Delta_{R}$ to model feature interactions. However, as mentioned earlier, **$\Delta_{R}$ cannot effectively capture arbitrary high-order feature interactions** (due to the exponential explosion issue). 
 In contrast, our approach ($\Delta_{RFM}$) not only has approximate results to $\Delta_{R}$ when the order is low, but also can scale to **arbitrarily high orders** of feature interactions, thereby illustrating the effectiveness and superiority of our proposed approach.

Reviewer comment (Novelty):
The motivation of the projecting features to the complex plane is not clear. Take AFN [2] as an example, it has similar capability to model arbitrary orders. What the diffidence between the existing solutions (e.g., Logarithmic Transformation) and yours should be discussed.

Author response:
In Section 3.1, we have discussed the main shortcomings of existing methods (e.g., AFN [4]) and the motivation behind our approach:

- First, the order of feature interactions in AFN [4] cannot take arbitrarily large values, due to the exponential explosion issue, e.g., considering the interaction term $\boldsymbol e_j^{\alpha_j}$ with embedding $\boldsymbol e_j = [0.5, 2]$, its order $\alpha_j$ cannot be 100, since $2^{100}$ is exceptionally large, which makes them suffer from the gradient explosion issue.
We have also provided the detailed analysis in the Section 4.3 (**Arbitrary-Order Learning Analysis**).
Such approaches  cannot scale to high-order cases in industrial scenarios.
As our solution, we project features to the complex plane (polar angles) that converts the interactions into complex rotations.
Such transformation can effectively avoid the exponential explosion issue and allows for the learning of arbitrary large order. (The detailed proofs are also provided in Section 3.1.)

- Second, the orders learned in field-aware and instance-aware methods cannot capture the dependencies between different features, i.e., the interaction order of each feature is independently learned.
However, as prior work [6] shows, in real-world applications, the importance  of a certain feature (controlled by the order) is often influenced by other features.
It is challenging for both
field-aware and instance-aware models to effectively capture the varied feature importance in different interaction contexts.
Unlike them, the interaction orders learned in our approach can effectively capture the dependencies between different features through the proposed self-attentive rotations.

Reviewer comment (Theory):
Is equation 3 a learning function for 1 feature interaction? Based on the equation and figure 2, it looks like they are an illustration on how to learn a specific feature interaction in the context of feature j. Because only the pairwise relationships between feature j and other features are modeled (i.e., x_j and x_1, x_j and x_l, and x_j and x_m), which are represented as a_{j,i}, a_{j,l}, and a_{j,m} respectively. How to model the relationship a_{1,m} between x_1 and x_m in this case? How to indicate the number of feature interactions? In my opinion, you might be referring to the Equ 7 in AFN [1], which is the output of the jth neuron. If yes, how to define the number of neurons (i.e., feature interactions)?

Author response:
- Answer (a): the equation (3) describes the interaction learning function with multiple interaction terms, and the set of order vectors is denoted as $\mathcal{A}$. For example, considering the interaction $\tilde{\boldsymbol e}_1 \odot \tilde{\boldsymbol e}_2 + \tilde{\boldsymbol e}_2 \odot \tilde{\boldsymbol e}_3 + \tilde{\boldsymbol e}_3 \odot \tilde{\boldsymbol e}_3$ with three fields ($m=3$), the set of order vectors is $\mathcal{A} = {[1,1,0], [0,1,1],[1,0,1]}$. (Specifically, $[1,1,0]$ for $\tilde{\boldsymbol e}_1 \odot \tilde{\boldsymbol e}_2$, $[0,1,1]$ for $\tilde{\boldsymbol e}_2 \odot \tilde{\boldsymbol e}_3$, and $[1,0,1]$ for $\tilde{\boldsymbol e}_1 \odot \tilde{\boldsymbol e}_3$) Therefore, each order vector corresponds to an interaction term, and the number of feature interaction terms is the number of order vectors.

- Answer (b): The feature interaction order $\alpha_{1,m}$ is modeled by the attention score between the query $\boldsymbol Q_1$ and the key $\boldsymbol K_{m}$. In our approach, we use self-attentive rotations to model relation-aware feature interactions. Specifically, we have $m$ queries ($\boldsymbol Q_1, \boldsymbol Q_2, ..., \boldsymbol Q_m$) and $m$ keys ($\boldsymbol K_1, \boldsymbol K_2, ..., \boldsymbol K_m$), and each query (e.g., $\boldsymbol Q_j$) will match all keys ($\boldsymbol K_1, \boldsymbol K_2, ..., \boldsymbol K_m$), generating $m$ interaction orders ($\alpha_{j,1}, \alpha_{j,2}, ..., \alpha_{j,m}$), constructing an interaction term $\tilde e_1^{\alpha_j,1} \odot \tilde e_2^{\alpha_j,2} \odot \cdots \odot\tilde e_m^{\alpha_j,m}$., measuring the relationships in the context of feature $j$. 
Therefore, the relationship $\alpha_{1,m}$ is modeled by the first query $\boldsymbol Q_1$ with the $m$-th Key $\boldsymbol K_{m}$.

- Answer (c): as mentioned in Answer (b), each query will construct a feature interaction term $\tilde{\boldsymbol e}_1^{\alpha_j,1} \odot \tilde{\boldsymbol e}_2^{\alpha_j,2} \odot \cdots \odot \tilde{\boldsymbol e}_m^{\alpha_j,m}$.
Therefore, the number of feature interactions is $m$ for a single head attention.
As introduced in Section 3.1.2 (**Multi-Head Rotation**), we use the head number $h$ to control the number of feature interaction terms (thus the total term number is $mh$).
All the output representations are concatenated as the input of the MLP, which enables it to filter out the useless interaction terms.

- Answer (d): As introduced in Section 3.1.2, a feature interaction term (the Equ 7 in AFN [1], output of the j-th neuron) is also described in Eq. (7) in our paper ($\mathcal{F}(\mathcal{A}_j):=\tilde{\boldsymbol e}_1^{\alpha_j,1} \odot \tilde{\boldsymbol e}_2^{\alpha_j,2} \odot \cdots \odot \tilde{\boldsymbol e}_m^{\alpha_j,m}$), where the order is generated by the attetnion score from $j$-th query.
Therefore, the number of feature interactions (neurons in AFN [1]) is the number of queries (i.e., $m$) for a single-head in our approach, and each query corresponds to a neuron.
For $h$ attention heads, the number of feature interactions is $mh$.

Reviewer comment (Evaluation):
Section 4 doesn't involve enough evidence that RFM is capturing feature interactions more efficiently, other than showing it has better prediction performance.

Author response:
**2. Section 4 doesn't involve enough evidence that RFM is capturing feature interactions more efficiently, other than showing it has better prediction performance.**

**We have discussed the inference efficiency of the model in Table 3 and analyzed the model complexity in Appendix E**.
Our model's overall complexity aligns with that of the transformer. To gain a more comprehensive understanding of the complexity information of the model, we present the training times for the experiments in the table below.
Please note that, due to the one-epoch phenomenon [1], we provide the training time (in seconds) for one epoch to facilitate a fair comparison.

| Model | Criteo | Avazu | ML-1M | ML-Tag | Frappe |
|----------|----------|----------|----------|----------|----------|
|   LR  |   355.51  |   252.98  |   2.42  |   4.30  |   0.76  |
|   FwFM  |   375.87  |   301.37  |   2.57  |   4.71  |   1.03  |
|   FmFM  |   610.29  |  530.49  |   2.99  |   6.94  |   1.26  |
|   NFM  |   565.35  |   464.63  |   5.07 |   10.30  |   2.76  |
|   CIN  |   803.92  |   713.59  |   9.93  |   15.51  |   3.51  |
|   CrossNet  |   452.21  |   403.73  |   3.39  |   11.47  |   1.37  |
|   PNN  |   654.77  |   332.21  |  4.84  |   10.65  |   1.60  |
|   Transformer  |  745.99   |   563.52  |   8.08  |   21.19  |   3.74  |
|   DeepFM  |   445.21  |  384.64  |   4.50  |   10.49  |  1.63  |
|  xDeepFM  |  857.64  |  724.78  |  10.94  |  17.77  |  3.84  |
|  DCNV2  |  474.63  |  410.39  | 3.70  |  12.57  |  1.45  |
|  AFN+  |  930.74  |  745.45  |  10.03  |  16.73  |  21.72 |
|  ARM-Net+  |  1754.36  |  1140.30  |  14.89 |  30.69  |  27.94  |
|  EulerNet  |  597.96  |  510.22  |  5.29  |  11.09  |  1.82  |
|  RFM  |  783.43  |  681.27  |  6.11  |  17.59  |  1.92  |

[1] Towards Understanding the Overfitting Phenomenon of Deep Click-Through Rate Prediction Models, CIKM 2022.

Reviewer comment (Experiments):
See my comments in the weakness section. I wonder if there could be more supporting discussions/materials that show how RFM's angular representations/rotation-based attention/modulus amplification are helping to better capture complex feature interactions. One thing that might help to convince the readers is to generate some synthetic data to show complex feature interactions are degenerated to simpler ones after these transformations.

Author response:
**3. One thing that might help to convince the readers is to generate some synthetic data to show complex feature interactions are degenerated to simpler ones after these transformations.**

Thanks for your careful reading of our paper. We have tried our best to elaborate the unclear points (High-order feature interaction learning analysis) and revised our paper accordingly. We have added the experiments using synthetic
data in Appendix I.
We can see that our approach can degenerated to simpler ones (traditional inner-product based interactions) after these transformations, which is consistent with our theoretical analysis.

Reviewer comment (Theory):
Question regarding the angular representations. Theoretically, the embedding layers could learn the exponential transformations. Say instead of x_j --> \theta_j (=E(x_j)) --> {\tilde e}_j (=e^{i\theta_j}, there could be a different embedding (\hat E) such that \hat E(x_j) = e^{iE(x_j)}. Why does explicitly adding this exponential transformation help? What's the hidden cost of it?

Author response:
**4. Question regarding the angular representations. Theoretically, the embedding layers could learn the exponential transformations. Say instead of x_j --> \theta_j (=E(x_j)) --> {\tilde e}_j (=e^{i\theta_j}, there could be a different embedding (\hat E) such that \hat E(x_j) = e^{iE(x_j)}. Why does explicitly adding this exponential transformation help? What's the hidden cost of it?**

 We greatly appreciate the reviewer for raising this question. The reason we use this form is primarily to explicitly emphasize that the feature embeddings we traditionally obtain correspond to the polar angles on the complex plane.
 Based on this concept, we can subsequently use a rotational mechanism to represent feature interactions.

 In other words, this is merely a conceptual shift rather than an actual transformation. 
 In the computer memory or practice implementation, we are still using the variable $E(\mathbf x_j)$ rather than $e^{iE(\mathbf x_j)}$, and the subsequent attention mechanism is designed for a series of rotation transformations based on $E(\mathbf x_j)$ (also $\mathbf \theta_j$).
 Based on this concept, after rotation, we use coordinate transformation (Eq. 12) to obtain the real and imaginary parts of the complex features, according to Euler's formula.

 In conclusion, it is merely an expression of an abstract concept rather than an actual transformation, hence there is no hidden cost.