Paper: TAFS: Task-aware Activation Function Search for Graph Neural Networks (ICLR-2024)

Reviewer comment (Experiments):
Graph classification is crucial in graph mining due to its emphasis on global topological structures. Given its distinct objectives from node classification, it's imperative for the author to include experiments on graph classification.

Author response:
Q1: *Graph classification is crucial in graph mining due to its emphasis on global topological structures. Given its distinct objectives from node classification, it's imperative for the author to include experiments on graph classification.*

**Answer**: We add below the performance of graph classification on the ogbg-molhiv dataset. We include baselines of classical GNN as well as Top-1 ranking solution PAS+FPS. In all cases, TAFS could achieve a performance gain.

**Graph level task** Dataset: ogbg-molhiv, #Graphs 41K, #Nodes per graph 25.5
| Method  | Activation | Test ROCAUC | Param  | Time     |
|---------|------------|-------------|--------|----------|
| GCN     | ReLU       | 0.7606      | 0.53M  | 40Min    |
|         | Tanh       | 0.7883      | 0.53M  | 40Min    |
|         | Swish      | 0.7577      | 0.53M  | 40Min    |
|         |  **TAFS**       | 0.7934      | +300   | 55Min    |
|         |            |             |        |          |
| GIN     | ReLU       | 0.7707      | 3.3M   | 53Min    |
|         | Tanh       | 0.778       | 3.3M   | 53Min    |
|         | Swish      | 0.7669      | 3.3M   | 53Min    |
|         |  **TAFS**       | 0.7883      | +300   | 67Min    |
|         |            |             |        |          |
| PAS+FPs | ReLU       | 0.8169      | 27M    | 17 Hours |
|         | Tanh       | 0.8201      | 27M    | 17 Hours |
|         | Swish      | 0.8152      | 27M    | 17 Hours |
|         |  **TAFS**       | 0.8203      | +0.01M | 23 Hours |

Reviewer comment (Experiments):
As a universal method, TAFS should be tested on as many GNN models as possible. GCN and GraphSAGE are quite similar, while GAT and GIN are different, which should also be used as backbone to evaluate the performance of TAFS.

Author response:
Q2. *As a universal method, TAFS should be tested on as many GNN models as possible. GCN and GraphSAGE are quite similar, while GAT and GIN are different, which should also be used as backbone to evaluate the performance of TAFS.*

**Answer**: We agree. Here are our detailed experiments on GAT/GIN across small and large datasets. As can be seen, TAFS achieves the best in most cases, showing the effectiveness of our method.

**GAT experiments**
|     |          |            | Cora  |      | DBLP  |      | Cornell |      | Texas |      | Chameleon |      |
|-----|----------|------------|-------|------|-------|------|---------|------|-------|------|-----------|------|
| GNN |          | Activation | mean  | std  | mean  | std  | mean    | std  | mean  | std  | mean      | std  |
| GAT | Stack    | ReLU       | 84.73 | 0.65 | 85.36 | 0.13 | 55.14   | 7.57 | 55.68 | 6.3  | 28.42     | 9.75 |
|     |          | Tanh       | 84.51 | 1.25 | 84.91 | 0.2  | 56.76   | 2.96 | 56.76 | 5.92 | 55.57     | 3.58 |
|     |          | Swish      | 84.44 | 1.4  | 85.34 | 0.28 | 56.76   | 2.42 | 60    | 8.09 | 22.81     | 1.04 |
|     |          | **TAFS**       | 86.14 | 1.68 | 85.91 | 0.13 | 56.59   | 3.15 | 58.92 | 7.91 | 55.97     | 3.16 |
|     | Residual | ReLU       | 83.96 | 0.6  | 85.16 | 0.09 | 55.14   | 6.07 | 54.59 | 6.92 | 61.18     | 4.19 |
|     |          | Tanh       | 85.62 | 0.18 | 85.25 | 0.3  | 54.59   | 5.24 | 54.05 | 2.42 | 64.65     | 2.25 |
|     |          | Swish      | 84.44 | 0.97 | 84.49 | 0.31 | 52.43   | 5.57 | 56.76 | 8.88 | 59.96     | 1.29 |
|     |          |  **TAFS**       | 86.47 | 1.25 | 85.99 | 0.22 | 55.11   | 7.33 | 56.81 | 8.44 | 65.66     | 3.78 |
|     | JKNet    | ReLU       | 86.43 | 0.25 | 84.77 | 0.23 | 78.92   | 8.61 | 79.46 | 5.01 | 57.68     | 0.46 |
|     |          | Tanh       | 86.73 | 0.92 | 85.07 | 0.13 | 75.14   | 5.24 | 78.38 | 2.96 | 58.25     | 1.01 |
|     |          | Swish      | 85.47 | 0.8  | 84.68 | 0.2  | 82.7    | 6.07 | 77.84 | 5.24 | 54.69     | 2.36 |
|     |          |  **TAFS**       | 87.21 | 0.47 | 85.22 | 0.18 | 82.84   | 6.26 | 79.38 | 4.52 | 64.82     | 2.01 |
|     | Mixhop   | ReLU       | 84.25 | 1.67 | 84.1  | 0.12 | 63.78   | 7.76 | 62.7  | 7.91 | 51.32     | 2.27 |
|     |          | Tanh       | 84.62 | 0.51 | 84.34 | 0.28 | 67.57   | 3.82 | 70.27 | 7.05 | 53.82     | 1.27 |
|     |          | Swish      | 84.4  | 0.38 | 84.39 | 0.35 | 64.86   | 3.82 | 65.95 | 6.07 | 50.26     | 2.93 |
|     |          |  **TAFS**       | 85.29 | 0.45 | 85.04 | 0.34 | 67.86   | 5.92 | 69.03 | 7.53 | 54.78     | 3.62 |

---
Dataset: ogbn-proteins
| Method    | Activation | Test ROCAUC | Param | Time    |
|-----------|------------|-------------|-------|---------|
| GAT       | ReLU       | 0.8176      | 2.48M | 33.6Min |
|           | Tanh       | 0.8003      | 2.48M | 33.6Min |
|           | Swish      | 0.8299      | 2.48M | 33.6Min |
|           |  **TAFS**       | 0.8682      | +300  | 41.5Min |

Reviewer comment (Evaluation):
The authors should conduct a time complexity analysis and compare it with GReLU.

Author response:
Q3. * The authors should conduct a time complexity analysis and compare it with GReLU.*

**Answer**: The setting of GReLU is actually very different from ours. GReLU designs a hyperfunction which includes additional graph convolutional layers, as a result, **GReLU is not a univariate function** as commonly required for activation functions (on the other hand, ReLU, Swish, Tanh and our TAFS as univariate functions). This can also be perceived by GReLU’s parameters. GReLU has N * C * K extra parameters, where N is number of nodes, C is the node feature dimensions and K is number of segments. So **GReLU is dependent on the graph dataset size**. TAFS is independent. Our TAFS brings extra parameter only dependent to number of graph layers, which is basically M * L, where M is the size of MLP (usually dozens) and L is the applied layers.

Reviewer comment (Experiments):
The search strategy presented in this paper does not support a larger GNN search space. For instance, if there is a need to search for aggregation functions or message passing functions, the optimal form of the activation function is likely to change.

Author response:
Actually, this is totally integratable. Our method TAFS could be applied after the other GNN search method finds the backbone model (for example, GraphNAS, 2019 Gao et al). We experiment this strategy on the OGB graph classification top 1 winner PAS+FPS method. It is a neural architecture search baseline that does exactly GNN space search. In the searched 14 layers GNN checkpoint, we replace its ReLU functions by TAFS, and fine tune further. As can be seen, within an affordable time, the performance could boost further, showing that TAFS is easily integrable with the GNN NAS methods. 

| Method  | Activation | Test ROCAUC | Param  | Time     |
|---------|------------|-------------|--------|----------|
| PAS+FPs | ReLU       | 0.8169      | 27M    | 17 Hours |
|         | TAFS       | 0.8203      | +0.01M | 23 Hours |

Reference:  https://ogb.stanford.edu/docs/leader_graphprop/#ogbg-molhiv

Reviewer comment (Experiments):
If TAFS employs an MLP to approximate the optimal activation function, the performance of activation function would also depend on the number of layers in the MLP and the non-linear transformation functions. This has not been discussed in this paper.

Author response:
**We highlight that MLP is a universal approximator which we choose on purpose as an expressive functional space. As a result, a normal MLP (not too large) would be sufficient in finding the suitable functions.**

We include the discussion on MLP hyperparameters in Figure 4(b)(c). It is true that this influences the performance. In order not to add too many extra parameters, we choose MLP with one-hidden layer (100 neurons) of Tanh activation function. No activation function is applied on the output neuron. We experiment below the differences of ReLU/Tanh used in MLP. Tanh is thus our final choice.

Dataset: DBLP

| Model        | Activation  | Accuracy |
|--------------|-------------|----------|
| GCN-Stack    | ReLU        | 83.06    |
|              | TAFS (relu) | 88.91    |
|              | TAFS (tanh) | 89.08    |
| GAT-residual | ReLU        | 83.96    |
|              | TAFS (relu) | 85.23    |
|              | TAFS (tanh) | 85.47    |

Reviewer comment (Experiments):
In experiments, although the paper introduces a Jacobian regularization term, the impact of this regularization has not been empirically tested.

Author response:
Thanks for pointing this out. We give below the influence of TAFS w./w.o. regularization. We will add this part to the paper later.

Dataset: DBLP
| Model        | Activation      | Accuracy |
|--------------|-----------------|----------|
| GCN-Stack    | ReLU            | 83.06    |
|              | TAFS (w.o. reg) | 87.51    |
|              | TAFS (w. Reg)   | 89.08    |
| GAT-residual | ReLU            | 83.96    |
|              | TAFS (w.o. reg) | 83.23    |
|              | TAFS (w. Reg)   | 85.47    |

Reviewer comment (Theory):
In TAFS, a stochastic relaxation is used, involving the reparameterization of the activation function parameters using a Gaussian distribution. However, this paper does not discuss the advantage of this strategy.

Author response:
The biggest advantage of the stochastic relaxation is to remove the differentiability requirement for the task evaluation metric (e.g. ROCAUC, Hit@20), which is very common in Graph tasks. By sampling the weights of activation functions from a probability distribution, we optimize the distribution parameters by proposition 1 in the paper. The proposition 1 gives the gradient of loss w.r.t distribution parameters, and there is no terms such as \nable M, avoiding the calculation of gradient of M. This makes the method more applicable in general tasks. The choice of Gaussian distribution is without loss of generality because we don’t have prior knowledge about this choice of distribution. In cases where we know certain probability distribution is better, we could definitely change to others such Poisson distribution.

We give below the influence of TAFS w./w.o. stochastic regularization. The full table has been updated in Table 3 in the paper.
| Dataset | Model   | Activation                    | ROCAUC | PRAUC |
|---------|---------|-------------------------------|--------|-------|
| DTI     | SkipGNN | ReLU                          | 0.922  | 0.928 |
|         |         |        TAFS w.o. relaxation   | 0.933  | 0.934 |
|         |         | TAFS                          | 0.952  | 0.954 |
|         | HOGCN   | ReLU                          | 0.927  | 0.929 |
|         |         |        TAFS w.o. relaxation   | 0.923  | 0.922 |
|         |         | TAFS                          | 0.943  | 0.94  |

Reviewer comment (Theory):
In a previous work [1], the best-performing activation function is referred to as Swish, which has a specific functional form. Similarly, can the best-performing activation function identified by TAFS be represented using a generic function?  
[1] Prajit Ramachandran, Barret Zoph, and Quoc V. Le. Searching for activation functions. In International Conference on Learning Representations, ICLR, 2018.

Author response:
This is an interesting question. We answer in two ways. Firstly, we could achieve the closed form equation by symbolic regression techniques. Because we have obtained the univariate function and we could simulate many (x, y) value pairs to symbolic regression to distill. Swish could do so because it uses a template based space which naturally leads to this symbolic form as a result. However, this has a big negative impact on the possible function space, i.e., it is very possible the suitable activation function is not in this manually-designed template space. Secondly, in terms of usage, we do not need to distill the explicit form because we open source the learned weights of the univariate function and for practitioners, they just need to load this small function and replace their ReLU. The usage is thus very easy.

We show below a case study of the symbolized version of TAFS searched results. On the Cora dataset with GCN-JK network baseline, the searched result is the blue line. We distill an explicit formula by symbolic regression: y = 0.6 Tanh(-x) and plug in this activation function back to the model. The performance is shown below and we provide a **visualization in the appendix Sec H**. There is a performance gap in the table below. This is because explicit symbolic space is not accurate, which shows further that our implicit functional space is expressive.

| Dataset | Baseline | Activation   | Accuracy |
|---------|----------|--------------|----------|
| Cora    | GCN-JK   | TAFS         | 89.08    |
|         |          | 0.6*Tanh(-x) | 87.89    |

Reviewer comment (Writing):
The sentence in the main text, "… how can we design GNN activation functions to adapt effectively to various graph-based tasks, creating task-aware activation functions??" seems to have an issue with the use of symbols.

Author response:
Thanks for your careful review. We have corrected these typos.

Reviewer comment (Writing):
In the text, "... and demote by ¯w_δ all the parameters of GNN, …", the word "demote" appears to be misspelled.

Author response:
Thanks for your careful review. We have corrected these typos.

Reviewer comment (Evaluation):
The experimental section lacks clear explanations of the metrics used in the tables. For example, the description of Table 2 does not specify whether the results represent accuracy, AUC, or something else.

Author response:
In Table 2, the standard accuracy is used. In Table 3, both ROCAUC and PRAUC are used. In our extended OGB experiments, we use  ROCAUC for node classification and graph classification, Hit@20 for link prediction, following the OGB standard.Thus, we have experimented on various tasks of different objective requirements.

Reviewer comment (Presentation):
In Figure 4, subfigure (a) shows that larger values of K yield better results but at a slower pace, demonstrating a "trade-off between accuracy and computation time." Is the Y-axis in (a) representing accuracy? And it's not clear how the relationship between K value and computation time is depicted.

Author response:
Yes, Y-axis is the downstream performance, here classification accuracy. For example, moving from K=32 towards K=64 increases 20% more computational times. As a result, it is truly a trade-off that should depend on the user's own preference. We will add this discussion in the paper.

Reviewer comment (Experiments):
Some of the experimental settings and results explanations in the paper are vague, such as Figure 1 in the Introduction section and Figure 3 in the Experiments section. I cannot understand how the experimental results in the figures were obtained, and what the data in the figures mean.

Author response:
Q1. *Some of the experimental settings and results explanations in the paper are vague, such as Figure 1 in the Introduction section and Figure 3 in the Experiments section. I cannot understand how the experimental results in the figures were obtained, and what the data in the figures mean.*

**Answer**: In a word, Figure 1 illustrates the impact of activation function in GNN and motivates our work to search activation functions. Figure 3 demonstrates the searched candidates from baselines and our proposed TAFS.

**Figure 1** shows the experiments on two GNN baselines (GCN and GraphSage) on two typical graph datasets (Core and DBLP), with different choices of activation functions (ReLU, Tanh, Leaky-Relu, Swish). The evaluation metric is node classification accuracy. We show by Figure 1 that activation functions in GNN, which was previously paid little attention, could have a huge impact on the performance. This motivates our study to design efficient activation function search algorithms in this paper. 

**Figure 3** demonstrates multiple things. Fig 3(a) gives an idea on the search space of our compared baselines Swish and APL to show what functions in their search space looks like. Fig 3(b)(c) shows our searched activation functions tailoring two datasets Cora and DBLP. Specifically, we show that (1) our method finds adaptive activation functions for different datasets (2) due to the efficiency of our method, we could adapt layer-wise activation functions without exploding the search space. So different layers find different activation functions (3) our found activation functions are distinct to already published manually-designed ones (4) our activation functions are smooth functions which potentially reflects our smoothness regularization.

Reviewer comment (Experiments):
I wonder why in the drug and protein interaction prediction experiments, only the results of TAFS and ReLU were provided, instead of comparing with multiple activation functions as in the node classification experiments.

Author response:
Q3. *I wonder why in the drug and protein interaction prediction experiments, only the results of TAFS and ReLU were provided, instead of comparing with multiple activation functions as in the node classification experiments.*

**Answer**: We add the other choices’ performance below. We didn’t include this to emphasize the impact by adopting TAFS.  The table will be merged to the main paper later.

| Dataset | Model   | Activation | ROCAUC | PRAUC |
|---------|---------|------------|--------|-------|
| DTI     | SkipGNN | ReLU       | 0.922  | 0.928 |
|         |         | Tanh       | 0.915  | 0.911 |
|         |         | Swish      | 0.931  | 0.935 |
|         |         | **TAFS**       | 0.952  | 0.954 |
|         | HOGCN   | ReLU       | 0.927  | 0.929 |
|         |         | Tanh       | 0.903  | 0.911 |
|         |         | Swish      | 0.925  | 0.921 |
|         |         | **TAFS**       | 0.943  | 0.94  |
|         |         |            |        |       |
| DDI     | SkipGNN | ReLU       | 0.886  | 0.866 |
|         |         | Tanh       | 0.903  | 0.872 |
|         |         | Swish      | 0.881  | 0.865 |
|         |         | **TAFS**       | 0.911  | 0.898 |
|         | HOGCN   | ReLU       | 0.898  | 0.881 |
|         |         | Tanh       | 0.887  | 0.876 |
|         |         | Swish      | 0.881  | 0.872 |
|         |         | **TAFS**       | 0.917  | 0.901 |
|         |         |            |        |       |
| PPI     | SkipGNN | ReLU       | 0.917  | 0.921 |
|         |         | Tanh       | 0.912  | 0.919 |
|         |         | Swish      | 0.92   | 0.921 |
|         |         | **TAFS**       | 0.927  | 0.937 |
|         | HOGCN   | ReLU       | 0.919  | 0.922 |
|         |         | Tanh       | 0.917  | 0.913 |
|         |         | Swish      | 0.915  | 0.914 |
|         |         | **TAFS**       | 0.923  | 0.929 |
|         |         |            |        |       |
| DGA     | SkipGNN | ReLU       | 0.912  | 0.915 |
|         |         | Tanh       | 0.912  | 0.914 |
|         |         | Swish      | 0.924  | 0.917 |
|         |         | **TAFS**       | 0.93   | 0.94  |
|         | HOGCN   | ReLU       | 0.927  | 0.934 |
|         |         | Tanh       | 0.918  | 0.929 |
|         |         | Swish      | 0.923  | 0.924 |
|         |         | **TAFS**       | 0.933  | 0.942 |

Reviewer comment (Experiments):
Since the authors compared the search efficiency with Swish and APL, why didn’t they involve the comparison with these two search-based methods in the effectiveness validation experiments (Table 2 and Table 3)?

Author response:
Q4. *Since the authors compared the search efficiency with Swish and APL, why didn’t they involve the comparison with these two search-based methods in the effectiveness validation experiments (Table 2 and Table 3)?*

**Answer**: Due to the inefficiency of Swish and APL, we could not practically obtain reasonable results by Searchable Swish and APL. For example, in the case of Table 4 in the paper, we can see that searchable Swish takes 50 Hours to run on Chameleon and APL costs 6 times more memory consumption than TAFS. In our table 2, we have included 5 datasets and 10 graph baselines, leading to a total of 50 sets of experiments. We find it not necessary to include the searchable Swish and APL in every single set of task. As a result, we included already in Table 2 and Table 3 the searched results of Swish paper (Swish), which is worse than TAFS. We also give below the APL results and searchable swish on two tasks for a general idea of their ineffectiveness. Note that both baselines could not be applied in OGB datasets due to inefficiency in time and memory, while TAFS could.

Dataset: DBLP

| Model        | Activation   | Accuracy |
|--------------|--------------|----------|
| GCN-Stack    | ReLU         | 83.06    |
|              | Swish search | 82.79    |
|              | APL search   | 81.03    |
|              | **TAFS**         | **89.08**    |
| GAT-residual | ReLU         | 83.96    |
|              | Swish search | 83.01    |
|              | APL search   | 81.22    |
|              | **TAFS**         | **85.47**    |

Reviewer comment (Experiments):
The experimental results are not very convincing. The experiments are all conducted on very small datasets, and I don't think the datasets for link prediction experiments are the commonly used ones in literature. I'd recommend the authors to use more standardized and commonly accepted benchmarks such as the OGB ones.

Author response:
We answer Q1 and Q2 together. The inappropriate comparison in the manuscript was due to the fact that the baseline was too small and as a result the computational head seems to be 10x more, though our method is already much more efficient than literature. Actually, **our method’s parameters have no dependency on the size of the graph**, making it easily scalable. We experimented on **OGB and much larger baselines** as in the table below. Three OGB datasets in node/link/graph level tasks are chosen and we experiment both classical GNN baselines and top-ranking baselines. As can be seen, on larger datasets (ogbn-molhiv) and larger baselines (PAS) which take over 17 hours, our method only adds about 35% more computational time, which is much more acceptable than baselines. Actually, both **Swish search and APL search could not be run on such larger datasets due to their inefficiency** in running time and overparameterization, which leads to months’ GPU hours or leads to Out-of-Memory error. 

---
**Node level task** Dataset: ogbn-proteins, Node 132,534, Edge 39,561,252

| Method    | Activation | Test ROCAUC | Param | Time       |
|-----------|------------|-------------|-------|------------|
| GCN       | ReLU       | 0.7219      | 100K  | 15.3Min    |
|           | Tanh       | 0.6521      | 100K  | 15.3Min    |
|           | Swish      | 0.7255      | 100K  | 15.3Min    |
|           | TAFS       | 0.7483      | +300  | 16.8Min    |
|           |            |             |       |            |
| GraphSage | ReLU       | 0.7567      | 193K  | 17.7Min    |
|           | Tanh       | 0.7442      | 193K  | 17.7Min    |
|           | Swish      | 0.7319      | 193K  | 17.7Min    |
|           | TAFS       | 0.7768      | +300  | 19.9Min    |
|           |            |             |       |            |
| GAT       | ReLU       | 0.8176      | 2.48M | 33.6Min    |
|           | Tanh       | 0.8003      | 2.48M | 33.6Min    |
|           | Swish      | 0.8299      | 2.48M | 33.6Min    |
|           | TAFS       | 0.8682      | +300  | 41.5Min    |
|           |            |             |       |            |
| GIPA      | ReLU       | 0.8917      | 17M   | 10.7 hours |
|           | Tanh       | 0.8809      | 17M   | 10.7 hours |
|           | Swish      | 0.8899      | 17M   | 10.7 hours |
|           | TAFS       | 0.8991      | +1500 | 14.9 hours |

---
**Link level task** dataset: ogbl-ddi, Node 4,267, Edge 1,334,889

| Method    | Activation | Test Hits@20 | Param | Time      |
|-----------|------------|--------------|-------|-----------|
| GCN       | ReLU       | 0.3707       | 1.29M | 7Min      |
|           | Tanh       | 0.3318       | 1.29M | 7Min      |
|           | Swish      | 0.3772       | 1.29M | 7Min      |
|           | TAFS       | 0.3991       | +300  | 10Min     |
|           |            |              |       |           |
| GraphSage | ReLU       | 0.5391       | 1.42M | 8Min      |
|           | Tanh       | 0.5228       | 1.42M | 8Min      |
|           | Swish      | 0.5377       | 1.42M | 8Min      |
|           | TAFS       | 0.5512       | +300  | 13Min     |
|           |            |              |       |           |
| E2N       | ReLU       | 0.9606       | 0.6M  | 4 Hours   |
|           | Tanh       | 0.9554       | 0.6M  | 4 Hours   |
|           | Swish      | 0.9515       | 0.6M  | 4 Hours   |
|           | TAFS       | 0.9681       | +300  | 4.7 Hours |

Reviewer comment (Evaluation):
Although the proposed method is already much more efficient than the baselines Swish and APL, it still takes about 10x times of runtime when compared with fixed activation functions. It's hard to tell whether the performance improvements worth such huge overhead on the time cost.

Author response:
We answer Q1 and Q2 together. The inappropriate comparison in the manuscript was due to the fact that the baseline was too small and as a result the computational head seems to be 10x more, though our method is already much more efficient than literature. Actually, **our method’s parameters have no dependency on the size of the graph**, making it easily scalable. We experimented on **OGB and much larger baselines** as in the table below. Three OGB datasets in node/link/graph level tasks are chosen and we experiment both classical GNN baselines and top-ranking baselines. As can be seen, on larger datasets (ogbn-molhiv) and larger baselines (PAS) which take over 17 hours, our method only adds about 35% more computational time, which is much more acceptable than baselines. Actually, both **Swish search and APL search could not be run on such larger datasets due to their inefficiency** in running time and overparameterization, which leads to months’ GPU hours or leads to Out-of-Memory error. 

---
**Node level task** Dataset: ogbn-proteins, Node 132,534, Edge 39,561,252

| Method    | Activation | Test ROCAUC | Param | Time       |
|-----------|------------|-------------|-------|------------|
| GCN       | ReLU       | 0.7219      | 100K  | 15.3Min    |
|           | Tanh       | 0.6521      | 100K  | 15.3Min    |
|           | Swish      | 0.7255      | 100K  | 15.3Min    |
|           | TAFS       | 0.7483      | +300  | 16.8Min    |
|           |            |             |       |            |
| GraphSage | ReLU       | 0.7567      | 193K  | 17.7Min    |
|           | Tanh       | 0.7442      | 193K  | 17.7Min    |
|           | Swish      | 0.7319      | 193K  | 17.7Min    |
|           | TAFS       | 0.7768      | +300  | 19.9Min    |
|           |            |             |       |            |
| GAT       | ReLU       | 0.8176      | 2.48M | 33.6Min    |
|           | Tanh       | 0.8003      | 2.48M | 33.6Min    |
|           | Swish      | 0.8299      | 2.48M | 33.6Min    |
|           | TAFS       | 0.8682      | +300  | 41.5Min    |
|           |            |             |       |            |
| GIPA      | ReLU       | 0.8917      | 17M   | 10.7 hours |
|           | Tanh       | 0.8809      | 17M   | 10.7 hours |
|           | Swish      | 0.8899      | 17M   | 10.7 hours |
|           | TAFS       | 0.8991      | +1500 | 14.9 hours |

---
**Link level task** dataset: ogbl-ddi, Node 4,267, Edge 1,334,889

| Method    | Activation | Test Hits@20 | Param | Time      |
|-----------|------------|--------------|-------|-----------|
| GCN       | ReLU       | 0.3707       | 1.29M | 7Min      |
|           | Tanh       | 0.3318       | 1.29M | 7Min      |
|           | Swish      | 0.3772       | 1.29M | 7Min      |
|           | TAFS       | 0.3991       | +300  | 10Min     |
|           |            |              |       |           |
| GraphSage | ReLU       | 0.5391       | 1.42M | 8Min      |
|           | Tanh       | 0.5228       | 1.42M | 8Min      |
|           | Swish      | 0.5377       | 1.42M | 8Min      |
|           | TAFS       | 0.5512       | +300  | 13Min     |
|           |            |              |       |           |
| E2N       | ReLU       | 0.9606       | 0.6M  | 4 Hours   |
|           | Tanh       | 0.9554       | 0.6M  | 4 Hours   |
|           | Swish      | 0.9515       | 0.6M  | 4 Hours   |
|           | TAFS       | 0.9681       | +300  | 4.7 Hours |

Reviewer comment (Novelty):
I'd appreciate if the authors can elaborate more on the difference of the proposed method versus the activation function search methods for CNNs and RNNs (as referenced in Sec. 1).

Author response:
The two most important activation function search methods are Swish and APL, which is explained and compared in Sec 2 and especially in Table 1. We give a detailed introduction in Appendix B. Swish is inefficient in running time because every inner optimization will train the network until convergence. APL is inefficient in parameters and memory because it parameterized each neuron’s activation by multiple hinges. In the case of ogbg-molhiv datasets (above) and 1st ranking solution PAS on graph classification, the total parameter is 27M and the running time is over 17 hours. APL will add over 150M parameters which leads to OOM and Swish will cost over 70 days even if we train only 100 epochs. Our method TAFS, takes only 23 hours and adds only 0.01M parameters, which is significantly more efficient than baselines, and as a result, paves the way for further study in activation function search. 

| Dataset     | Model   | Param  | Time     |
|-------------|---------|--------|----------|
| ogbg-molhiv | PAS+FPS | 27M    | 17Hours  |
|             | Swish   | NA     | >70days  |
|             | APL     | OOM (>150M)   | NA       |
|             | TAFS    | +0.01M | 23 Hours |