Paper: Video Caching at Data-drifting Network Edge: A KD-based Cross-domain Collaborative Solution (ICLR-2024)

Reviewer comment (Experiments):
The methods compared in the articles are all before 2020, so they are not timely and persuasive. It’s better to compare it with the latest methods, such as Edge Caching or KD related.

Author response:
##We also investigated the latest related methods, and found that they are not suitable for migrating to the edge cache scenario we set. Moreover, many methods are not open source, and forced migration will bring us a lot of work and is unfair.

Reviewer comment (Experiments):
Many of the hyper-parameters are given directly, without explanation and associated hyper-parameter analysis experiments.

Author response:
##Due to space limitations, our detailed analysis and results on hyperparameters are presented in the appendix (E.2).

Reviewer comment (Writing):
The readability of the article is not good, and the framework of the overall model may be better explained in the main content.

Author response:
##Your suggestion has been taken and our statement of the problem has been refined In subsection 2.2.

Reviewer comment (Evaluation):
For data analysis, the author proposed temporal drift and spatial drift, and designed different algorithms for them respectively. However, the experimental analysis lacks a stronger explanation for temporal and spatial, or the lack of relevant results indicates whether each part is right.

Author response:
##Due to space limitation, both our ablation study and real-world trace evaluation are presented in the appendix( E.1,E.3,F), and these supplementary experiments are intended to demonstrate the effectiveness of each module of our work.

Reviewer comment (Reproducibility):
How exactly is the data collected and will it be used as a public data set?

Author response:
##The data is collected by Kuaishou Company, but we are not clear about how it is collected. They only provide us with the latitude and longitude of the request, the time of the request and the video ID.

Reviewer comment (Theory):
In the system model, the state representation includes the number of requests over intervals for C cached videos and the top-C videos selected based on their feature modulus from the uncached pool. This approach might miss out on caching videos that are about to become popular, especially if their current feature modulus isn't among the top-C. An explanation on the rationale for equating the number of uncached videos to the cached ones in the state would be beneficial. Additionally, it would be insightful to provide an explanation on the decision to rely solely on the number of requests as a state.

Author response:
##We calculate the feature modulus of all videos in each feature extraction, which can largely avoid the problem of missing popular content. In the formulation of the Problem, we consider that $C$ contents in the cache should be replaced each time, and in the most extreme case, $C$ new contents should be filled. Then, we carry out slot-wise transformation to decouple the stored content from the storage location, so that EBS with different capacities can be adapted. We acknowledge that relying only on the request number to extract features is a simplification, but the features extracted from the number of requests are straightforward and effective.

Reviewer comment (Evaluation):
The paper could benefit from a complexity analysis of the proposed framework or comparison with other algorithms. Understanding its computational and memory requirements in various scenarios would be essential for practical deployment in real-world systems.

Author response:
##The experiments on the traffic cost of different baselines can reflect the cost of data exchange, which can reflect their complexity to a certain extent (for example, although the Baseline only relying on content-level cooperation can achieve a relatively close cache hit rate to our proposed algorithm, However, it faces a traffic cost much higher than our algorithm), the hardware used for training and the training time have been supplementary explained in the architectures and training parameters in section 4.

Reviewer comment (Writing):
In eq. (1), the terms r^i_t and r^j_t are defined as the cumulative numbers of requests to the content in i-th cache slot and j-th candidate video sequence, respectively. It might be clearer to use distinct notations for each, to avoid potential confusion between the cache slot and the candidate video sequence.

Author response:
##We adopted your suggestion and changed eq.(1) to a clearer form.

Reviewer comment (Theory):
The DRL formulation is not capturing some key factors of streaming systems dynamics including different video file-size distribution and EBS heterogeneities.

Author response:
Many similar works do not consider the file size, or treat it as the same size. Our focus is on how to enhance EBS collaboration on a strategy-level, so the size of the video is simplified, and our slot-wise design can be adjusted to almost all different EBS storage space.

Reviewer comment (Theory):
DRL state representation seems to only look at one aspect - the file request in a given past window and does not account for other factors such as its retrieval time (latency), size, popularity, etc. Size and retrieval time are both important factors and will play a role in transition from one state to another. How the proposed model will change accordingly?

Author response:
The number of requests in the past time windows is the core metric to evaluate popularity, as the input to the neural network. We follow the assumptions made in previous work and do not regard retrieval time (latency) and file size as the focus of this work.

Reviewer comment (Theory):
Reward seems to be designed to only capture the local reward and no global term. How to ensure consensus in terms of convergence? No convergence analysis is provided.

Author response:
The goal of all EBS is to improve their own cache hit rate, and the global goal is also consistent with this. Because in our framework, there is no content-level cooperation between EBS, so each EBS does not need to consider what content is cached by other EBS, and only needs to search for caching strategies to refer from beneficial partners.

Reviewer comment (Evaluation):
How are the three defined temporal metrics related to video file arrival rate? Why not adopting arrival rate instead since it is widely used and correlates/encapsulates at least two of these three metrics?

Author response:
The second metric is the average latency calculated with CHR, but it gives a more intuitive and significant representation of the latency experienced by the user. The communication overhead of the third metric is negatively correlated with CHR, and positively correlated with the number of information exchange between among EBSs and CDN when using different algorithms, which can comprehensively reflect the performance of the algorithm.

Reviewer comment (Writing):
The problem statement or definition of collaborative edge caching in this paper lacks clarity and support.

Author response:
##Your suggestion has been taken and our statement of the problem has been refined In subsection 2.2.

Reviewer comment (Theory):
The paper assumes that the EBSs collaborate by sharing caching policies rather than cached content. This assumption gives rise to two problems. Firstly, considering that each EBS has limited cache storage, if they don't share cached content among themselves, every cache miss would necessitate content retrieval from remote CDN servers. Consequently, the potential benefits of collaborative edge caching are constrained since it doesn't significantly reduce backbone traffic. Secondly, the paper also highlights the spatial drift and diversity of request patterns in different EBS domains. This raises a question about the rationale behind advocating the sharing of caching policies among EBSs. For example, if one EBS serves requests with a LRU pattern, while another EBS caters to a LFU pattern, how would these two EBSs benefit from sharing caching policies when their request patterns differ?

Author response:
##The increase of cache hit rate can reduce the traffic of the backbone network, and the focus of this paper is to optimize the strategy-level cooperation. As for the situation you mentioned that the request patterns of different regions are different, in the framework we designed, each EBS can adjust the appropriate cooperative partner according to the gain and loss of referring to the caching strategy of EBS in different regions.

Reviewer comment (Experiments):
This paper studies a caching problem with a fixed number of video contents, which significantly deviates from real caching systems where the number of videos or objects varies over time. Furthermore, the paper assumes the CDN server has cached all the videos. However, considering the substantial number of videos (i.e., 2.887 million) mentioned in the evaluation section, this assumption might be overly stringent, as CDNs typically have relatively limited storage capacity.

Author response:
##As we introduced in Section 3.1, we transform the problem into a slot-wise approach that can adapt to different cache sizes. According to our measurements, videos and their average requests show a long-tail distribution, and about 80% of video content has less than or equal to one request per day. Through simple filtering and screening, popular content can be stored in CDN to meet the requests of the vast majority of users, so as to effectively alleviate the storage pressure.

Reviewer comment (Presentation):
The presentation of the proposed framework in the paper lacks clarity, and certain claims about the techniques are not well substantiated.

Author response:
##Your suggestion has been taken and our statement of the problem has been refined In subsection 2.2.

Reviewer comment (Reproducibility):
The paper claims that the RL agent is slot-wise and can adapt to EBSs with different storage sizes. However, it's noted that the action space of the RL agent is C × C, where C seems a fixed number across all EBSs throughout the paper (including the evaluation section). If C does indeed differ among EBSs, then during knowledge distillation, how do you handle the variable input state size, which is 2C × F?

Author response:
##During the formulation of the Problem, we consider replacing C contents in the cache each action, so the initial dimension of the state space is $C \times F$. After we perform the slot-wise transformation, when cache misses, the features of the content in each slot are input into the popularity network for scoring, that is, the dimension of the network input vector is $1 \times F$, and then the content with the lowest score is selected for replacement.

Reviewer comment (Reproducibility):
How does the reference weight agent adapt to dynamic partner lists? If two EBSs are no longer partners, it seems the corresponding action network parameter in the reference weight agent should be zero. Then, how do you enforce this when training the reference weight agent?

Author response:
##The parameter of actor network of reference weight agent is the reference weight of EBS to each member in its partner list. The EBS not in partner list will not be referenced, and its reference weight will not be included in the actor network. The input dimension of actor network will be adjusted accordingly in this time slice.

Reviewer comment (Reproducibility):
It is unclear how the proposed framework updates the reference partner list.

Author response:
##The content of the adjustment of the partner list is introduced in the last paragraph of Section 3.3, and we have made additional annotations for your review.

Reviewer comment (Experiments):
The paper's evaluation results are not sufficiently robust or convincing. What is the reason to choose a cache size of 32? The caching size should be related to the total active video content number. Besides, the proposed framework should be evaluated on various caching size settings.

Author response:
##Experiments with different baseline performance versus cache size were moved to the appendix (E.2) due to space limitations.

Reviewer comment (Novelty):
The related work section misses literature about multi-agent RL and multi-agent RL with knowledge distillation. Examples include:  
Leonardos, Stefanos, et al. "Addressing Out-Of-Distribution Joint Actions in Offline Multi-Agent RL via Alternating Stationary Distribution Correction Estimation." Advances in Neural Information Processing Systems 36 (NeurIPS 2023). 2023.  
Tseng, Wei-Cheng, et al. "Offline Multi-Agent Reinforcement Learning with Knowledge Distillation." Advances in Neural Information Processing Systems 35 (2022): 226-237.  
Gao, Zijian, et al. "KnowSR: Knowledge Sharing among Homogeneous Agents in Multi-agent Reinforcement Learning." arXiv preprint arXiv:2105.11611 (2021).

Author response:
##New related work has been added to section 5.