Paper: PowerGPT: Foundation Model for Power Systems (ICLR-2024)

Reviewer comment (Writing):
The authors talk about load and electricity consumption but they didn't describe the difference. Usually load/demand and consumption are used as synonyms (except some cases) and, if there is not the case, authors should describe clearly.

Author response:
***Q1: The difference between load and electricity consumption.***

The sampling frequency of "load data" is 15 minutes, and its physical meaning is the total power consumption of various electrical devices borne by the user at a specific moment. The sampling frequency of "electricity consumption" is daily, and its physical meaning is the total electricity consumption by the user throughout the day.

Reviewer comment (Writing):
This issue regarding "load" and "consumption" terminology is present over all paper and the authors should describe the difference and modify accordingly.

Author response:
***Q1: The difference between load and electricity consumption.***

The sampling frequency of "load data" is 15 minutes, and its physical meaning is the total power consumption of various electrical devices borne by the user at a specific moment. The sampling frequency of "electricity consumption" is daily, and its physical meaning is the total electricity consumption by the user throughout the day.

Reviewer comment (Writing):
The meaning of "masked and unmasked patches of different nodes can overlap at temporal axis" is unclear. What is the unmasked patches? What is \(\tilde{N}\)?

Author response:
***Q2: What is the meaning of "masked and unmasked patches of different nodes can overlap at temporal axis"? What is the unmasked patches? What is \tilde{N}?***

Apologies for our poor expression. After careful review, we found that the correct wording for that section should be "at hierarchical axis" than "temporal axis". 

It means that for the hierarchical mask, it masks all the instance nodes' patches at the same moment, as shown by the gray mask in Figure 2(a). On the other hand, for the temporal mask, it randomly masks a certain number of patches along the temporal axis, as depicted by the blue mask in Figure 2(a). Therefore, different instance nodes have both masked and unmasked patches at the same moment, resulting in an overlap of these masked and unmasked patches. 

"\tilde{N}" represents the number of nodes sampled in the subgraph centered around the target node. In the text, it is discussed under the section __Randomly Masking__.

Reviewer comment (Writing):
It is unclear what the "learnable mask token" is. Is it a particular token that substitutes the missing data? How is this token learned?

Author response:
***Q3: What is the "learnable mask token"? A particular token that subtistute the missing data? How this token is learned?***

The "learnable mask token" refers to using learnable parameters instead of setting the mask values to zero. It serves as a specific token that substitutes the missing data. This approach, as mentioned earlier, has been demonstrated and applied in [1,2]. By using learnable parameters for the mask, the model can better learn the representation of masked positions and improve the learning process.

Reviewer comment (Reproducibility):
Will the authors plan to release the trained models, the code, and the dataset?

Author response:
***Q4: Will the authors plan to release the trained models? The code? The dataset?***

Yes, we plan to release the code and the trained weights. And due to privacy, we couldn’t release the dataset.

Reviewer comment (Reproducibility):
Moreover, I do not think the problem setting is of wide interest, as I am confident that geographical electricity time series data, down to the level of specific users linked on a graph, in actual cities is not in general publicly available data (see possible ethical concerns).

Author response:
***Q1: About ethical concerns.***

All our data and tasks have received ethical approval, and all data related to personal information have undergone de-identification processing. The fine-grained information only consists of electricity consumption curves at the user level, with no personally identifiable information (PII) involved.

From our understanding, deep learning models can be used to detect user anomalous behavior in various domains, including financial fraud[1,2] and telephone scams[3]. Platforms like Twitter and Instagram also utilize different user behavior patterns to customize recommendations for their users[4,5].These models do not surveil users; they simply classify based on user behavior. 

Just like the aforementioned tasks, the tasks “electricity theft detection” and “elder living alone” detection also involve the use of electricity consumption sequences for "anomaly detection" purposes. These tasks have undergone national ethical review to ensure their compliance with ethical standards.

Reviewer comment (Experiments):
The model is only evaluated on the State Grid test datasets/tasks, which is a dataset introduced by the authors and which has not been vetted by peer review. It would be recommended to conduct experiments on one or more established benchmarks as well. I can recommend BuildingsBench [2], which is a recently published benchmark of 7 electricity datasets for evaluating pretraining-the-finetuning approaches for short-term (24 hour) building load forecasting.

Author response:
***Q2: It would be recommended to conduct experiments on one or more established benchmarks as well.***

Thanks for your advices, in the future we will onduct extensive experiments on other publicly available datasets.

Reviewer comment (Reproducibility):
We do see significant improvements in forecasting performance by PowerGPT. However, it does use more information, especially the hierarchical information in the forecasting. It is not clear from the paper how those information are used. Do we get the same history window from different hierarchies? For example, when predicting at the district level, do we get the past 256 history values at city, province, industry and user level?

Author response:
***Q1: For example, when predicting at the district level, do we get the past 256 history values at city, province, industry and user level?***

Yes,when predicting an instance of some level, we will get other level instances’ history values.

When predicting a specific instance, such as a district, we simultaneously obtain a subgraph centered around that node. The subgraph includes its high-level instance: city and province, as well as its low-level instance: users. Each node contains a historical sequence of 256 points for the same time period. As shown in Figure 2 (during downstream tasks, there is no masking process), R-GCN models the connectivity between different instances in the subgraph, with different edges having distinct labels and weights. Therefore, when predicting the future sequence based on the past 256 points of a particular node, PowerGPT takes into account the values of the past 256 points of the node's high and low level nodes.

Reviewer comment (Experiments):
Although the authors claim that their PowerGPT is a foundational model, there are no transfer experiments on other datasets. We don't know how well the model might perform on electricity consumption data from other countries, or even a different province in China (since the dataset only contain 1 province). This makes the claim of a foundational model weak.

Author response:
***Q2:there are no transfer experiments on other datasets.***

Sorry, due to certain limitations, we have not yet obtained electricity datasets for other provinces or cities. In our future work, we plan to incorporate electricity data from additional regions to conduct more comprehensive transfer experiments.

Reviewer comment (Experiments):
From the results in the tables I cannot see any forecasting experiments on individual user electricity load. These time series are usually much more variable and interesting. Are they indicated by one of the rows 'exclusive' or 'public' in the tables? There are no explanations for those.

Author response:
***Q3: Are individual user indicated by one of the rows 'exclusive' or 'public' in the tables?***

In the power system, residential users are divided into two categories: Exclusive Transformer (ET) users and Public Transformer (PT) users. ET users are primarily large consumers from industrial, agricultural, office, and commercial sectors. They exhibit distinct electricity consumption patterns, with higher electricity usage and load values. ET users have their own transformers, which are invested in and established by the users themselves. On the other hand, most PT users are residential customers, and their transformers are invested in and established by the power companies to provide public electricity supply.

Classifying and predicting residential users in the power system has significant economic benefits. Therefore, in the paper, we present the results by categorizing users into 'exclusive' or 'public' groups separately.

Reviewer comment (Experiments):
Why is a history window of size 256 used? For 15-min user level data that's not even 1 week of consumption data to capture the weekly patterns. Have the authors tried longer windows sizes like 512 or 1024?

Author response:
***Q4: Why is a history window of size 256 used?***

According to the data we received, the number of points in the load data for the same number of days is 96 times higher than the electricity consumption data (due to the sampling frequency of the load being 96 times higher). As a result, when increasing the training window length, the amount of training data for electricity consumption decreases. To maintain a relatively balanced proportion of training data, we set the window to 256.

However, this may not be an optimal choice. In the future, we plan to increase the window length while performing upsampling to ensure an adequate amount of training data for electricity consumption. Additionally, we will also explore the use of non-uniform training lengths to enhance the robustness of the model.

Reviewer comment (Theory):
ETS data is not as structural as language tokens, which actually reflects the inherent physical laws of power systems, as well as human behaviors. These can hardly be captured by foundation model and also be predicted by autoregressive methods.

Author response:
Although ETS data is not as structured as language tokens, it shares inherent patterns of variation with any other time series in different domains. Nowadays, there are various works[1-9] dedicated to modeling and predicting time series data in publicly available datasets such as ETT, WEATHER, ILLNESS, ELECTRICITY, and CHARGE, covering domains like finance, disease, energy and weather. __And these models all use transformer to model time series__. Especially after patching, the time token is simialr with the language token in NLP, which is the embedding including the neighbor imformation.

These works demonstrate the feasibility of time series modeling and highlight their effectiveness in various tasks such as time series forecasting, classification, and anomaly detection. They provide evidence of the potential and usefulness of time series analysis in these domains.

Reviewer comment (Reproducibility):
The completion level of the article is low, and the dataset is not open-source. There are multiple referencing errors.

Author response:
And we apologize for the poor writing. We will rewrite most of the content and carefully check the spelling. We are sorry for any inconvenience caused to your reading and review.

Reviewer comment (Experiments):
Can PowerGPT adapt to topology changes without re-training?

Author response:
PowerGPT can adapt to the changes in the topology of downstream tasks. In various downstream tasks, such as those influenced by population mobility or changes in regional divisions, the instances forming the topology can vary significantly. Graph Neural Networks (GNNs) excel at modeling such changing topologies, which does not affect the prediction of single node due to changes in the overall topology.