openclimatefix

/

pvnet_uk_region

like 1

Follow

Open Climate Fix 36

Hi @kwon-encored , thanks for getting touch

Yea we pull the data from PVLive and then put it into a zarr file
https://github.com/openclimatefix/nowcasting_dataset/blob/main/scripts/generate_raw_data/get_raw_pv_gsp_data.py

However this repo is archived, so there's a chance some of the format is out of data. Ill keep looking for an update on this, and int he mean time, hopefully this helps you

@peterdudfield Thank you so much for your response

Do you think I can modify this code a little bit to convert from csv/json to zarr?
For the raw dataset, which link fit better for the list of GSP I can use for the aforementioned github link?
[1] https://api.solar.sheffield.ac.uk/pvlive/api/v4/gsp_list (from github repo's README.md)
[2] https://www.nationalgrideso.com/data-portal/gis-boundaries-gb-grid-supply-points/gsp_regions_20220314_geojson (from ICLR Paper)

I will keep an eye on this thread, thank you again for your time and consideration.

Hi @peterdudfield ,

Thank you for your advice.
Unfortunately, I'm still having trouble with the Zarr file issue, as I could not help myself with the algorithm you provided to convert the GSP into Zarr format.
Could you possibly share a Zarr file (or a part of it) that I could use to test and run your code?

Thank you again for your time and consideration!

What problems are you having? Ill try and get you an example of the data we use

Hi @peterdudfield ,
I downloaded the GSP file using the URL from the PVNet GitHub repository, but it only provided a JSON (or CSV) file.
After examining the code due to constant compile errors, I realized that it requires a .zarr file instead of a CSV or JSON.

Does this not create a zarr? https://github.com/openclimatefix/nowcasting_dataset/blob/main/scripts/generate_raw_data/get_raw_pv_gsp_data.py

@peterdudfield
Not really :( Unfortunately, the dataset that the code (from URL) required doesn't match the my GSP json format, so I wasn't able to generate a zarr as the script couldn't initialize correctly.

Hi @peterdudfield ,

I copied your code to generate the .zarr file, and it worked without any issues. However, when I run the model using this Zarr file, I encounter the following error:
KeyError: "Zarr object is missing the attribute _ARRAY_DIMENSIONS and the NCZarr metadata, which are required for xarray to determine variable dimensions.

It seems that the file is missing some attributes, even though I used your code directly.
Could you please provide some guidance on how to resolve this error?

Thank you very much!

Ok, great you copied the code, and made the zarr file.

Have you tried a minimum example, like xr.open_zarr("filename"), to see if it works?

(thank you for your assistance!!! GSP problem solved..now other compile errors :()