73 water vapor notebook by rogerkuou · Pull Request #74 · ESMValGroup/ClimaNet

rogerkuou · 2026-07-15T11:15:15Z

Fix #73 .

Should be merged after #96, since PR #96 contains improvement for this PR.

~~Should be merged after #92~~
~~Contains changes in #92 to fix gem embedding bug.~~
(#92 already merged on Sep 9)

Changes in code
Added check for empty dataset
Init loss in predict as a tensor

Changes in notebooks:
separated notebooks to sst and watervapor folder
added daily watervapor training notebook with local execution results
added hourly watervapor traning notebook with Levante execution results

rogerkuou · 2026-07-15T11:19:03Z

Hi @SarahAlidoost , while working on #73, I got an error in the notebook. This notebooks is adapted from the SST hourly one.

It fails on average_loss.item(), which hints that average_loss is a dictionany. Do you have clue on this? I am running with the CPU mode instead of GPU.

rogerkuou · 2026-07-16T13:55:53Z

Hi @SarahAlidoost , can you review this notwbook when you have time?

The error was caused by an empty dataset. I added a check to predict, and initiated loss as an zero Tensor.

Another thing maybe we need to pay attention, is the water wapor data is downsampled by the factor of 2 comparing to SST. This requires some attention to lsm data. In the notebook I used an average window to downsample the lsm, and set <0.5 to 0, and >=0.5 to 1. We can discuss if this makes sense.

SarahAlidoost

@rogerkuou Thanks! 👍 I added one comment on the changed code. Levante is not available today and I dont have data locally. I will check the notebook later. About the spatial resolution of water vapor, I expected era5 data have the same spatial resolution; this is something to check in our next update meeting. I couldnot also check the era5 data as CDS is also on maintenance.

SarahAlidoost · 2026-07-23T11:45:59Z

Hi @SarahAlidoost , can you review this notwbook when you have time?

The error was caused by an empty dataset. I added a check to predict, and initiated loss as an zero Tensor.

Another thing maybe we need to pay attention, is the water wapor data is downsampled by the factor of 2 comparing to SST. This requires some attention to lsm data. In the notebook I used an average window to downsample the lsm, and set <0.5 to 0, and >=0.5 to 1. We can discuss if this makes sense.

Indeed there are a few things about water vapor data to check:

the spatial resolution of data on levante is 0.5. The water vapor data is available on cds with resolution 0.25
data is cut between (-80, 80) in latitude dimension
the latitudes are not increasing they are from (80, -80) while longitude are i.e. (-180, 180)

rogerkuou · 2026-08-04T14:38:51Z

Hi @SarahAlidoost , can you review this notwbook when you have time?
The error was caused by an empty dataset. I added a check to predict, and initiated loss as an zero Tensor.
Another thing maybe we need to pay attention, is the water wapor data is downsampled by the factor of 2 comparing to SST. This requires some attention to lsm data. In the notebook I used an average window to downsample the lsm, and set <0.5 to 0, and >=0.5 to 1. We can discuss if this makes sense.

Indeed there are a few things about water vapor data to check:

the spatial resolution of data on levante is 0.5. The water vapor data is available on cds with resolution 0.25

data is cut between (-80, 80) in latitude dimension

the latitudes are not increasing they are from (80, -80) while longitude are i.e. (-180, 180)

During the meeting in July 24, we discussed the issues with Axel:

The resolution is coming from the fact the satellite observations on water vapor has resolution with 0.5 degress. Hence it does not add value to go to -.25 degree resolution
The data cut and lat coordinate order comes from how regredding is set. Axel is working on generating a new version of dataset with (-90, 90) and ascending ordering.

@SarahAlidoost shall we consider merging this PR for now? My motivation is that this PR also renames example notebooks to distinguish sst and water vapor. It can set good templates for other PRs.

rogerkuou · 2026-08-14T11:41:31Z

Updated on 20260814
Ihave confirmed with Axel that the satellite data's coverage is (-80, 80), and the resolution is 0.5 deg. Axel prefers not to create a new set of data, but use the current version. This means for the order of lat we can flip it by ourselves.

About the masking, I found CDS always provide 0.25 degrees. If using esemble mean to download, the downloaded mask are floating values. I check with xel he is fine with thresholding the floating mask with 0.5 values. Maybe let's do that and create another mask file matching the Water vapor data?

What do you think @SarahAlidoost and @meiertgrootes ?

rogerkuou · 2026-09-08T11:37:41Z

Hi @SarahAlidoost, in this PR I made example notebooks for watervapor. I executed the daily one locally and the hourly one on Levante.

I observed that the validation loss does not reduce, and the training stops with early stops. This is especially significant in the hourly training. I have tried using different month (I started with Jan, then switched to April. It the same as SST setting, three identical month from 2020-2022 for training, validation and testing), and reduced the learning rate to half (just in case if we are missing optimal). The validation loss curve stays flat. I hope for a full training with multiple month can help. But I will wait till your test with #92

(Updated on SEP 10)
Reagrding the small trial of validate/test only on training data, see this GH Gist. It should the traning loss continue to drop, and the prediction results are significantly improved comparing to baseline.
The "artificial oscillation" in the daily notebook is caused by an uncleared training history, where at same epoch there are multiple loss. To avoid conflicts, I solved this in #95. By removing caches and re-run we do not have this anymore. See the updated notebook.

I merged your fix in #92 to this PR, to include your fix for the bug. So it will make sense to review this after merging #92.

rogerkuou · 2026-09-10T09:52:09Z

Hi @SarahAlidoost , I brought this branch up to date. See my previous comment for details.

Meanwhile, following our discussion today, I will make another test on the training dataset to see if the training is able to overfit on the training data. This can validate if the problem is actually training on one month

SarahAlidoost · 2026-09-11T16:33:10Z

+<<<<<<< HEAD:notebooks/sst/training_hourly_sst.ipynb
+    "monthly_data_test = xr.open_mfdataset(data_folder / f\"202001_mon_ERA5dc_full_{var_name}.nc\")\n",
+=======
    "monthly_data_test = xr.open_mfdataset(data_folder / f\"202201_mon_ERA5dc_full_{var_name}.nc\")\n",
+>>>>>>> main:notebooks/example_hourly.ipynb


these are leftover from merging. Could you fix them?

SarahAlidoost · 2026-09-11T16:33:21Z

+<<<<<<< HEAD:notebooks/sst/training_hourly_sst.ipynb
+    "spatial_patch_size = monthly_da.shape[1:]  # the whole dataset \n",
+=======
    "spatial_crop_size = monthly_da.shape[1:]  # the whole dataset \n",
+>>>>>>> main:notebooks/example_hourly.ipynb


* implement lsm convertion func from sst to wv * add unit test; fix linting issue * update notebooks with lsm conversion func * linting * rerun daily notebook * Apply batched suggestions from code review Co-authored-by: SarahAlidoost <55081872+SarahAlidoost@users.noreply.github.com> * rename function * update variable name --------- Co-authored-by: SarahAlidoost <55081872+SarahAlidoost@users.noreply.github.com>

Co-authored-by: SarahAlidoost <55081872+SarahAlidoost@users.noreply.github.com>

rogerkuou · 2026-09-14T09:46:42Z

@SarahAlidoost thanks! I addressed your comments. shall we merge this PR now? Or do you want to wait till we perform a training with multiple months of watervapor?

SarahAlidoost

@rogerkuou thanks for addressing the comments. I'm reviewing the results. Meanwhile, there are several issues in the notebooks that should be fixed before merging. I couldn't comment directly on the notebooks, so I wrote them here:

In daily notebooks, under section "Prepare the test data", (in sst notebook see cell 20 and in watervapor notebook see cell 21), the variable spatial_patch_size should be replaced with spatial_crop_size.
In hourly watervapor notebook, the cell containing prediction_config = PredictionConfig(...) is duplicated.
There is an unused import add_month_day_dims in hourly notebooks.
For comparison purposes, we want same data in daily and hourly notebook. Please use month 01 in hourly notebook.
The variable name lsm_mask_sst is confusing as land-sea mask is not related to the climate variable like sst or water vapour. Please rename it.
Some training configs are changed in hourly notebook, like optimizer_lr. Could you fix them so notebooks are consistent.

SarahAlidoost · 2026-09-14T14:59:15Z

Meanwhile, following our discussion today, I will make another test on the training dataset to see if the training is able to overfit on the training data. This can validate if the problem is actually training on one month

@rogerkuou Hi, I wonder if you have done the test on the training dataset.

Oh, it seems that I missed the update in your comment.

rogerkuou · 2026-09-15T10:42:07Z

Hi @SarahAlidoost , thanks! I addressed your new comments:

In daily notebooks, under section "Prepare the test data", (in sst notebook see cell 20 and in watervapor notebook see cell 21), the variable spatial_patch_size should be replaced with spatial_crop_size. -> Updated the variable name
In hourly watervapor notebook, the cell containing prediction_config = PredictionConfig(...) is duplicated. -> removed duplicated cell
There is an unused import add_month_day_dims in hourly notebooks. -> removed unused import
For comparison purposes, we want same data in daily and hourly notebook. Please use month 01 in hourly notebook. changed to 01, need to be re-run, see item 7
The variable name lsm_mask_sst is confusing as land-sea mask is not related to the climate variable like sst or water vapour. Please rename it. -> renamed to lsm_mask_025 and added comments
Some training configs are changed in hourly notebook, like optimizer_lr. Could you fix them so notebooks are consistent. -> changed modified parameters back: patience=10, optimizer_lr=1e-3, verbose_epoch_interval=20
Rerun hourly notebook on Levante -> hourly notebook has been rerun

SarahAlidoost · 2026-09-15T12:56:07Z

(Updated on SEP 10) Reagrding the small trial of validate/test only on training data, see this GH Gist. It should the traning loss continue to drop, and the prediction results are significantly improved comparing to baseline.

The dropping loss and the comparison with the baseline are good signs that the model is learning from the data. But the train loss is still quite high (0.049759 after 100 epochs). My guess is that the training subset is small compared to the model size.

For the validation set, maybe using the full training dataset could help with the fitting issue. So for now, I’d say the results in the notebook are mainly for experimentation, and we probably shouldn’t draw any conclusions about the final error yet.

SarahAlidoost

@rogerkuou thanks for addressing the comments 👍 . Feel free to merge.

One small thing for future PRs: please run Ruff only on the files changed for the PR. For example, I don’t think we need to fix tuning.py as part of this PR.

rogerkuou · 2026-09-15T13:17:04Z

@rogerkuou thanks for addressing the comments 👍 . Feel free to merge.

One small thing for future PRs: please run Ruff only on the files changed for the PR. For example, I don’t think we need to fix tuning.py as part of this PR.

Thanks and noted! Do you know a practical way to do this? My formatting is done by pre-commit hooks. Maybe I will turn it off next time if I see irrelevant files. This time the formatting is introduced in the process of fixing conflicts.

Will merge this PR.

rogerkuou added 2 commits July 15, 2026 12:35

rename daily notebooks

67a043a

add draft example notebook for watervapor

5b3d0df

rogerkuou added 4 commits July 16, 2026 13:47

check empty dataset in prediction

5545f1f

update hourly watervapor notebook

0deab33

initiate loss as zero tensor

1dea21d

update notebook

3ac2df4

rogerkuou marked this pull request as ready for review July 16, 2026 13:52

SarahAlidoost reviewed Jul 20, 2026

View reviewed changes

Comment thread climanet/predict.py Outdated

changed loss to float

3e55c64

Merge branch 'main' into 73_water_vapor_notebook

c5a9337

SarahAlidoost added 15 commits August 26, 2026 13:01

rename patch with crop in dataset and predict module

9e289e3

fix geo_embeding in the model

9c523e5

remove spatial transfomer from the model

83a0a20

add some model improvemnets

63ba353

uncomment groupnorm in decoder

222f2e5

improve docstring

0ff075f

remove patch from dataset, adjust api in source modules

1fb8ba0

fix linters

09c55db

fix tests

814f151

fix docstring and comments

76842bd

fix nbs

e9cdd56

add model.train in train loop (a bug)

628507b

Merge branch 'main' into fix_geo_embeding

88382ce

update scripts

078121f

fix run_best_tuned_model script

b59ad45

add commented out code section for generating 0.5 deg lsm

f0409a7

rogerkuou marked this pull request as ready for review September 8, 2026 11:28

rogerkuou requested a review from SarahAlidoost September 8, 2026 11:37

solve confilcts

33795d1

This was referenced Sep 10, 2026

Add watervapor landmask creation to utils.py #95

Closed

95 watervapor lsm util func #96

Merged

SarahAlidoost reviewed Sep 11, 2026

View reviewed changes

Comment thread scripts/tuning.py Outdated

rogerkuou and others added 3 commits September 14, 2026 10:36

Apply suggestion from @SarahAlidoost

e0eefad

Co-authored-by: SarahAlidoost <55081872+SarahAlidoost@users.noreply.github.com>

solve conficts

8018eff

SarahAlidoost requested changes Sep 14, 2026

View reviewed changes

rogerkuou linked an issue Sep 15, 2026 that may be closed by this pull request

Add watervapor landmask creation to utils.py #95

Closed

rogerkuou added 6 commits September 15, 2026 09:53

change spatial_patch_size to spatial_crop_size

6b7dda2

remove duplicated cell

1150732

remove unuser import

5963654

use jan for hourly notebook

a06120b

update parameters in hourly notebook

34dc161

rerun hourly notenook

9feec31

SarahAlidoost approved these changes Sep 15, 2026

View reviewed changes

rogerkuou merged commit c62c724 into main Sep 15, 2026
6 checks passed

rogerkuou deleted the 73_water_vapor_notebook branch September 15, 2026 13:17

Conversation

rogerkuou commented Jul 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rogerkuou commented Jul 15, 2026

Uh oh!

rogerkuou commented Jul 16, 2026

Uh oh!

SarahAlidoost left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

SarahAlidoost commented Jul 23, 2026

Uh oh!

rogerkuou commented Aug 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rogerkuou commented Aug 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rogerkuou commented Sep 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rogerkuou commented Sep 10, 2026

Uh oh!

SarahAlidoost Sep 11, 2026

Choose a reason for hiding this comment

Uh oh!

SarahAlidoost Sep 11, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

rogerkuou commented Sep 14, 2026

Uh oh!

SarahAlidoost left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SarahAlidoost commented Sep 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rogerkuou commented Sep 15, 2026

Uh oh!

SarahAlidoost commented Sep 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

SarahAlidoost left a comment

Choose a reason for hiding this comment

Uh oh!

rogerkuou commented Sep 15, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

rogerkuou commented Jul 15, 2026 •

edited

Loading

rogerkuou commented Aug 4, 2026 •

edited

Loading

rogerkuou commented Aug 14, 2026 •

edited

Loading

rogerkuou commented Sep 8, 2026 •

edited

Loading

SarahAlidoost left a comment •

edited

Loading

SarahAlidoost commented Sep 14, 2026 •

edited

Loading

SarahAlidoost commented Sep 15, 2026 •

edited

Loading