Accessing S2 L2A data

Atmospheric Corrections Processor
1

Hello!

I am writing a script that creates cloud-free mosaics from Sen4CAP-derived S2 data.
Is there any way to fully access the downloaded and MAJA processed S2 L2A data?
I have found some data in \archive\maccs_def\sitename\l2a
However, for most dates there are some tiles missing - and it is not due to cloud cover, as the demmaccs reports state that cloud cover is well below the threshold. Is this a bug or is there another way to access the imagery?

Final lines from the demmaccs log:

2020-04-04 08:14:49.262733:[2595]:MACCS/MAJA did not processed any tiles for L1C product /mnt/archive/dwn_def/s2/default/latvija/S2A_MSIL1C_20200330T094031_N0209_R036_T35VLD_20200330T105655.SAFE
2020-04-04 08:14:49.282199:[2595]:Remove all the temporary files and directory
2020-04-04 08:14:49.885388:[2595]:Couldn’t remove the temp dir /mnt/archive/demmaccs_tmp/tmpSrgJCn
2020-04-04 08:14:49.887855:[2595]:Total execution 0:24:39.694268:

In the demmaccs_tilename log, there are some warnings which are not present for the correctly processed tiles or the ones that are excluded due to cloud cover:

ScientificProcessing: ITK Exception: The file doesn’t exist.
Filename = /mnt/archive/demmaccs_tmp/tmpMPV6gO/35VLD/S2_TEST_AUX_REFDE2_35VLD_20200330T094031_0001.DBL.DIR/S2_TEST_AUX_REFDE2_35VLD_0001_MSK.TIF
[otbImageFileReader.txx:612] [vnsMajaMainProcessor.cxx:main:130]
[vnsMajaMainProcessor.cxx:main:130]

ScientificProcessing: ITK Exception: The file doesn’t exist.
Filename = /mnt/archive/demmaccs_tmp/tmpv_s1Bf/35VLD/S2_TEST_AUX_REFDE2_35VLD_20200330T094031_0001.DBL.DIR/S2_TEST_AUX_REFDE2_35VLD_0001_ALT_R1.TIF
[otbImageFileReader.txx:612] [vnsMajaMainProcessor.cxx:main:130]
[vnsMajaMainProcessor.cxx:main:130]

ScientificProcessing: ITK Exception: The file doesn’t exist.
Filename = /mnt/archive/demmaccs_tmp/tmpSrgJCn/35VLD/SENTINEL2B_20200325-094435-516_L2A_T35VLD_C_V1-0/DATA/SENTINEL2B_20200325-094435-516_L2A_T35VLD_C_V1-0_PVD_ALL/RTA.tif
[otbImageFileReader.txx:612] [vnsMajaMainProcessor.cxx:main:130]
[vnsMajaMainProcessor.cxx:main:130]

Does the omission of these files mean that they are not used in the creation of higher level products as well?

EDIT: I can confirm that L1C data exist for all tiles, so the data acquisition functionality works as intended.

Thank you for your time!

Harijs

2

Hello,

The reasons why you don’t have L2A produced with MAJA can be several:

  • Wrong version of MAJA. The system provides the gipp for MAJA 3.2.2. If you installed MAJA 3.3.2 or above, then the gipp are not compatible and no L2A products will be created.
  • Execution rights for MAJA → If you installed MAJA before the system, you should check that the directory /opt/maja should have the read rights for all files (please check the SUM) for the command:

sudo chmod -R a+rx /opt/maja

  • Insufficient disk space?
  • Do you the gipp_maja installed in /mnt/archive? Please check that you have the directory /mnt/archive/gipp_maja. If you did not copied it in the installation package before installing the system, this might be a problem.
  • For a product that gave errors, you can go in the corresponding folder from /mnt/archive/maccs_def/sitename/l2a and open for example, the file demmaccs_35VLD.log. Please check after the command:

/usr/share/sen2agri/sen2agri-demmaccs/demmaccs.py --srtm /mnt/archive/srtm --swbd /mnt/archive/swbd --processes-number-dem 1 --processes-number-maccs 1 --gipp-dir /mnt/archive/gipp_maja --working-dir /mnt/archive/demmaccs_tmp/ --maccs-launcher /opt/maja/3.2.2/bin/maja --delete-temp True … (other parameters here)

  • Copy this command and perform the following operations:
  1. Stop the demmaccs services using:

sudo systemctl stop sen2agri-demmaccs.timer sen2agri-demmaccs

  1. Switch to user sen2agri-service

sudo su -l sen2agri-service

  1. Execute the above demmaccs.py command:

/usr/share/sen2agri/sen2agri-demmaccs/demmaccs.py --srtm /mnt/archive/srtm … (other parameters here)

  1. The log messages are more detailed here. Check the actual error you get. If you encountered a product with more than 90% clouds, you should check for another product this is why you should check before if your product is a cloud free product.
  2. If you finished investigating the logs, type exit to logout from the sen2agri-service user:

[sen2agri-service@aaa ~]$ exit

  1. After you corrected the problem, don’t forget to cleanup the failed products and restart the demmaccs services:

sudo -u postgres psql sen4cap -c “update downloader_history set status_id = 2 where status_id = 6”
sudo -u postgres psql sen4cap -c “delete from l1_tile_history”
sudo systemctl start sen2agri-demmaccs sen2agri-demmaccs.timer

Hope this helps.

Best regards,
Cosmin

P.S. I am not sure why you are trying to create a cloud free composite using the Sen4CAP system as Sen4CAP does not have such a processor. You can find a cloud free composite in the Sen2Agri system (the L3A processor) so maybe you are interested in that system. Sen4CAP system is based on the Sen2Agri system but does not imports its processors (except the L3B processor)

3

Hello, Cosmin!

Thank you very much for your reply and time. As I said in my original post, MAJA is mostly working correctly, but it skips some individual tiles from each acquisition - and not because of cloud cover. As a result I get L2A images with some missing data. In the detailed log files, the cloud cover is under 90%.

Here is an example from a detailed log:

No valid cam data found for product, using constant model
Cloud percentage computed is 48; the maximum cloud percentage threshold authorized is 90.
****************************************************************************************************** [vnsMajaMainProcessor.cxx:main:130]
ScientificProcessing: ITK Exception: The file doesn’t exist.
Filename = /mnt/archive/demmaccs_tmp/tmpMPV6gO/35VLD/S2_TEST_AUX_REFDE2_35VLD_20200330T094031_0001.DBL.DIR/S2_TEST_AUX_REFDE2_35VLD_0001_MSK.TIF
[otbImageFileReader.txx:612] [vnsMajaMainProcessor.cxx:main:130]
[vnsMajaMainProcessor.cxx:main:130]
****************************************************************************************************** [vnsMajaMainProcessor.cxx:main:130]

2020-04-03 07:27:08.371265:[4204]:dmworker_1: Starting the process for tile 35VLD
2020-04-03 07:27:08.387840:[4204]:dmworker_1: Starting demmaccs

As you can see, the tile processing is restarted.
A similar arror occurs twice more in the log (see my original post).

In this case, the cloudless composite was just my project to write a script that uses MAJA data to mask clouds from all images and then composites them together. But during this project I found that there are L2A data missing, so I became worried that this data is not used in creation of Level 4 products as well.

I have also uploaded an example log for a tile that is failing but cloud cover is under 90.

Thank you for your time, Cosmin! The system is great, and we just want to make sure it is running as expected.

demmaccs_35VLD.log (45.0 KB)

4

Hello Harijs,

Sometimes input products might not be valid and MAJA is not able to produce the L2A or it will produce a product but does not have the Validity Flags OK. In such cases, we do not insert in the database those product that do not have validity flags OK and we remove them.
In the log files we do not keep all the MAJA messages as, from our experience, sometimes they can get very large and might be not efficient to keep all of them (and occupy a lot of disk space).
This is why, if you are interested in more details on an error, you can follow the procedure I described above to get all the messages from MAJA and to see the actual reason why a particular product was not processed or why the validity flag was not OK. In your case, with the above procedure, you can try executing:

/usr/share/sen2agri/sen2agri-demmaccs/demmaccs.py --srtm /mnt/archive/srtm --swbd /mnt/archive/swbd --processes-number-dem 1 --processes-number-maccs 1 --gipp-dir /mnt/archive/gipp_maja --working-dir /mnt/archive/demmaccs_tmp/ --maccs-launcher /opt/maja/3.2.2/bin/maja --delete-temp True /mnt/archive/dwn_def/s2/default/latvija/S2A_MSIL1C_20200330T094031_N0209_R036_T35VLD_20200330T105655.SAFE /mnt/archive/maccs_def/latvija/l2a/S2A_MSIL2A_20200330T094031_N0209_R036_T35VLD_20200330T105655.SAFE/ --tiles-to-process 35VLD --prev-l2a-tiles 35VLD --prev-l2a-products-paths /mnt/archive/maccs_def/latvija/l2a/S2B_MSIL2A_20200325T094029_N0209_R036_T35VLD_20200325T122804.SAFE/

Best regards,
Cosmin

1 Like
5

Thank you for your reply, Cosmin!

We ran the command you provided, it indeed yielded a bit more detailed log. We, however, are still not sure what causes the tiles to be invalid. And this seems to be a recurring issue. Could you please take a look at the new log and help us figure out why a considerable amount of usable S2 data is left out?

Huge thanks in advance!

demmaccs_35VLD.log (49.9 KB)

6

Hello,
Could you please provide the full detailed log after executing manually the demmaccs command (and not only the log file found in the product directory)?

Best regards,
Cosmin