Error during S2 Preprocessing - ERROR - Creating sym links for NOMINAL MACCS/MAJA mod

MCa · June 23, 2023, 6:19am

Hello,

We are having issues during S2 preprocessing with MAJA using SEN4CAP 3.2.

Currently, we observe that:
• the MAJA processor effectively runs in our system but the preprocessing job still finishes systematically with error
• And S2 data are downloaded and available in the DB

The l2a job logs reports 2 issues:

Maja script still tries to create a symlink on “mnt/archive/maccs_def” folder without success.
For your information, we have the archive directory mounted.
No valid products JPI L2NOTV found
issue: Symlink on “archive/maccs_def”

Maja script still tries to create a symlink on “archive/maccs_def” folder without success.

2023.06.22-11:40:55 - ERROR - Creating sym links for NOMINAL MACCS/MAJA mode: l2a prev tiles [‘/mnt/archive/maccs_def/lux_2022_test3/l2a/2022/01/12/S2B_MSIL2A_20220112T104319_N0301_R008_T32ULA_20220112T113720.SAFE/SENTINEL2B_20220112-104659-911_L2A_T32ULA_C_V1-0’,
‘/mnt/archive/maccs_def/lux_2022_test3/l2a/2022/01/12/S2B_MSIL2A_20220112T104319_N0301_R008_T32ULA_20220112T113720.SAFE/mosaic.jpg’]
*2023.06.22-11:40:55 - ERROR - Created sym links for NOMINAL MACCS/MAJA mode for [‘/mnt/archive/maccs_def/lux_2022_test3/l2a/2022/01/12/S2B_MSIL2A_20220112T104319_N0301_R008_T32ULA_20220112T113720.SAFE/SENTINEL2B_20220112-104659-911_L2A_T32ULA_C_V1-0’, *
‘/mnt/archive/maccs_def/lux_2022_test3/l2a/2022/01/12/S2B_MSIL2A_20220112T104319_N0301_R008_T32ULA_20220112T113720.SAFE/mosaic.jpg’]

We did face the same issue but with the working directory “/archive/demmaccs_tmp/” and we finally “solved “ it by changing, via the L2A Atmospheric Corrections panel, the path to the demmaccs_tmp folder from a mounted location to local folder.

But now, the Sen4CAP “L2A Atmospheric Corrections interface” seems not to take into account the changes done for the default path L2A products “mnt/archive/maccs_def” ( the path is a priori configurable but the changes seems not to be considered after saving).

Question:
• Is there another way to change the path of this folder?
• Are other paths which require to be accessed via symlinks?

2.Issue: No valid products JPI L2NOTV found

This second issues appears systematically after the error reported above. So it could be they are correlated.
Nevertheless, in case it’s not, I take the opportunity to report it.

Below you can find an extract of the log of the l2a_1282 job where the issue can be found.

2023.06.22-12:01:22 - INFO - Checking MAJA JPI file /leo4cap/tmp/demmaccs_tmp/tmp_1283/maccs_32ULA/SENTINEL2B_20220201-104658-260_L2A_T32ULA_C_V1-0/DATA/SENTINEL2B_20220201-104658-260_L2A_T32ULA_C_V1-0_JPI_ALL.xml …
2023.06.22-12:01:22 - ERROR - L2NOTV found in the Validity_Flag Processing_Flags_And_Modes element from MAJA JPI file /leo4cap/tmp/demmaccs_tmp/tmp_1283/maccs_32ULA/SENTINEL2B_20220201-104658-260_L2A_T32ULA_C_V1-0/DATA/SENTINEL2B_20220201-104658-260_L2A_T32ULA_C_V1-0_JPI_ALL.xml. The product will be invalidated …
2023.06.22-12:01:22 - ERROR - No valid products (JPI L2NOTV status) found in: /leo4cap/tmp/demmaccs_tmp/tmp_1283/maccs_32ULA.
2023.06.22-12:01:22 - INFO - Erasing the MACCS/MAJA working directory: rmtree: /leo4cap/tmp/demmaccs_tmp/tmp_1283/maccs_32ULA
2023.06.22-12:01:22 - ERROR - MACCS/MAJA did not process any tiles for L1C product /leo4cap/tmp/demmaccs_tmp/archives/tmpIeA9kY/S2B_MSIL1C_20220201T104149_N0400_R008_T32ULA_20220201T113513.SAFE
2023.06.22-12:01:22 - INFO - Remove all the temporary files and directory
2023.06.22-12:01:22 - INFO - Total execution 0:22:08.977361:

Questions:
• is this issue linked to the symlink issue reported above?
• Otherwise, if there any hint of what the root cause could be?

Thank you very much in advance for your support.

Best regards,

Marta

cudroiu · June 23, 2023, 8:44am

Dear Marta,

Could you please provide the configuration values returned after this command:

psql -U admin sen4cap -c “select * from config where key like ‘%l2a%’;”

After you changed the value for /mnt/archive/maccs_def/… to something else, is it updated in the database for the key “processor.l2a.optical.output-path” (you can find this key also in the results of the above mentioned command)?
You can try updating manually the key in the database with a new value also using:

psql -U admin sen4cap -c "update config set value = ‘/<new_path>/maccs_def/{site}/{processor}/’ where key = ‘processor.l2a.optical.output-path’ "

Please note that <new_path> must be a path writable by the user “sen2agri-service”. Also, you should keep the tokens /{site}/{processor} at the end of the path.

Also, before doing this, you might consider perform the following operations, as the demmaccs might have already a queue with products into memory, with the old paths (probably, this might be a reason why you did not noticed something changed in execution after you updated the value):

sudo systemctl stop sen2agri-demmaccs.timer sen2agri-demmaccs

Optionally reset failed products :

sudo psql -U admin sen4cap -c " delete from l1_tile_history where downloader_history_id in (select id from downloader_history where status_id = 6 and satellite_id = 1) "
sudo psql -U admin sen4cap -c " update downloader_history set status_id = 2 where status_id = 6 and satellite_id = 1 "

Note: in the above 2 sql commands you can also add a filter by site adding :

… and satellite_id = 1 and site_id = <your_site_id> …

After you change the value for the key “processor.l2a.optical.output-path” as described above, you can do now:

sudo systemctl restart sen2agri-demmaccs.timer

Hope this helps.

Best regards,
Cosmin

P.S. Please update the ’ and the " when copy/paste the commands into the command line as the forum might have change them.

MCa · June 26, 2023, 9:32am

Hello Cosmin,

Thank you very much for your fast reply.

Please find below the configuration values returned after thecommand

psql -U admin sen4cap -c “select * from config where key like ‘%l2a%’;”

Effectively following your instructions we could change the maccs_def folder.

We did not need to reset failed products as there was still no processing done.

We will retrigger and let you know.

Best regards,

Marta

MCa · June 26, 2023, 10:04am

Hello Cosmin,

We did the changes you proposed but unfortunately the S2 processors seems not to process anymore.
Here an extract from the log from the demaccs service.

How could we make the pre-processing start reprocessing again?

Thank you in advance for your support.

Best rgards,

Marta

cudroiu · June 27, 2023, 2:03pm

Dear Marta,

From what I see, the demmaccs is running.
Can you check also the following file?
You can check the N last messages with a command like:

sudo tail -n 1000 -f /var/log/sen2agri/l2a_launcher.log

This file contains the second level of messages after the journalctl.
Further, you might find messages in some files like (full path indicated in the l2a_launcher.log):

/mnt/archive/maccs_def/<site_name>/l2a/output/<L2A_PRD>/l2a_<some_number>.log

Best regards,
Cosmin

ajames · June 28, 2023, 8:34am

Hi Cosmin,

Thank you for your answer. I’m James, Marta’s teammate.
Hereunder are details you asked for:

l2a_launcher.log:

2023.06.28-04:15:00 - INFO - <worker 0> Current worker stopped
2023.06.28-04:16:00 - INFO - <master> No more tiles to process
2023.06.28-04:16:00 - INFO - <master> Stoping workers
2023.06.28-04:16:00 - INFO - <worker 0> Current worker stopped
2023.06.28-04:17:00 - INFO - <master> No more tiles to process
2023.06.28-04:17:00 - INFO - <master> Stoping workers
2023.06.28-04:17:00 - INFO - <worker 0> Current worker stopped
2023.06.28-04:18:01 - INFO - <master> No more tiles to process
2023.06.28-04:18:01 - INFO - <master> Stoping workers
2023.06.28-04:18:01 - INFO - <worker 0> Current worker stopped
2023.06.28-04:19:01 - INFO - <master> No more tiles to process
2023.06.28-04:19:01 - INFO - <master> Stoping workers
2023.06.28-04:19:01 - INFO - <worker 0> Current worker stopped
2023.06.28-04:20:00 - INFO - <master> No more tiles to process
2023.06.28-04:20:00 - INFO - <master> Stoping workers
2023.06.28-04:20:00 - INFO - <worker 0> Current worker stopped
2023.06.28-04:21:00 - INFO - <master> No more tiles to process
2023.06.28-04:21:00 - INFO - <master> Stoping workers
2023.06.28-04:21:00 - INFO - <worker 0> Current worker stopped
2023.06.28-04:22:01 - INFO - <master> No more tiles to process
2023.06.28-04:22:01 - INFO - <master> Stoping workers
2023.06.28-04:22:01 - INFO - <worker 0> Current worker stopped
2023.06.28-04:23:01 - INFO - <master> No more tiles to process
2023.06.28-04:23:01 - INFO - <master> Stoping workers

Logs are constantly looping on that same pattern.

I also join the updated list of parameters for L2A processing:

 id  |                 key                  | site_id |                               value                                |         last_updated
-----+--------------------------------------+---------+--------------------------------------------------------------------+-------------------------------
   2 | archiver.max_age.l2a                 |         | 5                                                                  | 2015-07-20 13:31:33.655476+00
 110 | downloader.use.esa.l2a               |         | false                                                              | 2019-12-16 12:56:57.501918+00
 112 | executor.processor.l2a.name          |         | L2A                                                                | 2015-06-03 14:02:50.028002+00
 113 | executor.processor.l2a.path          |         | /bin/false                                                         | 2015-07-20 13:31:23.208369+00
 114 | processor.l2a.maja.gipp-path         |         | /mnt/archive/gipp/maja                                             | 2016-02-24 16:12:16.464479+00
 115 | processor.l2a.maja.remove-fre        |         | 0                                                                  | 2017-10-24 12:56:57.501918+00
 116 | processor.l2a.maja.remove-sre        |         | 1                                                                  | 2017-10-24 12:56:57.501918+00
 117 | processor.l2a.optical.cog-tiffs      |         | 0                                                                  | 2017-10-24 12:56:57.501918+00
 118 | processor.l2a.optical.compress-tiffs |         | 0                                                                  | 2017-10-24 12:56:57.501918+00
 119 | processor.l2a.optical.max-retries    |         | 3                                                                  | 2020-09-15 13:02:27.164968+00
 122 | processor.l2a.optical.retry-interval |         | 1 day                                                              | 2020-09-07 11:36:37.906825+00
 123 | processor.l2a.s2.implementation      |         | maja                                                               | 2020-09-07 11:17:52.846794+00
 124 | processor.l2a.sen2cor.gipp-path      |         | /mnt/archive/gipp/sen2cor                                          | 2020-09-15 13:48:05.415193+00
 125 | processor.l2a.srtm-path              |         | /mnt/archive/srtm                                                  | 2016-02-25 09:11:36.372405+00
 126 | processor.l2a.swbd-path              |         | /mnt/archive/swbd                                                  | 2016-02-25 09:12:04.008319+00
 128 | processor.l2a.processors_image       |         | sen4x/l2a-processors:0.2.3                                         | 2021-04-19 16:30:00+00
 129 | processor.l2a.sen2cor_image          |         | sen4x/sen2cor:2.10.01-ubuntu-20.04                                 | 2021-04-19 16:30:00+00
 130 | processor.l2a.maja_image             |         | sen4x/maja:4.5.4-centos-7                                          | 2021-04-19 16:30:00+00
 131 | processor.l2a.gdal_image             |         | osgeo/gdal:ubuntu-full-3.4.1                                       | 2021-04-19 16:30:00+00
 132 | processor.l2a.l8_align_image         |         | sen4x/l2a-l8-alignment:0.1.2                                       | 2021-04-19 16:30:00+00
 133 | processor.l2a.dem_image              |         | sen4x/l2a-dem:0.1.3                                                | 2021-04-19 16:30:00+00
 127 | processor.l2a.working-dir            |         | /leo4cap/tmp/demmaccs_tmp/                                         | 2016-02-25 15:31:06.01191+00
 121 | processor.l2a.optical.output-path    |         | /leo4cap/l2a_products/maccs_def/{site}/{processor}/                | 2016-02-24 16:09:17.379905+00
 120 | processor.l2a.optical.num-workers    |         | 1                                                                  | 2020-09-07 11:36:37.906825+00
 235 | general.scratch-path.l2a_msk         |         | /mnt/archive/orchestrator_temp/l2a_msk/{job_id}/{task_id}-{module} | 2021-05-18 14:54:17.288095+00
 237 | processor.l2a_msk.compress           |         | true                                                               | 2021-05-18 14:54:17.288095+00
 238 | processor.l2a_msk.cog                |         | false                                                              | 2021-05-18 14:54:17.288095+00
 239 | executor.processor.l2a_msk.slurm_qos |         | qosvaliditymsk                                                     | 2015-08-24 14:44:38.29255+00
 317 | processor.s4c_l4a.input_l2a          |         | N/A                                                                | 2023-03-04 09:09:58.820032+00
 398 | processor.s4c_mdb1.l2ab02_enabled    |         | false                                                              | 2021-05-16 15:31:06.01191+00
 399 | processor.s4c_mdb1.l2ab03_enabled    |         | false                                                              | 2021-05-16 15:31:06.01191+00
 400 | processor.s4c_mdb1.l2ab04_enabled    |         | false                                                              | 2021-05-16 15:31:06.01191+00
 401 | processor.s4c_mdb1.l2ab05_enabled    |         | false                                                              | 2021-05-16 15:31:06.01191+00
 402 | processor.s4c_mdb1.l2ab06_enabled    |         | false                                                              | 2021-05-16 15:31:06.01191+00
 403 | processor.s4c_mdb1.l2ab07_enabled    |         | false                                                              | 2021-05-16 15:31:06.01191+00
 404 | processor.s4c_mdb1.l2ab08_enabled    |         | false                                                              | 2021-05-16 15:31:06.01191+00
 405 | processor.s4c_mdb1.l2ab8a_enabled    |         | false                                                              | 2021-05-16 15:31:06.01191+00
 406 | processor.s4c_mdb1.l2ab11_enabled    |         | false                                                              | 2021-05-16 15:31:06.01191+00
 407 | processor.s4c_mdb1.l2ab12_enabled    |         | false                                                              | 2021-05-16 15:31:06.01191+00
 236 | processor.l2a_msk.enabled            |         | false                                                              | 2021-05-18 14:54:17.288095+00

Yesterday, we started over with a fresh environment to be sure there was no changes that could disturb Sen4CAP.

There is however something interesting :

You can see that a download is still pending.
I know that some downloads from LSA (Luxembourg Space Agency Data Center) failed. This could be the reason?

I join the sen2agri-services logs as well.
sen2agri-services.log (1.1 MB)

Thank you for your support.

Regards,
James

MCa · June 29, 2023, 2:25pm

Hello Cosmin,

Following your request, we posted in this topic ( my team mates James did) an extract from the l2a_launcher.log. Did you had the possibility to have a look?

Unfortunately, there is not many information in it which is helping us to understand the problem as the log show always the same pattern below.

From /mnt/archive/maccs_def/<site_name>/l2a/output/<L2A_PRD>/l2a_<some_number>.log, we can not share anything as no job is started.

James added in his post an extract from L2A preprocessing from the DB in case this could help.

Thank you very much in advance for your help!

Best regards,

Marta