Sentinel-2 L2A processing not fails

Dear all,

After an update to 3.0 demmaccs-processor stopped processing S2-Data (it works for Landsat8 now)

I use a local root with symlinks to get the Level1-Data from an archive.

Main error:
–docker-image-maja sen4x/maja:3.2.2-centos-el info finished with return code 126 in 0:00:00.33805
Can NOT run MAJA script, error code: 1
Could not read maja log at /mnt/archivR022_T33UWP_20181105T104758.SAFE/maja.log

full-log here:

 sen2agri-demmaccs.service - Runs MACCS on L1C products
   Loaded: loaded (/usr/lib/systemd/system/sen2agri-demmaccs.service; static; vendor preset: disabled)
   Active: active (running) since Thu 2022-01-13 15:21:58 UTC; 1min 34s ago
 Main PID: 17730 (python2)
    Tasks: 5
   Memory: 477.8M
   CGroup: /system.slice/sen2agri-demmaccs.service
           └─17730 python2 /usr/share/sen2agri/sen2agri-demmaccs/l2a_launcher.py

Jan 13 15:22:24 sen4cap-v2-25062021.novalocal l2a_launcher.py[17730]: <worker 0> Running L2a Processors, console outputSIL2A_20181105T101211_N0206_R022_T33UWP_20181105T104758.SAFE/l2a_14019.log
Jan 13 15:22:24 sen4cap-v2-25062021.novalocal l2a_launcher.py[17730]: <worker 0> Command docker run -v /var/run/docker.7 -v /etc/localtime:/etc/localtime -v /usr/share/zoneinfo:/usr/share/zoneinfo -v /mnt/archive/srtm:/mnt/archive/srtm -vja/UserConfiguration:/mnt/archive/gipp_maja/UserConfiguration -v /mnt/archive/gipp_maja:/mnt/archive/gipp_maja -v /mnt/hive/demmaccs_tmp/archives/tmppgNAFO/S2A_MSIL1C_20181105T101211_N0206_R022_T33UWP_20181105T104758.SAFE:/mnt/archive/dem6_R022_T33UWP_20181105T104758.SAFE -v /mnt/archive/maccs_def/uwp/l2a/output/S2A_MSIL2A_20181105T101211_N0206_R022_T33UW/S2A_MSIL2A_20181105T101211_N0206_R022_T33UWP_20181105T104758.SAFE --name l2a_processors_14019_I4JUIL0E sen4x/l2a-proceve/srtm --swbd /mnt/archive/swbd --conf /mnt/archive/gipp_maja/UserConfiguration --processes-number-dem 1 --processes-nr /mnt/archive/demmaccs_tmp/ --delete-temp --product-id 14019 /mnt/archive/demmaccs_tmp/archives/tmppgNAFO/S2A_MSIL1C_2rchive/maccs_def/uwp/l2a/output/S2A_MSIL2A_20181105T101211_N0206_R022_T33UWP_20181105T104758.SAFE --tiles-to-process 33ge-l8align sen4x/l2a-l8-alignment:0.1 --docker-image-dem sen4x/l2a-dem:0.1 --docker-image-maja sen4x/maja:3.2.2-centos-el info finished with return code 126 in 0:00:00.338057
Jan 13 15:22:24 sen4cap-v2-25062021.novalocal l2a_launcher.py[17730]: <worker 0> Can NOT run MAJA script, error code: 1
Jan 13 15:22:24 sen4cap-v2-25062021.novalocal l2a_launcher.py[17730]: <worker 0> Successful processing = False
Jan 13 15:22:24 sen4cap-v2-25062021.novalocal l2a_launcher.py[17730]: <worker 0> Invalid output format.
Jan 13 15:22:24 sen4cap-v2-25062021.novalocal l2a_launcher.py[17730]: <worker 0> Could not read maja log at /mnt/archivR022_T33UWP_20181105T104758.SAFE/maja.log
Jan 13 15:22:24 sen4cap-v2-25062021.novalocal l2a_launcher.py[17730]: <worker 0> Valid L2a product = False
Jan 13 15:22:24 sen4cap-v2-25062021.novalocal l2a_launcher.py[17730]: <master> Product 14019 was processed with errors,
Jan 13 15:22:24 sen4cap-v2-25062021.novalocal l2a_launcher.py[17730]: <worker 1> Successful pre-processing = True
Jan 13 15:22:24 sen4cap-v2-25062021.novalocal l2a_launcher.py[17730]: <worker 1> Running L2a Processors, console outputSIL2A_20181104T095159_N0206_R079_T33UWP_20181104T120209.SAFE/l2a_14018.log

Anyone an idea what happens here? Seems that there is a problem with the path to the log-file? @cudroiu

best regards

Dear Maatin,

Actually, the log error is not the cause but a consequence of a previous error. We shall update the text of the error, as it should say: can not determine the precise error since no maja log was written.
Anyway, I tried to look at the logs you sent. However, I think there is a problem with their rendering. For example, I can see -v /var/run/docker.7 where it should have been -v /var/run/docker.sock:/var/run/docker.sock
Thus, could you please provide the log files, which you will find in /var/log/sen2agri/, i.e. l2a_launcher.log and l2a_14019.log; Also, could you state the values of all database parameters starting with processor.l2a. from config table.

Dear @abecheru

thank you for reaching out to me. Please find the l2a-logs in the attachment and below the requested rows from the config-table:

sen4cap=# select * from config where key like '%processor.l2a%';
 id  |                 key                  | site_id |                   value                    |         last_updated
-----+--------------------------------------+---------+--------------------------------------------+-------------------------------
  76 | executor.processor.l2a.name          |         | L2A                                        | 2015-06-03 14:02:50.028002+00
  77 | executor.processor.l2a.path          |         | /bin/false                                 | 2015-07-20 13:31:23.208369+00
  14 | processor.l2a.srtm-path              |         | /mnt/archive/srtm                          | 2016-02-25 09:11:36.372405+00
  15 | processor.l2a.swbd-path              |         | /mnt/archive/swbd                          | 2016-02-25 09:12:04.008319+00
  16 | processor.l2a.working-dir            |         | /mnt/archive/demmaccs_tmp/                 | 2016-02-25 15:31:06.01191+00
  11 | processor.l2a.optical.output-path    |         | /mnt/archive/maccs_def/{site}/{processor}/ | 2016-02-24 16:09:17.379905+00
   9 | processor.l2a.maja.gipp-path         |         | /mnt/archive/gipp_maja                     | 2016-02-24 16:12:16.464479+00
   8 | processor.l2a.optical.compress-tiffs |         | true                                       | 2021-01-10 12:14:15.083382+00
   7 | processor.l2a.optical.cog-tiffs      |         | true                                       | 2021-01-10 12:14:15.083382+00
  12 | processor.l2a.maja.remove-fre        |         | false                                      | 2021-01-25 12:53:51.757887+00
  13 | processor.l2a.maja.remove-sre        |         | true                                       | 2021-01-10 12:14:15.083382+00
 435 | processor.l2a.optical.num-workers    |         | 4                                          | 2020-09-07 11:36:37.906825+00
 444 | processor.l2a.optical.max-retries    |         | 3                                          | 2020-09-15 13:02:27.164968+00
 446 | processor.l2a.optical.retry-interval |         | 1 day                                      | 2020-09-07 11:36:37.906825+00
 453 | processor.l2a.sen2cor.gipp-path      |         | /mnt/archive/gipp/sen2cor                  | 2020-09-15 13:48:05.415193+00
 433 | processor.l2a.s2.implementation      |         | maja                                       | 2020-09-07 11:17:52.846794+00
 505 | processor.l2a.sen2cor_image          |         | sen4x/sen2cor:2.9.0-ubuntu-20.04           | 2021-04-19 16:30:00+00
 504 | processor.l2a.processors_image       |         | sen4x/l2a-processors:0.1.2                 | 2021-04-19 16:30:00+00
 508 | processor.l2a.l8_align_image         |         | sen4x/l2a-l8-alignment:0.1                 | 2021-04-19 16:30:00+00
 509 | processor.l2a.dem_image              |         | sen4x/l2a-dem:0.1                          | 2021-04-19 16:30:00+00
 506 | processor.l2a.maja_image             |         | sen4x/maja                                 | 2021-04-19 16:30:00+00
 507 | processor.l2a.gdal_image             |         | osgeo/gdal:ubuntu-full-3.2.0               | 2021-04-19 16:30:00+00

l2a_log.zip (134.1 KB)

best regards

Dear maatin,

There is a permission issue while docker tries to mount some temporary/working files, see below an extract from l2a_14681.log:

/usr/bin/docker-current: Error response from daemon: error while creating mount source path '/mnt/archive/demmaccs_tmp/archives/tmpK49g7h/S2B_MSIL1C_20210505T100029_N0300_R122_T33UXP_20210505T123835.SAFE': mkdir /mnt/archive/demmaccs_tmp/archives/tmpK49g7h/S2B_MSIL1C_20210505T100029_N0300_R122_T33UXP_20210505T123835.SAFE: permission denied.

Please do check the permissions for /mnt/archive/demmaccs_tmp/, the user sen2agri-service should be able to read and write. Also, If there any folders within demmaccs_tmp, they should be removed, or at least make sure that sen2agri-service user is able to remove them.

Please keep me posted.

Dear @abecheru ,

hm I checked everything and the user sen2agri-service is able to write to and read in /mnt/archive/demaccs_tmp. In fact it’s the owner of the directory:

[sen2agri-service@sen4cap-v2-25062021 archive]$ ls -l
total 2123
drwxr-xr-x.  4 sen2agri-service sen2agri-service    4096 Jun 26  2021 at2021uxp
drwxrwxr-x.  3 centos           centos              4096 Jun 24  2021 change_scihub_jar_cfg
**drwxr-xr-x.  2 sen2agri-service sen2agri-service   32768 Jan 17 21:25 demmaccs_tmp**
drwxr-xr-x.  4 sen2agri-service sen2agri-service    4096 Jul  6  2021 dwn_def
drwxrwxrwx.  5 uba              uba                 4096 Jan 13 13:24 gipp_maja
drwxr-xr-x.  4 sen2agri-service sen2agri-service    4096 Feb  8  2021 gis_ela
drwxr-xr-x.  5 sen2agri-service sen2agri-service    4096 Jan 13 13:30 gump
drwxr-xr-x.  2 sen2agri-service sen2agri-service    4096 Oct  7  2020 L3B_GeneratedModels
drwxr-xr-x. 13 sen2agri-service sen2agri-service    4096 Jan 14 16:06 maccs_def
drwxrwxr-x.  3 centos           centos              4096 Apr  6  2021 MAJA421
drwxrwxr-x.  3 centos           centos              4096 Apr  6  2021 maja-gipp042021
drwxrwxrwx.  3 uba              uba                 4096 Dec 21  2020 orchestrator_temp
drwxr-xr-x.  2 uba              uba                 4096 Sep  7  2020 reference_data
drwxr-xr-x.  4 sen2agri-service sen2agri-service    4096 Jul 21 00:02 s33uvp_21
drwxr-xr-x.  4 sen2agri-service sen2agri-service    4096 Jul  7  2021 s33uwp_21
drwxrwxrwx.  2 uba              uba                32768 Sep  4  2020 srtm
drwxrwxr-x.  6 sen2agri-service sen2agri-service    4096 Sep 10 09:11 store
drwxrwxr-x.  3 sen2agri-service sen2agri-service    4096 Jan 17 13:03 storezip
drwxr-xr-x.  3 sen2agri-service sen2agri-service    4096 Jan 25  2021 stvn
drwxrwxrwx.  2 uba              uba              2097152 Jan  9  2021 swbd
drwxrwxrwx.  3       4294967294 uba                 4096 Jan 13 13:30 temp
drwxrwxr-x.  3 uba              uba                 4096 Sep 10  2020 upload
drwxr-xr-x.  4 sen2agri-service sen2agri-service    4096 Jan  7 00:04 uup21
drwxr-xr-x.  4 sen2agri-service sen2agri-service    4096 Dec 23 08:46 uxp0921on

I can also copy and delete file to the archive logged in as user sen2agri-service. And what is also strange is that the landsat scence are downloaded and processed but sentinel-2 scenes stuck somewhere.

Could the problem be that I’m using symlinks pointing to my localroot directory and the migration to a docker environment from version 2 to version 3 doesn’t support this?

best Martin