Hello @cudroiu, @lnicola
On Sen4CAP 2.0 there seems to be an issue with MAJA processing of S-2 data - the process is failing to initialize.
When exporting a log from demmaccs services, this message is printed on repeat:
Apr 22 21:59:02 gis-gpu.lad.gov.lv systemd[1]: Started Runs MACCS on L1C products.
Apr 22 21:59:02 gis-gpu.lad.gov.lv l2a_launcher.py[5878]: 2021-04-22 21:59:02.397119:[5878]:MainThread: Successful db fetch
Apr 22 21:59:02 gis-gpu.lad.gov.lv l2a_launcher.py[5878]: (launcher err) Invalid Maja configuration file /mnt/archive/gipp/maja/UserConfiguration.
Apr 22 21:59:02 gis-gpu.lad.gov.lv l2a_launcher.py[5878]: 2021-04-22 21:59:02.397413:[5878]:(launcher err) Invalid Maja configuration file.
Apr 22 21:59:02 gis-gpu.lad.gov.lv l2a_launcher.py[5878]: 2021-04-22 21:59:02.397442:[5878]:Invalid processing context
Apr 22 21:59:02 gis-gpu.lad.gov.lv systemd[1]: sen2agri-demmaccs.service: main process exited, code=exited, status=1/FAILURE
Apr 22 21:59:02 gis-gpu.lad.gov.lv systemd[1]: Unit sen2agri-demmaccs.service entered failed state.
Apr 22 21:59:02 gis-gpu.lad.gov.lv systemd[1]: sen2agri-demmaccs.service failed.
There is no directory UserConfiguration in the gipp/maja folder. Any idea how to solve this?
Dear Harijsi,
It seems that during install/update there was an error copying some of the MAJA configuration files.
In this case, you can try copy manually from the installation package the folder “UserConfiguration” into the MAJA gipp folder location:
sudo cp -fR <Install_Package_Location>/Sen4CAPDistribution/install_script/config/maja/UserConfiguration /mnt/archive/gipp_maja
Please note that for a fresh installation the target location is /mnt/archive/gipp/maja/ instead of /mnt/archive/gipp_maja.
Hope this helps.
Best regards,
Cosmin
1 Like
Dear @cudroiu,
Thank you for the tip. I copied the config file to the location. Now logs show that the process starts, but still fails on the same tile on repeat.
Apr 27 09:26:04 gis-gpu.lad.gov.lv systemd[1]: Started Runs MACCS on L1C products.
Apr 27 09:26:04 gis-gpu.lad.gov.lv l2a_launcher.py[11568]: 2021-04-27 09:26:04.814809:[11568]:MainThread: Successful db fetch
Apr 27 09:26:04 gis-gpu.lad.gov.lv l2a_launcher.py[11568]: 2021-04-27 09:26:04.827765:[11568]:MainThread: Successful db fetch
Apr 27 09:26:04 gis-gpu.lad.gov.lv l2a_launcher.py[11568]: 2021-04-27 09:26:04.872135:[11568]:MainThread: Successful db fetch
Apr 27 09:26:04 gis-gpu.lad.gov.lv l2a_launcher.py[11568]: 2021-04-27 09:26:04.883834:[11568]:MainThread: Successful db fetch
Apr 27 09:26:04 gis-gpu.lad.gov.lv l2a_launcher.py[11568]: 2021-04-27 09:26:04.888757:[11568]:MainThread: Successful db fetch
Apr 27 09:26:04 gis-gpu.lad.gov.lv l2a_launcher.py[11568]: (launcher info) <master>: product 342 assigned to <worker 0>
Apr 27 09:26:04 gis-gpu.lad.gov.lv l2a_launcher.py[11568]: #################### Tile & Site Info ####################
Apr 27 09:26:04 gis-gpu.lad.gov.lv l2a_launcher.py[11568]: (launcher info) <worker 0>: site_id = 4
Apr 27 09:26:04 gis-gpu.lad.gov.lv l2a_launcher.py[11568]: (launcher info) <worker 0>: satellite_id = 1
Apr 27 09:26:04 gis-gpu.lad.gov.lv l2a_launcher.py[11568]: (launcher info) <worker 0>: orbit_id = 36
Apr 27 09:26:04 gis-gpu.lad.gov.lv l2a_launcher.py[11568]: (launcher info) <worker 0>: tile_id = 34VEH
Apr 27 09:26:04 gis-gpu.lad.gov.lv l2a_launcher.py[11568]: (launcher info) <worker 0>: downloader_history_id = 342
Apr 27 09:26:04 gis-gpu.lad.gov.lv l2a_launcher.py[11568]: (launcher info) <worker 0>: path = /mnt/archive/dwn_def/s2/default/latvija_full_v2/S2B_MSIL1C_20210218T094029_N0209_R036_T34VEH_20210218T115327.SAFE
Apr 27 09:26:04 gis-gpu.lad.gov.lv l2a_launcher.py[11568]: (launcher info) <worker 0>: previous_l2a_path = None
Apr 27 09:26:04 gis-gpu.lad.gov.lv l2a_launcher.py[11568]: (launcher info) <worker 0>: site_short_name = latvija_full_v2
Apr 27 09:26:04 gis-gpu.lad.gov.lv l2a_launcher.py[11568]: (launcher info) <worker 0>: site_output_path = /mnt/archive/maccs_def/latvija_full_v2/l2a/
Apr 27 09:26:04 gis-gpu.lad.gov.lv l2a_launcher.py[11568]: 2021-04-27 09:26:04.889266:[11568]:<worker 0>: 0: Starting extract_from_archive_if_needed for tile 34VEH
Apr 27 09:26:04 gis-gpu.lad.gov.lv l2a_launcher.py[11568]: 2021-04-27 09:26:04.889462:[11568]:<worker 0>: This wasn't an archive, so continue as is.
Apr 27 09:26:04 gis-gpu.lad.gov.lv l2a_launcher.py[11568]: 2021-04-27 09:26:04.889561:[11568]:<worker 0>: 0: Ended extract_from_archive_if_needed for tile 34VEH
Apr 27 09:26:04 gis-gpu.lad.gov.lv l2a_launcher.py[11568]: (launcher info) <worker 0>: Successful pre-processing = True
Apr 27 09:26:04 gis-gpu.lad.gov.lv l2a_launcher.py[11568]: 2021-04-27 09:26:04.893132:[11568]:<worker 0>: Successful pre-processing = True
Apr 27 09:26:04 gis-gpu.lad.gov.lv systemd[1]: sen2agri-demmaccs.service: main process exited, code=exited, status=1/FAILURE
Apr 27 09:26:04 gis-gpu.lad.gov.lv systemd[1]: Unit sen2agri-demmaccs.service entered failed state.
Apr 27 09:26:04 gis-gpu.lad.gov.lv systemd[1]: sen2agri-demmaccs.service failed.
Apr 27 09:27:04 gis-gpu.lad.gov.lv systemd[1]: Started Runs MACCS on L1C products.
Not really sure what is causing this since there is no explicit error message here, just a notification that the process exited with a failure status. Maybe you have some ideas where I could look to investigate further?
Best,
Harijs
For additional info, it seems that the config values related to the l2a processor (for S4C version 2.0) are correct. Here is a screenshot from the database, in case there is something that I missed:
We also reinstalled MAJA (3.2.2), but that did not solve aforementioned error.
Dear Harijsi,
Is this a new installation or an upgraded system?
From the config keys seems a new installation. If a new installation, did you encountered any issues with the installation?
Best regards,
Cosmin
1 Like
Dear Harijsi,
More details on the error can be found in /mnt/archive/maccs_def/latvija_full_v2/l2a/output/[xyz];
where xyz is a directory with the name of the failed product. You will find more logs in that directory, which you can share with us to help mtigate the problem.
Also, could you show us what docker images are on the system, especially l2a-processors image?
All the best
1 Like
By running sudo docker images.
1 Like
Thank you for your replies and involvement, it is greatly appreciated in our PA!
@cudroiu
Yes, the installation was fresh. We removed the previous Sen4CAP version (1.2) and installed 2.0. There were no errors in this process. We did not reinstall MAJA during this procedure. However, once we encountered the issue described in this topic, we reinstalled MAJA (3.2.2) as well. All config paths match the MAJA installation, but the problem persists.
@abecheru
The logs for failed products in the maccs_def folder print just one line on repeat:
2021-04-28 10:59:10.285679:[372]:<worker 0>: Successful pre-processing = True
2021-04-28 11:00:01.455509:[485]:<worker 0>: Successful pre-processing = True
2021-04-28 11:01:00.402065:[591]:<worker 0>: Successful pre-processing = True
2021-04-28 11:02:00.461222:[714]:<worker 0>: Successful pre-processing = True
2021-04-28 11:03:00.518353:[820]:<worker 0>: Successful pre-processing = True
2021-04-28 11:04:00.585176:[927]:<worker 0>: Successful pre-processing = True
2021-04-28 11:05:00.643364:[1070]:<worker 0>: Successful pre-processing = True
2021-04-28 11:06:00.693779:[1193]:<worker 0>: Successful pre-processing = True
2021-04-28 11:07:00.753670:[1303]:<worker 0>: Successful pre-processing = True
2021-04-28 11:08:00.828243:[1411]:<worker 0>: Successful pre-processing = True
2021-04-28 11:09:00.898324:[1523]:<worker 0>: Successful pre-processing = True
2021-04-28 11:10:00.967593:[1629]:<worker 0>: Successful pre-processing = True
2021-04-28 11:11:01.029349:[1741]:<worker 0>: Successful pre-processing = True
2021-04-28 11:12:01.082532:[1853]:<worker 0>: Successful pre-processing = True
2021-04-28 11:13:01.151067:[1978]:<worker 0>: Successful pre-processing = True
2021-04-28 11:14:01.213612:[2105]:<worker 0>: Successful pre-processing = True
2021-04-28 11:15:01.273974:[2219]:<worker 0>: Successful pre-processing = True
2021-04-28 11:16:01.339194:[2326]:<worker 0>: Successful pre-processing = True
2021-04-28 11:17:01.423314:[2435]:<worker 0>: Successful pre-processing = True
Here is the docker image list from our machine:
Hope this information is of some use.
Also, a bit unrelated, but wanted to let you know:
Copernicus Scihub changed the address for their API, so for a while download queries failed with the “Too many requests” error.
We changed the scihub.search.url property in the scihub.properties file to: “https://apihub.copernicus.eu/apihub/search”. Now the downloads seem to have resumed, I hope this is not just a temporary solution.
Best,
Harijs
Dear Harijsi,
Thank you for the additional info, unfortunately we can not pinpoint a particular reason; although we have some working suppositions and are currently trying to replicate your problem.
Hence, could you upload here, or via email if not possible on the forum, the following full logs:
- l2a_launcher.log from /tmp
- maja.log. from /tmp
- any product log from /mnt/archive/maccs_def/latvija_full_v2/l2a/output/.
Thank you for your patience
Thank you for the effort!
Here is a Google Drive folder with the logs you asked for:
Best,
Harijs
Dear Harijsi,
Could you please confirm that you have the folders /mnt/archive/swbd and /mnt/archive/srtm and that they have the right files in them?
If not, you should download the srtm.zip and swbd.zip files and copy the files inside into the above locations.
Best regards,
Cosmin
