Hello,
I am currently processing a 2020 season over a region and I am currently facing some issues in L3B processing.
I give you some context.
My system is configured to handle both S1 SLC input products and S2 L2A (the one delivered on scihub, not maya processed) already stored in internal storages, mounted in custom paths on my processing virtual machine. I managed to configure sen4CAP properly, following the instructions descripted in the user manual, and indeed the S1 pre processing started and their output, together with the S2 L2A products, are correctly inventoried and showed in the GUI (tab products) and in the psql database:
S1 screenshot:
S2 screenshot:
The l2a-s1 processing tasks ended around two days ago and then I noticed that the L3B processing, scheduled every day at midnight, keeps failing, as you may see from the job history:
Looking at the output error of one of the failed tasks, there is a generic-error-step reporting that the system is not able to find the available S2 L2A input products:
“L3B Scheduled job with id 2044 for site 1 marked as done as no products are available for now to process”
That message looked strange to me and therefore I tried to trigger a custom job, via GUI, where indeed the S2 L2A input products for L3B processing are recognized:
The submitted custom job went anyway in error too, but with another output message related to the step GenerateLaiMonoDateMaskFlags:
2021 May 27 07:30:17 : Application.logger (FATAL) The following error occurred during application execution : /sen2agri/sen2agri-processors/ProductReaders/MetadataHelpers/src/MetadataHelperFactory.cpp:53: itk::ERROR: MetadataHelperFactory(0x198fa90): Unable to read metadata from /mnt/archive/input/S2-L2A/completed/2020/03/01/S2A_MSIL2A_20200301T105021_N0214_R051_T31UDS_20200301T122129.SAFE/MTD_MSIL2A.xml
It is worth to say that the indicated path exists and points to an existing MTD_MSIL2A.xml file, but the SAFE directory under /mnt/archive/input/S2-L2A/completed/2020/03/01/ is instead a symbolic link pointing to the physical product stored on another path. I did that following the indications of this post in order to made the inputs available to the downloader with the proper directory organization.
In order to debug a little bit this beahviour, I executed the command in error in the VM by terminal, getting unfortunately the same error:
docker run --rm -u 1001:1001 -v /mnt/archive/orchestrator_temp/l3b:/mnt/archive/orchestrator_temp/l3b -v /mnt/archive:/mnt/archive -v /mnt/data/sen4CAP/archive:/mnt/data/sen4CAP/archive sen4cap/processors:2.0.0 otbcli GenerateLaiMonoDateMaskFlags -inxml /mnt/archive/input/S2-L2A/completed/2020/03/01/S2A_MSIL2A_20200301T105021_N0214_R051_T31UDS_20200301T122129.SAFE/MTD_MSIL2A.xml -out /mnt/archive/orchestrator_temp/l3b/2047/25208-lai-processor-mask-flags/LAI_mono_date_msk_flgs_img.tif -outres 10 -outresampled /mnt/archive/orchestrator_temp/l3b/2047/25208-lai-processor-mask-flags/LAI_mono_date_msk_flgs_img_resampled.tif
2021 May 27 10:22:38 : Application.logger (FATAL) The following error occurred during application execution : /sen2agri/sen2agri-processors/ProductReaders/MetadataHelpers/src/MetadataHelperFactory.cpp:53:
itk::ERROR: MetadataHelperFactory(0x1b0ba90): Unable to read metadata from /mnt/archive/input/S2-L2A/completed/2020/03/01/S2A_MSIL2A_20200301T105021_N0214_R051_T31UDS_20200301T122129.SAFE/MTD_MSIL2A.xml
Looking at the docker mount points I noticed that there is just -v /mnt/archive:/mnt/archive, that I was saying is the path hosting the symbolic links to the input products physical location, thus they are reachable under that directory. Anyway I manually added the product physical location as additional docker mount point and the task succeded:
docker run --rm -u 1001:1001 -v /mnt/NAS3:/mnt/NAS3 -v /mnt/archive/orchestrator_temp/l3b:/mnt/archive/orchestrator_temp/l3b -v /mnt/archive:/mnt/archive -v /mnt/data/sen4CAP/archive:/mnt/data/sen4CAP/archive sen4cap/processors:2.0.0 otbcli GenerateLaiMonoDateMaskFlags -inxml /mnt/archive/input/S2-L2A/completed/2020/03/01/S2A_MSIL2A_20200301T105021_N0214_R051_T31UDS_20200301T122129.SAFE/MTD_MSIL2A.xml -out /mnt/archive/orchestrator_temp/l3b/2047/25208-lai-processor-mask-flags/LAI_mono_date_msk_flgs_img.tif -outres 10 -outresampled /mnt/archive/orchestrator_temp/l3b/2047/25208-lai-processor-mask-flags/LAI_mono_date_msk_flgs_img_resampled.tif
I realized that something was missing and then I found additional instructions in the user manual under Appendix E.15 Processors execution from docker or locally, to configure potential additional docker mount points when users are using custom paths from where products are imported in the system with the “Direct link to product” option (my case). Thus I updated directly the proper record on the db:
After restarting the services, the L3B processing is unfortunately still failing, both when it is triggered via the orchestrator scheduling and as a custom job, returning the same errors I previously reported in this thread.
The only way I am managing to process L3B products is manually triggering the job in the terminal following the instructions of the paragraph “4.4.4.2 Manual execution from terminal” of the user manual, but I’d rather let the system do that with its own orchestration to avoid to face potential issues when generating higher level products.
Could you please help me?
Thank you in advance,
Salvatore.