We have upgraded to version 2.0 from V 1.3 and configured instruction from Creodias to attach credential. After that, we created new site and we found that system cannot download any products and cannot do pre-processing product. Here is what we found in the log.
2021-Feb-23 03:04:24.204 WARN [DefaultQuartzScheduler_Worker-8] o.e.s
.scheduling.LookupJob - At least one query failed for site d12provinces (reason: ro.cs.tao.datasource.QueryException: Parameter [platformName] not supported by data source ‘Creo DIAS’ for sensor ‘Sentinel2’). It was saved in ‘/mnt/archive/dwn_def/s
2/default/d12provinces/failed_queries’ and will be retried later.
Please note that requested season is 2019 onwards and we confirmed that Creodias has the data we requested.
I went through forum and I found that there is a new jar file for Creodias as this post (Update to 1.3 - No more Pre-Processing). Copied new jar file (tao-datasources-creodias-1.0.3.5.jar) to sen4cap and restarted service accordingly but system is not able to generate pre-processing product. Noticed that there are two jar files under lib folder.
-rw-r–r--. 1 root root 92356 Feb 23 03:03 tao-datasources-creodias-1.0.3.2.jar
-rw-r–r--. 1 root root 102056 Feb 23 03:37 tao-datasources-creodias-1.0.3.5.jar
We didn’t upgrade MAYA and it is still 3.2.2. Let us know how we can solve it out.
You should keep only one version of the Creodias datasource plugin so if you copied the version 1.0.3.5 you should delete the version 1.0.3.2 and then restart the sen4cap services.
Could you please provide the logs for the sen4cap services?
Also, as there is a machine from Creodias, if you want, I can have a look if you provide me access to it.
I had a look over the machine and I identified the following:
somehow, the sen4cap services were not updated during the system upgrade to 2.0. Did you noticed any messages during the upgrade? What is strange is that the other components seem to be correctly upgraded. I updated the services to the latest version.
The error with the parameterName was because the previous settings in the table “datasource” were kept for the Creodias datasource (and a parameterName property was previously present) but which is not valid now anymore. To correct this there are 2 solutions:
First Solution:
Directly change the entries in the datasource table like the following:
psql -U admin sen4cap
sen4cap# update datasource set specific_params = ‘{“parameters”:[{“name”:“productType”,“type”:“java.lang.String”,“value”:“L1C”},{“name”:“s
tatus”,“type”:“java.lang.String”,“value”:“all”}]}’ where name = ‘Creo DIAS’ and satellite_id = 1;
sen4cap# update datasource set specific_params = ‘{“parameters”:[{“name”:“processingLevel”,“type”:“java.lang.String”,“value”:“LEVEL1TP”},{
“name”:“productType”,“type”:“java.lang.String”,“value”:“L1TP”},{“name”:“status”,“type”:“java.lang.String”,“value”:“all”},{“name”:“dataset”,“type”:“java.lang.String”,“value”:“ESA-DATASET”}]}’ where name = ‘Creo DIAS’ and satellite_id = 2;
sen4cap# update datasource set specific_params = ‘{“parameters”:[{“name”:“processingLevel”,“type”:“java.lang.String”,“value”:“LEVEL1”},{“n
ame”:“productType”,“type”:“java.lang.String”,“value”:“SLC”},{“name”:“sensorMode”,“type”:“java.lang.String”,“value”:“IW”},{“name”:“status”,“type”:“java.lang.String”,“value”:“all”}]}’ where name = ‘Creo DIAS’ and satellite_id = 3;
Please note that the query for S2 (satellite_id = 1) should be set to have “value”:“L2A” instead of “value”:“L1C” if your datasource is fetching L2A products from repository.
Second solution:
rename the jar /usr/share/sen2agri/sen2agri-services/lib/tao-datasources-creodias-1.0.3.5.jar into something not loadable by the services:
Go in the “datasources” tab and enable the Creodias datasources by setting:
Scope = Query and download
Fetch mode = Direct link to product
Local root =
L8 = /eodata/Landsat-8/OLI_TIRS/L1T/
S1 = /eodata/Sentinel-1/SAR/SLC/
S2 = /eodata/Sentinel-2/MSI/L1C/ or /eodata/Sentinel-2/MSI/L2A/ (depending on what level you are interested in fetching)
User/password = test/test or whatever you want (is not actually used)
Other parameters unchanged
Nevertheless, for your site d12provinces which is a multipolygon it seems that there are several queries made in parallel which in the end are rejected by Creodias Finder with the error: “Too Many Requests”. We are still analyzing this error but a quick fix might be to simplify the shapefile of your site.
When working with Creodias plugin as datasource, the following lines need to be added in the file /usr/share/sen2agri/sen2agri-services/config/services.properties:
Otherwise, incorrect product paths are inserted into the database.
After that, the incorrect entries in the downloader_history should be removed. To see the incorrect entries you can do something like:
psql -U admin sen4cap -c “select * from downloader_history where full_path like ‘/eodata/Sentinel-2/MSI/%/S2%/S2%.SAFE’”
And to remove them:
psql -U admin sen4cap -c “delete from downloader_history where full_path like ‘/eodata/Sentinel-2/MSI/%/S2%/S2%.SAFE’”
If there are any entries in l1_tile_history related to these entries (and the delete operation fails) you can do first:
psql -U admin sen4cap -c “delete from l1_tile_history where downloader_history_id in (select id from downloader_history where full_path like ‘/eodata/Sentinel-2/MSI/%/S2%/S2%.SAFE’)”
And then re-execute the first delete command on downloader_history.
I did the above operations for you and now the S2 L2 processing is running OK.