Version 2.0 - Creodias data source is not working

Dear @cudroiu and Sen4Cap,

We have upgraded to version 2.0 from V 1.3 and configured instruction from Creodias to attach credential. After that, we created new site and we found that system cannot download any products and cannot do pre-processing product. Here is what we found in the log.

2021-Feb-23 03:04:24.204 WARN [DefaultQuartzScheduler_Worker-8] o.e.s
.scheduling.LookupJob - At least one query failed for site d12provinces (reason: ro.cs.tao.datasource.QueryException: Parameter [platformName] not supported by data source ‘Creo DIAS’ for sensor ‘Sentinel2’). It was saved in ‘/mnt/archive/dwn_def/s
2/default/d12provinces/failed_queries’ and will be retried later.

Please note that requested season is 2019 onwards and we confirmed that Creodias has the data we requested.

I went through forum and I found that there is a new jar file for Creodias as this post (Update to 1.3 - No more Pre-Processing). Copied new jar file (tao-datasources-creodias-1.0.3.5.jar) to sen4cap and restarted service accordingly but system is not able to generate pre-processing product. Noticed that there are two jar files under lib folder.

-rw-r–r--. 1 root root 92356 Feb 23 03:03 tao-datasources-creodias-1.0.3.2.jar
-rw-r–r--. 1 root root 102056 Feb 23 03:37 tao-datasources-creodias-1.0.3.5.jar

We didn’t upgrade MAYA and it is still 3.2.2. Let us know how we can solve it out.

Thanks and Regards,
Henry

Dear Henry,

You should keep only one version of the Creodias datasource plugin so if you copied the version 1.0.3.5 you should delete the version 1.0.3.2 and then restart the sen4cap services.

Please let us know if that solves your issue.

Best regards,
Cosmin

Dear Cosmin,

Thank you for your tips. I deleted old version and restarted service. It didn’t help. Let me know if you need other logs for troubleshooting.

Thanks and Regards,
Henry

Dear Henry,

Could you please provide the logs for the sen4cap services?
Also, as there is a machine from Creodias, if you want, I can have a look if you provide me access to it.

Best regards,
Cosmin

Dear Cosmin,

I have PM you to access VM of Creodias.

Thanks and Regards,
Henry

Dear @cudroiu,

Could you have a quick look? i have given access of VM in the private message.

Thanks and Regards,
Henry

Hi Henry,

I had a look over the machine and I identified the following:

  • somehow, the sen4cap services were not updated during the system upgrade to 2.0. Did you noticed any messages during the upgrade? What is strange is that the other components seem to be correctly upgraded. I updated the services to the latest version.
  • The error with the parameterName was because the previous settings in the table “datasource” were kept for the Creodias datasource (and a parameterName property was previously present) but which is not valid now anymore. To correct this there are 2 solutions:

First Solution:
Directly change the entries in the datasource table like the following:

psql -U admin sen4cap
sen4cap# update datasource set specific_params = ‘{“parameters”:[{“name”:“productType”,“type”:“java.lang.String”,“value”:“L1C”},{“name”:“s
tatus”,“type”:“java.lang.String”,“value”:“all”}]}’ where name = ‘Creo DIAS’ and satellite_id = 1;
sen4cap# update datasource set specific_params = ‘{“parameters”:[{“name”:“processingLevel”,“type”:“java.lang.String”,“value”:“LEVEL1TP”},{
“name”:“productType”,“type”:“java.lang.String”,“value”:“L1TP”},{“name”:“status”,“type”:“java.lang.String”,“value”:“all”},{“name”:“dataset”,“type”:“java.lang.String”,“value”:“ESA-DATASET”}]}’ where name = ‘Creo DIAS’ and satellite_id = 2;
sen4cap# update datasource set specific_params = ‘{“parameters”:[{“name”:“processingLevel”,“type”:“java.lang.String”,“value”:“LEVEL1”},{“n
ame”:“productType”,“type”:“java.lang.String”,“value”:“SLC”},{“name”:“sensorMode”,“type”:“java.lang.String”,“value”:“IW”},{“name”:“status”,“type”:“java.lang.String”,“value”:“all”}]}’ where name = ‘Creo DIAS’ and satellite_id = 3;

Please note that the query for S2 (satellite_id = 1) should be set to have “value”:“L2A” instead of “value”:“L1C” if your datasource is fetching L2A products from repository.

Second solution:

  • rename the jar /usr/share/sen2agri/sen2agri-services/lib/tao-datasources-creodias-1.0.3.5.jar into something not loadable by the services:

sudo mv /usr/share/sen2agri/sen2agri-services/lib/tao-datasources-creodias-1.0.3.5.jar /usr/share/sen2agri/sen2agri-services/lib/tao-datasources-creodias-1.0.3.5.jar1

  • restart services:

sudo systemctl restart sen2agri-services

  • rename back the jar to its original name:

sudo mv /usr/share/sen2agri/sen2agri-services/lib/tao-datasources-creodias-1.0.3.5.jar1 /usr/share/sen2agri/sen2agri-services/lib/tao-datasources-creodias-1.0.3.5.jar

  • restart again the services:

sudo systemctl restart sen2agri-services

  • Go in the “datasources” tab and enable the Creodias datasources by setting:
    • Scope = Query and download
    • Fetch mode = Direct link to product
    • Local root =
      L8 = /eodata/Landsat-8/OLI_TIRS/L1T/
      S1 = /eodata/Sentinel-1/SAR/SLC/
      S2 = /eodata/Sentinel-2/MSI/L1C/ or /eodata/Sentinel-2/MSI/L2A/ (depending on what level you are interested in fetching)
    • User/password = test/test or whatever you want (is not actually used)
    • Other parameters unchanged

Nevertheless, for your site d12provinces which is a multipolygon it seems that there are several queries made in parallel which in the end are rejected by Creodias Finder with the error: “Too Many Requests”. We are still analyzing this error but a quick fix might be to simplify the shapefile of your site.

Cosmin

Dear Cosmin,

Thanks a lot for detail explanation and troubleshooting.

I will follow your steps accordingly and get back to you in my soonest.

Regards,
Henry

Dear @cudroiu,

After I followed your instruction. Here is what I got.

System can retrieve L1C data from Creodias after I changed AOI to smaller scale but it’s not generating pre-processing data though. Any tips?

Thanks and Regards,
Henry

Hello Henry,

When working with Creodias plugin as datasource, the following lines need to be added in the file /usr/share/sen2agri/sen2agri-services/config/services.properties:

CreoDiasDataSource.Sentinel1.path.suffix = .SAFE
CreoDiasDataSource.Sentinel1.product.format = folder
CreoDiasDataSource.Sentinel2.path.suffix = .SAFE
CreoDiasDataSource.Sentinel2.product.format = folder

Otherwise, incorrect product paths are inserted into the database.
After that, the incorrect entries in the downloader_history should be removed. To see the incorrect entries you can do something like:

psql -U admin sen4cap -c “select * from downloader_history where full_path like ‘/eodata/Sentinel-2/MSI/%/S2%/S2%.SAFE’”

And to remove them:

psql -U admin sen4cap -c “delete from downloader_history where full_path like ‘/eodata/Sentinel-2/MSI/%/S2%/S2%.SAFE’”

If there are any entries in l1_tile_history related to these entries (and the delete operation fails) you can do first:

psql -U admin sen4cap -c “delete from l1_tile_history where downloader_history_id in (select id from downloader_history where full_path like ‘/eodata/Sentinel-2/MSI/%/S2%/S2%.SAFE’)”

And then re-execute the first delete command on downloader_history.

I did the above operations for you and now the S2 L2 processing is running OK.

Best regards,
Cosmin

1 Like

Dear Cosmin,

What a relief! Thanks a lot for your support as usual.

I will monitor closely.

Regards,
Henry