Use of local product repository as S2 L1C datasource

mlop · September 4, 2020, 8:48am

Dear Sen4CAP team,

I have a historical season of S2 L1C products already downloaded in a local folder, that I would like to use as a local product repository for Sen4CAP.
I have followed the procedure detailled in the Software User Manuel ($4.2.1.2) to change the data source.
The structure of the folders is the default one (S2 L1C unzipped .SAFE folders organized as YYYY/MM/DD/product.SAFE). See screenshot below:

However, after uploading my new site, the system doesn’t include the corresponding S2 products in the database and thus doesn’t process them. It only includes the S1 products which data source is Scihub.

Here is a screenshot of the S2 data source configuration in the web interface:

And an extract from the /usr/share/sen2agri/sen2agri-services/config/services.properties configuration file (I’ve added the new S2 settings after ##Local repository):

server.port = 8080
spring.datasource.url=jdbc:postgresql://localhost:5432/sen4cap?stringtype=unspecified
spring.datasource.username=admin
spring.datasource.password=sen2agri

plugins.use.docker = false
# GDAL docker container if plugins.use.docker=true. Uncomment and change if different than geodata/gdal (default)
# docker.gdal.image = geodata/gdal

##################################################################
## Datatabase configuration polling (in minutes). 0 means disabled
database.config.polling=0
SciHubDataSource.Sentinel1.auto.uncompress=false
## IMPORTANT NOTE: if value for key "downloader.use.esa.l2a" is set to true in the config table, this value should be set on true
SciHubDataSource.Sentinel2.auto.uncompress=false
## What to do if, in local archive mode, a product is not found
## Possible values are: none|download
SciHubDataSource.Sentinel2.local.missing.action=download

###Local repository
SciHubDataSource.Sentinel2.local_archive_path=/mnt/archive/store
SciHubDataSource.Sentinel2.fetch_mode=4
SciHubDataSource.Sentinel2.local.archive.path.format = YYYY/MM/DD
SciHubDataSource.Sentinel2.path.suffix = .SAFE
SciHubDataSource.Sentinel2.product.format = folder
SciHubDataSource.Sentinel2.enabled=true
#SciHubDataSource.Sentinel2.scope=3

I’d be very grateful if you could help me figure out why Sen4Cap doesn’t want to use the already downloaded products.

Best regards

cudroiu · September 4, 2020, 8:58am

Hello,

Could you please try to set the scope of the datasource to “Query and Download” not just “Query”?
Are the products imported after the restart of the sen4cap services?

Please let us know.

Best regards,
Cosmin

mlop · September 4, 2020, 9:45am

Dear Cosmin,

Thank you for your reply!
I’ve changed the scope to “Query and download” and now the S2 products appear in the database, but with the incorrect path and a status_id = 3 with reason “Could not download or no tiles”. I suppose it tries to download the product from scihub.

Here an extract of the downloader_history table (it’s the same for all products):

id | site_id | satellite_id | product_name | full_path | created_timestamp | status_id | no_of_retries | product_date | orbit_id | orbit_type_id | status_reason | tiles | footprint

27512 | 8 | 1 | S2A_MSIL1C_20191229T172721_N0208_R012_T14SMJ_20191229T205120.SAFE | https://scihub.copernicus.eu/dhus/odata/v1/Products(‘b0e2fd0a-e033-4720-8c69-27c0f0ade6f2’)/$value
| 2020-09-04 09:03:44.331+00 | 3 | 2 | 2019-12-29 16:27:21+00 | 12 | 2 | Could not download or no tiles | {14SMJ} | 0103000020E61000000100000005000000E8A4F78DAF0
959C01427CC30AA6043407233DC80CFB858C0F1E2D73C62614340F168E388B5B858C0545A91F206E0434006F52D73BA0A59C0348B8B5748DF4340E8A4F78DAF0959C01427CC30AA604340
27511 | 8 | 1 | S2B_MSIL1C_20191227T173729_N0208_R055_T14SMJ_20191227T193002.SAFE | https://scihub.copernicus.eu/dhus/odata/v1/Products(‘0bf89f2e-e906-4cc1-9c2d-e6f7088b3708’)/$value
| 2020-09-04 09:03:44.223+00 | 3 | 2 | 2019-12-27 16:37:29+00 | 55 | 2 | Could not download or no tiles | {14SMJ} | 0103000020E6100000010000000C000000E8A4F78DAF0
959C01427CC30AA60434033E197FA79E758C0108BBD09F860434075029A081BE658C07252B674E668434050FC1873D7E258C02B9516ACA77B4340742497FF90DF58C054FB8A35688E43400551F70148DC58C0DF878BDB28A14340C2FA3F87F9D858C0FCC
27512E6B34340E8A4F78DAFD558C072BFDD59A3C64340A298BC0166D258C0F83030755ED94340C5C9FD0E45D158C0A1DC90DECDDF434006F52D73BA0A59C0348B8B5748DF4340E8A4F78DAF0959C01427CC30AA604340

Best regards

cudroiu · September 5, 2020, 10:40am

Hello,

Normally, if you set the “Local root”, the services will try to only import the products from the local repository specified.
Could you please provide me the content of the datasources table:

psql -U admin sen4cap -c “select * from datasource”

Also, could you please provide the full log of the sen4cap services during import?

sudo journalctl -fu sen2agri-services --since “3 days ago”

(the since should be adapted according to your system)

Best regards,
Cosmin

mlop · September 7, 2020, 8:32am

Dear Cosmin,

The content of the datasource table is the following (I have masked usernames and passwords):

id | satellite_id | name | scope | username | passwrd | fetch_mode | max_retries | retry_interval_minutes | download_path | specific_params | max_connections | local_root | enabled | site_id | secondary_datasource_id

3 | 1 | Scientific Data Hub | 3 | ****** | ****** | 5 | 1440 | 60 | /mnt/archive/dwn_def/s2/default | {“p
arameters”:[{“name”:“platformName”,“type”:“java.lang.String”,“value”:“Sentinel-2”},{“name”:“productType”,“type”:“java.lang.String”,“value”:“S2MSI1C”}]} | 2 | /mnt/archive/store | t |
|
1 | 2 | USGS | 3 | | | 1 | 72 | 60 | /mnt/archive/dwn_def/l8/default | {“p
arameters”:[{“name”:“platformName”,“type”:“java.lang.String”,“value”:“LANDSAT_8_C1”},{“name”:“minClouds”,“type”:“java.lang.Integer”,“value”:“0”},{“name”:“maxClouds”,“type”
:“java.lang.Integer”,“value”:“100”},{“name”:“productType”,“type”:“java.lang.String”,“value”:“L1TP”}]} | 1 | | f |
|
2 | 3 | Scientific Data Hub | 3 | ******* | ******* | 2 | 1440 | 60 | /mnt/archive/dwn_def/s1/default/ | {“p
arameters”:[{“name”:“platformName”,“type”:“java.lang.String”,“value”:“Sentinel-1”},{“name”:“productType”,“type”:“java.lang.String”,“value”:“SLC”},{“name”:“polarisation”,“t
ype”:“java.lang.String”,“value”:“VV+VH”},{“name”:“sensorOperationalMode”,“type”:“java.lang.String”,“value”:“IW”}]} | 2 | | t |
|

I changed the fetch mode to “Direct link to product” and it has just changed the status_reason to “Product XXXXXX not found in the local archive”.

I’ve attached the log of the sen2agri-services but not since 3 days ago, otherwise the file is too large to upload here.
sen2agri-services_1.log (1.5 MB)

I’ve noticed this

[Scientific Data Hub-S2-2] r.c.t.d.DefaultProductPathBuilder - Looking for product S2A_MSIL1C_20190516T172901_N0207_R055_T14SMJ_20190516T211022 into /mnt/archive/store/2019/05/136/S2A_MSIL1C_20190516T172901_N0207_R055_T14SMJ_20190516T211022.SAFE

It looks like the system is looking for the product acquisition day in the day of the year format. Should I change the service properties configuration file?

Thanks for your help!

mlop · September 7, 2020, 9:47am

Hello again,

In the services properties configuration file, I changed the Sentinel2 local archive path format to YYYY/MM/dd (instead of YYYY/MM/DD) and this solved my issue!
The S2 L1C product paths are now stored correctly in the database and processed by MAJA.

I think the procedure given in the SUM should be updated because it gives misleading information.

Thank you again for your help!

cudroiu · September 8, 2020, 11:11am

Hello,

Indeed, it seems that there is an issue in the SUM as DD is the doy and not the the day of the month. We did not encountered this issue so far as in the services.properties we always used dd.
Sorry for this inconvenience, we will update the SUM in the next version.
In any case, to avoid also an issue with the products in the last week of the year, you should use actually the format yyyy/MM/dd instead YYYY/MM/dd.

Best regards,
Cosmin

mlop · September 8, 2020, 11:50am

Dear Cosmin,

Thank you, this is really helpful.

Best regards

maatin · September 29, 2020, 10:07am

Dear all,

I have a similar issue here:

I use a local repository containing Sentinel-2 data in zip-archives.
The problem I have is that my directory-structure slightly differs from the proposed one: Sentinel-2A and 2B are in separate directories:
grafik

So I thought using a local repository like this could be a solution:

###Local repository
SciHubDataSource.Sentinel2.local_archive_path=/eodc/products/copernicus.eu
SciHubDataSource.Sentinel2.fetch_mode=4
SciHubDataSource.Sentinel2.local.archive.path.format = s2a_prd_msil1c/yyyy/MM/dd
SciHubDataSource.Sentinel2.auto.uncompress=true
SciHubDataSource.Sentinel2.path.suffix = none
SciHubDataSource.Sentinel2.product.format = zip
SciHubDataSource.Sentinel2.enabled=true
#SciHubDataSource.Sentinel2.scope=3

But the subfolder /s2a_prd_msil1c is not detected if I add it to the the local.archive.path.format before yyyy/MM/dd as it is suggested in the SUM in Section 4.2.1.2

I’m also not sure about the zip format. Is it possible to use symbolic links together with zipped datasets?

Do you think it’s possible to ingest the data like it is stored on our server?
Best M

cudroiu · September 30, 2020, 3:46pm

Hello,

The multiple directories as local root for the datasource is not supported in this moment (and I assume also not in the future) as there can be workarounds for this.
One workaround would be to create a script that creates in a whatever directory, let’s say /mnt/archive/repo with the subdirectories structure like yyyy/MM/dd in which you create symlinks (in the dd subdirectories leafs) to your S2A and S2B products in the /eodc/products/copernicus.eu/s2*_prd_msil1c/yyyy/MM/dd source directoris. This way, you will have a single root repository for all S2A and S2B products without the need of duplicating data.
Concerning zip format, yes, it is supported to ingest .zip products (of course if their internal structure is the correct one). But, it is true that we used it mostly when downloading products but not sure we tested when importing them from local root. Nevertheless, it should work also in this case and if not, you should follow the log files to see if you notice any relevant message.

Hope this helps.

Best regards,
Cosmin

maatin · October 12, 2020, 8:45am

Dear Cosmin,

finally the maja processor started processing:) the zip-files are supported by the local root. Using symlinks requires a “.SAFE” as file-extension of the symlink pointing to the zip-archive.

Best Martin

obu · December 9, 2020, 9:05am

Dear maatin,

how did you get the local archiive with S2 in zip folders to work?

I get the following log error:

The object (directory or file) /mnt/archive/dwn_def/s2/default/nw_nord/S2B_MSIL1C_20191104T104119_N0208_R008_T32UMC_20191104T124040.zip.SAFE/S2B_MSIL1C_20191104T104119_N0208_R008_T32UMC_20191104T124040.SAFE is not an archive: [Errno 20] Not a directory: ‘/mnt/archive/dwn_def/s2/default/nw_nord/S2B_MSIL1C_20191104T104119_N0208_R008_T32UMC_20191104T124040.zip.SAFE/S2B_MSIL1C_20191104T104119_N0208_R008_T32UMC_20191104T124040.SAFE’ !

in my service.properties if have the following setting:

##Local data repository
SciHubDataSource.Sentinel1.local_archive_path=/mnt/upload/L1C_rawdata/
SciHubDataSource.Sentinel1.local.archive.path.format = yyyy/MM/dd
SciHubDataSource.Sentinel2.path.suffix = none
SciHubDataSource.Sentinel2.product.format = .SAFE

I also tried “SciHubDataSource.Sentinel2.product.format = zip”, but that did not work either

Thanks
oliver

maatin · December 14, 2020, 9:32am

Dear obu,

my service properties file looks like this: my you need to add the correct path suffix:
`
###Local repository
SciHubDataSource.Sentinel2.local_archive_path=/mnt/archive/store
SciHubDataSource.Sentinel2.fetch_mode=4
SciHubDataSource.Sentinel2.local.archive.path.format = yyyy/MM/dd
SciHubDataSource.Sentinel2.path.suffix = .SAFE

my problem was that our archive with all the S2 zip-files is somewhere centralised and has the wrong structure. So I created symlinks of the zip-files pointing to my local storage having the right structure of yyyy/MM/dd. to make MAJA able to successfully read the symlinks which are pointing to the zip-files I had to change their extension to .SAFE. Not sure why this was necessary but then in the end maja started processing the files.

Hope this helps a bit.
best M

obu · December 16, 2020, 1:52pm

Thanks,

seems like an “error”. Changing the file-extension does not seem right. I got it working with the fetch mode “copy”, although the symlink option is definitely more useful.

Regards
O

henryhein · March 30, 2021, 5:40am

Dear @cudroiu and Sen4Cap,

I have similar issue and we have our local repository and its folder structure is same as sen4cap needed but Sen4Cap is not able to get it. Here is the log I got when I debug. How can I solve it out?

Here is our data source.

Here is configuration in services.properties

Local repository

SciHubDataSource.Sentinel2.local_archive_path=/eodata/products
SciHubDataSource.Sentinel2.local.archive.path.format=yyyy/MM/dd
SciHubDataSource.Sentinel2.path.suffix=none
SciHubDataSource.Sentinel2.fetch_mode=4
SciHubDataSource.Sentinel2.scope=3
SciHubDataSource.Sentinel2.enabled=true
SciHubDataSource.Sentinel2.product.format=folder

Here is log of sen2agri-services

Here is log from downloader_history table.

Here is log from failed_queries
[root@hadoop-srv-05 failed_queries]# cat 1_Sentinel2_20181101000000_20200131235959.json
{“user”:null,“password”:null,“id”:“a6449d7e-edcd-44a3-ae4c-ae5a2b66f88c”,“values”:{“footprint”:“POLYGON((100.419 14.08,100.422 14.086,100.425 14.082,100.423 14.078,100.419 14.08))”,“endDate”:[“2018-11-01 00:00:00”,“2020-01-31 23:59:59”],“tileId”:“47PPR”,“platformName”:“Sentinel-2”,“startDate”:[“2018-11-01 00:00:00”,“2020-01-31 23:59:59”],“productType”:“S2MSI1C”}}[root@havi /usr/share/sen2agri/sen2agri-services/config/services.properties

Thanks and Regards,
Henry

cudroiu · March 30, 2021, 9:04am

Dear Henry,

Are you using Creodias or AWS? If you are using AWS, you might need to use the SciHub datasource for query only and the AWS datasource for download only, using a local root and the Fetch mode to symlink or copy.
You can configure that directly in the web interface, for each datasource.

Hope this helps.

Best regards,
Cosmin

henryhein · March 31, 2021, 1:39am

Dear @cudroiu,

Thanks a lot for your advice as usual. Yes, It’s AWS repository. I have configured as below and status is changed to “In Progress” though but it is still not processing L2A. Is there anything I am missing?

Best Regards,
Henry

Here is folder structure and product info from AWS archive.