Use of local product repository as S2 L1C datasource

System Installation and Configuration
#1

Dear Sen4CAP team,

I have a historical season of S2 L1C products already downloaded in a local folder, that I would like to use as a local product repository for Sen4CAP.
I have followed the procedure detailled in the Software User Manuel ($4.2.1.2) to change the data source.
The structure of the folders is the default one (S2 L1C unzipped .SAFE folders organized as YYYY/MM/DD/product.SAFE). See screenshot below:
image

However, after uploading my new site, the system doesn’t include the corresponding S2 products in the database and thus doesn’t process them. It only includes the S1 products which data source is Scihub.

Here is a screenshot of the S2 data source configuration in the web interface:

image
image1008×348 22.9 KB

And an extract from the /usr/share/sen2agri/sen2agri-services/config/services.properties configuration file (I’ve added the new S2 settings after ##Local repository):

server.port = 8080
spring.datasource.url=jdbc:postgresql://localhost:5432/sen4cap?stringtype=unspecified
spring.datasource.username=admin
spring.datasource.password=sen2agri

plugins.use.docker = false
# GDAL docker container if plugins.use.docker=true. Uncomment and change if different than geodata/gdal (default)
# docker.gdal.image = geodata/gdal

##################################################################
## Datatabase configuration polling (in minutes). 0 means disabled
database.config.polling=0
SciHubDataSource.Sentinel1.auto.uncompress=false
## IMPORTANT NOTE: if value for key "downloader.use.esa.l2a" is set to true in the config table, this value should be set on true
SciHubDataSource.Sentinel2.auto.uncompress=false
## What to do if, in local archive mode, a product is not found
## Possible values are: none|download
SciHubDataSource.Sentinel2.local.missing.action=download

###Local repository
SciHubDataSource.Sentinel2.local_archive_path=/mnt/archive/store
SciHubDataSource.Sentinel2.fetch_mode=4
SciHubDataSource.Sentinel2.local.archive.path.format = YYYY/MM/DD
SciHubDataSource.Sentinel2.path.suffix = .SAFE
SciHubDataSource.Sentinel2.product.format = folder
SciHubDataSource.Sentinel2.enabled=true
#SciHubDataSource.Sentinel2.scope=3

I’d be very grateful if you could help me figure out why Sen4Cap doesn’t want to use the already downloaded products.

Best regards

#2

Hello,

Could you please try to set the scope of the datasource to “Query and Download” not just “Query”?
Are the products imported after the restart of the sen4cap services?

Please let us know.

Best regards,
Cosmin

#3

Dear Cosmin,

Thank you for your reply!
I’ve changed the scope to “Query and download” and now the S2 products appear in the database, but with the incorrect path and a status_id = 3 with reason “Could not download or no tiles”. I suppose it tries to download the product from scihub.

Here an extract of the downloader_history table (it’s the same for all products):

id | site_id | satellite_id | product_name | full_path | created_timestamp | status_id | no_of_retries | product_date | orbit_id | orbit_type_id | status_reason | tiles | footprint

27512 | 8 | 1 | S2A_MSIL1C_20191229T172721_N0208_R012_T14SMJ_20191229T205120.SAFE | https://scihub.copernicus.eu/dhus/odata/v1/Products(‘b0e2fd0a-e033-4720-8c69-27c0f0ade6f2’)/$value
| 2020-09-04 09:03:44.331+00 | 3 | 2 | 2019-12-29 16:27:21+00 | 12 | 2 | Could not download or no tiles | {14SMJ} | 0103000020E61000000100000005000000E8A4F78DAF0
959C01427CC30AA6043407233DC80CFB858C0F1E2D73C62614340F168E388B5B858C0545A91F206E0434006F52D73BA0A59C0348B8B5748DF4340E8A4F78DAF0959C01427CC30AA604340
27511 | 8 | 1 | S2B_MSIL1C_20191227T173729_N0208_R055_T14SMJ_20191227T193002.SAFE | https://scihub.copernicus.eu/dhus/odata/v1/Products(‘0bf89f2e-e906-4cc1-9c2d-e6f7088b3708’)/$value
| 2020-09-04 09:03:44.223+00 | 3 | 2 | 2019-12-27 16:37:29+00 | 55 | 2 | Could not download or no tiles | {14SMJ} | 0103000020E6100000010000000C000000E8A4F78DAF0
959C01427CC30AA60434033E197FA79E758C0108BBD09F860434075029A081BE658C07252B674E668434050FC1873D7E258C02B9516ACA77B4340742497FF90DF58C054FB8A35688E43400551F70148DC58C0DF878BDB28A14340C2FA3F87F9D858C0FCC
27512E6B34340E8A4F78DAFD558C072BFDD59A3C64340A298BC0166D258C0F83030755ED94340C5C9FD0E45D158C0A1DC90DECDDF434006F52D73BA0A59C0348B8B5748DF4340E8A4F78DAF0959C01427CC30AA604340

Best regards

#4

Hello,

Normally, if you set the “Local root”, the services will try to only import the products from the local repository specified.
Could you please provide me the content of the datasources table:

psql -U admin sen4cap -c “select * from datasource”

Also, could you please provide the full log of the sen4cap services during import?

sudo journalctl -fu sen2agri-services --since “3 days ago”

(the since should be adapted according to your system)

Best regards,
Cosmin

#5

Dear Cosmin,

The content of the datasource table is the following (I have masked usernames and passwords):

id | satellite_id | name | scope | username | passwrd | fetch_mode | max_retries | retry_interval_minutes | download_path | specific_params | max_connections | local_root | enabled | site_id | secondary_datasource_id

3 | 1 | Scientific Data Hub | 3 | ****** | ****** | 5 | 1440 | 60 | /mnt/archive/dwn_def/s2/default | {“p
arameters”:[{“name”:“platformName”,“type”:“java.lang.String”,“value”:“Sentinel-2”},{“name”:“productType”,“type”:“java.lang.String”,“value”:“S2MSI1C”}]} | 2 | /mnt/archive/store | t |
|
1 | 2 | USGS | 3 | | | 1 | 72 | 60 | /mnt/archive/dwn_def/l8/default | {“p
arameters”:[{“name”:“platformName”,“type”:“java.lang.String”,“value”:“LANDSAT_8_C1”},{“name”:“minClouds”,“type”:“java.lang.Integer”,“value”:“0”},{“name”:“maxClouds”,“type”
:“java.lang.Integer”,“value”:“100”},{“name”:“productType”,“type”:“java.lang.String”,“value”:“L1TP”}]} | 1 | | f |
|
2 | 3 | Scientific Data Hub | 3 | ******* | ******* | 2 | 1440 | 60 | /mnt/archive/dwn_def/s1/default/ | {“p
arameters”:[{“name”:“platformName”,“type”:“java.lang.String”,“value”:“Sentinel-1”},{“name”:“productType”,“type”:“java.lang.String”,“value”:“SLC”},{“name”:“polarisation”,“t
ype”:“java.lang.String”,“value”:“VV+VH”},{“name”:“sensorOperationalMode”,“type”:“java.lang.String”,“value”:“IW”}]} | 2 | | t |
|

I changed the fetch mode to “Direct link to product” and it has just changed the status_reason to “Product XXXXXX not found in the local archive”.

I’ve attached the log of the sen2agri-services but not since 3 days ago, otherwise the file is too large to upload here.
sen2agri-services_1.log (1.5 MB)

I’ve noticed this

[Scientific Data Hub-S2-2] r.c.t.d.DefaultProductPathBuilder - Looking for product S2A_MSIL1C_20190516T172901_N0207_R055_T14SMJ_20190516T211022 into /mnt/archive/store/2019/05/136/S2A_MSIL1C_20190516T172901_N0207_R055_T14SMJ_20190516T211022.SAFE

It looks like the system is looking for the product acquisition day in the day of the year format. Should I change the service properties configuration file?

Thanks for your help!

#6

Hello again,

In the services properties configuration file, I changed the Sentinel2 local archive path format to YYYY/MM/dd (instead of YYYY/MM/DD) and this solved my issue!
The S2 L1C product paths are now stored correctly in the database and processed by MAJA.

I think the procedure given in the SUM should be updated because it gives misleading information.

Thank you again for your help!

#7

Hello,

Indeed, it seems that there is an issue in the SUM as DD is the doy and not the the day of the month. We did not encountered this issue so far as in the services.properties we always used dd.
Sorry for this inconvenience, we will update the SUM in the next version.
In any case, to avoid also an issue with the products in the last week of the year, you should use actually the format yyyy/MM/dd instead YYYY/MM/dd.

Best regards,
Cosmin

#8

Dear Cosmin,

Thank you, this is really helpful.

Best regards

#9

Dear all,

I have a similar issue here:

I use a local repository containing Sentinel-2 data in zip-archives.
The problem I have is that my directory-structure slightly differs from the proposed one: Sentinel-2A and 2B are in separate directories:
grafik

So I thought using a local repository like this could be a solution:

###Local repository
SciHubDataSource.Sentinel2.local_archive_path=/eodc/products/copernicus.eu
SciHubDataSource.Sentinel2.fetch_mode=4
SciHubDataSource.Sentinel2.local.archive.path.format = s2a_prd_msil1c/yyyy/MM/dd
SciHubDataSource.Sentinel2.auto.uncompress=true
SciHubDataSource.Sentinel2.path.suffix = none
SciHubDataSource.Sentinel2.product.format = zip
SciHubDataSource.Sentinel2.enabled=true
#SciHubDataSource.Sentinel2.scope=3

But the subfolder /s2a_prd_msil1c is not detected if I add it to the the local.archive.path.format before yyyy/MM/dd as it is suggested in the SUM in Section 4.2.1.2

I’m also not sure about the zip format. Is it possible to use symbolic links together with zipped datasets?

Do you think it’s possible to ingest the data like it is stored on our server?
Best M

#10

Hello,

The multiple directories as local root for the datasource is not supported in this moment (and I assume also not in the future) as there can be workarounds for this.
One workaround would be to create a script that creates in a whatever directory, let’s say /mnt/archive/repo with the subdirectories structure like yyyy/MM/dd in which you create symlinks (in the dd subdirectories leafs) to your S2A and S2B products in the /eodc/products/copernicus.eu/s2*_prd_msil1c/yyyy/MM/dd source directoris. This way, you will have a single root repository for all S2A and S2B products without the need of duplicating data.
Concerning zip format, yes, it is supported to ingest .zip products (of course if their internal structure is the correct one). But, it is true that we used it mostly when downloading products but not sure we tested when importing them from local root. Nevertheless, it should work also in this case and if not, you should follow the log files to see if you notice any relevant message.

Hope this helps.

Best regards,
Cosmin

1 Like
#11

Dear Cosmin,

finally the maja processor started processing:) the zip-files are supported by the local root. Using symlinks requires a “.SAFE” as file-extension of the symlink pointing to the zip-archive.

Best Martin