Use of local product repository as S2 L1C datasource

Dear Sen4CAP team,

I have a historical season of S2 L1C products already downloaded in a local folder, that I would like to use as a local product repository for Sen4CAP.
I have followed the procedure detailled in the Software User Manuel ($ to change the data source.
The structure of the folders is the default one (S2 L1C unzipped .SAFE folders organized as YYYY/MM/DD/product.SAFE). See screenshot below:

However, after uploading my new site, the system doesn’t include the corresponding S2 products in the database and thus doesn’t process them. It only includes the S1 products which data source is Scihub.

Here is a screenshot of the S2 data source configuration in the web interface:

And an extract from the /usr/share/sen2agri/sen2agri-services/config/ configuration file (I’ve added the new S2 settings after ##Local repository):

server.port = 8080

plugins.use.docker = false
# GDAL docker container if plugins.use.docker=true. Uncomment and change if different than geodata/gdal (default)
# docker.gdal.image = geodata/gdal

## Datatabase configuration polling (in minutes). 0 means disabled
## IMPORTANT NOTE: if value for key "downloader.use.esa.l2a" is set to true in the config table, this value should be set on true
## What to do if, in local archive mode, a product is not found
## Possible values are: none|download

###Local repository
SciHubDataSource.Sentinel2.local.archive.path.format = YYYY/MM/DD
SciHubDataSource.Sentinel2.path.suffix = .SAFE
SciHubDataSource.Sentinel2.product.format = folder

I’d be very grateful if you could help me figure out why Sen4Cap doesn’t want to use the already downloaded products.

Best regards


Could you please try to set the scope of the datasource to “Query and Download” not just “Query”?
Are the products imported after the restart of the sen4cap services?

Please let us know.

Best regards,

Dear Cosmin,

Thank you for your reply!
I’ve changed the scope to “Query and download” and now the S2 products appear in the database, but with the incorrect path and a status_id = 3 with reason “Could not download or no tiles”. I suppose it tries to download the product from scihub.

Here an extract of the downloader_history table (it’s the same for all products):

id | site_id | satellite_id | product_name | full_path | created_timestamp | status_id | no_of_retries | product_date | orbit_id | orbit_type_id | status_reason | tiles | footprint

27512 | 8 | 1 | S2A_MSIL1C_20191229T172721_N0208_R012_T14SMJ_20191229T205120.SAFE |‘b0e2fd0a-e033-4720-8c69-27c0f0ade6f2’)/$value
| 2020-09-04 09:03:44.331+00 | 3 | 2 | 2019-12-29 16:27:21+00 | 12 | 2 | Could not download or no tiles | {14SMJ} | 0103000020E61000000100000005000000E8A4F78DAF0
27511 | 8 | 1 | S2B_MSIL1C_20191227T173729_N0208_R055_T14SMJ_20191227T193002.SAFE |‘0bf89f2e-e906-4cc1-9c2d-e6f7088b3708’)/$value
| 2020-09-04 09:03:44.223+00 | 3 | 2 | 2019-12-27 16:37:29+00 | 55 | 2 | Could not download or no tiles | {14SMJ} | 0103000020E6100000010000000C000000E8A4F78DAF0

Best regards


Normally, if you set the “Local root”, the services will try to only import the products from the local repository specified.
Could you please provide me the content of the datasources table:

psql -U admin sen4cap -c “select * from datasource”

Also, could you please provide the full log of the sen4cap services during import?

sudo journalctl -fu sen2agri-services --since “3 days ago”

(the since should be adapted according to your system)

Best regards,

Dear Cosmin,

The content of the datasource table is the following (I have masked usernames and passwords):

id | satellite_id | name | scope | username | passwrd | fetch_mode | max_retries | retry_interval_minutes | download_path | specific_params | max_connections | local_root | enabled | site_id | secondary_datasource_id

3 | 1 | Scientific Data Hub | 3 | ****** | ****** | 5 | 1440 | 60 | /mnt/archive/dwn_def/s2/default | {“p
arameters”:[{“name”:“platformName”,“type”:“java.lang.String”,“value”:“Sentinel-2”},{“name”:“productType”,“type”:“java.lang.String”,“value”:“S2MSI1C”}]} | 2 | /mnt/archive/store | t |
1 | 2 | USGS | 3 | | | 1 | 72 | 60 | /mnt/archive/dwn_def/l8/default | {“p
:“java.lang.Integer”,“value”:“100”},{“name”:“productType”,“type”:“java.lang.String”,“value”:“L1TP”}]} | 1 | | f |
2 | 3 | Scientific Data Hub | 3 | ******* | ******* | 2 | 1440 | 60 | /mnt/archive/dwn_def/s1/default/ | {“p
ype”:“java.lang.String”,“value”:“VV+VH”},{“name”:“sensorOperationalMode”,“type”:“java.lang.String”,“value”:“IW”}]} | 2 | | t |

I changed the fetch mode to “Direct link to product” and it has just changed the status_reason to “Product XXXXXX not found in the local archive”.

I’ve attached the log of the sen2agri-services but not since 3 days ago, otherwise the file is too large to upload here.
sen2agri-services_1.log (1.5 MB)

I’ve noticed this

[Scientific Data Hub-S2-2] r.c.t.d.DefaultProductPathBuilder - Looking for product S2A_MSIL1C_20190516T172901_N0207_R055_T14SMJ_20190516T211022 into /mnt/archive/store/2019/05/136/S2A_MSIL1C_20190516T172901_N0207_R055_T14SMJ_20190516T211022.SAFE

It looks like the system is looking for the product acquisition day in the day of the year format. Should I change the service properties configuration file?

Thanks for your help!

Hello again,

In the services properties configuration file, I changed the Sentinel2 local archive path format to YYYY/MM/dd (instead of YYYY/MM/DD) and this solved my issue!
The S2 L1C product paths are now stored correctly in the database and processed by MAJA.

I think the procedure given in the SUM should be updated because it gives misleading information.

Thank you again for your help!


Indeed, it seems that there is an issue in the SUM as DD is the doy and not the the day of the month. We did not encountered this issue so far as in the we always used dd.
Sorry for this inconvenience, we will update the SUM in the next version.
In any case, to avoid also an issue with the products in the last week of the year, you should use actually the format yyyy/MM/dd instead YYYY/MM/dd.

Best regards,

Dear Cosmin,

Thank you, this is really helpful.

Best regards

Dear all,

I have a similar issue here:

I use a local repository containing Sentinel-2 data in zip-archives.
The problem I have is that my directory-structure slightly differs from the proposed one: Sentinel-2A and 2B are in separate directories:

So I thought using a local repository like this could be a solution:

###Local repository
SciHubDataSource.Sentinel2.local.archive.path.format = s2a_prd_msil1c/yyyy/MM/dd
SciHubDataSource.Sentinel2.path.suffix = none
SciHubDataSource.Sentinel2.product.format = zip

But the subfolder /s2a_prd_msil1c is not detected if I add it to the the local.archive.path.format before yyyy/MM/dd as it is suggested in the SUM in Section

I’m also not sure about the zip format. Is it possible to use symbolic links together with zipped datasets?

Do you think it’s possible to ingest the data like it is stored on our server?
Best M


The multiple directories as local root for the datasource is not supported in this moment (and I assume also not in the future) as there can be workarounds for this.
One workaround would be to create a script that creates in a whatever directory, let’s say /mnt/archive/repo with the subdirectories structure like yyyy/MM/dd in which you create symlinks (in the dd subdirectories leafs) to your S2A and S2B products in the /eodc/products/*_prd_msil1c/yyyy/MM/dd source directoris. This way, you will have a single root repository for all S2A and S2B products without the need of duplicating data.
Concerning zip format, yes, it is supported to ingest .zip products (of course if their internal structure is the correct one). But, it is true that we used it mostly when downloading products but not sure we tested when importing them from local root. Nevertheless, it should work also in this case and if not, you should follow the log files to see if you notice any relevant message.

Hope this helps.

Best regards,

1 Like

Dear Cosmin,

finally the maja processor started processing:) the zip-files are supported by the local root. Using symlinks requires a “.SAFE” as file-extension of the symlink pointing to the zip-archive.

Best Martin

Dear maatin,

how did you get the local archiive with S2 in zip folders to work?

I get the following log error:

The object (directory or file) /mnt/archive/dwn_def/s2/default/nw_nord/ is not an archive: [Errno 20] Not a directory: ‘/mnt/archive/dwn_def/s2/default/nw_nord/’ !

in my if have the following setting:

##Local data repository
SciHubDataSource.Sentinel1.local.archive.path.format = yyyy/MM/dd
SciHubDataSource.Sentinel2.path.suffix = none
SciHubDataSource.Sentinel2.product.format = .SAFE

I also tried “SciHubDataSource.Sentinel2.product.format = zip”, but that did not work either


Dear obu,

my service properties file looks like this: my you need to add the correct path suffix:
###Local repository
SciHubDataSource.Sentinel2.local.archive.path.format = yyyy/MM/dd
SciHubDataSource.Sentinel2.path.suffix = .SAFE

my problem was that our archive with all the S2 zip-files is somewhere centralised and has the wrong structure. So I created symlinks of the zip-files pointing to my local storage having the right structure of yyyy/MM/dd. to make MAJA able to successfully read the symlinks which are pointing to the zip-files I had to change their extension to .SAFE. Not sure why this was necessary but then in the end maja started processing the files.

Hope this helps a bit.
best M


seems like an “error”. Changing the file-extension does not seem right. I got it working with the fetch mode “copy”, although the symlink option is definitely more useful.


Dear @cudroiu and Sen4Cap,

I have similar issue and we have our local repository and its folder structure is same as sen4cap needed but Sen4Cap is not able to get it. Here is the log I got when I debug. How can I solve it out?

Here is our data source.

Here is configuration in

Local repository


Here is log of sen2agri-services

Here is log from downloader_history table.

Here is log from failed_queries
[root@hadoop-srv-05 failed_queries]# cat 1_Sentinel2_20181101000000_20200131235959.json
{“user”:null,“password”:null,“id”:“a6449d7e-edcd-44a3-ae4c-ae5a2b66f88c”,“values”:{“footprint”:“POLYGON((100.419 14.08,100.422 14.086,100.425 14.082,100.423 14.078,100.419 14.08))”,“endDate”:[“2018-11-01 00:00:00”,“2020-01-31 23:59:59”],“tileId”:“47PPR”,“platformName”:“Sentinel-2”,“startDate”:[“2018-11-01 00:00:00”,“2020-01-31 23:59:59”],“productType”:“S2MSI1C”}}[root@havi /usr/share/sen2agri/sen2agri-services/config/

Thanks and Regards,

Dear Henry,

Are you using Creodias or AWS? If you are using AWS, you might need to use the SciHub datasource for query only and the AWS datasource for download only, using a local root and the Fetch mode to symlink or copy.
You can configure that directly in the web interface, for each datasource.

Hope this helps.

Best regards,

Dear @cudroiu,

Thanks a lot for your advice as usual. Yes, It’s AWS repository. I have configured as below and status is changed to “In Progress” though but it is still not processing L2A. Is there anything I am missing?

Best Regards,

Here is folder structure and product info from AWS archive.