I have a historical season of S2 L1C products already downloaded in a local folder, that I would like to use as a local product repository for Sen4CAP.
I have followed the procedure detailled in the Software User Manuel ($4.2.1.2) to change the data source.
The structure of the folders is the default one (S2 L1C unzipped .SAFE folders organized as YYYY/MM/DD/product.SAFE). See screenshot below:
However, after uploading my new site, the system doesn’t include the corresponding S2 products in the database and thus doesn’t process them. It only includes the S1 products which data source is Scihub.
Here is a screenshot of the S2 data source configuration in the web interface:
And an extract from the /usr/share/sen2agri/sen2agri-services/config/services.properties configuration file (I’ve added the new S2 settings after ##Local repository):
server.port = 8080
spring.datasource.url=jdbc:postgresql://localhost:5432/sen4cap?stringtype=unspecified
spring.datasource.username=admin
spring.datasource.password=sen2agri
plugins.use.docker = false
# GDAL docker container if plugins.use.docker=true. Uncomment and change if different than geodata/gdal (default)
# docker.gdal.image = geodata/gdal
##################################################################
## Datatabase configuration polling (in minutes). 0 means disabled
database.config.polling=0
SciHubDataSource.Sentinel1.auto.uncompress=false
## IMPORTANT NOTE: if value for key "downloader.use.esa.l2a" is set to true in the config table, this value should be set on true
SciHubDataSource.Sentinel2.auto.uncompress=false
## What to do if, in local archive mode, a product is not found
## Possible values are: none|download
SciHubDataSource.Sentinel2.local.missing.action=download
###Local repository
SciHubDataSource.Sentinel2.local_archive_path=/mnt/archive/store
SciHubDataSource.Sentinel2.fetch_mode=4
SciHubDataSource.Sentinel2.local.archive.path.format = YYYY/MM/DD
SciHubDataSource.Sentinel2.path.suffix = .SAFE
SciHubDataSource.Sentinel2.product.format = folder
SciHubDataSource.Sentinel2.enabled=true
#SciHubDataSource.Sentinel2.scope=3
I’d be very grateful if you could help me figure out why Sen4Cap doesn’t want to use the already downloaded products.
Could you please try to set the scope of the datasource to “Query and Download” not just “Query”?
Are the products imported after the restart of the sen4cap services?
Thank you for your reply!
I’ve changed the scope to “Query and download” and now the S2 products appear in the database, but with the incorrect path and a status_id = 3 with reason “Could not download or no tiles”. I suppose it tries to download the product from scihub.
Here an extract of the downloader_history table (it’s the same for all products):
Normally, if you set the “Local root”, the services will try to only import the products from the local repository specified.
Could you please provide me the content of the datasources table:
psql -U admin sen4cap -c “select * from datasource”
Also, could you please provide the full log of the sen4cap services during import?
sudo journalctl -fu sen2agri-services --since “3 days ago”
(the since should be adapted according to your system)
I changed the fetch mode to “Direct link to product” and it has just changed the status_reason to “Product XXXXXX not found in the local archive”.
I’ve attached the log of the sen2agri-services but not since 3 days ago, otherwise the file is too large to upload here. sen2agri-services_1.log (1.5 MB)
I’ve noticed this
[Scientific Data Hub-S2-2] r.c.t.d.DefaultProductPathBuilder - Looking for product S2A_MSIL1C_20190516T172901_N0207_R055_T14SMJ_20190516T211022 into /mnt/archive/store/2019/05/136/S2A_MSIL1C_20190516T172901_N0207_R055_T14SMJ_20190516T211022.SAFE
It looks like the system is looking for the product acquisition day in the day of the year format. Should I change the service properties configuration file?
In the services properties configuration file, I changed the Sentinel2 local archive path format to YYYY/MM/dd (instead of YYYY/MM/DD) and this solved my issue!
The S2 L1C product paths are now stored correctly in the database and processed by MAJA.
I think the procedure given in the SUM should be updated because it gives misleading information.
Indeed, it seems that there is an issue in the SUM as DD is the doy and not the the day of the month. We did not encountered this issue so far as in the services.properties we always used dd.
Sorry for this inconvenience, we will update the SUM in the next version.
In any case, to avoid also an issue with the products in the last week of the year, you should use actually the format yyyy/MM/dd instead YYYY/MM/dd.
I use a local repository containing Sentinel-2 data in zip-archives.
The problem I have is that my directory-structure slightly differs from the proposed one: Sentinel-2A and 2B are in separate directories:
So I thought using a local repository like this could be a solution:
But the subfolder /s2a_prd_msil1c is not detected if I add it to the the local.archive.path.format before yyyy/MM/dd as it is suggested in the SUM in Section 4.2.1.2
I’m also not sure about the zip format. Is it possible to use symbolic links together with zipped datasets?
Do you think it’s possible to ingest the data like it is stored on our server?
Best M
The multiple directories as local root for the datasource is not supported in this moment (and I assume also not in the future) as there can be workarounds for this.
One workaround would be to create a script that creates in a whatever directory, let’s say /mnt/archive/repo with the subdirectories structure like yyyy/MM/dd in which you create symlinks (in the dd subdirectories leafs) to your S2A and S2B products in the /eodc/products/copernicus.eu/s2*_prd_msil1c/yyyy/MM/dd source directoris. This way, you will have a single root repository for all S2A and S2B products without the need of duplicating data.
Concerning zip format, yes, it is supported to ingest .zip products (of course if their internal structure is the correct one). But, it is true that we used it mostly when downloading products but not sure we tested when importing them from local root. Nevertheless, it should work also in this case and if not, you should follow the log files to see if you notice any relevant message.
finally the maja processor started processing:) the zip-files are supported by the local root. Using symlinks requires a “.SAFE” as file-extension of the symlink pointing to the zip-archive.
how did you get the local archiive with S2 in zip folders to work?
I get the following log error:
The object (directory or file) /mnt/archive/dwn_def/s2/default/nw_nord/S2B_MSIL1C_20191104T104119_N0208_R008_T32UMC_20191104T124040.zip.SAFE/S2B_MSIL1C_20191104T104119_N0208_R008_T32UMC_20191104T124040.SAFE is not an archive: [Errno 20] Not a directory: ‘/mnt/archive/dwn_def/s2/default/nw_nord/S2B_MSIL1C_20191104T104119_N0208_R008_T32UMC_20191104T124040.zip.SAFE/S2B_MSIL1C_20191104T104119_N0208_R008_T32UMC_20191104T124040.SAFE’ !
in my service.properties if have the following setting:
my service properties file looks like this: my you need to add the correct path suffix:
`
###Local repository
SciHubDataSource.Sentinel2.local_archive_path=/mnt/archive/store
SciHubDataSource.Sentinel2.fetch_mode=4
SciHubDataSource.Sentinel2.local.archive.path.format = yyyy/MM/dd
SciHubDataSource.Sentinel2.path.suffix = .SAFE
my problem was that our archive with all the S2 zip-files is somewhere centralised and has the wrong structure. So I created symlinks of the zip-files pointing to my local storage having the right structure of yyyy/MM/dd. to make MAJA able to successfully read the symlinks which are pointing to the zip-files I had to change their extension to .SAFE. Not sure why this was necessary but then in the end maja started processing the files.
seems like an “error”. Changing the file-extension does not seem right. I got it working with the fetch mode “copy”, although the symlink option is definitely more useful.
I have similar issue and we have our local repository and its folder structure is same as sen4cap needed but Sen4Cap is not able to get it. Here is the log I got when I debug. How can I solve it out?
Are you using Creodias or AWS? If you are using AWS, you might need to use the SciHub datasource for query only and the AWS datasource for download only, using a local root and the Fetch mode to symlink or copy.
You can configure that directly in the web interface, for each datasource.
Thanks a lot for your advice as usual. Yes, It’s AWS repository. I have configured as below and status is changed to “In Progress” though but it is still not processing L2A. Is there anything I am missing?
Best Regards,
Henry
Here is folder structure and product info from AWS archive.
