Inserting downloaded products into PSQL database

General Discussions
1

Hello @cudroiu , @lnicola and rest of the Sen4CAP team,

I have noticed a number of S1 SLC products that have been queried but not downloaded (which is due to them not being available on scihub, already opened a ticket with them).

This causes the S1 processor to constantly skip these jobs with errored out jobs in the GUI. Here’s an excerpt from the log:

Product S1A_IW_SLC__1SDV_20210519T154733_20210519T154802_037959_047AEC_CF52.SAFE (status DOWNLOADING) was chosen as intersecting product
S1B_IW_SLC__1SDV_20210525T154705_20210525T154733_027063_033BBB_EB9F.SAFE, but its path [https://scihub.copernicus.eu/dhus/odata/v1/Products(‘930cb017-e72a-4cd2-a96b-3b1d7140c6f9’)/$value] looks like it has not been downloaded. It will be skipped.

So in the meantime, I figured I would try downloading them from asf.alaska.edu and manually copying them over to the dwn_def folder and inserting them into the database.

I just wanted to verify if this workflow I made will be enough:

  1. Download products and transfer to relevant s1 download folder
  2. In the downloader_history table of psql db modify:
  • full_path so it matches the location of my product in the dwn_def folder
  • Set the status_id to 2 (downloaded)
  • restart sen2agri-services

As there seem to be quite a few of these failed downloads with status 1, it seems quite necessary at this point to develop an alternative data ingestion flow to just using apihub. Would appreciate any tips!

EDIT:
Just checked the downloader_history database and it turns out about 10% of products for this season have failed with an error from scihub/apihub with a status reason:

Cannot download https://scihub.copernicus.eu/dhus/odata/v1/Products(‘20f4d438-16bc-4127-9717-af152efea095’)/$value: Remote file length is zero!

To anyone reading this and using the system operationally this year, check the downloader table and see if the same hasn’t happened to you.

Best,
Harijs

2

Hello Harijsi,

We are doing some tests with the system. And we had the same problem as you. Hopefully we have an internal system which is downloading the images in a local server in real time.
We have followed the same steps as you for importing the images in the system. The processing work well.

Nevertheless, we did not had the error ‘Remote file length is zero’.

I hope you will have all the images you need for your monitoring.
Regards,
Diane

3

Hello Diane,

It seems that the “Remote file length is zero” problem comes from SciHub side, as they said - they migrated their system and actually lost a part of the data. I am quite surprised, to put it mildly.

I managed to solve the issue by exporting a list of images from the psql database where status_id was 3. Then I wrote a quick Python script for ONDA DIAS to download the list through their API. Afterwards, I just copied the images inside the dwn_def folder and generated a list of PSQL UPDATE commands to set status_id to 2. This approach seems to work well. However, it is a bit of a hassle to have to re-query every week in order to see if SciHub dropped the ball on yet another acquisition.

Best,
Harijs