Processing of ESA archived data

#1

Dear all,

I would like to process a site with defined temporal extent 31.10.2019 - 30.09.2020.

It appears to me that S2 data is not processed for the whole period. Could it be connected or caused by the fact that ESA is not archiving S2 data for longer than a year?

The Download.service is logging something like " Cannot determine product size for S2B_MSIL1C_20191022T103059_N0208_R108_…" for all products before 19.11.2019, which seems the first date some L2A products have been produced by the sen4cap system.

I am working with a local storage and have the S2 files locally, but it seems, that due to the “response” from the ESA query, the further processing is not made.

Am I right, and if so, how could I trigger the processing of the locally stored L1C data?

Thanks
Oliver

#2

I can only guess that his may be caused by ESA marking products as “offline”.

The downloader will retry to fetch products in regular intervals, though by default I think they are big.

Since you have a local repository, a workaround could be the following:

Sen4CAP uses a postgres database to store state about the downloaded files and the processors. I don’t know all the details, but

  • look into the downloader_status table to see the different downloader statuses
  • look into the downloader_history table and find the a relevant product (S2B_MSIL1C_20191022T103059_N0208_R108_…)
  • check its status and path (and if necessary fill it in)
  • set the status to be a successful download

The tile should then be picked up by the l1-processor (maja/maccs), whose state is under the l1_tile_history table with its own statuses under l1_status.

PS: I wrote the above on top of my head. If you don’t see the tables, I may have misspelled them; the names should be fairly close.

#3

Your “work-around” helped.
I copied the relevant S2 scenes from my local archive to /mnt/archive/dwn_def/s1/default/SITE_NAME, changed the “full_path” in the table downloader_history to these locally stored zip files and set the status_id =2. Then restarted the sen2agri services and the l1 processor picked it up.

Thanks
Oliver

#4

It appears now that the products are processed by maja, but they do not show up in the web gui as available products.
However, if I check /mnt/archive/maccs_def/SITENAME/l2a/ the SAFE folders are there including processed scenes. e.g. S2A_MSIL2A_20190920T104021_N0208_R008_T32UMC_20190920T111121.SAFE
How do I get these products registered in the system?

#5

I see now what the situation is. My site season start is defined as 31.10.2019, therefore the 20190920 product is not shown in the site product list, but apparently processed by sen4cap. It seems that the season is internally prolonged. Does someone know for how long, why and if this can be changed?

#6

see the manual: http://esa-sen4cap.org/sites/default/files/Sen4CAP_System-Software-User-Manual_v1.3.pdf

  • The system starts to download / import optical products 2 months before the start of the season
    date. For example, considering a start of the season on 2018-05-01, products download will thus
    start on 2018-02-01. These 3 first months of data are needed to initialize the L2A processor (see
    section 4.4.2) but will not be used to produce the higher-level products;
  • Similarly, the system also tries to download products during 2 months after the end season (so
    considering an end of season of 2018-09-31, it means until 2018-10-31), to avoid losing any
    products inside the season which may be added after the end of season day

Sen4CAP Configurator
downloader.start.offset
2
Specifies for the S2/L8 products the interval before the start of the season where L1C/L1T need to be still downloaded for the MAJA training. To be set to 0 if L2A products are directly downloaded/fetched (no MAJA preprocessing).

From what I remember, the documentation is not correct - the right value seems to be 3, meaning that the extent is 3 months before the season date.