Hello everyone,
originally we defined a season from 2020-01-01 until 2020-07-15. So we only have pre-processed products until 15th of July.
Now I want to extend that.
I changed the season end to 15th of November with the web GUI and restarted the preprocessing via console:
S1:
psql -U admin sen4cap -c "update downloader_history set status_id=2 where id in (select downloader_history_id from l1_tile_history where status_id=3 and cloud_coverage is null);"
output: UPDATE 0
S2:
psql -U admin sen4cap -c "update downloader_history set status_id=2 where satellite_id=3;"
output: UPDATE 542
Then I restarted the sen2agri services.
In the web GUI I get a list of many L2-S1 Pre-Processor jobs that say “Finished”. I doubt that they are executed correctly as there are no products added to the folder and there was actually no processing time.
Clicking on “output” only returns “No jobs to be displayed".
I also thought about an issue with the download, but we don´t use downloads at CREODIAS as far as I know.
Another idea is deleting the site and set up a new one but I was hesitating as I am not sure whether the already processed data is safe. Is it?
Can somebody tell us how to produce pre-processed products for the extended period?
Cheers Jonathan
Hello Jonathan,
I had to do the same for one site and the option that I took is to create a new season with the extended period; here it would have been to add a new season to the site from 16th of July until 15th of November. By doing this, you don’t have to do anything, except maybe only restart the sen2agri-services.
For the option that you took (modify the end of the season), I think that you should not have changed the status of the data in the downloader_history table. What you did (for S1 -> there is a mix between S1 and S2 in your post) is to change the status_id of the data to 2 (= downloaded -> to preprocess) even for the data that are already preprocessed. Because of that, the system relaunches the preprocessing of all the low-level products (even in the former period). Fortunately, the system checks if it has already preprocessed products ready for each low-level product but this is pointless and it can explain the many jobs that you see in the GUI. What you should have done is just to relaunch the sen2agri-services or maybe to force the system to redo the query on SciHub platform to look for the new low-level products to preprocess corresponding to the new period. This can be done by activating to “true” the “downloader.s2.forcestart” and “downloader.s1.forcestart” in the config table, and then restart the sen2agri-services.
To fix your case, you should not delete the site. You should:
- wait for the system to recover the preprocessing (-> change the status) for the data that are already preprocessed and begin the preprocessing of the new data corresponding to the new period.
- or, to disable the current season (if the preprocessing was done for the former period) and launch a new season corresponding to the extended period.
You can check the status of the preprocessing at any time with:
For S2:
select status_id,count(*) from downloader_history where satellite_id=1 and site_id=? group by status_id;
For S1:
select status_id,count(*) from downloader_history where satellite_id=3 and site_id=? group by status_id;
And the last preprocessed data can be seen via:
select name,site_id,inserted_timestamp from product where site_id=? order by inserted_timestamp desc;
I hope it helps, don’t hesitate if it is not clear.
Best regards,
Philippe