Dear all,
I am having a problem for processing Sentinel-2 using Maja.I am trying to use the system in automatic mode. The process has started and processed 30 scenes out of 500 and it stopped.
I would be glad if someone could guide me to start the S2 processing again in automatic mode Any help will be highly appreciated. Thank you.
Hello,
Could you please provide the logs for the sen2agri-demmaccs service?
sudo journalctl -fu sen2agri-demmaccs --since “2 days ago”
“2 days ago” is just an example, it would be better to be “x days ago”, where x is the number of days since you created the site.
Best regards,
Cosmin
Dear Cosmin,
Thank ou for your reply. I followed your instrauction. Unfortuantely, the logs is very long and as a new user I cannot upload files. Therefore, I am copying have part of the logs here.
Sep 16 10:34:25 Lenovo2.cp demmaccs_launcher.py[126985]: dmworker_1 will consume: [[(13, 1, 108, ‘32UPD’, 13739, ‘/mnt/data/dwn_def/s2/default/st/S2A_MSIL1C_20200105T103421_N0208_R108_T32UPD_20200105T111147.zip’, None)]] | l2a | /mnt/archive/maccs_def/{site}/l2a/
Sep 16 10:34:25 Lenovo2.cp demmaccs_launcher.py[126985]: 2020-09-16 10:34:25.582929:[126985]:dmworker_1: Starting extract_from_archive_if_needed for tile 32UPD
Sep 16 10:34:25 Lenovo2.cp demmaccs_launcher.py[126985]: ARCHIVES DIRECTORY = /mnt/archive/demmaccs_tmp/archives/tmpS8fMHn
Sep 16 10:34:25 Lenovo2.cp demmaccs_launcher.py[126985]: is_tarfile: The object (directory or file) /mnt/data/dwn_def/s2/default/st/S2A_MSIL1C_20200105T103421_N0208_R108_T32UPD_20200105T111147.zip is not an archive: [Errno 2] No such file or directory: ‘/mnt/data/dwn_def/s2/default/st/S2A_MSIL1C_20200105T103421_N0208_R108_T32UPD_20200105T111147.zip’ !
Sep 16 10:34:25 Lenovo2.cp demmaccs_launcher.py[126985]: This wasn’t an archive, so continue as is. Deleting /mnt/archive/demmaccs_tmp/archives/tmpS8fMHn
Sep 16 10:34:25 Lenovo2.cp demmaccs_launcher.py[126985]: 2020-09-16 10:34:25.583400:[126985]:dmworker_1: Ended extract_from_archive_if_needed for tile 32UPD
Sep 16 10:34:25 Lenovo2.cp demmaccs_launcher.py[126985]: dmworker_1: site_id = 13
Sep 16 10:34:25 Lenovo2.cp demmaccs_launcher.py[126985]: dmworker_1: satellite_id = 1
Sep 16 10:34:25 Lenovo2.cp demmaccs_launcher.py[126985]: dmworker_1: orbit_id = 108
Sep 16 10:34:25 Lenovo2.cp demmaccs_launcher.py[126985]: dmworker_1: tile_id = 32UPD
Sep 16 10:34:25 Lenovo2.cp demmaccs_launcher.py[126985]: dmworker_1: dh_id = 13739
Sep 16 10:34:25 Lenovo2.cp demmaccs_launcher.py[126985]: dmworker_1: path = /mnt/data/dwn_def/s2/default/st/S2A_MSIL1C_20200105T103421_N0208_R108_T32UPD_20200105T111147.zip
Sep 16 10:34:25 Lenovo2.cp demmaccs_launcher.py[126985]: dmworker_1: prev_path = None
Sep 16 10:34:25 Lenovo2.cp demmaccs_launcher.py[126985]: Found key s2.enabled for all sites with value true
Sep 16 10:34:25 Lenovo2.cp demmaccs_launcher.py[126985]: ###################################
Sep 16 10:34:25 Lenovo2.cp demmaccs_launcher.py[126985]: Cannot check if product root dir path /mnt/data/dwn_def/s2/default/st/S2A_MSIL1C_20200105T103421_N0208_R108_T32UPD_20200105T111147.zip exists or it is a valid symlink. Error was: 2
Sep 16 10:34:25 Lenovo2.cp demmaccs_launcher.py[126985]: ###################################
Sep 16 10:34:25 Lenovo2.cp demmaccs_launcher.py[126985]: 2020-09-16 10:34:25.609281:[126985]:dmworker_1: The product /mnt/data/dwn_def/s2/default/st/S2A_MSIL1C_20200105T103421_N0208_R108_T32UPD_20200105T111147.zip is not valid or temporary unavailable…
Sep 16 10:34:26 Lenovo2.cp demmaccs_launcher.py[126985]: Demmaccs main thread: feeding the queue for workers with: [[(13, 1, 108, ‘32UQD’, 13742, ‘/mnt/data/dwn_def/s2/default/st/S2A_MSIL1C_20200105T103421_N0208_R108_T32UQD_20200105T111147.zip’, None)]]
Sep 16 10:34:26 Lenovo2.cp demmaccs_launcher.py[126985]: dmworker_0 will consume: [[(13, 1, 108, ‘32UQD’, 13742, ‘/mnt/data/dwn_def/s2/default/st/S2A_MSIL1C_20200105T103421_N0208_R108_T32UQD_20200105T111147.zip’, None)]] | l2a | /mnt/archive/maccs_def/{site}/l2a/
Sep 16 10:34:26 Lenovo2.cp demmaccs_launcher.py[126985]: 2020-09-16 10:34:26.604477:[126985]:dmworker_0: Starting extract_from_archive_if_needed for tile 32UQD
Sep 16 10:34:26 Lenovo2.cp demmaccs_launcher.py[126985]: ARCHIVES DIRECTORY = /mnt/archive/demmaccs_tmp/archives/tmpKbwGA6
Sep 16 10:34:26 Lenovo2.cp demmaccs_launcher.py[126985]: is_tarfile: The object (directory or file) /mnt/data/dwn_def/s2/default/st/S2A_MSIL1C_20200105T103421_N0208_R108_T32UQD_20200105T111147.zip is not an archive: [Errno 2] No such file or directory: ‘/mnt/data/dwn_def/s2/default/st/S2A_MSIL1C_20200105T103421_N0208_R108_T32UQD_20200105T111147.zip’ !
Sep 16 10:34:26 Lenovo2.cp demmaccs_launcher.py[126985]: This wasn’t an archive, so continue as is. Deleting /mnt/archive/demmaccs_tmp/archives/tmpKbwGA6
Sep 16 10:34:26 Lenovo2.cp demmaccs_launcher.py[126985]: 2020-09-16 10:34:26.604912:[126985]:dmworker_0: Ended extract_from_archive_if_needed for tile 32UQD
Sep 16 10:34:26 Lenovo2.cp demmaccs_launcher.py[126985]: dmworker_0: site_id = 13
Sep 16 10:34:26 Lenovo2.cp demmaccs_launcher.py[126985]: dmworker_0: satellite_id = 1
Sep 16 10:34:26 Lenovo2.cp demmaccs_launcher.py[126985]: dmworker_0: orbit_id = 108
Sep 16 10:34:26 Lenovo2.cp demmaccs_launcher.py[126985]: dmworker_0: tile_id = 32UQD
Sep 16 10:34:26 Lenovo2.cp demmaccs_launcher.py[126985]: dmworker_0: dh_id = 13742
Sep 16 10:34:26 Lenovo2.cp demmaccs_launcher.py[126985]: dmworker_0: path = /mnt/data/dwn_def/s2/default/st/S2A_MSIL1C_20200105T103421_N0208_R108_T32UQD_20200105T111147.zip
Sep 16 10:34:26 Lenovo2.cp demmaccs_launcher.py[126985]: dmworker_0: prev_path = None
Sep 16 10:34:26 Lenovo2.cp demmaccs_launcher.py[126985]: Found key s2.enabled for all sites with value true
Sep 16 10:34:26 Lenovo2.cp demmaccs_launcher.py[126985]: ###################################
Sep 16 10:34:26 Lenovo2.cp demmaccs_launcher.py[126985]: Cannot check if product root dir path /mnt/data/dwn_def/s2/default/st/S2A_MSIL1C_20200105T103421_N0208_R108_T32UQD_20200105T111147.zip exists or it is a valid symlink. Error was: 2
Sep 16 10:34:26 Lenovo2.cp demmaccs_launcher.py[126985]: ###################################
Sep 16 10:34:26 Lenovo2.cp demmaccs_launcher.py[126985]: 2020-09-16 10:34:26.627065:[126985]:dmworker_0: The product /mnt/data/dwn_def/s2/default/st/S2A_MSIL1C_20200105T103421_N0208_R108_T32UQD_20200105T111147.zip is not valid or temporary unavailable…
Sep 16 10:34:27 Lenovo2.cp demmaccs_launcher.py[126985]: Waiting for queue to join…
Sep 16 10:34:27 Lenovo2.cp demmaccs_launcher.py[126985]: dmworker_1 will consume: None | l2a | /mnt/archive/maccs_def/{site}/l2a/dmworker_0 will consume: None | l2a | /mnt/archive/maccs_def/{site}/l2a/
Sep 16 10:34:27 Lenovo2.cp demmaccs_launcher.py[126985]: 2020-09-16 10:34:27.621719:[126985]:dmworker_0: No tile to process. Gracefully closing…
Sep 16 10:34:27 Lenovo2.cp demmaccs_launcher.py[126985]: 2020-09-16 10:34:27.621899:[126985]:dmworker_1: No tile to process. Gracefully closing…
Sep 16 10:34:27 Lenovo2.cp demmaccs_launcher.py[126985]: 2020-09-16 10:34:27.622341:[126985]:dmworker_0: Exit thread
Sep 16 10:34:27 Lenovo2.cp demmaccs_launcher.py[126985]: 2020-09-16 10:34:27.622485:[126985]:dmworker_1: Exit thread
Sep 16 10:34:27 Lenovo2.cp demmaccs_launcher.py[126985]: All the workers finished their job. Exiting…
Looking forward for your reply.
Hello,
From what I see in your log messages, the products seem to be no longer in their location.
Could you please check, for example, the product /mnt/data/dwn_def/s2/default/st/S2A_MSIL1C_20200105T103421_N0208_R108_T32UQD_20200105T111147.zip? Is this product still on disk or accessible? Do you have them on a network or removable drive?
Also, if it exists, is it a valid zip archive?
Best regards,
Cosmin
Hello Cosmin,
I checked it and unfortuantely it is not there. So my question would be: how can i download these scenes which were not downloaded during the process and is there any way that user can get the info about the number of scenes that has not been downloaded?
And after solving that issue, how can I start the atmospheric correction again?
Thank you in advance.
Hello,
Can you please provide me the following results in 2 files?
ls -l /mnt/data/dwn_def/s2/default/st/ > files_in_dir.txt
psql -U admin sen4cap -c “select * from downloader_history where full_path like ‘%/mnt/data/dwn_def/s2/default/st/%’” > dwn_hist.txt
If you are sure that there are no files in /mnt/data/dwn_def/s2/default/st/ you could remove all the products from downloader_history corresponding to that site like this:
psql -U admin sen4cap -c “delete from l1_tile_history where downloader_history_id in (select id from downloader_history where full_path like ‘%/mnt/data/dwn_def/s2/default/st/%’)”
psql -U admin sen4cap -c “delete from downloader_history where full_path like ‘%/mnt/data/dwn_def/s2/default/st/%’”
And then restart the sen4cap services with:
sudo systemctl restart sen2agri-services
Normally, the system will re-download all S2 products for that site and will automatically pre-process with MAJA for the atmospheric correction.
Best regards,
Cosmin
Dear cosmin,
Thank you for your reply. I will definately provide you with the 2 txt files but before that I just want to clarify some points.
I haven’t stored S2 products in /mnt/data/dwn_def/s2/default/st/ but they are in /mnt/data/s2/default/st/. Therefore, should I change path where data are stored in database? if yes, in which table of database.
In /mnt/data/dwn_def/s2/default/st/ there are products but the the product that you have asked me to check is not there. That means some of the products has not been downloaded. So, in this case if i remove all the products from downloader_history corresponding to that site like this:
psql -U admin sen4cap -c “delete from l1_tile_history where downloader_history_id in (select id from downloader_history where full_path like ‘%/mnt/data/s2/default/st/%’)”
psql -U admin sen4cap -c “delete from downloader_history where full_path like ‘%/mnt/data/s2/default/st/%’”
Will it skip already downloaded & processed scenes and download & process only the failed scenes?
Thanks
Bishwa
Dear Bishwa,
I assume that you changed the path in the S2 datasource from the website interface.
Could you please tell me if you have any products downloaded in the /mnt/data/s2/default/st/ directory?
You could try first switch back into the datasource the directory /mnt/data/s2/default/st/ to /mnt/data/dwn_def/s2/default/st/ and then execute the commands you specified.
Nevertheless, to figure out more exactly what is happening, it would be helpful if you could provide me the results of the commands (some of them are the ones already specified or slightly modified):
ls -l /mnt/data/dwn_def/s2/default/st/ > files_in_dir.txt
ls -l /mnt/data/s2/default/st/ > files_in_dir2.txt
psql -U admin sen4cap -c “select * from downloader_history” > dwn_hist.txt
psql -U admin sen4cap -c “select * from datasource” > datasource.txt
psql -U admin sen4cap -c “select * from config” > config.txt
Best regards,
Cosmin
Dear Cosmin,
Thankn you for your response. Yes i changed the datasource for S1 and S2 from web interface. Regarding your question, My all data are stored in /mnt/data/s2/default/st/ and I donot have the directory /mnt/data/dwn_def/s2/default/st. But when I was changing the datasource I had used it once and I again updated it. Regarding the results of commands I have prepared a zip file where you can find all the text files.
Looking forward for your response.
Bishwatext_files.zip (303.1 KB)
Dear Bishwa,
From what I see in the downloader_history table, you had initially (in 14th of September) the setting in the datasource with “dwn_def” and then you changed it, at some point, to have the path without dwn_def.
It looks that the system downloaded some of the products in the previous location (the one with dwn_def) and they were inserted in the database with this location.
Did you removed the directory /mnt/data/dwn_def (apparently you did as I cannot find the products downloaded in dwn_def into the new location)?
If you just removed dwn_def, you should remove the products from the downloader_history pointing to the invalid files using something like this:
psql -U admin sen4cap -c “delete from downloader_history where full_path like ‘/mnt/data/dwn_def%’”
psql -U admin sen4cap -c “update config set value = true where key like ‘downloader.%.forcestart’”
Then restart the sen4cap services"
sudo systemctl restart sen2agri-services
This will delete them from downloader history and will also instruct services to start querying and downloading from the beginning of the season.
The configuration that you sent seems OK (at a first glance) and coherent.
Please let me know.
Best regards,
Cosmin
Dear Cosmin,
Thank you for your response. I followed your instruction but I faced an error while running the command for removing the products from the downloader_history pointing to the invalid files:
The error is:
[root@lenovoPc]# psql -U admin sen4cap -c “delete from downloader_history where full_path like ‘/mnt/data/dwn_def%’” update config set value = true where key like ‘downloader.%.forcestart’
psql: warning: extra command-line argument “update” ignored
psql: warning: extra command-line argument “config” ignored
psql: warning: extra command-line argument “set” ignored
psql: warning: extra command-line argument “value” ignored
psql: warning: extra command-line argument “=” ignored
psql: warning: extra command-line argument “true” ignored
psql: warning: extra command-line argument “where” ignored
psql: warning: extra command-line argument “key” ignored
psql: warning: extra command-line argument “like” ignored
psql: warning: extra command-line argument “downloader.%.forcestart” ignored
ERROR: update or delete on table “downloader_history” violates foreign key constraint “l1_tile_history_downloader_history_id_fkey” on table “l1_tile_history”
DETAIL: Key (id)=(13727) is still referenced from table “l1_tile_history”.
Thanks.
Bishwa
Dear Bishwa,
These are actually 2 commands but it seems that the editor altered my answer (sorry about this, I did not payed attention to this). The distinct commands are:
psql -U admin sen4cap -c “delete from downloader_history where full_path like ‘/mnt/data/dwn_def%’”
psql -U admin sen4cap -c “update config set value = true where key like ‘downloader.%.forcestart’”
Then:
sudo systemctl restart sen2agri-services
Best regards,
Cosmin
Dear cosmin,
psql -U admin sen4cap -c “delete from downloader_history where full_path like ‘/mnt/data/dwn_def%’”
ERROR: update or delete on table “downloader_history” violates foreign key constraint “l1_tile_history_downloader_history_id_fkey” on table “l1_tile_history”
DETAIL: Key (id)=(13727) is still referenced from table “l1_tile_history”.
still the same error 
Dear Bishwa,
Before deleting from the downloader_history, you must also delete from l1_tile_history (sorry I forgot to specify that):
psql -U admin sen4cap -c “delete from l1_tile_history where downloader_history_id in (select id from downloader_history where full_path like ‘/mnt/data/dwn_def%’)”
psql -U admin sen4cap -c “delete from downloader_history where full_path like ‘/mnt/data/dwn_def%’”
psql -U admin sen4cap -c “update config set value = true where key like ‘downloader.%.forcestart’”
sudo systemctl restart sen2agri-services
Please let me know.
Best regards,
Cosmin
Dear Cosmin,
Thank you. will try it and will update you.
Regards
Bishwa
Dear Cosmin,
Thank you for your supoort. I am able to solve the previous issue by following the steps that you have mentioined. I am back again with another strange issue. The system has processed 30 S2 files and it could not process the other. For your reference I am attaching one of the log file.
demmaccs_32UPB.log (2.1 KB)
Any help will be highly appreciated.
Thank you.
Regards
Bishwa
Dear @cudroiu ,
As we are getting more raw files of sentinel 2 in our repository, we want to generate products. And I have tried below command and it didn’t work out.
psql -U admin sen4cap -c “update config set value = true where key like ‘downloader.%.forcestart’”
sudo systemctl restart sen2agri-services
Could you advise how it can be triggered again?
Thanks and Regards,
Henry
Dear Cosmin,
I am still waiting for your response. I am looking forward for your support to solve the issue that I have mentioned above. Thank you in advance.
Regards
Bswa
Dear Bishwa,
From the logs it seems very strange the invocation of the MAJA.
Could you please provide the outputs of the command :
ls -al /mnt/data/s2/default/st/
Are the files there zip files or are actually some folders whose names are ending with .zip?
Also, could you provide the logs from demmaccs from the last few days, where there are still messages trying to process these files. For example (5 days ago or more):
sudo journalctl -fu sen2agri-demmaccs --since “5 days ago”
Best regards,
Cosmin
Dear Cosmin,
Thank you for your reply. Regarding your question,it has not created any folder or zipped files. My all Sentinel-2 Level 1C files are store in the following folder: /mnt/data/s2/default/st/.
For any progress, I will keep you updated.
Thank you.
Regards
Bishwa