Site cannot be deleted

Hi,
I am still having a hard time getting SEN4CAP to work as intended.

In the meantime I got to the point where I successfully ran the L4B processor. When trying to start the L4C processors as well for the season, the system kept falling into a continuous loop of the S4C Marker Database PR1, writing the entire HDD full of copies of the same database.

Now I would like to clean up again to start a new attempt. I would like to cleanly eliminate the entire site, but keep the pre-processed S1 & S2 data.

When I try to do it via the GUI, I only get the “ERROR” feedback (see screenshot).

Using sudo systemctl status -l sen2agri-services.service I get the following error message among others:

Mär 14 18:49:18 sen4rlp.codede.internal start.sh[13714]: 2022-Mar-14 18:49:18.465 ERROR [http-nio-8080-exec-1] o.e.s.s.AuthenticationFilter   - Request processing failed; nested exception is org.springframework.dao.DataIntegrityViolationException: StatementCallback; SQL [delete from site_tiles where (site_id = 1); delete from config where (site_id = 1); delete from step where task_id in (select id from task where job_id in (select id from job where site_id = 1)); delete from step_resource_log where task_id in (select id from task where job_id in (select id from job where site_id = 1)); delete from task where job_id in (select id from job where site_id = 1); delete from config_job where job_id in (select id from job where site_id = 1); delete from job where (site_id = 1); delete from scheduled_task_status where task_id in (select id from scheduled_task where site_id = 1); delete from scheduled_task where (site_id = 1); delete from l1_tile_history where downloader_history_id in (select id from downloader_history where site_id = 1); delete from downloader_history where (site_id = 1); delete from downloader_count where (site_id = 1); delete from product where (site_id = 1); delete from season where (site_id = 1); delete from config where (site_id = 1); delete from site where (id = 1)]; ERROR: update or delete on table "product" violates foreign key constraint "fk_product" on table "product_stats"
Mär 14 18:49:18 sen4rlp.codede.internal start.sh[13714]: Detail: Key (id)=(438) is still referenced from table "product_stats".; nested exception is org.postgresql.util.PSQLException: ERROR: update or delete on table "product" violates foreign key constraint "fk_product" on table "product_stats"
Mär 14 18:49:18 sen4rlp.codede.internal start.sh[13714]: Detail: Key (id)=(438) is still referenced from table "product_stats".

Seems like the database constraints prevent the deletion? Can I safely empty the product_stats table manually without breaking the system elsewhere? What exactly is the purpose of the table?

Can anyone here help me further on what the specific background of the error is and how best to work around it without crashing the whole system? I would be very pleased and will be happy to provide further information quickly if needed!

With best thanks and many greetings,
jab_lp

Ok, in the meantime I looked a little deeper into the system. The HDD was specifically written full by the orchestrator_temp directory.

du -hc --max-depth=1 /mnt/archive/orchestrator_temp/s4c_l4a/
1,5M	/mnt/archive/orchestrator_temp/s4c_l4a/5788
139G	/mnt/archive/orchestrator_temp/s4c_l4a/5785
208G	/mnt/archive/orchestrator_temp/s4c_l4a/5787
20K	/mnt/archive/orchestrator_temp/s4c_l4a/6087
32K	/mnt/archive/orchestrator_temp/s4c_l4a/624
32K	/mnt/archive/orchestrator_temp/s4c_l4a/627
1,7M	/mnt/archive/orchestrator_temp/s4c_l4a/5789
20K	/mnt/archive/orchestrator_temp/s4c_l4a/6089
20K	/mnt/archive/orchestrator_temp/s4c_l4a/6088
32K	/mnt/archive/orchestrator_temp/s4c_l4a/626
175G	/mnt/archive/orchestrator_temp/s4c_l4a/5786
20K	/mnt/archive/orchestrator_temp/s4c_l4a/6090
136G	/mnt/archive/orchestrator_temp/s4c_l4a/6086
32K	/mnt/archive/orchestrator_temp/s4c_l4a/625
32K	/mnt/archive/orchestrator_temp/s4c_l4a/623
657G	/mnt/archive/orchestrator_temp/s4c_l4a/
657G	insgesamt

Especially the cached SAR TIFs take up most of the memory here:

du -hc --max-depth=2 /mnt/archive/orchestrator_temp/s4c_l4a/5787/
20M	/mnt/archive/orchestrator_temp/s4c_l4a/5787/55885-product-formatter
208G	/mnt/archive/orchestrator_temp/s4c_l4a/5787/55884-s4c-crop-type/sar
43M	/mnt/archive/orchestrator_temp/s4c_l4a/5787/55884-s4c-crop-type/sar-merged
20M	/mnt/archive/orchestrator_temp/s4c_l4a/5787/55884-s4c-crop-type/features
16M	/mnt/archive/orchestrator_temp/s4c_l4a/5787/55884-s4c-crop-type/optical
208G	/mnt/archive/orchestrator_temp/s4c_l4a/5787/55884-s4c-crop-type
4,0K	/mnt/archive/orchestrator_temp/s4c_l4a/5787/55886-export-product-launcher
1,3M	/mnt/archive/orchestrator_temp/s4c_l4a/5787/55883-s4c-l4a-extract-parcels
208G	/mnt/archive/orchestrator_temp/s4c_l4a/5787/
208G	insgesamt

The directories do not seem to be deleted, although in the config table of the database the key “executor.processor.s4c_l4a.keep_job_folders” is set to 0.

Am I misunderstanding something here or shouldn’t the directories actually be deleted here after the L4A processing is complete?

If so, does anyone have any idea why they are not automatically deleted and what is the best way for me to do it myself?

Can I simply delete the directories manually or do I have to additionally manipulate entries in the database or so elsewhere?

Hello,

Normally, the orchestrator_temp directories should be automatically removed upon successful completion of a job. In the particular case of L4A processor, this is happening, if flag “executor.processor.s4c_l4a.keep_job_folders” is set to 0, after the “export-product-launcher” step.
Did you had any errors during the L4A processing (especially in the last step)?
Anyway, if your L4A ended, it is safe to delete also the directory
/mnt/archive/orchestrator_temp/s4c_l4a/<YOUR_JOB_ID>.
Also, if you have no other L3X/L4X processors running, is also safe to delete what is in /mnt/archive/orchestrator_temp directory.

Best regards,
Cosmin

Hi @cudroiu ,
Thank you very much for your feedback!

In the meantime, a few folders under /mnt/archive/orchestrator_temp/s4c_l4a/ have indeed been deleted from the system. Without me currently having any jobs running. Is there something like a cleanup routine here that goes through at longer intervals?

Anyway. I still have 15 folders under /mnt/archive/orchestrator_temp/s4c_l4a/ that occupy a lot of storage space. Thanks to your feedback, I now know that I can delete them safely. To answer your question I have tried to research their processing status and error logs via the “task” and “step_resource_log” tables.

(Using the GUI to find a job ID is a bit cumbersome. Perhaps one could implement a search/filter possibility here at opportunity or make clearly more jobs displayable on a side? But that’s just a side note.)

As far as I can see, there are three types of L4A tasks that did not complete successfully and were therefore not deleted correctly.

The early ones:
Error in step “s4c-l4a-extract-parcels”

Traceback (most recent call last):
  File "/usr/bin/extract-parcels.py", line 444, in <module>
    main()
  File "/usr/bin/extract-parcels.py", line 429, in main
    extract_lpis_path(conn, args.site_id, season_end, args.lpis_path)
  File "/usr/bin/extract-parcels.py", line 344, in extract_lpis_path
    path = cursor.fetchone()[0]
TypeError: 'NoneType' object has no attribute '__getitem__'

The middle ones:
Error in step “s4c-crop-type”

Empty stderr_text. But stdout_text like: s4c-crop-type_stdout_text.zip (206.8 KB)

The last ones:
Error in step “s4c-l4a-extract-parcels”

Traceback (most recent call last):
  File "/usr/bin/extract-parcels.py", line 444, in <module>
    main()
  File "/usr/bin/extract-parcels.py", line 409, in main
    extract_tile_footprints(conn, args.site_id, args.tile_footprints)
  File "/usr/bin/extract-parcels.py", line 190, in extract_tile_footprints
    save_to_csv(cursor, file, ["tile_id", "epsg_code", "geog"])
  File "/usr/bin/extract-parcels.py", line 69, in save_to_csv
    writer.writerow(row)
IOError: [Errno 28] No space left on device

The latter makes sense even to me directly :wink:

Can you see where the problem was, especially with the middle error message? Is there a general configuration problem here?

And do you have any hints how I can cleanly delete the whole site as desired (see first post)? I have the feeling that my many attempts here have already made a mess of things. I would like to set up the site again without having to go through the months of pre-processing again.

Either way thanks so much for the help!!!