L4B error on MowingInputShpGenerator

[Harijsi]

Hello Sen4CAP team and @cudriou,

We have been running in some issues with Sen4CAP L4B Grassland Mowing processor. It seems to be failing on the second step.

Here is the error text:

Traceback (most recent call last):
File “/usr/share/sen2agri/S4C_L4B_GrasslandMowing/Bin/generate_grassland_mowing_input_shp.py”, line 195, in
main()
File “/usr/share/sen2agri/S4C_L4B_GrasslandMowing/Bin/generate_grassland_mowing_input_shp.py”, line 167, in main
dataset.StartTransaction()
File “/usr/lib64/python2.7/site-packages/osgeo/ogr.py”, line 405, in
getattr = lambda self, name: _swig_getattr(self, DataSource, name)
File “/usr/lib64/python2.7/site-packages/osgeo/ogr.py”, line 54, in _swig_getattr
raise AttributeError(name)
AttributeError: StartTransaction

The output of this process is here:

/usr/share/sen2agri/S4C_L4B_GrasslandMowing/Bin/generate_grassland_mowing_input_shp.py is absolute!
No filter-ctnum provided!
No filter-ids-table provided!
sen2agri-service
sen2agri-service
Activating conda sen4cap for user sen2agri-service
Executing source ~/.bashrc && conda activate sen4cap && python /usr/share/sen2agri/S4C_L4B_GrasslandMowing/Bin/generate_grassland_mowing_input_shp.py --config-file /etc/sen2agri/sen2agri.conf --site-id 5 --path /mnt/archive/orchestrator_temp/s4c_l4b/1256/10018-s4c-grassland-gen-input-shp/SEN4CAP_L4B_GeneratedInputShp.shp --year 2020

        select short_name
        from site
        where id = 5

Starting executing the commands …
ogr2ogr -sql ‘select “NewID”, ori_hold as “Ori_hold”, ori_id as “Ori_id”, ori_crop as “Ori_crop”, “Area_meters” as “Area_meter”, wkb_geometry from “decl_rugaju_dwnld_2020”’ /mnt/archive/orchestrator_temp/s4c_l4b/1256/10018-s4c-grassland-gen-input-shp/SEN4CAP_L4B_GeneratedInputShp.shp ‘PG:dbname=sen4cap host=localhost port=5432 user=admin password=sen2agri’ -lco ENCODING=UTF-8
Finished executing commands!
5423 feature(s) found

Also, the output for the first step indicates several warnings of not being able to copy files, maybe that might be the reason why the function of the 2nd step can’t locate the attribute.
Output of ProductFormatter:

2020 Aug 10 18:40:40 : Application.logger (WARNING) Error copying file /mnt/archive/orchestrator_temp/s4c_l4b/1256/10021-product-formatter/SEN4CAP_L4B_S1_S2_MowingDetection.shp to file /mnt/archive/rugaju_dwnld/s4c_l4b//S2AGRI_S4C_L4B_PRD_S5_20200810T184040_V20200601T000000_20200815T000000/VECTOR_DATA/SEN4CAP_L4B_S1_S2_MowingDetection.shp
2020 Aug 10 18:40:40 : Application.logger (WARNING) Error copying file /mnt/archive/orchestrator_temp/s4c_l4b/1256/10021-product-formatter/SEN4CAP_L4B_S1_S2_MowingDetection.dbf to file /mnt/archive/rugaju_dwnld/s4c_l4b//S2AGRI_S4C_L4B_PRD_S5_20200810T184040_V20200601T000000_20200815T000000/VECTOR_DATA/SEN4CAP_L4B_S1_S2_MowingDetection.dbf
2020 Aug 10 18:40:40 : Application.logger (WARNING) Error copying file /mnt/archive/orchestrator_temp/s4c_l4b/1256/10021-product-formatter/SEN4CAP_L4B_S1_S2_MowingDetection.prj to file /mnt/archive/rugaju_dwnld/s4c_l4b//S2AGRI_S4C_L4B_PRD_S5_20200810T184040_V20200601T000000_20200815T000000/VECTOR_DATA/SEN4CAP_L4B_S1_S2_MowingDetection.prj
2020 Aug 10 18:40:40 : Application.logger (WARNING) Error copying file /mnt/archive/orchestrator_temp/s4c_l4b/1256/10021-product-formatter/SEN4CAP_L4B_S1_S2_MowingDetection.shx to file /mnt/archive/rugaju_dwnld/s4c_l4b//S2AGRI_S4C_L4B_PRD_S5_20200810T184040_V20200601T000000_20200815T000000/VECTOR_DATA/SEN4CAP_L4B_S1_S2_MowingDetection.shx
2020 Aug 10 18:40:40 : Application.logger (WARNING) Error copying file /mnt/archive/orchestrator_temp/s4c_l4b/1256/10021-product-formatter/SEN4CAP_L4B_S1_S2_MowingDetection.shp to file /mnt/archive/rugaju_dwnld/s4c_l4b//S2AGRI_S4C_L4B_PRD_S5_20200810T184040_V20200601T000000_20200815T000000/VECTOR_DATA/SEN4CAP_L4B_S1_S2_MowingDetection.shp
2020 Aug 10 18:40:40 : Application.logger (WARNING) Error copying file /mnt/archive/orchestrator_temp/s4c_l4b/1256/10020-s4c-grassland-mowing/SEN4CAP_L4B_S1_S2_MowingDetection.dbf to file /mnt/archive/rugaju_dwnld/s4c_l4b//S2AGRI_S4C_L4B_PRD_S5_20200810T184040_V20200601T000000_20200815T000000/VECTOR_DATA/SEN4CAP_L4B_S1_S2_MowingDetection.dbf
2020 Aug 10 18:40:40 : Application.logger (WARNING) Error copying file /mnt/archive/orchestrator_temp/s4c_l4b/1256/10020-s4c-grassland-mowing/SEN4CAP_L4B_S1_S2_MowingDetection.prj to file /mnt/archive/rugaju_dwnld/s4c_l4b//S2AGRI_S4C_L4B_PRD_S5_20200810T184040_V20200601T000000_20200815T000000/VECTOR_DATA/SEN4CAP_L4B_S1_S2_MowingDetection.prj
2020 Aug 10 18:40:40 : Application.logger (WARNING) Error copying file /mnt/archive/orchestrator_temp/s4c_l4b/1256/10020-s4c-grassland-mowing/SEN4CAP_L4B_S1_S2_MowingDetection.shx to file /mnt/archive/rugaju_dwnld/s4c_l4b//S2AGRI_S4C_L4B_PRD_S5_20200810T184040_V20200601T000000_20200815T000000/VECTOR_DATA/SEN4CAP_L4B_S1_S2_MowingDetection.shx
2020 Aug 10 18:40:40 : Application.logger (WARNING) Error copying file /mnt/archive/orchestrator_temp/s4c_l4b/1256/10021-product-formatter/executionInfos.txt to file /mnt/archive/rugaju_dwnld/s4c_l4b//S2AGRI_S4C_L4B_PRD_S5_20200810T184040_V20200601T000000_20200815T000000/AUX_DATA/S2AGRI_S4C_L4B_IPP_V20200601T000000_20200815T000000.txt
CoordRefSystem.HorizCSName = CoordRefSystem.HorizCSName.empty() =1

I added a system log for one of the failed attempts as well:
JOB_ID_1274.txt (17.1 KB)

Thanks a lot in advance!

[lnicola]

Hi,

For the permissions error, can you try

ls -ld /mnt/archive/rugaju_dwnld/
ls -ld /mnt/archive/rugaju_dwnld/s4c_l4b

They should be owned by the sen2agri-service user. They probably are, and the other error is responsible for your issue.

The first error might be related to a missing Python dependency in the Conda environment that this processor is using. We’ll come back with more questions or an update about that.

[Harijsi]

Hello,

You are right - the permissions are set correctly, we had checked this when encountered the error for the first time. Thank you for looking into this, much appreciated.

Best,
Harijs

[cudroiu]

Dear Harijs,

Could you please execute the following commands in a console and provide the outputs:

• Login under the user sen2agri-service (sudo su -l sen2agri-service)
• Create some paths needed for the commands:

mkdir -p /mnt/archive/orchestrator_temp/s4c_l4b/1274/10307-s4c-grassland-gen-input-shp/
mkdir -p /mnt/archive/orchestrator_temp/s4c_l4b/1274/10310-product-formatter/

• Execute the commands:

/usr/share/sen2agri/S4C_L4B_GrasslandMowing/Bin/generate_grassland_mowing_input_shp.sh -s 5 -y 2020 -o /mnt/archive/orchestrator_temp/s4c_l4b/1274/10307-s4c-grassland-gen-input-shp/SEN4CAP_L4B_GeneratedInputShp.shp -p /usr/share/sen2agri/S4C_L4B_GrasslandMowing/Bin/generate_grassland_mowing_input_shp.py -f 710,713,714,715,716,720,723,724,725,726,727,728,729,731,732,733,734,735,736,738,739

Please check that the shapefile /mnt/archive/orchestrator_temp/s4c_l4b/1274/10307-s4c-grassland-gen-input-shp/SEN4CAP_L4B_GeneratedInputShp.shp is correctly created and can be opened in QGis.

If OK, then execute the command:

/usr/share/sen2agri/S4C_L4B_GrasslandMowing/Bin/grassland_mowing.sh --script-path /usr/share/sen2agri/S4C_L4B_GrasslandMowing/Bin/src_s2/S2_main.py --site-id 5 --config-file /mnt/archive/grassland_mowing_files/rugaju_dwnld/2020/config/S4C_L4B_Config_2020.cfg --input-shape-file /mnt/archive/orchestrator_temp/s4c_l4b/1274/10307-s4c-grassland-gen-input-shp/SEN4CAP_L4B_GeneratedInputShp.shp --output-data-dir /mnt/archive/orchestrator_temp/s4c_l4b/1274/10308-s4c-grassland-mowing/SEN4CAP_L4B_S2_OutputData --start-date 2020-06-01 --end-date 2020-08-10 --seg-parcel-id-attribute NewID --output-shapefile /mnt/archive/orchestrator_temp/s4c_l4b/1274/10310-product-formatter/SEN4CAP_L4B_S1_S2_MowingDetection.shp --do-cmpl True --test True --input-products-list /mnt/archive/rugaju_dwnld/l3b_lai//S2AGRI_L3B_PRD_S5_20200807T093037_A20200625T093436 /mnt/archive/rugaju_dwnld/l3b_lai//S2AGRI_L3B_PRD_S5_20200807T094555_A20200720T093432 /mnt/archive/rugaju_dwnld/l3b_lai//S2AGRI_L3B_PRD_S5_20200809T002009_A20200608T094432 /mnt/archive/rugaju_dwnld/l3b_lai//S2AGRI_L3B_PRD_S5_20200809T004822_A20200613T094428 /mnt/archive/rugaju_dwnld/l3b_lai//S2AGRI_L3B_PRD_S5_20200809T011706_A20200618T094432 /mnt/archive/rugaju_dwnld/l3b_lai//S2AGRI_L3B_PRD_S5_20200809T014236_A20200623T094429 /mnt/archive/rugaju_dwnld/l3b_lai//S2AGRI_L3B_PRD_S5_20200809T020424_A20200628T094431 /mnt/archive/rugaju_dwnld/l3b_lai//S2AGRI_L3B_PRD_S5_20200809T022605_A20200703T094428 /mnt/archive/rugaju_dwnld/l3b_lai//S2AGRI_L3B_PRD_S5_20200809T025150_A20200708T094430 /mnt/archive/rugaju_dwnld/l3b_lai//S2AGRI_L3B_PRD_S5_20200809T032544_A20200713T094428 /mnt/archive/rugaju_dwnld/l3b_lai//S2AGRI_L3B_PRD_S5_20200809T035347_A20200718T094431 /mnt/archive/rugaju_dwnld/l3b_lai//S2AGRI_L3B_PRD_S5_20200809T042107_A20200802T094429 /mnt/archive/rugaju_dwnld/l3b_lai//S2AGRI_L3B_PRD_S5_20200809T044527_A20200807T094432 /mnt/archive/rugaju_dwnld/l3b_lai//S2AGRI_L3B_PRD_S5_20200810T002306_A20200605T093441 /mnt/archive/rugaju_dwnld/l3b_lai//S2AGRI_L3B_PRD_S5_20200810T004957_A20200625T093440 /mnt/archive/rugaju_dwnld/l3b_lai//S2AGRI_L3B_PRD_S5_20200810T011848_A20200720T093437 /mnt/archive/rugaju_dwnld/l3b_lai//S2AGRI_L3B_PRD_S5_20200810T014608_A20200725T093440 /mnt/archive/rugaju_dwnld/l3b_lai//S2AGRI_L3B_PRD_S5_20200810T021912_A20200809T093438 /mnt/archive/rugaju_dwnld/l3b_lai//S2AGRI_L3B_PRD_S5_20200811T002327_A20200608T094438 /mnt/archive/rugaju_dwnld/l3b_lai//S2AGRI_L3B_PRD_S5_20200811T005133_A20200613T094434 /mnt/archive/rugaju_dwnld/l3b_lai//S2AGRI_L3B_PRD_S5_20200811T011714_A20200618T094438 /mnt/archive/rugaju_dwnld/l3b_lai//S2AGRI_L3B_PRD_S5_20200811T014455_A20200623T094435 /mnt/archive/rugaju_dwnld/l3b_lai//S2AGRI_L3B_PRD_S5_20200811T020910_A20200628T094437 /mnt/archive/rugaju_dwnld/l3b_lai//S2AGRI_L3B_PRD_S5_20200811T023754_A20200708T094436 /mnt/archive/rugaju_dwnld/l3b_lai//S2AGRI_L3B_PRD_S5_20200811T031054_A20200713T094433 /mnt/archive/rugaju_dwnld/l3b_lai//S2AGRI_L3B_PRD_S5_20200811T034023_A20200718T094437 /mnt/archive/rugaju_dwnld/l3b_lai//S2AGRI_L3B_PRD_S5_20200811T040702_A20200723T094434 /mnt/archive/rugaju_dwnld/l3b_lai//S2AGRI_L3B_PRD_S5_20200811T043149_A20200807T094438

Please provide the full log messages for the two commands but also the file /mnt/archive/grassland_mowing_files/rugaju_dwnld/2020/config/S4C_L4B_Config_2020.cfg that you used.

Best regards,
Cosmin

[Harijsi]

Thank you for your reply, Cosmin

We will do this shortly. In the mean time, returning to the issue of Conda, our sysadmin indicated that there were some issues with it when setting up the system. When trying a repeat setup of Conda, this is what we get:

Downloading and Extracting Packages
Preparing transaction: …working… done
Verifying transaction: …working… done
Executing transaction: …working… failed
ERROR conda.core.link:_execute(481): An error occurred while installing package ‘conda-forge::astor-0.8.1-pyh9f0ad1d_0’.
FileNotFoundError(2, “No such file or directory: ‘/home/sen2agri-service/miniconda3/envs/sen4cap/bin/python3.6’”)
Attempting to roll back.
Rolling back transaction: …working… done
FileNotFoundError(2, “No such file or directory: ‘/home/sen2agri-service/miniconda3/envs/sen4cap/bin/python3.6’”)

Just in case this is relevant to the issue.

Best,
Harijs

[cudroiu]

Dear Harijs,

Yes, this is an important information. Apparently, conda did not installed properly in your case and this might be the reason the L4B is not running as the L4B processor is using python 3.6 from the conda environment. So I think this issue is very important to be solved before executing the steps from my previous message.
As conda is installing in /home/sen2agri-service/miniconda3, could you please check that you have at least 3.5GB free in /home before installing conda and the sen4cap environment?
If not, you might consider installing miniconda3 in another location where you have enough disk space (I can provide you the procedure to do this)

Please let me know.

Best regards,
Cosmin

[Harijsi]

Hello Cosmin,

We tried exploring the issue, but did not make any progress. The system has plenty of space, but, when setting up conda, we still get the same error. Would you have any other recommendations for troubleshooting/setting up conda?

Best,
Harry

[cudroiu]

Dear Harijsi,

Could you please provide more details on the errors that you obtain when you install conda?
Also, even if your system has plenty of disk space available, please note that a Centos has several partitions and the /home partition (where conda is installed) is usually mounted on a smaller partition.
You can provide additionally also the results of the command :

df -h

Best regards,
Cosmin

[Harijsi]

Dear Cosmin,
Thank you for your reply continuous commitment to our issue. It’s greatly appreciated.
I have attached two logs - one of the results after launcing the install_conda_sen4cap.sh script.
The other one showing our partition structure.

What we have deduced so far: Before attempting to reinstall, the miniconda3 folder is cleared, so everything within that directory is created by the script at installation. Judging by the error, during the install there is no conda env named sen4cap with Python 3.6. The only Python 3.6 is located in /home/sen2agri-service/miniconda3/bin/python3.6

df_h.txt (897 Bytes) conda_install.txt (35.9 KB)

[cudroiu]

Dear Harijsi,

Seems that there is a broken package so I have added the following line in the conda install script, right after the installation of Miniconda3-latest-Linux-x86_64.sh:

sudo su -l sen2agri-service -c bash -c “conda update -y -n base -c defaults conda”

Could you try the following operation:

1. Download and unzip the following file:install_conda_sen4cap.zip (904 Bytes)
2. Execute in the directory where you unzipped the above archive :

sudo rm -fr /home/sen2agri-services/miniconda3
sudo ./install_conda_sen4cap.sh

Please let me know.

Best regards,
Cosmin

[Harijsi]

Hello Cosmin,

Thank you for the updated script. Now Conda seems to install correctly and we can cross this issue off the list. However, the L4B issue still persists.
Now, returning to your previous suggestion, we tried to launch the commands you provided, but for the very first one (after creating directories) we encountered an error. Which is most likely the reason that the second command gives an index out of range error as well.

/usr/share/sen2agri/S4C_L4B_GrasslandMowing/Bin/generate_grassland_mowing_input_shp.sh -s 5 -y 2020 -o /mnt/archive/orchestrator_temp/s4c_l4b/1274/10307-s4c-grassland-gen-input-shp/SEN4CAP_L4B_GeneratedInputShp.shp -p
No script-path provided!
Usage: ./generate_grassland_mowing_input_shp.sh --user --script-path <script_path> --site-id <site_id> --config-file <config_file> --path <working_path> --year --filter-ctnum <filter_ct_num> --force

/usr/share/sen2agri/S4C_L4B_GrasslandMowing/Bin/generate_grassland_mowing_input_shp.py -f 710,713,714,715,716,720,723,724,725,726,727,728,729,731,732,733,734,735,736,738,739

        select short_name
        from site
        where id = NULL

Traceback (most recent call last):
File “/usr/share/sen2agri/S4C_L4B_GrasslandMowing/Bin/generate_grassland_mowing_input_shp.py”, line 195, in
main()
File “/usr/share/sen2agri/S4C_L4B_GrasslandMowing/Bin/generate_grassland_mowing_input_shp.py”, line 114, in main
site_name = get_site_name(conn, config.site_id)
File “/usr/share/sen2agri/S4C_L4B_GrasslandMowing/Bin/generate_grassland_mowing_input_shp.py”, line 68, in get_site_name
return rows[0][0]
IndexError: list index out of range

Best,
Harijs

[cudroiu]

Dear Harijsi,

I noticed some errors in your commands:

1. Did you executed the command manually? Could you try to add a Scheduled Job in the Dashboard with the desired date and see what is happening?
2. The first command seems to be incomplete (as the error said). If you ran the command manually, are you sure you copied correctly the command (the -p argument should have also the name of the script after)
3. Are you sure you have the site with id 5? If yes, are you sure this site is enabled?
   You can check that by executing in the console:

psql -U admin sen4cap -c “select * from site”

Best regards,
Cosmin

[Harijsi]

Thank you for the tip! Indeed, we misinterpreted the instructions as two commands. It turns out that the issue regarding .shp creation was incompatible data types in the declaration dataset. We solved that with:

alter table decl_rugaju_dwnld_2020
alter column ori_crop type int using ori_crop :: int,
alter column product_co type int using ori_crop :: int;
vacuum analyze decl_rugaju_dwnld_2020;

Now everything works properly regarding the initial shapefile creation. It can be opened in QGIS and has all the attributes. However, there still is an issue with the L4B processor itself.

“RuntimeError: CreateSpatialIndex : unsupported operation on a read-only datasource”

in S2Mowing and

“AttributeError: ‘NoneType’ object has no attribute ‘GetLayer’”

in S1Mowing step.

I have added a log from the commands that we had to run after creating the .shp.

L4B_Log_27082020.txt (97.5 KB)

Thank you so much for the help up to this point!

Best,
Harijs

[cudroiu]

Dear Harijsi,

This can be solved using the patch I posted here:

Error on S2MowingDetection Step

Hello, It seems that some updates were made in the gdal version from the conda (I assume there is a newly installed or updated machine) and in this new version were introduced some constraints. I am attaching 2 patches (S2_gmd.patch and S1_gmd.patch in the archive Sen4CAP_v_1_2_L4B_S1_gmd_patches.zip (790 Bytes) ) for this issue that can be installed with the following commands: unzip Sen4CAP_v_1_2_L4B_S1_gmd_patches.zip sudo patch -u /usr/share/sen2agri/S4C_L4B_GrasslandMowing/Bin/src_s2/S…

Please let me know.

Best regards,
Cosmin

[Harijsi]

The process finally finishes without errors and provides results from both S1 and S2. Thank you very much!

Best,
Harijs