Generating LPIS failed

System Installation and Configuration
1

Hello Sen4Cap,

I have successfully uploaded Subsidy application dataset and Crop Code Look LUT with help of Cosmin. Crop type is generated well but not LPIS. Even though it showed “Finished” on Monitoring page, there is no output under Product page. No Output error as well. Please kindly help to solve it out.

Thanks,
Henry

2

Here is the only error i noticed in error_log. Hope it helps. Let me know if you need more input. Thanks.

“[Tue Apr 28 16:53:39.885738 2020] [:error] [pid 13919] [client 1.10.193.103:54494] {“upload_sitename”:“LLK_training”,“upload_siteid”:“4”,“succ_up_files”:”",“err_up_files”:"",“upload_msgs”:"",“upload_year”:“2019”,“change_upload_year”:“Change year”}, referer: http://35.184.233.xxx/create_site.php"

3

Dear Henry,

Most probably, there is an inconsistence in the data uploaded and that is used for generating the LPIS product.
In order to debug the import of the files, you could do the following operations:

  • start logging the messages in the sen4cap services using:

sudo journalctl -fu sen2agri-services

  • start the upload of the declarations and LUT files from the web interface (sites -> edit site)
  • check the sen4cap services logs for the messages after a message like the following one:

/usr/bin/data-preparation.py -s 4 --year 2019 --parcel-id-cols PPP --holding-id-cols HHH --crop-code-col CCC --lpis LLL --lut UUU --mode < mode >

(Please note that PPP, HHH, CCC, LLL, UUU and < mode > are specific to your execution)

  • If you did not uploaded yet the files for the L4C processor, you can ignore the errors related to the export of L4C practices as they are normal (errors that occur after the command /usr/bin/s4c_l4c_export_all_practices.py -s llk_training -y 2019).
  • If you want to see only the messages for your LPIS import, you can take the above /usr/bin/data-preparation.py command and execute it in the following manner:

sudo su -l sen2agri-service
[sen2agri-service@aaa ~]$ /usr/bin/data-preparation.py -s 4 --year 2019 … (rest of command here)

Please let us know.

Best regards,
Cosmin

4

Dear Cosmin,

Thank you for detail explanation. Here is what i found in sen4cap services after uploading declarations and LUT files.

Apr 29 09:26:12 sen2cap start.sh[12398]: 2020-Apr-29 09:26:12.230 INFO [http-nio-8080-exec-9] r.c.t.s.c.ControllerBase$1 - Initializing ExecutorService
Apr 29 09:26:13 sen2cap start.sh[12398]: 2020-Apr-29 09:26:13.549 INFO [ControllerBase.1-1] o.e.s.s.web.ImportController - Processing LLK_2019_parcels_47PPR.shp completed
Apr 29 09:27:41 sen2cap start.sh[12398]: 2020-Apr-29 09:27:41.250 WARN [ControllerBase.1-1] o.e.s.s.web.ImportController - Sen4Cap_L4A_LLK_2019_CropCode_LUT.csv import ended with code 2
Apr 29 09:27:41 sen2cap start.sh[12398]: 2020-Apr-29 09:27:41.250 INFO [ControllerBase.1-1] o.e.s.s.web.ImportController - Processing Sen4Cap_L4A_LLK_2019_CropCode_LUT.csv completed

Here is command and output after i run this command with my specific parameters. Let me know if you need further information.

/usr/bin/data-preparation.py -s 4 --year 2019 --parcel-id-cols 31.0000001234567.001 --holding-id-cols 123456789 --crop-code-col 19 --lpis LLK_2019_parcels.shp --lut Sen4CAP_L4A_TH_2019_CropCode_LUT.csv --mode update

-> AVCE00' -> HTTP’
Preparing LUT
Traceback (most recent call last):
File “/usr/bin/data-preparation.py”, line 1534, in
main()
File “/usr/bin/data-preparation.py”, line 1515, in main
data_preparation.prepare_lut(args.lut)
File “/usr/bin/data-preparation.py”, line 516, in prepare_lut
cursor.execute(query)
psycopg2.errors.UndefinedTable: relation “lut_llk_training_2019” does not exist

Thanks and Regards,
Henry

5

Dear Henry,

From what I see, you did not provided the column names from the declarations shapefile but instead some values. The parameters --parcel-id-cols 31.0000001234567.001 --holding-id-cols 123456789 --crop-code-col 19 seem to contain values. Are the “31.0000001234567.001”, "123456789 " and “19” column names in the uploaded shapefile? These parameters should be the column names corresponding to the original parcel ids column(s), original holding id column(s) or original crop code column. Please check also the SUM for more details.

Best regards,
Cosmin

6

Dear Cosmin,

My bad. i have run command with column names as below. error output is attached.

/usr/bin/data-preparation.py -s 4 --year 2019 --parcel-id-cols FUNCTIONEE --holding-id-cols FakeBRSnr --crop-code-col GRONDBEDEK --lpis LLK_2019_parcels.shp --lut Sen4CAP_CropCode_LUT.csv --mode update

lpis_error.txt (862 Bytes)

Thanks and Regards,
Henry

7

Could you please post the full log of the command?
It doesn’t seem to be complete.

Cosmin

8

Dear Cosmin,

Oh… sorry. Full log of the command is attached.
lpis_full_error_Log.txt (1.5 KB)

Thanks and Regards,
Henry

9

Dear Henry,

Did you executed the command under user sen2agri-service (performing first the sudo su -l sen2agri-service command)?
Also, could you provide the LUT file?

Cosmin

10

Dear Cosmin,

Yes. it’s run under sen2agri. LUT file is attached. Changed extension to txt as fourm doesn’t allow csv format.

Sen4CAP_CropCode_LUT.txt (155 Bytes)

Value of parameters from shape file is also attached.

QGIS_shapefile_params

Thanks and Regards,
Henry

11

Dear Henry,

Could you please upgrade the system to the version from this location (you should run sudo ./update.sh) and try again performing the import:
https://we.tl/t-T7E1ira1Ge
This is the a new version of the system that we intend to release soon (I assume you use version 1.0.0).

Please let me know.

Best regards,
Cosmin

12

Dear Cosmin,

I will try it out and get back to you. Thanks a lot.

Best Regards,
Henry

13

Dear Cosmin,

I have updated accordingly and here is the log i got after i run the command.

I also attached sen2agri service log.

lpis_error_log.txt (1.7 KB)
sen2agri-services_log202000429.txt (2.5 KB)

Thanks and Regards,
Henry

14

Dear Cosmin,

have you found the issue? Let me know if you need more input to solve it out.

Thanks and Regards,
Henry

15

Dear Sen4Cap and Cosmin,

I am still puzzled why LPIS product is failed to generate even it shows “Finished” status on monitoring page. Any tips to debug?

Thanks and Regards,
Henry

16

Hello Henry,

I think that the problem could be that the crop code from the shapefile is in text format, while the crop code from the LUT table (Ori_crop) is recognized as a numerical value (integer) by the system. So the system cannot join the two datasets based on the crop code. What you should do is to check the format of the crop code in the shapefile. If it is not a numerical value (integer) you should convert it in integer. And then, reimport the parcels.

Let me know if that was the problem.

Best regards,

Philippe

17

And I forgot, after reimporting the parcels shp, to reimport again the LUT (to be sure).

Best regards,

Philippe

18

Hello Phillippe,

Thank you for your reply. I have checked the shapefile and I confirmed that crop type code is integer. Apparently, system can generate crop type product as well. Any other ways to debug?

Thanks and Regards,
Henry

19

Hello Henry,

Yes sorry, I forgot that the crop type was well generated so it is not this error.

If you want to extract the parcels with all the quality and information flags you can use this command in the console:
ogr2ogr -f “ESRI Shapefile” -lco ENCODING=UTF-8 /{path}/parcels.shp PG:“host=localhost user=admin dbname=sen4cap password=sen2agri port=5432” -sql “SELECT * from decl_{site_name}_{year}”

But it will not make it accessible in the products tab. For that, I cannot help. Let’s see if Cosmin finds a way to fix it.

Best regards,

Philippe

1 Like
20

Hello Philippe,

Thank you for the tips. Command is working after changing permission of postgres and gdal update. It did generate parcel shape file but it gave some warnings as below though.

Warning 6: Normalized/laundered field name: ‘Area_meters’ to ‘Area_meter’
Warning 6: Normalized/laundered field name: ‘inserted_timestamp’ to ‘inserted_t’
Warning 6: Field inserted_t create as date field, though DateTime requested.
Warning 6: Normalized/laundered field name: ‘updated_timestamp’ to ‘updated_ti’
Warning 6: Field updated_ti create as date field, though DateTime requested.
Warning 6: Normalized/laundered field name: ‘geom_change_ratio’ to ‘geom_chang’

Anything am I missing?

Thanks and Regards,
Henry