No predicted classes created in L4A processor

Jonathan · November 23, 2020, 9:54am

Hi everyone,

yes, we did upgrade to version 1.3, still we get no prediction results when running from GUI. Running from console does not produce the crop_type.csv. Instead the execution is not finished due to the same Error we got before updating:

Warning message:
NAs introduced by coercion
`summarise()` ungrouping output (override with `.groups` argument)
Warning message:
NAs introduced by coercion
Error in `levels<-`(`*tmp*`, value = as.character(levels)) :
  factor level [23] is duplicated
Calls: factor
Execution halted
Exit code: 1

Here I read “factor level duplicated”. In another forum (not Sen4CAP related) I found that this error is probably originating from an R script.

Another notice: To reduce the processing time while testing to get proper results, I choose a narrow season of only one month. I hope this does not effect the processing in a way that this error is produced?

Unfortunately we are in a hurry right now, as we did not get results due to this problem since about a month of trying different things. We really need these results as soon as possible.

Any help is very appreciated!

Best Regards
Jonathan

DianeHey · November 24, 2020, 9:24am

Dear Jonathan,

looking at your ouput I would says that the error comes from the creation of the accuracy metrics plot.
If your not interested in this plot, I would recommand to remove/comment this section in the script (crop_type.R) from the line 701 to 762.
The last section of the script (which is the creation of the crop_type.csv or now named Parcels_all_prediction.csv) should work even by removing the plot section.

Hopes it help,

Diane

lnicola · November 25, 2020, 7:24am

The issue was a couple of duplicate LUT entries (i.e. rows with the same CTL4A but different CTnumL4A values).

Okke_Gerhard · December 3, 2020, 7:01am

Hello @Philippe_Malcorps,
we are also running into some problems with the predicted classes for the L4A processor. I encountered this probleam already with version 1.2 but was hoping it would be solved by updating to version 1.3. As shown in the pervious posts, our results also don´t show any predicted classes. Moreover, the resuls we are getting only show following:

In the .shp it shows:

As for now, i did not run the “crop_type_unix” patch as i thought the problem was connected to version 1.2 and would be solved with 1.3.
We encountered the same problem with previous version (i think 1.0) and it was a problem with the data formats in the LUT (int), however our used LUT should have the right data formats.

In the “Error” tab in monitoring it shows no explicit Error Message, however the Code it provides is following:
Traceback (most recent call last):
File “/usr/bin/crop-type-parcels.py”, line 1489, in
main()
File “/usr/bin/crop-type-parcels.py”, line 1485, in main
process_radar(config, conn, pool)
File “/usr/bin/crop-type-parcels.py”, line 1144, in process_radar
products = get_radar_products(config, conn, config.site_id)
File “/usr/bin/crop-type-parcels.py”, line 803, in get_radar_products
cursor.execute(query)
psycopg2.InternalError: CircTreePIP: Unable to generate outside point!

Usage: merge-statistics mean.csv dev.csv mean-1.csv dev-1.csv count-1.csv […]
cut: sar-merged/mean-sar.csv: No such file or directory
cut: sar-merged/mean-sar.csv: No such file or directory
cut: sar-merged/dev-sar.csv: No such file or directory
cut: sar-merged/dev-sar.csv: No such file or directory

Attaching package: ‘dplyr’

The following objects are masked from ‘package:stats’:

filter, lag

The following objects are masked from ‘package:base’:

intersect, setdiff, setequal, union

Loading required package: lattice
Loading required package: ggplot2
Loading required package: proto
Warning message:
no DISPLAY variable so Tk is not available
── Attaching packages ─────────────────────────────────────── tidyverse 1.3.0 ──
tibble 3.0.3 stringr 1.4.0
tidyr 1.1.0 forcats 0.5.0
purrr 0.3.4
── Conflicts ────────────────────────────────────────── tidyverse_conflicts() ──
dplyr::filter() masks stats::filter()
dplyr::lag() masks stats::lag()
purrr::lift() masks caret::lift()

Attaching package: ‘data.table’

The following object is masked from ‘package:purrr’:

transpose

The following objects are masked from ‘package:dplyr’:

between, first, last

Warning message:
NAs introduced by coercion
Warning message:
NAs introduced by coercion
Error: Wrong argument: choose out of (1,2,3,4,5,0,‘All’,‘12’,‘123’,‘1234’)
Execution halted

Maybe it helps in finding a solution.

Please provide some solution for this problem.
Thank you very much.

Okke Gerhard

lnicola · December 3, 2020, 12:05pm

Dear Okee,

It looks like you’ve ran into an older issue that causes the S1 pre-processing to insert the wrong product footprints into the database. This was fixed in the meanwhile, but you might still have to update those manually.

I have a pair of scripts you can run for that. They’re very slow and ugly, but should hopefully do the job (make sure to replace $SITE_NAME below with the short site name):

$ sudo yum -y install jq
$ cd /mnt/archive/$SITE_NAME
$ find l2a-s1/ -name '*_AMP.tif' -print0 | while IFS= read -r -d '' line; do IFS='/' read -ra arr <<< $line; ( echo "UPDATE product SET geog = st_geogfromtext(TRIM('"; echo 'POLYGON(('; gdalinfo -json "$line" | jq -r '.wgs84Extent.coordinates[][]' | tr -s "[," " " | tr "]" "," | xargs | rev | cut -c 2- | rev; echo "))')) WHERE name = TRIM('"; echo ${arr[2]} | rev | cut -c 5- | rev; echo "');"; echo "UPDATE product SET footprint = (SELECT '(' || string_agg(REPLACE(replace(ST_AsText(geom) :: text, 'POINT', ''), ' ', ','), ',') || ')' from ST_DumpPoints(ST_Envelope(geog :: geometry)) WHERE path[2] IN (1, 3)) :: POLYGON WHERE name = TRIM('"; echo ${arr[2]} | rev | cut -c 5- | rev; echo "');" ) | cat | tr '\r\n\t' ' ' >> ~/sql_amp.txt; done;
$ find l2a-s1/ -name '*_COHE.tif' -print0 | while IFS= read -r -d '' line; do IFS='/' read -ra arr <<< $line; ( echo "UPDATE product SET geog = st_geogfromtext(TRIM('"; echo 'POLYGON(('; gdalinfo -json "$line" | jq -r '.wgs84Extent.coordinates[][]' | tr -s "[," " " | tr "]" "," | xargs | rev | cut -c 2- | rev; echo "))')) WHERE name = TRIM('"; echo ${arr[2]} | rev | cut -c 5- | rev; echo "');"; echo "UPDATE product SET footprint = (SELECT '(' || string_agg(REPLACE(replace(ST_AsText(geom) :: text, 'POINT', ''), ' ', ','), ',') || ')' from ST_DumpPoints(ST_Envelope(geog :: geometry)) WHERE path[2] IN (1, 3)) :: POLYGON WHERE name = TRIM('"; echo ${arr[2]} | rev | cut -c 5- | rev; echo "');" ) | cat | tr '\r\n\t' ' ' >> ~/sql_cohe.txt; done;
$ psql -U postgres sen4cap -f ~/sql_amp.txt
$ psql -U postgres sen4cap -f ~/sql_cohe.txt

Okke_Gerhard · December 3, 2020, 12:30pm

Hello @lnicola,
thank you for the scripts. I tried to run them, but encounter some difficulties.

It states, that there is no such file/directory as l2a-s1. However, there should be such directory as shown below.
grafik

I have accessed the proper folder (Side_Name) beforehand and installed jq.

Can you specify what you mean with “slow”. How long is it going to take to run the scripts? There should be over 1000 S1 images for this side. If i know it is going to take a longer time, i will shedule it accordingly.
Thank you very much for the help.

Okke

lnicola · December 3, 2020, 12:43pm

You need to go to the site directory (/mnt/archive/my-site). The screenshot shows that you’re running the command in /mnt/archive.

Can you specify what you mean with “slow”. How long is it going to take to run the scripts? There should be over 1000 S1 images for this side. If i know it is going to take a longer time, i will shedule it accordingly.

I don’t know, maybe 10 to 15 minutes?

Okke_Gerhard · December 3, 2020, 12:57pm

@lnicola
thank you very much. Now i am in the right directory and the script seems to run. I will let you know if that helped in my case.

Okke

Okke_Gerhard · December 4, 2020, 6:50am

Hello @lnicola,
i run the L4A processor and unfortunately, the results still do not show the predicted crop classes.
Still now error is shown, however in Monitoring --> Errors it shows following message:

WARNING: In /sen2agri/packaging/Sen2AgriPlatform/build/OTB-5.0/Modules/IO/ImageIO/include/otbImageFileWriter.txx, line 287
ImageFileWriter (0x23be110): Streaming configuration through extended filename is used. Any previous streaming configuration (ram value, streaming mode …) will be ignored.

WARNING: In /sen2agri/packaging/Sen2AgriPlatform/build/OTB-5.0/Modules/IO/ImageIO/include/otbImageFileWriter.txx, line 287
ImageFileWriter (0x2148670): Streaming configuration through extended filename is used. Any previous streaming configuration (ram value, streaming mode …) will be ignored.

WARNING: In /sen2agri/packaging/Sen2AgriPlatform/build/OTB-5.0/Modules/IO/ImageIO/include/otbImageFileWriter.txx, line 287
ImageFileWriter (0xd3fc40): Streaming configuration through extended filename is used. Any previous streaming configuration (ram value, streaming mode …) will be ignored.

WARNING: In /sen2agri/packaging/Sen2AgriPlatform/build/OTB-5.0/Modules/IO/ImageIO/include/otbImageFileWriter.txx, line 287
ImageFileWriter (0x264f650): Streaming configuration through extended filename is used. Any previous streaming configuration (ram value, streaming mode …) will be ignored.

WARNING: In /sen2agri/packaging/Sen2AgriPlatform/build/OTB-5.0/Modules/IO/ImageIO/include/otbImageFileWriter.txx, line 287
ImageFileWriter (0x1d7f680): Streaming configuration through extended filename is used. Any previous streaming configuration (ram value, streaming mode …) will be ignored.

WARNING: In /sen2agri/packaging/Sen2AgriPlatform/build/OTB-5.0/Modules/IO/ImageIO/include/otbImageFileWriter.txx, line 287
ImageFileWriter (0x1846a00): Streaming configuration through extended filename is used. Any previous streaming configuration (ram value, streaming mode …) will be ignored.

WARNING: In /sen2agri/packaging/Sen2AgriPlatform/build/OTB-5.0/Modules/IO/ImageIO/include/otbImageFileWriter.txx, line 287
ImageFileWriter (0x2148830): Streaming configuration through extended filename is used. Any previous streaming configuration (ram value, streaming mode …) will be ignored.

WARNING: In /sen2agri/packaging/Sen2AgriPlatform/build/OTB-5.0/Modules/IO/ImageIO/include/otbImageFileWriter.txx, line 287
ImageFileWriter (0x1c14a20): Streaming configuration through extended filename is used. Any previous streaming configuration (ram value, streaming mode …) will be ignored.

WARNING: In /sen2agri/packaging/Sen2AgriPlatform/build/OTB-5.0/Modules/IO/ImageIO/include/otbImageFileWriter.txx, line 287
ImageFileWriter (0x24e3610): Streaming configuration through extended filename is used. Any previous streaming configuration (ram value, streaming mode …) will be ignored.

WARNING: In /sen2agri/packaging/Sen2AgriPlatform/build/OTB-5.0/Modules/IO/ImageIO/include/otbImageFileWriter.txx, line 287
ImageFileWriter (0x2495be0): Streaming configuration through extended filename is used. Any previous streaming configuration (ram value, streaming mode …) will be ignored.

WARNING: In /sen2agri/packaging/Sen2AgriPlatform/build/OTB-5.0/Modules/IO/ImageIO/include/otbImageFileWriter.txx, line 287
ImageFileWriter (0x1c0ddc0): Streaming configuration through extended filename is used. Any previous streaming configuration (ram value, streaming mode …) will be ignored.

WARNING: In /sen2agri/packaging/Sen2AgriPlatform/build/OTB-5.0/Modules/IO/ImageIO/include/otbImageFileWriter.txx, line 287
ImageFileWriter (0x211ea00): Streaming configuration through extended filename is used. Any previous streaming configuration (ram value, streaming mode …) will be ignored.

WARNING: In /sen2agri/packaging/Sen2AgriPlatform/build/OTB-5.0/Modules/IO/ImageIO/include/otbImageFileWriter.txx, line 287
ImageFileWriter (0x17854d0): Streaming configuration through extended filename is used. Any previous streaming configuration (ram value, streaming mode …) will be ignored.

WARNING: In /sen2agri/packaging/Sen2AgriPlatform/build/OTB-5.0/Modules/IO/ImageIO/include/otbImageFileWriter.txx, line 287
ImageFileWriter (0x1698770): Streaming configuration through extended filename is used. Any previous streaming configuration (ram value, streaming mode …) will be ignored.

WARNING: In /sen2agri/packaging/Sen2AgriPlatform/build/OTB-5.0/Modules/IO/ImageIO/include/otbImageFileWriter.txx, line 287
ImageFileWriter (0x23caa60): Streaming configuration through extended filename is used. Any previous streaming configuration (ram value, streaming mode …) will be ignored.

WARNING: In /sen2agri/packaging/Sen2AgriPlatform/build/OTB-5.0/Modules/IO/ImageIO/include/otbImageFileWriter.txx, line 287
ImageFileWriter (0x2d5da60): Streaming configuration through extended filename is used. Any previous streaming configuration (ram value, streaming mode …) will be ignored.

WARNING: In /sen2agri/packaging/Sen2AgriPlatform/build/OTB-5.0/Modules/IO/ImageIO/include/otbImageFileWriter.txx, line 287
ImageFileWriter (0xbb3d60): Streaming configuration through extended filename is used. Any previous streaming configuration (ram value, streaming mode …) will be ignored.

WARNING: In /sen2agri/packaging/Sen2AgriPlatform/build/OTB-5.0/Modules/IO/ImageIO/include/otbImageFileWriter.txx, line 287
ImageFileWriter (0x1ca6e70): Streaming configuration through extended filename is used. Any previous streaming configuration (ram value, streaming mode …) will be ignored.

WARNING: In /sen2agri/packaging/Sen2AgriPlatform/build/OTB-5.0/Modules/IO/ImageIO/include/otbImageFileWriter.txx, line 287
ImageFileWriter (0x167b0e0): Streaming configuration through extended filename is used. Any previous streaming configuration (ram value, streaming mode …) will be ignored.

WARNING: In /sen2agri/packaging/Sen2AgriPlatform/build/OTB-5.0/Modules/IO/ImageIO/include/otbImageFileWriter.txx, line 287
ImageFileWriter (0x134aec0): Streaming configuration through extended filename is used. Any previous streaming configuration (ram value, streaming mode …) will be ignored.

WARNING: In /sen2agri/packaging/Sen2AgriPlatform/build/OTB-5.0/Modules/IO/ImageIO/include/otbImageFileWriter.txx, line 287
ImageFileWriter (0x1eaf480): Streaming configuration through extended filename is used. Any previous streaming configuration (ram value, streaming mode …) will be ignored.

WARNING: In /sen2agri/packaging/Sen2AgriPlatform/build/OTB-5.0/Modules/IO/ImageIO/include/otbImageFileWriter.txx, line 287
ImageFileWriter (0xc51db0): Streaming configuration through extended filename is used. Any previous streaming configuration (ram value, streaming mode …) will be ignored.

WARNING: In /sen2agri/packaging/Sen2AgriPlatform/build/OTB-5.0/Modules/IO/ImageIO/include/otbImageFileWriter.txx, line 287
ImageFileWriter (0x1e107d0): Streaming configuration through extended filename is used. Any previous streaming configuration (ram value, streaming mode …) will be ignored.

WARNING: In /sen2agri/packaging/Sen2AgriPlatform/build/OTB-5.0/Modules/IO/ImageIO/include/otbImageFileWriter.txx, line 287
ImageFileWriter (0x149e7d0): Streaming configuration through extended filename is used. Any previous streaming configuration (ram value, streaming mode …) will be ignored.

WARNING: In /sen2agri/packaging/Sen2AgriPlatform/build/OTB-5.0/Modules/IO/ImageIO/include/otbImageFileWriter.txx, line 287
ImageFileWriter (0x2e78430): Streaming configuration through extended filename is used. Any previous streaming configuration (ram value, streaming mode …) will be ignored.

WARNING: In /sen2agri/packaging/Sen2AgriPlatform/build/OTB-5.0/Modules/IO/ImageIO/include/otbImageFileWriter.txx, line 287
ImageFileWriter (0xb52870): Streaming configuration through extended filename is used. Any previous streaming configuration (ram value, streaming mode …) will be ignored.

WARNING: In /sen2agri/packaging/Sen2AgriPlatform/build/OTB-5.0/Modules/IO/ImageIO/include/otbImageFileWriter.txx, line 287
ImageFileWriter (0x1b7b610): Streaming configuration through extended filename is used. Any previous streaming configuration (ram value, streaming mode …) will be ignored.

WARNING: In /sen2agri/packaging/Sen2AgriPlatform/build/OTB-5.0/Modules/IO/ImageIO/include/otbImageFileWriter.txx, line 287
ImageFileWriter (0x1f75310): Streaming configuration through extended filename is used. Any previous streaming configuration (ram value, streaming mode …) will be ignored.

WARNING: In /sen2agri/packaging/Sen2AgriPlatform/build/OTB-5.0/Modules/IO/ImageIO/include/otbImageFileWriter.txx, line 287
ImageFileWriter (0x1a80030): Streaming configuration through extended filename is used. Any previous streaming configuration (ram value, streaming mode …) will be ignored.

WARNING: In /sen2agri/packaging/Sen2AgriPlatform/build/OTB-5.0/Modules/IO/ImageIO/include/otbImageFileWriter.txx, line 287
ImageFileWriter (0x120d3e0): Streaming configuration through extended filename is used. Any previous streaming configuration (ram value, streaming mode …) will be ignored.

WARNING: In /sen2agri/packaging/Sen2AgriPlatform/build/OTB-5.0/Modules/IO/ImageIO/include/otbImageFileWriter.txx, line 287
ImageFileWriter (0x134b4d0): Streaming configuration through extended filename is used. Any previous streaming configuration (ram value, streaming mode …) will be ignored.

WARNING: In /sen2agri/packaging/Sen2AgriPlatform/build/OTB-5.0/Modules/IO/ImageIO/include/otbImageFileWriter.txx, line 287
ImageFileWriter (0x1c71890): Streaming configuration through extended filename is used. Any previous streaming configuration (ram value, streaming mode …) will be ignored.

WARNING: In /sen2agri/packaging/Sen2AgriPlatform/build/OTB-5.0/Modules/IO/ImageIO/include/otbImageFileWriter.txx, line 287
ImageFileWriter (0x20b7810): Streaming configuration through extended filename is used. Any previous streaming configuration (ram value, streaming mode …) will be ignored.

WARNING: In /sen2agri/packaging/Sen2AgriPlatform/build/OTB-5.0/Modules/IO/ImageIO/include/otbImageFileWriter.txx, line 287
ImageFileWriter (0x1b13e00): Streaming configuration through extended filename is used. Any previous streaming configuration (ram value, streaming mode …) will be ignored.

WARNING: In /sen2agri/packaging/Sen2AgriPlatform/build/OTB-5.0/Modules/IO/ImageIO/include/otbImageFileWriter.txx, line 287
ImageFileWriter (0x26d6bb0): Streaming configuration through extended filename is used. Any previous streaming configuration (ram value, streaming mode …) will be ignored.

WARNING: In /sen2agri/packaging/Sen2AgriPlatform/build/OTB-5.0/Modules/IO/ImageIO/include/otbImageFileWriter.txx, line 287
ImageFileWriter (0x1967f60): Streaming configuration through extended filename is used. Any previous streaming configuration (ram value, streaming mode …) will be ignored.

WARNING: In /sen2agri/packaging/Sen2AgriPlatform/build/OTB-5.0/Modules/IO/ImageIO/include/otbImageFileWriter.txx, line 287
ImageFileWriter (0xbedf60): Streaming configuration through extended filename is used. Any previous streaming configuration (ram value, streaming mode …) will be ignored.

WARNING: In /sen2agri/packaging/Sen2AgriPlatform/build/OTB-5.0/Modules/IO/ImageIO/include/otbImageFileWriter.txx, line 287
ImageFileWriter (0x2ad2490): Streaming configuration through extended filename is used. Any previous streaming configuration (ram value, streaming mode …) will be ignored.

WARNING: In /sen2agri/packaging/Sen2AgriPlatform/build/OTB-5.0/Modules/IO/ImageIO/include/otbImageFileWriter.txx, line 287
ImageFileWriter (0x12c6a80): Streaming configuration through extended filename is used. Any previous streaming configuration (ram value, streaming mode …) will be ignored.

WARNING: In /sen2agri/packaging/Sen2AgriPlatform/build/OTB-5.0/Modules/IO/ImageIO/include/otbImageFileWriter.txx, line 287
ImageFileWriter (0x1e761a0): Streaming configuration through extended filename is used. Any previous streaming configuration (ram value, streaming mode …) will be ignored.

WARNING: In /sen2agri/packaging/Sen2AgriPlatform/build/OTB-5.0/Modules/IO/ImageIO/include/otbImageFileWriter.txx, line 287
ImageFileWriter (0x2867840): Streaming configuration through extended filename is used. Any previous streaming configuration (ram value, streaming mode …) will be ignored.

WARNING: In /sen2agri/packaging/Sen2AgriPlatform/build/OTB-5.0/Modules/IO/ImageIO/include/otbImageFileWriter.txx, line 287
ImageFileWriter (0x1b7de00): Streaming configuration through extended filename is used. Any previous streaming configuration (ram value, streaming mode …) will be ignored.

WARNING: In /sen2agri/packaging/Sen2AgriPlatform/build/OTB-5.0/Modules/IO/ImageIO/include/otbImageFileWriter.txx, line 287
ImageFileWriter (0xa81da0): Streaming configuration through extended filename is used. Any previous streaming configuration (ram value, streaming mode …) will be ignored.

WARNING: In /sen2agri/packaging/Sen2AgriPlatform/build/OTB-5.0/Modules/IO/ImageIO/include/otbImageFileWriter.txx, line 287
ImageFileWriter (0x19fceb0): Streaming configuration through extended filename is used. Any previous streaming configuration (ram value, streaming mode …) will be ignored.

WARNING: In /sen2agri/packaging/Sen2AgriPlatform/build/OTB-5.0/Modules/IO/ImageIO/include/otbImageFileWriter.txx, line 287
ImageFileWriter (0xbf6eb0): Streaming configuration through extended filename is used. Any previous streaming configuration (ram value, streaming mode …) will be ignored.

WARNING: In /sen2agri/packaging/Sen2AgriPlatform/build/OTB-5.0/Modules/IO/ImageIO/include/otbImageFileWriter.txx, line 287
ImageFileWriter (0x13244b0): Streaming configuration through extended filename is used. Any previous streaming configuration (ram value, streaming mode …) will be ignored.

WARNING: In /sen2agri/packaging/Sen2AgriPlatform/build/OTB-5.0/Modules/IO/ImageIO/include/otbImageFileWriter.txx, line 287
ImageFileWriter (0x12fa0d0): Streaming configuration through extended filename is used. Any previous streaming configuration (ram value, streaming mode …) will be ignored.

WARNING: In /sen2agri/packaging/Sen2AgriPlatform/build/OTB-5.0/Modules/IO/ImageIO/include/otbImageFileWriter.txx, line 287
ImageFileWriter (0x2713650): Streaming configuration through extended filename is used. Any previous streaming configuration (ram value, streaming mode …) will be ignored.

WARNING: In /sen2agri/packaging/Sen2AgriPlatform/build/OTB-5.0/Modules/IO/ImageIO/include/otbImageFileWriter.txx, line 287
ImageFileWriter (0x1479510): Streaming configuration through extended filename is used. Any previous streaming configuration (ram value, streaming mode …) will be ignored.

WARNING: In /sen2agri/packaging/Sen2AgriPlatform/build/OTB-5.0/Modules/IO/ImageIO/include/otbImageFileWriter.txx, line 287
ImageFileWriter (0x2986650): Streaming configuration through extended filename is used. Any previous streaming configuration (ram value, streaming mode …) will be ignored.

WARNING: In /sen2agri/packaging/Sen2AgriPlatform/build/OTB-5.0/Modules/IO/ImageIO/include/otbImageFileWriter.txx, line 287
ImageFileWriter (0x23ec6e0): Streaming configuration through extended filename is used. Any previous streaming configuration (ram value, streaming mode …) will be ignored.

WARNING: In /sen2agri/packaging/Sen2AgriPlatform/build/OTB-5.0/Modules/IO/ImageIO/include/otbImageFileWriter.txx, line 287
ImageFileWriter (0x2be5620): Streaming configuration through extended filename is used. Any previous streaming configuration (ram value, streaming mode …) will be ignored.

WARNING: In /sen2agri/packaging/Sen2AgriPlatform/build/OTB-5.0/Modules/IO/ImageIO/include/otbImageFileWriter.txx, line 287
ImageFileWriter (0x10dd7b0): Streaming configuration through extended filename is used. Any previous streaming configuration (ram value, streaming mode …) will be ignored.

WARNING: In /sen2agri/packaging/Sen2AgriPlatform/build/OTB-5.0/Modules/IO/ImageIO/include/otbImageFileWriter.txx, line 287
ImageFileWriter (0x2895bf0): Streaming configuration through extended filename is used. Any previous streaming configuration (ram value, streaming mode …) will be ignored.

WARNING: In /sen2agri/packaging/Sen2AgriPlatform/build/OTB-5.0/Modules/IO/ImageIO/include/otbImageFileWriter.txx, line 287
ImageFileWriter (0x1780660): Streaming configuration through extended filename is used. Any previous streaming configuration (ram value, streaming mode …) will be ignored.

WARNING: In /sen2agri/packaging/Sen2AgriPlatform/build/OTB-5.0/Modules/IO/ImageIO/include/otbImageFileWriter.txx, line 287
ImageFileWriter (0x19da660): Streaming configuration through extended filename is used. Any previous streaming configuration (ram value, streaming mode …) will be ignored.

WARNING: In /sen2agri/packaging/Sen2AgriPlatform/build/OTB-5.0/Modules/IO/ImageIO/include/otbImageFileWriter.txx, line 287
ImageFileWriter (0x18c17b0): Streaming configuration through extended filename is used. Any previous streaming configuration (ram value, streaming mode …) will be ignored.

WARNING: In /sen2agri/packaging/Sen2AgriPlatform/build/OTB-5.0/Modules/IO/ImageIO/include/otbImageFileWriter.txx, line 287
ImageFileWriter (0xd55d60): Streaming configuration through extended filename is used. Any previous streaming configuration (ram value, streaming mode …) will be ignored.

WARNING: In /sen2agri/packaging/Sen2AgriPlatform/build/OTB-5.0/Modules/IO/ImageIO/include/otbImageFileWriter.txx, line 287
ImageFileWriter (0x2c8db50): Streaming configuration through extended filename is used. Any previous streaming configuration (ram value, streaming mode …) will be ignored.

WARNING: In /sen2agri/packaging/Sen2AgriPlatform/build/OTB-5.0/Modules/IO/ImageIO/include/otbImageFileWriter.txx, line 287
ImageFileWriter (0x9d46f0): Streaming configuration through extended filename is used. Any previous streaming configuration (ram value, streaming mode …) will be ignored.

WARNING: In /sen2agri/packaging/Sen2AgriPlatform/build/OTB-5.0/Modules/IO/ImageIO/include/otbImageFileWriter.txx, line 287
ImageFileWriter (0x19e2d80): Streaming configuration through extended filename is used. Any previous streaming configuration (ram value, streaming mode …) will be ignored.

WARNING: In /sen2agri/packaging/Sen2AgriPlatform/build/OTB-5.0/Modules/IO/ImageIO/include/otbImageFileWriter.txx, line 287
ImageFileWriter (0x1686d80): Streaming configuration through extended filename is used. Any previous streaming configuration (ram value, streaming mode …) will be ignored.

WARNING: In /sen2agri/packaging/Sen2AgriPlatform/build/OTB-5.0/Modules/IO/ImageIO/include/otbImageFileWriter.txx, line 287
ImageFileWriter (0xa29e80): Streaming configuration through extended filename is used. Any previous streaming configuration (ram value, streaming mode …) will be ignored.

WARNING: In /sen2agri/packaging/Sen2AgriPlatform/build/OTB-5.0/Modules/IO/ImageIO/include/otbImageFileWriter.txx, line 287
ImageFileWriter (0x23a2140): Streaming configuration through extended filename is used. Any previous streaming configuration (ram value, streaming mode …) will be ignored.

WARNING: In /sen2agri/packaging/Sen2AgriPlatform/build/OTB-5.0/Modules/IO/ImageIO/include/otbImageFileWriter.txx, line 287
ImageFileWriter (0x1b4f5d0): Streaming configuration through extended filename is used. Any previous streaming configuration (ram value, streaming mode …) will be ignored.

WARNING: In /sen2agri/packaging/Sen2AgriPlatform/build/OTB-5.0/Modules/IO/ImageIO/include/otbImageFileWriter.txx, line 287
ImageFileWriter (0x2abedc0): Streaming configuration through extended filename is used. Any previous streaming configuration (ram value, streaming mode …) will be ignored.

WARNING: In /sen2agri/packaging/Sen2AgriPlatform/build/OTB-5.0/Modules/IO/ImageIO/include/otbImageFileWriter.txx, line 287
ImageFileWriter (0x202adc0): Streaming configuration through extended filename is used. Any previous streaming configuration (ram value, streaming mode …) will be ignored.

WARNING: In /sen2agri/packaging/Sen2AgriPlatform/build/OTB-5.0/Modules/IO/ImageIO/include/otbImageFileWriter.txx, line 287
ImageFileWriter (0x81589a0): Streaming configuration through extended filename is used. Any previous streaming configuration (ram value, streaming mode …) will be ignored.

WARNING: In /sen2agri/packaging/Sen2AgriPlatform/build/OTB-5.0/Modules/IO/ImageIO/include/otbImageFileWriter.txx, line 287
ImageFileWriter (0x665efe0): Streaming configuration through extended filename is used. Any previous streaming configuration (ram value, streaming mode …) will be ignored.

WARNING: In /sen2agri/packaging/Sen2AgriPlatform/build/OTB-5.0/Modules/IO/ImageIO/include/otbImageFileWriter.txx, line 287
ImageFileWriter (0x6c81610): Streaming configuration through extended filename is used. Any previous streaming configuration (ram value, streaming mode …) will be ignored.

WARNING: In /sen2agri/packaging/Sen2AgriPlatform/build/OTB-5.0/Modules/IO/ImageIO/include/otbImageFileWriter.txx, line 287
ImageFileWriter (0x75ad5e0): Streaming configuration through extended filename is used. Any previous streaming configuration (ram value, streaming mode …) will be ignored.

WARNING: In /sen2agri/packaging/Sen2AgriPlatform/build/OTB-5.0/Modules/IO/ImageIO/include/otbImageFileWriter.txx, line 287
ImageFileWriter (0x8440610): Streaming configuration through extended filename is used. Any previous streaming configuration (ram value, streaming mode …) will be ignored.

WARNING: In /sen2agri/packaging/Sen2AgriPlatform/build/OTB-5.0/Modules/IO/ImageIO/include/otbImageFileWriter.txx, line 287
ImageFileWriter (0x71a9b20): Streaming configuration through extended filename is used. Any previous streaming configuration (ram value, streaming mode …) will be ignored.

WARNING: In /sen2agri/packaging/Sen2AgriPlatform/build/OTB-5.0/Modules/IO/ImageIO/include/otbImageFileWriter.txx, line 287
ImageFileWriter (0x757c5e0): Streaming configuration through extended filename is used. Any previous streaming configuration (ram value, streaming mode …) will be ignored.

WARNING: In /sen2agri/packaging/Sen2AgriPlatform/build/OTB-5.0/Modules/IO/ImageIO/include/otbImageFileWriter.txx, line 287
ImageFileWriter (0x6d3cfe0): Streaming configuration through extended filename is used. Any previous streaming configuration (ram value, streaming mode …) will be ignored.

WARNING: In /sen2agri/packaging/Sen2AgriPlatform/build/OTB-5.0/Modules/IO/ImageIO/include/otbImageFileWriter.txx, line 287
ImageFileWriter (0x8520180): Streaming configuration through extended filename is used. Any previous streaming configuration (ram value, streaming mode …) will be ignored.

WARNING: In /sen2agri/packaging/Sen2AgriPlatform/build/OTB-5.0/Modules/IO/ImageIO/include/otbImageFileWriter.txx, line 287
ImageFileWriter (0x82f1830): Streaming configuration through extended filename is used. Any previous streaming configuration (ram value, streaming mode …) will be ignored.

WARNING: In /sen2agri/packaging/Sen2AgriPlatform/build/OTB-5.0/Modules/IO/ImageIO/include/otbImageFileWriter.txx, line 287
ImageFileWriter (0x8970dc0): Streaming configuration through extended filename is used. Any previous streaming configuration (ram value, streaming mode …) will be ignored.

WARNING: In /sen2agri/packaging/Sen2AgriPlatform/build/OTB-5.0/Modules/IO/ImageIO/include/otbImageFileWriter.txx, line 287
ImageFileWriter (0x718a430): Streaming configuration through extended filename is used. Any previous streaming configuration (ram value, streaming mode …) will be ignored.

WARNING: In /sen2agri/packaging/Sen2AgriPlatform/build/OTB-5.0/Modules/IO/ImageIO/include/otbImageFileWriter.txx, line 287
ImageFileWriter (0x75361f0): Streaming configuration through extended filename is used. Any previous streaming configuration (ram value, streaming mode …) will be ignored.

WARNING: In /sen2agri/packaging/Sen2AgriPlatform/build/OTB-5.0/Modules/IO/ImageIO/include/otbImageFileWriter.txx, line 287
ImageFileWriter (0x85c4ab0): Streaming configuration through extended filename is used. Any previous streaming configuration (ram value, streaming mode …) will be ignored.

WARNING: In /sen2agri/packaging/Sen2AgriPlatform/build/OTB-5.0/Modules/IO/ImageIO/include/otbImageFileWriter.txx, line 287
ImageFileWriter (0x76329d0): Streaming configuration through extended filename is used. Any previous streaming configuration (ram value, streaming mode …) will be ignored.

WARNING: In /sen2agri/packaging/Sen2AgriPlatform/build/OTB-5.0/Modules/IO/ImageIO/include/otbImageFileWriter.txx, line 287
ImageFileWriter (0x7d2b720): Streaming configuration through extended filename is used. Any previous streaming configuration (ram value, streaming mode …) will be ignored.

Attaching package: ‘dplyr’

The following objects are masked from ‘package:stats’:

filter, lag

The following objects are masked from ‘package:base’:

intersect, setdiff, setequal, union

Loading required package: lattice
Loading required package: ggplot2
Loading required package: proto
Warning message:
no DISPLAY variable so Tk is not available
── Attaching packages ─────────────────────────────────────── tidyverse 1.3.0 ──
tibble 3.0.3 stringr 1.4.0
tidyr 1.1.0 forcats 0.5.0
purrr 0.3.4
── Conflicts ────────────────────────────────────────── tidyverse_conflicts() ──
dplyr::filter() masks stats::filter()
dplyr::lag() masks stats::lag()
purrr::lift() masks caret::lift()

Attaching package: ‘data.table’

The following object is masked from ‘package:purrr’:

transpose

The following objects are masked from ‘package:dplyr’:

between, first, last

Warning message:
NAs introduced by coercion
Warning message:
NAs introduced by coercion
Error: Wrong argument: choose out of (1,2,3,4,5,0,‘All’,‘12’,‘123’,‘1234’)
Execution halted

(I deleted some lines of the Error-Message, as it was too long for the Post. However, they were all the same.)

As mentioned before, i run all the script you descriped in the earlier posts.
As a result, now the S1 products Coherence/Amplitude are shown with their correct location in the products tab.
I would be happy, if you could provide a solution for this problem.
Thank you very much!

Okke

lnicola · December 4, 2020, 1:04pm

Hi,

The real error seems to be Error: Wrong argument: choose out of (1,2,3,4,5,0,‘All’,‘12’,‘123’,‘1234’). This is probably a version mismatch between two different scripts. Can you attach you versions of /usr/bin/crop-type-wrapper.py and /usr/bin/crop_type.R?

Okke_Gerhard · December 7, 2020, 7:24am

@lnicola,
here are the files:
crop-type-wrapper and R.zip (9.3 KB)
Hope this will help.

Okke

lnicola · December 7, 2020, 2:37pm

The two files look fine. Can you try a manual run by running the L4A wrapper script? You should be able to see the command line next to the logs, it’s something like crop-type-wrapper.py -s 1 --season-start 2020-05-01 --season-end 2020-05-15. You can use a shorter season to speed up the processing.

Okke_Gerhard · December 7, 2020, 3:58pm

Dear @lnicola,
i run the command, however it appears it still does not work. I got following message:

1660 entries skipped
0 entries skipped
0 entries skipped
0 entries skipped
0 entries skipped
16156 entries skipped
0 entries skipped
0 entries skipped
merge-statistics sar-merged/mean-sar.csv sar-merged/dev-sar.csv sar-merged/SEN4CAP_L2A_PRD_S1_32UQB_MEAN.csv sar-merged/SEN4CAP_L2A_PRD_S1_32UQB_DEV.csv sar-merged/SEN4CAP_L2A_PRD_S1_32UQB_COUNT.csv sar-merged/SEN4CAP_L2A_PRD_S1_32UQC_MEAN.csv sar-merged/SEN4CAP_L2A_PRD_S1_32UQC_DEV.csv sar-merged/SEN4CAP_L2A_PRD_S1_32UQC_COUNT.csv sar-merged/SEN4CAP_L2A_PRD_S1_32UQA_MEAN.csv sar-merged/SEN4CAP_L2A_PRD_S1_32UQA_DEV.csv sar-merged/SEN4CAP_L2A_PRD_S1_32UQA_COUNT.csv sar-merged/SEN4CAP_L2A_PRD_S1_33UUS_MEAN.csv sar-merged/SEN4CAP_L2A_PRD_S1_33UUS_DEV.csv sar-merged/SEN4CAP_L2A_PRD_S1_33UUS_COUNT.csv sar-merged/SEN4CAP_L2A_PRD_S1_33UUR_MEAN.csv sar-merged/SEN4CAP_L2A_PRD_S1_33UUR_DEV.csv sar-merged/SEN4CAP_L2A_PRD_S1_33UUR_COUNT.csv sar-merged/SEN4CAP_L2A_PRD_S1_33UUT_MEAN.csv sar-merged/SEN4CAP_L2A_PRD_S1_33UUT_DEV.csv sar-merged/SEN4CAP_L2A_PRD_S1_33UUT_COUNT.csv sar-merged/SEN4CAP_L2A_PRD_S1_33UVS_MEAN.csv sar-merged/SEN4CAP_L2A_PRD_S1_33UVS_DEV.csv sar-merged/SEN4CAP_L2A_PRD_S1_33UVS_COUNT.csv sar-merged/SEN4CAP_L2A_PRD_S1_33UVT_MEAN.csv sar-merged/SEN4CAP_L2A_PRD_S1_33UVT_DEV.csv sar-merged/SEN4CAP_L2A_PRD_S1_33UVT_COUNT.csv
sh -c ‘cut -d, -f1-31 sar-merged/mean-sar.csv > sar-merged/mean-simple-sar.csv’
sh -c ‘cut -d, -f1,32- sar-merged/mean-sar.csv > sar-merged/mean-temporal-sar.csv’
sh -c ‘cut -d, -f1-31 sar-merged/dev-sar.csv > sar-merged/dev-simple-sar.csv’
sh -c ‘cut -d, -f1,32- sar-merged/dev-sar.csv > sar-merged/dev-temporal-sar.csv’
sh -c ‘cut -d, -f2- sar-merged/dev-simple-sar.csv > /tmp/tmpLVYZta.csv’
sh -c ‘paste -d, sar-merged/mean-simple-sar.csv /tmp/tmpLVYZta.csv >> sar-merged/sar-features.csv’
sh -c ‘cut -d, -f2- sar-merged/dev-temporal-sar.csv > /tmp/tmpVQKlO6.csv’
sh -c ‘paste -d, sar-merged/mean-temporal-sar.csv /tmp/tmpVQKlO6.csv >> sar-merged/sar-temporal.csv’
extract-parcels.py -s 1 -y 2020 ./parcels.csv ./lut.csv

        select short_name
        from site
        where id = 1


        select lpis."NewID",
            lpis."Area_meters",
            lpis."S1Pix",
            lpis."S2Pix",
            lut.ctnuml4a as "CTnumL4A",
            lut.ctl4a as "CTL4A",
            lut.lc
        from "decl_sachsen_2020_2020" lpis
        natural join "lut_sachsen_2020_2020" lut
        where "GeomValid"
        and not "Duplic"
        and not "Overlap"
        and not is_deleted
        order by "NewID"

select ctnuml4a as “CTnumL4A”, ctl4a as “CTL4A” from “lut_sachsen_2020_2020”
crop_type.R ./ ./features/sar-features.csv ./features/optical-features.csv ./features/optical-features-re.csv ./features/sar-temporal.csv ./parcels.csv CTnumL4A 1234 Area_meters 3 1 30 10 4000 1333 0.25 0.75 Smote 1000 5 300 10 ./lut.csv

Attaching package: ‘dplyr’

The following objects are masked from ‘package:stats’:

filter, lag

The following objects are masked from ‘package:base’:

intersect, setdiff, setequal, union

Loading required package: lattice
Loading required package: ggplot2
Loading required package: proto
── Attaching packages ─────────────────────────────────────── tidyverse 1.3.0 ──
tibble 3.0.3 stringr 1.4.0
tidyr 1.1.0 forcats 0.5.0
purrr 0.3.4
── Conflicts ────────────────────────────────────────── tidyverse_conflicts() ──
dplyr::filter() masks stats::filter()
dplyr::lag() masks stats::lag()
purrr::lift() masks caret::lift()

Attaching package: ‘data.table’

The following object is masked from ‘package:purrr’:

transpose

The following objects are masked from ‘package:dplyr’:

between, first, last

[1] “./” “./features/sar-features.csv”
[3] “./features/optical-features.csv” “./features/optical-features-re.csv”
[5] “./features/sar-temporal.csv” “./parcels.csv”
[7] “CTnumL4A” “1234”
[9] “Area_meters” “3”
[11] “1” “30”
[13] “10” “4000”
[15] “1333” “0.25”
[17] “0.75” “Smote”
[19] “1000” “5”
[21] “300” “10”
[23] “./lut.csv”
Warning message:
NAs introduced by coercion
Warning message:
NAs introduced by coercion
Error: Wrong argument: choose out of (1,2,3,4,5,0,‘All’,‘12’,‘123’,‘1234’)
Execution halted
Exit code: 1

I dont see any output beeing created in the corresponding folder.

lnicola · December 7, 2020, 4:36pm

That’s strange. Can you send me an archive with lut.csv, parcels.csv and features?

The CSVs only contain identifiers and names and the directory contains reflectance series without any geometry information, but if you feel more comfortable you can send it to me at lnicola@c-s.ro.

lnicola · December 8, 2020, 10:29am

For some reason I still can’t reproduce your issue. I get a different error instead:

Error in `levels<-`(`*tmp*`, value = as.character(levels)) : 
  factor level [12] is duplicated
Calls: factor
Execution halted

But that’s caused by a known problem (the same one that @Jonathan had earlier): your LUT contains duplicate CTnumL4A values.

I hope he won’t mind, but I’ll paste what I wrote him:

PS: to elaborate a little bit on what I meant by duplicate entries, the LUT allows you to group different classes into one for the purpose of the L4A processor. This way, if you have similar classes, they will be considered identical, for example:

 710030 | phaseolus vulgaris                  |     1122 | beans
 220000 | vicia faba                          |     1122 | beans

Or even:

 730000 | vegetable under glass and plastic   |     1153 | vegetable
 710009 | other vegetable                     |     1153 | vegetable
 710000 | vegetable                           |     1153 | vegetable

(which, by the way, might be a bad idea, because for the classifier, the vegetables covered with plastic will bear no resemblance with those that are uncovered)

The problem is with these entries:

 418370 | arable grass                        |     1136 | arable grass
 418400 | arable grass                        |     1137 | arable grass 
 950010 | not defined crop type               |     1191 | not defined crop type
 930400 | not defined crop type               |     1188 | not defined crop type
 790444 | not defined crop type               |     1185 | not defined crop type

These have different CTnumL4A values but the same CTL4A description. The script presumably tries to map from the descriptions to the codes when building the legend for a plot. It could probably be fixed, but I’m not really familiar with the R language and, unless you disagree, I propose changing those entries to use the same CTnumL4A value, e.g.:

 418370 | arable grass                        |     1136 | arable grass
 418400 | arable grass                        |     1136 | arable grass 
 950010 | not defined crop type               |     1185 | not defined crop type
 930400 | not defined crop type               |     1185 | not defined crop type
 790444 | not defined crop type               |     1185 | not defined crop type

Basically you have to update the LUT and re-upload it (or update it directly into the database). After that, you can re-run only the classification using:

 extract-parcels.py -s 1 -y 2020 ./parcels.csv ./lut.csv
 crop_type.R ./ ./features/sar-features.csv ./features/optical-features.csv ./features/optical-features-re.csv ./features/sar-temporal.csv ./parcels.csv CTnumL4A 1234 Area_meters 3 1 30 10 4000 1333 0.25 0.75 Smote 1000 5 300 10 ./lut.csv

lnicola · December 9, 2020, 8:56am

Dear @Okke_Gerhard,

You had two versions of the crop_type.R script – one installed by the system and one in /usr/local/bin. They differ in the parameter parsing (the latter had one parameter missing and force-set to 0). I suspect this was done to make a newer version of the R script work with an older version of the system.

I moved the modified file to ~eouser and verified that the classification is now running correctly.

Okke_Gerhard · December 9, 2020, 11:03am

Dear @lnicola,
thank you very much for the support. I just finished a first run of the L4A for a short periode of time. It now works and we will procceed with our testing, hopefully it will all work from this point on.

Okke