L4B processor is not running

Grassland Mowing (L4B) processor
1

We are currently running a VM at Creodias to test whether the SEN4CAP toolbox is suitable for our purposes in operational use.

For this purpose, I have set up a test site across three tiles via the web interface. We are mainly interested in the L4B processor for the detection of mowing events, so I disabled all other processors. At first everything worked as expected and over a month all S1 (Amplitude, Coherence) & S2 (Atmospheric correction) data was preprocessed.

After that the processing stopped. The L4B processor was not completed successfully. Several error messages were displayed.

I was able to fix some of them already, e.g. with Cosmin’s patches. Thanks for that!

Unfortunately the L4B processor still terminates with different error messages. Both in the S1MowingDetection step and in the S2MowingDetection step.

error message S1MowingDetection:

Traceback (most recent call last):
File “/usr/share/sen2agri/S4C_L4B_GrasslandMowing/Bin/src_s1/S1_main.py”, line 852, in
main()
File “/usr/share/sen2agri/S4C_L4B_GrasslandMowing/Bin/src_s1/S1_main.py”, line 846, in main
do_cmpl=s4cConfig.do_cmpl, test=s4cConfig.test)
File “/usr/share/sen2agri/S4C_L4B_GrasslandMowing/Bin/src_s1/S1_main.py”, line 585, in main_run
compliancy_conf_file=configFile)
File “/usr/share/sen2agri/S4C_L4B_GrasslandMowing/Bin/src_s1/S1_main.py”, line 140, in run_proc
Layer = ogr_data.GetLayer(0)
AttributeError: ‘NoneType’ object has no attribute ‘GetLayer’

error message S2MowingDetection:

/home/sen2agri-service/miniconda3/envs/sen4cap/lib/python3.6/site-packages/scipy/ndimage/measurements.py:668: RuntimeWarning: invalid value encountered in true_divide
return sum / numpy.asanyarray(count).astype(numpy.float)
/usr/share/sen2agri/S4C_L4B_GrasslandMowing/Bin/src_s2/S2_main.py:413: RuntimeWarning: invalid value encountered in less
d[‘VI’][ d[‘VI’] < d[‘corrupted_th’] ] = np.nan
/usr/share/sen2agri/S4C_L4B_GrasslandMowing/Bin/src_s2/S2_main.py:514: RuntimeWarning: invalid value encountered in less
NDVI_det_cube[:,t] *= ((NDVI_last[:,1] - NDVI_last[:,0]) / (NDVI_last_doy[:,1] - NDVI_last_doy[:,0]) < decreasing_rate_th) # detection only if the decreasing is enough fast
/usr/share/sen2agri/S4C_L4B_GrasslandMowing/Bin/src_s2/S2_main.py:515: RuntimeWarning: invalid value encountered in less
NDVI_det_cube[:,t] *= ((NDVI_seg[:,t] - NDVI_last[:,1]) / (DOYList[t] - NDVI_last_doy[:,1]) < increasing_rate_th) # detection only if the successive VI increasing is not too fast
/usr/share/sen2agri/S4C_L4B_GrasslandMowing/Bin/src_s2/S2_main.py:516: RuntimeWarning: invalid value encountered in less
NDVI_det_cube[:,t] *= (NDVI_last[:,1] < high_abs_th)
/usr/share/sen2agri/S4C_L4B_GrasslandMowing/Bin/src_s2/S2_main.py:517: RuntimeWarning: invalid value encountered in greater
NDVI_det_cube[:,t] *= (NDVI_last[:,0] > low_abs_th)
/usr/share/sen2agri/S4C_L4B_GrasslandMowing/Bin/src_s2/S2_main.py:556: RuntimeWarning: invalid value encountered in greater
det_cube[det_cube>0] = S2_gmd.norm_fun(det_cube[det_cube>0], alpha, bounds=(0.5,1.0))
/usr/share/sen2agri/S4C_L4B_GrasslandMowing/Bin/src_s2/fusion.py:151: RuntimeWarning: invalid value encountered in greater
detections = confidence > 0
Traceback (most recent call last):
File “/usr/share/sen2agri/S4C_L4B_GrasslandMowing/Bin/src_s2/S2_main.py”, line 971, in
main()
File “/usr/share/sen2agri/S4C_L4B_GrasslandMowing/Bin/src_s2/S2_main.py”, line 965, in main
do_cmpl=s4cConfig.do_cmpl, test=s4cConfig.test)
File “/usr/share/sen2agri/S4C_L4B_GrasslandMowing/Bin/src_s2/S2_main.py”, line 740, in main_run
test=test)
File “/usr/share/sen2agri/S4C_L4B_GrasslandMowing/Bin/src_s2/S2_main.py”, line 569, in run_proc
compliancy.do_compliancy(outputShapeFile, compliance_config_filename=compliancy_conf_file)
File “/usr/share/sen2agri/S4C_L4B_GrasslandMowing/Bin/src_s2/compliancy.py”, line 50, in do_compliancy
assert len(cnt_crop_code) == len(cnt_crop_TR)
AssertionError

Probably I have a fundamental error in the configuration somewhere?

I would be very grateful for hints where I can/should search. I would also be happy to provide further information as soon as I know which ones could be helpful.

It would be a pity if I have to cancel the SEN4CAP test without results, I think the toolbox has great potential for us!

So thank you very much in advance for any help.
Dominik

2

Hello Dominik,

The last line in the S2 seem to indicate indeed an error in the config file, apparently in the [compliancy] section where the 3 fields (crop_codes, crop_time_intervals, crop_rule) seem to not have the same lengths. Could you please check this section or provide the configuration file that you use?
Also, as no output results shapefile results after S2 extraction, it is normal to have the above error in S1. The first thing would be to solve the S2 issue.

Best regards,
Cosmin

3

Hi Cosmin,

Thank you very much for the quick feedback and for pointing out where the error could be.

The fields all had the same length, but the error occurred because the S2_main.py lines 658-660 do not take into account the case that only one crop code could be given respectively I did not know that in this case for the field “crop_time_intervals” a different input format is necessary than for the other two fields.

I had only entered one crop code for testing purposes, which leads to ast.literal_eval interpreting my one date range as a tuple of two strings and not as a tuple of one tuple as in all cases where I enter more than one crop code. In line 50 of compliancy.py this is now interpreted as different lengths of the fields crop_codes and crop_time_intervals.

As a simple workaround it helps to add a “,” behind the one date range.

Is there actually an instruction/documentation for the configuration files? If so, I could need it in general, because I am not sure about the meaning of every field. And maybe you could also place a note in this documentation (if existing) about the required format with the closing comma or implement the case with only one crop code in the S2_main.py?

Now the L4B processor finally ran through and I can continue to experiment with it! :slight_smile: So thanks again for your help!

Is there a way to get the underlying S1 & S2 time series to run your own analyses on it, too?