L4 jobs submitted but not running after update to 2.0

System Installation and Configuration
#1

Hi,

we are running Sen4CAP at CreoDIAS. We started with the prepared image of version 1.3 and recently followed the instructions in the manual to update to version 2.0. The update appears to have been successful, system works and UI now contains the S4C marker database job.

When we try to run this job via the custom job tab it shows up in job history and system overview, but the status is “Submitted” and none of the steps are running. We are testing it on a relatively small site, and the status of the job (and any of the steps) has not changed for 12 hours already. When we run L4A or L4B jobs they too show up as submitted, but nothing appears to be going on.

At the same time, the CPU utilization is full even though none of the jobs seem to be running. S1 preprocessing jobs get triggered automatically and finish successfully, but show no steps in the job history.

Do you have any suggestions what could be wrong? Or is it working as it should?
Thanks in advance.

Adam

1 Like
#2

Hello,

Could you try performing the following operations:

sudo su -l sen2agri-service
srun ls -al

Is the operation successful and shows the files in the current directory?
If not, you can try execute the script Sen4CAPDistribution/install_script/install_slurm_only.sh

sudo ./install_slurm_only.sh

This will reinstall slurm but you will have to relaunch the processors again.

Hope this helps.

Best regards,
Cosmin

#3

Hello,

thanks for the advice, but it didn’t help. The ls command ran successfully, and I reinstalled slurm just in case, with no apparent effect.

When I open the steps in job history and try to run any of them manually (the docker commands) as sen2agri-service user, they work without any errors and produce outputs as expected. The job itself has still status “submitted” and none of the steps are being executed.

Adam

#4

Could you please also check that the sen2agri-orchestrator and sen2agri-executor are running properly and if needed, start them?

Cosmin

2 Likes
#5

I checked the status and the sen2agri-executor was inactive, I started it and the processing is running now.
I presumed all of these services get started automatically at boot (and I have rebooted the server a couple times), so it didn’t occur to me to check this. Still not sure why wasn’t it running, but I will know what to check if it happens again.

Thank you very much for your help!

Adam

#6

Actually, in the previous versions sen2agri-executor was not enabled but starting with this version it should be, otherwise after a reboot it will not automatically start.
I think doing :

sudo systemctl enable sen2agri-executor

Will solve the issue also after rebooting the machine.

Cosmin

1 Like