Upgrade from 1.2 to 1.3: sen2agri-orchestrator.service failed to start

System Installation and Configuration
1

Hello,

I previously installed 1.2 release and downloaded latest Sen4Cap release. I ran the update script:

sudo ./update.sh

And get the following message:

Updating R packages...
Job for sen2agri-orchestrator.service failed because a fatal signal was delivered to the control process. See "systemctl status sen2agri-orchestrator.service" and "journalctl -xe" for details.

As recommended, I ran:

systemctl status sen2agri-orchestrator.service

With the following result:

? sen2agri-orchestrator.service - Orchestrator for Sen2Agri
   Loaded: loaded (/usr/lib/systemd/system/sen2agri-orchestrator.service; enabled; vendor preset: disabled)
   Active: activating (auto-restart) (Result: signal) since Fri 2020-11-13 17:19:03 CET; 1s ago
  Process: 25393 ExecStart=/usr/bin/sen2agri-orchestrator (code=killed, signal=ABRT)
 Main PID: 25393 (code=killed, signal=ABRT)

And also ran:

sudo journalctl -xe

With the following result (sequence repeated):

Nov 13 17:22:22 va-esa-01-eu systemd[1]: sen2agri-orchestrator.service holdoff time over, scheduling restart.
Nov 13 17:22:22 va-esa-01-eu systemd[1]: Stopped Orchestrator for Sen2Agri.
-- Subject: Unit sen2agri-orchestrator.service has finished shutting down
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit sen2agri-orchestrator.service has finished shutting down.
Nov 13 17:22:22 va-esa-01-eu systemd[1]: Starting Orchestrator for Sen2Agri...
-- Subject: Unit sen2agri-orchestrator.service has begun start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit sen2agri-orchestrator.service has begun starting up.
Nov 13 17:22:22 va-esa-01-eu sen2agri-orchestrator[25682]: Reading settings from /etc/sen2agri/sen2agri.conf
Nov 13 17:22:22 va-esa-01-eu sen2agri-orchestrator[25682]: Invalid processor configuration found in database: l2-s1, igoring it as no handler is available for it!
Nov 13 17:22:22 va-esa-01-eu sen2agri-orchestrator[25682]: Invalid processor configuration found in database: lpis, igoring it as no handler is available for it!
Nov 13 17:22:22 va-esa-01-eu sen2agri-orchestrator[25682]: QThread: Destroyed while thread is still running
Nov 13 17:22:22 va-esa-01-eu systemd[1]: sen2agri-orchestrator.service: main process exited, code=killed, status=6/ABRT
Nov 13 17:22:22 va-esa-01-eu systemd[1]: Failed to start Orchestrator for Sen2Agri.
-- Subject: Unit sen2agri-orchestrator.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit sen2agri-orchestrator.service has failed.
--
-- The result is failed.
Nov 13 17:22:22 va-esa-01-eu systemd[1]: Unit sen2agri-orchestrator.service entered failed state.
Nov 13 17:22:22 va-esa-01-eu systemd[1]: sen2agri-orchestrator.service failed.

Did I miss something? How can I investigate or solve this issue?

Any help is welcome.

Thanks.

2

Hello,

Searching a bit more, I found a post in Sen2agri forum with similar issue: https://forum.esa-sen2agri.org/t/l3a-failed-to-start-orchestrator-for-sen2agri/320/5.

I removed jobs that were not yet processed from DB. Yet it did not solved the issue.

Thus I ran the strace and gdb commands on sen2agri-orchestrator and results in the following attachments:
orchestrator_gdb_2020_11_23_01.txt (6.8 KB)
orchestrator_strace_2020_11_23_01.txt (89.1 KB)

I’m not sure how to interpret those files but I hope someone can help!

Thanks.

3

Hello Guillaume,

Not sure if you solved this issue but if you did, we would be interested how you did it.
Otherwise, we noticed that sometimes the orchestrator does not start due to an error in DBus but seems that a simple restart of the machine (if possible) solves the issue.
Also, before restarting the machine you should go into the “System overview” tab and “pause” any L3B, L4A, L4B or L4C running at that moment (no need to pause the S1 pre-processing ones). You will have to resume them after the restart.

NOTE: The operation of pausing/resuming these jobs should be done also during system upgrades (before and after running update.sh)

Hope this helps.

Best regards,
Cosmin

4

Hi Cosmin,

Thanks for your reply.

I could not solve the issue yet. As a work around, I started to run processes manually.

There is a L3B process currently running that should be completed at some moment next week. Once completed, I will restart the server and let you know if I can restart the orchestrator then.

Kindest,
Guillaume.

5

Hi Cosmin,

I could restart the server today and also restart the orchestrator. There is no longer error. Thanks for your help.

Regards,
Guillaume.