SEN4CAp 3.2: Error Preprocessing - unable to download orbit files

1

Hello,

We are currently having an issue with Sen4CAP downloading some reference data durihng preprocessing. We have a Sen4CAP installation in 2 different environments:

  • Sandbox, without proxy or firewall
  • Acceptance, with proxy and firewall

We are using Sen4CAP 3.2.

In both environments we only use Sentinel 1 and 2 satellite data which are downloaded from an LSA Datasource:

Sentinel 2 - LSA Data Center
Scope : Query and Download
Fetch Mode: Overwrite
Max connection: 1
Download path: /mnt/archive/dwn_def/s2/default
Max Retries: 72
Retry Interval: 60
User: ********
Password: **********

Sentinel 1 - LSA Data Center
Scope : Query and Download
Fetch Mode: Overwrite
Max connection: 1
Download path: /mnt/archive/dwn_def/s1/default
Max Retries: 72
Retry Interval: 60
User: ********
Password: **********

** In the Acceptance environment, there are several configurations:

  • At the CentOS level, the environment variables are (/etc/environment):
    http_proxy=http://some-proxy:8080
    https_proxy=http://some-proxy:8080
    ftp_proxy=http://some-proxy:8080
    no_proxy=localhost,127.0.0.0,127.0.1.1,127.0.1.1,[some internal services and repositories]]
    HTTP_PROXY=http://some-proxy:8080
    HTTPS_PROXY=http://some-proxy:8080

  • In Sen4CAP configuration file (/usr/share/sen2agri/sen2agri-services/config/services.properties)
    …
    proxy.host=some-proxy
    proxy.port=8080
    proxy.user=
    proxy.password=
    proxy.type=HTTP
    …

  • For Docker configuration (/etc/systemd/system/docker.service.d/http-proxy.conf):
    …
    [Service]
    Environment=“HTTPS_PROXY=http://some-proxy:8080”
    …

** In Sandbox environment, there is no specific configuration

In both environments, the download of Sentinel 1 and 2 images works correctly. But the S1 and S2 pre-processing does not work in the Acceptance environment.
It seems it is related to the download of Orbit files:

Jun 14 08:51:30 MA-DEV start.sh[36382]: INFO: org.esa.s2tbx.dataio.gdal.GDALVersion: Internal GDAL 3.0.0 set to be used by SNAP.
Jun 14 08:51:31 MA-DEV start.sh[36382]: 2023-Jun-14 08:51:31.253 TRACE [process-exec-2] r.c.t.utils.executors.Executor - Executing processing graph
Jun 14 08:51:33 MA-DEV start.sh[36382]: INFO: org.hsqldb.persist.Logger: dataFileCache open start
Jun 14 08:51:54 MA-DEV start.sh[36382]: WARNING: org.esa.s1tbx.orbits.gpf.ApplyOrbitFileOp: No valid orbit file found for 26-JAN-2022 17:23:22.419728
Jun 14 08:51:54 MA-DEV start.sh[36382]: Orbit files may be downloaded from https://scihub.copernicus.eu/gnss/odata/v1/
Jun 14 08:51:54 MA-DEV start.sh[36382]: and placed in /home/.snap/auxdata/Orbits/Sentinel-1/POEORB/S1A/2022/01
Jun 14 08:51:54 MA-DEV start.sh[36382]: 2023-Jun-14 08:51:54.541 TRACE [process-exec-2] r.c.t.utils.executors.Executor - OpenSearch: https://scihub.copernicus.eu/gnss/search?q=platformname:Sentinel-1 AND platformnumber:A AND producttype:AUX_RESORB AND beginposition:[2022-01-01T00:00:000Z TO 2022-01-31T24:00:000Z]
Jun 14 08:54:01 MA-DEV start.sh[36382]: 281529 [main] INFO org.apache.commons.httpclient.HttpMethodDirector - I/O exception (java.net.ConnectException) caught when processing request: Connection timed out (Connection timed out)
Jun 14 08:54:01 MA-DEV start.sh[36382]: 281529 [main] INFO org.apache.commons.httpclient.HttpMethodDirector - Retrying request
Jun 14 08:56:09 MA-DEV start.sh[36382]: 408761 [main] INFO org.apache.commons.httpclient.HttpMethodDirector - I/O exception (java.net.ConnectException) caught when processing request: Connection timed out (Connection timed out)
Jun 14 08:56:09 MA-DEV start.sh[36382]: 408761 [main] INFO org.apache.commons.httpclient.HttpMethodDirector - Retrying request

And also for Elevation files:

Jun 13 11:27:33 MA-DEV start.sh[1915]: SEVERE: org.esa.snap.core.dataop.dem.ElevationFile: Connection timed out (Connection timed out)
Jun 13 11:27:33 MA-DEV start.sh[1915]: INFO: org.esa.snap.core.dataop.dem.ElevationFile: http retrieving http://step.esa.int/auxdata/dem/SRTMGL1/N50E006.SRTMGL1.hgt.zip
Jun 13 11:27:35 MA-DEV start.sh[1915]: SEVERE: org.esa.snap.core.dataop.dem.ElevationFile: Connection timed out (Connection timed out)
Jun 13 11:27:35 MA-DEV start.sh[1915]: INFO: org.esa.snap.core.dataop.dem.ElevationFile: http retrieving http://step.esa.int/auxdata/dem/SRTMGL1/N50E007.SRTMGL1.hgt.zip

Connection timeout is maybe due to proxy/firewall is filtering the request. Indeed, network admin team indicates that request are sent out of HTTP proxy depite the configuration made.

  1. Can you confirm that https://scihub.copernicus.eu or http://step.esa.int should be made through proxy thanks to the Sen4CAP proxy configuration? (It is actually working for image download) Or there any additional configuration to be made?
    A curl request from CentOS command line shows that access is allowed by proxy/firewall in Acceptance environment.

  2. How is Sen4CAP able to access to this resource without SciHub credentials? Isn’t it supposed to be password protected?

  3. Do you have a list of sites that need to be accessed by SEN4CAP during operation ( preprocessing/ processors jobs execution)?

Thank you in advanced for your support.

Best regards,

Marta

2

Dear Marta,

Sen4CAP is using internally SNAP and the messages you displayed are actually from SNAP gpt which is running from a docker container. I am not an expert on how SNAP is trying internally to retrieve its orbit files but what I can confirm is that the step.esa.int/auxdata/dem/ files can be accessed without the need of an user and a password. On this subject, I think you can search or ask for more details directly on SNAP forum (here No response to regular proxy configuration mechanisms of Java - #3 by marpet - Problem Reports - STEP Forum seems to be a thread similar with your issue).
Concerning the proxy, unfortunately, in this moment, the proxy configuration that you added into services.properties is not passed also to the SNAP gpt (that tries itself connecting to some external sites, out of our control). We will try to correct this in a future version but it might take some time until the new release is ready.
What you can try in this moment, you can create a new SNAP docker image locally on the machine with proxy configuration following the steps:

  • Into the installation package, go into the directory Sen4CAPDistribution/install_script/docker/snap8
  • Edit the Dockerfile in order to add the following lines (you will have to replace the NNNN, MMMM and https.proxy.ip with your own values):

RUN sed -i ‘$a-Dhttp.proxyHost=http.proxy.ip’ /opt/snap/bin/gpt.vmoptions
RUN sed -i ‘$a-Dhttp.proxyPort=NNNN’ /opt/snap/bin/gpt.vmoptions
RUN sed -i ‘$a-Dhttps.proxyHost=https.proxy.ip’ /opt/snap/bin/gpt.vmoptions
RUN sed -i ‘$a-Dhttps.proxyPort=MMMM’ /opt/snap/bin/gpt.vmoptions

right after the line:

RUN sed -i ‘$a-Dsnap.extraClusters=/opt/snap/extra’ /opt/snap/bin/gpt.vmoptions

  • create and copy into the same directory a script like the one attached (you can change the extension to .sh to the file I am attaching):
    build.txt (1.1 KB)

  • run the script build.sh (with an user being in docker group) and after it finishes, you should have a new image “sen4cap/snap:8.0-proxy”

  • edit /usr/share/sen2agri/sen2agri-services/config/services.properties to change the snap image name to the new one:

docker.snap.image = sen4cap/snap:8.0-proxy

  • Restart the sen2agri-services with :

sudo systemctl restart sen2agri-services

Hope this works.

Best regards,
Cosmin