Dear Sen4Cap,
I have upgraded successfully to V1.1 but all site information are gone including postgres db. Generated files are still there though. Is there any way to recover data including postgres db?
Thanks and Regards,
Henry
Hello Henry,
Could you please provide us the messages given during the update?
Also can you please check if you have the script /tmp/db.sql?
If yes, please run :
psql -U postgres -f /tmp/db.sql
You can try after that running again the ./update.sh script and see if there is any other error.
Best regards,
Cosmin
Hello Cosmin,
Here is the message I got when I updated. there’s db.sql file under tmp file but there’s no content in it. Please see attached screenshot.
Thanks and Regards,
Henry
- DONE (tidyverse)
Making ‘packages.html’ … done
The downloaded source packages are in
‘/tmp/RtmpOw0RyT/downloaded_packages’
Updating HTML index of packages in ‘.Library’
Making ‘packages.html’ … done
Setting report_errors to false…
Traceback (most recent call last):
File “/home/sen2agri-service/miniconda3/bin/conda”, line 7, in
from conda.cli import main
ModuleNotFoundError: No module named ‘conda’
Traceback (most recent call last):
File “/home/sen2agri-service/miniconda3/bin/conda”, line 7, in
from conda.cli import main
ModuleNotFoundError: No module named ‘conda’
Creating sen4cap conda environment …
Traceback (most recent call last):
File “/home/sen2agri-service/miniconda3/bin/conda”, line 7, in
from conda.cli import main
ModuleNotFoundError: No module named ‘conda’
Printing current environments …
Traceback (most recent call last):
File “/home/sen2agri-service/miniconda3/bin/conda”, line 7, in
from conda.cli import main
ModuleNotFoundError: No module named ‘conda’
Job for sen2agri-orchestrator.service failed because the control process exited with error code. See “systemctl status sen2agri-orchestrator.service” and “journalctl -xe” for details.
[root@sen2cap install_script]# systemctl status sen2agri-orchestrator.service
● sen2agri-orchestrator.service - Orchestrator for Sen2Agri
Loaded: loaded (/usr/lib/systemd/system/sen2agri-orchestrator.service; enabled; vendor preset: disabled)
Active: activating (auto-restart) (Result: exit-code) since Tue 2020-06-09 05:20:05 UTC; 3s ago
Process: 19202 ExecStart=/usr/bin/sen2agri-orchestrator (code=exited, status=1/FAILURE)
Main PID: 19202 (code=exited, status=1/FAILURE)
Jun 09 05:20:05 sen2cap systemd[1]: Failed to start Orchestrator for Sen2Agri.
Jun 09 05:20:05 sen2cap systemd[1]: Unit sen2agri-orchestrator.service entered failed state.
Jun 09 05:20:05 sen2cap systemd[1]: sen2agri-orchestrator.service failed.
Hello,
It’s a bit involved, but there might be a way for you to recover the old database.
- check that the old database still exists:
ls -ld /var/lib/{pgsql,postgresql}
- check that Docker was installed:
sudo docker ps
- prepare a container with Postgres 9.4
mkdir pg94
cd pg94
vim Dockerfile
and paste the following:
FROM centos:7
RUN yum install -y epel-release && \
yum update -y epel-release && \
yum install -y http://yum.postgresql.org/9.4/redhat/rhel-7.3-x86_64/pgdg-centos94-9.4-3.noarch.rpm && \
yum install -y postgresql94-server postgresql94-contrib postgis22_94
- build the container
docker build -t pg94 .
- start it
$ sudo docker run --rm -it -v /var/lib/pgsql:/var/lib/postgresql -p 5433:5432 -u 26:26 pg94 /usr/pgsql-9.4/bin/postgres -D /var/lib/postgresql/9.4/data/
< 2020-06-10 13:43:13.384 EEST >LOG: redirecting log output to logging collector process
< 2020-06-10 13:43:13.384 EEST >HINT: Future log output will appear in directory "pg_log".
- in another console, make a backup:
pg_dump -h 127.0.0.1 -p 5433 -U admin -f sen4cap.sql sen4cap
Hello Inicola,
I have tried out your instruction and it was working well until building docker. Unfortunately, it’s stuck when i run the command to restore. I have attached the result and error I got. How can I solve it out?
Thanks and Regards,
Henry
I’ve never seen that happen. Does psql -h 127.0.0.1 -p 5433 -U admin sen4cap work? And does /var/lib/pgsql/9.4/data/pg_log (I think?) contain any errors?
that doesn’t work. Here is the error log I got under pg_log folder. Let me know if you need further info.
Thanks and Regards,
Henry
[root@sen2cap pg_log]# ll
total 740
-rw-------. 1 postgres postgres 143642 Jun 5 23:35 postgresql-Fri.log
-rw-------. 1 postgres postgres 88395 Jun 8 13:38 postgresql-Mon.log
-rw-------. 1 postgres postgres 44610 Jun 6 22:22 postgresql-Sat.log
-rw-------. 1 postgres postgres 87733 Jun 7 23:23 postgresql-Sun.log
-rw-------. 1 postgres postgres 205408 Jun 4 23:23 postgresql-Thu.log
-rw-------. 1 postgres postgres 326 Jun 9 05:10 postgresql-Tue.log
-rw-------. 1 postgres postgres 170420 Jun 10 14:14 postgresql-Wed.log
[root@sen2cap pg_log]# tail postgresql-Wed.log -f
< 2020-06-03 23:23:44.406 UTC >ERROR: invalid input syntax for type date: “V”
< 2020-06-03 23:23:44.406 UTC >STATEMENT: SELECT dh.id, dh.site_id, dh.satellite_id, dh.product_name, dh.full_path, dh.created_timestamp, dh.status_id, dh.no_of_retries, dh.product_date, dh.orbit_id, dh.status_reason, dh.tiles, dh.footprint, dh.orbit_type_id FROM public.downloader_history dh JOIN (SELECT d2.product_name, i.product_name AS intr, d2.site_id, d2.orbit_id, d2.satellite_id FROM public.downloader_history d2 JOIN public.downloader_history i ON i.site_id = d2.site_id AND i.orbit_id = d2.orbit_id AND i.satellite_id = d2.satellite_id WHERE ST_INTERSECTS(i.footprint, d2.footprint) AND DATE_PART(‘day’, d2.product_date - i.product_date) BETWEEN $1 AND $2 AND d2.satellite_id = 3) AS intersections ON intersections.site_id = dh.site_id AND intersections.orbit_id = dh.orbit_id AND intersections.satellite_id = dh.satellite_id AND intersections.product_name = dh.product_name LEFT JOIN (SELECT full_path, name, site_id, orbit_id, satellite_id, inserted_timestamp, GREATEST(substr(split_part(name, ‘’, 6), 2, 15)::date, substr(split_part(name, '’, 7), 1, 15)::date) as master_date FROM public.product) AS p ON p.site_id = dh.site_id AND p.orbit_id = dh.orbit_id AND p.satellite_id = dh.satellite_id AND CAST(dh.product_date as date) = p.master_date WHERE dh.satellite_id = 3 AND dh.site_id = $3 AND dh.status_id = 5 AND p.full_path IS NULL ORDER BY dh.product_date, dh.orbit_id, dh.product_name
< 2020-06-03 23:23:44.462 UTC >ERROR: invalid input syntax for type date: “V”
< 2020-06-03 23:23:44.462 UTC >STATEMENT: SELECT dh.id, dh.site_id, dh.satellite_id, dh.product_name, dh.full_path, dh.created_timestamp, dh.status_id, dh.no_of_retries, dh.product_date, dh.orbit_id, dh.status_reason, dh.tiles, dh.footprint, dh.orbit_type_id FROM public.downloader_history dh JOIN (SELECT d2.product_name, i.product_name AS intr, d2.site_id, d2.orbit_id, d2.satellite_id FROM public.downloader_history d2 JOIN public.downloader_history i ON i.site_id = d2.site_id AND i.orbit_id = d2.orbit_id AND i.satellite_id = d2.satellite_id WHERE ST_INTERSECTS(i.footprint, d2.footprint) AND DATE_PART(‘day’, d2.product_date - i.product_date) BETWEEN $1 AND $2 AND d2.satellite_id = 3) AS intersections ON intersections.site_id = dh.site_id AND intersections.orbit_id = dh.orbit_id AND intersections.satellite_id = dh.satellite_id AND intersections.product_name = dh.product_name LEFT JOIN (SELECT full_path, name, site_id, orbit_id, satellite_id, inserted_timestamp, GREATEST(substr(split_part(name, ‘’, 6), 2, 15)::date, substr(split_part(name, '’, 7), 1, 15)::date) as master_date FROM public.product) AS p ON p.site_id = dh.site_id AND p.orbit_id = dh.orbit_id AND p.satellite_id = dh.satellite_id AND CAST(dh.product_date as date) = p.master_date WHERE dh.satellite_id = 3 AND dh.site_id = $3 AND dh.status_id = 5 AND p.full_path IS NULL ORDER BY dh.product_date, dh.orbit_id, dh.product_name
< 2020-06-03 23:35:54.068 UTC >ERROR: invalid input syntax for type date: “V”
< 2020-06-03 23:35:54.068 UTC >STATEMENT: SELECT dh.id, dh.site_id, dh.satellite_id, dh.product_name, dh.full_path, dh.created_timestamp, dh.status_id, dh.no_of_retries, dh.product_date, dh.orbit_id, dh.status_reason, dh.tiles, dh.footprint, dh.orbit_type_id FROM public.downloader_history dh JOIN (SELECT d2.product_name, i.product_name AS intr, d2.site_id, d2.orbit_id, d2.satellite_id FROM public.downloader_history d2 JOIN public.downloader_history i ON i.site_id = d2.site_id AND i.orbit_id = d2.orbit_id AND i.satellite_id = d2.satellite_id WHERE ST_INTERSECTS(i.footprint, d2.footprint) AND DATE_PART(‘day’, d2.product_date - i.product_date) BETWEEN $1 AND $2 AND d2.satellite_id = 3) AS intersections ON intersections.site_id = dh.site_id AND intersections.orbit_id = dh.orbit_id AND intersections.satellite_id = dh.satellite_id AND intersections.product_name = dh.product_name LEFT JOIN (SELECT full_path, name, site_id, orbit_id, satellite_id, inserted_timestamp, GREATEST(substr(split_part(name, ‘’, 6), 2, 15)::date, substr(split_part(name, '’, 7), 1, 15)::date) as master_date FROM public.product) AS p ON p.site_id = dh.site_id AND p.orbit_id = dh.orbit_id AND p.satellite_id = dh.satellite_id AND CAST(dh.product_date as date) = p.master_date WHERE dh.satellite_id = 3 AND dh.site_id = $3 AND dh.status_id = 5 AND p.full_path IS NULL ORDER BY dh.product_date, dh.orbit_id, dh.product_name
< 2020-06-10 14:14:02.326 UTC >LOG: database system was shut down at 2020-06-09 05:10:14 UTC
< 2020-06-10 14:14:02.466 UTC >LOG: MultiXact member wraparound protections are now enabled
< 2020-06-10 14:14:02.468 UTC >LOG: database system is ready to accept connections
< 2020-06-10 14:14:02.468 UTC >LOG: autovacuum launcher started
The ERROR: invalid input syntax for type date: “V” messages are unrelated and I see no other error. What error do you get when you run psql?
Strange. No other errors.
Here is the error of running psql.
[root@sen2cap ~]# psql -h 127.0.0.1 -p 5433 -U admin sen4cap
psql: error: could not connect to server: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
[root@sen2cap ~]#
Here is the error log from pg_log folder.
[root@sen2cap pg_log]# tail postgresql-Wed.log | grep “ERROR”
< 2020-06-03 23:23:44.406 UTC >ERROR: invalid input syntax for type date: “V”
< 2020-06-03 23:23:44.462 UTC >ERROR: invalid input syntax for type date: “V”
< 2020-06-03 23:35:54.068 UTC >ERROR: invalid input syntax for type date: “V”
Am I missing something?
Thanks and Regards,
Henry
I’m not sure. It might be a version conflict (although connecting to Postgres 9.4 with psql 12 works fine for me) or a problem like file corruption. If you can send me the contents of /var/lib/pgsql/9.4/ as an archive (you’ll need to create it as the postgres user or root), I’ll take a look.
Thanks for your prompt support. Here is a link to download the content. Let me know if you need more import.
Regards,
Henry
Thanks, I’ll take a look tomorrow. You can remove the file from Dropbox now.
It almost worked when I tried it. I couldn’t connect to the instance, but I also got this hint in the Postgres logs:
$ docker run --rm -it -v $PWD/var/lib/pgsql:/var/lib/postgresql -p 5433:5432 -u 26:26 pg94 /usr/pgsql-9.4/bin/postgres -D /var/lib/postgresql/9.4/data/
< 2020-06-11 05:57:14.881 UTC >LOG: could not bind IPv6 socket: Cannot assign requested address
< 2020-06-11 05:57:14.881 UTC >HINT: Is another postmaster already running on port 5432? If not, wait a few seconds and retry.
< 2020-06-11 05:57:14.901 UTC >LOG: redirecting log output to logging collector process
< 2020-06-11 05:57:14.901 UTC >HINT: Future log output will appear in directory "pg_log".
To fix that, I edited /var/lib/postgresql/9.4/data/postgresql.conf to set listen_addresses to * (it’s localhost by default).
I’ll send you a database dump in a private message, but you might still need help restoring it and applying the schema migrations for the latest version.
1 Like
Hello Inicola,
Thanks a lot. Db backup is well received. I think this port is using by another docker if i am not wrong. That could be the reason it didn’t work out. Please see the screenshot for docker list.
What’s the next step to restore db to match with new schema?
Thansk and Regards,
Henry
I think this port is using by another docker if i am not wrong. That could be the reason it didn’t work out.
I don’t think so. The first port (5433) is on the host, while the second (5432) is in the container (guest). I didn’t check, but I would expect them to run in different network namespaces, so it’s fine if two containers use the same port as long as the host uses different ones.
What’s the next step to restore db to match with new schema?
First of all, you should stop the running services as described in the manual. Then (untested):
psql -U postgres -c "alter database sen4cap rename to sen4cap_bad"
pg_restore -U postgres -f db.sql -d sen4cap
Then run the update script and restart the services.
1 Like
Hello Inicola,
Database is successfully restored by using psql. It seems upgraded to V1.1 but I still got same conda related error. Please see attached photo.
Thanks and Regards,
Henry
Dear Henry,
Could you please check the disk space in /home? Is there any disk space available there?
My assumption is that miniconda did not completely installed (by default in /home/sen2agri-service).
Normally, the miniconda occupies around 5.2 GB of disk space so if you don’t have this space in /home I will send you the updated script to install it in /mnt/archive
Please let me know.
Best regards,
Cosmin
Dear Cosmin,
I have checked disk space and it still left plenty as below.
[root@sen2cap ~]# df -h /home
Filesystem Size Used Avail Use% Mounted on
/dev/sda2 3.0T 2.8T 212G 93% /
Looking forward to trying out the updated script.
Thanks and Regards,
Henry
Dear Henry,
Apparently, you have enough space on /home. Could you try removing the directory /home/sen2agri-service/miniconda3 and try again with update.sh?
Normally, the update script checks if miniconda was already installed but if somehow it was incompletely install, could bring to the behaviour you have.
Please let me know.
Best regards,
Cosmin
1 Like
Dear Cosmin,
Thank you for your instruction. I will run that out.
Regards,
Henry