Commit Graph

132 Commits

Author SHA1 Message Date
Patrick C. F. Ernzer
65182ea0f2 fixes #1365
docker-ce does not like quotes around the shm-size value
2021-10-30 13:13:30 +02:00
Slavi Pantaleev
fc593bc8ab Increase shm-size for Postgres container to fix VACUUM
Related to https://github.com/spantaleev/matrix-docker-ansible-deploy/issues/1268

Related to https://github.com/spantaleev/matrix-docker-ansible-deploy/issues/1364

Fixes https://github.com/spantaleev/matrix-docker-ansible-deploy/issues/1362
2021-10-30 09:53:56 +03:00
Slavi Pantaleev
09ac950d17 Fix dump importing (backup restore) into Postgres v14
In short, the problem is that older Postgres versions store passwords
hashed as md5. When you dump such a database, the dump naturally also
contains md5-hashed passwords.
Restoring from that dump used to create users and updates their passwords
with these md5 hashes.
However, Postgres v14 prefers does not like md5-hashed passwords now (by default),
which breaks connectivity. Postgres v14 prefers `scram-sha-256` for
authentication.

Our solution is to just ignore setting passwords (`ALTER ROLE ..`
statements) when restoring dumps. We don't need to set passwords as
defined in the dump anyway, because the playbook creates users
and manages their passwords by itself.

Fixes https://github.com/spantaleev/matrix-docker-ansible-deploy/issues/1340
2021-10-21 16:38:56 +03:00
Slavi Pantaleev
096c960b84 Add support for Postgres v14 2021-10-01 11:27:40 +03:00
sakkiii
48548eb561
Postgres Minor Updates 2021-08-22 18:45:25 +05:30
sakkiii
d338090f00
postgres minor updates 2021-06-30 10:00:52 +05:30
pushytoxin
bee14550ab Fix local/bin scripts autocompletion by adding rx perms to everyone
It's mildly annoying when trying to execute these scripts while logged
in as a regular user, as the missing execute permissions will hinder
autocompletion even when trying to use with sudo.

These shell scripts don't contain secrets, but may fail when ran by a
regular user. The failure is due to the lack of access to the /matrix
directory, and does not result in any damage.
2021-05-28 10:39:27 +02:00
Slavi Pantaleev
1ab507349c Fix self-building for various components on Ansible < 2.8
Fixes https://github.com/spantaleev/matrix-docker-ansible-deploy/issues/1070
2021-05-20 08:43:20 +03:00
Ahmad Haghighi
e335f3fc77 rename matrix_global_registry to matrix_container_global_registry_prefix related to #990
Signed-off-by: Ahmad Haghighi <haghighi@fedoraproject.org>
2021-04-12 17:23:55 +04:30
Ahmad Haghighi
f52a8b6484 use custom docker registry 2021-04-12 17:23:55 +04:30
Slavi Pantaleev
108aed53be Fix invalid matrix-postgres.service when matrix_postgres_process_extra_arguments is empty
This only seems to be affecting some people badly enough to cause
matrix-postgres not to start. Certain systemd versions probably handle
it better or something.

Fixes https://github.com/spantaleev/matrix-docker-ansible-deploy/issues/889
(hopefully)
2021-02-19 16:33:23 +02:00
Slavi Pantaleev
1dbdfeec07 Fix matrix-postgres stopping for consistency with other services
This probably got lost somehow in all the work that happened in
https://github.com/spantaleev/matrix-docker-ansible-deploy/pull/456
2021-02-19 15:53:30 +02:00
Slavi Pantaleev
5cfeae806b Merge branch 'master' into synapse-workers 2021-02-14 13:00:57 +02:00
Slavi Pantaleev
7d39e5153a Upgrade Postgres minor versions 2021-02-14 09:12:29 +02:00
Julian Foad
d1f28d17bb Allow psql args to be given to matrix-postgres-cli
This passes any arguments given to 'matrix-postgres-cli' to the 'psql' command.

Examples:
  $ # start an interactive shell connected to a given db
  $ sudo matrix-postgres-cli -d synapse
  $ # run a query, non-interactively
  $ sudo matrix-postgres-cli -d synapse -c 'SELECT group_id FROM groups;'
2021-02-03 12:59:21 +00:00
o8F0LY
0a0c9a4efc Add double quotes to avoid synatx errors 2021-01-30 22:54:51 +01:00
Slavi Pantaleev
d98a1ceadd Merge branch 'master' into synapse-workers 2021-01-27 10:27:17 +02:00
Slavi Pantaleev
512f42aa76 Do not report docker kill/rm attempts as errors
These are just defensive cleanup tasks that we run.
In the good case, there's nothing to kill or remove, so they trigger an
error like this:

> Error response from daemon: Cannot kill container: something: No such container: something

and:

> Error: No such container: something

People often ask us if this is a problem, so instead of always having to
answer with "no, this is to be expected", we'd rather eliminate it now
and make logs cleaner.

In the event that:
- a container is really stuck and needs cleanup using kill/rm
- and cleanup fails, and we fail to report it because of error
suppression (`2>/dev/null`)

.. we'd still get an error when launching ("container name already in use .."),
so it shouldn't be too hard to investigate.
2021-01-27 10:22:46 +02:00
Marcel Partap
183adec3d8 Merge remote-tracking branch 'origin/master' into synapse-workers 2021-01-23 15:04:11 +01:00
Slavi Pantaleev
a56cb34850 Notify people if /matrix/postgres/data-auto-upgrade-backup exists 2021-01-23 14:14:45 +02:00
Slavi Pantaleev
a2422c458a Notify of remaining matrix-postgres local data in a better way 2021-01-23 14:04:51 +02:00
Slavi Pantaleev
f085362149 Fix some Postgres CLI scripts to target the correct database
Fixes a regression introduced in 95346f3117.

Fixes https://github.com/spantaleev/matrix-docker-ansible-deploy/issues/814

Using `matrix_synapse_` variables in the `matrix-postgres` role is not
ideal, but.. this script belongs neither here, nor there.
We'll have it be like that for now.
2021-01-23 11:38:34 +02:00
Slavi Pantaleev
f9968b6981 Fix matrix_postgres_connection_password length check 2021-01-22 21:22:58 +02:00
Slavi Pantaleev
88addd71fc Fix Postgres imports going to the matrix DB by default
Well, they still do go to that DB by default,
but our docs give a better command to users, which would do the right
thing.
2021-01-22 17:39:08 +02:00
Slavi Pantaleev
bef0702fea Wait some more when starting Postgres during setup on ARM 2021-01-22 16:21:30 +02:00
Slavi Pantaleev
f9c1d62435 Fix Postgres database (-alpine) failing to start on ARM32 2021-01-22 13:52:55 +02:00
Slavi Pantaleev
95346f3117 Reorganize Postgres access (breaking change)
In short, this makes Synapse a 2nd class citizen,
preparing for a future where it's just one-of-many homeserver software
options.

We also no longer have a default Postgres superuser password,
which improves security.

The changelog explains more as to why this was done
and how to proceed from here.
2021-01-22 13:26:12 +02:00
Slavi Pantaleev
1692a28fe4 Work around annoying Docker warning about undefined $HOME
> WARNING: Error loading config file: .dockercfg: $HOME is not defined

.. which appeared in Docker 20.10.
2021-01-15 00:23:01 +02:00
Slavi Pantaleev
05ca9357a8 Add .service suffix to systemd units list
We'll be adding `.timer` units later on, so it's good to be
more explicit.
2021-01-14 23:02:10 +02:00
Marcel Partap
cd8100544b Merge remote-tracking branch 'origin/master' into synapse-workers
Sync with upstream
2021-01-08 20:58:50 +01:00
Slavi Pantaleev
fb83eccf99 Relocate SQL template file 2021-01-03 00:58:31 +02:00
Slavi Pantaleev
86da489b9b Never fail when stopping systemd service during (SQLite -> Postgres) migration
We need to suppress systemd service-stopping requests in certain rare
cases like https://github.com/spantaleev/matrix-docker-ansible-deploy/issues/771

That issue seems to describe a case, where a migration from mxisd to
ma1sd was happening (DB files had just been moved), and then we were
attemping to stop `matrix-ma1sd.service` so we could import that database into
Postgres. However, there's neither `matrix-mxisd.service`, nor
`matrix-ma1sd.service` after `migrate_mxisd.yml` had just run, so
stopping `matrix-ma1sd.service` was failing.
2020-12-29 10:31:20 +02:00
Slavi Pantaleev
21662af3be Archive database only after additional_psql_statements_list had executed 2020-12-23 14:12:11 +02:00
Stuart Mumford
019a4d7dcd Use role relative paths for things 2020-12-23 11:34:48 +00:00
Slavi Pantaleev
ad1425eee4 Add pgloader self-building support (for ARM) 2020-12-23 09:08:54 +02:00
Slavi Pantaleev
8675dedbdb Add support for automatic (nedb -> Postgres) migration to matrix-appservice-slack 2020-12-22 19:56:52 +02:00
Slavi Pantaleev
715bdf2c64 Add support for automatic (nedb -> Postgres) migration to mx-appservice-irc 2020-12-22 19:32:43 +02:00
Slavi Pantaleev
1bd5c240e5 Add support for executing additional DB migration statements
In cases where pgloader is not enough and we need to do some additional
migration work after it, we can now use
`additional_psql_statements_list` and
`additional_psql_statements_db_name`.

This is to be used when migrating `matrix-registration`'s data at the
very least.
2020-12-15 23:18:29 +02:00
Slavi Pantaleev
dd797ba6a7 Fix Postgres database importing/upgrading conflicts
We were running into conflicts, because having initialized
the roles (users) and databases, trying to import leads to
errors (role XXX already exists, etc.).

We were previously ignoring the Synapse database (`homeserver`)
when upgrading/importing, because that one gets created by default
whenever the container starts.

For our additional databases, it's a similar situation now.
It's not created by default as soon as Postgres starts with an empty
database, but rather we create it as part of running the playbook.

So we either need to skip those role/database creation statements
while upgrading/importing, or to avoid creating the additional database
and rely on the import for that. I've gone for the former, because
it's already similar to what we were doing and it's simpler
(it lets `setup_postgres.yml` be the same in all scenarios).
2020-12-14 22:28:20 +02:00
transcaffeine
13d8a9b39c
hint supported automatic migration nedb->postgres 2020-12-14 16:33:40 +01:00
Slavi Pantaleev
af3ea67bba Add (SQLite + Postgres) support and automatic migration to matrix-ma1sd 2020-12-14 17:16:25 +02:00
Slavi Pantaleev
c1431b28f0 Make use of matrix_postgres_db_migration_request.caller 2020-12-14 16:13:57 +02:00
Slavi Pantaleev
dc7850e83c Fix wording and variable names a bit 2020-12-14 16:03:40 +02:00
Slavi Pantaleev
bc376c2fb2 Add database migration utility to matrix-postgres role 2020-12-14 16:03:40 +02:00
Slavi Pantaleev
5dba0c038b Make --tags=import-generic-sqlite-db commands not pass a sensitive connection string around
Instead of passing the connection string, we can now pass a name of a
variable, which contains a connection string.

Both are supported for having extra flexibility.
2020-12-14 11:47:00 +02:00
Slavi Pantaleev
f1e85f7112 Don't mention Postgres roles, just say users 2020-12-14 10:04:37 +02:00
Slavi Pantaleev
cb969c6ca2 Add --tags=import-generic-sqlite-db (pgloader import)
This can be used by various bridges, etc., to import an SQLite
(or some other supported) database into Postgres.
2020-12-14 02:23:29 +02:00
Slavi Pantaleev
c66c084027 Merge branch 'master' into postgres-per-default 2020-12-14 01:51:15 +02:00
Slavi Pantaleev
6e1dfb62f0 Rename some doc files and commands related to importing
Since we'll likely have generic SQLite database importing
via [pgloader](https://pgloader.io/) for migrating bridge
databases from SQLite to Postgres, we'd rather avoid
calling the "import Synapse SQLite database" command
as just `--tags=import-sqlite-db`.

Similarly, for the media store, we'd like to mention that it's
related to Synapse as well.

We'd like to be more explicit, so as to be less confusing,
especially in light of other homeserver implementations
coming in the future.
2020-12-14 01:51:00 +02:00
Slavi Pantaleev
183d2a10db Ensure matrix-postgres.service is started before creating additional users/databases 2020-12-14 00:59:59 +02:00