Commit Graph

6131 Commits

Author SHA1 Message Date
Slavi Pantaleev
5dba0c038b Make --tags=import-generic-sqlite-db commands not pass a sensitive connection string around
Instead of passing the connection string, we can now pass a name of a
variable, which contains a connection string.

Both are supported for having extra flexibility.
2020-12-14 11:47:00 +02:00
Slavi Pantaleev
2a99e84b5b Rename Postgres-related variables (matrix*database_* -> matrix*postgres_*)
`matrix_SERVICE_database_engine` remains as it is - the only one with a
`_database_` infix. Postgres-related ones move to `_postgres_`.
2020-12-14 11:19:18 +02:00
Slavi Pantaleev
c0edacbefd Fix matrix_postgres_additional_databases syntax problems 2020-12-14 11:17:12 +02:00
Slavi Pantaleev
d91aa5a060 Do not introduce sub-variables exposing implementation details 2020-12-14 10:52:07 +02:00
Slavi Pantaleev
f1e85f7112 Don't mention Postgres roles, just say users 2020-12-14 10:04:37 +02:00
transcaffeine
b217a2ed69
postgres: set default in playbook to postgres but fallback to sqlite 2020-12-14 07:54:24 +01:00
Slavi Pantaleev
4617984b9f Add (SQLite -> Postgres) migration instructions 2020-12-14 02:24:32 +02:00
Slavi Pantaleev
cb969c6ca2 Add --tags=import-generic-sqlite-db (pgloader import)
This can be used by various bridges, etc., to import an SQLite
(or some other supported) database into Postgres.
2020-12-14 02:23:29 +02:00
Slavi Pantaleev
c66c084027 Merge branch 'master' into postgres-per-default 2020-12-14 01:51:15 +02:00
Slavi Pantaleev
6e1dfb62f0 Rename some doc files and commands related to importing
Since we'll likely have generic SQLite database importing
via [pgloader](https://pgloader.io/) for migrating bridge
databases from SQLite to Postgres, we'd rather avoid
calling the "import Synapse SQLite database" command
as just `--tags=import-sqlite-db`.

Similarly, for the media store, we'd like to mention that it's
related to Synapse as well.

We'd like to be more explicit, so as to be less confusing,
especially in light of other homeserver implementations
coming in the future.
2020-12-14 01:51:00 +02:00
Slavi Pantaleev
b87b754372 Fail if appservice-discord wants Postgres, but has leftover SQLite data 2020-12-14 01:36:15 +02:00
Slavi Pantaleev
dd994995bc Simplify password for additional Postgres databases
Using the result of `password_hash` works for creating them,
but authentication seems to be failing with some tools like pgloader.

It's possible that we're not escaping things properly somewhere.
Ideally, it'd be nice to solve that. But the easier (and still
relatively safe/good) solution is to just turn that password hash
into a UUID that's safe for passing around without worrying about
escaping.
2020-12-14 01:22:23 +02:00
Slavi Pantaleev
183d2a10db Ensure matrix-postgres.service is started before creating additional users/databases 2020-12-14 00:59:59 +02:00
Slavi Pantaleev
a374d309c8 Make appservice-discord support both SQLite and Postgres
People can toggle between them now. The playbook also defaults
to using SQLite if an external Postgres server is used.

Ideally, we'd be able to create databases/users in external Postgres
servers as well, but our initialization logic (and `docker run` command,
etc.) hardcode too many things right now.
2020-12-14 00:52:25 +02:00
Slavi Pantaleev
46a4034d3e Use "password" for additional Postgres databases, not "pass"
Being more explicit sounds better.
2020-12-14 00:43:03 +02:00
Slavi Pantaleev
3a037a5993 Ensure additional databases contain all the keys that we expect 2020-12-14 00:39:38 +02:00
Slavi Pantaleev
da4cb2f639 Do not use the postgresql_user/postgresql_db modules
While these modules are really nice and helpful, we can't use them
for at least 2 reasons:

- for us, Postgres runs in a container on a private Docker network
(`--network=matrix`) without usually being exposed to the host.
These modules execute on the host so they won't be able to reach it.

- these modules require `psycopg2`, so we need to install it before
using it. This might or might not be its own can of worms.
2020-12-14 00:31:38 +02:00
Slavi Pantaleev
bbc09d013b Do not execute additional databases creation code if not necessary
The tasks in `create_additional_databases.yml` will likely
ensure `matrix-postgres.service` is started, etc.

If no additional databases are defined, we'd rather not execute that
file and all these tasks that it may do in the future.
2020-12-13 23:46:05 +02:00
Slavi Pantaleev
c765ceb270 Prevent weird loop error
> Invalid data passed to 'loop', it requires a list, got this instead: matrix_postgres_additional_databases. Hint: If you passed a list/dict of just one element, try adding wantlist=True to your lookup invocation or use q/query instead of lookup.

Well, or working around it, as I've done in this commit (which seems
more sane than `wantlist=True` stuff).
2020-12-13 22:56:56 +02:00
Slavi Pantaleev
e2952f16f7 Determine matrix-postgres IP address without relying on jq
To avoid needing to have `jq` installed on the machine, we could:
- try to run jq in a Docker container using some small image providing
that
- better yet, avoid `jq` altogether
2020-12-13 22:45:48 +02:00
Slavi Pantaleev
f47e8a97e6 Make use of matrix_host_command_docker instead of hardcoding 2020-12-13 22:38:35 +02:00
Slavi Pantaleev
d251764c16 Fix syntax issues in matrix_postgres_additional_databases
Quotes are necessary around dictionary field names.

There was a missing `}` as well.
2020-12-13 22:37:04 +02:00
Slavi Pantaleev
0641106370 Allow username of additional Postgres databases to be different
We'll most likely use one that matches the database name, but
it's better to have it configurable.
2020-12-13 22:37:04 +02:00
Slavi Pantaleev
527d5f57d5 Relocate Postgres additional database creation logic
Moving it above the "uninstalling" set of tasks is better.
Extracting it out to another file at the same time, for readability,
especially given that it will probably have to become more complex in
the future (potentially installing `jq`, etc.)
2020-12-13 22:37:04 +02:00
Slavi Pantaleev
dac0d3a682 Add default matrix_postgres_additional_databases 2020-12-13 21:07:16 +02:00
Slavi Pantaleev
77a5c7cf3c Merge branch 'master' into postgres-per-default 2020-12-13 21:04:15 +02:00
Slavi Pantaleev
9dc023c558
Merge pull request #750 from chagai95/patch-1
Adding jitsi as well
2020-12-12 01:44:50 +02:00
Slavi Pantaleev
bae4113422
Mention that we possibly obtain a Jitsi certificate 2020-12-12 01:44:34 +02:00
chagai95
d96d7f2a43
Adding jitsi as well 2020-12-12 00:24:38 +01:00
Slavi Pantaleev
47613e5a27 Remove synapse-janitor support
Fixes https://github.com/spantaleev/matrix-docker-ansible-deploy/issues/746
2020-12-11 23:24:42 +02:00
Slavi Pantaleev
86988ae180 Switch matrix-registration to v0.7.1
Now that a new release has been made, we no longer need to use
`latest` / `master`.

Related to 0a9109771d and https://github.com/ZerataX/matrix-registration/issues/43
2020-12-11 22:52:42 +02:00
Slavi Pantaleev
0a9109771d Use latest/master version of matrix-registration
v0.7.0 is broken right now, because it calls
`/_matrix/client/r0/admin/register`, which is now at
`/_synapse/admin/v1/register`.

This has been fixed here: 6b26255fea

.. but it's not part of any release.

Switching to `master` (`docker.io/devture/zeratax-matrix-registration:latest`) until it gets resolved.

Reported upstream here: https://github.com/ZerataX/matrix-registration/issues/43
2020-12-11 22:22:07 +02:00
Slavi Pantaleev
7f1f3fd97b
Merge pull request #749 from aaronraimist/synapse-admin-0.6.1
Upgrade synapse-admin (v0.5.0 -> 0.6.1)
2020-12-11 00:46:12 +02:00
Aaron Raimist
3c2a644e5c
Upgrade synapse-admin (v0.5.0 -> 0.6.1) 2020-12-10 16:28:48 -06:00
Slavi Pantaleev
eae4f67470 Mention Docker 20.10 in the changelog 2020-12-11 00:13:10 +02:00
Slavi Pantaleev
7593d969e3 Make matrix-mailer not occupy matrix_server_fqn_matrix
Starting with Docker 20.10, `--hostname` seems to have the side-effect
of making Docker's internal DNS server resolve said hostname to the IP
address of the container.

Because we were giving the mailer service a hostname of `matrix.DOMAIN`,
all requests destined for `matrix.DOMAIN` originating from other
services on the container network were resolving to `matrix-mailer`.
This is obviously wrong.

Initially reported here: https://github.com/spantaleev/matrix-docker-ansible-deploy/pull/748

We normally try to not use the public hostname (and IP address) on the
container network and try to make services talk to one another locally,
but it sometimes could happen.

With this, we use a `matrix-mailer` hostname for the matrix-mailer
container. My testing shows that it doesn't cause any trouble with
email deliverability.
2020-12-10 23:51:11 +02:00
transcaffeine
d9f4914e0d
WIP: postgres: create databases for all services
If a service is enabled, a database for it is created in postgres with a uniqque password. The service can then use this database for data storage instead of relying on sqlite.
2020-12-10 18:26:22 +01:00
Slavi Pantaleev
d08b27784f Fix systemd services autostart problem with Docker 20.10
The Docker 19.04 -> 20.10 upgrade contains the following change
in `/usr/lib/systemd/system/docker.service`:

```
-BindsTo=containerd.service
-After=network-online.target firewalld.service containerd.service
+After=network-online.target firewalld.service containerd.service multi-user.target
-Requires=docker.socket
+Requires=docker.socket containerd.service
Wants=network-online.target
```

The `multi-user.target` requirement in `After` seems to be in conflict
with our `WantedBy=multi-user.target` and `After=docker.service` /
`Requires=docker.service` definitions, causing the following error on
startup for all of our systemd services:

> Job matrix-synapse.service/start deleted to break ordering cycle starting with multi-user.target/start

A workaround which appears to work is to add `DefaultDependencies=no`
to all of our services.
2020-12-10 11:43:20 +02:00
Slavi Pantaleev
327ca847b6
Merge pull request #748 from jgoerzen/patch-1
Correct inabillity for appservice-discord to connect
2020-12-10 10:50:52 +02:00
John Goerzen
673e19f830
Correct inabillity for appservice-discord to connect
After recently updating my matrix-docker-ansible-deploy installation, matrix-appservice-discord would refuse to start, logging ECONNREFUSED to https://matrix.[mydomain]:443, which was resolving to 172.18.0.2 due to the `--hostname` in mailer grabbing that hostname.

Curious why the IRC bridge didn't have this issue, I looked into it, and it was connecting to `http://matrix-synapse:8008`.  Correcting this one to that URL resolved the issue.
2020-12-09 21:20:06 -06:00
Slavi Pantaleev
245b749946 Upgrade Synapse for ARM (v1.23.0 -> v1.24.0)
Continuation of aa86e0dac6, now that ARM images are out.
2020-12-09 20:54:18 +02:00
Slavi Pantaleev
aa86e0dac6 Upgrade Synapse (v1.23.0 -> v1.24.0)
Because the ARM images are not pushed yet, we hold back to v1.23.0
for now.
2020-12-09 13:31:10 +02:00
Slavi Pantaleev
f453d28f0e
Merge pull request #747 from benkuly/master
updated matrix-sms-bridge
2020-12-09 11:08:49 +02:00
benkuly
ad92c61fdd updated matrix-sms-bridge 2020-12-09 09:45:44 +01:00
Slavi Pantaleev
c07c927d9f Automatically enable openid listeners when ma1sd enabled
ma1sd requires the openid endpoints for certain functionality.
Example: 90b2b5301c/src/main/java/io/kamax/mxisd/auth/AccountManager.java (L67-L99)

If federation is disabled, we still need to expose these openid APIs on the
federation port.

Previously, we were doing similar magic for Dimension.
As per its documentation, when running unfederated, one is to enable
the openid listener as well. As per their recommendation, people
are advised to do enable it on the Client-Server API port
and use the `federationUrl` variable to override where the federation
port is (making federation requests go to the Client-Server API).

Because ma1sd always uses the federation port (unless you do some
DNS overwriting magic using its configuration -- which we'd rather not
do), it's better if we just default to putting the `openid` listener
where it belongs - on the federation port.

With this commit, we retain the "automatically enable openid APIs" thing
we've been doing for Dimension, but move it to the federation port instead.
We also now do the same thing when ma1sd is enabled.
2020-12-08 16:59:20 +02:00
Slavi Pantaleev
8c02f7b79b Upgrade services 2020-12-07 15:18:03 +02:00
Slavi Pantaleev
d556aa943f Update docker-ce.repo to not hardcode $releasever=7
This keeps it in line with https://download.docker.com/linux/centos/docker-ce.repo

Whether or not Docker works well on CentOS 8 for our purposes
hasn't been verified yet.

Related to https://github.com/spantaleev/matrix-docker-ansible-deploy/issues/300
2020-12-07 07:20:47 +02:00
Slavi Pantaleev
1e0777f2ed
Merge pull request #745 from wabuMike/patch-1
fixes config.js
2020-12-07 00:46:14 +02:00
Mike
9713ac9617
fixes config.js
comments in config.js must begin with two slashes
2020-12-06 23:42:44 +01:00
Slavi Pantaleev
7372480e95 Properly serialize some ma1sd configuration values
We've had a report of the `connection` value getting cut off,
supposedly because it contains something that breaks off the string.

Using `|to_json` takes care of it.
2020-12-06 23:59:58 +02:00