A lot of services are yet to be updated to start connecting to
`devture_postgres_container_network` as an additional network.
Many are already done, but I'll go through all the others later.
Downsides: decreasing security slightly due to less networking isolation
Benefits:
- decreased complexity
- having a generically-named `matrix-addons` network we may use for other things now (client apps, etc.)
- not exhausting the container networks pool with 2 (or more) networks and using just 1
This also updates validation tasks and documentation, pointing to
variables in the matrix-synapse role which don't currently exist yet
(e.g. `matrix_synapse_container_labels_client_synapse_admin_api_enabled`).
These variables will be added soon, as Traefik labels are added to the
`matrix-synapse` role. At that point, the `matrix-synapse-reverse-proxy-companion` role
will be updated to also use them.
matrix-nginx-proxy is going away and this is one of the features it
offered.
This feature will have no equivalent in our new Traefik-only
setup, although it's possible to implement it manually by using
`matrix_client_element_container_labels_additional_labels`
This is still very far from usable.
Various bridges and bots are still talking to
`matrix-nginx-proxy` instead of the new `matrix-homeserver-proxy` role.
These services need to be reworked. While reworking them,
various cleanups are being done as well as adding Traefik-labels to
those that need them.
It seems like connectivity is problematic, even though the networks
appear to be configured correctly:
> [ioredis] Unhandled error event: Error: connect ECONNREFUSED 172.22.0.2:6739
> at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1595:16)
For now, I disable pointing the queue host to Redis to avoid it.
It should be investigated.
People who enable Hookshot's new experimental encryption may encounter
this also.
Related to https://github.com/spantaleev/matrix-docker-ansible-deploy/pull/3042
Squashed based on the work done in https://github.com/spantaleev/matrix-docker-ansible-deploy/pull/3042
commit 49932b8f3c
Author: Slavi Pantaleev <slavi@devture.com>
Date: Sat Dec 16 09:21:31 2023 +0200
Fix syntax in matrix-bridge-hookshot/tasks/reset_encryption.yml
Also, this task always does work and side-effects, so it should always report changes
(`changed_when: true`).
commit 6bdf7a9dcb
Author: Slavi Pantaleev <slavi@devture.com>
Date: Sat Dec 16 09:12:41 2023 +0200
Add Hookshot validation task to ensure queue settings are set when encryption is enabled
commit 8c531b7971
Author: Slavi Pantaleev <slavi@devture.com>
Date: Sat Dec 16 09:10:17 2023 +0200
Add missing variables rewiring in group_vars/matrix_servers for Hookshot
commit 7d26dabc2f
Author: Slavi Pantaleev <slavi@devture.com>
Date: Sat Dec 16 09:08:19 2023 +0200
Add defaults for matrix_hookshot_queue_host and matrix_hookshot_queue_port
commit 74f91138c9
Author: Slavi Pantaleev <slavi@devture.com>
Date: Sat Dec 16 09:06:17 2023 +0200
Fix syntax for connecting to additional networks for Hookshot
commit ca7b41f3f2
Author: Slavi Pantaleev <slavi@devture.com>
Date: Sat Dec 16 09:05:28 2023 +0200
Fix indentation and remove unnecessary if-statements
commit ac4a918d58
Author: Slavi Pantaleev <slavi@devture.com>
Date: Sat Dec 16 09:04:44 2023 +0200
Add missing --network for Hookshot
This seems to have been removed by accident.
commit 6a81fa208f
Author: Slavi Pantaleev <slavi@devture.com>
Date: Sat Dec 16 09:02:47 2023 +0200
Make automatic Redis enabling safer, when Hookshot encryption enabled
If we ever default encryption to enabled for Hookshot, we only wish to force-enable Redis if Hookshot is actually enabled.
commit 75a8e0f2a6
Author: Slavi Pantaleev <slavi@devture.com>
Date: Sat Dec 16 09:01:10 2023 +0200
Fix typo
commit 98ad182eac
Author: Joshua Hoffmann <joshua.hoffmann@b1-systems.de>
Date: Fri Dec 15 22:37:40 2023 +0100
Add defaults for Hookshot's encryption
commit 29fa9fab15
Author: Joshua Hoffmann <joshua.hoffmann@b1-systems.de>
Date: Fri Dec 15 22:35:11 2023 +0100
Improve wording of Hookshot's encryption section
commit 4f835e0560
Author: Joshua Hoffmann <joshua.hoffmann@b1-systems.de>
Date: Fri Dec 15 22:28:52 2023 +0100
use safer mount options for the container's files
commit 8c93327e25
Author: Joshua Hoffmann <joshua.hoffmann@b1-systems.de>
Date: Fri Dec 15 22:26:01 2023 +0100
fix filename
commit 03a7bb6e77
Merge: e55d769406047763
Author: Joshua Hoffmann <joshua.hoffmann@b1-systems.de>
Date: Fri Dec 15 22:23:44 2023 +0100
Merge branch 'HarHarLinks/hookshot-encryption' of https://github.com/real-joshua/matrix-docker-ansible-deploy into HarHarLinks/hookshot-encryption
commit 06047763bb
Author: Joshua Hoffmann <joshua.hoffmann@b1-systems.de>
Date: Fri Dec 15 22:15:54 2023 +0100
Update roles/custom/matrix-bridge-hookshot/templates/config.yml.j2
change the if statement to not require a variable with a length > 0 and add a filter to json for the redis host
Co-authored-by: Slavi Pantaleev <slavi@devture.com>
commit e55d769465
Author: Joshua Hoffmann <joshua.hoffmann@b1-systems.de>
Date: Fri Dec 15 22:13:50 2023 +0100
clarify that Redis is required, standardadise on Hookshot with an upper-case first letter for consistency
commit 66706e4535
Author: Joshua Hoffmann <joshua.hoffmann@b1-systems.de>
Date: Fri Dec 15 22:08:20 2023 +0100
Update roles/custom/matrix-bridge-hookshot/templates/config.yml.j2
fix for a typo
Co-authored-by: Slavi Pantaleev <slavi@devture.com>
commit f6aaeb9a16
Merge: e5d34002869dd33f
Author: Joshua Hoffmann <joshua.hoffmann@b1-systems.de>
Date: Fri Dec 15 00:22:34 2023 +0100
Merge branch 'master' into HarHarLinks/hookshot-encryption
commit e5d34002fd
Author: Joshua Hoffmann <joshua.hoffmann@b1-systems.de>
Date: Fri Dec 15 00:09:27 2023 +0100
Add Jinja loop to allow adding multiple networks
commit 69f947782d
Author: Joshua Hoffmann <joshua.hoffmann@b1-systems.de>
Date: Thu Dec 14 23:52:41 2023 +0100
split if statements for the message queue and experimental encryption support into seperate statements
commit 4c13be1c89
Author: Joshua Hoffmann <joshua.hoffmann@b1-systems.de>
Date: Thu Dec 14 23:31:19 2023 +0100
change variable name per spantaleev's suggestion (https://github.com/spantaleev/matrix-docker-ansible-deploy/pull/2979#discussion_r1379015551)
commit 9905309aa9
Author: HarHarLinks <kim.brose@rwth-aachen.de>
Date: Wed Nov 1 16:14:04 2023 +0100
amend docs
commit 94abf2d5bd
Author: HarHarLinks <kim.brose@rwth-aachen.de>
Date: Wed Nov 1 16:05:22 2023 +0100
draft encryption support for hookshot
Currently v3.0.0 tested with no issues.
So remove matrix_user_verification_service_docker_image from groups_vars.
/.npm must be writable or an error will be reported.
* Inital work, copeid from mautrix-amp PR
* Some fixes leftover code copeid over from whatsapp
* Got it to run and register
* Fixed service issue with docker image
* I now realize I need 2 roles wsproxy and imessage
* Got someting working, still rough
* Closer to working but still not working
* reverting ports
* Update main.yml
* Add matrix-nginx-proxy config for mautrix-wsproxy
* Changed
* Add back file
* fix for error hopefully
* Changed the the way nginx was recieved
* basically did not add anything ugh
* Added some arguments
* just trying stuff now
* Ugh i messed up port number
* Changed docs
* Change dns config
* changed generic secret key
* Testing new nginx proxy
* test
* Fix linting errors
* Add mautrix syncproxy to wsproxy for Android SMS
* WIP
* Move wsproxy to custom
* Squashed commit of the following:
commit 943189a9aa
Merge: 4a229d68f5a09f30
Author: Slavi Pantaleev <slavi@devture.com>
Date: Sun Nov 13 08:54:32 2022 +0200
Merge pull request #2259 from throny/patch-3
warn users about upgrading to pg15 when using borg
commit 4a229d6870
Merge: 9b326e08c68def08
Author: Slavi Pantaleev <slavi@devture.com>
Date: Sun Nov 13 08:53:13 2022 +0200
Merge pull request #2260 from etkecc/patch-117
Update ntfy 1.28.0 -> 1.29.0
commit f5a09f30b7
Author: throny <m.throne12@gmail.com>
Date: Sat Nov 12 23:48:57 2022 +0100
Update maintenance-postgres.md
commit b12cdbd99d
Author: throny <m.throne12@gmail.com>
Date: Sat Nov 12 23:40:46 2022 +0100
Update maintenance-postgres.md
commit c68def0809
Author: Aine <97398200+etkecc@users.noreply.github.com>
Date: Sat Nov 12 22:01:31 2022 +0000
Update ntfy 1.28.0 -> 1.29.0
commit adbc09f152
Author: throny <m.throne12@gmail.com>
Date: Sat Nov 12 11:20:43 2022 +0100
warn users about upgrading to pg15 when using borg
* Fix linting errors
* Cleanup after merge
* Correct outdated variable names
* Enable both Android and iMessage with wsproxy
* Restructure wsproxy service defs and nginx config
* Fix linter errors
* Apply suggestions from code review
Co-authored-by: Slavi Pantaleev <slavi@devture.com>
* Fix comments for documentation, volumes and ports
* Correct mount syntax
* Complete network and traefik support for wsproxy
* Remove wsproxy data_path
* Fix wsproxy service definitions
* Actually include syncproxy service
* Remove wsproxy PathPrefix, it needs a subdomain
There's no setting in the iMessage bridge that allows a path.
Also don't bind port by default, wsproxy has no TLS.
Syncproxy should never expose a port, it's only internal.
---------
Co-authored-by: hanthor <jreilly112@gmail.com>
Co-authored-by: Miguel Alatzar <miguel@natrx.io>
Co-authored-by: Shreyas Ajjarapu <github.tzarina@aleeas.com>
Co-authored-by: Slavi Pantaleev <slavi@devture.com>
* intial commit
* changed
* Reorderd
* merge old changes
* added changes to matrix_servers
* Remove duplicate discord
* Update main.yml
* added google message to configuring-playbook.md
* Changed docs to add new changes
* Changed bug?
* Removed problem j2 values
* Rename a service files
* change how password hash string
* Changed port number
* Change how the local part works
* Revert "Merge pull request #8 from shreyasajj/wsproxy"
This reverts commit bb1b8fc67c, reversing
changes made to cce6ba5f9d.
---------
Co-authored-by: Shreyas Ajjarapu <github.tzarina@aleeas.com>
Making the group match the Ansible task tags allows people to do `just install-service matrix-media-repo` and have that trigger both `--tags=matrix-media-repo` and also restart just that single group (`matrix-media-repo`).
Previously `just install-service buscarron` would not fully work,
because:
- the systemd services were indeed tagged with `buscarron`
- however, the actual installation tasks are not
`install-buscarron`/`setup-buscarron`, but rather
`install-bot-buscarron`/`setup-bot-buscarron`
Services are now tagged with the `bot-` prefix to match the tags.
This allows people to try out the new Element X clients, which need to
run against the sliding-sync proxy (https://github.com/matrix-org/sliding-sync).
Supersedes https://github.com/spantaleev/matrix-docker-ansible-deploy/pull/2515
The code is based on the existing PR (#2515), but heavily reworked. Major changes:
- lots of internal refactoring and variable renaming
- fixed self-building to support non-amd64 architectures
- changed to talk to the homeserver locally, over the container network (not
publicly)
- no more matrix-nginx-proxy support due to complexity (see below)
- no more `matrix_server_fqn_sliding_sync_proxy` in favor of
`matrix_sliding_sync_hostname` and `matrix_sliding_sync_path_prefix`
- runs on `matrix.DOMAIN/sliding-sync` by default, so it can tried
easily without having to create new DNS records
The variable was necessary when multiple playbooks could have
potentially tried to manage a shared `devture-traefik.serivce` systemd service
and shared `/devture-traefik` directory.
Since adcc6d9723, we use our own `/matrix/traefik`
(`matrix-traefik.service`) installation and no conflicts can arise.
It's safe to always enable the role, just like we do with all the other roles.
The migration is automatic. Existing users should experience a bit of
downtime until the playbook runs to completion, but don't need to do
anything manually.
This change is provoked by https://github.com/spantaleev/matrix-docker-ansible-deploy/pull/2535
While my statements there ("Traefik is a shared component among
sibling/related playbooks and should retain its global
non-matrix-prefixed name and path") do make sense, there's another point
of view as well.
With the addition of docker-socket-proxy support in bf2b540807,
we potentially introduced another non-`matrix-`-prefixed systemd service
and global path (`/devture-container-socket-proxy`). It would have
started to become messy.
Traefik always being called `devture-traefik.service` and using the `/devture-traefik` path
has the following downsides:
- different playbooks may write to the same place, unintentionally,
before you disable the Traefik role in some of them.
If each playbook manages its own installation, no such conflicts
arise and you'll learn about the conflict when one of them starts its
Traefik service and fails because the ports are already in use
- the data is scattered - backing up `/matrix` is no longer enough when
some stuff lives in `/devture-traefik` or `/devture-container-socket-proxy` as well;
similarly, deleting `/matrix` is no longer enough to clean up
For this reason, the Traefik instance managed by this playbook
will now be called `matrix-traefik` and live under `/matrix/traefik`.
This also makes it obvious to users running multiple playbooks, which
Traefik instance (powered by which playbook) is the active one.
Previously, you'd look at `devture-traefik.service` and wonder which
role was managing it.
We don't need these 2 roughly-the-same settings related to the
traefik-certs-dumper role.
For Traefik, it makes sense, because it's a component used by the
various related playbooks and they could step onto each other's toes
if the role is enabled, but Traefik is disabled (in that case, uninstall
tasks will run).
As for Traefik certs dumper, the other related playbooks don't have it,
so there's no conflict. Even if they used it, each one would use its own
instance (different `devture_traefik_certs_dumper_identifier`), so there
wouldn't be a conflict and uninstall tasks can run without any danger.
This allows people wishing to change or unset the resolver,
to have a single variable which they can toggle.
Unsetting the resolver is useful for using your own certificates
(not coming from a certificate resolver).
The newly extracted role also has native Traefik support,
so we no longer need to rely on `matrix-nginx-proxy` for
reverse-proxying to Ntfy.
The new role uses port `80` inside the container (not `8080`, like
before), because that's the default assumption of the officially
published container image. Using a custom port (like `8080`), means the
default healthcheck command (which hardcodes port `80`) doesn't work.
Instead of fiddling to override the healthcheck command, we've decided
to stick to the default port instead. This only affects the
inside-the-container port, not any external ports.
The new role also supports adding the network ranges of the container's
multiple additional networks as "exempt hosts". Previously, only one
network's address range was added to "exempt hosts".
Previously, it had to go through matrix-nginx-proxy.
It's exposed to Traefik directly via container labels now
Serving at a path other than `/` doesn't work well yet.
We'd like to auto-enable traefik-certs-dumper for these setups.
`devture_traefik_certs_dumper_ssl_dir_path` will be empty though,
so the role's validation will point people in the right direction.
This gets us started on adding a Traefik role and hooking Traefik:
- directly to services which support Traefik - we only have a few of
these right now, but the list will grow
- to matrix-nginx-proxy for most services that integrate with
matrix-nginx-proxy right now
Traefik usage should be disabled by default for now and nothing should
change for people just yet.
Enabling these experiments requires additional configuration like this:
```yaml
devture_traefik_ssl_email_address: '.....'
matrix_playbook_traefik_role_enabled: true
matrix_playbook_traefik_labels_enabled: true
matrix_ssl_retrieval_method: none
matrix_nginx_proxy_https_enabled: false
matrix_nginx_proxy_container_http_host_bind_port: ''
matrix_nginx_proxy_container_federation_host_bind_port: ''
matrix_nginx_proxy_trust_forwarded_proto: true
matrix_nginx_proxy_x_forwarded_for: '$proxy_add_x_forwarded_for'
matrix_coturn_enabled: false
```
What currently works is:
reverse-proxying for all nginx-proxy based services **except** for the Matrix homeserver
(both Client-Server an Federation traffic for the homeserver don't work yet)
Related to https://github.com/spantaleev/matrix-docker-ansible-deploy/pull/2427
This just enables the endpoint, which is somewhat helpful, but not
really enough to scrape them. Ideally, we'd be injecting these targets
into the Prometheus scrape config too.
For now, registering targets with Prometheus is very manual
(`matrix_prometheus_scraper_postgres_enabled`, `matrix_prometheus_scraper_hookshot_enabled`, ..).
This should be redone - e.g. a new `matrix_prometheus_scrape_config_jobs_auto` variable,
which is dynamically built in `group_vars/matrix_servers`.
selectattr() returns a generator object, an iterator. This leads to an exception later, lists can't concated to iterators, only to other lists. So '| list' converts the iterator to a list and the script runs happily.
This extends the collection with support for seamless authentication at the Jitsi server using Matrix OpenID.
1. New role for installing the [Matrix User Verification Service](https://github.com/matrix-org/matrix-user-verification-service)
2. Changes to Jitsi role: Installing Jitsi Prosody Mods and configuring Jitsi Auth
3. Changes to Jitsi and nginx-proxy roles: Serving .well-known/element/jitsi from jitsi.DOMAIN
4. We updated the Jitsi documentation on authentication and added documentation for the user verification service.
* add prometheus-nginxlog-exporter role
* Rename matrix_prometheus_nginxlog_exporter_container_url to matrix_prometheus_nginxlog_exporter_container_hostname
* avoid referencing variables from other roles, handover info using group_vars/matrix_servers
* fix: stop service when uninstalling
fix: typo
move available arch's into a var
fix: text
* fix: prometheus enabled condition
Co-authored-by: ikkemaniac <ikkemaniac@localhost>
This role is usable on its own and it's not tied to Matrix, so
extracting it out into an independent role that we install via
ansible-galaxy makes sense.
This also fixes the confusion from the other day, where
`matrix_postgres_*` had to be renamed to `devture_postgres_*`
(unless it was about `matrix_postgres_backup_*`).
We now can safely say that ALL `matrix_postgres_*` variables need to be
renamed.
Fixes https://github.com/spantaleev/matrix-docker-ansible-deploy/issues/2305
There was no need to add `synapse` to the list manually
and then add all other additional databases.
When the `synapse` database was the main database, this made sense.
Since a long time ago already, ALL databases are "additional" databases,
so the `synapse` database is part of that list.
We could additional add the main (`matrix`) database to this list,
but there's probably no point in backing that one up.
* added dendrite captcha options
* added hcaptcha doc
* proper url
* Apply suggestions from code review
Co-authored-by: Slavi Pantaleev <slavi@devture.com>
* Update main.yml
* renamed captcha vars to new naming scheme
* change vars to new format
* Rename back some incorrect renamed variables
These variables are either not just part of the `client_api` subsection,
or are not even part of that section at all. They shouldn't have been
renamed in baaef2ed616e2645550d9
* Fix up naming inconsistencies
Some of these variables had been renamed in one place,
but not in other places, so it couldn't have worked that way.
* Add validation/deprecation for renamed Dendrite variables
Related to 4097898f885cf4c73, baaef2ed616e2645550, 68f4418092fa8ad
and a0b4a0ae6b2f1f18
Co-authored-by: Slavi Pantaleev <slavi@devture.com>
Ansible recently started showing warnings about `crypt` being
deprecated. If one installs `passlib`, the `password_hash` values that
are generated would be different by default. With this patch, we ensure
consistency regardless of which one is used.
After this patch, password hashes (and UUIDs derived from them) will
change once, but they should be stable after that.
These hashes changing is not a problem, because the playbook
changes all references to the new values. Changes are only a problem if
they're done partially and with different tools.
For example:
- `--tags=setup-COMPONENT` with `passlib`
- `--tags=setup-postgres` with `crypt` (no `passlib`)
If so, the Postgres database password's value will differ for the
configuration generated for `COMPONENT`.
The `rounds=` value is arbitrary. It doesn't matter what it is,
as long as it's different than the default for `crypt` (5000)
and the default for `passlib` for `sha512` (656000).
Source (https://docs.ansible.com/ansible/latest/playbook_guide/playbooks_filters.html):
> To ensure idempotency, specify rounds to be neither crypt’s nor passlib’s default, which is 5000 for crypt and a variable value (535000 for sha256, 656000 for sha512) for passlib
We need this to control whether `('matrix-' + matrix_homeserver_implementation + '.service')`
would get injected into `devture_systemd_service_manager_services_list_auto`
We no longer ask users to create Matrix user accounts for these bots:
- Postmoogle
- Honoroit
- Reminder Bot
Other bots and services (matrix-registration-bot, maubot, mjolnir,
Dimension, etc.) require an Access Token to run (not a password),
so this new role doesn't help for them.
It does help for the above bots though, and for defining your own
"initial user accounts" in the `matrix_user_creator_users_additional`
variable.
