Related to 0241c71a4c
Related to https://github.com/spantaleev/matrix-docker-ansible-deploy/pull/3270#issuecomment-2143782962
With this change, it should be possible for people to adjust the Docker
dependency from `docker.service` to something else (e.g. `pkg-ContainerManager-dockerd.service`),
or to completely eliminate it by setting `devture_systemd_docker_base_docker_service_name` to an empty string.
This makes it easier for people to use the playbook against a Synology DSM server.
This is based on the PR (https://github.com/spantaleev/matrix-docker-ansible-deploy/pull/3241)
by Tobias Diez (https://github.com/tobiasdiez).
I've refactored some parts, made it more configurable, polished it up,
and it's integrated into the playbook now.
Both the WeChat bridge and WeChat agent appear to be working.
The WeChat bridge joins rooms and responds as expected.
That said, end-to-end testing (actually bridging to a WeChat account) has not been done yet.
Fixes https://github.com/spantaleev/matrix-docker-ansible-deploy/issues/701
Fixes https://github.com/spantaleev/matrix-docker-ansible-deploy/issues/3092
This is sponsored https://etke.cc/ work related to https://gitlab.com/etke.cc/ansible/-/issues/2
Squashed commit of the following:
commit fdd37f02472a0b83d61b4fac80650442f90e7629
Author: Slavi Pantaleev <slavi@devture.com>
Date: Mon Jun 3 21:05:53 2024 +0300
Add documentation for WeChat bridge
commit 8426fc8b95bb160ea7f9659bd45bc59cf1326614
Author: Slavi Pantaleev <slavi@devture.com>
Date: Mon Jun 3 20:59:42 2024 +0300
Rename directory for matrix_wechat_agent_container_src_files_path
commit da200df82bbc9153d307095dd90e4769c400ea1e
Author: Slavi Pantaleev <slavi@devture.com>
Date: Mon Jun 3 20:58:26 2024 +0300
Make WeChat listen_secret configurable and auto-configured via matrix_homeserver_generic_secret_key
commit 4022cb1355828ac16af7d9228cb1066962bb35f5
Author: Slavi Pantaleev <slavi@devture.com>
Date: Mon Jun 3 20:54:56 2024 +0300
Refactor install.yml for WeChat a bit (using blocks, etc.)
commit d07a39b4c4f6b93d04204e13e384086d5a242d52
Author: Slavi Pantaleev <slavi@devture.com>
Date: Mon Jun 3 20:52:35 2024 +0300
Rename WeChat Agent configuration file
This makes it more clear that it belongs to the agent.
Otherwise, `config.yaml` and `configure.yaml` make you wonder.
commit ccca72f8d1e602f7c42f4bd552193afa153c9b9d
Author: Slavi Pantaleev <slavi@devture.com>
Date: Mon Jun 3 20:49:06 2024 +0300
Move WeChat agent configuration to a template
commit a4047d94d8877b4095712dfc76ac3082a1edca28
Author: Slavi Pantaleev <slavi@devture.com>
Date: Mon Jun 3 20:47:17 2024 +0300
Mount WeChat config as readonly and instruct bridge to not update it
commit bc0e89f345bf14bbdbfd574bb60d93918c2ac053
Author: Slavi Pantaleev <slavi@devture.com>
Date: Mon Jun 3 20:46:33 2024 +0300
Sync WeChat config with upstream
Brings up-to-date with:
https://github.com/duo/matrix-wechat/commits/0.2.4/example-config.yaml
commit a46f5b9cbc8bf16042685a18c77d25a606bc8232
Author: Slavi Pantaleev <slavi@devture.com>
Date: Mon Jun 3 19:48:17 2024 +0300
Rename some files
commit 3877679040cffc4ca6cccfa21a7335f8f796f06e
Author: Slavi Pantaleev <slavi@devture.com>
Date: Mon Jun 3 19:47:10 2024 +0300
Update WeChat logging config
This brings it up-to-date with what mautrix-go uses.
Otherwise, on startup we see:
> Migrating legacy log config
.. and it gets migrated to what we've done here.
commit e3e95ab234651867c7a975a08455549b31db4172
Author: Slavi Pantaleev <slavi@devture.com>
Date: Mon Jun 3 19:43:37 2024 +0300
Make sure matrix-wechat-agent runs as 1000:1000
It needs to write stuff to `/home/user/.vnc`.
`/home/user` is owned by `user:group` (`1000:1000`), so it cannot run
any other way.
Previously, if the `matrix` user was uid=1000 by chance, it would work,
but that's pure luck.
commit 4d5748ae9b84c81d6b48b0a41b790339d9ac4724
Author: Slavi Pantaleev <slavi@devture.com>
Date: Mon Jun 3 18:57:09 2024 +0300
Pin wechat and wechat-agent versions
commit 40d40009f19ebceed4126146cbb510a2c95af671
Author: Slavi Pantaleev <slavi@devture.com>
Date: Mon Jun 3 18:53:58 2024 +0300
docker_image -> container_image for WeChat bridge
commit cc33aff592541913070d13288d17b04ed6243176
Author: Slavi Pantaleev <slavi@devture.com>
Date: Mon Jun 3 18:00:25 2024 +0300
docker_src -> container_src in WeChat bridge
commit 42e6ae9a6483c8ca6d53b8052058d41d90d93797
Author: Slavi Pantaleev <slavi@devture.com>
Date: Mon Jun 3 17:54:24 2024 +0300
matrix_go_wechat_ -> matrix_wechat_
The bridge is written in Go, but does not include Go anywhere in its
name. As such, it's mostly useless to use `matrix_go_wechat` as the
prefix.
commit d6662a69d1916d215d5184320c36d2ef73afd3e9
Author: Tobias Diez <code@tobiasdiez.de>
Date: Mon Mar 25 10:55:16 2024 +0800
Add wechat bridge
The conditional check 'matrix_hookshot_experimental_encryption_enabled and matrix_hookshot_cache_redisUri == ''' failed. The error was: An unhandled exception occurred while templating '{{ ('redis://' + matrix_hookshot_cache_redis_host + ':' + matrix_hookshot_cache_redis_port) if matrix_hookshot_cache_redis_host else '' }}'. Error was a <class 'ansible.errors.AnsibleError'>, original message: Unexpected templating type error occurred on ({{ ('redis://' + matrix_hookshot_cache_redis_host + ':' + matrix_hookshot_cache_redis_port) if matrix_hookshot_cache_redis_host else '' }}): can only concatenate str (not \"int\") to str. can only concatenate str (not \"int\") to str
Appservice Draupnir for All required Develop before the release of 1.87.0 to work at all in the playbook. Now that we have a release to pin to we will return to being pinned to a release. Especially as Draupnir 2.0.0 push is happening now in main. This will mean that Draupnir develop is expected to be much more unstable than usual for a bit so its important that we pin to a stable release. These releases are validated due to having been dogfooded ever since D4A was merged into the playbook.
commit cf8637efaca0a0be3609fd6add0dff893a0a9194
Author: Slavi Pantaleev <slavi@devture.com>
Date: Sun Mar 24 19:14:57 2024 +0200
Make devture_systemd_docker_base_ipv6_enabled automatically reconfigure geerlingguy/ansible-role-docker
Related to https://github.com/spantaleev/matrix-docker-ansible-deploy/pull/3218
commit dc7af3bc7d25f321bf409477d823e43ea8a05803
Author: Slavi Pantaleev <slavi@devture.com>
Date: Sun Mar 24 19:10:31 2024 +0200
Replace matrix_ipv6_enabled with devture_systemd_docker_base_ipv6_enabled
Related to https://github.com/spantaleev/matrix-docker-ansible-deploy/pull/3218
commit 07e900d6a2
Author: Slavi Pantaleev <slavi@devture.com>
Date: Sun Mar 24 19:01:51 2024 +0200
Improve matrix_ipv6_enabled comments
commit 3f03ca7f69
Author: Tilo Spannagel <development@tilosp.de>
Date: Sat Mar 9 19:27:50 2024 +0000
Add setting to enable ipv6
* Draupnir for all Role
* Draupnir for all Documentation
* Pin D4A to Develop until D4A patches are in a release.
* Update D4A Docs to mention pros and cons of D4A mode compared to normal
* Change Documentation to mention a fixed simpler provisioning flow.
Use of /plain allows us to bypass the bugs encountered during the development of this role with clients attempting to escape our wildcards causing the grief that led to using curl.
This reworded commit does still explain you can automatically inject stuff into the room if you wanted to.
* Emphasise the State of D4A mode
* Link to Draupnir-for-all docs and tweak the docs some
* Link to Draupnir-for-all from Draupnir documentation page
* Announce Draupnir-for-all
---------
Co-authored-by: Slavi Pantaleev <slavi@devture.com>
We still remain on v0.1.0 for now, even though that's quite old nowadays
and the bridge is moving quickly.
Still, one could now pin to a specific commit like this:
```yml
matrix_mautrix_meta_messenger_version: 682c4d75b0fdfe102af4b6d88bb5c76453adc86d
matrix_mautrix_meta_instagram_version: 682c4d75b0fdfe102af4b6d88bb5c76453adc86d
```
* mautrix-meta: enable spaces; add a hint into the display name
* use the meta mode to determine displayname suffix
* Allow for people to easily unset the mautrix-meta displayname suffix
Previously, unsetting `matrix_mautrix_meta_messenger_bridge_displayname_suffix`
or (`matrix_mautrix_meta_instagram_bridge_displayname_suffix`) variable would
make you end up witha trailing space in `displayname`.
It's possible that mautrix-meta trims this, but I haven't checked. It's
better not to risk it anyway.
---------
Co-authored-by: Slavi Pantaleev <slavi@devture.com>
We noticed that the reporting function in Element is broken, at least when using the 'specialized-workers' preset.
This changes the `main_override_locations_regex` of the reverse proxy companion to ensure that requests to `/_matrix/client/v3/rooms/<roomid>/report/<message>` always land on the main process.
We may have had another migration task before, but I cannot find it now.
Some people have reported a leftover systemd timer and service,
so it's evident that not everyone has gone through that previous migration.
The new version is very broken. It has at least 2 issues.
The first one is:
```
Error: maxPromptTokens + max_tokens (3097 + 1024 = 4121) must be less than or equal to maxContextTokens (4097)
at ChatGPTClient.setOptions (file:///usr/src/app/node_modules/@waylaidwanderer/chatgpt-api/src/ChatGPTClient.js:72:19)
at new ChatGPTClient (file:///usr/src/app/node_modules/@waylaidwanderer/chatgpt-api/src/ChatGPTClient.js:23:14)
at main (file:///usr/src/app/dist/index.js:62:21)
at file:///usr/src/app/dist/index.js:94:1
at ModuleJob.run (node:internal/modules/esm/module_job:218:25)
at async ModuleLoader.import (node:internal/modules/esm/loader:329:24)
at async loadESM (node:internal/process/esm_loader:28:7)
at async handleMainPromise (node:internal/modules/run_main:113:12)
```
Likely related to:
- https://github.com/matrixgpt/matrix-chatgpt-bot/issues/246
- https://github.com/matrixgpt/matrix-chatgpt-bot/pull/248
It can be tweaked around by overriding some default environment
variables (`roles/custom/matrix-bot-chatgpt/templates/env.j2`) in order to tweak them:
```
CHATGPT_MAX_CONTEXT_TOKENS=4097
CHATGPT_MAX_PROMPT_TOKENS=2500
```
This leads us to another issue:
```
node:internal/process/promises:289
triggerUncaughtException(err, true /* fromPromise */);
^
[Error: Failed to deserialize or serialize a JSON value missing field `version` at line 1 column 6704] {
code: 'GenericFailure'
}
Node.js v20.11.1
error Command failed with exit code 1.
```
... whatever that means.
* feat: auto-accept-invite module and docs
* fix: name typos and some forgot to adjust variables
* fix: accept only direct messages should work now and better wording
* changed: only_direct_messages variable naming
* feat: add logger, add synapse workers config
* Fix typo and add details about synapse-auto-acccept-invite
* Add newline at end of file
* Fix alignment
* Fix logger name for synapse_auto_accept_invite
The name of the logger needs to match the name of the Python module.
Ref: d673c67678/synapse_auto_accept_invite/__init__.py (L20)
* Add missing document start YAML annotation
* Remove trailing spaces
---------
Co-authored-by: Slavi Pantaleev <slavi@devture.com>
Media didn't work before this patch, likely because this feature is broken:
> N.B. Discord now requires signed expiring download links, which means this solution no longer works. In the future, a more dynamic solution may be implemented where requests go to the bridge and the bridge and the bridge refetches the message if necessary.
Source: https://docs.mau.fi/bridges/go/discord/direct-media.html
Moreover, most users more likely don't want this behavior and would
prefer to keep a complete mirror of the media on Matrix, instead of
going through two 3rd party servers to fetch the media on demand.
The default config for the bridge
(https://github.com/mautrix/discord/blob/main/example-config.yaml)
actually does not enable it.
It seems like 4ed522e8fe
(https://github.com/spantaleev/matrix-docker-ansible-deploy/pull/3133)
lied to us as to what upstream does. Poor PR review lead to this
anti-feature making it into the playbook.
Fixup for https://github.com/spantaleev/matrix-docker-ansible-deploy/pull/3017
This reverts 1cd82cf068 and also multiplies results by `1024`
so as to pass bytes to Synapse, not KB (as done before).
1cd82cf068 was correctly documenting what we were doing (passing KB values),
but that's incorrect.
Synapse's Config Conventions
(https://matrix-org.github.io/synapse/latest/usage/configuration/config_documentation.html#config-conventions)
are supposed to clear it up, but they don't currently state what happens when you pass a plain number (without a unit suffix).
Thankfully, the source code tells us:
bc1db16086/synapse/config/_base.py (L181-L206)
> If an integer is provided it is treated as bytes and is unchanged.
>
> String byte sizes can have a suffix of ...
> No suffix is understood as a plain byte count.
We were previously passing strings, but that has been improved in 3d73ec887a.
Regardless, non-suffixed values seem to be treated as bytes by Synapse,
so this patch changes the variables to use bytes.
Moreover, we're moving from `matrix_synapse_memtotal_kb` to
`matrix_synapse_cache_size_calculations_memtotal_bytes` as working with
the base unit everywhere is preferrable.
Here, we also introduce 2 new variables to allow for the caps to be
tweaked:
- `matrix_synapse_cache_size_calculations_max_cache_memory_usage_cap_bytes`
- `matrix_synapse_cache_size_calculations_target_cache_memory_usage_cap_bytes`
We're casting everything it `int`, but since Jinja templates are
involved, these values end up as strings anyway.
Doing `| int | to_json` is good, but we should only cast numbers to
integer, not empty strings, as that (0) may be interpreted differently
by Synapse.
To turn of auto-tuning, one is possibly supposed to pass empty strings:
> This option defaults to off, enable it by providing values for the sub-options listed below.
It could be that `0` is also considered "no value provided", but I
haven't verified that.
Related to https://github.com/spantaleev/matrix-docker-ansible-deploy/pull/3017
* Modify Synapse Cache Factor to use Auto Tune
Synapse has the ability to as it calls in its config auto tune caches.
This ability lets us set very high cache factors and then instead limit our resource use.
Defaults for this commit are 1/10th of what Element apparently runs for EMS stuff and matrix.org on Cache Factor and upstream documentation defaults for auto tune.
* Add vars to Synapse main.yml to control cache related config
This commit adds various cache related vars to main.yml for Synapse.
Some are auto tune and some are just adding explicit ways to control upstream vars.
* Updated Auto Tune figures
Autotuned figures have been bumped in consultation with other community members as to a reasonable level. Please note these defaults are more on the one of each workers side than they are on the monolith Side.
* Fix YML Error
The playbook is not happy with the previous state of this patch so this commit hopefully fixes it
* Add to_json to various Synapse tuning related configs
* Fix incorrect indication in homeserver.yaml.j2
* Minor cleanups
* Synapse Cache Autotuning Documentation
* Upgrade Synapse Cache Autotune to auto configure memory use
* Update Synapse Tuning docs to reflect automatic memory use configuration
* Fix Linting errors in synapses main.yml
* Rename variables for consistency (matrix_synapse_caches_autotuning_* -> matrix_synapse_cache_autotuning_*)
* Remove FIX ME comment about Synapse's `cache_autotuning`
`docs/maintenance-synapse.md` and `roles/custom/matrix-synapse/defaults/main.yml`
already contains documentation about these variables and the default values we set.
* Improve "Tuning caches and cache autotuning" documentation for Synapse
* Announce larger Synapse caches and cache auto-tuning
---------
Co-authored-by: Slavi Pantaleev <slavi@devture.com>
`matrix_synapse_federation_port_enabled` is defined like this:
```
matrix_synapse_federation_port_enabled: "{{ matrix_synapse_federation_enabled or matrix_synapse_federation_port_openid_resource_required }}"
```
Previously, people that disabled federation, but needed the `openid`
listener were running without these federation-related labels.
In this patch, we're also dropping the `not matrix_synapse_workers_enabled` condition,
because.. none of the Matrix-related labels would be applied anyway when
workers are enabled, thanks to `matrix_synapse_container_labels_matrix_related_labels_enabled`.
Fixes https://github.com/spantaleev/matrix-docker-ansible-deploy/issues/3127
This is a break in backward-compatibility for people disabling
`index.html` creation via the playbook but are managing their static
website files in another way (AUX role, etc).
* Fix s3-storage migrate and shell: container needs attachment to postgres network also
* Connect to s3-storage-provider migrate to multiple networks in multiple steps
Multiple `--network` calls lead to:
> docker: Error response from daemon: Container cannot be connected to network endpoints: NETWORK_1 NETWORK_2.
* Connect to s3-storage-provider shell to multiple networks in multiple steps
---------
Co-authored-by: Slavi Pantaleev <slavi@devture.com>
Most addons live in the same network by default (matrix-addons) right now,
so this network would have usually been created by some other addon.
Howevre, if this is the only addon someone uses, it may have remained
uncreated causing a problem.
I believe `specialized-workers` is a better name than `room-workers`,
because when enabled, 4 different types of specialized workers are
created:
- Room workers
- Sync workers
- Client readers
- Federation readers
Only one of these is called room-workers.
In the future, more specialized workers may be added, making the
`room-workers` preset name an even poorer choice.
Related to https://github.com/spantaleev/matrix-docker-ansible-deploy/pull/3100
Until now, the validation check would only get tripped up
if generic workers are used, combined with at least one EACH
other type of specialized workers.
This means that someone doing this:
```
matrix_synapse_workers_preset: one-of-each
matrix_synapse_workers_client_reader_workers_count: 5
```
.. would not have triggered this safety check.
Related to https://github.com/spantaleev/matrix-docker-ansible-deploy/pull/3100
Hookshot wants a trailing slash for this route.
If we let Hookshot redirect, it goes to `/widgetapi/v1/static/`,
instead of `/hookshot/widgetapi/v1/static/`, so we take this matter into our
own hands.
Public URLs are like: `/hookshot/widgetapi/v1/static/`
.. which get translated to requests for: `/widgetapi/v1/static/`
Previously, we were stripping the whole `/hookshot/widgetapi` prefix,
which is wrong.
Most of these files were defining a service, usually toward the end.
These lines have been moved upward.
Some components (mautrix-signal, mautrix-gmessages, etc.) were defining
a service conditionally (only if metrics are exposed, etc). This was
causing issues like these in the Traefik logs:
> level=error msg="service \"matrix-mautrix-twitter\" error: port is missing" providerName=docker container=matrix-mautrix-twitter-..
This changes the behavior of
`matrix_playbook_migration_matrix_nginx_proxy_uninstallation_enabled`
and is against what we initially described in the changelog entry,
but I've discovered some problems when the `matrix-nginx-proxy` service
and container remain running. They need to go.
After some checking, it seems like there's `/_synapse/client/oidc`,
but no such thing as `/_synapse/oidc`.
I'm not sure why we've been reverse-proxying these paths for so long
(even in as far back as the `matrix-nginx-proxy` days), but it's time we
put a stop to it.
The OIDC docs have been simplified. There's no need to ask people to
expose the useless `/_synapse/oidc` endpoint. OIDC requires
`/_synapse/client/oidc` and `/_synapse/client` is exposed by default
already.
Issues and Pull Requests were not migrated to the new
organization/repository, so `matrix-org/synapse/pull` and
`matrix-org/synapse/issues` references were kept as-is.
`matrix-org/synapse-s3-storage-provider` references were also kept,
as that module still continues living under the `matrix-org` organization.
This patch mainly aims to change documentation-related things, not actual
usage in full yet. For polish that, another more comprehensive patch is coming later.
This moves the comments from being just in Jinja,
to actually ending up in the generated `labels` file,
which makes inspection of the final result easier.
Also, some new lines were added here and there to make labels
more legible.
The generated file may still include weird new-lines due to
various `if` statements yielding content or not, but that's not so ugly
anymore - now that we have proper start/end sections that are visible in
the final `labels` file.
The old variables still work. The global lets us avoid
auto-detection logic like we're currently doing for
`matrix_nginx_proxy_proxy_matrix_federation_api_enabled`.
In the future, we'd just be able to reference
`matrix_homeserver_federation_enabled` and know the up-to-date value
regardless of homeserver.
This was meant to serve as an intermediary for services needing to reach
the homeserver. It was used like that for a while in this
`bye-bye-nginx-proxy` branch, but was never actually public.
It has recently been superseded by homeserver-like services injecting
themselves into a new internal Traefik entrypoint
(see `matrix_playbook_internal_matrix_client_api_traefik_entrypoint_*`),
so `matrix-homeserver-proxy` is no longer necessary.
---
This is probably a good moment to share some benchmarks and reasons
for going with the internal Traefik entrypoint as opposed to this nginx
service.
1. (1400 rps) Directly to Synapse (`ab -n 1000 -c 100 http://matrix-synapse:8008/_matrix/client/versions`
2. (~900 rps) Via `matrix-homeserver-proxy` (nginx) proxying to Synapse (`ab -n 1000 -c 100 http://matrix-homeserver-proxy:8008/_matrix/client/versions`)
3. (~1200 rps) Via the new internal entrypoint of Traefik (`matrix-internal-matrix-client-api`) proxying to Synapse (`ab -n 1000 -c 100 http://matrix-traefik:8008/_matrix/client/versions`)
Besides Traefik being quicker for some reason, there are also other
benefits to not having this `matrix-homeserver-proxy` component:
- we can reuse what we have in terms of labels. Services can register a few extra labels on the new Traefik entrypoint
- we don't need services (like `matrix-media-repo`) to inject custom nginx configs into `matrix-homeserver-proxy`. They just need to register labels, like they do already.
- Traefik seems faster than nginx on this benchmark for some reason, which is a nice bonus
- no need to run one extra container (`matrix-homeserver-proxy`) and execute one extra Ansible role
- no need to maintain a setup where some people run the `matrix-homeserver-proxy` component (because they have route-stealing services like `matrix-media-repo` enabled) and others run an optimized setup without this component and everything needs to be rewired to talk to the homeserver directly. Now, everyone can go through Traefik and we can all run an identical setup
Downsides of the new Traefik entrypoint setup are that:
- all addon services that need to talk to the homeserver now depend on Traefik
- people running their own Traefik setup will be inconvenienced - they
need to manage one additional entrypoint
We'd be adding integration with an internal Traefik entrypoint
(`matrix_playbook_internal_matrix_client_api_traefik_entrypoint`),
so renaming helps disambiguate things.
There's no need for deperecation tasks, because the old names
have only been part of this `bye-bye-nginx-proxy` branch and not used by
anyone publicly.
This is a bit of a compatibility break.
The role was defaulting the Postgres password to `some-password` and we
auto-generate it now.
However, rebuilding both Postgres and this service should unify the
database credentials and the service configs to the new value.
