add synapse-auto-compressor workaround, fixes #3397 (#3473)

* add synapse-auto-compressor workaround, fixes #3397

* Clarify what the PG-prefixed variables are for

---------

Co-authored-by: Slavi Pantaleev <slavi@devture.com>

CHANGELOG.md

@@ -1,3 +1,16 @@
+# 2024-08-15
+
+## matrix-media-repo now configured for Authenticated Media
+
+Thanks to [Michael Hollister](https://github.com/Michael-Hollister) from [FUTO](https://www.futo.org/), our matrix-media-repo implementation now automatically [sets up signing keys](https://docs.t2bot.io/matrix-media-repo/v1.3.5/installation/signing-key/) for Authenticated Media (as per [MSC3916](https://github.com/matrix-org/matrix-spec-proposals/pull/3916)).
+
+If you had never heard of Authenticated Media before, the [Sunsetting unauthenticated media](https://matrix.org/blog/2024/06/26/sunsetting-unauthenticated-media/) article on [matrix.org](https://matrix.org/) is a good introduction.
+
+This feature is enabled for matrix-media-repo installations by default and will append an additional (matrix-media-repo-generated signing key) to your homeserver's (Synapse or Dendrite) signing key. See the [Signing keys](./docs/configuring-playbook-matrix-media-repo.md#signing-keys) and [Key backup and revoking](./docs/configuring-playbook-matrix-media-repo.md#key-backup-and-revoking) sections of the matrix-media-repo documentation for more details.
+
+If you'd like to avoid this new feature, you can disable it by setting `matrix_media_repo_generate_signing_key: false` in your `vars.yml` configuration file.
+
+
 # 2024-08-08
 
 ## (Backward Compatibility Break) matrix-corporal has been upgraded to v3

docs/configuring-playbook-matrix-media-repo.md

@@ -27,8 +27,6 @@ The repo is pre-configured for integrating with the Postgres database, Traefik p
 
 By default, the media-repo will use the local filesystem for data storage. You can alternatively use a `s3` cloud backend as well. Access token caching is also enabled by default since the logout endpoints are proxied through the media repo.
 
-**Note:** If you want to use authenticated media endpoints ([MSC3916](https://github.com/matrix-org/matrix-spec-proposals/pull/3916)), you must configure a signing key for your MMR instance to authorize outbound federation requests. See https://docs.t2bot.io/matrix-media-repo/v1.3.5/installation/signing-key/ for more details on how to configure your server with a signing key.
-
 ## Configuring the media-repo
 
 Additional common configuration options:
@@ -91,6 +89,26 @@ matrix_media_repo_datastore_s3_opts_bucket_name: "your-media-bucket"
 
 Full list of configuration options with documentation can be found in [`roles/custom/matrix-media-repo/defaults/main.yml`](https://github.com/spantaleev/matrix-docker-ansible-deploy/blob/master/roles/custom/matrix-media-repo/defaults/main.yml)
 
+## Signing Keys
+
+Authenticated media endpoints ([MSC3916](https://github.com/matrix-org/matrix-spec-proposals/pull/3916)) requires MMR to have a configured signing key to authorize outbound federation requests. Additionally, the signing key must be merged with your homeserver's signing key file.
+
+The playbook default is to generate a MMR signing key when invoking the setup role and merge it with your homeserver if you are using Synapse or Dendrite. This can be disabled if desired by setting the option in your inventory:
+
+```yaml
+matrix_media_repo_generate_signing_key: false
+```
+
+If you wish to manually generate the signing key and merge it with your homeserver's signing key file, see https://docs.t2bot.io/matrix-media-repo/v1.3.5/installation/signing-key/ for more details.
+
+**Note that if you uninstall MMR from the playbook, it will not remove the old MMR signing key from your homeserver's signing key file. You will have to remove it manually.**
+
+### Key backup and revoking
+
+Since your homeserver signing key file is modified by the playbook, a backup will be created in `HOMESERVER_DIR/config/DOMAIN.signing.key.backup`. If you need to remove/revoke old keys, you can restore from this backup or remove the MMR key id from your `DOMAIN.signing.key` file.
+
+Additionally, its recommended after revoking a signing key to update your homeserver config file (`old_signing_keys` field for Synapse and `old_private_keys` for Dendrite). See your homeserver config file for further documentation on how to populate the field.
+
 ## Importing data from an existing media store
 
 If you want to add this repo to an existing homeserver managed by the playbook, you will need to import existing media into MMR's database or you will lose access to older media while it is active. MMR versions up to `v1.3.3` only support importing from Synapse, but newer versions (at time of writing: only `latest`) also support importing from Dendrite.

docs/configuring-playbook-ssl-certificates.md

@@ -98,3 +98,29 @@ aux_file_definitions:
               certFile: /ssl/cert.pem
               keyFile: /ssl/privkey.pem
 ```
+
+## Using a DNS-01 ACME challenge type, instead of HTTP-01
+
+You can configure Traefik to use the [DNS-01 challenge type](https://letsencrypt.org/docs/challenge-types/#dns-01-challenge) for Let's Encrypt. This is less commonly used than the default [HTTP-01 challenge type](https://letsencrypt.org/docs/challenge-types/#http-01-challenge), but it can be helpful to:
+
+- hide your public IP from Let's Encrypt logs
+- allow you to obtain SSL certificates for servers which are not accessible (via HTTP) from the public internet (and for which the HTTP-01 challenge would fail)
+
+This is an example for how to edit the `vars.yml` file if you're using Cloudflare:
+
+```yaml
+devture_traefik_config_certificatesResolvers_acme_dnsChallenge_enabled: true
+devture_traefik_config_certificatesResolvers_acme_dnsChallenge_provider: "cloudflare"
+devture_traefik_config_certificatesResolvers_acme_dnsChallenge_delayBeforeCheck: 60
+devture_traefik_config_certificatesResolvers_acme_dnsChallenge_resolvers:
+  - "1.1.1.1:53"
+devture_traefik_environment_variables_additional_variables: |
+  CF_API_EMAIL=redacted
+  CF_ZONE_API_TOKEN=redacted
+  CF_DNS_API_TOKEN=redacted
+  LEGO_DISABLE_CNAME_SUPPORT=true
+```
+
+Make sure to change the value of "provider" to your particular DNS solution, and provide the appropriate environment variables. The full list of supported providers is available [here](https://doc.traefik.io/traefik/https/acme/#providers).
+
+This example assumes you're using Cloudflare to manage your DNS zone. Note that it requires the use of two tokens: one for reading all zones (`CF_ZONE_API_TOKEN`) and another that must be able to edit the particular domain you're using (`CF_DNS_API_TOKEN`). For security, it's recommended that you create two fine-grained tokens for this purpose, but you might choose to use the same token for both.

group_vars/matrix_servers

@@ -3653,6 +3653,11 @@ matrix_media_repo_systemd_required_services_list_auto: |
     ([devture_postgres_identifier ~ '.service'] if devture_postgres_enabled and matrix_media_repo_database_hostname == devture_postgres_connection_hostname else [])
   }}
 
+matrix_media_repo_generate_signing_key: "{{ matrix_homeserver_implementation == 'synapse' or matrix_homeserver_implementation == 'dendrite'}}"
+matrix_media_repo_homeserver_signing_key: "{{ matrix_media_repo_synapse_signing_key if matrix_homeserver_implementation == 'synapse' else (matrix_media_repo_dendrite_signing_key if matrix_homeserver_implementation == 'dendrite' else '') }}"
+matrix_media_repo_synapse_signing_key: "{{ matrix_synapse_config_dir_path }}/{{ matrix_server_fqn_matrix }}.signing.key"
+matrix_media_repo_dendrite_signing_key: "{{ matrix_dendrite_config_dir_path }}/{{ matrix_server_fqn_matrix }}.signing.pem"
+
 # Auto configured server setup by the playbook
 matrix_media_repo_homeservers_auto:
   -  # Keep the dash from this line.
@@ -3677,7 +3682,7 @@ matrix_media_repo_homeservers_auto:
     # The signing key to use for authorizing outbound federation requests. If not specified,
     # requests will not be authorized. See https://docs.t2bot.io/matrix-media-repo/v1.3.5/installation/signing-key/
     # for details.
-    signingKeyPath: ""
+    signingKeyPath: "{{ '/config/' + matrix_media_repo_identifier + '.signing.key' if matrix_media_repo_generate_signing_key else '' }}"
 
 matrix_media_repo_homeserver_federation_enabled: "{{ matrix_homeserver_federation_enabled }}"
 
@@ -4509,6 +4514,11 @@ matrix_synapse_report_stats_endpoint: "http://{{ matrix_synapse_usage_exporter_i
 # Not enabled by default, for now
 matrix_synapse_auto_compressor_enabled: false
 
+matrix_synapse_auto_compressor_uid: "{{ matrix_user_uid }}"
+matrix_synapse_auto_compressor_gid: "{{ matrix_user_gid }}"
+
+matrix_synapse_auto_compressor_postgres_image: "{{ devture_postgres_container_image_to_use }}"
+
 matrix_synapse_auto_compressor_container_image_self_build: "{{ matrix_architecture not in ['amd64'] }}"
 
 matrix_synapse_auto_compressor_container_network: "{{ (devture_postgres_container_network if (devture_postgres_enabled and matrix_synapse_auto_compressor_database_hostname == matrix_synapse_database_host and matrix_synapse_database_host == devture_postgres_connection_hostname) else 'matrix-synapse-auto-compressor') }}"

requirements.yml

@@ -22,7 +22,7 @@
   version: v4.98-r0-1-0
   name: exim_relay
 - src: git+https://gitlab.com/etke.cc/roles/grafana.git
-  version: v11.1.3-1
+  version: v11.1.4-0
   name: grafana
 - src: git+https://github.com/mother-of-all-self-hosting/ansible-role-jitsi.git
   version: v9646-0

roles/custom/matrix-media-repo/defaults/main.yml

@@ -319,6 +319,14 @@ matrix_media_repo_homeservers_auto: []
 # Additional servers to be managed by MMR
 matrix_media_repo_homeservers_additional: []
 
+# If true, the playbook will generate a signing key when the setup role is invoked.
+# This is adjusted in `group_vars/matrix_servers` depending on the homeserver implementation.
+matrix_media_repo_generate_signing_key: false
+
+# Path where the homeserver signing key is located. Set automatically in
+# `group_vars/matrix_servers` depending on which homeserver is being used.
+matrix_media_repo_homeserver_signing_key: ""
+
 # Options for controlling how access tokens work with the media repo. It is recommended that if
 # you are going to use these options that the `/logout` and `/logout/all` client-server endpoints
 # be proxied through this process. They will also be called on the homeserver, and the response

roles/custom/matrix-media-repo/tasks/setup_install.yml

@@ -77,6 +77,63 @@
       changed_when: true
       when: "matrix_media_repo_git_pull_results.changed | bool or matrix_media_repo_docker_image_check_result.stdout == ''"
 
+- name: Check existence of media-repo signing key
+  ansible.builtin.stat:
+    path: "{{ matrix_media_repo_config_path }}/{{ matrix_media_repo_identifier }}.signing.key"
+  register: matrix_media_repo_signing_key_stat
+
+- when: "matrix_media_repo_generate_signing_key | bool and not (matrix_media_repo_signing_key_stat.stat.exists | bool)"
+  block:
+    - name: Generate media-repo signing key
+      ansible.builtin.command:
+        cmd: |
+          {{ devture_systemd_docker_base_host_command_docker }} run
+          --rm
+          --name={{ matrix_media_repo_identifier }}
+          --user={{ matrix_synapse_uid }}:{{ matrix_synapse_gid }}
+          --cap-drop=ALL
+          --mount type=bind,src={{ matrix_media_repo_config_path }},dst=/config
+          --workdir='/config'
+          --entrypoint='generate_signing_key'
+          {{ matrix_media_repo_docker_image }}
+          -output {{ matrix_media_repo_identifier }}.signing.key.TEMP
+        creates: "{{ matrix_media_repo_config_path }}/{{ matrix_media_repo_identifier }}.signing.key.TEMP"
+
+    - name: Merge media-repo signing key with homeserver signing key
+      ansible.builtin.command:
+        cmd: |
+          {{ devture_systemd_docker_base_host_command_docker }} run
+          --rm
+          --name={{ matrix_media_repo_identifier }}
+          --user={{ matrix_synapse_uid }}:{{ matrix_synapse_gid }}
+          --cap-drop=ALL
+          --mount type=bind,src={{ matrix_media_repo_config_path }},dst=/config
+          --mount type=bind,src={{ matrix_media_repo_homeserver_signing_key | dirname }},dst=/homeserver-signing-key-dir
+          --workdir='/config'
+          --entrypoint='combine_signing_keys'
+          {{ matrix_media_repo_docker_image }}
+          -format {{ matrix_homeserver_implementation }} -output /homeserver-signing-key-dir/{{ matrix_media_repo_homeserver_signing_key | basename }}.merged /homeserver-signing-key-dir/{{ matrix_media_repo_homeserver_signing_key | basename }} {{ matrix_media_repo_identifier }}.signing.key.TEMP
+        creates: "{{ matrix_media_repo_homeserver_signing_key }}.merged"
+
+    - name: Backup existing homeserver signing key before replacing it
+      ansible.builtin.copy:
+        remote_src: true
+        src: "{{ matrix_media_repo_homeserver_signing_key }}"
+        dest: "{{ matrix_media_repo_homeserver_signing_key }}.{{ matrix_homeserver_implementation }}.backup"
+        mode: 0644
+        owner: "{{ matrix_user_username }}"
+        group: "{{ matrix_user_groupname }}"
+
+    - name: Replace homeserver signing key with merged signing key
+      ansible.builtin.command:
+        cmd: "mv {{ matrix_media_repo_homeserver_signing_key }}.merged {{ matrix_media_repo_homeserver_signing_key }}"
+        removes: "{{ matrix_media_repo_homeserver_signing_key }}.merged"
+
+    - name: Finalize media-repo signing key setup
+      ansible.builtin.command:
+        cmd: "mv {{ matrix_media_repo_config_path }}/{{ matrix_media_repo_identifier }}.signing.key.TEMP {{ matrix_media_repo_config_path }}/{{ matrix_media_repo_identifier }}.signing.key"
+        removes: "{{ matrix_media_repo_config_path }}/{{ matrix_media_repo_identifier }}.signing.key.TEMP"
+
 - name: Ensure media-repo container network is created
   community.general.docker_network:
     enable_ipv6: "{{ devture_systemd_docker_base_ipv6_enabled }}"

roles/custom/matrix-synapse-auto-compressor/defaults/main.yml

@@ -8,6 +8,13 @@ matrix_synapse_auto_compressor_enabled: true
 # renovate: datasource=docker depName=registry.gitlab.com/etke.cc/rust-synapse-compress-state
 matrix_synapse_auto_compressor_version: v0.1.4
 
+# note: UID/GID better to match the UID/GID of the postgres container, but it doesn't really matter, as volumes are not used here
+matrix_synapse_auto_compressor_uid: ''
+matrix_synapse_auto_compressor_gid: ''
+
+# note: the postgres image version doesn't matter, as it is used for a SQL script that doesn't use any specific features
+matrix_synapse_auto_compressor_postgres_image: "postgres"
+
 matrix_synapse_auto_compressor_base_path: "{{ matrix_base_data_path }}/synapse-auto-compressor"
 matrix_synapse_auto_compressor_container_src_files_path: "{{ matrix_synapse_auto_compressor_base_path }}/container-src"
 

roles/custom/matrix-synapse-auto-compressor/tasks/install.yml

@@ -14,7 +14,7 @@
     - path: "{{ matrix_synapse_auto_compressor_container_src_files_path }}"
       when: "{{ matrix_synapse_auto_compressor_container_image_self_build }}"
 
-- name: Ensure synapse-auto-compressor labels installed
+- name: Ensure synapse-auto-compressor env file is installed
   ansible.builtin.template:
     src: "{{ role_path }}/templates/env.j2"
     dest: "{{ matrix_synapse_auto_compressor_base_path }}/env"
@@ -22,6 +22,14 @@
     owner: "{{ matrix_user_username }}"
     group: "{{ matrix_user_groupname }}"
 
+- name: Ensure synapse-auto-compressor workaround script is installed
+  ansible.builtin.template:
+    src: "{{ role_path }}/templates/matrix-synapse-auto-compressor-fix.sh.j2"
+    dest: "{{ matrix_synapse_auto_compressor_base_path }}/matrix-synapse-auto-compressor-fix.sh"
+    mode: 0750
+    owner: "{{ matrix_user_username }}"
+    group: "{{ matrix_user_groupname }}"
+
 - name: Ensure synapse-auto-compressor image is pulled
   community.docker.docker_image:
     name: "{{ matrix_synapse_auto_compressor_container_image }}"

roles/custom/matrix-synapse-auto-compressor/templates/env.j2

@@ -1 +1,7 @@
 POSTGRES_LOCATION={{ matrix_synapse_auto_compressor_environment_variable_postgres_location }}
+
+# PG-prefixed variables below are for the matrix-synapse-auto-compressor-fix.sh script
+PGHOST={{ matrix_synapse_auto_compressor_database_hostname }}
+PGDATABASE={{ matrix_synapse_auto_compressor_database_name }}
+PGUSER={{ matrix_synapse_auto_compressor_database_username }}
+PGPASSWORD={{ matrix_synapse_auto_compressor_database_password }}

roles/custom/matrix-synapse-auto-compressor/templates/matrix-synapse-auto-compressor-fix.sh.j2

@@ -0,0 +1,47 @@
+#jinja2: lstrip_blocks: "True"
+#!/bin/bash
+
+# this script is used to workaround the https://github.com/matrix-org/rust-synapse-compress-state/issues/78,
+# and it is based on postgres' cli-non-interactive and https://github.com/matrix-org/rust-synapse-compress-state/issues/78#issuecomment-1409932869
+
+docker run \
+	--rm \
+	--user={{ matrix_synapse_auto_compressor_uid }}:{{ matrix_synapse_auto_compressor_gid }} \
+	--cap-drop=ALL \
+	--interactive \
+	--network={{ matrix_synapse_auto_compressor_container_network }} \
+	--env-file={{ matrix_synapse_auto_compressor_base_path }}/env \
+	{{ matrix_synapse_auto_compressor_postgres_image }} \
+	psql -h {{ matrix_synapse_auto_compressor_database_hostname }} \
+	<<_EOF
+BEGIN;
+
+DELETE
+FROM state_compressor_state AS scs
+WHERE NOT EXISTS
+    (SELECT *
+     FROM rooms AS r
+     WHERE r.room_id = scs.room_id);
+
+DELETE
+FROM state_compressor_state AS scs
+WHERE scs.room_id in
+    (SELECT DISTINCT room_id
+     FROM state_compressor_state AS scs2
+     WHERE scs2.current_head IS NOT NULL
+       AND NOT EXISTS
+         (SELECT *
+          FROM state_groups AS sg
+          WHERE sg.id = scs2.current_head));
+
+DELETE
+FROM state_compressor_progress AS scp
+WHERE NOT EXISTS
+    (SELECT *
+     FROM state_compressor_state AS scs
+     WHERE scs.room_id = scp.room_id);
+
+COMMIT;
+_EOF
+
+# vim: ft=sh

roles/custom/matrix-synapse-auto-compressor/templates/matrix-synapse-auto-compressor.service.j2

@@ -15,6 +15,7 @@ Type=oneshot
 Environment="HOME={{ devture_systemd_docker_base_systemd_unit_home_path }}"
 ExecStartPre=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} stop --time={{ devture_systemd_docker_base_container_stop_grace_time_seconds }} matrix-synapse-auto-compressor 2>/dev/null || true'
 ExecStartPre=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} rm matrix-synapse-auto-compressor 2>/dev/null || true'
+ExecStartPre=-{{ devture_systemd_docker_base_host_command_sh }} {{ matrix_synapse_auto_compressor_base_path }}/matrix-synapse-auto-compressor-fix.sh
 
 ExecStartPre={{ devture_systemd_docker_base_host_command_docker }} create \
 			--rm \
@@ -22,7 +23,7 @@ ExecStartPre={{ devture_systemd_docker_base_host_command_docker }} create \
 			--log-driver=none \
 			--cap-drop=ALL \
 			--read-only \
-			--user={{ matrix_user_uid }}:{{ matrix_user_gid }} \
+			--user={{ matrix_synapse_auto_compressor_uid }}:{{ matrix_synapse_auto_compressor_gid }} \
 			--network={{ matrix_synapse_auto_compressor_container_network }} \
 			--env-file={{ matrix_synapse_auto_compressor_base_path }}/env \
 			--entrypoint=/bin/sh \