From 226c550ffafc08d3d34d4a92882ed3729124faa0 Mon Sep 17 00:00:00 2001
From: Slavi Pantaleev <slavi@devture.com>
Date: Thu, 15 Sep 2022 07:05:25 +0300
Subject: [PATCH 01/10] Add support for stream writer Synapse workers

As stream writer workers are also powered by the `generic_worker`
Synapse app, this necessitated that we provide means for distinguishing
between them and regular `generic_workers`.

I've also taken the time to optimize nginx configuration generation
(more Jinja2 macro usage, less duplication).

Worker names have also changed.
Workers are now named sequentially like this:
- `matrix-synapse-worker-0-generic`
- `matrix-synapse-worker-1-stream-writer-typing`
- `matrix-synapse-worker-2-pusher`

instead of `matrix-synapse-worker_generic_worker-18111` (indexed with a
port number).

People who modify `matrix_synapse_workers_enabled_list` directly will
need to adjust their configuration.
---
 CHANGELOG.md                                  |  39 ++++++
 ...configuring-playbook-prometheus-grafana.md |   8 +-
 group_vars/matrix_servers                     |   7 +
 roles/matrix-nginx-proxy/defaults/main.yml    |   5 +
 .../nginx/conf.d/matrix-synapse.conf.j2       | 125 +++++++++++-------
 .../templates/prometheus.yml.j2               |   5 +-
 roles/matrix-synapse/defaults/main.yml        | 112 ++++++++++++++--
 roles/matrix-synapse/tasks/init.yml           |  10 +-
 .../tasks/synapse/workers/init.yml            |  50 ++++++-
 .../inject_systemd_services_for_worker.yml    |  19 ---
 .../synapse/workers/util/inject_worker.yml    |  65 +++++++++
 .../workers/util/setup_files_for_worker.yml   |  10 +-
 .../matrix-synapse/tasks/validate_config.yml  |   5 +
 .../templates/synapse/homeserver.yaml.j2      |   8 +-
 .../external_prometheus.yml.example.j2        |   7 +-
 .../systemd/matrix-synapse-worker.service.j2  |   2 +-
 .../synapse/systemd/matrix-synapse.service.j2 |   2 +-
 .../templates/synapse/worker.yaml.j2          |  36 +++--
 roles/matrix-synapse/vars/main.yml            |  55 ++++++++
 19 files changed, 449 insertions(+), 121 deletions(-)
 delete mode 100644 roles/matrix-synapse/tasks/synapse/workers/util/inject_systemd_services_for_worker.yml
 create mode 100644 roles/matrix-synapse/tasks/synapse/workers/util/inject_worker.yml

diff --git a/CHANGELOG.md b/CHANGELOG.md
index df4957eb5..2b9b149e1 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,3 +1,42 @@
+# 2022-09-15
+
+## (Potential Backward Compatibility Break) Major changes to Synapse workers
+
+People who are interested in running a Synapse worker setup should know that **our Synapse worker implementation is much more powerful now**:
+
+- we've added [Stream writers support](#stream-writers-support)
+- see the [Potential Backward Incompatibilities after these Synapse worker changes](#potential-backward-incompatibilities-after-these-synapse-worker-changes)
+
+### Stream writers support
+
+From now on, the playbook lets you easily set up various [stream writer workers](https://matrix-org.github.io/synapse/latest/workers.html#stream-writers) which can handle different streams (`events` stream; `typing` URL endpoints, `to_device` URL endpoints, `account_data` URL endpoints, `receipts` URL endpoints, `presence` URL endpoints). All of this work was previously handled by the main Synapse process, but can now be offloaded to stream writer worker processes.
+
+If you're using `matrix_synapse_workers_preset: one-of-each`, you'll automatically get 6 additional workers (one for each of the above stream types). Our `little-federation-helper` preset (meant to be quite minimal and focusing in improved federation performance) does not include stream writer workers.
+
+If you'd like to customize the number of workers we also make that possible using these variables:
+
+```yaml
+# Synapse only supports more than 1 worker for the `events` stream.
+# All other streams can utilize either 0 or 1 workers, not more than that.
+matrix_synapse_workers_stream_writer_events_stream_workers_count: 5
+matrix_synapse_workers_stream_writer_typing_stream_workers_count: 1
+matrix_synapse_workers_stream_writer_to_device_stream_workers_count: 1
+matrix_synapse_workers_stream_writer_account_data_stream_workers_count: 1
+matrix_synapse_workers_stream_writer_receipts_stream_workers_count: 1
+matrix_synapse_workers_stream_writer_presence_stream_workers_count: 1
+```
+
+### Potential Backward Incompatibilities after these Synapse worker changes
+
+Below we'll discuss **potential backward incompatibilities**.
+
+- **Worker names** (container names, systemd services, worker configuration files) **have changed**. Workers are now labeled sequentially (e.g. `matrix-synapse-worker_generic_worker-18111` -> `matrix-synapse-worker-generic-0`). The playbook will handle these changes automatically.
+
+- **Metric endpoints have also changed** (`/metrics/synapse/worker/generic_worker-18111` -> `/metrics/synapse/worker/generic-worker-0`). If you're [collecting metrics to an external Prometheus server](docs/configuring-playbook-prometheus-grafana.md#collecting-metrics-to-an-external-prometheus-server), consider revisiting our [Collecting Synapse worker metrics to an external Prometheus server](docs/configuring-playbook-prometheus-grafana.md#collecting-synapse-worker-metrics-to-an-external-prometheus-server) docs and updating your Prometheus configuration. **If you're collecting metrics to the integrated Prometheus server** (not enabled by default), **your Prometheus configuration will be updated automatically**. Old data (from before this change) may stick around though.
+
+- **the format of `matrix_synapse_workers_enabled_list` has changed**. You were never advised to use this variable for directly creating workers (we advise people to control workers using `matrix_synapse_workers_preset` or by tweaking `matrix_synapse_workers_*_workers_count` variables only), but some people may have started using the `matrix_synapse_workers_enabled_list` variable to gain more control over workers. If you're one of them, you'll need to adjust its value. See `roles/matrix-synapse/defaults/main.yml` for more information on the new format. The playbook will also do basic validation and complain if you got something wrong.
+
+
 # 2022-09-09
 
 ## Cactus Comments support
diff --git a/docs/configuring-playbook-prometheus-grafana.md b/docs/configuring-playbook-prometheus-grafana.md
index b2878c12b..f86566a94 100644
--- a/docs/configuring-playbook-prometheus-grafana.md
+++ b/docs/configuring-playbook-prometheus-grafana.md
@@ -90,11 +90,11 @@ matrix_nginx_proxy_proxy_matrix_metrics_additional_user_location_configuration_b
 
 Using `matrix_nginx_proxy_proxy_matrix_metrics_additional_user_location_configuration_blocks` only takes effect if `matrix_nginx_proxy_proxy_matrix_metrics_enabled: true` (see above).
 
-Note : The playbook will hash the basic_auth password for you on setup. Thus, you need to give the plain-text version of the password as a variable. 
+Note : The playbook will hash the basic_auth password for you on setup. Thus, you need to give the plain-text version of the password as a variable.
 
 ### Collecting Synapse worker metrics to an external Prometheus server
 
-If you are using workers (`matrix_synapse_workers_enabled: true`) and have enabled `matrix_synapse_metrics_proxying_enabled` as described above, the playbook will also automatically expose all Synapse worker threads' metrics to `https://matrix.DOMAIN/metrics/synapse/worker/TYPE-ID`, where `TYPE` corresponds to the type and `ID` to the instanceId of a worker as exemplified in `matrix_synapse_workers_enabled_list`.
+If you are using workers (`matrix_synapse_workers_enabled: true`) and have enabled `matrix_synapse_metrics_proxying_enabled` as described above, the playbook will also automatically expose all Synapse worker threads' metrics to `https://matrix.DOMAIN/metrics/synapse/worker/ID`, where `ID` corresponds to the worker `id` as exemplified in `matrix_synapse_workers_enabled_list`.
 
 The playbook also generates an exemplary config file (`/matrix/synapse/external_prometheus.yml.template`) with all the correct paths which you can copy to your Prometheus server and adapt to your needs. Make sure to edit the specified `password_file` path and contents and path to your `synapse-v2.rules`.
 It will look a bit like this:
@@ -111,8 +111,8 @@ scrape_configs:
         labels:
           job: "master"
           index: 1
-  - job_name: 'synapse-generic_worker-1'
-    metrics_path: /metrics/synapse/worker/generic_worker-18111
+  - job_name: 'matrix-synapse-synapse-worker-generic-worker-0'
+    metrics_path: /metrics/synapse/worker/generic-worker-0
     scheme: https
     basic_auth:
       username: prometheus
diff --git a/group_vars/matrix_servers b/group_vars/matrix_servers
index c0d831e1e..e891a3d9a 100755
--- a/group_vars/matrix_servers
+++ b/group_vars/matrix_servers
@@ -1760,6 +1760,11 @@ matrix_nginx_proxy_synapse_workers_enabled: "{{ matrix_synapse_workers_enabled }
 matrix_nginx_proxy_synapse_workers_list: "{{ matrix_synapse_workers_enabled_list }}"
 matrix_nginx_proxy_synapse_generic_worker_client_server_locations: "{{ matrix_synapse_workers_generic_worker_client_server_endpoints }}"
 matrix_nginx_proxy_synapse_generic_worker_federation_locations: "{{ matrix_synapse_workers_generic_worker_federation_endpoints }}"
+matrix_nginx_proxy_synapse_stream_writer_typing_stream_worker_client_server_locations: "{{ matrix_synapse_workers_stream_writer_typing_stream_worker_client_server_endpoints }}"
+matrix_nginx_proxy_synapse_stream_writer_to_device_stream_worker_client_server_locations: "{{ matrix_synapse_workers_stream_writer_to_device_stream_worker_client_server_endpoints }}"
+matrix_nginx_proxy_synapse_stream_writer_account_data_stream_worker_client_server_locations: "{{ matrix_synapse_workers_stream_writer_account_data_stream_worker_client_server_endpoints }}"
+matrix_nginx_proxy_synapse_stream_writer_receipts_stream_worker_client_server_locations: "{{ matrix_synapse_workers_stream_writer_receipts_stream_worker_client_server_endpoints }}"
+matrix_nginx_proxy_synapse_stream_writer_presence_stream_worker_client_server_locations: "{{ matrix_synapse_workers_stream_writer_presence_stream_worker_client_server_endpoints }}"
 matrix_nginx_proxy_synapse_media_repository_locations: "{{matrix_synapse_workers_media_repository_endpoints|default([]) }}"
 matrix_nginx_proxy_synapse_user_dir_locations: "{{ matrix_synapse_workers_user_dir_endpoints|default([]) }}"
 
@@ -1767,6 +1772,8 @@ matrix_nginx_proxy_systemd_wanted_services_list: |
   {{
     ['matrix-' + matrix_homeserver_implementation + '.service']
     +
+    (matrix_synapse_webserving_workers_systemd_services_list if matrix_homeserver_implementation == 'synapse' and matrix_synapse_workers_enabled)
+    +
     (['matrix-corporal.service'] if matrix_corporal_enabled else [])
     +
     (['matrix-ma1sd.service'] if matrix_ma1sd_enabled else [])
diff --git a/roles/matrix-nginx-proxy/defaults/main.yml b/roles/matrix-nginx-proxy/defaults/main.yml
index a74842158..88c99ea9b 100644
--- a/roles/matrix-nginx-proxy/defaults/main.yml
+++ b/roles/matrix-nginx-proxy/defaults/main.yml
@@ -623,6 +623,11 @@ matrix_nginx_proxy_synapse_workers_enabled: false
 matrix_nginx_proxy_synapse_workers_list: []
 matrix_nginx_proxy_synapse_generic_worker_client_server_locations: []
 matrix_nginx_proxy_synapse_generic_worker_federation_locations: []
+matrix_nginx_proxy_synapse_stream_writer_typing_stream_worker_client_server_locations: []
+matrix_nginx_proxy_synapse_stream_writer_to_device_stream_worker_client_server_locations: []
+matrix_nginx_proxy_synapse_stream_writer_account_data_stream_worker_client_server_locations: []
+matrix_nginx_proxy_synapse_stream_writer_receipts_stream_worker_client_server_locations: []
+matrix_nginx_proxy_synapse_stream_writer_presence_stream_worker_client_server_locations: []
 matrix_nginx_proxy_synapse_media_repository_locations: []
 matrix_nginx_proxy_synapse_user_dir_locations: []
 
diff --git a/roles/matrix-nginx-proxy/templates/nginx/conf.d/matrix-synapse.conf.j2 b/roles/matrix-nginx-proxy/templates/nginx/conf.d/matrix-synapse.conf.j2
index 81e31a7c0..bbce7462b 100644
--- a/roles/matrix-nginx-proxy/templates/nginx/conf.d/matrix-synapse.conf.j2
+++ b/roles/matrix-nginx-proxy/templates/nginx/conf.d/matrix-synapse.conf.j2
@@ -1,23 +1,52 @@
 #jinja2: lstrip_blocks: "True"
 
 {% set generic_workers = matrix_nginx_proxy_synapse_workers_list | selectattr('type', 'equalto', 'generic_worker') | list %}
+{% set stream_writer_typing_stream_workers = matrix_nginx_proxy_synapse_workers_list | selectattr('type', 'equalto', 'stream_writer') | selectattr('stream_writer_stream', 'equalto', 'typing') | list %}
+{% set stream_writer_to_device_stream_workers = matrix_nginx_proxy_synapse_workers_list | selectattr('type', 'equalto', 'stream_writer') | selectattr('stream_writer_stream', 'equalto', 'to_device') | list %}
+{% set stream_writer_account_data_stream_workers = matrix_nginx_proxy_synapse_workers_list | selectattr('type', 'equalto', 'stream_writer') | selectattr('stream_writer_stream', 'equalto', 'account_data') | list %}
+{% set stream_writer_receipts_stream_workers = matrix_nginx_proxy_synapse_workers_list | selectattr('type', 'equalto', 'stream_writer') | selectattr('stream_writer_stream', 'equalto', 'receipts') | list %}
+{% set stream_writer_presence_stream_workers = matrix_nginx_proxy_synapse_workers_list | selectattr('type', 'equalto', 'stream_writer') | selectattr('stream_writer_stream', 'equalto', 'presence') | list %}
 {% set media_repository_workers = matrix_nginx_proxy_synapse_workers_list | selectattr('type', 'equalto', 'media_repository') | list %}
 {% set user_dir_workers = matrix_nginx_proxy_synapse_workers_list | selectattr('type', 'equalto', 'user_dir') | list %}
+
+{% macro render_worker_upstream(name, workers, matrix_nginx_proxy_enabled) %}
+{% if workers | length > 0 %}
+	upstream {{ name }} {
+		{% for worker in workers %}
+			{% if matrix_nginx_proxy_enabled %}
+				server "{{ worker.name }}:{{ worker.port }}";
+			{% else %}
+				server "127.0.0.1:{{ worker.port }}";
+			{% endif %}
+		{% endfor %}
+	}
+{% endif %}
+{% endmacro %}
+
+{% macro render_locations_to_upstream(locations, upstream_name) %}
+	{% for location in locations %}
+	location ~ {{ location }} {
+		proxy_pass http://{{ upstream_name }}$request_uri;
+		proxy_set_header Host $host;
+	}
+	{% endfor %}
+{% endmacro %}
+
 {% if matrix_nginx_proxy_synapse_workers_enabled %}
 	{% if matrix_nginx_proxy_synapse_cache_enabled %}
     	proxy_cache_path  {{ matrix_nginx_proxy_synapse_cache_path }} levels=1:2 keys_zone={{ matrix_nginx_proxy_synapse_cache_keys_zone_name }}:{{ matrix_nginx_proxy_synapse_cache_keys_zone_size }} inactive={{ matrix_nginx_proxy_synapse_cache_inactive_time }} max_size={{ matrix_nginx_proxy_synapse_cache_max_size_mb }}m;
 	{% endif %}
 	# Round Robin "upstream" pools for workers
 
-	{% if generic_workers %}
-	upstream generic_worker_upstream {
+	{% if generic_workers |length > 0 %}
+	upstream generic_workers_upstream {
 		# ensures that requests from the same client will always be passed
 		# to the same server (except when this server is unavailable)
 		hash $http_x_forwarded_for;
 
 		{% for worker in generic_workers %}
 			{% if matrix_nginx_proxy_enabled %}
-				server "matrix-synapse-worker-{{ worker.type }}-{{ worker.instanceId }}:{{ worker.port }}";
+				server "{{ worker.name }}:{{ worker.port }}";
 			{% else %}
 				server "127.0.0.1:{{ worker.port }}";
 			{% endif %}
@@ -25,29 +54,15 @@
 	}
 	{% endif %}
 
-	{% if media_repository_workers %}
-	upstream media_repository_upstream {
-		{% for worker in media_repository_workers %}
-			{% if matrix_nginx_proxy_enabled %}
-				server "matrix-synapse-worker-{{ worker.type }}-{{ worker.instanceId }}:{{ worker.port }}";
-			{% else %}
-				server "127.0.0.1:{{ worker.port }}";
-			{% endif %}
-		{% endfor %}
-	}
-	{% endif %}
+	{{ render_worker_upstream('stream_writer_typing_stream_workers_upstream', stream_writer_typing_stream_workers, matrix_nginx_proxy_enabled) }}
+	{{ render_worker_upstream('stream_writer_to_device_stream_workers_upstream', stream_writer_to_device_stream_workers, matrix_nginx_proxy_enabled) }}
+	{{ render_worker_upstream('stream_writer_account_data_stream_workers_upstream', stream_writer_account_data_stream_workers, matrix_nginx_proxy_enabled) }}
+	{{ render_worker_upstream('stream_writer_receipts_stream_workers_upstream', stream_writer_receipts_stream_workers, matrix_nginx_proxy_enabled) }}
+	{{ render_worker_upstream('stream_writer_presence_stream_workers_upstream', stream_writer_presence_stream_workers, matrix_nginx_proxy_enabled) }}
 
-	{% if user_dir_workers %}
-	upstream user_dir_upstream {
-		{% for worker in user_dir_workers %}
-			{% if matrix_nginx_proxy_enabled %}
-				server "matrix-synapse-worker-{{ worker.type }}-{{ worker.instanceId }}:{{ worker.port }}";
-			{% else %}
-				server "127.0.0.1:{{ worker.port }}";
-			{% endif %}
-		{% endfor %}
-	}
-	{% endif %}
+	{{ render_worker_upstream('media_repository_workers_upstream', media_repository_workers, matrix_nginx_proxy_enabled) }}
+
+	{{ render_worker_upstream('user_dir_workers_upstream', user_dir_workers, matrix_nginx_proxy_enabled) }}
 {% endif %}
 
 server {
@@ -65,21 +80,41 @@ server {
 	{% if matrix_nginx_proxy_synapse_workers_enabled %}
 		{# Workers redirects BEGIN #}
 
-		{% if generic_workers %}
+		{% if generic_workers | length > 0 %}
 			# https://github.com/matrix-org/synapse/blob/master/docs/workers.md#synapseappgeneric_worker
-			{% for location in matrix_nginx_proxy_synapse_generic_worker_client_server_locations %}
-			location ~ {{ location }} {
-				proxy_pass http://generic_worker_upstream$request_uri;
-				proxy_set_header Host $host;
-			}
-			{% endfor %}
+			{{ render_locations_to_upstream(matrix_nginx_proxy_synapse_generic_worker_client_server_locations, 'generic_workers_upstream') }}
 		{% endif %}
 
-		{% if media_repository_workers %}
+		{% if stream_writer_typing_stream_workers | length > 0 %}
+			# https://matrix-org.github.io/synapse/latest/workers.html#the-typing-stream
+			{{ render_locations_to_upstream(matrix_nginx_proxy_synapse_stream_writer_typing_stream_worker_client_server_locations, 'stream_writer_typing_stream_workers_upstream') }}
+		{% endif %}
+
+		{% if stream_writer_to_device_stream_workers | length > 0 %}
+			# https://matrix-org.github.io/synapse/latest/workers.html#the-to_device-stream
+			{{ render_locations_to_upstream(matrix_nginx_proxy_synapse_stream_writer_to_device_stream_worker_client_server_locations, 'stream_writer_to_device_stream_workers_upstream') }}
+		{% endif %}
+
+		{% if stream_writer_account_data_stream_workers | length > 0 %}
+			# https://matrix-org.github.io/synapse/latest/workers.html#the-account_data-stream
+			{{ render_locations_to_upstream(matrix_nginx_proxy_synapse_stream_writer_account_data_stream_worker_client_server_locations, 'stream_writer_account_data_stream_workers_upstream') }}
+		{% endif %}
+
+		{% if stream_writer_receipts_stream_workers | length > 0 %}
+			# https://matrix-org.github.io/synapse/latest/workers.html#the-receipts-stream
+			{{ render_locations_to_upstream(matrix_nginx_proxy_synapse_stream_writer_receipts_stream_worker_client_server_locations, 'stream_writer_receipts_stream_workers_upstream') }}
+		{% endif %}
+
+		{% if stream_writer_presence_stream_workers | length > 0 %}
+			# https://matrix-org.github.io/synapse/latest/workers.html#the-presence-stream
+			{{ render_locations_to_upstream(matrix_nginx_proxy_synapse_stream_writer_presence_stream_worker_client_server_locations, 'stream_writer_presence_stream_workers_upstream') }}
+		{% endif %}
+
+		{% if media_repository_workers | length > 0 %}
 			# https://github.com/matrix-org/synapse/blob/master/docs/workers.md#synapseappmedia_repository
 			{% for location in matrix_nginx_proxy_synapse_media_repository_locations %}
 			location ~ {{ location }} {
-				proxy_pass http://media_repository_upstream$request_uri;
+				proxy_pass http://media_repository_workers_upstream$request_uri;
 				proxy_set_header Host $host;
 
 				client_body_buffer_size 25M;
@@ -97,15 +132,10 @@ server {
 			{% endfor %}
 		{% endif %}
 
-		{% if user_dir_workers %}
+		{% if user_dir_workers | length > 0 %}
 			# FIXME: obsolete if matrix_nginx_proxy_proxy_matrix_user_directory_search_enabled is set
 			# https://github.com/matrix-org/synapse/blob/master/docs/workers.md#synapseappuser_dir
-			{% for location in matrix_nginx_proxy_synapse_user_dir_locations %}
-			location ~ {{ location }} {
-				proxy_pass http://user_dir_upstream$request_uri;
-				proxy_set_header Host $host;
-			}
-			{% endfor %}
+			{{ render_locations_to_upstream(matrix_nginx_proxy_synapse_user_dir_locations, 'user_dir_workers_upstream') }}
 		{% endif %}
 		{# Workers redirects END #}
 	{% endif %}
@@ -150,20 +180,15 @@ server {
 	gzip_types text/plain application/json;
 
 	{% if matrix_nginx_proxy_synapse_workers_enabled %}
-		{% if generic_workers %}
+		{% if generic_workers | length > 0 %}
 			# https://github.com/matrix-org/synapse/blob/master/docs/workers.md#synapseappgeneric_worker
-			{% for location in matrix_nginx_proxy_synapse_generic_worker_federation_locations %}
-			location ~ {{ location }} {
-				proxy_pass http://generic_worker_upstream$request_uri;
-				proxy_set_header Host $host;
-			}
-			{% endfor %}
+			{{ render_locations_to_upstream(matrix_nginx_proxy_synapse_generic_worker_federation_locations, 'generic_workers_upstream') }}
 		{% endif %}
-		{% if media_repository_workers %}
+		{% if media_repository_workers | length > 0 %}
 			# https://github.com/matrix-org/synapse/blob/master/docs/workers.md#synapseappmedia_repository
 			{% for location in matrix_nginx_proxy_synapse_media_repository_locations %}
 			location ~ {{ location }} {
-				proxy_pass http://media_repository_upstream$request_uri;
+				proxy_pass http://media_repository_workers_upstream$request_uri;
 				proxy_set_header Host $host;
 
 				client_body_buffer_size 25M;
diff --git a/roles/matrix-prometheus/templates/prometheus.yml.j2 b/roles/matrix-prometheus/templates/prometheus.yml.j2
index f3262f485..628f4f4f9 100644
--- a/roles/matrix-prometheus/templates/prometheus.yml.j2
+++ b/roles/matrix-prometheus/templates/prometheus.yml.j2
@@ -37,11 +37,12 @@ scrape_configs:
         index: 0
   {% for worker in matrix_prometheus_scraper_synapse_workers_enabled_list %}
   {% if worker.metrics_port != 0 %}
-    - targets: ['matrix-synapse-worker-{{ worker.type }}-{{ worker.instanceId }}:{{ worker.metrics_port }}']
+    - targets: ['{{ worker.name }}:{{ worker.metrics_port }}']
       labels:
         instance: {{ matrix_domain }}
+		worker_id: {{ worker.id }}
         job: {{ worker.type }}
-        index: {{ worker.instanceId }}
+		app: {{ worker.app }}
   {% endif %}
   {% endfor %}
   {% endif %}
diff --git a/roles/matrix-synapse/defaults/main.yml b/roles/matrix-synapse/defaults/main.yml
index a7dab5b67..4213c4f18 100644
--- a/roles/matrix-synapse/defaults/main.yml
+++ b/roles/matrix-synapse/defaults/main.yml
@@ -398,6 +398,12 @@ matrix_synapse_workers_presets:
     federation_sender_workers_count: 1
     media_repository_workers_count: 0
     user_dir_workers_count: 0
+    stream_writer_events_stream_workers_count: 0
+    stream_writer_typing_stream_workers_count: 0
+    stream_writer_to_device_stream_workers_count: 0
+    stream_writer_account_data_stream_workers_count: 0
+    stream_writer_receipts_stream_workers_count: 0
+    stream_writer_presence_stream_workers_count: 0
   one-of-each:
     generic_workers_count: 1
     pusher_workers_count: 1
@@ -409,6 +415,12 @@ matrix_synapse_workers_presets:
     # user_dir workers are deprecated since Synapse v1.59. This will be removed.
     # See: https://github.com/matrix-org/synapse/blob/v1.59.0/docs/upgrade.md#deprecation-of-the-synapseappappservice-and-synapseappuser_dir-worker-application-types
     user_dir_workers_count: 0
+    stream_writer_events_stream_workers_count: 1
+    stream_writer_typing_stream_workers_count: 1
+    stream_writer_to_device_stream_workers_count: 1
+    stream_writer_account_data_stream_workers_count: 1
+    stream_writer_receipts_stream_workers_count: 1
+    stream_writer_presence_stream_workers_count: 1
 
 # Controls whether the matrix-synapse container exposes the various worker ports
 # (see `port` and `metrics_port` in `matrix_synapse_workers_enabled_list`) outside of the container.
@@ -421,6 +433,71 @@ matrix_synapse_workers_generic_workers_count: "{{ matrix_synapse_workers_presets
 matrix_synapse_workers_generic_workers_port_range_start: 18111
 matrix_synapse_workers_generic_workers_metrics_range_start: 19111
 
+# matrix_synapse_workers_stream_writer_events_stream_workers_count controls how many stream writers that handle the `events` stream to spawn.
+# More than 1 worker is also supported of this type.
+matrix_synapse_workers_stream_writer_events_stream_workers_count: "{{ matrix_synapse_workers_presets[matrix_synapse_workers_preset]['stream_writer_events_stream_workers_count'] }}"
+
+# matrix_synapse_workers_stream_writer_typing_stream_workers_count controls how many stream writers that handle the `typing` stream to spawn.
+# The count of these workers can only be 0 or 1.
+matrix_synapse_workers_stream_writer_typing_stream_workers_count: "{{ matrix_synapse_workers_presets[matrix_synapse_workers_preset]['stream_writer_typing_stream_workers_count'] }}"
+
+# matrix_synapse_workers_stream_writer_to_device_stream_workers_count controls how many stream writers that handle the `to_device` stream to spawn.
+# The count of these workers can only be 0 or 1.
+matrix_synapse_workers_stream_writer_to_device_stream_workers_count: "{{ matrix_synapse_workers_presets[matrix_synapse_workers_preset]['stream_writer_to_device_stream_workers_count'] }}"
+
+# matrix_synapse_workers_stream_writer_account_data_stream_workers_count controls how many stream writers that handle the `account_data` stream to spawn.
+# The count of these workers can only be 0 or 1.
+matrix_synapse_workers_stream_writer_account_data_stream_workers_count: "{{ matrix_synapse_workers_presets[matrix_synapse_workers_preset]['stream_writer_account_data_stream_workers_count'] }}"
+
+# matrix_synapse_workers_stream_writer_receipts_stream_workers_count controls how many stream writers that handle the `receipts` stream to spawn.
+# The count of these workers can only be 0 or 1.
+matrix_synapse_workers_stream_writer_receipts_stream_workers_count: "{{ matrix_synapse_workers_presets[matrix_synapse_workers_preset]['stream_writer_receipts_stream_workers_count'] }}"
+
+# matrix_synapse_workers_stream_writer_presence_stream_workers_count controls how many stream writers that handle the `presence` stream to spawn.
+# The count of these workers can only be 0 or 1.
+matrix_synapse_workers_stream_writer_presence_stream_workers_count: "{{ matrix_synapse_workers_presets[matrix_synapse_workers_preset]['stream_writer_presence_stream_workers_count'] }}"
+
+# A list of stream writer workers to enable. This list is built automatically based on other variables.
+# You're encouraged to enable/disable stream writer workers by setting `matrix_synapse_workers_stream_writer_*_stream_workers_count` variables, instead of adjusting this list manually.
+matrix_synapse_workers_stream_writers: |
+  {{
+    []
+    +
+    ([{'stream': 'events'}] * matrix_synapse_workers_stream_writer_events_stream_workers_count | int)
+    +
+    ([{'stream': 'typing'}] * matrix_synapse_workers_stream_writer_typing_stream_workers_count | int)
+    +
+    ([{'stream': 'to_device'}] * matrix_synapse_workers_stream_writer_to_device_stream_workers_count | int)
+    +
+    ([{'stream': 'account_data'}] * matrix_synapse_workers_stream_writer_account_data_stream_workers_count | int)
+    +
+    ([{'stream': 'receipts'}] * matrix_synapse_workers_stream_writer_receipts_stream_workers_count | int)
+    +
+    ([{'stream': 'presence'}] * matrix_synapse_workers_stream_writer_presence_stream_workers_count | int)
+  }}
+
+# matrix_synapse_stream_writers populates the `stream_writers` Synapse configuration used when Synapse workers are in use (`matrix_synapse_workers_enabled`).
+# What you see below is an initial default value which will be adjusted at runtime based on the value of `matrix_synapse_workers_stream_writers`.
+# Adjusting this value manually is generally not necessary.
+#
+# It's tempting to initialize this like this:
+# matrix_synapse_stream_writers:
+#   - typing: []
+#   - events: []
+#   - to_device: []
+#   - account_data: []
+#   - receipts: []
+#   - presence: []
+# .. but Synapse does not like empty lists (see https://github.com/matrix-org/synapse/issues/13804)
+matrix_synapse_stream_writers: {}
+
+# `matrix_synapse_workers_stream_writer_workers_` variables control the port numbers of various stream writer workers
+# defined in `matrix_synapse_workers_stream_writers`.
+# It should be noted that not all of the background worker types will need to expose HTTP services, etc.
+matrix_synapse_workers_stream_writer_workers_http_port_range_start: 20011
+matrix_synapse_workers_stream_writer_workers_replication_port_range_start: 25011
+matrix_synapse_workers_stream_writer_workers_metrics_range_start: 19111
+
 # matrix_synapse_workers_pusher_workers_count can only be 0 or 1 for now.
 # More instances are not supported due to a playbook limitation having to do with keeping `pusher_instances` in `homeserver.yaml` updated.
 # See https://github.com/matrix-org/synapse/commit/ddfdf945064925eba761ae3748e38f3a1c73c328
@@ -463,25 +540,36 @@ matrix_synapse_workers_user_dir_workers_metrics_range_start: 19661
 # as certain workers can only be spawned just once.
 #
 # Each worker instance in the list defines the following fields:
-# - `type` - the type of worker (`generic_worker`, etc.)
-# - `instanceId` - a string that identifies the worker. The combination of (`type` + `instanceId`) represents the name of the worker and must be unique.
+# - `id` - a string that uniquely identifies the worker
+# - `name` - a string that will be used as the container and systemd service name
+# - `type` - the type of worker (`generic_worker`, `stream_writer`, `pusher`, etc.)
+# - `app` - the Synapse app (https://matrix-org.github.io/synapse/latest/workers.html#available-worker-applications) that powers this worker (`generic_worker`, `federation_sender`, etc.).
+#           The `app` usually matches the `type`, but not always. For example, `type = stream_writer` workers are served by the `generic_worker` type.
 # - `port` - an HTTP port where the worker listens for requests (can be `0` for workers that don't do HTTP request processing)
 # - `metrics_port` - an HTTP port where the worker exports Prometheus metrics
+# - `replication_port` - an HTTP port where the worker serves `replication` endpoints (used by stream writers, etc.)
+# - `webserving` - tells whether this type of worker serves web (client or federation) requests, so that it can be injected as a dependency to the reverse-proxy
 #
 # Example of what this needs to look like, if you're defining it manually:
 # matrix_synapse_workers_enabled_list:
-#   - { type: generic_worker, instanceId: '18111', port: 18111, metrics_port: 19111 }
-#   - { type: generic_worker, instanceId: '18112', port: 18112, metrics_port: 19112 }
-#   - { type: generic_worker, instanceId: '18113', port: 18113, metrics_port: 19113 }
-#   - { type: generic_worker, instanceId: '18114', port: 18114, metrics_port: 19114 }
-#   - { type: generic_worker, instanceId: '18115', port: 18115, metrics_port: 19115 }
-#   - { type: generic_worker, instanceId: '18116', port: 18116, metrics_port: 19116 }
-#   - { type: pusher, instanceId: '0', port: 0, metrics_port: 19200 }
-#   - { type: appservice, instanceId: '0', port: 0, metrics_port: 19300 }
-#   - { type: federation_sender, instanceId: '0', port: 0, metrics_port: 19400 }
-#   - { type: media_repository, instanceId: '18551', port: 18551, metrics_port: 19551 }
+#   - { 'id': 'generic-worker-0', 'name': 'matrix-synapse-worker-generic-0', 'type': 'generic_worker', 'app': 'generic_worker', 'port': 18111, 'metrics_port': 19111, 'webserving': true }
+#   - { 'id': 'generic-worker-1', 'name': 'matrix-synapse-worker-generic-1', 'type': 'generic_worker', 'app': 'generic_worker', 'port': 18112, 'metrics_port': 19112, 'webserving': true }
+#   - { 'id': 'generic-worker-2', 'name': 'matrix-synapse-worker-generic-2', 'type': 'generic_worker', 'app': 'generic_worker', 'port': 18113, 'metrics_port': 19113, 'webserving': true }
+#   - { 'id': 'generic-worker-3', 'name': 'matrix-synapse-worker-generic-3', 'type': 'generic_worker', 'app': 'generic_worker', 'port': 18114, 'metrics_port': 19114, 'webserving': true }
+#   - { 'id': 'generic-worker-4', 'name': 'matrix-synapse-worker-generic-4', 'type': 'generic_worker', 'app': 'generic_worker', 'port': 18115, 'metrics_port': 19115, 'webserving': true }
+#   - { 'id': 'generic-worker-5', 'name': 'matrix-synapse-worker-generic-5', 'type': 'generic_worker', 'app': 'generic_worker', 'port': 18116, 'metrics_port': 19116, 'webserving': true }
+#   - { 'id': 'stream-writer-0-events', 'name': 'matrix-synapse-worker-stream-writer-0-events', 'type': 'stream_writer', 'app': 'generic_worker', 'stream_writer_stream': 'events', 'port': 0, 'replication_port': 25011, metrics_port: 19111, 'webserving': false }
+#   - { 'id': 'stream-writer-1-typing', 'name': 'matrix-synapse-worker-stream-writer-1-typing', 'type': 'stream_writer', 'app': 'generic_worker', 'stream_writer_stream': 'typing', 'port': 20012, 'replication_port': 25012, metrics_port: 19112, 'webserving': true }
+#   - { 'id': 'pusher-0', 'name': 'matrix-synapse-worker-pusher-0', 'type': 'pusher', 'app': 'pusher', 'port': 0, 'metrics_port': 19200, 'webserving': false }
+#   - { 'id': 'appservice-0', 'name': 'matrix-synapse-worker-appservice-0', 'type': 'appservice', 'port': 0, 'metrics_port': 19300, 'webserving': false }
+#   - { 'id': 'federation-sender-0', 'name': 'matrix-synapse-worker-federation-sender-0', 'type': 'federation_sender', 'port': 0, 'metrics_port': 19400, 'webserving': false }
+#   - { 'id': 'media-repository-0', 'name': 'matrix-synapse-worker-media-repository-0', 'type': 'media_repository', 'port': 18551, 'metrics_port': 19551, 'webserving': true }
 matrix_synapse_workers_enabled_list: []
 
+# matrix_synapse_instance_map holds the instance map used for mapping worker names (for certain generic workers only!) to where they live (host, port which handles replication traffic).
+# This is populated automatically based on `matrix_synapse_workers_enabled_list` during runtime, so you're not required to tweak it manually.
+matrix_synapse_instance_map: {}
+
 # Redis information
 matrix_synapse_redis_enabled: false
 matrix_synapse_redis_host: ""
diff --git a/roles/matrix-synapse/tasks/init.yml b/roles/matrix-synapse/tasks/init.yml
index 77696bced..8610a33b7 100644
--- a/roles/matrix-synapse/tasks/init.yml
+++ b/roles/matrix-synapse/tasks/init.yml
@@ -8,15 +8,15 @@
 
 # Unless `matrix_synapse_workers_enabled_list` is explicitly defined,
 # we'll generate it dynamically.
-- ansible.builtin.import_tasks: "{{ role_path }}/tasks/synapse/workers/init.yml"
+- ansible.builtin.include_tasks: "{{ role_path }}/tasks/synapse/workers/init.yml"
   when: "matrix_synapse_enabled and matrix_synapse_workers_enabled and matrix_synapse_workers_enabled_list|length == 0"
 
 - ansible.builtin.set_fact:
     matrix_systemd_services_list: "{{ matrix_systemd_services_list + ['matrix-synapse.service'] }}"
   when: matrix_synapse_enabled | bool
 
-- name: Ensure systemd services for workers are injected
-  ansible.builtin.include_tasks: "{{ role_path }}/tasks/synapse/workers/util/inject_systemd_services_for_worker.yml"
+- name: Ensure workers are injected into various places
+  ansible.builtin.include_tasks: "{{ role_path }}/tasks/synapse/workers/util/inject_worker.yml"
   with_items: "{{ matrix_synapse_workers_enabled_list }}"
   loop_control:
     loop_var: matrix_synapse_worker_details
@@ -65,9 +65,9 @@
         matrix_synapse_worker_nginx_metrics_configuration_block: |
           {% for worker in matrix_synapse_workers_enabled_list %}
           {% if worker.metrics_port != 0 %}
-          location /metrics/synapse/worker/{{ worker.type }}-{{ worker.instanceId }} {
+          location /metrics/synapse/worker/{{ worker.id }} {
             resolver 127.0.0.11 valid=5s;
-            set $backend "matrix-synapse-worker-{{ worker.type }}-{{ worker.instanceId }}:{{ worker.metrics_port }}";
+            set $backend "{{ worker.name }}:{{ worker.metrics_port }}";
             proxy_pass http://$backend/_synapse/metrics;
             proxy_set_header Host $host;
           }
diff --git a/roles/matrix-synapse/tasks/synapse/workers/init.yml b/roles/matrix-synapse/tasks/synapse/workers/init.yml
index 3aa619238..fe613e70d 100644
--- a/roles/matrix-synapse/tasks/synapse/workers/init.yml
+++ b/roles/matrix-synapse/tasks/synapse/workers/init.yml
@@ -7,18 +7,47 @@
 - name: Build generic workers
   ansible.builtin.set_fact:
     worker:
+      id: "generic-worker-{{ item }}"
+      name: "matrix-synapse-worker-generic-{{ item }}"
       type: 'generic_worker'
-      instanceId: "{{ matrix_synapse_workers_generic_workers_port_range_start + item }}"
+      app: 'generic_worker'
+      webserving: true
       port: "{{ matrix_synapse_workers_generic_workers_port_range_start + item }}"
       metrics_port: "{{ matrix_synapse_workers_generic_workers_metrics_range_start + item }}"
   register: "matrix_synapse_workers_list_results_generic_workers"
   loop: "{{ range(0, matrix_synapse_workers_generic_workers_count | int) | list }}"
 
+- name: Build stream writer workers
+  ansible.builtin.set_fact:
+    worker:
+      id: "stream-writer-{{ item }}-{{ item.stream }}"
+      # Names must not include understores. Certain stream writer streams (to_device, account_data, ..) do, so we fix them up.
+      name: "matrix-synapse-worker-stream-writer-{{ idx }}-{{ item.stream | replace('_', '-') }}"
+      type: 'stream_writer'
+      app: "generic_worker"
+      webserving: "{{ item.stream in matrix_synapse_workers_webserving_stream_writer_types }}"
+      stream_writer_stream: "{{ item.stream }}"
+      port: "{{ matrix_synapse_workers_stream_writer_workers_http_port_range_start + idx }}"
+      replication_port: "{{ matrix_synapse_workers_stream_writer_workers_replication_port_range_start + idx }}"
+      metrics_port: "{{ matrix_synapse_workers_stream_writer_workers_metrics_range_start + idx }}"
+  register: "matrix_synapse_workers_list_results_stream_writer_workers"
+  loop: "{{ matrix_synapse_workers_stream_writers }}"
+  loop_control:
+    index_var: idx
+
+- name: Populate matrix_synapse_stream_writers from enabled stream writer workers list
+  ansible.builtin.set_fact:
+    matrix_synapse_stream_writers: "{{ matrix_synapse_stream_writers | combine ({item.ansible_facts.worker.stream_writer_stream: [item.ansible_facts.worker.name]}) }}"
+  with_items: "{{ matrix_synapse_workers_list_results_stream_writer_workers.results }}"
+
 - name: Build federation sender workers
   ansible.builtin.set_fact:
     worker:
+      id: "federation-sender-{{ item }}"
+      name: "matrix-synapse-worker-federation-sender-{{ item }}"
       type: 'federation_sender'
-      instanceId: "{{ item }}"
+      app: 'federation_sender'
+      webserving: false
       port: 0
       metrics_port: "{{ matrix_synapse_workers_federation_sender_workers_metrics_range_start + item }}"
   register: "matrix_synapse_workers_list_results_federation_sender_workers"
@@ -28,8 +57,11 @@
 - name: Build pusher workers
   ansible.builtin.set_fact:
     worker:
+      id: "pusher-{{ item }}"
+      name: "matrix-synapse-worker-pusher-{{ item }}"
       type: 'pusher'
-      instanceId: "{{ item }}"
+      app: 'pusher'
+      webserving: false
       port: 0
       metrics_port: "{{ matrix_synapse_workers_pusher_workers_metrics_range_start + item }}"
   register: "matrix_synapse_workers_list_results_pusher_workers"
@@ -39,8 +71,11 @@
 - name: Build appservice workers
   ansible.builtin.set_fact:
     worker:
+      id: "appservice-{{ item }}"
+      name: "matrix-synapse-worker-appservice-{{ item }}"
       type: 'appservice'
-      instanceId: "{{ item }}"
+      app: 'appservice'
+      webserving: false
       port: 0
       metrics_port: "{{ matrix_synapse_workers_appservice_workers_metrics_range_start + item }}"
   register: "matrix_synapse_workers_list_results_appservice_workers"
@@ -49,8 +84,11 @@
 - name: Build media_repository workers
   ansible.builtin.set_fact:
     worker:
+      id: "media-repository-{{ item }}"
+      name: "matrix-synapse-worker-media-repository-{{ item }}"
       type: 'media_repository'
-      instanceId: "{{ matrix_synapse_workers_media_repository_workers_port_range_start + item }}"
+      app: 'media_repository'
+      webserving: true
       port: "{{ matrix_synapse_workers_media_repository_workers_port_range_start + item }}"
       metrics_port: "{{ matrix_synapse_workers_media_repository_workers_metrics_range_start + item }}"
   register: "matrix_synapse_workers_list_results_media_repository_workers"
@@ -62,6 +100,8 @@
     {{
       matrix_synapse_workers_list_results_generic_workers.results
       +
+      matrix_synapse_workers_list_results_stream_writer_workers.results
+      +
       matrix_synapse_workers_list_results_federation_sender_workers.results
       +
       matrix_synapse_workers_list_results_pusher_workers.results
diff --git a/roles/matrix-synapse/tasks/synapse/workers/util/inject_systemd_services_for_worker.yml b/roles/matrix-synapse/tasks/synapse/workers/util/inject_systemd_services_for_worker.yml
deleted file mode 100644
index 2ecb3f2bf..000000000
--- a/roles/matrix-synapse/tasks/synapse/workers/util/inject_systemd_services_for_worker.yml
+++ /dev/null
@@ -1,19 +0,0 @@
----
-# The tasks below run before `validate_config.yml`.
-# To avoid failing with a cryptic error message, we'll do validation here.
-#
-# This check is mostly relevant to people who explicitly define `matrix_synapse_workers_enabled_list`
-# (Synapse Workers users from the earlier days of this PR - https://github.com/spantaleev/matrix-docker-ansible-deploy/pull/456).
-#
-# In the future, it should be possible to remove this check.
-# Our own code which dynamically builds `matrix_synapse_workers_enabled_list` does things right.
-- name: Fail if instanceId not defined for worker
-  ansible.builtin.fail:
-    msg: "Synapse workers (like {{ matrix_synapse_worker_details | to_json }}) need to define an instanceId property (type + instanceId must be unique)"
-  when: "'instanceId' not in matrix_synapse_worker_details"
-
-- ansible.builtin.set_fact:
-    matrix_synapse_worker_systemd_service_name: "matrix-synapse-worker-{{ matrix_synapse_worker_details.type }}-{{ matrix_synapse_worker_details.instanceId }}.service"
-
-- ansible.builtin.set_fact:
-    matrix_systemd_services_list: "{{ matrix_systemd_services_list + [matrix_synapse_worker_systemd_service_name] }}"
diff --git a/roles/matrix-synapse/tasks/synapse/workers/util/inject_worker.yml b/roles/matrix-synapse/tasks/synapse/workers/util/inject_worker.yml
new file mode 100644
index 000000000..4f6b43986
--- /dev/null
+++ b/roles/matrix-synapse/tasks/synapse/workers/util/inject_worker.yml
@@ -0,0 +1,65 @@
+---
+# The tasks below run before `validate_config.yml`.
+# To avoid failing with a cryptic error message, we'll do validation here.
+#
+# This check is mostly relevant to people who explicitly define `matrix_synapse_workers_enabled_list`
+# (Synapse Workers users from the earlier days of this PR - https://github.com/spantaleev/matrix-docker-ansible-deploy/pull/456).
+#
+# In the future, it should be possible to remove this check.
+# Our own code which dynamically builds `matrix_synapse_workers_enabled_list` does things right.
+- name: Fail if required property not defined for worker
+  ansible.builtin.fail:
+    msg: "Synapse workers (like {{ matrix_synapse_worker_details | to_json }}) need to define a `{{ item }}` property"
+  with_items:
+    - id
+    - name
+    - type
+    - app
+    - port
+    - webserving
+  when: "item not in matrix_synapse_worker_details"
+
+# Names are used for container names and systemd services.
+# Routing happens based on container names, so Synapse processes that try to route to workers with underscores in the name will complain. Example:
+# > InvalidCodepoint Codepoint U+005F at position 46 of 'matrix-synapse-worker-stream-writer-3-account_data' not allowed
+- name: Fail if worker name includes underscore
+  ansible.builtin.fail:
+    msg: "Unrecognized Synapse worker `name`: `{{ matrix_synapse_worker_details.name }}`. It must not include underscores"
+  when: "'_' in matrix_synapse_worker_details.name"
+
+- name: Fail if worker type unknown
+  ansible.builtin.fail:
+    msg: "Unrecognized Synapse worker `type`: `{{ matrix_synapse_worker_details.type }}`. Supported types are: {{ matrix_synapse_known_worker_types | join(', ') }}"
+  when: "matrix_synapse_worker_details.type not in matrix_synapse_known_worker_types"
+
+- name: Fail if worker app unknown
+  ansible.builtin.fail:
+    msg: "Unrecognized Synapse worker `app`: `{{ matrix_synapse_worker_details.app }}`. Supported types are: {{ matrix_synapse_workers_avail_list | join(', ') }}"
+  when: "matrix_synapse_worker_details.app not in matrix_synapse_workers_avail_list"
+
+- block:
+    - name: Fail if stream_writer_stream not defined for stream_writer worker
+      ansible.builtin.fail:
+        msg: >-
+          Synapse stream_writer workers (such as {{ item }}) need to define a valid `stream_writer_stream` property
+          (not `{{ matrix_synapse_worker_details.stream_writer_stream|default('undefined') }}`).
+          Supported types are: {{ matrix_synapse_workers_known_stream_writer_stream_types | join(', ') }}
+      when: "'stream_writer_stream' not in matrix_synapse_worker_details or matrix_synapse_worker_details.stream_writer_stream not in matrix_synapse_workers_known_stream_writer_stream_types"
+
+    - name: Fail if replication_port not defined for stream_writer worker
+      ansible.builtin.fail:
+        msg: "Synapse background workers of type stream_writer (such as {{ item }}) need to define a valid `replication_port` property"
+      when: "'replication_port' not in matrix_synapse_worker_details"
+  when: "matrix_synapse_worker_details.type == 'stream_writer'"
+
+- ansible.builtin.set_fact:
+    matrix_systemd_services_list: "{{ matrix_systemd_services_list + [matrix_synapse_worker_details.name + '.service'] }}"
+
+- ansible.builtin.set_fact:
+    matrix_synapse_webserving_workers_systemd_services_list: "{{ matrix_synapse_webserving_workers_systemd_services_list + [matrix_synapse_worker_details.name + '.service'] }}"
+  when: matrix_synapse_worker_details.webserving | bool
+
+# Inject stream writers and various other background workers into the instance map.
+- ansible.builtin.set_fact:
+    matrix_synapse_instance_map: "{{ matrix_synapse_instance_map | combine({matrix_synapse_worker_details.name: {'host': matrix_synapse_worker_details.name, 'port': matrix_synapse_worker_details.replication_port}}) }}"
+  when: matrix_synapse_worker_details.type in matrix_synapse_known_instance_map_eligible_worker_types
diff --git a/roles/matrix-synapse/tasks/synapse/workers/util/setup_files_for_worker.yml b/roles/matrix-synapse/tasks/synapse/workers/util/setup_files_for_worker.yml
index d6d4924fb..d3f309175 100644
--- a/roles/matrix-synapse/tasks/synapse/workers/util/setup_files_for_worker.yml
+++ b/roles/matrix-synapse/tasks/synapse/workers/util/setup_files_for_worker.yml
@@ -1,13 +1,9 @@
 ---
 
 - ansible.builtin.set_fact:
-    matrix_synapse_worker_systemd_service_name: "matrix-synapse-worker-{{ matrix_synapse_worker_details.type }}-{{ matrix_synapse_worker_details.instanceId }}"
-
-- ansible.builtin.set_fact:
-    matrix_synapse_worker_container_name: "{{ matrix_synapse_worker_systemd_service_name }}"
-
-- ansible.builtin.set_fact:
-    matrix_synapse_worker_config_file_name: "worker.{{ matrix_synapse_worker_details.type }}_{{ matrix_synapse_worker_details.instanceId }}.yaml"
+    matrix_synapse_worker_systemd_service_name: "{{ matrix_synapse_worker_details.name }}"
+    matrix_synapse_worker_container_name: "{{ matrix_synapse_worker_details.name }}"
+    matrix_synapse_worker_config_file_name: "worker.{{ matrix_synapse_worker_details.name }}.yaml"
 
 - name: Ensure configuration exists for {{ matrix_synapse_worker_systemd_service_name }}
   ansible.builtin.template:
diff --git a/roles/matrix-synapse/tasks/validate_config.yml b/roles/matrix-synapse/tasks/validate_config.yml
index bcb71c756..1fc708166 100644
--- a/roles/matrix-synapse/tasks/validate_config.yml
+++ b/roles/matrix-synapse/tasks/validate_config.yml
@@ -32,6 +32,11 @@
   with_items:
     - "matrix_synapse_workers_pusher_workers_count"
     - "matrix_synapse_workers_federation_sender_workers_count"
+    - "matrix_synapse_workers_stream_writer_typing_stream_workers_count"
+    - "matrix_synapse_workers_stream_writer_to_device_stream_workers_count"
+    - "matrix_synapse_workers_stream_writer_account_data_stream_workers_count"
+    - "matrix_synapse_workers_stream_writer_receipts_stream_workers_count"
+    - "matrix_synapse_workers_stream_writer_presence_stream_workers_count"
 
 - name: (Deprecation) Catch and report renamed settings
   ansible.builtin.fail:
diff --git a/roles/matrix-synapse/templates/synapse/homeserver.yaml.j2 b/roles/matrix-synapse/templates/synapse/homeserver.yaml.j2
index 04c4b3cf5..86d03be41 100644
--- a/roles/matrix-synapse/templates/synapse/homeserver.yaml.j2
+++ b/roles/matrix-synapse/templates/synapse/homeserver.yaml.j2
@@ -352,13 +352,13 @@ worker_app: synapse.app.homeserver
 
 # thx https://oznetnerd.com/2017/04/18/jinja2-selectattr-filter/
 # reduce the main worker's offerings to core homeserver business
-{% if matrix_synapse_workers_enabled_list | selectattr('type', 'equalto', 'federation_sender') | list  %}
+{% if matrix_synapse_workers_enabled_list | selectattr('type', 'equalto', 'federation_sender') | list | length > 0  %}
 send_federation: false
 {% endif %}
-{% if matrix_synapse_workers_enabled_list | selectattr('type', 'equalto', 'media_repository') | list  %}
+{% if matrix_synapse_workers_enabled_list | selectattr('type', 'equalto', 'media_repository') | list | length > 0  %}
 enable_media_repo: false
 {% endif %}
-{% if matrix_synapse_workers_enabled_list | selectattr('type', 'equalto', 'pusher') | list  %}
+{% if matrix_synapse_workers_enabled_list | selectattr('type', 'equalto', 'pusher') | list | length > 0  %}
 start_pushers: false
 {% endif %}
 
@@ -2870,6 +2870,7 @@ opentracing:
 #  worker1:
 #    host: localhost
 #    port: 8034
+instance_map: {{ matrix_synapse_instance_map | to_json }}
 
 # Experimental: When using workers you can define which workers should
 # handle event persistence and typing notifications. Any worker
@@ -2878,6 +2879,7 @@ opentracing:
 #stream_writers:
 #  events: worker1
 #  typing: worker1
+stream_writers: {{ matrix_synapse_stream_writers | to_json }}
 
 # The worker that is used to run background tasks (e.g. cleaning up expired
 # data). If not provided this defaults to the main process.
diff --git a/roles/matrix-synapse/templates/synapse/prometheus/external_prometheus.yml.example.j2 b/roles/matrix-synapse/templates/synapse/prometheus/external_prometheus.yml.example.j2
index b194c3c2d..1501697d7 100644
--- a/roles/matrix-synapse/templates/synapse/prometheus/external_prometheus.yml.example.j2
+++ b/roles/matrix-synapse/templates/synapse/prometheus/external_prometheus.yml.example.j2
@@ -24,8 +24,8 @@ scrape_configs:
           job: "master"
           index: "0"
 {% for worker in matrix_synapse_workers_enabled_list %}
-  - job_name: 'synapse-{{ worker.type }}-{{ worker.instanceId }}'
-    metrics_path: /metrics/synapse/worker/{{ worker.type }}-{{ worker.instanceId }}
+  - job_name: '{{ worker.name }}'
+    metrics_path: /metrics/synapse/worker/{{ worker.id }}
     scheme: {{ 'https' if matrix_nginx_proxy_https_enabled|default(true) else 'http' }}
 {% if matrix_nginx_proxy_proxy_matrix_metrics_basic_auth_enabled|default(true) %}
     basic_auth:
@@ -35,6 +35,7 @@ scrape_configs:
     static_configs:
       - targets: ['{{ matrix_server_fqn_matrix }}:{{ matrix_nginx_proxy_container_https_host_bind_port|default(443) if matrix_nginx_proxy_https_enabled|default(true) else matrix_nginx_proxy_container_http_host_bind_port|default(80) }}']
         labels:
+          worker_id: {{ worker.id }}
           job: "{{ worker.type }}"
-          index: "{{ worker.instanceId }}"
+          app: {{ worker.app }}
 {% endfor %}
diff --git a/roles/matrix-synapse/templates/synapse/systemd/matrix-synapse-worker.service.j2 b/roles/matrix-synapse/templates/synapse/systemd/matrix-synapse-worker.service.j2
index 4a38251d7..96b65a0ae 100644
--- a/roles/matrix-synapse/templates/synapse/systemd/matrix-synapse-worker.service.j2
+++ b/roles/matrix-synapse/templates/synapse/systemd/matrix-synapse-worker.service.j2
@@ -43,7 +43,7 @@ ExecStart={{ matrix_host_command_docker }} run --rm --name {{ matrix_synapse_wor
 			{{ arg }} \
 			{% endfor %}
 			{{ matrix_synapse_docker_image }} \
-			run -m synapse.app.{{ matrix_synapse_worker_details.type }} -c /data/homeserver.yaml -c /data/{{ matrix_synapse_worker_config_file_name }}
+			run -m synapse.app.{{ matrix_synapse_worker_details.app }} -c /data/homeserver.yaml -c /data/{{ matrix_synapse_worker_config_file_name }}
 
 
 ExecStop=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} kill {{ matrix_synapse_worker_container_name }} 2>/dev/null || true'
diff --git a/roles/matrix-synapse/templates/synapse/systemd/matrix-synapse.service.j2 b/roles/matrix-synapse/templates/synapse/systemd/matrix-synapse.service.j2
index 027114fb3..eed50ad1e 100644
--- a/roles/matrix-synapse/templates/synapse/systemd/matrix-synapse.service.j2
+++ b/roles/matrix-synapse/templates/synapse/systemd/matrix-synapse.service.j2
@@ -12,7 +12,7 @@ Wants={{ service }}
 
 {% if matrix_synapse_workers_enabled %}
 {% for matrix_synapse_worker_details in matrix_synapse_workers_enabled_list %}
-Wants=matrix-synapse-worker-{{ matrix_synapse_worker_details.type }}-{{ matrix_synapse_worker_details.port }}.service
+Wants={{ matrix_synapse_worker_details.name }}.service
 {% endfor %}
 {% endif %}
 
diff --git a/roles/matrix-synapse/templates/synapse/worker.yaml.j2 b/roles/matrix-synapse/templates/synapse/worker.yaml.j2
index 33789b0ca..c9637a839 100644
--- a/roles/matrix-synapse/templates/synapse/worker.yaml.j2
+++ b/roles/matrix-synapse/templates/synapse/worker.yaml.j2
@@ -1,27 +1,38 @@
 #jinja2: lstrip_blocks: "True"
-worker_app: synapse.app.{{ matrix_synapse_worker_details.type }}
-worker_name: {{ matrix_synapse_worker_details.type ~ ':' ~ matrix_synapse_worker_details.port }}
+worker_app: synapse.app.{{ matrix_synapse_worker_details.app }}
+worker_name: {{ matrix_synapse_worker_details.name }}
 
 {% if matrix_synapse_replication_listener_enabled %}
 worker_replication_host: matrix-synapse
 worker_replication_http_port: {{ matrix_synapse_replication_http_port }}
 {% endif %}
 
-{% set has_listeners = (matrix_synapse_worker_details.type not in [ 'appservice', 'federation_sender', 'pusher' ] or matrix_synapse_metrics_enabled) %}
-
 {% set http_resources = [] %}
 
-{% if matrix_synapse_worker_details.type in ['generic_worker', 'user_dir'] %}
+{% if matrix_synapse_worker_details.type == 'user_dir' %}
   {% set http_resources = http_resources + ['client'] %}
 {% endif %}
-{% if matrix_synapse_worker_details.type in ['generic_worker'] %}
-  {% set http_resources = http_resources+ ['federation'] %}
+{% if matrix_synapse_worker_details.type == 'generic_worker' %}
+  {% set http_resources = http_resources + ['client', 'federation'] %}
 {% endif %}
-{% if matrix_synapse_worker_details.type in ['media_repository'] %}
+{#
+  None of the background workers need to handle federation traffic.
+  Only some of the stream writers need to handle client traffic.
+#}
+{% if matrix_synapse_worker_details.type == 'stream_writer' and matrix_synapse_worker_details.webserving %}
+  {% set http_resources = http_resources + ['client'] %}
+{% endif %}
+{% if matrix_synapse_worker_details.type == 'media_repository' %}
   {% set http_resources = http_resources + ['media'] %}
 {% endif %}
 
-{% if http_resources|length > 0 or matrix_synapse_metrics_enabled %}
+{% set replication_http_resources = [] %}
+{% if matrix_synapse_worker_details.type == 'stream_writer' %}
+  {# All background workers need to handle replication traffic. #}
+  {% set replication_http_resources = replication_http_resources + ['replication'] %}
+{% endif %}
+
+{% if http_resources|length > 0 or matrix_synapse_metrics_enabled or replication_http_resources|length > 0 %}
 worker_listeners:
 {% if http_resources|length > 0 %}
   - type: http
@@ -36,6 +47,13 @@ worker_listeners:
     bind_addresses: ['0.0.0.0']
     port: {{ matrix_synapse_worker_details.metrics_port }}
 {% endif %}
+{% if replication_http_resources|length > 0 %}
+  - type: http
+    bind_addresses: ['::']
+    port: {{ matrix_synapse_worker_details.replication_port }}
+    resources:
+      - names: {{ replication_http_resources|to_json }}
+{% endif %}
 {% endif %}
 
 {% if matrix_synapse_worker_details.type == 'generic_worker' %}
diff --git a/roles/matrix-synapse/vars/main.yml b/roles/matrix-synapse/vars/main.yml
index 2d9b62cf7..e8b0e49c6 100644
--- a/roles/matrix-synapse/vars/main.yml
+++ b/roles/matrix-synapse/vars/main.yml
@@ -36,3 +36,58 @@ matrix_synapse_workers_generic_worker_federation_endpoints: "{{ matrix_synapse_w
 # matrix_synapse_workers_generic_worker_federation_endpoints_regex contains the regex used in matrix_synapse_workers_generic_worker_federation_endpoints.
 # It's intentionally put in a separate variable, to avoid tripping ansible-lint's var-spacing rule.
 matrix_synapse_workers_generic_worker_federation_endpoints_regex: '.*(/_matrix/federation|/_matrix/key).*'
+
+# matrix_synapse_workers_stream_writer_typing_stream_worker_client_server_endpoints contains the endpoints serviced by the `typing` stream writer.
+# See: https://matrix-org.github.io/synapse/latest/workers.html#the-typing-stream
+matrix_synapse_workers_stream_writer_typing_stream_worker_client_server_endpoints:
+  - ^/_matrix/client/(api/v1|r0|v3|unstable)/rooms/.*/typing
+
+# matrix_synapse_workers_stream_writer_to_device_stream_worker_client_server_endpoints contains the endpoints serviced by the `to_device` stream writer.
+# See: https://matrix-org.github.io/synapse/latest/workers.html#the-to_device-stream
+matrix_synapse_workers_stream_writer_to_device_stream_worker_client_server_endpoints:
+  - ^/_matrix/client/(r0|v3|unstable)/sendToDevice/
+
+# matrix_synapse_workers_stream_writer_account_data_stream_worker_client_server_endpoints contains the endpoints serviced by the `account_data` stream writer.
+# See: https://matrix-org.github.io/synapse/latest/workers.html#the-account_data-stream
+matrix_synapse_workers_stream_writer_account_data_stream_worker_client_server_endpoints:
+  - ^/_matrix/client/(r0|v3|unstable)/.*/tags
+  - ^/_matrix/client/(r0|v3|unstable)/.*/account_data
+
+# matrix_synapse_workers_stream_writer_receipts_stream_worker_client_server_endpoints contains the endpoints serviced by the `recepts` stream writer.
+# See: https://matrix-org.github.io/synapse/latest/workers.html#the-receipts-stream
+matrix_synapse_workers_stream_writer_receipts_stream_worker_client_server_endpoints:
+  - ^/_matrix/client/(r0|v3|unstable)/rooms/.*/receipt
+  - ^/_matrix/client/(r0|v3|unstable)/rooms/.*/read_markers
+
+# matrix_synapse_workers_stream_writer_presence_stream_worker_client_server_endpoints contains the endpoints serviced by the `presence` stream writer.
+# See: https://matrix-org.github.io/synapse/latest/workers.html#the-presence-stream
+matrix_synapse_workers_stream_writer_presence_stream_worker_client_server_endpoints:
+  - ^/_matrix/client/(api/v1|r0|v3|unstable)/presence/
+
+# matrix_synapse_workers_known_stream_writer_stream_types contains the list of stream writer stream types that the playbook recognizes.
+# This is used for validation purposes. If adding support for a new type, besides adding it to this list,
+# don't forget to actually configure it where appropriate (see worker.yaml.j2`, the nginx proxy configuration, etc).
+matrix_synapse_workers_known_stream_writer_stream_types: ['events', 'typing', 'to_device', 'account_data', 'receipts', 'presence']
+
+# matrix_synapse_workers_webserving_stream_writer_types contains a list of stream writer types that serve web (client) requests.
+# Not all stream writers serve web requests. Some just perform background tasks.
+matrix_synapse_workers_webserving_stream_writer_types: ['typing', 'to_device', 'account_data', 'receipts', 'presence']
+
+# matrix_synapse_workers_systemd_services_list contains a list of systemd services (one for each worker systemd service which serves web requests).
+# This list is built during runtime.
+# Not all workers serve web requests. Those that don't won't be injected here.
+matrix_synapse_webserving_workers_systemd_services_list: []
+
+# matrix_synapse_known_worker_types contains the list of known worker types.
+# A worker type is different than a worker app (e.g. `generic_worker`).
+# For example, the `stream_writer` worker type is served by the `generic_worker` app, but is a separate type that we recognize.
+matrix_synapse_known_worker_types: |
+  {{
+    matrix_synapse_workers_avail_list
+    +
+    ['stream_writer']
+  }}
+
+# matrix_synapse_known_instance_map_eligible_worker_types contains the list of worker types that are to be injected into `matrix_synapse_instance_map`.
+matrix_synapse_known_instance_map_eligible_worker_types:
+  - stream_writer

From ec654ca91ee32dcac5fefd5d750bb6aa4e47896f Mon Sep 17 00:00:00 2001
From: Slavi Pantaleev <slavi@devture.com>
Date: Thu, 15 Sep 2022 08:13:08 +0300
Subject: [PATCH 02/10] Add support for multiple federation sender workers

---
 CHANGELOG.md                                        |  8 +++++++-
 roles/matrix-synapse/defaults/main.yml              | 13 +++++++++++--
 roles/matrix-synapse/tasks/synapse/workers/init.yml |  7 +++++++
 roles/matrix-synapse/tasks/validate_config.yml      |  1 -
 .../templates/synapse/homeserver.yaml.j2            |  5 ++---
 5 files changed, 27 insertions(+), 7 deletions(-)

diff --git a/CHANGELOG.md b/CHANGELOG.md
index 2b9b149e1..4a1fbb2d6 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -4,7 +4,8 @@
 
 People who are interested in running a Synapse worker setup should know that **our Synapse worker implementation is much more powerful now**:
 
-- we've added [Stream writers support](#stream-writers-support)
+- we've added support for [Stream writers](#stream-writers-support)
+- we've added support for [multiple federation sender workers](#multiple-federation-sender-workers-support)
 - see the [Potential Backward Incompatibilities after these Synapse worker changes](#potential-backward-incompatibilities-after-these-synapse-worker-changes)
 
 ### Stream writers support
@@ -26,6 +27,11 @@ matrix_synapse_workers_stream_writer_receipts_stream_workers_count: 1
 matrix_synapse_workers_stream_writer_presence_stream_workers_count: 1
 ```
 
+### Multiple federation sender workers support
+
+Until now, we only supported a single `federation_sender` worker (`matrix_synapse_workers_federation_sender_workers_count` could either be `0` or `1`).
+From now on, you can have as many as you want to help with your federation traffic.
+
 ### Potential Backward Incompatibilities after these Synapse worker changes
 
 Below we'll discuss **potential backward incompatibilities**.
diff --git a/roles/matrix-synapse/defaults/main.yml b/roles/matrix-synapse/defaults/main.yml
index 4213c4f18..6349fa6f1 100644
--- a/roles/matrix-synapse/defaults/main.yml
+++ b/roles/matrix-synapse/defaults/main.yml
@@ -510,12 +510,21 @@ matrix_synapse_workers_pusher_workers_metrics_range_start: 19200
 matrix_synapse_workers_appservice_workers_count: 0
 matrix_synapse_workers_appservice_workers_metrics_range_start: 19300
 
-# matrix_synapse_workers_federation_sender_workers_count can only be 0 or 1 for now.
-# More instances are not supported due to a playbook limitation having to do with keeping `federation_sender_instances` in `homeserver.yaml` updated.
+# matrix_synapse_workers_federation_sender_workers_count controls the number of federation sender workers to spawn.
 # See https://github.com/matrix-org/synapse/blob/master/docs/workers.md#synapseappfederation_sender
 matrix_synapse_workers_federation_sender_workers_count: "{{ matrix_synapse_workers_presets[matrix_synapse_workers_preset]['federation_sender_workers_count'] }}"
 matrix_synapse_workers_federation_sender_workers_metrics_range_start: 19400
 
+# matrix_synapse_federation_sender_instances populates the `federation_sender_instances` Synapse configuration used when Synapse workers are in use (`matrix_synapse_workers_enabled`).
+# What you see below is an initial default value which will be adjusted at runtime based on the value of `matrix_synapse_workers_federation_sender_workers_count` or `matrix_synapse_workers_enabled_list`.
+# Adjusting this value manually is generally not necessary.
+matrix_synapse_federation_sender_instances: []
+
+# matrix_synapse_send_federation controls if theh main Synapse process should send federation traffic.
+# This is allowed if workers are disabled, or if there are no federation sender workers.
+# Adjusting this value manually is generally not necessary.
+matrix_synapse_send_federation: "{{ not matrix_synapse_workers_enabled or (matrix_synapse_workers_enabled_list | selectattr('type', 'equalto', 'federation_sender') | list | length > 0) }}"
+
 matrix_synapse_workers_media_repository_workers_count: "{{ matrix_synapse_workers_presets[matrix_synapse_workers_preset]['media_repository_workers_count'] }}"
 matrix_synapse_workers_media_repository_workers_port_range_start: 18551
 matrix_synapse_workers_media_repository_workers_metrics_range_start: 19551
diff --git a/roles/matrix-synapse/tasks/synapse/workers/init.yml b/roles/matrix-synapse/tasks/synapse/workers/init.yml
index fe613e70d..6ab6784f3 100644
--- a/roles/matrix-synapse/tasks/synapse/workers/init.yml
+++ b/roles/matrix-synapse/tasks/synapse/workers/init.yml
@@ -53,6 +53,13 @@
   register: "matrix_synapse_workers_list_results_federation_sender_workers"
   loop: "{{ range(0, matrix_synapse_workers_federation_sender_workers_count | int) | list }}"
 
+- name: Populate matrix_synapse_federation_sender_instances from enabled federation sender workers list
+  ansible.builtin.set_fact:
+    matrix_synapse_federation_sender_instances: "{{ matrix_synapse_federation_sender_instances + [item.ansible_facts.worker.name] }}"
+  with_items: "{{ matrix_synapse_workers_list_results_federation_sender_workers.results }}"
+
+- debug: var="matrix_synapse_federation_sender_instances"
+
 # This type of worker can only have a count of 1, at most
 - name: Build pusher workers
   ansible.builtin.set_fact:
diff --git a/roles/matrix-synapse/tasks/validate_config.yml b/roles/matrix-synapse/tasks/validate_config.yml
index 1fc708166..04a04cd51 100644
--- a/roles/matrix-synapse/tasks/validate_config.yml
+++ b/roles/matrix-synapse/tasks/validate_config.yml
@@ -31,7 +31,6 @@
   when: "vars[item]|int > 1"
   with_items:
     - "matrix_synapse_workers_pusher_workers_count"
-    - "matrix_synapse_workers_federation_sender_workers_count"
     - "matrix_synapse_workers_stream_writer_typing_stream_workers_count"
     - "matrix_synapse_workers_stream_writer_to_device_stream_workers_count"
     - "matrix_synapse_workers_stream_writer_account_data_stream_workers_count"
diff --git a/roles/matrix-synapse/templates/synapse/homeserver.yaml.j2 b/roles/matrix-synapse/templates/synapse/homeserver.yaml.j2
index 86d03be41..9f6679fc4 100644
--- a/roles/matrix-synapse/templates/synapse/homeserver.yaml.j2
+++ b/roles/matrix-synapse/templates/synapse/homeserver.yaml.j2
@@ -352,9 +352,6 @@ worker_app: synapse.app.homeserver
 
 # thx https://oznetnerd.com/2017/04/18/jinja2-selectattr-filter/
 # reduce the main worker's offerings to core homeserver business
-{% if matrix_synapse_workers_enabled_list | selectattr('type', 'equalto', 'federation_sender') | list | length > 0  %}
-send_federation: false
-{% endif %}
 {% if matrix_synapse_workers_enabled_list | selectattr('type', 'equalto', 'media_repository') | list | length > 0  %}
 enable_media_repo: false
 {% endif %}
@@ -2851,6 +2848,7 @@ opentracing:
 # Uncomment if using a federation sender worker.
 #
 #send_federation: false
+send_federation: {{ matrix_synapse_send_federation | to_json }}
 
 # It is possible to run multiple federation sender workers, in which case the
 # work is balanced across them.
@@ -2862,6 +2860,7 @@ opentracing:
 #
 #federation_sender_instances:
 #  - federation_sender1
+federation_sender_instances: {{ matrix_synapse_federation_sender_instances | to_json }}
 
 # When using workers this should be a map from `worker_name` to the
 # HTTP replication listener of the worker, if configured.

From 22cf259155e0a3cac5d0d53fd330e7ea028bcf13 Mon Sep 17 00:00:00 2001
From: Slavi Pantaleev <slavi@devture.com>
Date: Thu, 15 Sep 2022 08:14:47 +0300
Subject: [PATCH 03/10] Put common worker configuration options at the top

---
 .../templates/synapse/worker.yaml.j2               | 14 +++++++-------
 1 file changed, 7 insertions(+), 7 deletions(-)

diff --git a/roles/matrix-synapse/templates/synapse/worker.yaml.j2 b/roles/matrix-synapse/templates/synapse/worker.yaml.j2
index c9637a839..2b0df98dd 100644
--- a/roles/matrix-synapse/templates/synapse/worker.yaml.j2
+++ b/roles/matrix-synapse/templates/synapse/worker.yaml.j2
@@ -2,11 +2,18 @@
 worker_app: synapse.app.{{ matrix_synapse_worker_details.app }}
 worker_name: {{ matrix_synapse_worker_details.name }}
 
+worker_daemonize: false
+worker_log_config: /data/{{ matrix_server_fqn_matrix }}.log.config
+
 {% if matrix_synapse_replication_listener_enabled %}
 worker_replication_host: matrix-synapse
 worker_replication_http_port: {{ matrix_synapse_replication_http_port }}
 {% endif %}
 
+{% if matrix_synapse_worker_details.type == 'generic_worker' %}
+worker_main_http_uri: http://matrix-synapse:{{ matrix_synapse_container_client_api_port }}
+{% endif %}
+
 {% set http_resources = [] %}
 
 {% if matrix_synapse_worker_details.type == 'user_dir' %}
@@ -55,10 +62,3 @@ worker_listeners:
       - names: {{ replication_http_resources|to_json }}
 {% endif %}
 {% endif %}
-
-{% if matrix_synapse_worker_details.type == 'generic_worker' %}
-worker_main_http_uri: http://matrix-synapse:{{ matrix_synapse_container_client_api_port }}
-{% endif %}
-
-worker_daemonize: false
-worker_log_config: /data/{{ matrix_server_fqn_matrix }}.log.config

From b842447047be77bcf209d9f3ed0609e063b8a192 Mon Sep 17 00:00:00 2001
From: Slavi Pantaleev <slavi@devture.com>
Date: Thu, 15 Sep 2022 08:32:55 +0300
Subject: [PATCH 04/10] Add support for multiple pusher workers

---
 CHANGELOG.md                                  |  7 +++++++
 roles/matrix-synapse/defaults/main.yml        | 19 ++++++++++++++-----
 .../matrix-synapse/tasks/validate_config.yml  |  3 +--
 .../templates/synapse/homeserver.yaml.j2      |  6 +++---
 4 files changed, 25 insertions(+), 10 deletions(-)

diff --git a/CHANGELOG.md b/CHANGELOG.md
index 4a1fbb2d6..90b9b8f76 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -6,6 +6,7 @@ People who are interested in running a Synapse worker setup should know that **o
 
 - we've added support for [Stream writers](#stream-writers-support)
 - we've added support for [multiple federation sender workers](#multiple-federation-sender-workers-support)
+- we've added support for [multiple pusher workers](#multiple-pusher-workers-support)
 - see the [Potential Backward Incompatibilities after these Synapse worker changes](#potential-backward-incompatibilities-after-these-synapse-worker-changes)
 
 ### Stream writers support
@@ -32,6 +33,12 @@ matrix_synapse_workers_stream_writer_presence_stream_workers_count: 1
 Until now, we only supported a single `federation_sender` worker (`matrix_synapse_workers_federation_sender_workers_count` could either be `0` or `1`).
 From now on, you can have as many as you want to help with your federation traffic.
 
+### Multiple pusher workers support
+
+Until now, we only supported a single `pusher` worker (`matrix_synapse_workers_pusher_workers_count` could either be `0` or `1`).
+From now on, you can have as many as you want to help with pushing notifications out.
+
+
 ### Potential Backward Incompatibilities after these Synapse worker changes
 
 Below we'll discuss **potential backward incompatibilities**.
diff --git a/roles/matrix-synapse/defaults/main.yml b/roles/matrix-synapse/defaults/main.yml
index 6349fa6f1..ca84ff38d 100644
--- a/roles/matrix-synapse/defaults/main.yml
+++ b/roles/matrix-synapse/defaults/main.yml
@@ -498,12 +498,21 @@ matrix_synapse_workers_stream_writer_workers_http_port_range_start: 20011
 matrix_synapse_workers_stream_writer_workers_replication_port_range_start: 25011
 matrix_synapse_workers_stream_writer_workers_metrics_range_start: 19111
 
-# matrix_synapse_workers_pusher_workers_count can only be 0 or 1 for now.
-# More instances are not supported due to a playbook limitation having to do with keeping `pusher_instances` in `homeserver.yaml` updated.
-# See https://github.com/matrix-org/synapse/commit/ddfdf945064925eba761ae3748e38f3a1c73c328
+# matrix_synapse_workers_pusher_workers_count controls the number of pusher workers (workers who push out notifications) to spawn.
+# See https://matrix-org.github.io/synapse/latest/workers.html#synapseapppusher
 matrix_synapse_workers_pusher_workers_count: "{{ matrix_synapse_workers_presets[matrix_synapse_workers_preset]['pusher_workers_count'] }}"
 matrix_synapse_workers_pusher_workers_metrics_range_start: 19200
 
+# matrix_synapse_federation_pusher_instances populates the `pusher_instances` Synapse configuration used when Synapse workers are in use (`matrix_synapse_workers_enabled`).
+# What you see below is an initial default value which will be adjusted at runtime based on the value of `matrix_synapse_workers_pusher_workers_count` or `matrix_synapse_workers_enabled_list`.
+# Adjusting this value manually is generally not necessary.
+matrix_synapse_federation_pusher_instances: []
+
+# matrix_synapse_start_pushers controls if theh main Synapse process should push out notifications or if it should be left to pusher workers (see `matrix_synapse_federation_pusher_instances`).
+# This is allowed if workers are disabled, or if there are no pusher workers.
+# Adjusting this value manually is generally not necessary.
+matrix_synapse_start_pushers: "{{ not matrix_synapse_workers_enabled or (matrix_synapse_workers_enabled_list | selectattr('type', 'equalto', 'pusher') | list | length > 0) }}"
+
 # matrix_synapse_workers_appservice_workers_count can only be 0 or 1. More instances are not supported.
 # appservice workers are deprecated since Synapse v1.59. This will be removed.
 # See: https://github.com/matrix-org/synapse/blob/v1.59.0/docs/upgrade.md#deprecation-of-the-synapseappappservice-and-synapseappuser_dir-worker-application-types
@@ -511,7 +520,7 @@ matrix_synapse_workers_appservice_workers_count: 0
 matrix_synapse_workers_appservice_workers_metrics_range_start: 19300
 
 # matrix_synapse_workers_federation_sender_workers_count controls the number of federation sender workers to spawn.
-# See https://github.com/matrix-org/synapse/blob/master/docs/workers.md#synapseappfederation_sender
+# See https://matrix-org.github.io/synapse/latest/workers.html#synapseappfederation_sender
 matrix_synapse_workers_federation_sender_workers_count: "{{ matrix_synapse_workers_presets[matrix_synapse_workers_preset]['federation_sender_workers_count'] }}"
 matrix_synapse_workers_federation_sender_workers_metrics_range_start: 19400
 
@@ -520,7 +529,7 @@ matrix_synapse_workers_federation_sender_workers_metrics_range_start: 19400
 # Adjusting this value manually is generally not necessary.
 matrix_synapse_federation_sender_instances: []
 
-# matrix_synapse_send_federation controls if theh main Synapse process should send federation traffic.
+# matrix_synapse_send_federation controls if theh main Synapse process should send federation traffic or if it should be left to federation_sender workers (see `matrix_synapse_federation_sender_instances`).
 # This is allowed if workers are disabled, or if there are no federation sender workers.
 # Adjusting this value manually is generally not necessary.
 matrix_synapse_send_federation: "{{ not matrix_synapse_workers_enabled or (matrix_synapse_workers_enabled_list | selectattr('type', 'equalto', 'federation_sender') | list | length > 0) }}"
diff --git a/roles/matrix-synapse/tasks/validate_config.yml b/roles/matrix-synapse/tasks/validate_config.yml
index 04a04cd51..faab7daa1 100644
--- a/roles/matrix-synapse/tasks/validate_config.yml
+++ b/roles/matrix-synapse/tasks/validate_config.yml
@@ -12,7 +12,7 @@
     - "matrix_synapse_database_password"
     - "matrix_synapse_database_database"
 
-- name: Fail if asking to configure deprecaed workers (appservice, userdir)
+- name: Fail if asking to configure deprecated workers (appservice, userdir)
   ansible.builtin.fail:
     msg: >-
       `{{ item }}` cannot be more than 0.
@@ -30,7 +30,6 @@
       `{{ item }}` cannot be more than 1. This is a single-instance worker.
   when: "vars[item]|int > 1"
   with_items:
-    - "matrix_synapse_workers_pusher_workers_count"
     - "matrix_synapse_workers_stream_writer_typing_stream_workers_count"
     - "matrix_synapse_workers_stream_writer_to_device_stream_workers_count"
     - "matrix_synapse_workers_stream_writer_account_data_stream_workers_count"
diff --git a/roles/matrix-synapse/templates/synapse/homeserver.yaml.j2 b/roles/matrix-synapse/templates/synapse/homeserver.yaml.j2
index 9f6679fc4..5e60f1c03 100644
--- a/roles/matrix-synapse/templates/synapse/homeserver.yaml.j2
+++ b/roles/matrix-synapse/templates/synapse/homeserver.yaml.j2
@@ -355,9 +355,6 @@ worker_app: synapse.app.homeserver
 {% if matrix_synapse_workers_enabled_list | selectattr('type', 'equalto', 'media_repository') | list | length > 0  %}
 enable_media_repo: false
 {% endif %}
-{% if matrix_synapse_workers_enabled_list | selectattr('type', 'equalto', 'pusher') | list | length > 0  %}
-start_pushers: false
-{% endif %}
 
 daemonize: false
 {% endif %}
@@ -2862,6 +2859,9 @@ send_federation: {{ matrix_synapse_send_federation | to_json }}
 #  - federation_sender1
 federation_sender_instances: {{ matrix_synapse_federation_sender_instances | to_json }}
 
+pusher_instances: {{ matrix_synapse_federation_pusher_instances | to_json }}
+start_pushers: {{ matrix_synapse_start_pushers | to_json }}
+
 # When using workers this should be a map from `worker_name` to the
 # HTTP replication listener of the worker, if configured.
 #

From 8cd7c1ec2f85d80659f33fe7334f4a4ddfd34ac4 Mon Sep 17 00:00:00 2001
From: Slavi Pantaleev <slavi@devture.com>
Date: Thu, 15 Sep 2022 08:37:19 +0300
Subject: [PATCH 05/10] Relocate enable_media_repo and make it configurable via
 matrix_synapse_enable_media_repo

.. although.. manual configuration is discouraged in most cases.
---
 roles/matrix-synapse/defaults/main.yml                | 11 ++++++++---
 .../templates/synapse/homeserver.yaml.j2              |  8 +-------
 2 files changed, 9 insertions(+), 10 deletions(-)

diff --git a/roles/matrix-synapse/defaults/main.yml b/roles/matrix-synapse/defaults/main.yml
index ca84ff38d..05cd767a6 100644
--- a/roles/matrix-synapse/defaults/main.yml
+++ b/roles/matrix-synapse/defaults/main.yml
@@ -508,8 +508,8 @@ matrix_synapse_workers_pusher_workers_metrics_range_start: 19200
 # Adjusting this value manually is generally not necessary.
 matrix_synapse_federation_pusher_instances: []
 
-# matrix_synapse_start_pushers controls if theh main Synapse process should push out notifications or if it should be left to pusher workers (see `matrix_synapse_federation_pusher_instances`).
-# This is allowed if workers are disabled, or if there are no pusher workers.
+# matrix_synapse_start_pushers controls if the main Synapse process should push out notifications or if it should be left to pusher workers (see `matrix_synapse_federation_pusher_instances`).
+# This is enabled if workers are disabled, or if there are no pusher workers.
 # Adjusting this value manually is generally not necessary.
 matrix_synapse_start_pushers: "{{ not matrix_synapse_workers_enabled or (matrix_synapse_workers_enabled_list | selectattr('type', 'equalto', 'pusher') | list | length > 0) }}"
 
@@ -529,7 +529,7 @@ matrix_synapse_workers_federation_sender_workers_metrics_range_start: 19400
 # Adjusting this value manually is generally not necessary.
 matrix_synapse_federation_sender_instances: []
 
-# matrix_synapse_send_federation controls if theh main Synapse process should send federation traffic or if it should be left to federation_sender workers (see `matrix_synapse_federation_sender_instances`).
+# matrix_synapse_send_federation controls if the main Synapse process should send federation traffic or if it should be left to federation_sender workers (see `matrix_synapse_federation_sender_instances`).
 # This is allowed if workers are disabled, or if there are no federation sender workers.
 # Adjusting this value manually is generally not necessary.
 matrix_synapse_send_federation: "{{ not matrix_synapse_workers_enabled or (matrix_synapse_workers_enabled_list | selectattr('type', 'equalto', 'federation_sender') | list | length > 0) }}"
@@ -538,6 +538,11 @@ matrix_synapse_workers_media_repository_workers_count: "{{ matrix_synapse_worker
 matrix_synapse_workers_media_repository_workers_port_range_start: 18551
 matrix_synapse_workers_media_repository_workers_metrics_range_start: 19551
 
+# matrix_synapse_enable_media_repo controls if the main Synapse process should serve media repository endpoints or if it should be left to media_repository workers (see `matrix_synapse_workers_media_repository_workers_count`).
+# This is enabled if workers are disabled, or if there are no media repository workers.
+# Adjusting this value manually is generally not necessary.
+matrix_synapse_enable_media_repo: "{{ not matrix_synapse_workers_enabled or (matrix_synapse_workers_enabled_list | selectattr('type', 'equalto', 'media_repository') | list | length > 0) }}"
+
 # Disabled until https://github.com/matrix-org/synapse/issues/8787 is resolved.
 # user_dir workers are deprecated since Synapse v1.59. This will be removed.
 # See: https://github.com/matrix-org/synapse/blob/v1.59.0/docs/upgrade.md#deprecation-of-the-synapseappappservice-and-synapseappuser_dir-worker-application-types
diff --git a/roles/matrix-synapse/templates/synapse/homeserver.yaml.j2 b/roles/matrix-synapse/templates/synapse/homeserver.yaml.j2
index 5e60f1c03..a635559f2 100644
--- a/roles/matrix-synapse/templates/synapse/homeserver.yaml.j2
+++ b/roles/matrix-synapse/templates/synapse/homeserver.yaml.j2
@@ -349,13 +349,6 @@ listeners:
 
 # c.f. https://github.com/matrix-org/synapse/tree/master/contrib/systemd-with-workers/README.md
 worker_app: synapse.app.homeserver
-
-# thx https://oznetnerd.com/2017/04/18/jinja2-selectattr-filter/
-# reduce the main worker's offerings to core homeserver business
-{% if matrix_synapse_workers_enabled_list | selectattr('type', 'equalto', 'media_repository') | list | length > 0  %}
-enable_media_repo: false
-{% endif %}
-
 daemonize: false
 {% endif %}
 
@@ -1017,6 +1010,7 @@ federation_rr_transactions_per_room_per_second: {{ matrix_synapse_federation_rr_
 # following if you are using a separate media store worker.
 #
 #enable_media_repo: false
+enable_media_repo: {{ matrix_synapse_enable_media_repo | to_json }}
 
 # Directory where uploaded images and attachments are stored.
 #

From 5f3f460cda5653232087c146b4cd0fa74dff1c96 Mon Sep 17 00:00:00 2001
From: Slavi Pantaleev <slavi@devture.com>
Date: Thu, 15 Sep 2022 10:00:57 +0300
Subject: [PATCH 06/10] Restore support for appservice and user_dir workers

---
 CHANGELOG.md                                  | 25 +++++++++++-
 group_vars/matrix_servers                     |  2 +-
 .../nginx/conf.d/matrix-synapse.conf.j2       |  5 ++-
 roles/matrix-synapse/defaults/main.yml        | 38 ++++++++++---------
 .../tasks/synapse/workers/init.yml            | 20 ++++++++--
 .../synapse/workers/util/inject_worker.yml    |  2 +-
 .../matrix-synapse/tasks/validate_config.yml  | 14 +------
 .../templates/synapse/homeserver.yaml.j2      |  8 ++++
 roles/matrix-synapse/vars/main.yml            | 23 +++++++++--
 roles/matrix-synapse/vars/workers.yml         |  4 +-
 10 files changed, 100 insertions(+), 41 deletions(-)

diff --git a/CHANGELOG.md b/CHANGELOG.md
index 90b9b8f76..37032735e 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,12 +1,14 @@
 # 2022-09-15
 
-## (Potential Backward Compatibility Break) Major changes to Synapse workers
+## (Potential Backward Compatibility Break) Major improvements to Synapse workers
 
 People who are interested in running a Synapse worker setup should know that **our Synapse worker implementation is much more powerful now**:
 
 - we've added support for [Stream writers](#stream-writers-support)
 - we've added support for [multiple federation sender workers](#multiple-federation-sender-workers-support)
 - we've added support for [multiple pusher workers](#multiple-pusher-workers-support)
+- we've restored support for [`appservice` workers](#appservice-worker-support-is-back)
+- we've restored support for [`user_dir` workers](#user-directory-worker-support-is-back)
 - see the [Potential Backward Incompatibilities after these Synapse worker changes](#potential-backward-incompatibilities-after-these-synapse-worker-changes)
 
 ### Stream writers support
@@ -38,6 +40,25 @@ From now on, you can have as many as you want to help with your federation traff
 Until now, we only supported a single `pusher` worker (`matrix_synapse_workers_pusher_workers_count` could either be `0` or `1`).
 From now on, you can have as many as you want to help with pushing notifications out.
 
+### Appservice worker support is back
+
+We previously had an `appservice` worker type, which [Synapse deprecated in v1.59.0](https://github.com/matrix-org/synapse/blob/v1.59.0/docs/upgrade.md#deprecation-of-the-synapseappappservice-and-synapseappuser_dir-worker-application-types). So did we, at the time.
+
+The new way to implement such workers is by using a `generic_worker` and dedicating it to the task of talking to Application Services.
+From now on, we have support for this.
+
+With `matrix_synapse_workers_preset: one-of-each`, you'll get one `appserice` worker automatically.
+You can also control the `appserice` workers count with `matrix_synapse_workers_appserice_workers_count`. Only  `0` or `1` workers of this type are supported by Synapse.
+
+### User Directory worker support is back
+
+We previously had a `user_dir` worker type, which [Synapse deprecated in v1.59.0](https://github.com/matrix-org/synapse/blob/v1.59.0/docs/upgrade.md#deprecation-of-the-synapseappappservice-and-synapseappuser_dir-worker-application-types). So did we, at the time.
+
+The new way to implement such workers is by using a `generic_worker` and dedicating it to the task of serving the user directory.
+From now on, we have support for this.
+
+With `matrix_synapse_workers_preset: one-of-each`, you'll get one `user_dir` worker automatically.
+You can also control the `user_dir` workers count with `matrix_synapse_workers_user_dir_workers_count`. Only  `0` or `1` workers of this type are supported by Synapse.
 
 ### Potential Backward Incompatibilities after these Synapse worker changes
 
@@ -45,6 +66,8 @@ Below we'll discuss **potential backward incompatibilities**.
 
 - **Worker names** (container names, systemd services, worker configuration files) **have changed**. Workers are now labeled sequentially (e.g. `matrix-synapse-worker_generic_worker-18111` -> `matrix-synapse-worker-generic-0`). The playbook will handle these changes automatically.
 
+- Due to increased worker types support above, people who use `matrix_synapse_workers_preset: one-of-each` should be aware that with these changes, **the playbook will deploy 8 additional workers** (6 stream writers, 1 `appservice` worker, 1 `user_dir` worker). This **may increase RAM/CPU usage**, etc. If you find your server struggling, consider disabling some workers with the appropriate `matrix_synapse_workers_*_workers_count` variables.
+
 - **Metric endpoints have also changed** (`/metrics/synapse/worker/generic_worker-18111` -> `/metrics/synapse/worker/generic-worker-0`). If you're [collecting metrics to an external Prometheus server](docs/configuring-playbook-prometheus-grafana.md#collecting-metrics-to-an-external-prometheus-server), consider revisiting our [Collecting Synapse worker metrics to an external Prometheus server](docs/configuring-playbook-prometheus-grafana.md#collecting-synapse-worker-metrics-to-an-external-prometheus-server) docs and updating your Prometheus configuration. **If you're collecting metrics to the integrated Prometheus server** (not enabled by default), **your Prometheus configuration will be updated automatically**. Old data (from before this change) may stick around though.
 
 - **the format of `matrix_synapse_workers_enabled_list` has changed**. You were never advised to use this variable for directly creating workers (we advise people to control workers using `matrix_synapse_workers_preset` or by tweaking `matrix_synapse_workers_*_workers_count` variables only), but some people may have started using the `matrix_synapse_workers_enabled_list` variable to gain more control over workers. If you're one of them, you'll need to adjust its value. See `roles/matrix-synapse/defaults/main.yml` for more information on the new format. The playbook will also do basic validation and complain if you got something wrong.
diff --git a/group_vars/matrix_servers b/group_vars/matrix_servers
index e891a3d9a..78d1ed68a 100755
--- a/group_vars/matrix_servers
+++ b/group_vars/matrix_servers
@@ -1766,7 +1766,7 @@ matrix_nginx_proxy_synapse_stream_writer_account_data_stream_worker_client_serve
 matrix_nginx_proxy_synapse_stream_writer_receipts_stream_worker_client_server_locations: "{{ matrix_synapse_workers_stream_writer_receipts_stream_worker_client_server_endpoints }}"
 matrix_nginx_proxy_synapse_stream_writer_presence_stream_worker_client_server_locations: "{{ matrix_synapse_workers_stream_writer_presence_stream_worker_client_server_endpoints }}"
 matrix_nginx_proxy_synapse_media_repository_locations: "{{matrix_synapse_workers_media_repository_endpoints|default([]) }}"
-matrix_nginx_proxy_synapse_user_dir_locations: "{{ matrix_synapse_workers_user_dir_endpoints|default([]) }}"
+matrix_nginx_proxy_synapse_user_dir_locations: "{{ matrix_synapse_workers_user_dir_worker_client_server_endpoints|default([]) }}"
 
 matrix_nginx_proxy_systemd_wanted_services_list: |
   {{
diff --git a/roles/matrix-nginx-proxy/templates/nginx/conf.d/matrix-synapse.conf.j2 b/roles/matrix-nginx-proxy/templates/nginx/conf.d/matrix-synapse.conf.j2
index bbce7462b..488ced5a8 100644
--- a/roles/matrix-nginx-proxy/templates/nginx/conf.d/matrix-synapse.conf.j2
+++ b/roles/matrix-nginx-proxy/templates/nginx/conf.d/matrix-synapse.conf.j2
@@ -133,8 +133,9 @@ server {
 		{% endif %}
 
 		{% if user_dir_workers | length > 0 %}
-			# FIXME: obsolete if matrix_nginx_proxy_proxy_matrix_user_directory_search_enabled is set
-			# https://github.com/matrix-org/synapse/blob/master/docs/workers.md#synapseappuser_dir
+			# https://matrix-org.github.io/synapse/latest/workers.html#updating-the-user-directory
+			# If matrix_nginx_proxy_proxy_matrix_user_directory_search_enabled is set, requests may not reach here,
+			# but could be captured early on (see `matrix-domain.conf.j2`) and forwarded elsewhere (to an identity server, etc.).
 			{{ render_locations_to_upstream(matrix_nginx_proxy_synapse_user_dir_locations, 'user_dir_workers_upstream') }}
 		{% endif %}
 		{# Workers redirects END #}
diff --git a/roles/matrix-synapse/defaults/main.yml b/roles/matrix-synapse/defaults/main.yml
index 05cd767a6..7f6ce40fa 100644
--- a/roles/matrix-synapse/defaults/main.yml
+++ b/roles/matrix-synapse/defaults/main.yml
@@ -394,9 +394,9 @@ matrix_synapse_workers_presets:
   little-federation-helper:
     generic_workers_count: 0
     pusher_workers_count: 0
-    appservice_workers_count: 0
     federation_sender_workers_count: 1
     media_repository_workers_count: 0
+    appservice_workers_count: 0
     user_dir_workers_count: 0
     stream_writer_events_stream_workers_count: 0
     stream_writer_typing_stream_workers_count: 0
@@ -407,14 +407,10 @@ matrix_synapse_workers_presets:
   one-of-each:
     generic_workers_count: 1
     pusher_workers_count: 1
-    # appservice workers are deprecated since Synapse v1.59. This will be removed.
-    appservice_workers_count: 0
     federation_sender_workers_count: 1
     media_repository_workers_count: 1
-    # Disabled until https://github.com/matrix-org/synapse/issues/8787 is resolved.
-    # user_dir workers are deprecated since Synapse v1.59. This will be removed.
-    # See: https://github.com/matrix-org/synapse/blob/v1.59.0/docs/upgrade.md#deprecation-of-the-synapseappappservice-and-synapseappuser_dir-worker-application-types
-    user_dir_workers_count: 0
+    appservice_workers_count: 1
+    user_dir_workers_count: 1
     stream_writer_events_stream_workers_count: 1
     stream_writer_typing_stream_workers_count: 1
     stream_writer_to_device_stream_workers_count: 1
@@ -513,12 +509,6 @@ matrix_synapse_federation_pusher_instances: []
 # Adjusting this value manually is generally not necessary.
 matrix_synapse_start_pushers: "{{ not matrix_synapse_workers_enabled or (matrix_synapse_workers_enabled_list | selectattr('type', 'equalto', 'pusher') | list | length > 0) }}"
 
-# matrix_synapse_workers_appservice_workers_count can only be 0 or 1. More instances are not supported.
-# appservice workers are deprecated since Synapse v1.59. This will be removed.
-# See: https://github.com/matrix-org/synapse/blob/v1.59.0/docs/upgrade.md#deprecation-of-the-synapseappappservice-and-synapseappuser_dir-worker-application-types
-matrix_synapse_workers_appservice_workers_count: 0
-matrix_synapse_workers_appservice_workers_metrics_range_start: 19300
-
 # matrix_synapse_workers_federation_sender_workers_count controls the number of federation sender workers to spawn.
 # See https://matrix-org.github.io/synapse/latest/workers.html#synapseappfederation_sender
 matrix_synapse_workers_federation_sender_workers_count: "{{ matrix_synapse_workers_presets[matrix_synapse_workers_preset]['federation_sender_workers_count'] }}"
@@ -543,13 +533,27 @@ matrix_synapse_workers_media_repository_workers_metrics_range_start: 19551
 # Adjusting this value manually is generally not necessary.
 matrix_synapse_enable_media_repo: "{{ not matrix_synapse_workers_enabled or (matrix_synapse_workers_enabled_list | selectattr('type', 'equalto', 'media_repository') | list | length > 0) }}"
 
-# Disabled until https://github.com/matrix-org/synapse/issues/8787 is resolved.
-# user_dir workers are deprecated since Synapse v1.59. This will be removed.
-# See: https://github.com/matrix-org/synapse/blob/v1.59.0/docs/upgrade.md#deprecation-of-the-synapseappappservice-and-synapseappuser_dir-worker-application-types
-matrix_synapse_workers_user_dir_workers_count: 0
+# matrix_synapse_workers_appservice_workers_count can only be 0 or 1. More instances are not supported.
+# appservice workers were deprecated since Synapse v1.59 (see: https://github.com/matrix-org/synapse/blob/v1.59.0/docs/upgrade.md#deprecation-of-the-synapseappappservice-and-synapseappuser_dir-worker-application-types).
+# Our implementation uses generic worker services and assigns them to perform appservice work using the `notify_appservices_from_worker` Synapse option.
+matrix_synapse_workers_appservice_workers_count: "{{ matrix_synapse_workers_presets[matrix_synapse_workers_preset]['appservice_workers_count'] }}"
+matrix_synapse_workers_appservice_workers_metrics_range_start: 19300
+
+# matrix_synapse_notify_appservices_from_worker populates the `notify_appservices_from_worker` Synapse configuration used when Synapse workers are in use (`matrix_synapse_workers_enabled`).
+# `notify_appservices_from_worker` is meant to point to a worker, which is dedicated to sending output traffic to Application Services.
+matrix_synapse_notify_appservices_from_worker: "{{ (matrix_synapse_workers_enabled_list | selectattr('type', 'equalto', 'appservice') | list)[0].name if (matrix_synapse_workers_enabled and matrix_synapse_workers_enabled_list | selectattr('type', 'equalto', 'appservice') | list | length > 0) else '' }}"
+
+# matrix_synapse_workers_user_dir_workers_count can only be 0 or 1. More instances are not supported.
+# user_dir workers were deprecated since Synapse v1.59 (see: https://github.com/matrix-org/synapse/blob/v1.59.0/docs/upgrade.md#deprecation-of-the-synapseappappservice-and-synapseappuser_dir-worker-application-types).
+# Our implementation uses generic worker services and assigns them to perform appservice work using the `update_user_directory_from_worker` Synapse option.
+matrix_synapse_workers_user_dir_workers_count: "{{ matrix_synapse_workers_presets[matrix_synapse_workers_preset]['user_dir_workers_count'] }}"
 matrix_synapse_workers_user_dir_workers_port_range_start: 18661
 matrix_synapse_workers_user_dir_workers_metrics_range_start: 19661
 
+# matrix_synapse_update_user_directory_from_worker populates the `update_user_directory_from_worker` Synapse configuration used when Synapse workers are in use (`matrix_synapse_workers_enabled`).
+# `update_user_directory_from_worker` is meant to point to a worker, which is dedicated to updating the user directory and servicing some user directory URL endpoints (`matrix_synapse_workers_user_dir_worker_client_server_endpoints`).
+matrix_synapse_update_user_directory_from_worker: "{{ (matrix_synapse_workers_enabled_list | selectattr('type', 'equalto', 'user_dir') | list)[0].name if (matrix_synapse_workers_enabled and matrix_synapse_workers_enabled_list | selectattr('type', 'equalto', 'user_dir') | list | length > 0) else '' }}"
+
 # Default list of workers to spawn.
 #
 # Unless you populate this manually, this list is dynamically generated
diff --git a/roles/matrix-synapse/tasks/synapse/workers/init.yml b/roles/matrix-synapse/tasks/synapse/workers/init.yml
index 6ab6784f3..c5a532979 100644
--- a/roles/matrix-synapse/tasks/synapse/workers/init.yml
+++ b/roles/matrix-synapse/tasks/synapse/workers/init.yml
@@ -58,8 +58,6 @@
     matrix_synapse_federation_sender_instances: "{{ matrix_synapse_federation_sender_instances + [item.ansible_facts.worker.name] }}"
   with_items: "{{ matrix_synapse_workers_list_results_federation_sender_workers.results }}"
 
-- debug: var="matrix_synapse_federation_sender_instances"
-
 # This type of worker can only have a count of 1, at most
 - name: Build pusher workers
   ansible.builtin.set_fact:
@@ -81,13 +79,27 @@
       id: "appservice-{{ item }}"
       name: "matrix-synapse-worker-appservice-{{ item }}"
       type: 'appservice'
-      app: 'appservice'
+      app: 'generic_worker'
       webserving: false
       port: 0
       metrics_port: "{{ matrix_synapse_workers_appservice_workers_metrics_range_start + item }}"
   register: "matrix_synapse_workers_list_results_appservice_workers"
   loop: "{{ range(0, matrix_synapse_workers_appservice_workers_count | int) | list }}"
 
+# This type of worker can only have a count of 1, at most
+- name: Build user_dir workers
+  ansible.builtin.set_fact:
+    worker:
+      id: "user-dir-{{ item }}"
+      name: "matrix-synapse-worker-user-dir-{{ item }}"
+      type: 'user_dir'
+      app: 'generic_worker'
+      webserving: true
+      port: "{{ matrix_synapse_workers_user_dir_workers_port_range_start + item }}"
+      metrics_port: "{{ matrix_synapse_workers_user_dir_workers_metrics_range_start + item }}"
+  register: "matrix_synapse_workers_list_results_user_dir_workers"
+  loop: "{{ range(0, matrix_synapse_workers_user_dir_workers_count | int) | list }}"
+
 - name: Build media_repository workers
   ansible.builtin.set_fact:
     worker:
@@ -115,6 +127,8 @@
       +
       matrix_synapse_workers_list_results_appservice_workers.results
       +
+      matrix_synapse_workers_list_results_user_dir_workers.results
+      +
       matrix_synapse_workers_list_results_media_repository_workers.results
     }}
 
diff --git a/roles/matrix-synapse/tasks/synapse/workers/util/inject_worker.yml b/roles/matrix-synapse/tasks/synapse/workers/util/inject_worker.yml
index 4f6b43986..0a52db327 100644
--- a/roles/matrix-synapse/tasks/synapse/workers/util/inject_worker.yml
+++ b/roles/matrix-synapse/tasks/synapse/workers/util/inject_worker.yml
@@ -59,7 +59,7 @@
     matrix_synapse_webserving_workers_systemd_services_list: "{{ matrix_synapse_webserving_workers_systemd_services_list + [matrix_synapse_worker_details.name + '.service'] }}"
   when: matrix_synapse_worker_details.webserving | bool
 
-# Inject stream writers and various other background workers into the instance map.
+# Inject stream writers into the instance map.
 - ansible.builtin.set_fact:
     matrix_synapse_instance_map: "{{ matrix_synapse_instance_map | combine({matrix_synapse_worker_details.name: {'host': matrix_synapse_worker_details.name, 'port': matrix_synapse_worker_details.replication_port}}) }}"
   when: matrix_synapse_worker_details.type in matrix_synapse_known_instance_map_eligible_worker_types
diff --git a/roles/matrix-synapse/tasks/validate_config.yml b/roles/matrix-synapse/tasks/validate_config.yml
index faab7daa1..f6b1fe688 100644
--- a/roles/matrix-synapse/tasks/validate_config.yml
+++ b/roles/matrix-synapse/tasks/validate_config.yml
@@ -12,24 +12,14 @@
     - "matrix_synapse_database_password"
     - "matrix_synapse_database_database"
 
-- name: Fail if asking to configure deprecated workers (appservice, userdir)
-  ansible.builtin.fail:
-    msg: >-
-      `{{ item }}` cannot be more than 0.
-      This type of worker has been deprecated since Synapse v1.59.
-      Please remove your `{{ item }}` configuration to solve this problem.
-      See: https://github.com/matrix-org/synapse/blob/v1.59.0/docs/upgrade.md#deprecation-of-the-synapseappappservice-and-synapseappuser_dir-worker-application-types
-  when: "vars[item]|int != 0"
-  with_items:
-    - "matrix_synapse_workers_appservice_workers_count"
-    - "matrix_synapse_workers_user_dir_workers_count"
-
 - name: Fail if asking for more than 1 instance of single-instance workers
   ansible.builtin.fail:
     msg: >-
       `{{ item }}` cannot be more than 1. This is a single-instance worker.
   when: "vars[item]|int > 1"
   with_items:
+    - "matrix_synapse_workers_appservice_workers_count"
+    - "matrix_synapse_workers_user_dir_workers_count"
     - "matrix_synapse_workers_stream_writer_typing_stream_workers_count"
     - "matrix_synapse_workers_stream_writer_to_device_stream_workers_count"
     - "matrix_synapse_workers_stream_writer_account_data_stream_workers_count"
diff --git a/roles/matrix-synapse/templates/synapse/homeserver.yaml.j2 b/roles/matrix-synapse/templates/synapse/homeserver.yaml.j2
index a635559f2..ef8684abd 100644
--- a/roles/matrix-synapse/templates/synapse/homeserver.yaml.j2
+++ b/roles/matrix-synapse/templates/synapse/homeserver.yaml.j2
@@ -2874,6 +2874,14 @@ instance_map: {{ matrix_synapse_instance_map | to_json }}
 #  typing: worker1
 stream_writers: {{ matrix_synapse_stream_writers | to_json }}
 
+{% if matrix_synapse_notify_appservices_from_worker != '' %}
+notify_appservices_from_worker: {{ matrix_synapse_notify_appservices_from_worker | to_json }}
+{% endif %}
+
+{% if matrix_synapse_update_user_directory_from_worker != '' %}
+update_user_directory_from_worker: {{ matrix_synapse_update_user_directory_from_worker | to_json }}
+{% endif %}
+
 # The worker that is used to run background tasks (e.g. cleaning up expired
 # data). If not provided this defaults to the main process.
 #
diff --git a/roles/matrix-synapse/vars/main.yml b/roles/matrix-synapse/vars/main.yml
index e8b0e49c6..2dc00feaf 100644
--- a/roles/matrix-synapse/vars/main.yml
+++ b/roles/matrix-synapse/vars/main.yml
@@ -64,6 +64,11 @@ matrix_synapse_workers_stream_writer_receipts_stream_worker_client_server_endpoi
 matrix_synapse_workers_stream_writer_presence_stream_worker_client_server_endpoints:
   - ^/_matrix/client/(api/v1|r0|v3|unstable)/presence/
 
+# matrix_synapse_workers_user_dir_worker_client_server_endpoints contains the endpoints serviced by the `type = user_dir` (`app = generic_worker`) worker.
+# See: https://matrix-org.github.io/synapse/latest/workers.html#updating-the-user-directory
+matrix_synapse_workers_user_dir_worker_client_server_endpoints:
+  - ^/_matrix/client/(r0|v3|unstable)/user_directory/search$
+
 # matrix_synapse_workers_known_stream_writer_stream_types contains the list of stream writer stream types that the playbook recognizes.
 # This is used for validation purposes. If adding support for a new type, besides adding it to this list,
 # don't forget to actually configure it where appropriate (see worker.yaml.j2`, the nginx proxy configuration, etc).
@@ -79,13 +84,25 @@ matrix_synapse_workers_webserving_stream_writer_types: ['typing', 'to_device', '
 matrix_synapse_webserving_workers_systemd_services_list: []
 
 # matrix_synapse_known_worker_types contains the list of known worker types.
+#
 # A worker type is different than a worker app (e.g. `generic_worker`).
 # For example, the `stream_writer` worker type is served by the `generic_worker` app, but is a separate type that we recognize.
+#
+# Some other types (`appservice` and `user_dir`) used to be Synapse worker apps, which got subsequently deprecated.
+# We still allow these types of workers and map them to the `generic_worker` app,
+# which is why we make sure they're part of the list below.
+# We use the `unique` filter because they're part of `matrix_synapse_workers_avail_list` too (for now; scheduled for removal).
 matrix_synapse_known_worker_types: |
   {{
-    matrix_synapse_workers_avail_list
-    +
-    ['stream_writer']
+    (
+      matrix_synapse_workers_avail_list
+      +
+      ['stream_writer']
+      +
+      ['appservice']
+      +
+      ['user_dir']
+    ) | unique
   }}
 
 # matrix_synapse_known_instance_map_eligible_worker_types contains the list of worker types that are to be injected into `matrix_synapse_instance_map`.
diff --git a/roles/matrix-synapse/vars/workers.yml b/roles/matrix-synapse/vars/workers.yml
index 3f34bcb2d..bc49e8354 100644
--- a/roles/matrix-synapse/vars/workers.yml
+++ b/roles/matrix-synapse/vars/workers.yml
@@ -90,7 +90,9 @@ matrix_synapse_workers_generic_worker_endpoints:
 #  - ^/_matrix/client/(api/v1|r0|v3|unstable)/presence/
 
   # User directory search requests
-  - ^/_matrix/client/(r0|v3|unstable)/user_directory/search$
+  # Any worker can handle these, but we have a dedicated user_dir worker for this,
+  # so we'd like for other generic workers to not try and capture these requests.
+  # - ^/_matrix/client/(r0|v3|unstable)/user_directory/search$
 
   # Additionally, the following REST endpoints can be handled for GET requests:
 

From a1fb0826183af614c277430f257bfcc729026d53 Mon Sep 17 00:00:00 2001
From: Slavi Pantaleev <slavi@devture.com>
Date: Thu, 15 Sep 2022 10:32:51 +0300
Subject: [PATCH 07/10] Add support for running background tasks on a worker

---
 CHANGELOG.md                                     | 10 +++++++++-
 roles/matrix-synapse/defaults/main.yml           | 11 +++++++++++
 .../tasks/synapse/workers/init.yml               | 16 ++++++++++++++++
 roles/matrix-synapse/tasks/validate_config.yml   |  1 +
 .../templates/synapse/homeserver.yaml.j2         |  3 +++
 roles/matrix-synapse/vars/main.yml               |  2 ++
 6 files changed, 42 insertions(+), 1 deletion(-)

diff --git a/CHANGELOG.md b/CHANGELOG.md
index 37032735e..06cac7a7d 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -7,6 +7,7 @@ People who are interested in running a Synapse worker setup should know that **o
 - we've added support for [Stream writers](#stream-writers-support)
 - we've added support for [multiple federation sender workers](#multiple-federation-sender-workers-support)
 - we've added support for [multiple pusher workers](#multiple-pusher-workers-support)
+- we've added support for [running background tasks on a worker](#background-tasks-can-run-on-a-worker)
 - we've restored support for [`appservice` workers](#appservice-worker-support-is-back)
 - we've restored support for [`user_dir` workers](#user-directory-worker-support-is-back)
 - see the [Potential Backward Incompatibilities after these Synapse worker changes](#potential-backward-incompatibilities-after-these-synapse-worker-changes)
@@ -40,6 +41,13 @@ From now on, you can have as many as you want to help with your federation traff
 Until now, we only supported a single `pusher` worker (`matrix_synapse_workers_pusher_workers_count` could either be `0` or `1`).
 From now on, you can have as many as you want to help with pushing notifications out.
 
+### Background tasks can run on a worker
+
+From now on, you can put [background task processing on a worker](https://matrix-org.github.io/synapse/latest/workers.html#background-tasks).
+
+With `matrix_synapse_workers_preset: one-of-each`, you'll get one `background` worker automatically.
+You can also control the `background` workers count with `matrix_synapse_workers_background_workers_count`. Only  `0` or `1` workers of this type are supported by Synapse.
+
 ### Appservice worker support is back
 
 We previously had an `appservice` worker type, which [Synapse deprecated in v1.59.0](https://github.com/matrix-org/synapse/blob/v1.59.0/docs/upgrade.md#deprecation-of-the-synapseappappservice-and-synapseappuser_dir-worker-application-types). So did we, at the time.
@@ -66,7 +74,7 @@ Below we'll discuss **potential backward incompatibilities**.
 
 - **Worker names** (container names, systemd services, worker configuration files) **have changed**. Workers are now labeled sequentially (e.g. `matrix-synapse-worker_generic_worker-18111` -> `matrix-synapse-worker-generic-0`). The playbook will handle these changes automatically.
 
-- Due to increased worker types support above, people who use `matrix_synapse_workers_preset: one-of-each` should be aware that with these changes, **the playbook will deploy 8 additional workers** (6 stream writers, 1 `appservice` worker, 1 `user_dir` worker). This **may increase RAM/CPU usage**, etc. If you find your server struggling, consider disabling some workers with the appropriate `matrix_synapse_workers_*_workers_count` variables.
+- Due to increased worker types support above, people who use `matrix_synapse_workers_preset: one-of-each` should be aware that with these changes, **the playbook will deploy 9 additional workers** (6 stream writers, 1 `appservice` worker, 1 `user_dir` worker, 1 background task worker). This **may increase RAM/CPU usage**, etc. If you find your server struggling, consider disabling some workers with the appropriate `matrix_synapse_workers_*_workers_count` variables.
 
 - **Metric endpoints have also changed** (`/metrics/synapse/worker/generic_worker-18111` -> `/metrics/synapse/worker/generic-worker-0`). If you're [collecting metrics to an external Prometheus server](docs/configuring-playbook-prometheus-grafana.md#collecting-metrics-to-an-external-prometheus-server), consider revisiting our [Collecting Synapse worker metrics to an external Prometheus server](docs/configuring-playbook-prometheus-grafana.md#collecting-synapse-worker-metrics-to-an-external-prometheus-server) docs and updating your Prometheus configuration. **If you're collecting metrics to the integrated Prometheus server** (not enabled by default), **your Prometheus configuration will be updated automatically**. Old data (from before this change) may stick around though.
 
diff --git a/roles/matrix-synapse/defaults/main.yml b/roles/matrix-synapse/defaults/main.yml
index 7f6ce40fa..672285cc7 100644
--- a/roles/matrix-synapse/defaults/main.yml
+++ b/roles/matrix-synapse/defaults/main.yml
@@ -398,6 +398,7 @@ matrix_synapse_workers_presets:
     media_repository_workers_count: 0
     appservice_workers_count: 0
     user_dir_workers_count: 0
+    background_workers_count: 0
     stream_writer_events_stream_workers_count: 0
     stream_writer_typing_stream_workers_count: 0
     stream_writer_to_device_stream_workers_count: 0
@@ -411,6 +412,7 @@ matrix_synapse_workers_presets:
     media_repository_workers_count: 1
     appservice_workers_count: 1
     user_dir_workers_count: 1
+    background_workers_count: 1
     stream_writer_events_stream_workers_count: 1
     stream_writer_typing_stream_workers_count: 1
     stream_writer_to_device_stream_workers_count: 1
@@ -554,6 +556,15 @@ matrix_synapse_workers_user_dir_workers_metrics_range_start: 19661
 # `update_user_directory_from_worker` is meant to point to a worker, which is dedicated to updating the user directory and servicing some user directory URL endpoints (`matrix_synapse_workers_user_dir_worker_client_server_endpoints`).
 matrix_synapse_update_user_directory_from_worker: "{{ (matrix_synapse_workers_enabled_list | selectattr('type', 'equalto', 'user_dir') | list)[0].name if (matrix_synapse_workers_enabled and matrix_synapse_workers_enabled_list | selectattr('type', 'equalto', 'user_dir') | list | length > 0) else '' }}"
 
+# matrix_synapse_workers_background_workers_count can only be 0 or 1. More instances are not supported.
+# Our implementation uses a generic worker and assigns Synapse to perform background work on this worker using the `run_background_tasks_on` Synapse option.
+matrix_synapse_workers_background_workers_count: "{{ matrix_synapse_workers_presets[matrix_synapse_workers_preset]['background_workers_count'] }}"
+matrix_synapse_workers_background_workers_metrics_range_start: 19700
+
+# matrix_synapse_run_background_tasks_on populates the `run_background_tasks_on` Synapse configuration used when Synapse workers are in use (`matrix_synapse_workers_enabled`).
+# `run_background_tasks_on` is meant to point to a worker, which is dedicated to processing background tasks.
+matrix_synapse_run_background_tasks_on: "{{ (matrix_synapse_workers_enabled_list | selectattr('type', 'equalto', 'background') | list)[0].name if (matrix_synapse_workers_enabled and matrix_synapse_workers_enabled_list | selectattr('type', 'equalto', 'background') | list | length > 0) else '' }}"
+
 # Default list of workers to spawn.
 #
 # Unless you populate this manually, this list is dynamically generated
diff --git a/roles/matrix-synapse/tasks/synapse/workers/init.yml b/roles/matrix-synapse/tasks/synapse/workers/init.yml
index c5a532979..0c2b8a793 100644
--- a/roles/matrix-synapse/tasks/synapse/workers/init.yml
+++ b/roles/matrix-synapse/tasks/synapse/workers/init.yml
@@ -100,6 +100,20 @@
   register: "matrix_synapse_workers_list_results_user_dir_workers"
   loop: "{{ range(0, matrix_synapse_workers_user_dir_workers_count | int) | list }}"
 
+# This type of worker can only have a count of 1, at most
+- name: Build background workers
+  ansible.builtin.set_fact:
+    worker:
+      id: "background-{{ item }}"
+      name: "matrix-synapse-worker-background-{{ item }}"
+      type: 'background'
+      app: 'generic_worker'
+      webserving: false
+      port: 0
+      metrics_port: "{{ matrix_synapse_workers_background_workers_metrics_range_start + item }}"
+  register: "matrix_synapse_workers_list_results_background_workers"
+  loop: "{{ range(0, matrix_synapse_workers_background_workers_count | int) | list }}"
+
 - name: Build media_repository workers
   ansible.builtin.set_fact:
     worker:
@@ -130,6 +144,8 @@
       matrix_synapse_workers_list_results_user_dir_workers.results
       +
       matrix_synapse_workers_list_results_media_repository_workers.results
+      +
+      matrix_synapse_workers_list_results_background_workers.results
     }}
 
 - ansible.builtin.set_fact:
diff --git a/roles/matrix-synapse/tasks/validate_config.yml b/roles/matrix-synapse/tasks/validate_config.yml
index f6b1fe688..79e581549 100644
--- a/roles/matrix-synapse/tasks/validate_config.yml
+++ b/roles/matrix-synapse/tasks/validate_config.yml
@@ -20,6 +20,7 @@
   with_items:
     - "matrix_synapse_workers_appservice_workers_count"
     - "matrix_synapse_workers_user_dir_workers_count"
+    - "matrix_synapse_workers_background_workers_count"
     - "matrix_synapse_workers_stream_writer_typing_stream_workers_count"
     - "matrix_synapse_workers_stream_writer_to_device_stream_workers_count"
     - "matrix_synapse_workers_stream_writer_account_data_stream_workers_count"
diff --git a/roles/matrix-synapse/templates/synapse/homeserver.yaml.j2 b/roles/matrix-synapse/templates/synapse/homeserver.yaml.j2
index ef8684abd..e330617fd 100644
--- a/roles/matrix-synapse/templates/synapse/homeserver.yaml.j2
+++ b/roles/matrix-synapse/templates/synapse/homeserver.yaml.j2
@@ -2886,6 +2886,9 @@ update_user_directory_from_worker: {{ matrix_synapse_update_user_directory_from_
 # data). If not provided this defaults to the main process.
 #
 #run_background_tasks_on: worker1
+{% if matrix_synapse_run_background_tasks_on != '' %}
+run_background_tasks_on: {{ matrix_synapse_run_background_tasks_on | to_json }}
+{% endif %}
 
 # A shared secret used by the replication APIs to authenticate HTTP requests
 # from workers.
diff --git a/roles/matrix-synapse/vars/main.yml b/roles/matrix-synapse/vars/main.yml
index 2dc00feaf..b403b461b 100644
--- a/roles/matrix-synapse/vars/main.yml
+++ b/roles/matrix-synapse/vars/main.yml
@@ -102,6 +102,8 @@ matrix_synapse_known_worker_types: |
       ['appservice']
       +
       ['user_dir']
+      +
+      ['background']
     ) | unique
   }}
 

From 8887aed500db8f0cc9de30170e9d272d029d14b8 Mon Sep 17 00:00:00 2001
From: Slavi Pantaleev <slavi@devture.com>
Date: Thu, 15 Sep 2022 10:34:02 +0300
Subject: [PATCH 08/10] Do not add federation_sender_instances/pusher_instances
 configuration if not necessary

This keeps the configuration cleaner and hopefully prevents odd issues
if Synapse (by any chance) happens to interpret `pusher_instances: []`
as "no pushers", instead of as "push from the master process".

I haven't confirmed what an empty pushers/federation-senders list means,
so it's safer to just avoid it.
---
 roles/matrix-synapse/templates/synapse/homeserver.yaml.j2 | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/roles/matrix-synapse/templates/synapse/homeserver.yaml.j2 b/roles/matrix-synapse/templates/synapse/homeserver.yaml.j2
index e330617fd..3cd76bf95 100644
--- a/roles/matrix-synapse/templates/synapse/homeserver.yaml.j2
+++ b/roles/matrix-synapse/templates/synapse/homeserver.yaml.j2
@@ -2851,9 +2851,13 @@ send_federation: {{ matrix_synapse_send_federation | to_json }}
 #
 #federation_sender_instances:
 #  - federation_sender1
+{% if matrix_synapse_federation_sender_instances | length > 0 %}
 federation_sender_instances: {{ matrix_synapse_federation_sender_instances | to_json }}
+{% endif %}
 
+{% if matrix_synapse_federation_pusher_instances | length > 0 %}
 pusher_instances: {{ matrix_synapse_federation_pusher_instances | to_json }}
+{% endif %}
 start_pushers: {{ matrix_synapse_start_pushers | to_json }}
 
 # When using workers this should be a map from `worker_name` to the

From 109e1addb882304a4fa31db7ae253a72c1c867a8 Mon Sep 17 00:00:00 2001
From: Slavi Pantaleev <slavi@devture.com>
Date: Thu, 15 Sep 2022 10:45:03 +0300
Subject: [PATCH 09/10] Improve reliability when using more than 1
 media_repository worker

---
 CHANGELOG.md                                              | 8 ++++++++
 roles/matrix-synapse/defaults/main.yml                    | 5 +++++
 roles/matrix-synapse/templates/synapse/homeserver.yaml.j2 | 4 ++++
 3 files changed, 17 insertions(+)

diff --git a/CHANGELOG.md b/CHANGELOG.md
index 06cac7a7d..95356d8cd 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -10,6 +10,7 @@ People who are interested in running a Synapse worker setup should know that **o
 - we've added support for [running background tasks on a worker](#background-tasks-can-run-on-a-worker)
 - we've restored support for [`appservice` workers](#appservice-worker-support-is-back)
 - we've restored support for [`user_dir` workers](#user-directory-worker-support-is-back)
+- we've made it possible to [reliably use more than 1 `media_repository` worker](#using-more-than-1-media-repository-worker-is-now-more-reliable)
 - see the [Potential Backward Incompatibilities after these Synapse worker changes](#potential-backward-incompatibilities-after-these-synapse-worker-changes)
 
 ### Stream writers support
@@ -68,6 +69,13 @@ From now on, we have support for this.
 With `matrix_synapse_workers_preset: one-of-each`, you'll get one `user_dir` worker automatically.
 You can also control the `user_dir` workers count with `matrix_synapse_workers_user_dir_workers_count`. Only  `0` or `1` workers of this type are supported by Synapse.
 
+### Using more than 1 media repository worker is now more reliable
+
+With `matrix_synapse_workers_preset: one-of-each`, we only launch one `media_repository` worker.
+
+If you've been configuring `matrix_synapse_workers_media_repository_workers_count` manually, you may have increased that to more workers.
+When multiple media repository workers are in use, background tasks related to the media repository must always be configured to run on a single `media_repository` worker via `media_instance_running_background_jobs`. Until now, we weren't doing this correctly, but we now are.
+
 ### Potential Backward Incompatibilities after these Synapse worker changes
 
 Below we'll discuss **potential backward incompatibilities**.
diff --git a/roles/matrix-synapse/defaults/main.yml b/roles/matrix-synapse/defaults/main.yml
index 672285cc7..dab098fb0 100644
--- a/roles/matrix-synapse/defaults/main.yml
+++ b/roles/matrix-synapse/defaults/main.yml
@@ -535,6 +535,11 @@ matrix_synapse_workers_media_repository_workers_metrics_range_start: 19551
 # Adjusting this value manually is generally not necessary.
 matrix_synapse_enable_media_repo: "{{ not matrix_synapse_workers_enabled or (matrix_synapse_workers_enabled_list | selectattr('type', 'equalto', 'media_repository') | list | length > 0) }}"
 
+# matrix_synapse_media_instance_running_background_jobs populates the `media_instance_running_background_jobs` Synapse configuration used when Synapse workers are in use (`matrix_synapse_workers_enabled`).
+# `media_instance_running_background_jobs` is meant to point to a single media-repository worker, which is dedicated to running background tasks that maintain the media repository.
+# Multiple `media_repository` workers may be enabled. We always pick the first one as the background tasks worker.
+matrix_synapse_media_instance_running_background_jobs: "{{ (matrix_synapse_workers_enabled_list | selectattr('type', 'equalto', 'media_repository') | list)[0].name if (matrix_synapse_workers_enabled and matrix_synapse_workers_enabled_list | selectattr('type', 'equalto', 'media_repository') | list | length > 0) else '' }}"
+
 # matrix_synapse_workers_appservice_workers_count can only be 0 or 1. More instances are not supported.
 # appservice workers were deprecated since Synapse v1.59 (see: https://github.com/matrix-org/synapse/blob/v1.59.0/docs/upgrade.md#deprecation-of-the-synapseappappservice-and-synapseappuser_dir-worker-application-types).
 # Our implementation uses generic worker services and assigns them to perform appservice work using the `notify_appservices_from_worker` Synapse option.
diff --git a/roles/matrix-synapse/templates/synapse/homeserver.yaml.j2 b/roles/matrix-synapse/templates/synapse/homeserver.yaml.j2
index 3cd76bf95..ae71b7aec 100644
--- a/roles/matrix-synapse/templates/synapse/homeserver.yaml.j2
+++ b/roles/matrix-synapse/templates/synapse/homeserver.yaml.j2
@@ -2894,6 +2894,10 @@ update_user_directory_from_worker: {{ matrix_synapse_update_user_directory_from_
 run_background_tasks_on: {{ matrix_synapse_run_background_tasks_on | to_json }}
 {% endif %}
 
+{% if matrix_synapse_media_instance_running_background_jobs != '' %}
+media_instance_running_background_jobs: {{ matrix_synapse_media_instance_running_background_jobs | to_json }}
+{% endif %}
+
 # A shared secret used by the replication APIs to authenticate HTTP requests
 # from workers.
 #

From ef112181a1b4dae89642e2189da610244f03602f Mon Sep 17 00:00:00 2001
From: Slavi Pantaleev <slavi@devture.com>
Date: Thu, 15 Sep 2022 10:46:17 +0300
Subject: [PATCH 10/10] Update some Synapse documentation URLs

---
 .../templates/nginx/conf.d/matrix-synapse.conf.j2         | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/roles/matrix-nginx-proxy/templates/nginx/conf.d/matrix-synapse.conf.j2 b/roles/matrix-nginx-proxy/templates/nginx/conf.d/matrix-synapse.conf.j2
index 488ced5a8..1d6f2106a 100644
--- a/roles/matrix-nginx-proxy/templates/nginx/conf.d/matrix-synapse.conf.j2
+++ b/roles/matrix-nginx-proxy/templates/nginx/conf.d/matrix-synapse.conf.j2
@@ -81,7 +81,7 @@ server {
 		{# Workers redirects BEGIN #}
 
 		{% if generic_workers | length > 0 %}
-			# https://github.com/matrix-org/synapse/blob/master/docs/workers.md#synapseappgeneric_worker
+			# https://matrix-org.github.io/synapse/latest/workers.html#synapseappgeneric_worker
 			{{ render_locations_to_upstream(matrix_nginx_proxy_synapse_generic_worker_client_server_locations, 'generic_workers_upstream') }}
 		{% endif %}
 
@@ -111,7 +111,7 @@ server {
 		{% endif %}
 
 		{% if media_repository_workers | length > 0 %}
-			# https://github.com/matrix-org/synapse/blob/master/docs/workers.md#synapseappmedia_repository
+			# https://matrix-org.github.io/synapse/latest/workers.html#synapseappmedia_repository
 			{% for location in matrix_nginx_proxy_synapse_media_repository_locations %}
 			location ~ {{ location }} {
 				proxy_pass http://media_repository_workers_upstream$request_uri;
@@ -182,11 +182,11 @@ server {
 
 	{% if matrix_nginx_proxy_synapse_workers_enabled %}
 		{% if generic_workers | length > 0 %}
-			# https://github.com/matrix-org/synapse/blob/master/docs/workers.md#synapseappgeneric_worker
+			# https://matrix-org.github.io/synapse/latest/workers.html#synapseappgeneric_worker
 			{{ render_locations_to_upstream(matrix_nginx_proxy_synapse_generic_worker_federation_locations, 'generic_workers_upstream') }}
 		{% endif %}
 		{% if media_repository_workers | length > 0 %}
-			# https://github.com/matrix-org/synapse/blob/master/docs/workers.md#synapseappmedia_repository
+			# https://matrix-org.github.io/synapse/latest/workers.html#synapseappmedia_repository
 			{% for location in matrix_nginx_proxy_synapse_media_repository_locations %}
 			location ~ {{ location }} {
 				proxy_pass http://media_repository_workers_upstream$request_uri;