matrix-docker-ansible-deploy/docs/maintenance-synapse.md

105 lines
7.5 KiB
Markdown
Raw Normal View History

# Synapse maintenance
This document shows you how to perform various maintenance tasks related to the Synapse chat server.
Table of contents:
- [Purging old data with the Purge History API](#purging-old-data-with-the-purge-history-api), for when you wish to delete in-use (but old) data from the Synapse database
- [Compressing state with rust-synapse-compress-state](#compressing-state-with-rust-synapse-compress-state)
- [Browse and manipulate the database](#browse-and-manipulate-the-database), for when you really need to take matters into your own hands
- [Make Synapse faster](#make-synapse-faster)
## Purging old data with the Purge History API
You can use the **[Purge History API](https://github.com/element-hq/synapse/blob/master/docs/admin_api/purge_history_api.md)** to delete old messages on a per-room basis. **This is destructive** (especially for non-federated rooms), because it means **people will no longer have access to history past a certain point**.
To make use of this Synapse Admin API, **you'll need an admin access token** first. Refer to the documentation on [how to obtain an access token](obtaining-access-tokens.md).
Synapse's Admin API is not exposed to the internet by default, following [official Synapse reverse-proxying recommendations](https://github.com/element-hq/synapse/blob/master/docs/reverse_proxy.md#synapse-administration-endpoints). To expose it you will need to add `matrix_synapse_container_labels_public_client_synapse_admin_api_enabled: true` to your `vars.yml` file.
2021-05-08 22:07:54 +02:00
Follow the [Purge History API](https://github.com/element-hq/synapse/blob/master/docs/admin_api/purge_history_api.md) documentation page for the actual purging instructions.
After deleting data, you may wish to run a [`FULL` Postgres `VACUUM`](./maintenance-postgres.md#vacuuming-postgresql).
## Compressing state with rust-synapse-compress-state
2021-05-08 22:07:54 +02:00
[rust-synapse-compress-state](https://github.com/matrix-org/rust-synapse-compress-state) can be used to optimize some `_state` tables used by Synapse. If your server participates in large rooms this is the most effective way to reduce the size of your database.
**Note**: besides running the `rust-synapse-compress-state` tool manually, you can also enable its `synapse-auto-compressor` tool by [Setting up synapse-auto-compressor](configuring-playbook-synapse-auto-compressor.md). The automatic tool will run on a schedule every day and you won't have to compress state manually ever again.
`rust-synapse-compress-state` should be safe to use (even when Synapse is running), but it's always a good idea to [make Postgres backups](./maintenance-postgres.md#backing-up-postgresql) first.
To ask the playbook to run rust-synapse-compress-state, execute:
```sh
ansible-playbook -i inventory/hosts setup.yml --tags=rust-synapse-compress-state
```
Edit descriptions about installation of components (#3842) * Replace installation command shortcut for the "just" program with the most conservative raw ansible-playbook command This commit replaces installation command shortcut ("recipe") for the "just" program with the raw ansible-playbook command, so that the shortcut will be added to it later. The command is so conservative that failure of the command will mean something is clearly broken. Signed-off-by: Suguru Hirahara <acioustick@noreply.codeberg.org> * Add comments about using setup-all instead of install-all Signed-off-by: Suguru Hirahara <acioustick@noreply.codeberg.org> * Add description about shortcut command with the "just" program to the ansible-playbook command with "setup-all" and "start" tags It also explains difference between "just install-all" and "just setup-all" recipes. The explanation is based on docs/playbook-tags.md Signed-off-by: Suguru Hirahara <acioustick@noreply.codeberg.org> * Update raw ansible-playbook command to have it do what "just install-all" or "just setup-all" does Since "just install-all" or "just setup-all" invokes "ensure-matrix-users-created" as well, it needs adding to the raw ansible-playbook command. Signed-off-by: Suguru Hirahara <acioustick@noreply.codeberg.org> * Remove "ensure-matrix-users-created" from the raw ansible-playbook command which does not need it Also: update the "just" recipes accordingly. "just install-all" and "just setup-all" run "ensure-matrix-users-created" tag as well, therefore they need to be replaced with "run-tags" recipes to skip "ensure-matrix-users-created" Signed-off-by: Suguru Hirahara <acioustick@noreply.codeberg.org> * Update docs/configuring-playbook-etherpad.md: add ensure-matrix-users-created to the raw ansible-playbook Signed-off-by: Suguru Hirahara <acioustick@noreply.codeberg.org> * Add description about "ensure-matrix-users-created" and create a list with description about shortcut commands with "just" This commit also fixes list item capitalization and punctuation. Signed-off-by: Suguru Hirahara <acioustick@noreply.codeberg.org> * Add notes bullet lists Signed-off-by: Suguru Hirahara <acioustick@noreply.codeberg.org> * Update docs/configuring-playbook-matrix-corporal.md and docs/configuring-playbook-email2matrix.md: adopt common instructions Signed-off-by: Suguru Hirahara <acioustick@noreply.codeberg.org> * Replace "run the installation command" with "run the playbook with tags" Now that shortcut commands for the "just" program are displayed along with the existing "installation command", this commit replaces "run the installation command" with "run the playbook with tags" in order to prevent misunderstanding and confusion. Signed-off-by: Suguru Hirahara <acioustick@noreply.codeberg.org> * Add notes about changing passwords of users specified on vars.yml Signed-off-by: Suguru Hirahara <acioustick@noreply.codeberg.org> * Update docs/configuring-playbook-synapse-admin.md: add the playbook command and just recipes Signed-off-by: Suguru Hirahara <acioustick@noreply.codeberg.org> * Remove redundant blank lines Signed-off-by: Suguru Hirahara <acioustick@noreply.codeberg.org> * Update docs/configuring-playbook-alertmanager-receiver.md: remove the direction to proceed to Usage Such a kind of direction is not used on other documentation, so it should be fine to just remove it. Signed-off-by: Suguru Hirahara <acioustick@noreply.codeberg.org> * Update docs/importing-synapse-media-store.md: code block for ansible-playbook Signed-off-by: Suguru Hirahara <acioustick@noreply.codeberg.org> --------- Signed-off-by: Suguru Hirahara <acioustick@noreply.codeberg.org> Co-authored-by: Suguru Hirahara <acioustick@noreply.codeberg.org>
2024-12-01 08:42:30 +01:00
The shortcut command with `just` program is also available: `just run-tags rust-synapse-compress-state`
By default, all rooms with more than `100000` state group rows will be compressed. If you need to adjust this, pass: `--extra-vars='matrix_synapse_rust_synapse_compress_state_min_state_groups_required=SOME_NUMBER_HERE'` to the command above.
After state compression, you may wish to run a [`FULL` Postgres `VACUUM`](./maintenance-postgres.md#vacuuming-postgresql).
## Browse and manipulate the database
When the [Synapse Admin API](https://github.com/element-hq/synapse/tree/master/docs/admin_api) and the other tools do not provide a more convenient way, having a look at synapse's postgresql database can satisfy a lot of admins' needs.
2021-05-08 22:07:54 +02:00
Editing the database manually is not recommended or supported by the Synapse developers. If you are going to do so you should [make a database backup](./maintenance-postgres.md#backing-up-postgresql).
First, set up an SSH tunnel to your Matrix server (skip if it is your local machine):
```sh
# you may replace 1799 with an arbitrary port unbound on both machines
ssh -L 1799:localhost:1799 matrix.example.com
```
2020-05-11 10:09:22 +02:00
Then start up an ephemeral [adminer](https://www.adminer.org/) container on the Matrix server, connecting it to the `matrix` network and linking the postgresql container:
```sh
docker run --rm --publish 1799:8080 --link matrix-postgres --net matrix adminer
```
You should then be able to browse the adminer database administration GUI at http://localhost:1799/ after entering your DB credentials (found in the `host_vars` or on the server in `{{matrix_synapse_config_dir_path}}/homeserver.yaml` under `database.args`)
⚠️️ Be **very careful** with this, there is **no undo** for impromptu DB operations.
## Make Synapse faster
2021-04-22 13:22:07 +02:00
Synapse's presence feature which tracks which users are online and which are offline can use a lot of processing power. You can disable presence by adding `matrix_synapse_presence_enabled: false` to your `vars.yml` file.
If you have enough compute resources (CPU & RAM), you can make Synapse better use of them by [enabling load-balancing with workers](configuring-playbook-synapse.md#load-balancing-with-workers).
2024-05-10 15:11:13 +02:00
[Tuning your PostgreSQL database](maintenance-postgres.md#tuning-postgresql) could also improve Synapse performance. The playbook tunes the integrated Postgres database automatically, but based on your needs you may wish to adjust tuning variables manually. If you're using an [external Postgres database](configuring-playbook-external-postgres.md), you will also need to tune Postgres manually.
Increase Synapse caches and enable cache-autotuning by default (#3017) * Modify Synapse Cache Factor to use Auto Tune Synapse has the ability to as it calls in its config auto tune caches. This ability lets us set very high cache factors and then instead limit our resource use. Defaults for this commit are 1/10th of what Element apparently runs for EMS stuff and matrix.org on Cache Factor and upstream documentation defaults for auto tune. * Add vars to Synapse main.yml to control cache related config This commit adds various cache related vars to main.yml for Synapse. Some are auto tune and some are just adding explicit ways to control upstream vars. * Updated Auto Tune figures Autotuned figures have been bumped in consultation with other community members as to a reasonable level. Please note these defaults are more on the one of each workers side than they are on the monolith Side. * Fix YML Error The playbook is not happy with the previous state of this patch so this commit hopefully fixes it * Add to_json to various Synapse tuning related configs * Fix incorrect indication in homeserver.yaml.j2 * Minor cleanups * Synapse Cache Autotuning Documentation * Upgrade Synapse Cache Autotune to auto configure memory use * Update Synapse Tuning docs to reflect automatic memory use configuration * Fix Linting errors in synapses main.yml * Rename variables for consistency (matrix_synapse_caches_autotuning_* -> matrix_synapse_cache_autotuning_*) * Remove FIX ME comment about Synapse's `cache_autotuning` `docs/maintenance-synapse.md` and `roles/custom/matrix-synapse/defaults/main.yml` already contains documentation about these variables and the default values we set. * Improve "Tuning caches and cache autotuning" documentation for Synapse * Announce larger Synapse caches and cache auto-tuning --------- Co-authored-by: Slavi Pantaleev <slavi@devture.com>
2024-02-14 11:02:06 +01:00
### Tuning caches and cache autotuning
Tuning Synapse's cache factor is useful for performance increases but also as part of controlling Synapse's memory use. Use the variable `matrix_synapse_caches_global_factor` to set the cache factor as part of this process.
**The playbook defaults the global cache factor to a large value** (e.g. `10`). A smaller value (e.g. `0.5`) will decrease the amount used for caches, but will [not necessarily decrease RAM usage as a whole](https://github.com/matrix-org/synapse/issues/3939).
Tuning the cache factor is useful only to a limited degree (as its crude to do in isolation) and therefore users who are tuning their cache factor should likely look into tuning autotune variables as well (see below).
Cache autotuning is **enabled by default** and controlled via the following variables:
Fix Synapse cache auto-tuning variables to use bytes, not KB Fixup for https://github.com/spantaleev/matrix-docker-ansible-deploy/pull/3017 This reverts 1cd82cf06823fb7680 and also multiplies results by `1024` so as to pass bytes to Synapse, not KB (as done before). 1cd82cf06823fb7680 was correctly documenting what we were doing (passing KB values), but that's incorrect. Synapse's Config Conventions (https://matrix-org.github.io/synapse/latest/usage/configuration/config_documentation.html#config-conventions) are supposed to clear it up, but they don't currently state what happens when you pass a plain number (without a unit suffix). Thankfully, the source code tells us: https://github.com/element-hq/synapse/blob/bc1db16086d0718c9c0bb61b32b825ba62049bb0/synapse/config/_base.py#L181-L206 > If an integer is provided it is treated as bytes and is unchanged. > > String byte sizes can have a suffix of ... > No suffix is understood as a plain byte count. We were previously passing strings, but that has been improved in 3d73ec887aec27483. Regardless, non-suffixed values seem to be treated as bytes by Synapse, so this patch changes the variables to use bytes. Moreover, we're moving from `matrix_synapse_memtotal_kb` to `matrix_synapse_cache_size_calculations_memtotal_bytes` as working with the base unit everywhere is preferrable. Here, we also introduce 2 new variables to allow for the caps to be tweaked: - `matrix_synapse_cache_size_calculations_max_cache_memory_usage_cap_bytes` - `matrix_synapse_cache_size_calculations_target_cache_memory_usage_cap_bytes`
2024-02-14 12:39:40 +01:00
- `matrix_synapse_cache_autotuning_max_cache_memory_usage` - defaults to 1/8 of total RAM with a cap of 2GB; values are specified in bytes
- `matrix_synapse_cache_autotuning_target_cache_memory_usage` - defaults to 1/16 of total RAM with a cap of 1GB; values are specified in bytes
Increase Synapse caches and enable cache-autotuning by default (#3017) * Modify Synapse Cache Factor to use Auto Tune Synapse has the ability to as it calls in its config auto tune caches. This ability lets us set very high cache factors and then instead limit our resource use. Defaults for this commit are 1/10th of what Element apparently runs for EMS stuff and matrix.org on Cache Factor and upstream documentation defaults for auto tune. * Add vars to Synapse main.yml to control cache related config This commit adds various cache related vars to main.yml for Synapse. Some are auto tune and some are just adding explicit ways to control upstream vars. * Updated Auto Tune figures Autotuned figures have been bumped in consultation with other community members as to a reasonable level. Please note these defaults are more on the one of each workers side than they are on the monolith Side. * Fix YML Error The playbook is not happy with the previous state of this patch so this commit hopefully fixes it * Add to_json to various Synapse tuning related configs * Fix incorrect indication in homeserver.yaml.j2 * Minor cleanups * Synapse Cache Autotuning Documentation * Upgrade Synapse Cache Autotune to auto configure memory use * Update Synapse Tuning docs to reflect automatic memory use configuration * Fix Linting errors in synapses main.yml * Rename variables for consistency (matrix_synapse_caches_autotuning_* -> matrix_synapse_cache_autotuning_*) * Remove FIX ME comment about Synapse's `cache_autotuning` `docs/maintenance-synapse.md` and `roles/custom/matrix-synapse/defaults/main.yml` already contains documentation about these variables and the default values we set. * Improve "Tuning caches and cache autotuning" documentation for Synapse * Announce larger Synapse caches and cache auto-tuning --------- Co-authored-by: Slavi Pantaleev <slavi@devture.com>
2024-02-14 11:02:06 +01:00
- `matrix_synapse_cache_autotuning_min_cache_ttl` - defaults to `30s`
You can **learn more about cache-autotuning and the global cache factor settings** in the [Synapse's documentation on caches and associated values](https://matrix-org.github.io/synapse/latest/usage/configuration/config_documentation.html#caches-and-associated-values).
To **disable cache auto-tuning**, unset all values:
```yaml
Increase Synapse caches and enable cache-autotuning by default (#3017) * Modify Synapse Cache Factor to use Auto Tune Synapse has the ability to as it calls in its config auto tune caches. This ability lets us set very high cache factors and then instead limit our resource use. Defaults for this commit are 1/10th of what Element apparently runs for EMS stuff and matrix.org on Cache Factor and upstream documentation defaults for auto tune. * Add vars to Synapse main.yml to control cache related config This commit adds various cache related vars to main.yml for Synapse. Some are auto tune and some are just adding explicit ways to control upstream vars. * Updated Auto Tune figures Autotuned figures have been bumped in consultation with other community members as to a reasonable level. Please note these defaults are more on the one of each workers side than they are on the monolith Side. * Fix YML Error The playbook is not happy with the previous state of this patch so this commit hopefully fixes it * Add to_json to various Synapse tuning related configs * Fix incorrect indication in homeserver.yaml.j2 * Minor cleanups * Synapse Cache Autotuning Documentation * Upgrade Synapse Cache Autotune to auto configure memory use * Update Synapse Tuning docs to reflect automatic memory use configuration * Fix Linting errors in synapses main.yml * Rename variables for consistency (matrix_synapse_caches_autotuning_* -> matrix_synapse_cache_autotuning_*) * Remove FIX ME comment about Synapse's `cache_autotuning` `docs/maintenance-synapse.md` and `roles/custom/matrix-synapse/defaults/main.yml` already contains documentation about these variables and the default values we set. * Improve "Tuning caches and cache autotuning" documentation for Synapse * Announce larger Synapse caches and cache auto-tuning --------- Co-authored-by: Slavi Pantaleev <slavi@devture.com>
2024-02-14 11:02:06 +01:00
matrix_synapse_cache_autotuning_max_cache_memory_usage: ''
matrix_synapse_cache_autotuning_target_cache_memory_usage: ''
matrix_synapse_cache_autotuning_min_cache_ttl: ''
```
Users who wish to lower Synapse's RAM footprint should look into lowering the global cache factor and tweaking the autotune variables (or disabling auto-tuning). If your cache factor is too low for a given auto tune setting your caches will not reach autotune thresholds and autotune won't be able to do its job. Therefore, when auto-tuning is enabled (which it is by default), it's recommended to have your cache factor be large.
See also [How do I optimize this setup for a low-power server?](faq.md#how-do-i-optimize-this-setup-for-a-low-power-server).