mirror/matrix-docker-ansible-deploy
1
0
Fork 0
mirror of https://github.com/spantaleev/matrix-docker-ansible-deploy.git synced 2025-01-16 05:03:10 +01:00
Code Issues Packages Projects Releases Wiki Activity

Update docs/configuring-playbook-synapse-s3-storage-provider.md (#3932)

Browse Source 
* Update docs/configuring-playbook-synapse-s3-storage-provider.md: adopt the common instruction

Signed-off-by: Suguru Hirahara <acioustick@noreply.codeberg.org>

* Update docs/configuring-playbook-synapse-s3-storage-provider.md: add the sections "Extending the configuration" and "Usage"

Signed-off-by: Suguru Hirahara <acioustick@noreply.codeberg.org>

* Update docs/configuring-playbook-synapse-s3-storage-provider.md: minor changes

Signed-off-by: Suguru Hirahara <acioustick@noreply.codeberg.org>

---------

Signed-off-by: Suguru Hirahara <acioustick@noreply.codeberg.org>
Co-authored-by: Suguru Hirahara <acioustick@noreply.codeberg.org>
This commit is contained in:
Suguru Hirahara 2025-01-12 20:19:08 +09:00 committed by GitHub
parent 031c83b690
commit 04cb2f8fa5
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
1 changed files with 20 additions and 12 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

32
docs/configuring-playbook-synapse-s3-storage-provider.md
View File

@ -23,7 +23,7 @@ While you will need some local disk space around, it's only to accommodate usage
## Adjusting the playbook configuration ## Adjusting the playbook configuration
After [creating the S3 bucket and configuring it](configuring-playbook-s3.md#bucket-creation-and-security-configuration), you can proceed to configure `s3-storage-provider` in your configuration file (`inventory/host_vars/matrix.example.com/vars.yml`): After [creating the S3 bucket and configuring it](configuring-playbook-s3.md#bucket-creation-and-security-configuration), add the following configuration to your `inventory/host_vars/matrix.example.com/vars.yml` file:
```yaml ```yaml
matrix_synapse_ext_synapse_s3_storage_provider_enabled: true matrix_synapse_ext_synapse_s3_storage_provider_enabled: true
@ -43,10 +43,18 @@ matrix_synapse_ext_synapse_s3_storage_provider_config_storage_class: STANDARD #
# This only works on AWS when your server is hosted on an EC2 instance with the correct instance profile set. # This only works on AWS when your server is hosted on an EC2 instance with the correct instance profile set.
# Uncomment the variable below to use it. # Uncomment the variable below to use it.
# matrix_synapse_ext_synapse_s3_storage_provider_config_ec2_instance_profile: true # matrix_synapse_ext_synapse_s3_storage_provider_config_ec2_instance_profile: true
# For additional advanced settings, take a look at `roles/custom/matrix-synapse/defaults/main.yml`
``` ```
### Extending the configuration
There are some additional things you may wish to configure about the server.
Take a look at:
- `roles/custom/matrix-synapse/defaults/main.yml` for some variables that you can customize via your `vars.yml` file
## Usage
If you have existing files in Synapse's media repository (`/matrix/synapse/storage/media-store/…`): If you have existing files in Synapse's media repository (`/matrix/synapse/storage/media-store/…`):
- new files will start being stored both locally and on the S3 store - new files will start being stored both locally and on the S3 store
@ -55,16 +63,16 @@ If you have existing files in Synapse's media repository (`/matrix/synapse/stora
Regardless of whether you need to [Migrate your existing files to the S3 store](#migrating-your-existing-media-files-to-the-s3-store) or not, make sure you've familiarized yourself with [How it works?](#how-it-works) above and [Periodically cleaning up the local filesystem](#periodically-cleaning-up-the-local-filesystem) below. Regardless of whether you need to [Migrate your existing files to the S3 store](#migrating-your-existing-media-files-to-the-s3-store) or not, make sure you've familiarized yourself with [How it works?](#how-it-works) above and [Periodically cleaning up the local filesystem](#periodically-cleaning-up-the-local-filesystem) below.
## Migrating your existing media files to the S3 store ### Migrating your existing media files to the S3 store
Migrating your existing data can happen in multiple ways: Migrating your existing data can happen in multiple ways:
- [using the `s3_media_upload` script from `synapse-s3-storage-provider`](#using-the-s3_media_upload-script-from-synapse-s3-storage-provider) (very slow when dealing with lots of data) - [using the `s3_media_upload` script from `synapse-s3-storage-provider`](#using-the-s3_media_upload-script-from-synapse-s3-storage-provider) (very slow when dealing with lots of data)
- [using another tool in combination with `s3_media_upload`](#using-another-tool-in-combination-with-s3_media_upload) (quicker when dealing with lots of data) - [using another tool in combination with `s3_media_upload`](#using-another-tool-in-combination-with-s3_media_upload) (quicker when dealing with lots of data)
### Using the `s3_media_upload` script from `synapse-s3-storage-provider` 💡 **Note**: instead of using `s3_media_upload` directly, which is very slow and painful for an initial data migration, we recommend [using another tool in combination with `s3_media_upload`](#using-another-tool-in-combination-with-s3_media_upload).
Instead of using `s3_media_upload` directly, which is very slow and painful for an initial data migration, we recommend [using another tool in combination with `s3_media_upload`](#using-another-tool-in-combination-with-s3_media_upload). #### Using the `s3_media_upload` script from `synapse-s3-storage-provider`
To copy your existing files, SSH into the server and run `/matrix/synapse/ext/s3-storage-provider/bin/shell`. To copy your existing files, SSH into the server and run `/matrix/synapse/ext/s3-storage-provider/bin/shell`.
@ -87,7 +95,7 @@ Instead of running the above commands manually in the shell, you can also run th
- includes progress bars and detailed output for each command - includes progress bars and detailed output for each command
- allows you to easily interrupt slow-running commands, etc. (the `/matrix/synapse/ext/s3-storage-provider/bin/migrate` starts a container without interactive TTY support, so `Ctrl+C` may not work and you and require killing via `docker kill …`) - allows you to easily interrupt slow-running commands, etc. (the `/matrix/synapse/ext/s3-storage-provider/bin/migrate` starts a container without interactive TTY support, so `Ctrl+C` may not work and you and require killing via `docker kill …`)
### Using another tool in combination with `s3_media_upload` #### Using another tool in combination with `s3_media_upload`
To migrate your existing local data to S3, we recommend to: To migrate your existing local data to S3, we recommend to:
@ -95,7 +103,7 @@ To migrate your existing local data to S3, we recommend to:
- **only then** [use the `s3_media_upload` tool to finish the migration](#using-the-s3_media_upload-script-from-synapse-s3-storage-provider) (this checks to ensure all files are uploaded and then deletes the local files) - **only then** [use the `s3_media_upload` tool to finish the migration](#using-the-s3_media_upload-script-from-synapse-s3-storage-provider) (this checks to ensure all files are uploaded and then deletes the local files)
#### Copying data to Amazon S3 ##### Copying data to Amazon S3
To copy to AWS S3, start a container on the Matrix server like this: To copy to AWS S3, start a container on the Matrix server like this:
@ -109,13 +117,13 @@ docker.io/amazon/aws-cli:2.9.16 \
-c 'aws s3 sync /work/. s3://$BUCKET/' -c 'aws s3 sync /work/. s3://$BUCKET/'
``` ```
#### Copying data to an S3 alternative using the aws-s3 tool ##### Copying data to an S3 alternative using the aws-s3 tool
To copy to a provider other than AWS S3 (e.g. Wasabi, Digital Ocean Spaces, etc.), you can use the command for [Copying data to Amazon S3](#copying-data-to-amazon-s3) with an added `--endpoint-url=$ENDPOINT` argument. To copy to a provider other than AWS S3 (e.g. Storj, Wasabi, Digital Ocean Spaces, etc.), you can use the command for [Copying data to Amazon S3](#copying-data-to-amazon-s3) with an added `--endpoint-url=$ENDPOINT` argument.
Add this argument to the command **as-is** (`$ENDPOINT` is an environment variable corresponding to `matrix_synapse_ext_synapse_s3_storage_provider_config_endpoint_url`, so you don't need to touch it). Make sure to add the argument **before** the final quote (`'`) of the command. Add this argument to the command **as-is** (`$ENDPOINT` is an environment variable corresponding to `matrix_synapse_ext_synapse_s3_storage_provider_config_endpoint_url`, so you don't need to touch it). Make sure to add the argument **before** the final quote (`'`) of the command.
#### Copying data to Backblaze B2 ##### Copying data to Backblaze B2
You can copy files to Backblaze B2 either by following the [Copying data to an S3 alternative using the aws-s3 tool](#copying-data-to-an-s3-alternative-using-the-aws-s3-tool) or by using the B2-specific [b2 command-line tool](https://www.backblaze.com/b2/docs/quick_command_line.html) as described below. You can copy files to Backblaze B2 either by following the [Copying data to an S3 alternative using the aws-s3 tool](#copying-data-to-an-s3-alternative-using-the-aws-s3-tool) or by using the B2-specific [b2 command-line tool](https://www.backblaze.com/b2/docs/quick_command_line.html) as described below.
@ -133,7 +141,7 @@ docker.io/tianon/backblaze-b2:3.6.0 \
-c 'b2 authorize-account $B2_KEY_ID $B2_KEY_SECRET && b2 sync /work b2://$B2_BUCKET_NAME --skipNewer' -c 'b2 authorize-account $B2_KEY_ID $B2_KEY_SECRET && b2 sync /work b2://$B2_BUCKET_NAME --skipNewer'
``` ```
## Periodically cleaning up the local filesystem ### Periodically cleaning up the local filesystem
As described in [How it works?](#how-it-works) above, when new media is uploaded to the Synapse homeserver, it's first stored locally and then also stored on the remote S3 storage. As described in [How it works?](#how-it-works) above, when new media is uploaded to the Synapse homeserver, it's first stored locally and then also stored on the remote S3 storage.