2022-10-14 16:41:41 +02:00
# Storing Matrix media files on Amazon S3 with Goofys (optional)
2024-10-18 19:32:11 +02:00
If you'd like to store Synapse's content repository (`media_store`) files on Amazon S3 (or other S3-compatible service), you can let this playbook configure [Goofys ](https://github.com/kahing/goofys ) for you.
2022-10-14 16:41:41 +02:00
Another (and better performing) way to use S3 storage with Synapse is [synapse-s3-storage-provider ](configuring-playbook-synapse-s3-storage-provider.md ).
Using a Goofys-backed media store works, but performance may not be ideal. If possible, try to use a region which is close to your Matrix server.
If you'd like to move your locally-stored media store data to Amazon S3 (or another S3-compatible object store), we also provide some migration instructions below.
2024-10-12 13:48:24 +02:00
## Adjusting the playbook configuration
2022-10-14 16:41:41 +02:00
2024-10-17 15:17:56 +02:00
After [creating the S3 bucket and configuring it ](configuring-playbook-s3.md#bucket-creation-and-security-configuration ), add the following configuration to your `inventory/host_vars/matrix.example.com/vars.yml` file (adapt to your needs):
2022-10-14 16:41:41 +02:00
```yaml
matrix_s3_media_store_enabled: true
matrix_s3_media_store_bucket_name: "your-bucket-name"
matrix_s3_media_store_aws_access_key: "access-key-goes-here"
matrix_s3_media_store_aws_secret_key: "secret-key-goes-here"
matrix_s3_media_store_region: "eu-central-1"
```
You can use any S3-compatible object store by **additionally** configuring these variables:
```yaml
matrix_s3_media_store_custom_endpoint_enabled: true
matrix_s3_media_store_custom_endpoint: "https://your-custom-endpoint"
```
If you have local media store files and wish to migrate to Backblaze B2 subsequently, follow our [migration guide to Backblaze B2 ](#migrating-to-backblaze-b2 ) below instead of applying this configuration as-is.
## Migrating from local filesystem storage to S3
It's a good idea to [make a complete server backup ](faq.md#how-do-i-backup-the-data-on-my-server ) before migrating your local media store to an S3-backed one.
2024-10-24 20:20:38 +02:00
After making the backup, follow one of the guides below for a migration path from a locally-stored media store to one stored on S3-compatible storage:
2022-10-14 16:41:41 +02:00
2024-10-24 20:14:53 +02:00
- [Migrating to any S3-compatible storage (universal, but likely slow) ](#migrating-to-any-s3-compatible-storage-universal-but-likely-slow )
- [Migrating to Backblaze B2 ](#migrating-to-backblaze-b2 )
2022-10-14 16:41:41 +02:00
### Migrating to any S3-compatible storage (universal, but likely slow)
1. Proceed with the steps below without stopping Matrix services
2. Start by adding the base S3 configuration in your `vars.yml` file (seen above, may be different depending on the S3 provider of your choice)
3. In addition to the base configuration you see above, add this to your `vars.yml` file:
2024-10-24 19:30:02 +02:00
```yaml
matrix_s3_media_store_path: /matrix/s3-media-store
```
2022-10-14 16:41:41 +02:00
2024-10-24 19:30:02 +02:00
This enables S3 support, but mounts the S3 storage bucket to `/matrix/s3-media-store` without hooking it to your homeserver yet. Your homeserver will still continue using your local filesystem for its media store.
2022-10-14 16:41:41 +02:00
2024-10-19 14:33:49 +02:00
4. Run the playbook to apply the changes: `ansible-playbook -i inventory/hosts setup.yml --tags=setup-all,start`
2022-10-14 16:41:41 +02:00
2024-10-19 14:33:49 +02:00
5. Do an **initial sync of your files** by running this **on the server** (it may take a very long time):
2022-10-14 16:41:41 +02:00
2024-10-24 19:30:02 +02:00
```sh
sudo -u matrix -- rsync --size-only --ignore-existing -avr /matrix/synapse/storage/media-store/. /matrix/s3-media-store/.
```
2022-10-14 16:41:41 +02:00
2024-10-24 19:30:02 +02:00
You may need to install `rsync` manually.
2022-10-14 16:41:41 +02:00
2024-10-19 14:33:49 +02:00
6. Stop all Matrix services (`ansible-playbook -i inventory/hosts setup.yml --tags=stop`)
2022-10-14 16:41:41 +02:00
2024-10-19 14:33:49 +02:00
7. Start the S3 service by running this **on the server** : `systemctl start matrix-goofys`
2022-10-14 16:41:41 +02:00
2024-10-25 04:10:15 +02:00
8. Sync the files again by re-running the `rsync` command you see in step #5
2022-10-14 16:41:41 +02:00
2024-10-19 14:33:49 +02:00
9. Stop the S3 service by running this **on the server** : `systemctl stop matrix-goofys`
2022-10-14 16:41:41 +02:00
2024-10-19 14:33:49 +02:00
10. Get the old media store out of the way by running this command on the server:
2022-10-14 16:41:41 +02:00
2024-10-24 19:30:02 +02:00
```sh
mv /matrix/synapse/storage/media-store /matrix/synapse/storage/media-store-local-backup
```
2022-10-14 16:41:41 +02:00
2024-10-19 14:33:49 +02:00
11. Remove the `matrix_s3_media_store_path` configuration from your `vars.yml` file (undoing step #3 above)
2022-10-14 16:41:41 +02:00
2024-10-19 14:33:49 +02:00
12. Run the playbook: `ansible-playbook -i inventory/hosts setup.yml --tags=setup-all,start`
2022-10-14 16:41:41 +02:00
2024-10-19 14:33:49 +02:00
13. You're done! Verify that loading existing (old) media files works and that you can upload new ones.
2022-10-14 16:41:41 +02:00
2024-10-19 14:33:49 +02:00
14. When confident that it all works, get rid of the local media store directory: `rm -rf /matrix/synapse/storage/media-store-local-backup`
2022-10-14 16:41:41 +02:00
### Migrating to Backblaze B2
1. While all Matrix services are running, run the following command on the server:
2024-10-24 19:30:02 +02:00
(you need to adjust the 3 `--env` line below with your own data)
```sh
docker run -it --rm -w /work \
--env='B2_KEY_ID=YOUR_KEY_GOES_HERE' \
--env='B2_KEY_SECRET=YOUR_SECRET_GOES_HERE' \
--env='B2_BUCKET_NAME=YOUR_BUCKET_NAME_GOES_HERE' \
--mount type=bind,src=/matrix/synapse/storage/media-store,dst=/work,ro \
--entrypoint=/bin/sh \
docker.io/tianon/backblaze-b2:3.6.0 \
-c 'b2 authorize-account $B2_KEY_ID $B2_KEY_SECRET & & b2 sync /work b2://$B2_BUCKET_NAME --skipNewer'
```
2022-10-14 16:41:41 +02:00
2024-10-24 19:30:02 +02:00
This is some initial file sync, which may take a very long time.
2022-10-14 16:41:41 +02:00
2. Stop all Matrix services (`ansible-playbook -i inventory/hosts setup.yml --tags=stop`)
3. Run the command from step #1 again.
2024-10-24 19:30:02 +02:00
Doing this will sync any new files that may have been created locally in the meantime.
2022-10-14 16:41:41 +02:00
2024-10-24 19:30:02 +02:00
Now that Matrix services aren't running, we're sure to get Backblaze B2 and your local media store fully in sync.
2022-10-14 16:41:41 +02:00
4. Get the old media store out of the way by running this command on the server:
2024-10-24 19:30:02 +02:00
```sh
mv /matrix/synapse/storage/media-store /matrix/synapse/storage/media-store-local-backup
```
2022-10-14 16:41:41 +02:00
2024-10-24 19:55:15 +02:00
5. Put the [Backblaze B2 settings ](configuring-playbook-s3.md#backblaze-b2 ) in your `vars.yml` file
2022-10-14 16:41:41 +02:00
6. Run the playbook: `ansible-playbook -i inventory/hosts setup.yml --tags=setup-all,start`
7. You're done! Verify that loading existing (old) media files works and that you can upload new ones.
8. When confident that it all works, get rid of the local media store directory: `rm -rf /matrix/synapse/storage/media-store-local-backup`