Infrastructure Changes #46

Open
opened 2024-06-22 15:25:10 +02:00 by lukas · 9 comments
Owner

The armisael.libre.moe server is currently not used to it's potential and can be downscaled to a lower type. As the SSD cannot be downsized, a full rebuild of the server is necessary, which shouldn't be that complicated. Most scripts, configs and podman volumes can be simply moved over.

A real world benchmark between Hetzners CX21 and CAX11 should be made to determine which hosts brings the most performance for Seafile and Nextcloud.

ToDo

  • Remove Jitsi from Wiki and DNS
  • Compare CX21 and CCX11
  • Validate if not using --fast-list fixes the memory usage issue
  • Complete checklist for Seafile changes
  • Reduce armisael DNS TTY to 1min
  • Setup new server
  • Maintenance: stop and move all data, change DNS, start new server
  • shutdown old server
  • take snapshot of old server and delete it
  • Verify operation of cron scripts
The armisael.libre.moe server is currently not used to it's potential and can be downscaled to a lower type. As the SSD cannot be downsized, a full rebuild of the server is necessary, which shouldn't be that complicated. Most scripts, configs and podman volumes can be simply moved over. A real world benchmark between Hetzners CX21 and CAX11 should be made to determine which hosts brings the most performance for Seafile and Nextcloud. # ToDo - [x] Remove Jitsi from Wiki and DNS - [x] Compare CX21 and CCX11 - [x] Validate if not using `--fast-list` fixes the [memory usage issue](https://git.libre.moe/KomuSolutions/igot99issues/issues/46#issuecomment-2755) - [ ] Complete [checklist](https://git.libre.moe/KomuSolutions/igot99issues/issues/46#issuecomment-2806) for Seafile changes - [ ] Reduce armisael DNS TTY to 1min - [ ] Setup new server - [ ] [Maintenance](https://git.libre.moe/KomuSolutions/igot99issues/issues/46#issuecomment-2740): stop and move all data, change DNS, start new server - [ ] shutdown old server - [ ] take snapshot of old server and delete it - [ ] Verify operation of cron scripts
lukas added the
Breaking
Domain
LibreMoe
Priority
Medium
Kind
Enhancement
labels 2024-06-22 15:25:10 +02:00
lukas self-assigned this 2024-06-22 15:25:10 +02:00
lukas added this to the Issue Board project 2024-06-22 15:25:10 +02:00
Author
Owner

Maintenance Schedule

  • Verify setup of new server
  • Change DNS to new server, up TTL to 1h
  • Start maintenance
  • Safely shut down all services on armisael
  • Run backup to B2
  • Copy data from B2 to new server
  • Import volumes
  • Verify successful start of services
  • Get TLS certificates
  • Start nginx
  • Verify that services are reachable through domain
  • End maintenance
# Maintenance Schedule - [ ] Verify setup of new server - [ ] Change DNS to new server, up TTL to 1h - [ ] Start maintenance - [ ] Safely shut down all services on armisael - [ ] Run backup to B2 - [ ] Copy data from B2 to new server - [ ] Import volumes - [ ] Verify successful start of services - [ ] Get TLS certificates - [ ] Start nginx - [ ] Verify that services are reachable through domain - [ ] End maintenance
Author
Owner

Comparison of server types as shown in this thread seems to show that CX22 is comparable to CAX11, with the ARM-based CAX11 still being a bit faster in benchmarks. Real world tests using Seafile and Nextcloud need to be done.

Comparison of server types as shown in [this thread](https://lowendtalk.com/discussion/195428/hetzners-new-shared-intel-x86-cpus-for-cheap) seems to show that CX22 is comparable to CAX11, with the ARM-based CAX11 still being a bit faster in benchmarks. Real world tests using Seafile and Nextcloud need to be done.
Author
Owner

During backup tasks rclone with 16 threads consume ~1.7 GiB of RAM. With RAM usage of services being ~2.8 GiB, the usage in total without the OS and reserved memory itself is already above 4 GiB, meaning a server with only 4 GiB is already not sufficient.

During backup tasks `rclone` with 16 threads consume ~1.7 GiB of RAM. With RAM usage of services being ~2.8 GiB, the usage in total without the OS and reserved memory itself is already above 4 GiB, meaning a server with only 4 GiB is already not sufficient.
Author
Owner

New idea:

  • change armisael from CAX21 to CAX11, remove IPv4
  • add a new IPv6-only CAX11 server. move nextcloud and office to there
  • change DNS entries to map each servers A-record to arael and AAAA to the actual IPv6
  • make arael proxy IPv4 traffic to each server. as IPv6 is preferred when available, this should be very little
  • profit.
New idea: - change armisael from CAX21 to CAX11, remove IPv4 - add a new IPv6-only CAX11 server. move nextcloud and office to there - change DNS entries to map each servers A-record to arael and AAAA to the actual IPv6 - make arael proxy IPv4 traffic to each server. as IPv6 is preferred when available, this should be very little - profit.
lukas changed title from Armisael rebuild to Infrastructure Changes 2024-07-02 00:56:57 +02:00
lukas added
Priority
Low
and removed
Priority
Medium
labels 2024-07-02 00:57:12 +02:00
Author
Owner

IPv6 servers are blocked because Backblaze B2 doesn't support IPv6

IPv6 servers are blocked because Backblaze B2 doesn't support IPv6
Author
Owner

I am looking to also make a change to Seafile's underlying data storage. JuiceFS looks like a promising option to store data on object storage without using Seafile (bad) integrated (and enterprise-walled) feature.

Checklist

  • Setup test server
  • Test performance of a new instance
  • Copy actual Seafile data to new instance and test all possibilities and evaluate results
I am looking to also make a change to Seafile's underlying data storage. [JuiceFS](https://juicefs.com) looks like a promising option to store data on object storage without using Seafile (bad) integrated (and enterprise-walled) feature. # Checklist - [ ] Setup test server - [ ] Test performance of a new instance - [ ] Copy actual Seafile data to new instance and test **all** possibilities and evaluate results
Author
Owner

So juicefs looks promising, claiming a strong consistency, reliability and also performance.
copying hundreds of small files to B2 from my workstation ... was rather slow. Could be because of Backblaze or my connection - needs further testing with actual redis db, server internet and also different backends

So juicefs looks promising, claiming a strong consistency, reliability and also performance. copying hundreds of small files to B2 from my workstation ... was rather slow. Could be because of Backblaze or my connection - needs further testing with actual redis db, server internet and also different backends
Author
Owner

but we have encryption, so this could provide a good backend for general purpose storage needs while still keeping the data safe and encrypted

but we have encryption, so this could provide a good backend for general purpose storage needs while still keeping the data safe and encrypted
Author
Owner

the idea currently looks like this: setup a new server (cloud cube) on IONOS with ~4GB RAM, setup a juicefs with based on IONOS managed db and IONOS S3 object storage and migrate seafile blocks data to it. Seafile's fs and commits will stay local to allow fast general operation (since those folders only really grow up to afew megabytes in size and include hundreds of thousands of small files.

The OnlyOffice server will stay with Hetzner for cheap computing and memory. This system should be fast, reliable and cost effective. Backups will be made from the view of the server filesystem, so that in the event that the juicefs becomes corrupted or otherwise unusable, a recovery without juicefs is possible.

the idea currently looks like this: setup a new server (cloud cube) on IONOS with ~4GB RAM, setup a juicefs with based on IONOS managed db and IONOS S3 object storage and migrate seafile blocks data to it. Seafile's `fs` and `commits` will stay local to allow fast general operation (since those folders only really grow up to afew megabytes in size and include hundreds of thousands of small files. The OnlyOffice server will stay with Hetzner for cheap computing and memory. This system should be fast, reliable and cost effective. Backups will be made from the view of the server filesystem, so that in the event that the juicefs becomes corrupted or otherwise unusable, a recovery without juicefs is possible.
Sign in to join this conversation.
No Milestone
No project
No Assignees
1 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: KomuSolutions/igot99issues#46
No description provided.