Our first full day Open House! Friday 14 2025, full day 09:00 to 17:00 UTC. #overseers on irc.libera.chat
Discussing Migration2025
Hardware resources
See Migration2025.
OSUOSL is offering to replace our current 2 x86_64 container builders with one bigger server.
- Old 48G/62G mem, 260GB/450GB disk, 16/24 cores
- New 2x28 cores (2x56 threads), 768GB RAM machine. Need to provide disks.
- Recommend Sourceware PLC to buy 2 x NVMe M.2 1TB (VMs/containers) and one 2.5" SATA drive of 500GB (base OS).
Security
- Red Hat IT does scans of our servers and we discuss findings
We last updated the STARTLS crypto algos for forge.sourceware.org based on their recommendations
Container package security checks
- builder.sourceware.org has various container builders. Some build gnu tools from source.
- https://sourceware.org/cgit/builder/tree/builder/containers
- Sam sent a patch to always check signatures for source tars
- https://inbox.sourceware.org/buildbot/168d47063b5286251d749a350b38939d23b77794.1763093965.git.sam@gentoo.org/
- Discussion if/how we can trust https://ftp.gnu.org/gnu/gnu-keyring.gpg when using ftpmirror
- Pulled in glibc hackers to show how they can impove their own build-many-glibcs.py script
The new server1 VM-first setup
- New server (already in RDU3) has been setup with a minimal RHEL 10 setup just for running VMs
- Access is currently only available to fche and mjw. No direct public internet access. We could setup a jumphost.
- But probably ok to have a (read-only) git repo with just the libvirt configs
- Currently running just three VMs
- forge-stage-deb13 (4GB mem, 2 vcpu, 20 GiB lv), https://forge-stage.sourceware.org, ansible managed
- debuginfod-elfutils (8GB mem, 2 vcpu, 50 GiB lv), https://debuginfod.elfutils.org, already in production
- sourceware (256GB mem, 8 vcpu, 3.91 TiB lv), hosting everything from server2 which isn't already in a separate VM
- Other proposed VMs from Migration2025
- inbox (16GB mem, 6 vcpu, 8GB + 120GB lvs), current version 1.9.0 (from epel), latest 2.0.0
- patchwork (12GB mem, 4 vcpu, 12GB + 12GB lvs), current version v3.1.1 [django 3.2.x] (in python env with local patches), latest v3.2.1 [django 5.2.x]
- buildbot (32GB mem, 4 vcpu, 8GB + 30GB lvs), current version 3.9.2 (python env with 1 local patch, see SETUP), latest 4.3.0
- forge (32GB mem, 8 vcpu, 12GB + 128GB lvs), as soon as forge-stage ansible setup works out for production usage
- bunsen (12GB mem, 8 vcpu, 500GB lvs)
- cygwin, gitweb/cgit, bugzilla, dwarfstd, valgrind
Network
For network setup we have a /24 with 8 public addresses that can be staticly assigned to each VM (if we need more, we can get more).
Migration experiment, add the following to your /etc/hosts (make sure to take it out again!)
38.145.34.32 sourceware.org 38.145.34.32 gcc.gnu.org 38.145.34.32 elfutils.org 38.145.34.32 dwarfstd.org 38.145.34.32 lists.dwarfstd.org 38.145.34.32 valgrind.org 38.145.34.32 cygwin.com 38.145.34.32 cygwin.org 38.145.34.32 cygwin.net 38.145.34.32 www.cygwin.com 38.145.34.32 www.cygwin.org 38.145.34.32 www.cygwin.net 38.145.34.32 ftp.cygwin.com 38.145.34.32 ftp.cygwin.org 38.145.34.32 ftp.cygwin.net{www,builder,inbox,patchwork,ecos,gcc}.sourceware.org are all CNAMEs so don't need the (new) address.
forge.sourceware.org uses an RH OSCI VM
snapshots.sourceware.org uses a server at osuosl.
forge-stage.sourceware.org is already vm08 (doesn't use CNAME but A record for MX).
debuginfod.elfutils.org is already an alias for vm07
{www,git,wiki}.dwarfstd.org are CNAMEs for sourceware.org
www.valgrind.org is a CNAME for sourceware.org
systemtap.org is maintained by Sourceware PLC/Conservancy, but DNS is handled by elastic.org
Note {www,ftp}.cygwin.{com,org,net} are CNAMEs for server2.sourceware.org (so above they get an new address)
Also note that there are no IPv6 addresses available in RDU3 (yet?). And rDNS isn't mapped (yet?).
Backups
- Should be done through lvm snapshots. Which can then be rsynced to the "old" server2 or server3.
- Some debate what to do about databases, should they be put in "backup mode", dumped regularly as .sql files?