The overseers sometimes hear questions similar to "how secure is sourceware"? While such a vague question may deserve a half-hearted "secure enough" response, let's address it deeper.

History

Sourceware has been operating since 1998. During those 25+ years of operation, we recall a total of two security incidents, with the last one occurring over a decade ago. We know of no case of a software repo suffering unauthorized change.

Principles

  • Simplicity

We run the simplest software configuration possible to get the job done. Old school UNIX packages and mechanisms are lightweight and well-understood, so we prefer them when they are sufficient. The infrastructure is minimal and fits well even a single server.

  • Intimacy

Sourceware caters to a relatively small user base of a few thousand developers for a few dozen projects. At that scale, we can keep open direct communications with the project maintainers and even users. There is no business-driven or informal urge to keep growing, so we can stay a personal scale community.

  • Free/Open-Source

Naturally, every snippet of software running is free/open-source software.

  • No secrets

Sourceware aims to store no secrets in the form of credentials, crypto data, or even user content that would be a sensible target for an intruder. There is approximately nothing confidential worth breaking in for. Even accidental exposure of much configuration or other such data is harmless, because confidentiality is not a factor.

  • Choice

The needs of software projects hosted at sourceware are similar but also vary. Because sourceware is small, we can generally offer special exceptions or extensions needed by individual projects.

Security measures

How do we accomplish those goals?

  • Overseers

Sourceware is looked after by a small group of "overseers", with decades of experience in software development as well as systems administration. They work partly on a volunteer and partly on a part-of-the-job basis, on different continents. Red Hat staff look after the physical hardware at the RDU colocation facility.

  • Operating system

Sourceware runs on a fully supported, centrally monitored copy of Red Hat Enterprise Linux. It self-updates via the Red Hat Network frequently. Most of the software is stock from Red Hat or Fedora EPEL. A few exceptions are extra infrastructure that comes from unpackaged free software sources compiled locally.

  • Network security

We operate the usual suite of firewalls at the TCP/IP and HTTP levels, plus fail2ban. There is no DoS frontend server at the moment, so the system attempts to protect itself from overloads by rate-limiting certain operations via systemd and httpd configuration. Common attack attempts against the infrastructure are detected by fail2ban filters monitoring log files and result in auto-firewalling of attacking IP addresses or subnets.

  • GIT

Git is the primary software repository engine on sourceware. It provides plenty of native protection against accidental data corruption, and is configured to run with some repo self-checking that is not otherwise default. If somehow raw repository history were to be corrupted, the many clones that exist would immediately make this obvious, and recovery would be possible.

Several projects use git hooks to perform fine-grained access control, bugzilla interface. Some projects use gitsigur, a homegrown script for enforcing authenticated signed git commits, to protect repos from misattributed commits.

  • Groups

Different sourceware hosted projects are assigned distinct groupids in /etc/group. Their code repositories are writable exclusively by members of that groupid. Some projects share repos ("src", containing both binutils and gdb.) Most projects are fully isolated from one another.

  • Logins

Commit-capable contributors to most of our hosted projects receive credentials to log onto sourceware, to individual /etc/passwd userids created for them. These userids are accessed exclusively through ssh public keys (no secrets!) The userids are associated with groupids for the projects they received authorization for, which gives them write access to only those git repository contents. Users that have left projects have their group memberships cleaned. Users that leave sourceware entirely are considered "emeritus", and cannot log on at all. However, their stub userid is reserved and preserved for history.

While contributors can log on, almost all of them are subject to a homebrew restricted shell (long predating "ssh-shell") that permits them to do a very limited number of operations, such as pushing git commits. Most contributors (1100+) cannot access a general shell on sourceware at all.

Some project maintainers receive normal shell access, which lets them hand-manage repositories, release directories, web documents etc., but these people are few (60+), and highly trusted senior members of their respective projects. Some projects have additional administrative userids, usually shell-capable, which are shared and managed by multiple maintainers. These accounts sometimes perform automated local work on the repositories.

Instead of first-class userids for contributors, some projects use gitolite with its nested authentication system. Contributors for these projects do not have distinct userids, and are instead managed by the projects' gitolite administrators.

Only the handful of overseers can login to the root account via ssh.

Password authentication for shell/git access is entirely disabled (ssh pubkeys only), partly to prevent dictionary type attacks.

Long-unused userids are periodically retired to "emeritus" status.

  • Web services

Sourceware operates several web services (bugzilla, wikis, patchwork, bunsen, gitweb, cygwin catalogs). All but the trivial ones tend to run as reverse-proxied daemons running under distinct unprivileged userids. There is some denial-of-service protection in place.

Some web services such as bugzilla and the wikis have their own separate authentication system based on passwords, but those credentials cannot be used to log into system shells.

  • Non-local builds

Sourceware runs a buildbot server, which is tasked by several projects to build snapshots or test builds. The slave workers run on donated hardware resources running elsewhere. A few projects elect to package pre-release source snapshots on sourceware itself, but they do not build actual project binaries. That makes sourceware a less juicy target for "reflections on trusting trust" type infrastructure attacks.

  • Monitoring

Sourceware's performance is remotely monitored via prometheus node_exporter, RHEL Insights and other technologies. Remote access to the server console is available to a few key individuals. System logs are kept for a long time, on and off site.

  • Disaster recovery

Sourceware runs on an industrial Dell server, with RAID6 local storage. Hard drives have died and have been replaced, generally via live hotswaps while online. It has been amazingly reliable: your scribe does not remember a single outright system crash. Sourceware has a warm backup twin nearby, which receives daily snapshots of the main sourceware project hosting data. A far off-site server also receives intermittent snapshots of the sourceware content. Many projects also duplicate their repositories on mirrors and other git forge type services. Therefore, data should be fairly safe, even in the case of catastrophic damage at the physical colocation facility.