Review of a secure software development process for glibc.

[Mark Wielaard]

Hi Joseph,

On Mon, May 12, 2025 at 03:34:42PM +0000, Joseph Myers wrote:
> On Sat, 10 May 2025, Mark Wielaard wrote:
> 
> > Although security is a layered concept, this glibc project security
> > policy could be much better/clearer if it just concentrated on the
> > glibc project secure software development process itself. We have
> > better/separate documents for the hosting infrastructure security
> > parts.
> 
> I think the nature of a large server running many different services is by 
> far the greatest security weakness around the toolchain development.  All 
> the other things around development process for each toolchain project are 
> secondary.

All layers are important. Of course you are right to point out places
where we can improve isolation at the infrastructure level. We are
certainly working on it. But you should also anticipate that every
layer can be breached. There will always be some administrators with
priviliges that you will have to trust, but who can of course be
comprimised. Which is why we are also trying to make the actual
project secure software development process so that it is resiliant
against a service compromise. Ideally the project processes are so
that a breach of the infrastructure is non-fatal.

That is why we are (also) pushing for secure development policies for
projects: https://sourceware.org/cyber-security-faq.html#policy-checklist
Please (also) read them thinking how they could help against an evil
service [admin].

> As I said at the 2022 Cauldron, maybe the Sourceware admins could move to 
> a modern high-security setup with isolated services (preferably isolated 
> (project, service) pairs, so a compromise in the context of one project 
> doesn't affect any other project), but such an arrangement with maybe 50 
> or more VMs would look very different from Sourceware as it was then - or 
> as it is now.
> 
> And I haven't really seen much progress on such isolation since 2022.  
> There have been some security improvements (in particular disabling 
> inactive user accounts), and new services may be better isolated, but I 
> simply haven't seen activity on isolating existing services.

Yes, we are working on it. Sorry the progress isn't as fast as you
like. And sorry not everything is as visible. Happy you did see some
improvements. New services like the builder and the forge have their
setup/config fully in public git already. Older configs are also in
git, but not public yet.

I know you would love everything in a separate VM and don't believe
process isolation is "enough", but we did start with that, using
systemd services and resource controls:
https://www.freedesktop.org/software/systemd/man/latest/systemd.resource-control.html

Also while most services still go through one httpd service things
have been split up now so that they don't have to, serveral services
actually "loop back" into the main httpd process, but could now easily
use their own process/container/VM.

In Q3, when the Red Hat community cage servers will move, we should
get an extra server which will be an oppertunity to turn some of the
above into "real" separate containers/VMs running on separate
hardware.

> > Every system can be gamed, so in general we should trust but
> > verify. In general it isn't good if we constrain/deny people making
> > any changes without strict formal/technical approvals. But it should
> > be easy to detect if unwanted changes were made, either by accident or
> > by malice.
> 
> I think this applies very much to Sourceware administrative practices even 
> though you wrote it regarding changes committed to project source code.  
> Knowing that some configuration change was made, or some administrative 
> action was done, seems to depend largely on being on IRC at the time it 
> happened.  That doesn't provide any effective oversight from the various 
> projects concerned, it doesn't provide any effective way that someone who 
> wasn't on IRC can know that something happened and why and challenge 
> whether it was actually a desirable change in the first place.
> 
> I think we need administrative practice that every system configuration 
> change or administrative action, large or small, is properly reported / 
> logged somewhere people can see and query later, without depending on 
> being on IRC when the change was made, so that the individual projects can 
> effectively consider whether such changes were actually desirable for 
> their projects or not.

Totally agreed all security layers should be as transparent as
possible. We do try to go through bugzilla for changes. And as said
above all configs are in git. We should definitely look into making
more of that public like we already do for newer services.

Sorry if it feels you have to be on IRC to participate, we do have
other channels of communication:
https://sourceware.org/mission.html#organization

Cheers,

Mark