WIP: Secure Software Development Life-cycle for the GNU C Library
This document is a work in progres
This document present information that is not yet finalized and still under discussion.
Contents
-
WIP: Secure Software Development Life-cycle for the GNU C Library
- High level practices for secure develoment
-
Prepare the glibc community (PO, page 5)
- Security for Software Development Infrastructure (PO.1.1 page 5)
- Security of glibc and development process (PO.1.2, page 5)
- Communicate the requirements for glibc development (PO.1.3, page 5)
- Implement roles and responsibilities to support glibc (PO.2.1, page 6)
- Provide support and documentation for those roles for glibc (PO.2.2, page 6)
- Gather consensus for secure development of glibc (PO.2.3, page 7)
- Specify which tools are going to be used for secure development (PO.3.1, page 7)
- Follow the written security practice (PO.3.2, page 7)
- Generate artifacts that support secure practices (PO.3.3, page 7)
- Define software security checks (PO.4.1, page 8)
- Implement processes to gather and safeguard security check data (PO.4.2, page 8)
- Separate and protect each environment involved in development (PO.5.1, page 8)
- Secure and harden development endpoints (PO.5.2, page 8)
- Protect glibc's software components (PS, page 9)
-
Produce secure glibc releases (PW, page 11)
- Assess the security risk of glibc (PW.1.1, page 11)
- Track and maintain security requirements for glibc (PW.1.2, page 11)
- Build in support for standardized security features for glibc (PW.1.3, page 11)
- Review secure software design frequently (PW.2.1, page 12)
- Use well-secured and maintained software components (PW.4.1, page 12)
- Create and maintain well-secured software components for use by glibc (PW.4.2, page 12)
- Verify that all the components we use comply with our own requirements (PW.4.4, page 13)
- Follow secure coding practices for glibc (PW.5.1, page 13)
- Use build tools that offer features to improve binary artifact security (PW.6.1, page 14)
- Determine which build tools we should use and how to configure them (PW.6.2, page 14)
- Determine if code review, automated or by a human should be used (PW.7.1, page 14)
- Perform code review based on the secure coding standards (PW.7.2, page 14)
- Determine if binary artifact review is required to find vulnerabilities (PW.8.1, page 15)
- Perform testing and document results (PW.8.2, page 15)
- Define a secure baseline for configuring all components (PW.9.1, page 16)
- Implement the default secure settings (PW.9.2, page 16)
-
Respond to vulnerabilities in glibc (RV, page 16)
- Gather information about glibc vulnerabilties (RV.1.1, page 16)
- Review, analyze and confirm vulnerabilities (RV.1.2, page 16)
- Have a policy and process required to address vulnerabilities (RV.1.3, page 17)
- Analyze each vulnerability and plan response (RV.2.1, page 17)
- Implement vulnerability response (RV.2.2, page 18)
- Analyze and document root cause of vulnerability (RV.3.1, page 18)
- Analyze root causes over time (RV.3.2, page 18)
- Review root causes over time to remove classes of vulnerabilities (RV.3.3, page 18)
- Review the glibc development process and update to remove root causes of issues (RV.3.4, page 18)
This document follows the context laid out in the Secure Software Development Life-cycle for the GNU Toolcahin document.
The following documents are specific policies for glibc.
The structure follows the same structure as NIST SP 800-218 (Secure Software Development Framework (SSDF) Version 1.1).
High level practices for secure develoment
The following are the top-level practices that we should follow in order develop a secure and robust GNU C Library.
The underlying implementation principles are taken from adhering to and following best practices for zero-trust environments.
Zero-trust as a principle is considered industry best practice and you can follow along here:
It is not that we don't trust the FSF, the GNU Project, GNU Maintainers, Sourceware, or our developers, but in a zero-trust environment we assume that one or more of the entities involved in the process have been compromised and we attempt to limit the consequences of such a compromise.
Practice |
Task |
Prepare the glibc community (PO, page 5) |
Identify, document, and implement security best practices for the glibc community. |
Protect glibc's software components (PS, page 9) |
Implement, support, and maintain the protections required to avoid unauthorized access or tampering of glibc sources. |
Produce secure glibc releases (PW, page 11) |
Identify, and document the processes by which glibc produces secure software, and how it addresses remaining risks. |
Respond to vulnerabilities in glibc (RV, page 16) |
Identify, confirm, and remediate vulnerabilities in glibc. |
We will break down each practice and the impact it has on software and the infrastructure involved in those practices.
Prepare the glibc community (PO, page 5)
Tasks/Service |
CTI |
Sourceware |
GNU Savannah |
Define security requirements for development (PO.1, page 5). |
Yearly review. Service related Critical or Major CVE review. Implementation must meet requirements. |
Likewise. |
Likewise. |
Define and implement roles and responsibilities for the project (PO.2, page 6). |
Roles must be defined in the implementation. |
Likewise. |
Likewise. |
Implement and maintain supporting security tooling (PO.3, page 7). |
Implementation must be operated by service provider. |
Likewise. |
Likewise. |
Define and implement software security checks (PO.4, page 8). |
Security checks must be run by service provider. |
Likewise. |
Likewise. |
Implement and maintain secure environments for software development (PO.5, page 8). |
Service provider must continuously check environments. |
Likewise. |
Likewise. |
Security for Software Development Infrastructure (PO.1.1 page 5)
Identifying, documenting and maintaining requirements for secure software development infrastructure is critically important.
These requirements set the baseline for what we expect from our infrastructure.
CTI on behalf of the GNU Toolchain carried out a services enumeration for glibc here: https://cti.coretoolchain.dev/projects/enum.html
The service enumeration is valuable since each of the services need to meet specific criteria for securing software development infrastructure.
Subsequently a statement of work was prepared by the Linux Foundation IT team: https://lore.kernel.org/cti-tac/2186f48a-12cd-4630-8944-26b9042e924a@redhat.com/T/#u
The following core principles have come out of the review, based on glibc's desire for a secure and robust infrastructure:
- System and service isolation as a critical underpinning of the availability of developer services e.g. git, mailing lists, bugzilla, patchwork.
- Service availability has a direct impact on our ability to remediate vulnerabilities.
- Secure developer endpoints reduce the risk of supply chain attacks.
- Secure build systems and attestation systems for verifying artifact integrity e.g. reproducibility, source to downstream chain of custody.
The following is a table of existing infrastructure for use by associated services.
Tasks/Service |
CTI |
Sourceware |
GNU Savannah |
System isolation |
Yes. Multiple hardware nodes with live-failover |
Yes, two system for essential services. Primary and backup. One additional system for snapshots. |
Hardware isolation unknown. |
Service isolation |
Yes. Distinct containers, VMs, and even distinct systems. |
Partially. Primarily via containers. VMs possible. All on a single primary system. |
Partially. Primarily via VMs. Containers possible. Hardware isolation unknown. |
Secure developer endpoints |
Partially, via FOSS hardware access token program supported by LF IT |
Possibility to use hardware access tokens. |
Possibility to use hardware access tokens. No GNU Prject program for hardware token support. |
Secure build systems and attestation systems for verifying artifact integrity |
No support currently. |
File delivery via Apache from ssh-based uploads. |
GNU Upload with GPG signing. Access to FSF fencepost with ssh key. |
Security team requirements and service availability
We are only going to talk about availability from the perspective of security.
One specific aspect here is that glibc becoming a CNA in the CVE MITRE program allows glibc to reduce the latency to respond to security issues. This adopts overall best practice for handling security issues in that the subject matter experts can review the incoming requests and respond with cogent arguments for or against the issue, but it does create a service that attackers may want to exploit.
The glibc CNA is a potential target for attackers that want to reduce or degrade upstream's ability to respond to critical or important CVEs in a timely fashion. This means that it is paramount that mailing lists as a service, particularly the glibc CNA mailing list should have a high degree of availability.
Well resourced attackers, after finding a critical or important vulnerability, could attempt to disrupt mail delivery for the glibc security team while they use this time to carry out specific attacks.
While it is possible to report the issue downstream to the Debian, Red Hat, SUSE or other teams, or directly to linux-distros, the lack of availability of a service introduces a possible uncertainty and delay in response.
In general it is becoming more and more expected that upstream is able to help handle security issues, and as such the services like a mailing list for communication become more critical.
The alternative is to give up on handling security issues in the upstream projects, but this is not a good position to take because it is in direct opposition to what other projects are doing to make security handling more responsive and valuable.
Thus we have an availability requirement: Mailing list service availability is paramount to handling security issues in a timely fashion.
Similarly the project git repository and the main branch advisories directory is the single source of truth for project published advisories and must be accessible with a high degree of availability via read-only access.
Security of glibc and development process (PO.1.2, page 5)
Generally speaking all defects in glibc should where possible be corrected with a regression test added to address the defect.
Dynamic loader
Where possible the dynamic loader should assume defaults that protect dynamic loader internals from modification and misuse.
DNS stub resolver
The DNS stub resolver should aim to handle malformed packets and degrade gracefully when communicating with upstream recursive resolvers.
Userspace memory allocator
The malloc family API functions should aim to limit how much is cached in userspace to avoid OOM scenarios.
The implementation should take steps to avoid unbounded allocations in userspace for a bounded set of API requests.
The goal is to avoid OOM scenarios becoming denial of service security issues.
Documented release process
The glibc project should document in detail the release process and what needs to be archived.
The process is currently documented here: https://sourceware.org/glibc/wiki/Release/
Documented security policy
The glibc project should document in detail the security policy.
The glibc security policy is documented here: https://sourceware.org/git/?p=glibc.git;a=blob;f=SECURITY.md
Documented security process
The glibc project should document in detail a security process.
The glibc security process is documented here: https://sourceware.org/glibc/security.html
Communicate the requirements for glibc development (PO.1.3, page 5)
The requirements in this document should be communicated at least annually with the developers on the main development mailing list.
The requirements in this document should be communicated at least annually with the service provider.
The glibc project should review if the service provider meets the requirements set out in this document and discuss deficiencies.
Implement roles and responsibilities to support glibc (PO.2.1, page 6)
The glibc project in general has the following roles:
- GNU Project Maintainers - Support the project in making decisions.
- GNU Project uploader - Manages uploading GNU Project source archives.
- Maintainers for code - Developing code for the project. Includes global, machine, os, and distribution maintainers.
- Maintainers for mailing lists - Managing the mailing list, spam, moderation etc.
- Maintainers for the website - Developing the project website.
- Maintainers for the wiki - Documenting process, design docs, and updating instructions.
- Maintainers for patchwork - Developing the patchwork patch tracking instance.
- Maintainers for git configuration - Developing and supporting source control.
- Maintainers for bugzilla - Maintain the release process updates integrated into bugzilla.
- Security team for the project - Review and respond to incoming security issues.
- Code of conduct committee - Review and respond to incoming CoC issues and product annual transparency report.
Each role should have distinct access controls, though it would be allowable to implement SSO across all services.
Provide support and documentation for those roles for glibc (PO.2.2, page 6)
Supporting documentation for the roles is provided here:
Release process: https://sourceware.org/glibc/wiki/Release/
Code of conduct committee: https://sourceware.org/glibc/wiki/CoC
GNU Maintainers guide: https://www.gnu.org/prep/maintain/maintain.html
Gather consensus for secure development of glibc (PO.2.3, page 7)
Currently Carlos O'Donell is the primary author for this document and a GNU Project maintainer for glibc working on defining a secure software development life-cycle process for glibc.
Specify which tools are going to be used for secure development (PO.3.1, page 7)
The project should use gcc and clang to detect and correct warnings produced by both compilers to reduce project defects.
Follow the written security practice (PO.3.2, page 7)
The project, in collaboration with downstream, should run a static analyzer in pre-commit CI or post-commit.
Generate artifacts that support secure practices (PO.3.3, page 7)
Artifacts shall be recorded in patchwork tracking that a patch passed the required checks for inclusion in the project.
There is a security gap here in that we do not verify what was tested is what is committed, but this is a limit of the current process.
Post-commit testing must still be carried out in downstream distributions to make up for the process gap.
For example Fedora does a weekly synchronization with upstream glibc and tests regularly with artifacts of testing recorded e.g. https://bodhi.fedoraproject.org/updates/FEDORA-2025-54758e9af3
Define software security checks (PO.4.1, page 8)
The glibc regression testsuite must run without failures on at least one architecture.
At the release boundary build-many-glibcs.py should run to completion correctly without defects for at least one architecture.
Access to external source code repositories that are downloaded in the process of testing should always be via HTTPS.
As an implementation detail build-many-glibcs.py should use HTTPS for all urls.
The release process dictates the tests that should be run at release: https://sourceware.org/glibc/wiki/Release/
Implement processes to gather and safeguard security check data (PO.4.2, page 8)
Security check data stored in patchwork should be present permanently and the patchwork database should be safeguarded and backed up.
Tasks/Service |
CTI |
Sourceware |
GNU Savannah |
Record security pre-commit CI test |
Yes. via Patchwork |
Yes. via Patchwork |
Not easily via Savannah patch tracking |
Store security test results |
Yes. via database and backups. |
Yes. via database and hot standby |
Unknown. |
Separate and protect each environment involved in development (PO.5.1, page 8)
Each of the environments and services involved in development, from the mailing lists to git shall be separated to ensure freedom from interference.
The goal is to deploy a zero-trust infrastructure where each of the services can be degraded and have minimal if no impact on other services.
The specific services in question here are:
- git
- mailing lists and email
- website
- wiki
- patchwork
- bugzilla
Tasks/Service |
CTI |
Sourceware |
GNU Savannah |
Distinct system or VM handling git with high availability for read-only requests |
Yes. |
No. Process isolation only. High-availability via non-authoritative mirrors. |
Yes. Multiple VMs. |
Distinct system or VM handling bugzilla |
Yes. |
No. Process isolation only. |
Yes. Distinct VM (shared web/db). |
Distinct system or VM handling mailing lists |
Yes. |
No. Process isolation only. |
Yes. Distinct VM (not shared). |
Distinct system or VM handling patchwork |
Yes. |
No. Process isolation only. |
Yes. Distinct VM (shared as web/db). |
Distinct system or VM handling website |
Yes. |
No. Process isolation only, coupled with Apache running multiple services. |
Yes. Distinct VM (shared web/db). |
This does not include additional systems that are not in the direct line of critical services e.g. buildbot, pre/post-commit CI systems, archival systems.
At least annually the project should audit that infrastructure and services are up to date with respect to most recent patch levels. Service providers should provide sufficient information to the project to show all the in-use software has been updated and this should be recorded.
Secure and harden development endpoints (PO.5.2, page 8)
Developers should develop in an environment that is supported and secure and not end of life (EOL) e.g. https://docs.fedoraproject.org/en-US/releases/eol/, https://wiki.debian.org/DebianReleases.
Developers should store ssh keys on secure open hardware devices e.g. Nitrokey.
The glibc project should have a once-a-release reminder that developers should be using a development environment that is supported and has security vulnerabilities corrected.
Protect glibc's software components (PS, page 9)
Protect, verify, and archive software produced using the provided infrastructure (PS)
Tasks/Service |
CTI |
Sourceware |
GNU Savannah |
Protect code from unauthorized access and tampering (PS.1, page 9) |
Allow signing of a release, storage, and limited write access. |
Likewise. |
Likewise. |
Provide mechanisms for verifying software release integrity (PS.2, page 10) |
Allow verification of release with transparency logs |
Likewise. |
Likewise. |
Archive and protect each software release (PS.3, page 10) |
Archive each release and protect them from loss. |
Likewise. |
Likewise. |
Store all forms of code based on principle of least privilege (PS.1.1, page 9)
Code read access is open to the public for review.
As an implementation detail all read-only git access should be carried out over HTTPS to support transport layer security. Recommending HTTPS in glibc git information pages.
Code write access is restricted to developers of the project i.e. Maintainers for code.
Code write access should be possible to audit by placing keys in cgit and allowing GNU Project maintainers to approve and remove access following policy.
Code write access should be disabled if writes to either the glibc or glibc-access repo (a temporary repo to touch for access) have not occurred within the last 12 months.
- Note that the "glibc-access" repo has not yet been created.
Code write access can be reinstated upon written request to a GNU Project maintainer.
Approved top-tier branches should have identified owners and only they should have code write access.
At least once a year the top-tier branches via git ls-remote should be reviewed and ownership assessed and discussed with the community.
Sourceware website code write access should be limited to maintainers for the website.
GNU Project website code write access should be limited to GNU Project maintainers.
Code write access to git repo configuration should be limited to GNU Project maintainers.
Project branch tags must be signed by release manager gpg key.
Account key access policies should be documented and account keys deactivated where accounts have been inactive for more than 12 months (2 glibc release cycles). Account roles remain the same.
Account activity includes: authenticated git access (pull or push), wiki page updates, bugzilla updates, patchwork updates, in general any interaction with services that can be verified.
Account key access deactivation should have a minimum of 3 notification attempts by email.
Reactivating accounts should require going through a more stringent validation.
Make software integrity verification information available (PS.2.1, page 10)
Project source archives released on GNU Project server must include signature file for verification that GNU Project maintainer or uploader signed the archive.
The goal is to ensure the source archive was not tampered between the point the GNU Project maintainer or uploader created it and the point at which it arrived on the GNU project server for download.
Secure archive necessary files and supporting data (PS.3.1, page 10)
Access to uploaded archives on the GNU Project server is restricted to the FSF admins, GNU Project maintainers, and uploaders.
Collect, safeguard, maintain and share provenance data (PS.3.2, page 10)
Existing releases are archived on the GNU Project servers i.e. https://ftp.gnu.org/gnu/glibc/
Produce secure glibc releases (PW, page 11)
Produce, review and test glibc using secure and state of the art infrastructure (PW)
Tasks/Service |
CTI |
Sourceware |
GNU Savannah |
Design software to meet security requirements and mitigate security risks (PW.1, page 11) |
NA |
Likewise. |
Likewise. |
Review secure software design frequently (PW.2, page 12) |
NA |
Likewise. |
Likewise. |
Verify third-party sofware (PW.3, page 12) |
Moved to PO.1 and PW.4. |
Likewise. |
Likewise. |
Reuse existing software (PW.4, page 12) |
NA |
Likewise. |
Likewise. |
Following secure coding practices (PW.5, page 13) |
NA |
Likewise. |
Likewise. |
Configure the toolchain to improve executable security (PW.6, page 14) |
NA |
Likewise. |
Likewise. |
Review and audit code for security defects (PW.7, page 14) |
NA |
Likewise. |
Likewise. |
Test code to identify vulnerabilities (PW.8, page 15) |
NA |
Likewise. |
Likewise. |
Secure by default settings (PW.9, page 16) |
NA |
Likewise. |
Likewise. |
Assess the security risk of glibc (PW.1.1, page 11)
Constantly review code posted to the mailing list, reviewing code manuall for potential security risks.
Run gcc -fanalyzer in pre-commit CI configuration to attempt to detect security issue in patches.
Track and maintain security requirements for glibc (PW.1.2, page 11)
Run gcc -fanalyzer on glibc sources and ensure no failures.
Support mechanism that allow glibc to transition to read-only memory those mappings that can be made read-only e.g. PT_GNU_RELRO.
Write a security requirements design document for glibc.
Build in support for standardized security features for glibc (PW.1.3, page 11)
Move towards building in PT_GNU_RELRO support.
Move towards using PT_GNU_STACK set to non-executable stack.
Review secure software design frequently (PW.2.1, page 12)
Review security requirements design document once every 6 months (before release).
Use well-secured and maintained software components (PW.4.1, page 12)
Ensure that bundled software included and tracked in SHARED-FILES top level file are all updated and free from defects.
Carry out review every 6 months (before release).
Create and maintain well-secured software components for use by glibc (PW.4.2, page 12)
All additional components created by the project for the project are included in the project sources.
Verify that all the components we use comply with our own requirements (PW.4.4, page 13)
Work through the set of requirements in INSTALL and update them to the latest versions.
Ensure that requirements in INSTALL are themselves as secure as we expect e.g. gcc, binutils, python etc.
Follow secure coding practices for glibc (PW.5.1, page 13)
Follow secure coding guidelines for C, including giving strong consideration to guidance from SEI CERT C Coding Standard.
Use build tools that offer features to improve binary artifact security (PW.6.1, page 14)
Use gcc flags for a good combination of security features including Source Foritifcation level 3.
Not all security features can be enabled by default because of their performance cost, including those for SPECTRE and MELTDOWN CPU mitigations.
Determine which build tools we should use and how to configure them (PW.6.2, page 14)
We should use all the build tools as documented in INSTALL.
Determine if code review, automated or by a human should be used (PW.7.1, page 14)
Reach for the goal of one human reviewer for all commits to the project.
All patches to have been tested by pre-commit CI for at least one target.
Perform code review based on the secure coding standards (PW.7.2, page 14)
Code review for patches carried out on the libc-alpha mailing list and supported by patchwork instance.
Determine if binary artifact review is required to find vulnerabilities (PW.8.1, page 15)
The project does not deliver binary artifacts, but does contain pre-generated sources for configure scripts.
Regenerate files once per release (as documented in the Release process) to ensure they can be regenerated and commit them to git.
Perform testing and document results (PW.8.2, page 15)
Results of testing are documented per release e.g. glibc 2.41 test results
Define a secure baseline for configuring all components (PW.9.1, page 16)
Default "./configure --prefix=/usr" defines the security baseline.
Implement the default secure settings (PW.9.2, page 16)
Default "./configure --prefix=/usr; make;" implements the recommended secure build.
Respond to vulnerabilities in glibc (RV, page 16)
Identify, confirm, and respond to vulnerabilities in the glibc using services provided by highly available infrastructure (RV)
Tasks/Service |
CTI |
Sourceware |
GNU Savannah |
Continuously identify and confirm vulnerabilities (RV.1, page 16) |
NA |
Likewise. |
Likewise. |
Asses, prioritize and remediate vulnerabilities (RV.2, page 17) |
NA |
Likewise. |
Likewise. |
Identify vulnerability root causes (RV.3, page 18) |
NA |
Likewise. |
Likewise. |
Gather information about glibc vulnerabilties (RV.1.1, page 16)
The glibc CNA gathers information about security vulnerabilities following https://sourceware.org/glibc/security.html.
Review, analyze and confirm vulnerabilities (RV.1.2, page 16)
The glibc security team analyzes and confirms reported vulnerabilities.
Have a policy and process required to address vulnerabilities (RV.1.3, page 17)
The glibc security policy for processing is recorded here: https://sourceware.org/glibc/wiki/CNA which includes a response guide: https://sourceware.org/glibc/wiki/CNA/Response
Analyze each vulnerability and plan response (RV.2.1, page 17)
The glibc security team analyzes each reported vulnerability and plans a response following: https://sourceware.org/glibc/wiki/CNA/Response
Implement vulnerability response (RV.2.2, page 18)
Response is implemented following: https://sourceware.org/glibc/wiki/CNA/Response
Analyze and document root cause of vulnerability (RV.3.1, page 18)
The glibc security team includes relevant community members to root cause the issue.
Analyze root causes over time (RV.3.2, page 18)
Issues are recorded in the advisories directory of the project e.g. https://sourceware.org/cgit/glibc/tree/advisories
At least once a release the previous 6 months of vulnerabilities should be reviewed.
Review root causes over time to remove classes of vulnerabilities (RV.3.3, page 18)
At least once a release the previous 6 months of vulnerabilities should be reviewed to determine if a trend exists.
Review the glibc development process and update to remove root causes of issues (RV.3.4, page 18)
At least once a release the previous 6 months we should review updating pre-commit CI to address and remove root causes.
Notes:
- MITRE program uses CNA mailing list as the single point of contact for escalations and discussions.
References: