Bug 25420 - Race condition in resolv_conf.c can result in caching stale configuration forever
Summary: Race condition in resolv_conf.c can result in caching stale configuration for...
Status: RESOLVED FIXED
Alias: None
Product: glibc
Classification: Unclassified
Component: network (show other bugs)
Version: 2.27
: P2 normal
Target Milestone: 2.32
Assignee: Florian Weimer
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2020-01-20 13:37 UTC by Filip Ochnik
Modified: 2020-02-14 12:13 UTC (History)
2 users (show)

See Also:
Host:
Target:
Build:
Last reconfirmed: 2020-01-20 00:00:00
fweimer: security-


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Filip Ochnik 2020-01-20 13:37:23 UTC
There is a TOCTOU race condition in resolv/resolv_conf.c:__resolv_conf_get_current between calling stat64 on /etc/resolv.conf and parsing its contents.
If /etc/resolv.conf is a symlink that changes what it points to between 2 different, existing files, we can end up in a situation when stat64 is called on one file and parsing is done on the other.


Consider the following scenario:
1. /etc/resolv.conf points to file A
2. __resolv_conf_get_current is called
3. stat64 is called on file A
4. we decide to update global cached config based on stat64
5. /etc/resolv.conf is changed to point to file B
6. __resolv_conf_load is called on file B and we use its contents as the new config
7. /etc/resolv.conf is changed back to point to file A

After these steps, the cached configuration is based on file B yet /etc/resolv.conf points to file A. The cache won't ever be updated, because we saved output from stat64 on file A.


I believe this issue could be solved by using fstat64 instead and then parsing from fd.


I also include a fully working PoC, which is a bit involved and perhaps convoluted since I'm not a C programmer. Apologies for that in advance. In any case, here are the steps to reproduce this issue exactly. 
I uploaded the needed files here: https://gist.github.com/filipochnik/edbbf8e1eac56057d2a521840461639d

1. Compile main.c with gcc main.c -o main and renamer.c with gcc renamer.c -o renamer
2. Prepare 2 symlinks: /etc/resolv.conf pointing to good.conf and /etc/resolv.tmp.conf pointing to bad.conf. (I assumed that there's a local nameserver running 127.0.0.53 and none or 127.0.0.1, these values may need adjusting)
3. Run run.sh

Description:

This setup changes in a loop what /etc/resolv.conf points to and tries to hit the described race condition. 

renamer.c simply switches what /etc/resolv.conf points to in a quick loop.

main.c runs renamer in the backgroud and resolves google.com in a loop. If we get an error on a first try it means we loaded the invalid configuration and possibly hit the race condition. We then kill renamer process, which leaves /etc/resolv.conf pointing to the correct configuration. In theory, we would expect that this change will be picked up soon but every so often it never happens. This means we hit the race condition. 

If you run this PoC you should hit a race condition in under a minute.
Once the race condition is hit, the bug can finally be confirmed by running touch on the correct config. The main process will then see that stat64 output changed, load the config again and resolve google.com successfully.
Comment 1 Sourceware Commits 2020-02-14 11:32:52 UTC
The master branch has been updated by Florian Weimer <fw@sourceware.org>:

https://sourceware.org/git/gitweb.cgi?p=glibc.git;h=fa00db0a6eb755837ae5d413515e0da582b304f3

commit fa00db0a6eb755837ae5d413515e0da582b304f3
Author: Florian Weimer <fweimer@redhat.com>
Date:   Tue Jan 21 17:38:15 2020 +0100

    resolv: Fix ABA race in /etc/resolv.conf change detection [BZ #25420]
    
    __resolv_conf_get_current should only record the initial file
    change data if after verifying that file just read matches the
    original measurement.  Fixes commit aef16cc8a4c670036d45590877
    ("resolv: Automatically reload a changed /etc/resolv.conf file
    [BZ #984]").
    
    Reviewed-by: Adhemerval Zanella <adhemerval.zanella@linaro.org>
Comment 2 Florian Weimer 2020-02-14 11:52:09 UTC
Fixed for glibc 2.32.

I tried the reproducer, but it is a bit of a fork bomb, so it did not work for me. We can add a proper test case later, maybe once we have a port directive for /etc/resolv.conf.
Comment 3 Florian Weimer 2020-02-14 12:13:10 UTC
.