This came up as a continuation of PR#29911 one thing that FchE noticed was that fedabipkgdiff was downloading debuginfo packages rather than fetching the data from debuginfod even though libabigail being based on elfutils has the capability to fetch debuginfod. What we figured out was that somewhere in fedabipkgdiff it was deciding to and fetching the debuginfod from koji and then it would prefer that over debuginfod data. It was FchE that noticed this but once I started thinking about it, I realized that this could greatly improve fedabipkgdiff. As I was running my full distro test this is the kind of thing that I noticed in my top session as I monitored progress. Note how many processes are running the cpio to unpack a package vs how many jobs actually running abipkgdiff. top - 18:30:34 up 1 day, 9:34, 4 users, load average: 31.53, 37.87, 42.06 Tasks: 581 total, 16 running, 565 sleeping, 0 stopped, 0 zombie %Cpu(s): 39.6 us, 12.4 sy, 0.0 ni, 23.8 id, 24.0 wa, 0.1 hi, 0.1 si, 0.0 st MiB Mem : 128119.5 total, 65441.2 free, 5400.2 used, 58509.8 buff/cache MiB Swap: 32124.0 total, 31700.5 free, 423.5 used. 122719.3 avail Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 3199639 ben 20 0 1635092 1.3g 131052 S 99.7 1.1 0:18.00 abipkgdiff 3199358 ben 20 0 722808 596612 47728 S 99.3 0.5 0:08.26 abipkgdiff 3183615 ben 20 0 189168 91344 12564 S 99.7 0.1 4:30.37 abipkgdiff 3087305 ben 20 0 268540 42156 15720 R 99.7 0.0 28:23.80 python 3155605 ben 20 0 266940 40540 15676 R 100.0 0.0 9:20.09 python 3199415 ben 20 0 266036 39764 15728 R 100.0 0.0 0:22.19 python 3195139 ben 20 0 265384 39100 15732 R 100.0 0.0 1:41.93 python 3197990 ben 20 0 265384 39084 15716 R 100.0 0.0 0:48.76 python 3156907 ben 20 0 265384 39076 15740 R 99.7 0.0 10:42.88 python 3199255 ben 20 0 265388 38964 15788 R 99.7 0.0 0:30.69 python 3172092 ben 20 0 265388 38896 15712 R 100.0 0.0 7:19.67 python 3185309 ben 20 0 265388 38868 15712 R 99.7 0.0 4:01.49 python 3188347 ben 20 0 264996 38720 15792 R 99.7 0.0 3:21.35 python 3196118 ben 20 0 265132 38656 15696 R 100.0 0.0 1:21.71 python 3081909 ben 20 0 264996 38616 15712 R 94.7 0.0 24:21.68 python 3199647 ben 20 0 264996 38584 15724 S 99.7 0.0 0:18.55 python 3200028 ben 20 0 20860 14364 4752 D 0.7 0.0 0:00.04 rpm2cpio 3200000 ben 20 0 20868 14328 4716 D 1.3 0.0 0:00.07 rpm2cpio 3199717 ben 20 0 20784 14260 4492 D 2.7 0.0 0:00.25 rpm2cpio 3191525 ben 20 0 20784 14216 4712 D 1.0 0.0 0:00.92 rpm2cpio 3199737 ben 20 0 20860 14212 4600 D 0.7 0.0 0:00.07 rpm2cpio 3189094 ben 20 0 20860 14168 4552 D 1.3 0.0 0:02.47 rpm2cpio 3199830 ben 20 0 20824 14148 4536 D 2.3 0.0 0:00.19 rpm2cpio 3200088 ben 20 0 20860 14104 4488 R 16.9 0.0 0:00.51 rpm2cpio 3199794 ben 20 0 20860 14064 4716 D 0.7 0.0 0:00.10 rpm2cpio 3192390 ben 20 0 20860 14008 4460 D 0.3 0.0 0:00.92 rpm2cpio 3199496 ben 20 0 20856 14004 4552 D 0.3 0.0 0:00.10 rpm2cpio 3192391 ben 20 0 221068 1172 1080 S 0.3 0.0 0:01.02 cpio 3191526 ben 20 0 221068 1168 1076 S 1.0 0.0 0:00.84 cpio 3199718 ben 20 0 221068 1160 1068 S 3.0 0.0 0:00.31 cpio 3189095 ben 20 0 221068 1156 1064 S 1.3 0.0 0:02.02 cpio 3200029 ben 20 0 221068 1156 1064 S 1.0 0.0 0:00.04 cpio 3199497 ben 20 0 221068 1136 1044 S 0.3 0.0 0:00.07 cpio 3199831 ben 20 0 221068 1132 1040 S 2.7 0.0 0:00.21 cpio 3200089 ben 20 0 221068 1132 1040 R 18.3 0.0 0:00.55 cpio 3199795 ben 20 0 221068 1108 1012 S 0.3 0.0 0:00.09 cpio 3200001 ben 20 0 221068 1108 1012 S 1.7 0.0 0:00.09 cpio 20964 root 20 0 0 0 0 S 0.3 0.0 0:47.66 NFSv4 callback 2612147 root 20 0 0 0 0 I 0.3 0.0 0:00.51 kworker/9:2-events_freezable 2858412 root 20 0 0 0 0 I 0.3 0.0 0:05.50 kworker/u256:6-events_unbound 2953630 root 20 0 0 0 0 I 0.3 0.0 0:04.37 kworker/u256:0-rpciod 3096956 root 20 0 0 0 0 I 0.3 0.0 0:01.41 kworker/u256:5-events_unbound 3182633 root 20 0 0 0 0 I 0.3 0.0 0:00.34 kworker/u256:9-rpciod Furthermore fedabipkgdiff creates a cache directory of rpms. I have frequently had test terminate because I've run out of space in this directory. A notable percentage of the files that end up in the cache directory are the debuginfo packages.
fedabipkgdiff does a busy loop on waitpid() on the abipkgdiff child process: @log_call def abipkgdiff(cmp_half1, cmp_half2): [...] proc = subprocess.Popen(' '.join(cmd), shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, universal_newlines=True) # So we could have done: stdout, stderr = proc.communicate() # But then the documentatin of proc.communicate says: # # Note: The data read is buffered in memory, so do not use this # method if the data size is large or unlimited. " # # In practice, we are seeing random cases where this # proc.communicate() function does *NOT* terminate and seems to be # in a deadlock state. So we are avoiding it altogether. We are # then busy looping, waiting for the spawn process to finish, and # then we get its output. # while True: if proc.poll() != None: break Not sure that interpretation of python docs is correct, but even then, a wee bit of sleep wouldn't hurt.
One thing that I can say from experience is when there are a lot of differences, the size of output files is a few hundred thousand lines and probably amounts to a couple of megabytes of text. This is tiny in relation to the size of memory used by abipkgdiff itself when building the corpus. (I've seen up to 25G just casually watching top.) So to me it sounds like the concern that led to that design decision is unwarranted.