Displaced stepping not always working as expected

[Marc Khouzam <marc dot khouzam at ericsson dot com>]

Hi,

I just need a hint on where next to look...

I've been asked to look into problems with non-stop on 
a user-mode-linux virtual machine
(http://user-mode-linux.sourceforge.net/)

On that AMD 64bit machine, I cannot step or resume past a breakpoint
when using non-stop with a multi-threaded program _if_ any of the
threads is still running.  If I interrupt all threads, then displaced
stepping works.

During the failure case, I confirmed that the displaced
instruction does _not_ get executed (the memory it should have 
changed stays the same).  So, the PC stays in the same place
and the step does not move forward.

I tried to turn on 'set debug infrun 1', but I get the exact same
logs during the failure as during a success case.

Sometimes, if I keep trying to step, it will finally work (could be
after 3 attempts, could be after 100 attempts or more).  It seems
related to what the other running thread is doing at the time.

Can someone let me know where in GDB I can look to see why a displaced
instruction is not being executed?  Or maybe other debug logs to enable?

For more details, below are stripped logs showing the problem as
concisely as possible.

Thanks a lot for any guidance

Marc




Displaced logs showing PC stuck:
===============================
displaced: stepping Thread 0x40b21940 (LWP 763) now
displaced: saved 0x4006d2: 49 89 d1 5e 48 89 e2 48 83 e4 f0 50 54 49 c7 c0 
displaced: copy 0x40083e->0x4006d2: 83 6d fc 01 8b 75 fc bf 8c 09 40 00 b8 00 00 00 
====> PC being displaced
displaced: displaced pc to 0x4006d2
====> Instruction being run
displaced: run 0x4006d2: 83 6d fc 01
               ^^^^^^^^
displaced: restored 0x4006d2
displaced: fixup (0x40083e, 0x4006d2), insn = 0x83 0x6d ...
====> PC being relocated from the same address 
====> as the displaced instruction!
displaced: relocated %rip from 0x4006d2 to 0x40083e
                               ^^^^^^^^
'next' operation stuck at line 9 of my program:
==============================================
(gdb) n
infrun: clear_proceed_status_thread (Thread 0x40b21940 (LWP 763))
infrun: proceed (addr=0xffffffffffffffff, signal=144, step=1)
infrun: resume (step=1, signal=0), trap_expected=1
displaced: stepping Thread 0x40b21940 (LWP 763) now
displaced: saved 0x4006d2: 49 89 d1 5e 48 89 e2 48 83 e4 f0 50 54 49 c7 c0 
displaced: copy 0x40083e->0x4006d2: 83 6d fc 01 8b 75 fc bf 8c 09 40 00 b8 00 00 00 
displaced: displaced pc to 0x4006d2
displaced: run 0x4006d2: 83 6d fc 01 
infrun: target_wait (-1, status) =
infrun:   760 [Thread 0x40b21940 (LWP 763)],
infrun:   status->kind = stopped, signal = SIGTRAP
infrun: Switching context from Thread 0x40b21940 (LWP 763) to Thread 0x40b21940 (LWP 763)
infrun: infwait_normal_state
infrun: TARGET_WAITKIND_STOPPED
displaced: restored 0x4006d2
displaced: fixup (0x40083e, 0x4006d2), insn = 0x83 0x6d ...
displaced: relocated %rip from 0x4006d2 to 0x40083e
infrun: stop_pc = 0x40083e
infrun: BPSTAT_WHAT_STOP_NOISY
infrun: stop_stepping

Breakpoint 2, thread_exec1 (ptr=0x40095c) at multithread.c:9
9               i--;
(gdb) infrun: target_wait (-1, status) =
infrun:   -1 [process -1],
infrun:   status->kind = ignore
infrun: TARGET_WAITKIND_IGNORE
infrun: prepare_to_wait


'next' operation that finally gets to line 10:
(exact same output as failure except PC gets incremented)
=========================================================
(gdb) n
infrun: clear_proceed_status_thread (Thread 0x40b21940 (LWP 763))
infrun: proceed (addr=0xffffffffffffffff, signal=144, step=1)
infrun: resume (step=1, signal=0), trap_expected=1
displaced: stepping Thread 0x40b21940 (LWP 763) now
displaced: saved 0x4006d2: 49 89 d1 5e 48 89 e2 48 83 e4 f0 50 54 49 c7 c0 
displaced: copy 0x40083e->0x4006d2: 83 6d fc 01 8b 75 fc bf 8c 09 40 00 b8 00 00 00 
displaced: displaced pc to 0x4006d2
displaced: run 0x4006d2: 83 6d fc 01 
infrun: target_wait (-1, status) =
infrun:   760 [Thread 0x40b21940 (LWP 763)],
infrun:   status->kind = stopped, signal = SIGTRAP
infrun: Switching context from Thread 0x40b21940 (LWP 763) to Thread 0x40b21940 (LWP 763)
infrun: infwait_normal_state
infrun: TARGET_WAITKIND_STOPPED
displaced: restored 0x4006d2
displaced: fixup (0x40083e, 0x4006d2), insn = 0x83 0x6d ...
displaced: relocated %rip from 0x4006d6 to 0x400842
infrun: stop_pc = 0x400842
infrun: stepped to a different line
infrun: stop_stepping
10              printf("in the second thread %d\n", i);
(gdb) infrun: target_wait (-1, status) =
infrun:   -1 [process -1],
infrun:   status->kind = ignore
infrun: TARGET_WAITKIND_IGNORE
infrun: prepare_to_wait