Breakpoint meeting

[Nurdin Premji <npremji at redhat dot com>]

So we had a meeting about the current and future state of breakpoints in
frysk. Here is a rough outline of what we talked about. 

We plan to have about 2 levels of breakpoints: High Level and Low level.
(I believe 2-3 where the exact words used but I can't see a third)

High Level Issues:

Usually  the user looks at the source code and adds breakpoints, then
starts debugging. Tagsets are these predefined breakpoints.

The high level breakpoints are from the user/source code side of things
where a user can put a break point on a function or line or such.

The issue we have in frysk is that these breakpoints must be projected
down to the currently running processes and libraries and new running
processes and libraries.

Frysk must look at every process and library and check if these break
points apply.

The reason for this is because we really don't know which processes and
libraries are going to use that code before hand. For example putting a
break point on printf.

We have to prepare for the worst case and for us the worst case is that
we know nothing (about the source code and libraries that are going to
be used)

We have to be able to handle inline functions, and inheritance and all
that object oriented, event driven fun stuff.

Linking Between high and low:
So high level user breakpoints should accumulate low level address
breakpoints.

If we implement the low level breakpoints, high level breakpoints will
follow.

We can implement a 2 level breakpoint table.

The relationship between high level breakpoints to low level breakpoints
is many to many (n to n). That is, a high level breakpoint can have can
have many low level address breakpoints. And a low level breakpoint can
be associated with many high level breakpoints.

Low Level Issues:

Low level breakpoints should keep addresses per process (since tasks can
share breakpoints)

Side note: When a process gets a trap signal, only the triggered thread
is stopped.

Thread Hop: When we have a breakpoint, the natural thing to do is have
the thread run until it hits that breakpoint, stop, and then continue
and retain the thread.

The obvious way to do this is to (on a breakpoint):
stop the thread, 
remove the breakpoint, 
single step the thread, 
add the breakpoint back, 
and then continue.

However other threads (in a multi-threaded environment) could miss the
breakpoint when we (temporarily) remove that breakpoint.

So the first possible solution is to: 
stop all threads, 
remove bp
single step the single thread,
add the bp back,
and then continue all threads. 

This adds a lot of overhead because stopping all the threads and
restarting them takes some time.

In frysk we want to try to keep things running as long as possible,
which brings up the second possible solution.

Second method involves kprobes.

When we insert the breakpoint, we know the instruction it is replacing.
So we copy the entire instruction, put it somewhere else* in the running
process
Set the thread pc to point to this copy,
single step execute,
put pc back to where it should be after the instruction,
and continue.

somewhere else*: Possible candidates: 
1. the process stack. However there may be security issues.
2. the starting point address (however only one thread can use this at
any time so we need locking)

The next issue with the low level threads is a 386 complexity
decraPCAfterBreak: For every architecture (other than 386) the
instruction address is the address of the breakpoint. For 386 the
instruction address is the address AFTER the breakpoint, thus we have to
backtrack.

So look at the instruction you are on -1, determine if that is a
breakpoint and react accordingly.

Also we have to ignore system calls because we can't do system call
tracing and single stepping at the same time. This is apparently a
kernel bug.