Differences between revisions 9 and 10
Revision 9 as of 2018-09-16 18:04:27
Size: 16759
Editor: StanCox
Comment:
Revision 10 as of 2019-08-23 21:15:03
Size: 17644
Editor: StanCox
Comment:
Deletions are marked like this. Additions are marked like this.
Line 1: Line 1:
h## Please edit system and help pages ONLY in the master wiki! ## Please edit system and help pages ONLY in the master wiki!
Line 35: Line 35:
  * [[https://copr.fedorainfracloud.org/coprs/scox/gdbserver/|copr build of gdbserver]]
Line 37: Line 38:
  * Current gdbserver work is checked into:   * [[https://copr.fedorainfracloud.org/coprs/scox/strace/|copr build of strace]]
  * A more complete (see Future below) gdbserver work is checked into:
Line 51: Line 53:
    * int attached_to_client;
Line 69: Line 70:
  * Add a client_states mapping from file descriptor to client_state
    * client_states.current_cs
 
* There is always a current client; this points to it
 * client_states.current_fd
* get_client_state() returns client_states.current_cs

* Add a multi_client_states mapping from file descriptor to client_state
   * "indexed" by file descriptor of the client
   * add_client_by_pid add another client state coordinating the same pid
    * e
.g. client 1 does 'run program' and client 2 is an strace that attaches to the resulting pid
   * client_states.current_cs
* There is always a current client; this points to it
     * get_client_state() returns client_states.current_cs
Line 86: Line 90:
 * event-loop.c
   * wait_for_event :: now adds a select in case another client tries to connect. add a new client if so.
   * handle_file_event :: Set the client state if we are attaching to a client already managed by gdbserver
 * remote-utils.c
   * get_remote_desc, get_listen_desc :: allow fds to be accessed outside remote-utils.c
   * handle_accept_event :: Set the default client_state to the new fd and don't close the listen_desc
   * write_prim, read_prim, readchar, putpkt, getpkt :: add a parameter for fd
Line 207: Line 218:
     * There can 1 or more notifications in a notification packet      * There can be 1 or more notifications in a notification packet
Line 268: Line 279:
  * free_client_state: Remove a client state
Line 286: Line 296:
  * mem-break.c: Allow per client breakpoints
  * has_client_breakpoint_at: Given multiple clients for the same process does clientA have a break at location addr.
Line 318: Line 326:
  * mem-break.c: Allow per client breakpoints
  * has_client_breakpoint_at: Given multiple clients for the same process does clientA have a break at location addr.
Line 326: Line 336:
 * Dyninst take 1 added dyninst-low.cc
  dyninst-low.cc needed to mimic linux-low.c functionality. Now that
  gdb is moving towards g++, a better approach may be:
  * abstraction layer for the linux-low.c ptrace interface
  * since dyninst manages the ptrace itself.
 * An early effort to use Dyninst as a backend added dyninst-low.cc
  dyninst-low.cc needed to mimic linux-low.c functionality. A better approach may be:
  similar to the strace approach:
  * add an abstraction layer for the linux-low.c ptrace interface
  * since dyninst manages ptrace itself.

Multiple Clients

This page describes the gdbserver multi-client project.

Goal

  • Allow multiple clients that use the gdb remote protocol to multiplex the same gdbserver inferior
    • Often mentioned: gdb client and an strace client
      • strace reports syscalls in gdb's execution flow
      • stays synchronized with the gdb client
      • strace client example
        • gdb client

          attached strace client

          Process nnnnn attached in non-stop mode

          close (-1);

          waiting

          (gdb) next

          waiting

          waiting

          close(-1) = -1 EBADF (Bad file descriptor)

          chroot (".");

          waiting

          (gdb) next

          waiting

          waiting

          chroot(0x400970) = -1 EPERM (Operation not permitted)

Status

  • Current effort concentrates on supporting the gdb remote protocol aware strace client

Plan

Current effort to support strace

  • Adds multiplexing to gdbserver and uses the existing linux-low backend
  • Each client needs its own state information
    • current upstream gdbserver has global data moved into a client_state structure
    • Enhancements to support multiple clients
      • Add to client_state
        • gdb_fildes_t file_desc;
        • int packet_type;
          • identifies the type of packet for this and previous packet
          • other_packet, vContc, vConts, vContr, vContt, vRun, vAttach, Hg, g_or_m, Detach, vStopped
        • int last_packet_type;
        • displaced_step_type
          • uses a heuristic, 'G' packet followed by a vConts to indicate that we may be doing displaced stepping
        • int pending;
          • none_pending, pending_waitee, pending_cont_waiter
        • int nonstop_pending;
          • no_notifier_pending, pending_notifier, pending_notified
        • int catch_syscalls;
        • int syscall_op
          • NO_SYSCALL, ANY_SYSCALL, XXX_SYSCALL
          • whether or not and which syscalls to monitor
        • ptid_t last_cont_ptid;
        • ptid_t new_general_thread;
        • struct target_waitstatus last_cont_status;
    • Add a multi_client_states mapping from file descriptor to client_state
      • "indexed" by file descriptor of the client
      • add_client_by_pid add another client state coordinating the same pid
        • e.g. client 1 does 'run program' and client 2 is an strace that attaches to the resulting pid
      • client_states.current_cs
        • There is always a current client; this points to it
        • get_client_state() returns client_states.current_cs

access functions

  • add_client_by_pid add another client state coordinating the same pid
    • for example client 1 does 'run program' and client 2 is an strace that attaches to the resulting pid
  • client_state::copy_status copies those state items that are in common for both clients
  • get_first_client
    • The initial client holds the state when gdb begins in captured_main
    • get_first_client returns the first active client.
  • have_multiple_clients
    • Is there more than one client?

when an additional client connects

  • event-loop.c
    * wait_for_event
    now adds a select in case another client tries to connect. add a new client if so.
    * handle_file_event
    Set the client state if we are attaching to a client already managed by gdbserver
  • remote-utils.c
    * get_remote_desc, get_listen_desc
    allow fds to be accessed outside remote-utils.c
    * handle_accept_event
    Set the default client_state to the new fd and don't close the listen_desc
    * write_prim, read_prim, readchar, putpkt, getpkt
    add a parameter for fd
  • start_event_loop => process_event => handle_file_event

    • => handle_accept_event => add_file_handler to add an additional client's file descriptor

  • start_event_loop => wait_for_event to add another client

  • process_serial_event
    • set_client_state creates a new client state
  • If this is a client attaching to an existing gdbserver executable
    • e.g. an strace client
    • add_client_by_pid adds another client_state to an existing server_state
  • Example
    • Packet/FileID (Args)
    • getpkt/9 ("vAttach;PID")
    • putpkt/9 ("$OK")
    • getpkt/9 ("vCont;t:pPID.-1")
    • putpkt/9 ("$OK")
    • putpkt/9 ("%Stop:T05swbreak:;06:..")

    • getpkt/9 ("vStopped")
    • results in:

    • fd type pending current client
    • 4 g_or_m not-waiting y gdb
    • 9 vStopped not-waiting n strace

primary additions for multiplexing clients

  • clients have a state and an input, and are managed as a state machine
    • states: not-waiting waitee waiter step-waiter
    • notifier states: none notifier notified
      • these are used for non-stop notifications
    • inputs: other, vContc vConts vContr vContt vRun vAttach Hg g_or_m Detach vStopped
      • these correspond to input packets and are returned by get_packet_type
    • get_packet_type sets the packet's state and input
  • One client is always current and one is waiting.
    • For a syscall the syscall client becomes the current client
    • While the gdb client waits
    • For a continue, next, or step the gdb client potentially waits if a syscall is reached
  • setup_multiplexing is invoked before packet handling in server.c::process serial event
    • setup_multiplexing: Determine the state of all the clients for a given inferior prior to handle_serial_event's packet handling switch
    • If current client was doing continue or next, make it wait if the strace client is handling a syscall
    • These situations do not happen often with a gdb / strace client combination
      • do_multiplexing (below) handles most of the client switch situations+
    • e.g. client 1 is at gdb request level
    • current client 1 "not waiting" / "vContc" and client 2 "not waiting"
      • becomes client 1 "waiter" / "vContc" (or "vConts") and current client 2 "waitee"
      • no further processing is done for client 1 so it waits
    • another example: client 1 "waiting" / "vConts" and current client 2 "not waiting" "vContc"
      • becomes current client 1 "waitee" / "vConts" and client 2 "waiter" "vContc"
    • another example: client 1 "waiter" / "vContc" and current client 2 "not waiting" "vContc"
      • No change, the client 2 vContc is handled
    • client1 packet type / client1 state / client2 state: new states
    • vContc / * / none pending: cont waiter / waitee
    • vConts / waitee / none pending: step waiter / waitee
    • vContc / waitee / step waiter: cont waiter / waitee
    • vConts / waitee / step waiter: step waiter / waitee
    • vContc / waitee / cont waiter: cont waiter / *
    • vConts / waitee / cont waiter: step waiter / *
    • vContc / * / waitee: cont waiter / *
    • vConts / * / waitee: step waiter / *
    • : If transitioning to a waiter then return without handling the
      • packet The client awaits a reply.
  • do_multiplexing is invoked after packet handling
    • Also a state machine that determines if a client's waiting should be resolved
    • do_multiplexing: Resolve the state of all the clients for a given inferior after the packet handling switch
    • Handles the case where the current client is continuing: has a vContc or vConts packet
    • typically for an all stop client
    • e.g. when a gdb client does a continue and a syscall is hit,
    • resolve_waiter does the work of replying to a waiting client
      • handle_status gets the status which is passed to the waiting client
    • The non stop case just sets the client state to pending_notifier
    • The remainder of the non stop work is done in notify_clients
    • An exception is vCont;t, e.g. when attaching, which is special cased
    • Example
      • getpkt/9 ("vCont;c")
      • ...
      • getpkt/4 ("vCont;s:p25f0.25f0")
      • ...
      • getpkt/4 ("vCont;c:p25f0.-1")
      • begin do_multiplexing

      • fd type pending current client
      • 4 vContc waitee y gdb client
      • 9 vContc waiter n syscall client
      • putpkt/9 ("$T05syscall_entry:e;06:xxx;07:xxx...")
      • end do_multiplexing the client state becomes

      • 4 vContc waiter y
      • 9 vContc waitee n
      • getpkt/9 ("vCont;c")
      • begin do_multiplexing

      • client 9 continue causes client 4 breakpoint to be hit

      • putpkt/4 ("$T05swbreak:;06:xxx;07:xxx...")
      • end do_multiplexing the client state becomes

      • 4 vContc waitee n
      • 9 vContc waiter y
  • resolve_waiter wakes up a waiting client
    • If a client does a continue then it goes into the waiter state and the other client is in the waitee state.
    • When the waitee continues and the execution point reaches the waiter breakpoint then that client is resolved, as if it had not been waiting.
    • for a vContc packet
      • for all_stop mode: call handle_status and putpkt the result
      • for non stop mode: set pending_notifier, indicating it is handled later in notify_clients
    • for a vConts packet
      • for all_stop mode: the current client packet is handled and the status is available so send it to the waiting client
      • for non stop mode: set pending_notifier
    • for a vContt packet
      • the notification is created on the fly and sent to the waiting client
  • notify_clients notifies an attached client for non-stop mode
    • A notification has been created and is about to be sent to the waiting client
    • invoked by notif_write_event when it outputs successive notifications.
    • determines if the notification should be sent to another client
      • as for above do_multiplexing case, if client 2 has reached a client 1 breakpoint then client 1 is sent the notification
      • additionally, syscall clients, i.e. strace, are always sent syscall notifications
    • notify_clients replies to a waiting non stop "pending_notifier" client
      • similar in organization to do_multiplexing
        • putpkt the buffer instead of resolve_waiter
    • notify_clients is called by the notification mechanism
      • There can be 1 or more notifications in a notification packet
      • invoked by notif_push when it outputs the first notification
        • process_event calls handle_file_event => handle_target_event => push_stop_notification => notif_push => { notif_event_enque;

          • (as np->write)vstop_notif_reply => prepare_resume_reply;

            • notify_clients; putpkt_notif}
      • invoked by notif_write_event when it outputs successive notifications.
        • handle_v_requests => handle_notif_ack => notif_write_event => { (as notif->write)vstop_notif_reply ; notify_clients }

      • The non stop case is passed the status buffer by the notify mechanism
      • Example
        • getpkt/4 ("vCont;c:p2bbd.2bbd")
        • putpkt/4 ("$OK#9a")
        • begin notify_clients

        • fd type pending current client
        • 4 vContc waitee y
        • 9 vContc waiter n
        • putpkt/9 ("$OK#9a")
        • putpkt/9 ("%Stop:T05syscall_entry:e;06:xxx;07:xxx...")

        • end notify_clients the client state becomes

        • 4 vContc waiter y
        • 9 vContc waitee n
        • getpkt/11 ("vStopped")
        • putpkt/11 ("OK#9a")
        • getpkt/11 ("vCont;c")
        • putpkt/11 ("OK#9a")
        • begin notify_clients notify is for gdb client breakpoint

        • putpkt/4 ("%Stop:T05swbreak:;06:xxx;07:xxx...")

        • end notify_clients the client state becomes

        • 4 vContc waitee n
        • 9 vContc waiter y
        • getpkt/4 ("vStopped")
    • Two thread components in a stop reply
      • There are situations where the reported thread does not match the
        • general thread.
      • strace is interested in the reported thread but the waiting gdb
        • client is interested in the general thread.
      • To assist strace a non stop reply has two thread components
      • The first component is the syscall thread and the second is the general thread.
      • Example
        • getpkt/4 ("vCont;c:p2bbd.2bbd")
        • ...
        • putpkt/9 ("%Stop:T05syscall_entry:111;06:xxx,;07:xxx;thread:p2bbd.2bc7;thread:p2bbd.2bbd;core:4;");

        • # 1st thread is the syscall thread, 2nd thread is the general thread

        • # since the two ptids don't match we don't continue a specific ptid

        • getpkt/9 ("vCont;c")
        • putpkt/9 ("%Stop:T05syscall_entry:111;06:xxx,;07:xxx;thread:p2bbd.2bc8;thread:p2bbd.2bc8;core:4;");

        • # The syscall thread and general thread now match so we continue the syscall ptid

        • getpkt/9 ("vCont;c:p2bbd.2bc8")

Other additions for multiplexing clients

  • server.c
    • get_client_state: There is always a current client; this returns a pointer to that client
    • set_client_state: This changes the current client; creating it if it does not exist
    • attached_to_same_proc: Determines if two clients share the same inferior
    • dump_client_state: Dump the client state table
    • add_client_by_pid: Setup clients to share a common inferior's server
      • state
    • get_first_client: Get the first client in the client list
    • have_multiple_clients: Is there more than one client?
    • delete_client_state: Remove the client state corresponding to a fd.
    • get_packet_type: Get the type of the packet, e.g. vCont;c, vRun, ...
    • analyze_group: Determines if all the clients for a given inferior are
      • waiters or waitees
  • event-loop.c
    • wait_for_event: now adds a select in case another client tries to connect. add a new client if so.
    • handle_file_event: Set the client state if we are attaching to a client already managed by gdbserver
  • remote-utils.c
    • get_remote_desc, get_listen_desc: allow fds to be accessed outside remote-utils.c
    • handle_accept_event: Set the default client_state to the new fd and don't close the listen_desc
      • write_prim, read_prim, readchar, putpkt, getpkt: add a parameter for fd

Future

support an additional gdb client

  • gdb client and another gdb client, e.g. customer and support engineer
    • clients stay synchronized so both have the same concept of the PC
    • if client 1 continues, client 1 waits
      • until client2 execution reaches the client 1 breakpoint
    • if client 1 steps, client 1 waits
      • until client 2 steps
    • gdb clients example
      • gdb client 1

        gdb client 2

        Breakpoint 1, main () at bench.c:1148

        main () at bench.c:1148

        permute_cycles = calloc(cycles_hbound,sizeof(long));

        permute_cycles = calloc(cycles_hbound,sizeof(long));

        (gdb) b queens

        (gdb) b puzzle

        (gdb) continue

        (gdb)

        (gdb) waiting

        (gdb) continue

        Breakpoint 2, queens () at bench.c:443

        waiting

        for (i = 1; i <= 50; i++)

        waiting

        (gdb) next

        waiting

        (gdb) step

        waiting

        doit () at bench.c:422

        waiting

        i = 0 - 7;

        waiting

        (gdb) continue

        waiting

        waiting

        Breakpoint 1, puzzle () at bench.c:613

        waiting

        for (m = 0; m <= puzzle_size; m++)

        waiting

        (gdb) print puzzl

        waiting

        $2 = {0 <repeats 512 times>}

  • mem-break.c: Allow per client breakpoints
  • has_client_breakpoint_at: Given multiple clients for the same process does clientA have a break at location addr.
  • add_client_breakpoint: Each client has a breakpoint view, which may be a
    • subset of the breakpoints managed by gdbserver for the inferior
  • delete_client_breakpoint: Likewise
  • To Be Continued...

Dyninst take 2

  • An early effort to use Dyninst as a backend added dyninst-low.cc
    • dyninst-low.cc needed to mimic linux-low.c functionality. A better approach may be: similar to the strace approach:
    • add an abstraction layer for the linux-low.c ptrace interface
    • since dyninst manages ptrace itself.
    • allow either dyninst or ptrace equivalents to be plugged in.

None: MultiClient (last edited 2019-08-23 21:15:03 by StanCox)

All content (C) 2008 Free Software Foundation. For terms of use, redistribution, and modification, please see the WikiLicense page.