• MultiClient

Multiple Clients

This page describes the gdbserver multi-client project.

Goal

• Allow multiple clients that use the gdb remote protocol to multiplex the same gdbserver inferior
  • Often mentioned: gdb client and an strace client
    • strace reports syscalls in gdb's execution flow
    • stays synchronized with the gdb client
    • strace client example

      • gdb client	attached strace client
        	Process nnnnn attached in non-stop mode
        close (-1);	waiting
        (gdb) next	waiting
        waiting	close(-1) = -1 EBADF (Bad file descriptor)
        chroot (".");	waiting
        (gdb) next	waiting
        waiting	chroot(0x400970) = -1 EPERM (Operation not permitted)

Status

• Current effort concentrates on supporting the gdb remote protocol aware strace client

  • archer on the scox/strace branch

  • copr build of gdbserver

  • The gdb remote protocol aware strace

    • github strace on the gdbserver0 branch

  • copr build of strace

  • A more complete (see Future below) gdbserver work is checked into:

  • archer on the scox/globalstate branch

Plan

Current effort to support strace

• Adds multiplexing to gdbserver and uses the existing linux-low backend
• Each client needs its own state information
  • current upstream gdbserver has global data moved into a client_state structure
  • Enhancements to support multiple clients
    • Add to client_state
      • gdb_fildes_t file_desc;
      • int packet_type;
        • identifies the type of packet for this and previous packet
        • other_packet, vContc, vConts, vContr, vContt, vRun, vAttach, Hg, g_or_m, Detach, vStopped
      • int last_packet_type;
      • displaced_step_type
        • uses a heuristic, 'G' packet followed by a vConts to indicate that we may be doing displaced stepping
      • int pending;
        • none_pending, pending_waitee, pending_cont_waiter
      • int nonstop_pending;
        • no_notifier_pending, pending_notifier, pending_notified
      • int catch_syscalls;
      • int syscall_op
        • NO_SYSCALL, ANY_SYSCALL, XXX_SYSCALL
        • whether or not and which syscalls to monitor
      • ptid_t last_cont_ptid;
      • ptid_t new_general_thread;
      • struct target_waitstatus last_cont_status;
  • Add a multi_client_states mapping from file descriptor to client_state
    • "indexed" by file descriptor of the client
    • add_client_by_pid add another client state coordinating the same pid
      • e.g. client 1 does 'run program' and client 2 is an strace that attaches to the resulting pid
    • client_states.current_cs
      • There is always a current client; this points to it
      • get_client_state() returns client_states.current_cs

access functions

• add_client_by_pid add another client state coordinating the same pid
  • for example client 1 does 'run program' and client 2 is an strace that attaches to the resulting pid
• client_state::copy_status copies those state items that are in common for both clients
• get_first_client
  • The initial client holds the state when gdb begins in captured_main
  • get_first_client returns the first active client.
• have_multiple_clients
  • Is there more than one client?

when an additional client connects

• event-loop.c

  * wait_for_event

  now adds a select in case another client tries to connect. add a new client if so.

  * handle_file_event

  Set the client state if we are attaching to a client already managed by gdbserver

• remote-utils.c

  * get_remote_desc, get_listen_desc

  allow fds to be accessed outside remote-utils.c

  * handle_accept_event

  Set the default client_state to the new fd and don't close the listen_desc

  * write_prim, read_prim, readchar, putpkt, getpkt

  add a parameter for fd

• start_event_loop => process_event => handle_file_event

  • => handle_accept_event => add_file_handler to add an additional client's file descriptor

• start_event_loop => wait_for_event to add another client

• process_serial_event
  • set_client_state creates a new client state
• If this is a client attaching to an existing gdbserver executable
  • e.g. an strace client
  • add_client_by_pid adds another client_state to an existing server_state
• Example
  • Packet/FileID (Args)
  • getpkt/9 ("vAttach;PID")
  • putpkt/9 ("$OK")
  • getpkt/9 ("vCont;t:pPID.-1")
  • putpkt/9 ("$OK")

  • putpkt/9 ("%Stop:T05swbreak:;06:..")

  • getpkt/9 ("vStopped")

  • results in:

  • fd type pending current client
  • 4 g_or_m not-waiting y gdb
  • 9 vStopped not-waiting n strace

primary additions for multiplexing clients

• clients have a state and an input, and are managed as a state machine
  • states: not-waiting waitee waiter step-waiter
  • notifier states: none notifier notified
    • these are used for non-stop notifications
  • inputs: other, vContc vConts vContr vContt vRun vAttach Hg g_or_m Detach vStopped
    • these correspond to input packets and are returned by get_packet_type
  • get_packet_type sets the packet's state and input
• One client is always current and one is waiting.
  • For a syscall the syscall client becomes the current client
  • While the gdb client waits
  • For a continue, next, or step the gdb client potentially waits if a syscall is reached
• setup_multiplexing is invoked before packet handling in server.c::process serial event
  • setup_multiplexing: Determine the state of all the clients for a given inferior prior to handle_serial_event's packet handling switch
  • If current client was doing continue or next, make it wait if the strace client is handling a syscall
  • These situations do not happen often with a gdb / strace client combination
    • do_multiplexing (below) handles most of the client switch situations+
  • e.g. client 1 is at gdb request level
  • current client 1 "not waiting" / "vContc" and client 2 "not waiting"
    • becomes client 1 "waiter" / "vContc" (or "vConts") and current client 2 "waitee"
    • no further processing is done for client 1 so it waits
  • another example: client 1 "waiting" / "vConts" and current client 2 "not waiting" "vContc"
    • becomes current client 1 "waitee" / "vConts" and client 2 "waiter" "vContc"
  • another example: client 1 "waiter" / "vContc" and current client 2 "not waiting" "vContc"
    • No change, the client 2 vContc is handled
  • client1 packet type / client1 state / client2 state: new states
  • vContc / * / none pending: cont waiter / waitee
  • vConts / waitee / none pending: step waiter / waitee
  • vContc / waitee / step waiter: cont waiter / waitee
  • vConts / waitee / step waiter: step waiter / waitee
  • vContc / waitee / cont waiter: cont waiter / *
  • vConts / waitee / cont waiter: step waiter / *
  • vContc / * / waitee: cont waiter / *
  • vConts / * / waitee: step waiter / *
  • : If transitioning to a waiter then return without handling the
    • packet The client awaits a reply.
• do_multiplexing is invoked after packet handling
  • Also a state machine that determines if a client's waiting should be resolved
  • do_multiplexing: Resolve the state of all the clients for a given inferior after the packet handling switch
  • Handles the case where the current client is continuing: has a vContc or vConts packet
  • typically for an all stop client
  • e.g. when a gdb client does a continue and a syscall is hit,
  • resolve_waiter does the work of replying to a waiting client
    • handle_status gets the status which is passed to the waiting client
  • The non stop case just sets the client state to pending_notifier
  • The remainder of the non stop work is done in notify_clients
  • An exception is vCont;t, e.g. when attaching, which is special cased
  • Example
    • getpkt/9 ("vCont;c")
    • ...
    • getpkt/4 ("vCont;s:p25f0.25f0")
    • ...
    • getpkt/4 ("vCont;c:p25f0.-1")

    • begin do_multiplexing

    • fd type pending current client
    • 4 vContc waitee y gdb client
    • 9 vContc waiter n syscall client
    • putpkt/9 ("$T05syscall_entry:e;06:xxx;07:xxx...")

    • end do_multiplexing the client state becomes

    • 4 vContc waiter y
    • 9 vContc waitee n
    • getpkt/9 ("vCont;c")

    • begin do_multiplexing

    • client 9 continue causes client 4 breakpoint to be hit

    • putpkt/4 ("$T05swbreak:;06:xxx;07:xxx...")

    • end do_multiplexing the client state becomes

    • 4 vContc waitee n
    • 9 vContc waiter y
• resolve_waiter wakes up a waiting client
  • If a client does a continue then it goes into the waiter state and the other client is in the waitee state.
  • When the waitee continues and the execution point reaches the waiter breakpoint then that client is resolved, as if it had not been waiting.
  • for a vContc packet
    • for all_stop mode: call handle_status and putpkt the result
    • for non stop mode: set pending_notifier, indicating it is handled later in notify_clients
  • for a vConts packet
    • for all_stop mode: the current client packet is handled and the status is available so send it to the waiting client
    • for non stop mode: set pending_notifier
  • for a vContt packet
    • the notification is created on the fly and sent to the waiting client
• notify_clients notifies an attached client for non-stop mode
  • A notification has been created and is about to be sent to the waiting client
  • invoked by notif_write_event when it outputs successive notifications.
  • determines if the notification should be sent to another client
    • as for above do_multiplexing case, if client 2 has reached a client 1 breakpoint then client 1 is sent the notification
    • additionally, syscall clients, i.e. strace, are always sent syscall notifications
  • notify_clients replies to a waiting non stop "pending_notifier" client
    • similar in organization to do_multiplexing
      • putpkt the buffer instead of resolve_waiter
  • notify_clients is called by the notification mechanism
    • There can be 1 or more notifications in a notification packet
    • invoked by notif_push when it outputs the first notification

      • process_event calls handle_file_event => handle_target_event => push_stop_notification => notif_push => { notif_event_enque;

        • (as np->write)vstop_notif_reply => prepare_resume_reply;

          • notify_clients; putpkt_notif}
    • invoked by notif_write_event when it outputs successive notifications.

      • handle_v_requests => handle_notif_ack => notif_write_event => { (as notif->write)vstop_notif_reply ; notify_clients }

    • The non stop case is passed the status buffer by the notify mechanism
    • Example
      • getpkt/4 ("vCont;c:p2bbd.2bbd")
      • putpkt/4 ("$OK#9a")

      • begin notify_clients

      • fd type pending current client
      • 4 vContc waitee y
      • 9 vContc waiter n
      • putpkt/9 ("$OK#9a")

      • putpkt/9 ("%Stop:T05syscall_entry:e;06:xxx;07:xxx...")

      • end notify_clients the client state becomes

      • 4 vContc waiter y
      • 9 vContc waitee n
      • getpkt/11 ("vStopped")
      • putpkt/11 ("OK#9a")
      • getpkt/11 ("vCont;c")
      • putpkt/11 ("OK#9a")

      • begin notify_clients notify is for gdb client breakpoint

      • putpkt/4 ("%Stop:T05swbreak:;06:xxx;07:xxx...")

      • end notify_clients the client state becomes

      • 4 vContc waitee n
      • 9 vContc waiter y
      • getpkt/4 ("vStopped")
  • Two thread components in a stop reply
    • There are situations where the reported thread does not match the
      • general thread.
    • strace is interested in the reported thread but the waiting gdb
      • client is interested in the general thread.
    • To assist strace a non stop reply has two thread components
    • The first component is the syscall thread and the second is the general thread.
    • Example
      • getpkt/4 ("vCont;c:p2bbd.2bbd")
      • ...

      • putpkt/9 ("%Stop:T05syscall_entry:111;06:xxx,;07:xxx;thread:p2bbd.2bc7;thread:p2bbd.2bbd;core:4;");

      • # 1st thread is the syscall thread, 2nd thread is the general thread

      • # since the two ptids don't match we don't continue a specific ptid

      • getpkt/9 ("vCont;c")

      • putpkt/9 ("%Stop:T05syscall_entry:111;06:xxx,;07:xxx;thread:p2bbd.2bc8;thread:p2bbd.2bc8;core:4;");

      • # The syscall thread and general thread now match so we continue the syscall ptid

      • getpkt/9 ("vCont;c:p2bbd.2bc8")

Other additions for multiplexing clients

• server.c
  • get_client_state: There is always a current client; this returns a pointer to that client
  • set_client_state: This changes the current client; creating it if it does not exist
  • attached_to_same_proc: Determines if two clients share the same inferior
  • dump_client_state: Dump the client state table
  • add_client_by_pid: Setup clients to share a common inferior's server
    • state
  • get_first_client: Get the first client in the client list
  • have_multiple_clients: Is there more than one client?
  • delete_client_state: Remove the client state corresponding to a fd.
  • get_packet_type: Get the type of the packet, e.g. vCont;c, vRun, ...
  • analyze_group: Determines if all the clients for a given inferior are
    • waiters or waitees
• event-loop.c
  • wait_for_event: now adds a select in case another client tries to connect. add a new client if so.
  • handle_file_event: Set the client state if we are attaching to a client already managed by gdbserver
• remote-utils.c
  • get_remote_desc, get_listen_desc: allow fds to be accessed outside remote-utils.c
  • handle_accept_event: Set the default client_state to the new fd and don't close the listen_desc
    • write_prim, read_prim, readchar, putpkt, getpkt: add a parameter for fd

Future

support an additional gdb client

• gdb client and another gdb client, e.g. customer and support engineer
  • clients stay synchronized so both have the same concept of the PC
  • if client 1 continues, client 1 waits
    • until client2 execution reaches the client 1 breakpoint
  • if client 1 steps, client 1 waits
    • until client 2 steps
  • gdb clients example

    • gdb client 1	gdb client 2
      Breakpoint 1, main () at bench.c:1148	main () at bench.c:1148
      permute_cycles = calloc(cycles_hbound,sizeof(long));	permute_cycles = calloc(cycles_hbound,sizeof(long));
      (gdb) b queens	(gdb) b puzzle
      (gdb) continue	(gdb)
      (gdb) waiting	(gdb) continue
      Breakpoint 2, queens () at bench.c:443	waiting
      for (i = 1; i <= 50; i++)	waiting
      (gdb) next	waiting
      (gdb) step	waiting
      doit () at bench.c:422	waiting
      i = 0 - 7;	waiting
      (gdb) continue	waiting
      waiting	Breakpoint 1, puzzle () at bench.c:613
      waiting	for (m = 0; m <= puzzle_size; m++)
      waiting	(gdb) print puzzl
      waiting	$2 = {0 <repeats 512 times>}

• mem-break.c: Allow per client breakpoints
• has_client_breakpoint_at: Given multiple clients for the same process does clientA have a break at location addr.
• add_client_breakpoint: Each client has a breakpoint view, which may be a
  • subset of the breakpoints managed by gdbserver for the inferior
• delete_client_breakpoint: Likewise
• To Be Continued...

Dyninst take 2

• An early effort to use Dyninst as a backend added dyninst-low.cc
  • dyninst-low.cc needed to mimic linux-low.c functionality. A better approach may be: similar to the strace approach:
  • add an abstraction layer for the linux-low.c ptrace interface
  • since dyninst manages ptrace itself.
  • allow either dyninst or ptrace equivalents to be plugged in.