One usually needs to type more commands than just "set sysroot remote:".

(gdb) set sysroot remote:
(gdb) target extended-remote hostname:port
(gdb) set remote exec-file /path/to/program
(gdb) file /path/to/program - local copy
(gdb) start
 - GDB could download even exec_bfd from remote:.

(gdb) set sysroot remote:
(gdb) target remote hostname:port
(gdb) symbol-file remote:/path/to/program
(gdb) tbreak main
(gdb) continue
(gdb) sharedlibrary
 - why is symbol-file needed here when remote gdbserver already has the file?
(Unrelated bug - "sharedlibrary" command should not be needed there.)

These example come from my experiments in 2012:
  http://people.redhat.com/jkratoch/gdb2012.pdf