Inconsistent usage on onebyte_modrm and twobyte_modrm table in x86 disassembler and gdb?

[Jiang, Haochen]

Hi all,

Recently I happened to have a look at the gdb code. At gdb/amd64-tdep.c
L1102 comment, it mentioned that:

/* WARNING: Keep onebyte_has_modrm, twobyte_has_modrm in sync with
   ../opcodes/i386-dis.c (until libopcodes exports them, or an alternative,
   at which point delete these in favor of libopcodes' versions).  */

This means the table content and usage should be the same as gas.

However, when we are using the table in disassembler at opcode/i386-dis.c
L9877, it is:

  /* REX2.M in rex2 prefix represents map0 or map1.  */
  if (ins.last_rex2_prefix < 0 ? *ins.codep == 0x0f : (ins.rex2 & REX2_M))
    {
      if (!ins.rex2)
        {
          ins.codep++;
          if (!fetch_code (info, ins.codep + 1))
            goto fetch_error_out;
        }

      dp = &dis386_twobyte[*ins.codep];
      ins.need_modrm = twobyte_has_modrm[*ins.codep];
    }
  else
    {
      dp = &dis386[*ins.codep];
      ins.need_modrm = onebyte_has_modrm[*ins.codep];
    }

It will use the very first byte of the bytecode.

On the other hand, in gdb, let's take VEX prefix as example at
gdb/amd64-tdep.c L1349, the logic is:

  /* Skip REX/VEX instruction encoding prefixes.  */
  ...
  else if (vex2_prefix_p (*insn))
    {
      details->enc_prefix_offset = insn - start;
      insn += 2;
    }
  else if (vex3_prefix_p (*insn))
    {
      details->enc_prefix_offset = insn - start;
      insn += 3;
    }
  ...
  if (prefix != nullptr && rex2_prefix_p (*prefix))
    {
      ...
    }
  else if (prefix != nullptr && vex2_prefix_p (*prefix))
    {
      need_modrm = twobyte_has_modrm[*insn];
      details->opcode_len = 2;
    }
  else if (prefix != nullptr && vex3_prefix_p (*prefix))
    {
      need_modrm = twobyte_has_modrm[*insn];
      ...
}
...

It will skip the VEX prefix and use twobyte_has_modrm table instead of
onebyte_has_modrm[0xc4/c5] in disassembler. The table usage are totally
different although the table itself is the same. It will cause the need_modrm
value different eventually. For example, opcode for VPBLENDW under 128 bit
is "VEX.128.66.0F3A.WIG 0E /r ib". The need_modrm would be false in gdb
since twobyte_has_modrm[0x0e] is false.

Does anyone know the reason on that? It is weird to me.

BTW, the content of the two table is inconsistent currently due to the ud*
added in 2019. No matter what the answer of the previous question is, we
need to fix it. Maybe we should also add a comment at opcode/i386-dis.c
to have a reminder.

Thx,
Haochen
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://sourceware.org/pipermail/binutils/attachments/20250825/3f1cc90e/attachment-0001.htm>