[rfa] i386-tdep.c: add some insn's to prologue analyzer

[Mark Kettenis]

   Date: Mon, 4 Aug 2003 21:33:19 -0400
   From: Michael Elizabeth Chastain <mec@shout.net>

   This patch improves the i386 prologue analyzer so that it can
   handle a few more instructions in the prologue.  This fixes
   pr gdb/1253 and pr gdb/1255.

Thanks, Michael.  A minor nit though.  When rewriting the prologue
analyzer I made an effort to make sure that the returned PC points at
the first instruction after the prologue.  I think we should consider
the "xor" and "sub" instructions only part of the prologue if they're
followed by the "mov %esp, %ebp".  So I rewrote your patch a bit.  I
checked in the attached on mainline.  Can you check whether it really
solves the problems you were seeing.  I added a testcase to the
testsuite, and there it seems to do the job.  I'll move this over to
the branch in a few days if things turn out to be allright.

Mark

Index: ChangeLog
from  Mark Kettenis  <kettenis@gnu.org>

	Based on a patch from Michael Elizabeth Chastain (mec@shout.net):
	* i386-tdep.c (i386_analyze_frame_setup): Recognize more
	instructions that GCC likes to mingle into the prologue.  Fixes
	gdb/1253 and gdb/1255.

Index: i386-tdep.c
===================================================================
RCS file: /cvs/src/src/gdb/i386-tdep.c,v
retrieving revision 1.162
diff -u -p -r1.162 i386-tdep.c
--- i386-tdep.c 18 Jul 2003 21:31:50 -0000 1.162
+++ i386-tdep.c 10 Aug 2003 12:00:13 -0000
@@ -439,6 +439,7 @@ i386_analyze_frame_setup (CORE_ADDR pc, 
 			  struct i386_frame_cache *cache)
 {
   unsigned char op;
+  int skip = 0;
 
   if (current_pc <= pc)
     return current_pc;
@@ -456,25 +457,61 @@ i386_analyze_frame_setup (CORE_ADDR pc, 
       if (current_pc <= pc + 1)
 	return current_pc;
 
-      /* Check for `movl %esp, %ebp' -- can be written in two ways.  */
       op = read_memory_unsigned_integer (pc + 1, 1);
+
+      /* Check for some special instructions that might be migrated
+	 by GCC into the prologue.  We check for
+
+	    xorl %ebx, %ebx
+	    xorl %ecx, %ecx
+	    xorl %edx, %edx
+
+	 and the equivalent
+
+	    subl %ebx, %ebx
+	    subl %ecx, %ecx
+	    subl %edx, %edx
+
+	 Make sure we only skip these instructions if we later see the
+	 `movl %esp, %ebp' that actually sets up the frame.  */
+      while (op == 0x29 || op == 0x31)
+	{
+	  op = read_memory_unsigned_integer (pc + skip + 2, 1);
+	  switch (op)
+	    {
+	    case 0xdb:	/* %ebx */
+	    case 0xc9:	/* %ecx */
+	    case 0xd2:	/* %edx */
+	      skip += 2;
+	      break;
+	    default:
+	      return pc + 1;
+	    }
+
+	  op = read_memory_unsigned_integer (pc + skip + 1, 1);
+	}
+
+      /* Check for `movl %esp, %ebp' -- can be written in two ways.  */
       switch (op)
 	{
 	case 0x8b:
-	  if (read_memory_unsigned_integer (pc + 2, 1) != 0xec)
+	  if (read_memory_unsigned_integer (pc + skip + 2, 1) != 0xec)
 	    return pc + 1;
 	  break;
 	case 0x89:
-	  if (read_memory_unsigned_integer (pc + 2, 1) != 0xe5)
+	  if (read_memory_unsigned_integer (pc + skip + 2, 1) != 0xe5)
 	    return pc + 1;
 	  break;
 	default:
 	  return pc + 1;
 	}
 
-      /* OK, we actually have a frame.  We just don't know how large it is
-	 yet.  Set its size to zero.  We'll adjust it if necessary.  */
+      /* OK, we actually have a frame.  We just don't know how large
+	 it is yet.  Set its size to zero.  We'll adjust it if
+	 necessary.  We also now commit to skipping the special
+	 instructions mentioned before.  */
       cache->locals = 0;
+      pc += skip;
 
       /* If that's all, return now.  */
       if (current_pc <= pc + 3)