[PATCH] x86: tighten .insn SAE and broadcast checking

[Jan Beulich]

SAE / embedded rounding are invalid when there's the memory operand, as
the bit encoding this specifies broadcast in that case.

Broadcast needs to be specified on the memory operand.
---
Arguably, until the AVX10 spec spells out how embedded rounding is to be
encoded for 256-bit operations, we may need to further insist on EVEX512
or EVEXLIG to be specified. In practice that would end up overly
limiting though, as can be observed when using .insn to encode APX insns
(where EVEX.brs is overloaded by EVEX.nd).

--- a/gas/config/tc-i386.c
+++ b/gas/config/tc-i386.c
@@ -11538,8 +11538,9 @@ s_insn (int dummy ATTRIBUTE_UNUSED)
 	  ? i.broadcast.type || i.broadcast.bytes
 	    || i.rounding.type != rc_none
 	    || i.mask.reg
-	  : (i.broadcast.type || i.broadcast.bytes)
-	    && i.rounding.type != rc_none))
+	  : (i.mem_operands && i.rounding.type != rc_none)
+	    || ((i.broadcast.type || i.broadcast.bytes)
+		&& !(i.flags[i.broadcast.operand] & Operand_Mem))))
     {
       as_bad (_("conflicting .insn operands"));
       goto done;