i will have to check the rest but at least 'mte' is not an architecture extension in the gnu tools, there is 'memtag' instead

i.e. -march=armv8.5-a+memtag is valid but -march=armv8.5-a+mte is not.

I think a problem we've got here is that in LLVM there are all these "backend features" that do not correspond to the notional user-facing features in Clang, but which are nevertheless exposed to users through the assembler and .arch_extension.

Thus, GCC and Clang agree on +memtag being the option to enable MTE. But gas follows GCC and uses +memtag to enable it, whereas in LLVM the "internal feature" ends up being "+mte", which is what the assembler demands instead of +memtag.

Similar for 'tlb-rmi', I think introduced through https://github.com/llvm/llvm-project/commit/9c9067316be2b802a3af689b94aadc2740a47bcc

Reading through the rationale at http://lists.llvm.org/pipermail/llvm-dev/2018-September/126346.html
it looks like LLVM made the conscious step of breaking compatibility with GNU, which is unfortunate.

For the future, it would be great if both the Clang-level and the assembly-level feature strings in LLVM aligned (and we can ensure they're aligned with GCC and gas).

For the current inconsistencies, we have some options of resolving the pain.
LLVM can add +memtag as an alias for +mte to its "backend feature".

The other extension we need to resolve is 'tlb-rmi'. It is a mandatory part of Armv8.4-A. It cannot be enabled at the Clang level, only at the assembler level, as it's one of those "backend features". gas doesn't support the extension string and gates these instructions on -march=armv8.4-a. I think that behaviour is aligned with what LLVM did with https://github.com/llvm/llvm-project/commit/9c9067316be2b802a3af689b94aadc2740a47bcc

Any ideas on how to proceed?

> I think that behaviour is aligned with what LLVM did with
> https://github.com/llvm/llvm-project/commit/
> 9c9067316be2b802a3af689b94aadc2740a47bcc
> 

Sorry, I meant to say "what LLVM did before the commit"

>Reading through the rationale at http://lists.llvm.org/pipermail/llvm-dev/2018-September/126346.html
it looks like LLVM made the conscious step of breaking compatibility with GNU, which is unfortunate.

FWIW none of that is implemented yet and I don't think the mismatching names is any intention, it's just due to the timing when they were implemented. (mte is possibly the only one with this issue)

Happy to discuss that RFC if you've got specific concerns, it's not set in stone by any means. (I think there was some GNU input but it's been a while)

> For the future, it would be great if both the Clang-level and the assembly-level feature strings in LLVM aligned (and we can ensure they're aligned with GCC and gas).

+1

> For the current inconsistencies, we have some options of resolving the pain.
LLVM can add +memtag as an alias for +mte to its "backend feature".

Sounds good to me.

> The other extension we need to resolve is 'tlb-rmi'. It is a mandatory part of Armv8.4-A.

Do you have an example of what we want this to look like? Is there existing code that uses it with gas perhaps.

(In reply to David Spickett from comment #4)
> Is there existing
> code that uses it with gas perhaps.

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/arch/arm64/include/asm/tlbflush.h

In the Linux kernel header that uses tlb-rmi.  It's C, but the inline assembly is what's tripping us up.  It's pretty well commented what the functions in the header are for.

Specifically, it looks like the functions flush_tlb_all/flush_tlb_mm/flush_tlb_page_nosync/flush_tlb_kernel_range/__flush_tlb_kernel_pgtable try to use `tlbi ` vmalle1/vmalle1is/aside1is/vale1is/vaale1is/vaae1is respectively.

For out of line assembly files using `tlbi`, see:
- arch/arm64/mm/proc.S
- arch/arm64/kernel/hibernate-asm.S
- arch/arm64/kernel/head.S
- arch/arm64/kernel/entry.S
- arch/arm64/kvm/hyp/nvhe/hyp-init.S

I can also see the use of the tlbi related macros using tlbi in inline asm in some of the virtualization code:
- arch/arm64/kvm/hyp/nvhe/tlb.c
- arch/arm64/kvm/hyp/vhe/tlb.c
- arch/arm64/kernel/sys_compat.c

Is the tlbi instruction what was added in ARMv8.4 (tlb-rmi), or only certain operands?

> Is the tlbi instruction what was added in ARMv8.4 (tlb-rmi), or only certain operands?

(I'm reading this from the ARMARM for v8, "C6.2.328 TLBI")

tlbi is a sys instruction alias which I think means that it's always been available in some form. Clang takes this approach, so just armv8.0-a will give you access to tlbi itself.

v8.4-a added more operands (some of which take registers) to the list. These are the "outer shareable" and "range maintanence". (added to llvm in 35bd8f5d1e31c7f57ddd56b12fba302f54274548, note that since then a "tlb-rmi" sub feature was added)

So if you look at the instruction description, <tlbi_op> will tell you which operands were added for FEAT_TLBIOS and FEAT_TLBIRANGE which are enabled for v8.4-a.

Clang example:
$ cat /tmp/test.s
tlbi vmalle1os
$ ./clang -target aarch64-arm-none-eabi -march=armv8.3-a -c /tmp/test.s -o /dev/null
/tmp/test.s:1:6: error: TLBI VMALLE1OS requires tlb-rmi
tlbi vmalle1os
     ^
$ ./clang -target aarch64-arm-none-eabi -march=armv8.4-a -c /tmp/test.s -o /dev/null

This will also do the same thing from the assembler if you want to avoid the clang only tlb-rmi name:
.arch armv8.4-a
tlbi vmalle1os

(if that helps at all)

(LLVM patch for memtag: https://reviews.llvm.org/D85620).