Re: MIPS simulator is broken

["Maciej W. Rozycki" <macro at imgtec dot com>]

On Wed, 10 Feb 2016, Mike Frysinger wrote:

> > > the mips address logic was fundamentally broken.  my changes merely
> > > uncovered that braindeadness rather than caused it.  if you look at
> > > that commit and the AddressTranslation definition, you'll see:
> > >   /* For a simple (flat) memory model, we simply pass virtual
> > >      addressess through (mostly) unchanged. */
> > >   vAddr &= 0xFFFFFFFF;
> > > 
> > > this is forcing all addresses to be 32-bit.  when i saw this, i assumed
> > > it was just a dummy function someone didn't get around to finishing.
> > 
> >  Not unreasonable unless you want to simulate gigabytes of memory (or a 
> > TLB -- does the MIPS port of sim support it?).  Most programs run under 
> > sim won't need that though.  There is a mistake there however, you really 
> > want the mask to be 0x1FFFFFFF, to handle the architecture's 512MB KSEG0 
> > and KSEG1 unmapped segments and their aliasing correctly -- some programs 
> > rely on that.  Beyond that you need some more intelligence.
> 
> the sim uses a sparse memory map, so it doesn't need to fill the address
> space.  it's pretty common for some arches to allocate maps at the top
> (for things like hardware) while the low addresses have the RAM.  i don't
> know much this applies to existing mips usage.

 Due to how kernel segments, exception handler locations and the reset 
vector have been defined in the MIPS architecture, there's RAM expected to 
be there from the physical address 0 up, and ROM from the physical address 
0x1fc00000 up.  These will normally be accessed via the KSEG0 and KSEG1 
fixed virtual mappings respectively.  Specifically 32-bit MIPS processors 
will access exception handlers from 0x80000000 virtual/0 physical up and 
the reset vector 0xbfc00000 virtual/0x1fc00000 physical up.  And likewise 
64-bit MIPS processors will access these from 0xffffffff80000000 virtual/0 
physical and 0xffffffffbfc00000 virtual/0x1fc00000 physical respectively.  

 A bootstrap setting is available so that exception handlers are initially 
in ROM, so that any initialisation firmware can handle exceptional 
conditions before caches and any DRAM controller has been initialised, 
however applications will necessarily install their handlers in RAM so 
that they have control over them.

 Recently a possibility to relocate application exception handlers has 
been added, mainly to allow distinct handlers in SMP systems, however as a 
side effect there's no need for RAM at 0 physical anymore.

 NB I've mentioned it for illustration purposes only, as all except the 
virtual-to-physical mappings quoted above is external to the CPU.  It does 
have implications on how the mappings have been made in the first place 
though.

> for this specific case, i don't see how the mask as implemented could ever
> be correct.  it always threw away the top 32-bits, so when you try to use
> any address above 0xffffffff, it'd be silently changed to the lower 32-bits.

 Few bare-metal apps require or expect physical memory beyond 4GB, so this 
is mostly correct, but see below for the exact mapping details.

> playing with some n32 programs on a mips64 system shows that it often gets
> maps created both above and below that point.  i don't see how chopping the
> top bits could possibly yield correct behavior.

 N32 by definition uses 32-bit addressing and cannot ever go beyond 512MB 
via KSEG0/KSEG1 (because they strip the 3 MSBs of the address).  It could 
go via TLB, but this I gather is not implemented in GNU sim.  The 64-bit 
XKPHYS segment is by definition inaccesible to n32 programs, though indeed 
this is seldom enforced in hardware with the use of the CP0 Status.PX bit, 
due to historical baggage.  The XKPHYS mapping chops off the 5 MSBs BTW, 
that is bits 63:59.

 Can you give me an example of an n32 program going beyond 4GB?

 NB if you have bare-metal programs using both user (positive) and kernel 
(negative) addresses, then you really need to implement some sort of MMU 
to make them work correctly.

 Actually I think I wasn't explicit enough about this previously: the sign 
bit of any address is really the user/kernel* bit in the MIPS architecture
-- negative addresses are accessible to the kernel only, while positive 
addresses can be accessed both from the user and the kernel mode.  The 
implication of this arrangement and the observation that in 32-bit 
implementations it is bit #31 that is the sign bit and in 64-bit ones
it is bit #63 that is the sign bit is that you need to sign-extend 32-bit 
kernel addresses so that bit #31 is propagated to bit #63 in hardware.  
For 32-bit user addresses it does not matter if you sign- or zero-extend 
addresses, because bit #31 is zero anyway.  NB user addresses are mapped 
via the MMU, except when CP0 Status.ERL bit is set.

* There's some complication due to the optional supervisor mode, invented 
  miserably to support Windows NT on MIPS processors (ironically enough, 
  the architecture the OS was originally written for), but let's keep 
  things simple for the purpose of this consideration -- except from the 
  oddball Windows NT port for the DECstation (little-endian, of course!), 
  I haven't heard of any other use of the supervisor mode and the kernel 
  can treat any supervisor-mode addresses as plain kernel-mode addresses; 
  they're inaccessible to user-mode software.

  And yes, there's a further recent complication where you can shift the
  user/kernel split to 0xc0000000 instead in 32-bit processors; more on 
  this below.

> let me outline a few general points i think apply here:
> i see the simulator core as a bunch of generic building blocks.  the arch port
> itself only adds support for the ISA (insns & registers) to the equation.  once
> you get beyond that (e.g. the memory or devices), now you're talking about more
> building blocks than stuff that should be baked in/architecture defaults.  so
> even if today all of the systems that have a mips cpu also wire up the system
> mem in a specific way, the mips arch core should not be doing any of that.
> this makes it very easy to take just the mips ISA and construct a new cpu from
> scratch that has new/different behavior and play around with things.  when you
> do want behavior that matches an existing board, that's where the model
> framework comes into play.  you can define specific cpu/system combinations
> that match existing products and users can pick those via the --model flag.

 Listen, in the MIPS platform the memory architecture is wired to the CPU, 
not any external arrangement like straps, board circuitry or whatever; CP0 
is a part of the architecture, the privileged side.  The MIPS architecture 
may be doing it a bit differently from other architectures, but that's 
just how things are, every architecture is a bit unique in some respect.  
You can't have a MIPS implementation without these mappings within the CPU 
itself, whether externally there's memory, MMIO or nothing.

 Yes, you can wire different types of MMU, which will do different address 
mappings, e.g. BAT rather than TLB, but segments are going to stay there.  
In more recent architecture revisions there is an optional extension 
available to set segment attributes more flexibly, e.g. you can control 
privilege levels required for access or enable/disable mapping via the 
MMU.  This is how the user/kernel split at 0xc0000000 has been 
implemented.

 If you don't feel like implementing TLB or any fancy MMU for the MIPS 
port, then it's fine, such MIPS processors even have been made.  But then 
you need to provide at least the simplest KSEG0/KSEG1/XKPHYS fixed mapping 
I outlined for any MIPS software to work correctly.

 To give you a more illustrative example: ignoring the presence of MIPS 
segments is like ignoring the presence of segment registers on x86 even in 
their simplest real-mode form.  Yes, you can have an x86 simulation with 
the 8/16 GPRs only (and the PC aka IP), but is it still a proper x86 
implementation?

> it is a little difficult currently to reconcile virtual-vs-physical behavior
> in the sim as there is only a single physical map and no virt-to-phys trans.
> we've just ignored this so far and said virtual==physical.  which is why the
> sim throws an error when you try to put a virtual 0xffffffff80000000 address
> onto the physical bus.

 So yes, if you're missing a building block for what is an MMU in any 
processor that implements virtual addressing, then yes, I think you need 
to add it at least conceptually, and let ports wire it as they see it fit.  
Ones that have real addressing only will map 1-to-1, keeping this block 
null.  Other ones will wire it as appropriate, maybe in a generic way if 
there's any, or maybe in an implementation-dependent way, different for 
individual processor models simulated.

> since 64-bit address aren't actually being used in the 32-bit env, why bother
> using them ?  seems like it'd be much easier to just use 32-bit addresses and
> be done.

 This is why I suggested taking the simplest approach and masking off 
anything outside low 512MB, that is applying a mask of 0x1FFFFFFF still 
quoted at the top, matching KSEG0 vs KSEG1 semantics in real hardware and 
satisfying virtually all bare-metal software (let's exclude OS kernels 
such as Linux from here even though technically they're bare-metal apps 
too).  As a side effect you'll avoid the sign-extension that is 
architectural and which BFD gets right and you're so unhappy about.  
Merely chopping off upper 32 bits from a 64-bit virtual address isn't 
compliant with any architecturally permitted address mapping and won't 
work with most MIPS software.

 So what's wrong with this proposal?

 This isn't correct for user-mode addresses that need to go via the MMU, 
but GNU sim does not implement any MIPS MMU, so this will be as good as 
any.  In fact a mapping like this just could be treated as a simple fixed 
MMU, though to be fully compliant, user-mode addresses would have to be 
remapped according to the architecture manual[1], i.e. conceptually 
(ignoring CP0 Status.ERL for simplicity; trivial to add):

	if (addr >= 0)
		addr = (addr & 0x7fffffff) + 0x40000000;
	else if (addr <= -0x40000000)
		addr &= 0x1fffffff;
	else
		addr &= 0xffffffff;

Maybe we really ought to do this instead until we have a proper wireable 
MMU building block.  It is permitted by the architecture to implement a 
64-bit processor with 32-bit addressing only.

References:

[1] "MIPS Architecture For Programmers, Volume III: The MIPS64 and
    microMIPS64 Privileged Resource Architecture", MIPS Technologies,
    Inc., Document Number: MD00091, Revision 5.04, January 15, 2014,
    Section A.1 "Fixed Mapping MMU", p. 286
    <https://imgtec.com/mips/architectures/mips64/>

  Maciej