Subversion Repositories planix.SVN

.FP palatino

.TM

.TL

Plan 9 on the Mikrotik RB450G Routerboard

.AU

Geoff Collyer

.AI

.MH

.NH 1

Motivation

.LP

I ported Plan 9 to the Routerboard mainly to verify

that Plan 9's MIPS-related code

(compiler, assembler, loader,

.CW libmach ,

etc.) was still in working order and would

work on newer machines than the 1993-era ones that we last owned

(MIPS Magnum, SGI Challenge, Carrera and the like).

The verdict is that,

with a few surprising exceptions, the code still works on newish machines

(the MIPS 24K CPU in the Routerboard dates to about 2003 originally;

this revision is from about 2005).

So we now have a

machine on which to test MIPS executables.

.LP

The other reason I did the port was

as an incremental step toward

running Plan 9 on a MIPS64 machine (e.g., the dual-core, dual-issue

Cavium CN5020 in the Ubiquiti Edgerouter Lite 3).

.NH 1

The new MIPS world

.LP

These newer MIPS systems are aimed at embedded applications, so they

typically lack FPUs and may also lack L2 caches or have small TLBs;

the MIPS 24K in the Atheros 7161 SoC lacks FPU and L2 cache, and has a

16-entry TLB.

It is a MIPS32R2 architecture system and lacks the 64-bit instructions

of the R4000.

These new MIPS systems are still big-endian,

so provide a useful test case to expose byte-ordering bugs.

.NH 1

Plan 9 changes and additions

.NH 2

CPU Bug Workarounds

.LP

The Linux MIPS people cite MIPS 24K erratum 48:

3 consecutive stores lose data.

MIPS only distribute their errata lists under NDA and to their

corporate partners, so we have only the Linux report to go on.

The fix requires

.I both

write-through data cache and

no more than two consecutive single-word stores in all executables.

I have made a crude optional change to

.I vl

to generate a NOP before every third consecutive store.

The fix could be better, in particular the technique for

keeping stores out of branch delay slots.

.NH 2

Driver for Undocumented Ethernet Controller

.LP

The FreeBSD Atheros

.I arge

driver

(in

.CW /usr/src/sys/mips/atheros )

provided inspiration for our Gigabit Ethernet driver, since the

hardware is otherwise largely undocumented.

I haven't got the second

Ethernet controller entirely working yet;

it's perhaps complicated by having a switch attached to it (the Atheros 8316).

At minimum, it probably needs MII or PHY initialisation.

.NH 2

Floating-point Emulation

.LP

Floating-point emulation works but is

.I very

slow:

.I astro

takes about 8 seconds.

I added an

.CW fpemudebug

command to

.CW /dev/archctl ;

it

takes a number as argument corresponding to the

.CW Dbg*

bits in

.CW fpimips.c ,

but requires the kernel to be compiled with

.CW FPEMUDEBUG

defined.

.NH 3

\&... in Locking Code

.LP

The big surprises included that

.CW /sys/src/libc/mips/lock.c

read

.CW FCR0

to

choose the locking style.

That's been broken out into

.CW c_fcr0.s

so that we can change it, but the kernel also emulates the

.CW MOVW

.CW FCR0,R1

(and via a fast code path), to keep alive the possibility of running

old binaries from the dump.

.NH 2

No 64-bit Instructions

.LP

The other big surprise was that

.CW /sys/src/libmp/mips/mpdigdiv.s

used 64-bit instructions (SLLV, SRLV, ADDVU, DIVVU).

For now I've resolved the problem by pushing it into a

subdirectory (\c

.CW r4k )

and editing the

.CW mkfile s

to use the

.CW port

version

(and similarly in APE).

.br

.ne 8

.NH 2

Page Size vs TLB Faults

.LP

I started out with a 4K page size and reduced the number of TLB

entries reserved for the kernel to 2, leaving 14 for user programs,

but

.CW /dev/sysstat

was reporting 6 times as many TLB faults as page

faults, and the number increased at a furious rate.

.LP

So I switched to

a 16K page size, adjusted

.CW vl

.CW -H2

accordingly and recompiled the

.CW /mips

world.

This reduced the TLB faults to just 10% more than the number of page faults.

(That number is now around 15% more, due to a better soft-TLB hash function

that makes the soft TLB more effective.)

16K pages also produce consecutive (even recursive) page faults

for the same address at the same PC

and the system runs at about 10% of its normal speed,

so 4K pages are currently the only sensible choice;

we'll just live with the absurdly-high number of TLB faults

(around 20k–30k per second).

It probably doesn't help that one 16K page is half of the L1 data cache

and one quarter of the L1 instruction cache.

.LP

Page size is controlled by

.CW BIGPAGES

in

.CW mem.h .

.NH 3

Combined TLB Pool

.LP

I also changed

.CW mmu.c

to collapse the separate kernel and user TLB pools into one,

once user processes start running,

but that only helps to reduce TLB faults a little.

.

.br

.ne 8

.

.NH 1

Remaining Problems

.LP

Interrupt-driven UART output isn't quite right.

It can get stuck and then input makes it resume.

The UART is apparently connected via the APB and requires

interrupt unmasking in the APB (which we now do).

There's some kludgey stuff in

.CW uarti8250.c

that makes output work most of the time

(characters do sometimes get dropped).

.LP

The Ethernet driver currently does not

dig out the MAC addresses from the hardware,

so you'll need to edit the

.CW rb

configuration file for each Routerboard; the format should be obvious.

I don't have the stomach to dig the MAC address out of the hardware

via SPI or whatever vile interface it requires.