Microsoft’s first 32bit OS, Xenix

Well back before, I had been looking into old linkers for 386 OMF, I knew I’d found some fun with some old GNU tools going back to the heyday of Xenix 386. As kind of expected the tools used to build Xenix, along with it’s SDK were in fact Microsoft C/MASM. So yes way back in 1987 Microsoft not only has MASM386, but they also had a 32bit Microsoft C 5.00. Let that sink in for a moment as OS/2 had been forced into 16-bit land despite FOOTBALL, and Windows/386 being a 386 VM86 multitasker. So in a weird way all the parts were in existence.

Back in the old days of GCC 1.x there was a bit of excitement about the file masm386.c in the old GCC source directory, sadly despite it being updated, there was no real public push on modifying GCC to support non AT&T assemblers. Instead something unexpected (well to me!) happened is that GAS had been modified to output OMF.

I tested this on MinGW with some simple stuff, and sure enough it links just fine. Considering its what is on the GCC on Xenix port I’m sure it’s pretty solid.

Enter OS/2

Now this is more fun, and again kind of sad that GCC didn’t take on the ability to target other assemblers (just look at the x68000!), Maybe they didn’t see GAS for OMF, or just didn’t know. Instead a more aggressive choice was made, to alter the binary output. Linking on OS/2 with EMX involved 2 very different and incompatible paths, the first one is the ancient Unix i386 a.out format, which then a utility called emxbind will convert into a loader & stub that OS/2 can run. Think of it as an OS/2 extender to take simple Unix programs (which is what they are) to run on OS/2. Neat!

The second more ‘native’ approach is to convert the binary a.out files into what is known as OMF files, which non GNU linkers like Wlink from Watcom or Link386 from Microsoft can then link into direct native EXE’s or DLL’s.

There had been an experimental ELF build of the EMX toolchain on OS/2 but I think it may have died? So as crazy as it seems bigger and crazier programs need to be built on systems like Windows or Linux and linked outside of OS/2 to get around the old memory limits. It’s really hard to say as I’ve never used it, although being able to do the link outside of OS/2 would be an advantage.

I’ve found 2 programs to convert the a.out objects into OMF, the first and oldest being o2obj. The one drawback I’ve had is that this doesn’t play so well with the Watcom 386 OMF linker. Instead the much later RSXNT/LIBC0.6 project’s emxomf. I’ve done some painful hacking but it appears to do what it should do. A simple omfdump seems to be spilling stuff out.

Of course the alternative is to use a 64bit linker, and since a.out has been pulled from binutils the only real hope is the Watcom linker which is now running in a 64bit address space. And the Watcom chain won’t understand ancient i386 a.out, however Microsoft 386 OMF it certainly will, although it appears to be based around something later than the aforementioned o2obj, which is why I ahd to do the emxomf.

I know as this stands it’s not very satisfying but I kind of wanted to push this out the door as I’d been hacking from time to time on it, and didn’t want to leave it to rot completely. The EMX tools remind me of the NeXT stuff where everyone goes native platform wild never imagining a day when remaining portable would mean it’d be easier to target more software.

The one thing I wonder about sometimes if there was some kind of secret Microsoft extended DOS/Windows that relied on OMF & Link386 that predated the NT team and their switch to COFF? Obviously it’d be super obsolete and would have been something like the first PharLap 386 stuff. But I’ve only owned a disk dump of v4, and a legit copy of TNT v6. Old 386/DPMI/Extender tools are hard to find.

Linkers & loaders, along with binary formats are too hard for me, but I thought I’d share at least what I’d been able to conjure with MinGW. I’ll have to touch on EMX to native OMF linking later.