Chasing more 386 OMF

Microsoft’s first 32bit OS, Xenix

Well back before, I had been looking into old linkers for 386 OMF, I knew I’d found some fun with some old GNU tools going back to the heyday of Xenix 386. As kind of expected the tools used to build Xenix, along with it’s SDK were in fact Microsoft C/MASM. So yes way back in 1987 Microsoft not only has MASM386, but they also had a 32bit Microsoft C 5.00. Let that sink in for a moment as OS/2 had been forced into 16-bit land despite FOOTBALL, and Windows/386 being a 386 VM86 multitasker. So in a weird way all the parts were in existence.

Back in the old days of GCC 1.x there was a bit of excitement about the file masm386.c in the old GCC source directory, sadly despite it being updated, there was no real public push on modifying GCC to support non AT&T assemblers. Instead something unexpected (well to me!) happened is that GAS had been modified to output OMF.

I tested this on MinGW with some simple stuff, and sure enough it links just fine. Considering its what is on the GCC on Xenix port I’m sure it’s pretty solid.

Enter OS/2

Now this is more fun, and again kind of sad that GCC didn’t take on the ability to target other assemblers (just look at the x68000!), Maybe they didn’t see GAS for OMF, or just didn’t know. Instead a more aggressive choice was made, to alter the binary output. Linking on OS/2 with EMX involved 2 very different and incompatible paths, the first one is the ancient Unix i386 a.out format, which then a utility called emxbind will convert into a loader & stub that OS/2 can run. Think of it as an OS/2 extender to take simple Unix programs (which is what they are) to run on OS/2. Neat!

The second more ‘native’ approach is to convert the binary a.out files into what is known as OMF files, which non GNU linkers like Wlink from Watcom or Link386 from Microsoft can then link into direct native EXE’s or DLL’s.

There had been an experimental ELF build of the EMX toolchain on OS/2 but I think it may have died? So as crazy as it seems bigger and crazier programs need to be built on systems like Windows or Linux and linked outside of OS/2 to get around the old memory limits. It’s really hard to say as I’ve never used it, although being able to do the link outside of OS/2 would be an advantage.

I’ve found 2 programs to convert the a.out objects into OMF, the first and oldest being o2obj. The one drawback I’ve had is that this doesn’t play so well with the Watcom 386 OMF linker. Instead the much later RSXNT/LIBC0.6 project’s emxomf. I’ve done some painful hacking but it appears to do what it should do. A simple omfdump seems to be spilling stuff out.

Of course the alternative is to use a 64bit linker, and since a.out has been pulled from binutils the only real hope is the Watcom linker which is now running in a 64bit address space. And the Watcom chain won’t understand ancient i386 a.out, however Microsoft 386 OMF it certainly will, although it appears to be based around something later than the aforementioned o2obj, which is why I ahd to do the emxomf.

I know as this stands it’s not very satisfying but I kind of wanted to push this out the door as I’d been hacking from time to time on it, and didn’t want to leave it to rot completely. The EMX tools remind me of the NeXT stuff where everyone goes native platform wild never imagining a day when remaining portable would mean it’d be easier to target more software.

The one thing I wonder about sometimes if there was some kind of secret Microsoft extended DOS/Windows that relied on OMF & Link386 that predated the NT team and their switch to COFF? Obviously it’d be super obsolete and would have been something like the first PharLap 386 stuff. But I’ve only owned a disk dump of v4, and a legit copy of TNT v6. Old 386/DPMI/Extender tools are hard to find.

Linkers & loaders, along with binary formats are too hard for me, but I thought I’d share at least what I’d been able to conjure with MinGW. I’ll have to touch on EMX to native OMF linking later.

5 thoughts on “Chasing more 386 OMF

  1. Thank You!, I will have a look at this at the near future.

    “The one thing I wonder about sometimes if there was some kind of secret Microsoft extended DOS/Windows that relied on OMF & Link386”
    I seem to recall that when Cutler and his people started working in OS2/NT-Windows NT they compiled and run an emulator of an Intel i860 in some kind of OS/2, like taking the early OS/2 2.00 source code and compiling with their tools.
    Microsoft still had access to OS/2 source code like IBM had access to Windows code.
    You could see at “OS/2 Museum” that versions of OS/2 2.00 before release used another executable format.
    I should look where I read this, but it’s not easy to find.

  2. You said in the article: “Windows/386 being a 386 VM86 multitasker”.
    But in the linked article, there’s no evidence of “386 VM86 multitasker”. Can you explain more?
    Thank you 🙂

    • Well, the explanation on Gunkies was basically written by Neozeed:

      There’s been a lot of conversation about this release by people who hang out here since it’s such an oddball. Is there anything specific you want to know about it?

      What seemed most odd to me is that Windows/386 is basically a hypervisor underneath a regular Windows/286. Win386.exe sets up EMS emulation and provides services for VM86 boxen, then launches, a regular Windows. So a copy of Windows/386 can be run in fully real mode by running directly, or under its bundled hypervisor. But what’s even crazier is the hypervisor can be put under Windows/286, or even under, to achieve EMM386-like behavior, before EMM386 existed. I think doing this under Windows/286 won’t support VM86 boxes, but it will use emulated EMS.

Leave a Reply