Rebuilding Darwin from source: Part 2 Building the kernel

Re-creating the steps from 7 years ago the first phase was to build the Darwin kernel. Like everything else, once you know what is involved, it’s not all that difficult. But as always finding out the steps to get there is half the fun!

I’m going to assume if you want to follow along, that you’ve completed the first part of this exercise, and you have a Rhapsody DR 2 system up and running. Due to some issues I’ve had with creating a lot of files & filesystem corruption, we are going to create and add two more disks to the system. On Qemu we need to add them via the CLI:

qemu-img create -f vmdk source.vmdk 8G
qemu-img create -f vmdk scratch.vmdk 8G

Adding them to the command line gives us something like this:

qemu -L pc-bios -m 512 ^
-k en-us ^
-rhapsodymouse ^
-hda rhapsody.vmdk ^
-hdb source.vmdk ^
-hdd scratch.vmdk ^
-cdrom darwin-builder-04-23-2024.iso ^
-fda nic.flp ^
-net nic,model=ne2k_pci,vlan=1 ^
-net socket,udp=127.0.0.1:5001,remote=127.0.0.1:5000,vlan=1 ^
-boot c ^
%1 %2 %3 %4 %5 %6 %7

Additionally you’ll also need to download the current ‘darwin builder’ ISO that I’ve put up on sourceforge. As of today it is darwin-builder-04-23-2024.iso

Step one is to boot into single user mode. As we need to prep & format the disks under Darwin before the system starts up.

We need to check the hard disk, and then create the device names for the third hard disk.

fsck -y /dev/rhd0a
mount -w /
cd /dev
./MAKEDEV hd2

Now we need to run the ‘disk’ command which will abstract the whole volume creation. There are numerous flags, but we don’t need all that many.

disk -i -l 'src' /dev/rhd1a
disk -i -l 'scratch' /dev/rhd2a

The output scrolls off the screen, so I didn’t capture it, but you’ll see all the inodes being created, it’s a lot of output!

With the disks created, we can now shut down the VM

shutdown -h now

and then restart Qemu, and let it boot up normally. We’ll get to the login screen, login as the root user.

The first thing I’d recommend is to drag the Terminal.app from /System/Administration to the desktop to make it easier to get to. Rhapsody, unlike NeXTSTEP & OPENSTEP doesn’t have any dock, as the goal back then was to make Rhapsody look and feel more like Platinum MacOS.

The next thing to do is to make the system very insecure by allowing remote root logins. It’s just easier to deal with. You could use sed or just copy the one I provided from the CD-ROM.

cp /source/ttys /etc

And with that in place, its easy enough to telnet into the VM so you can copy/paste stuff in and out with ease!

You should now be able to verify that all 3 disks are mounted:

# mount | grep hd.a
/dev/hd0a on / (local)
/dev/hd1a on /src (local)
/dev/hd2a on /scratch (local)

From here it should be very simple to kick off the build process:

/source/phase1.sh

And this will kick off the build, recreating all the fun steps I’d gone through so many years ago. These projects now are building in the following order:

  • kernel-1
  • driverkit-139.1-1
  • cc-798
  • bootstrap_cmds-1
  • objc-1

The first phase of the script will unpack both the kernel & driverkit and install their respective header files into the OS. NeXT a bunch of symlinks are created to link the system to the driverkit. Next I decided to build the ObjectiveC compiler from 10.0, hoping it’s more bugfixed and slightly more optimized than what was available back in 1999. Building the compiler is a little involved, as a good GCC tradition is to be cross compiled first, then re-compile itself with itself, then do that again and verify that the 3rd recompile outputs the same as the second one. Yes it’s a thing. Yes it’s slow. Yes you are lucky to live in the future, this was really painful back in the day.

With the kernel compiled, we can then compile the bootstrap commands, and the objectiveC runtime that is used by the kernel. Nothing too exciting here.

DriverKit however….

The PCMCIA code was not included in any of the 0.x Darwins, so for laptop enthusiasts you are basically SOL. As a matter of fact, a lot of weird stuff was pruned out, that either could be ‘touched’ or borrowed from the PowerPC port and massaged into place. Luckily I had at least figured out a simple fix for PCIKernBus.h so at least PCI works.

Likewise for the kernel, there was some guessing on the EISA config, which also overlaps ISA, along with having to remove the PCMCIA cardbus .. bus.

APM crash

I had issues with the APM (Advanced Power Management), another laptopisim I suspect. I had to amputate that.

for testing purposes

Naturally the cpuid code is broken much like early NT (I wonder if both were contributed by intel?), so it doesn’t detect any half way modern Pentium processors correctly which causes it to fall all the way back to the i386, which unfortunately, Rhapsody is compiled as 486 (remember NeXTSTEP had fat binaries allowing you to recompile for different processors and ship a single binary that can be ‘lipo’d into the appropriate one for the host). So being degraded to a 386 means nothing works.

bad CPU type in library!

yay.

Luckily patching the cpuid was pretty simple just force it always to be a Pentium. It is 1999 afterall.

I’ve done my best to make this a single script to run, and all being well you’ll get something that looks like errors, but it should be fine?!

System/Library/Frameworks/System.framework/Versions/B/Headers/bsd/i386/table.h
System/Library/Frameworks/System.framework/Versions/B/Headers/bsd/i386/types.h
System/Library/Frameworks/System.framework/Versions/B/Headers/bsd/i386/user.h
System/Library/Frameworks/System.framework/Versions/B/Headers/bsd/i386/vmparam.h
private/
private/dev/
private/dev/MAKEDEV
private/tftpboot/
private/tftpboot/mach_kernel
mach_kernel
tar: private/dev: Could not change access and modification times: Permission denied
tar: private/dev: Cannot change mode to 0755: Permission denied
tar: private/dev: Cannot chown to uid 0 gid 0: Permission denied
tar: Error exit delayed from previous errors

A quick look around shows that there is tgz files indicating that things have been compiled. I did backup the old original kernel as “rhapsody-gcc.tgz” in case you ever need it. Can’t imagine why but who knows?

qemu:13# ls -l /usr/src/*.tgz
-rw-r--r--  1 root  wheel   173706 Apr 23 15:25 /usr/src/bootstrap_cmds.bin.tgz
-rw-r--r--  1 root  wheel  2184460 Apr 23 15:33 /usr/src/cc-798-bin.tgz
-rw-r--r--  1 root  wheel  2747289 Apr 23 15:36 /usr/src/driverkit-kern-bin.tgz
-rw-r--r--  1 root  wheel  1264957 Apr 23 15:26 /usr/src/kernel-driverkit-hdrs.tgz
-rw-r--r--  1 root  wheel   116343 Apr 23 15:26 /usr/src/objc-bin.tgz
-rw-r--r--  1 root  wheel  2173005 Apr 23 15:26 /usr/src/rhapsody-gcc.tgz
qemu:14# ls -l /mach_kernel*
-r--r--r--  2 root  wheel  1459520 Apr 23 15:36 /mach_kernel
-r--r--r--  1 root  wheel  1404116 Apr 23 15:38 /mach_kernel-rhapsody

You should now be able to reboot into the kernel that you’d compiled!

Next up is Phase 2, where we compile the tools to enter the dark magic that is the Debian build system. Yes, you read that right, Apple/NeXT was all in on Debian.

Looking at UnixWare 7.1.1 on VMware & Qemu yet again!

Same old OS, same old problems.

My UnixWare 7.1.1 box

The UnixWare 7.1.1 install program has a date & time Y2k problem. And this always ends in whatever licensing you give it to install will expire and be nullified. Luckily this time while re-installing on VMware I saw if you defer the license on install, It’ll grant you a temporary eval license. It’s not going to matter as it’ll immediately expire, but it get’s us past the install.

Qemu however let’s you rev up the time machine and specify a starting time

-rtc base=1999-09-29T15:00:00

As simple as that. I found for installing with Qemu 8.0 (Latest win32) binary it worked well enough like this:

"c:\Program Files\qemu\qemu-system-x86_64.exe" ^
-m 1024 ^
-hda UnixWare711.vmdk ^
-cdrom SCO_UnixWare711.iso ^
-net nic,model=pcnet ^
-rtc base=2010-09-29T15:00:00

Installation in a stock boring VM goes fine, there is built in support for the AMD PCnet driver, so things ‘just work’. And then on the VMware reboot it never launches X11.

Starting Desktop works fine on Qemu

Under Qemu, I’m greeted by CDE and the login page. On VMware however…

Starting Desktop never starts

I know t his used to work on VMware, but there is some regression in the VESA video driver. The fix was to use scoadmin and knock the video settings down to stock VGA. Luckily I have an X server running on Windows, so I could just export the display and set it up.

Video Configuration on Qemu

Whereas I had to set VMware to VGA:

Video Configuration on VMware

And one more reboot, and I was at least given a graphical console:

Now able to login to VMware graphically

Inputting the licenses

Being a commercial Unix from back in the day, it relies on License Numbers, and activation codes to actually use the software. I have the box, so I have codes so yay me. Post install, I could remove the expired licenses, and then input the ones that were in the box.

These are 5 user licenses, just enough to show off the system, I suppose: The advanced features of the day are nothing special these days, but it’s nice to have the PCC derived compiler, if not to compile GCC but more so for SYSV code from back in the day.

Networking

Networking for VMware is straightforward, I use the NAT interface VMnet8 that is installed by default, selecting a valid Tcp/IP address on the interface range gives me not only full internet access, but also allows me to easily telnet into the VM.

Qemu however…

"c:\Program Files\qemu\qemu-system-x86_64.exe" ^
-m 1024 ^
-hda UnixWare711.vmdk ^
-net nic,model=pcnet ^
-net user,hostfwd=tcp::42323-:23

I had been using the user mode SLiRP code for ages, but after all the MIT PC/IP fun, I was thinking I bet modern Qemu supports UDP transport for network traffic, and that it’d just integrate with HecnetNT just fine. And it does!

"c:\Program Files\qemu\qemu-system-x86_64.exe" ^
-m 1024 ^
-hda UnixWare711.vmdk ^
-net nic,model=pcnet,netdev=hecnet ^
-netdev socket,id=hecnet,udp=127.0.0.1:5001,localaddr=127.0.0.1:5000

Configuring the HetnetNT bridge was simple, as always make sure you have Wireshark/pcapng installed and simply run ethlist to get the list of interfaces:

D:\qemu>ethlist.exe
Network devices:
  Number       NAME                                     (Description)
  0  \Device\NPF_{E79F6278-3E7E-4547-955A-2080A0473AD6} (Local Area Connection* 8)
  1  \Device\NPF_{1D960E08-2A3A-43F7-BAD6-21FCB466717B} (Local Area Connection* 7)
  2  \Device\NPF_{98053A85-B049-45A0-AC33-961E2C136FCA} (Local Area Connection* 6)
  3  \Device\NPF_{BFA868ED-E508-4436-B085-EC815C4C544C} (LoopBack)
  4  \Device\NPF_{C75EAF23-0FA3-433B-B271-9CB0F5EB92D0} (VMware Network Adapter VMnet8)
  5  \Device\NPF_{B615DE21-AEC3-4347-916C-332AC4A4DA70} (VMware Network Adapter VMnet1)
  6  \Device\NPF_{82E5A370-6D3D-40AD-A9D5-C4E0E0C50F0D} (Ethernet)

And then create a simple bridge.conf file with the VMnet8 adapter and the UDP pariing to talk to Qemu:

[bridge]
update 127.0.0.1:5000
vmnet8 \Device\NPF_{C75EAF23-0FA3-433B-B271-9CB0F5EB92D0}


[tcpip]
update
vmnet8

Then launch the bridge program listening on port 5001:

D:\qemu>hecnet.exe 5001
Config filename: bridge.conf
Adding router ''update''. 0100007f:5000
Opening pcap \Device\NPF_{C75EAF23-0FA3-433B-B271-9CB0F5EB92D0}
Adding router ''vmnet8''. 00000000:0
Host table:
0: update 127.0.0.1:5000 (Rx: 0 Tx: 0 (Drop rx: 0)) Active: 1 Throttle: 0(000)
1: vmnet8 0.0.0.0:0 (Rx: 0 Tx: 0 (Drop rx: 0)) Active: 1 Throttle: 0(000)
Hash of known destinations:
Adding new hash entry [52:54:00:12:34:56]. Port is 0
Adding new hash entry [00:50:56:c0:00:08]. Port is 1
Adding new hash entry [00:50:56:f1:dd:d0]. Port is 1
Adding new hash entry [00:0c:29:9a:2b:fb]. Port is 1

It’s a little bit more involved to setup as we have to link the 2 programs via UDP, but I can say it’s totally worth it.

“It just works!” – Sydney

I can now easily FTP files into Qemu, and of course telnet as much as I want to. I don’t see why NFS wouldn’t work either.

Which brings us to the bigger elephant in the room, which one is ‘worth the squeeze’?!

I thought it’d be fun to do a totally unfair CPU intensive thing like building GCC. I would do a quick stage 3 compile blindly running this:

./configure --host=i386-sysv4 --target=i386-sysv4 --prefix=/usr/local/gcc-2.5.8
make
make stage1
make CC="stage1/xgcc -Bstage1/" CFLAGS="-g -O"
make stage2
make CC="stage2/xgcc -Bstage2/" CFLAGS="-g -O"
make stage3
make CC="stage3/xgcc -Bstage3/" CFLAGS="-g -O"

This way we can just look at the timestamps between completed releases. It does build C++ & ObjectiveC as well, and compared to machines from 1999 this is amazing!

-rwxr-xr-x 1 neozeed other 3495688 Mar 29 12:42 ./cc1
-rwxr-xr-x 1 neozeed other 2646888 Mar 29 12:37 ./stage1/cc1
-rwxr-xr-x 1 neozeed other 3495720 Mar 29 12:39 ./stage2/cc1
-rwxr-xr-x 1 neozeed other 3495688 Mar 29 12:40 ./stage3/cc1

Qemu timing

-rwxr-xr-x 1 neozeed other 3884076 Mar 28 20:12 ./cc1
-rwxr-xr-x 1 neozeed other 2647116 Mar 28 20:11 ./stage1/cc1
-rwxr-xr-x 1 neozeed other 3884124 Mar 28 20:11 ./stage2/cc1
-rwxr-xr-x 1 neozeed other 3884076 Mar 28 20:12 ./stage3/cc1

VMware timing

As you can see VMware is substantially faster when it comes to computation. This shouldn’t come to anyone as any surprise. And this isn’t a fair competition, but it does show that you can stage GCC on Qemu just fine, so that’s actually great!

Now let’s mix in some more nonsense, I have a Merge license so let’s try it! First off it really wants Windows 95 from CD-ROM. It will not accept anything else. I have a hacked-up copy of the floppy version of Windows 95 on CD-ROM, and it accepted that just fine, it appears to search through.CAB files looking for files to setup it’s preferred environment. I’m not all that familiar with the whole thing as PC’s are cheap, and virtual machines are even cheaper!

Merge setup on VMware

After the setup completed, I thought I’d try my Sarien 286/386 ports.

no DPMI for Merge

Sadly, neither worked. Maybe it’d have better luck with Windows 95 actually installed. I wanted some high colours so I went over to Qemu and found out that it cannot run Merge.

No Merge!

The error lies in a missing opcode 000000FF. Maybe it’s invalid to trigger an exception to call between DOS and the supervisor?

unknown opcode 000000FF

Either way it doesn’t matter, it doesn’t work. I did get feedback that it does run under KVM.

I don’t know why I didn’t think about doing the HecnetNT bridge earlier as it gives things far more flexibility for tapping into networks, or even being transported. I guess I should look at other transport mechanisms besides UDP since it’s 1:1. Also, it might be worth dropping the protocol restrictive filters to allow everything on the wire to flow.

Using MIT PC/IP with SLiRP & a virtual Cisco Router

In this video I’ll be covering the circa 1985 MIT PC/IP stack running on 86box trying to connect to a user mode network library, SLiRP. After that fails I’ll show how to break the stack apart so we can use WireShark to inspect the traffic, then how to replace the direct connection to SLiRP by using Dynamips to emulate a cisco 7200 router.

Caution it’s all command line!

I’ll cover adding a loop back adapter, installing WireShark, how to find the GUID’s of the interfaces, how to configure a HecNET bridge, and set it up to relay to a stand-alone version of SLiRP, then how to setup a virtual cisco router to do NAT, and also forward to SLiRP, along with taking network captures to show what is really going on!

Software used (in no particular order)

Or for those who prefer the written way…

what is going on?

In 86box, you have the ability to use the SLiRP library directly from the emulator. Which is all find and good, but sadly it gives you no visibility when things go wrong. And with MIT PC/IP things go wrong. Looking at the data through Wireshark sure would be nice, but how to we get it into there?

Well the simplest way is to just break it apart.

Broken apart into it’s components

Adding in a KM TEST loopback adapter to Windows now gives us a private network we can now attach programs to via the pcap API. Hecnet is a l2 bridge that can inspect and look for l3 traffic and then forward it via udp to another program. In this case I had made a version of SLiRP that will communicate via UDP, perfect for stuff like this!

One thing to keep in mind is that the ‘GUIDS’ of the network interfaces are unique to each system, the ethlist program will show you which is which. It’s also why renaming interfaces only helps you when dealing with old libpcap stuff!

Rename stuff so it makes sense! Otherwise, everything is Ethernet

Becomes:

C:\hecnet>ethlist.exe
Network devices:
  Number       NAME                                     (Description)
  0  \Device\NPF_{E7EB72FA-7850-4864-B721-2A3B38737214} (KM-TEST)
  1  \Device\NPF_{649448CA-969D-486E-AAC8-99F1993C701A} (Ethernet0)
Press Enter to continue...

C:\hecnet>

With this information in hand, creating the bridge configuration is quite simple:

[bridge]
Loop \Device\NPF_{E7EB72FA-7850-4864-B721-2A3B38737214}
update 127.0.0.1:5001

[tcpip]
Loop
update

The bridge is for an uncompressed normal bridge connection between the KM TEST loopback interface and a UDP connection listening on port 5001 on localhost. Of note it’ll be forwarding TCP/IP related packets. Since we want the bridge to listen on UDP port 5000 we simply run it like this:

hecnet.exe 5000

Running the SLiRP redirector is just a simple matter of telling it which port to listen on, and where to forward traffic. In this case we’ll listen on port 5001 and forward traffic to 5000 on the localhost

slirp_rdr.exe 5001 127.0.0.1 5000

Thankfully, it’s that simple!

Running a ping fails (yet again) but this time we can see that they are doing ARP but for some reason PC/IP does not acknowledge the SLiRP library.

Just to verify, the HecnetNT bridge does see the source and destination address, and the SLiRP does indicate traffic in and out as expected.

Clearly the fault is on the PC/IP side, and most likely because it’s so old.

I then decided to build another network, this time using Dynamips to add in a virtual router.

Adding in a router

This complicates things as I’m not sure how to control the internal routing of the SLiRP library so the router has to NAT the PC/IP traffic to SLiRP, which in turn ‘NATs’ it to the internet. But rest assured double NAT (or even more) is quite common these days.

Configuring the router is somewhat straight forward, we are going to use pcap to listen on the KM TEST loopack, replacing the HecnetNT bridge. But it’s going to talk to the SLiRP redirector in the same manner:

set loopback=\Device\NPF_{E7EB72FA-7850-4864-B721-2A3B38737214}
set IOS=..\c7200-is-mz.19991126.bin
set NPE=npe-100
set RAM=64 -X

@attrib *.* -r
..\dynamips.exe -P 7200 %IOS%  ^
-m %RAM% ^
-t %NPE%  ^
-p 0:C7200-IO-FE ^
-p 1:PA-4E  ^
-s1:0:gen_eth:%loopback% ^
-s1:1:udp:5000:127.0.0.1:5001

This creates a basic 7200 router with a 4 port ethernet card, with one port connected to the KM TEST loopback, and the other connected to the SLiRP library.

Configuration of the router is not very complicated either:

!
no ip domain-lookup
!
interface Ethernet1/0
 ip address 192.168.1.1 255.255.255.0
 no ip directed-broadcast
 ip nat inside
!
interface Ethernet1/1
 ip address 10.0.2.15 255.255.255.0
 no ip directed-broadcast
 ip nat outside
!
ip default-gateway 10.0.2.2
ip nat inside source list 1 interface Ethernet1/1 overload
ip classless
ip route 0.0.0.0 0.0.0.0 10.0.2.2
no ip http server
!
access-list 1 permit 192.168.1.0 0.0.0.255
!

This defines our default route for both the routing table, and the management engine to the SLiRP library, defines the NAT inside/outside interfaces along with specifying the ‘overload’ address will be the 10.0.2.15 NAT’ing the PC/IP traffic behind the usual SLiRP user address.

Pinging the SLiRP gateway

This allows us to ping SLiRP, and get the expected response.

Working ARP/ICMP with cisco router

Checking the capture, we can see that yes ARP is working as expected, and the ping works without any issues.

On the router we can see the NAT translation.

Router#show ip nat translations
Pro Inside global      Inside local       Outside local      Outside global
tcp 10.0.2.15:4376     192.168.1.5:4376   71.95.196.34:23    71.95.196.34:23
Router#

And we can also check the SLiRP redirector for information on the current session.

SLiRP redirector started!
Press 's' for SLiRP stats
Press 'e' to exit.

Sent:           Recv:
stats!          4859

Proto[state]     Sock     Local Address, Port  Remote Address, Port RecvQ SendQ
tcp[ESTABLISHED]  632         10.0.2.15  4376    71.95.196.34    23     0   600
tcp[REDIRECT]     616         10.0.2.15    23        10.0.2.2 42323     0     0

Plus, we also have the Wireshark capture going showing the start of the TCP conversation

TCP connected!
Connected to VERT

So now we’ve connected to the internet and by breaking the process appart we can now inspect what is going on, and made modifications like adding a cisco router.

I figured that this may be something that other people may be interested in, as you can build far more complex virtual networks this way!

Announcing Cockatrice III 0.5g

Now with better pcap filter in place. I had an issue with 86box and how NetBEUI wasn’t working. It didn’t hit me at the time but cold-brewed caught it, that the 86box didn’t have the multicast addresses in the default pcap filter.. We need the filter to not send EVERY packet to the VM, and to also filter out the VM’s own packets so it doesn’t loop and send traffic to itself (which hopefully would just get discarded, but there is no point doing it in the first place!)

Before I had it set to this:

(ether dst 09:00:07:ff:ff:ff) or (ether dst ff:ff:ff:ff:ff:ff)

It’s allowing one Multicast address for AppleTalk and the general ethernet broadcast address. Which is more or less okay but for everything else, you want to catch a wider net. The better choice is to use built in pcap macros:

((ether broadcast) or (ether multicast)

I can confirm that these do work for 86box, so I’ve copied the same into Cockatrice III. For those who wonder what the difference between Basilisk II & Cockatrice III is, I basically took a super old version of Basilisk II, ripped out as much platform code as I could, re-ported it to basic SDL, removed all the fancy clipboard/meta/drive sharing integration code, added a pcap network option to use the raw network for AppleTalk, and took the SCSI emulation from Previous so I can partition and format virtual ‘SCSI’ disks that I’ve even been able to use in a BlueSCSI!. I’ve sat down with a debugger some years ago and went through the SLiRP code catching as many faults when using Internet Explorer as I could, I think most of the ‘fix’ involved renaming clashing symbols, and while it’s not perfect it was a lot more stable than the default stuff. Although I haven’t touched it in years, and probably should look to borrowing a more modern version from elsewhere.

0.5g in action

Linking this thing with TDM GCC is becoming a bit of a challenge so in case I forget here is how I’m currently statically linking libgcc/libg++ along with winpthread. All it should need now is SDL & working WinSock which every Win32 should have!

c++ -o CockatriceIII.exe        obj/main.o obj/main_sdl.o obj/prefs.o obj/prefs_dummy.o obj/sys_unix_sdl.o obj/rom_patches.o obj/slot_rom.o obj/rsrc_patches.o obj/emul_op.o obj/macos_util.o obj/xpram.o obj/xpram_dummy.o obj/timer.o obj/timer_common.o obj/clip_dummy.o obj/adb.o obj/serial.o obj/serial_dummy.o obj/ether.o obj/sony.o obj/disk.o obj/cdrom.o obj/scsi.o obj/video.o obj/video_sdl.o obj/audio.o obj/extfs.o obj/user_strings.o obj/user_strings_unix.o obj/sdl_pcap.o obj/scsi_dummy.o obj/audio_sdl.o obj/prefs_editor_dummy.o obj/basilisk_glue.o obj/memory.o obj/newcpu.o obj/readcpu.o obj/cpustbl.o obj/cpudefs.o obj/cpuemu.o obj/fpu_ieee.o cockatricerc.o ..\slirp\libslirp.a ..\drmingw-0.7.7-win32\lib\libexchndl.a -Wl,-Bdynamic -lSDL -lws2_32  -liphlpapi -Wl,-Bstatic -lgcc -lstdc++ -lwinpthread -static -static-libgcc -static-libstdc++

I’ve been able to map to my internal servers just fine, and connect using Internet Explorer so I suppose everything is fine!

All the cool kids love github so here you got, you can get it from the latest releases:

Releases · neozeed/CockatriceIII (github.com)

I’ll have to re-sync sourceforge. Although I do keep more resources on sourceforge since it’s clearly a lot better at file hosting.

Building MIT PC/IP, or making apple pie

“If you want to make a pie from scratch, you must first create the universe”

–Carl Sagan

A little while ago I had touched briefly on the availability of of a PCC port to the 8086 done back in the early 1980’s that was hosted on VAX running 4.1BSD. I’d ruled it basically useless as you are restricted to 64kb .COM files, and I couldn’t get much of anything interesting running on it.

After all the fun of setting up NetManage Chameleon on Windows 3.0, over on IRC lys had mentioned I should try the MIT PC/IP stack. I never knew anything about it’s history but it became the first PC TCP/IP stack. You can read more about it from Internaut?

Dave Clark had gone to England for sabbatical and while he was there, had implemented TCP/IP in BCPL for the TRIPOS operating system, a predecessor of the Commodore AMIGA operating system. He brought the TCP/IP code back with him, and the Lab for Computer Science had a bunch of Xerox Alto workstations, and someone at LCS ported Dave’s TCP/IP to the Alto.

Then someone ported it to Version 6 UNIX and rewrote it in C. And that was what we took, and ported to PC DOS. At that point there were no C compilers that ran on the PC, and we were using a compiler that ran on a PDP 11/45 on Version 6 UNIX, and then on a VAX 750 running BSD v4.1. That was the AT&T; portable C compiler, and a group of people on the fourth floor of the LCS had written an 8088 code generator for it. This was before Microsoft C, and before 4.2 BSD.

Our first tasks were to bring up TFTP, TCP, and a telnet client under DOS. Several people were involved. Lou Konopelski did the initial TCP and telnet work, and Dave Brigham did similar work to what I did.

John Romkey – InternautHow PC-IP Came to Be

What is even more notable about PCIP is that it’s the reason the whole ‘MIT License’ even exists, as word seems to have spread quickly about a TCP/IP stack for the IBM PC compatible market. Jerome Saltzer has documented it all, if you want to read about it (WARNING PDF!)

Romkey would even then go on to found FTP software in that wonderful pre public internet era, before the eternal September.

Over on bitsavers there are 3 files:

[   ]8086_C_19850820.tar2019-03-12 14:36750K 
[   ]PC-IP_19850124.tar2019-03-12 11:534.6M 
[   ]PC-IP_19860403.tar2019-03-12 11:536.9M 
bitsavers.trailing-edge.com/bits/MIT/pc-ip/

Of course, the one thing that stands out is that these files look tiny. They aren’t even compressed! PCC, or the Portable C Compiler was released from AT&T, itself written in C, to make porting the C compiler easier to further allow Unix to be further easily ported. Now that I kind of had a mission, I decided to take the 8086 PCC leap, again.

Get the time machine ready!

A man, his best friend and a time machine! – Microsoft Sydney

Thankfully I had that 4.1c BSD image still up on sourceforge, aptly named: 4.1c BSD.7z, so I could follow my old instructions to create the tap file and start working with 8086 C, going back from 1985. And in no time, I had re-built the compiler, and assembler up and running. But I wanted more, as much fun as 4.1BSD is, I wanted to run everything natively on Windows.

At this point I’d remembered that this setup is a bit odd in that the object files that the assembler produces are OMAGIC (type 0407) a.out files. Thinking back to my old project of building Ancient Linux on Windows using vintage tools, it also uses OMAGIC a.out files! It’s not that unexpected as the original GNU ld linker from 1986 has hooks for both old magic & new magic (OMAGIC/NMAGIC) files, as this would be consistent from the era. Thinking this was my out, I might have a way of migrating the build process off of the VAX.

The first pass was to try to pull in all the VAX includes into my native Visual C++ 1.0, and get it to build with Microsoft C/C++ 8.0. The next thing to do of course, is look for where it’s doing anything with binary files and make sure it’s all set to O_BINARY/”rb”/”wb” where appropriate as MS-DOS/Win32/OS2 all handle text files differently from binary data. There is also fights with mktemp along with procedure name creep, like how ’round’ wasn’t a thing in 1980 but it sure is by 1993! Before the era of the 486DX/68040/Pentium not everyone had a math co-processor (FPU) so it’d make sense that a lot of things wouldn’t have this by default.

As a quick refresher the following diagram may be specific to the GNU GCC compiler, but the older PCC compiler uses the same methodology of first pre-processing files, then compiling them, assembling the resulting compiler output, then finally linking to an executable program. ( See – “So it turns out GCC could have been available on Windows NT the entire time“)

a rough explanation of how old C compilers work in stages

The steps for PCC 8086 are quite similar but to spell them out:

  • Pre-process with GNU cpp
  • Compile with PCC’s c86
  • Assemble with PCC’s a86
  • Link with GNU’s ld
  • Extract the MS-DOS .COM file with cvt86

There isn’t much to talk about the pre-processor, so I’ll skip it, suffice to say from my cl386 research, and porting GCC to OS2/NT, it just worked.

Compiling the compiler

Surprisingly getting the compiler running wasn’t too difficult. I do remember getting this running before, so seeing it run again wasn’t too much of a surprise. The simple C program:

main(){
printf("hi from 8086 pcc\n");
}

Gives us the following generated assembly:

        .data
        .text
        .globl  _main
_main:
        push    bp
        mov     bp,sp
        push    si
        push    di
        sub     sp,#LF1
        mov     ax,#L14
        push    ax
        call    _printf
        pop     cx
L12:
        lea     sp,*-4(bp)
        pop     di
        pop     si
        pop     bp
        ret
        LF1 = 0
        .data
L14:
        .byte   104,105,32,102,114,111,109,32
        .byte   56,48,56,54,32,112,99,99
        .byte   10,0

So far, so good, right! Even for fun, I was able to build it using Microsoft C 6.0! I figured I may as well try to get as much out of that purchase as possible.

Strage binary formats in a strange world

One thing that was a constant problem for me is that the assembler generated garbage, it never gave me the a.out OMAGIC file. Opening it up in a hex editor, Hex Editor Neo, and it showed problem, right away.

A simple do nothing program, assembled on the VAX

The OMAGIC sequence in hex should be written down as 07 01, but when I looked at the output from my PC port the file was not only too big but it didn’t have the headder:

The same program assembled on the PC

As you can see it’s just a bunch of zeros up front. Later on, I’d realize this was a ‘pad’ so it could be filled in later by the assembler when doing relocations. The file in question was rel.c which also should have been the hint. I don’t know if anyone saw it, but let me highlight it for you:

The OMAGIC header is being appended!

As you can see, where the file ends on the VAX, on the PC the OMAGIC header is being appended. I did a simple cut & paste in the editor, and the object file validated just fine. So why was this happening? In my rush to just add ‘binary’ flags to any file operations I had seen this in rel.c:

		(dout = fopen(Rel_name, "a")) == NULL)

I had taken this be an ‘append’ for whatever reason it would need to do that kind of thing. Well maybe that’s what it means in 1993, but in 1981 it doesn’t append, instead it just opens it normally. Is this a bug in the assembler, or a feature of 4.1BSD? Without debugging it, I just modified it to not append, as this was the only occurrence of an explicit append in the source code I could see.

		(dout = fopen(Rel_name, "wb")) == NULL)

And that did the trick, I now had a working assembler running on my PC!

But we are not out of the woods yet!

Naturally trying to build a much ‘larger’ Fibonacci program crashed the assembler. Luckily debugging it was a snap to find out that it was trying to free a static pointer. Or so I think. Today, in the future RAM is cheap, so I just commented out the offending free and now it was off to the linker.

When is advanced optimization a bad idea?

When the machine you wrote this for no longer exists. In the middle of the ld86 linker is this line:

		asm("movc3 r8,(r11),(r7)");

I have no idea why it’s there.

I don’t know what it should be doing.

This makes the linker un-portable.

However, as I’d mentioned before I did have the GNU linker that I’d successfully used to build Linux kernels, so there was hope!

C:\msvc32s\proj86pcc>\aoutgcc\bin\ld.exe -X -r -o hi.out crt0.b hi.b ./libc.a
C:\msvc32s\proj86pcc>cvt86 hi.out hi.com
C:\msvc32s\proj86pcc>msdos hi.com
hello from pcc for 8086!

I had now successfully run my first program without using the VAX. Although I had not mentioned a step yet, cvt86, this utility is described as creating a MS-DOS .COM file from an a.out file. I didn’t look into how it accomplishes this, but basically, it’s another linker. And this issue would come to complicate things as I had thought that everything was working.

libc & the heart of C

Libc, is simply put the central C library for getting everything done. While crt0 will setup the C environment everything else core from opening files, writing to the screen, and reading keyboard input is done through libc. So I thought re-building libc would be easy enough. To build the library you ‘archive’ them with the ‘ar’ archiver, then index them with ‘ranlib’. And again, from my a.out adventures building Linux I had both tools, however no matter what I was doing they did not work with cvt86. I wen’t back and rebuilt some kernels to verify, and yes it does generate archives but cvt86 was not happy.

I still had the SIMH VAX running in case I needed it, so I just broke down and figured I’d just port the VAX ar/ranlib to the PC. I don’t know off hand what the problem was, and I didn’t feel like spending an eternity to try to correct it, and both of the programs are somewhat portable. But of course it wasn’t that simple as ar opens files in strange ways that work on 4.1BSD but yeild chaos on the PC.

#define ARMAG   "!<arch>\n"

#define SARMAG  8

#define ARFMAG  "`\n"

‘ar’ has it’s own magic, a simple !<arch> and a ` followed by a new line. On any UNIX the \n is 10 in decimal 0xa in hex. But on the PC it’s carriage return and linefeed! And yes despite setting all the opens to binary, it was constantly injecting carriage returns & linefeeds all over the place! Some-how the file was being opened in text mode. Thankfully debugging even in old Visual C is great and inspecting the temporary files lead me to find this part:

			f = creat(file, larbuf.lar_mode & 0777);

In a few places it uses the creat (or create call. In an interview Dennis Ritchie had mentioned that this was one of his regrets in life, not naming creat create) call, which of course never has to set a mode, as everything is binary in Unix, unlike the PC. Great.

Luckily the fix was very simple after every creat, simply set the file mode to binary.

			setmode(f,O_BINARY);

Great!

Apple pie!

Fibonacci sequence

Now I could re-build libc from source and link it to the Fibonacci program!

Yes it’d take this long to get to the point where I can now easily edit file on a modern machine and have a Win32 native toolchain! VAX no longer required! We’ve successfully extracted everything we needed from the 1980’s!

First contact!

Now it’s time to look at what brought us on this adventure, MIT PC/IP.

The MIT PC/IP (PCIP) does require changes to the libc to work correctly. Unfortunately, they didn’t provide the full copy of the libc, but rather some ed scripts. So, the first question is do I even have the version these are based off of to start? I don’t know, so I had guessed, and I had guessed incorrectly.

3com 3c501

Configuring PCIP is somewhat involved, first you need MS-DOS 2.00 or greater which thankfully in the future is FREE! The next thing you need is a 3com 3c501 card. This is going to be a challenge but it’s just as any good time to mention 86box, and the latest version that I’ve been using that of course supports this kind of setup!

New version 4.1.1

I can’t recommend it enough, 86box is like all the inconveniences of old software & hardware without having to spend a fortune for weird combinations, fighting to have space for it. I naturally setup a 386sx with CGA, 20Mb hard disk and a 3c501 card. It’s nice being able to also be very task specific, this doesn’t have to be a DooM/Quake machine!

the first thing you need to do is add the netdev.sys device driver that is created as part of building PCIP from source. a simple:

DEVICE=\NETDEV.SYS

in your config.sys is more than enough. However, how do you configure it? Well it’s the ‘custom’ program that binary edit’s the device driver.

YES, IT EDITS THE DEVICE DRIVER.

Setting stuff up is somewhat straight forward, however it displays TCP/IP information in decimal not in hex. I haven’t even looked into the how or why.

The first page

The first page options are kind of banal, it’s back in the day when people would use finger to find people on the internet and call them up or send emails. How cute. 1985 was a different world!

hardware options

In the hardware options the only thing to check is the I/O base, IRQ & DMA for the Ethernet card. I just configured the card around 0x300/5/1 as it’s great being purpose built!

telnet options

There is a separate window for telnet options. Naturally high speed connections frame far too fast for something built from 1985. I found lowering the TCP windows really helped with pacing.

Site config

As I had mentioned the site configuration displays all the information in decimal. Also, I’m wasn’t sure what is going on with the netmask, but looking at the old Windows calculator revealed it was being stored in OCTAL. It’s probably why the addresses have commas instead of periods. Although I had configured DNS it didn’t work, as it uses UDP port 42. It’s clearly doing something very early 1980’s.

The status CR/LF is broken!

So close and yet so far away. The only thing to do was try to figure out which of the libc stuff was ‘newest’ in the pure state and try to figure out where to go from there.

Redo!

While I had not configured the libc correctly, I had partial success! I could actually establish a telnet session! Libc wasn’t working correctly as every line feed didn’t generate a carriage return, as you’d need for MS-DOS leaving it with broken status.

But at the same time, despite all the weird ‘challenges’ for the most part ‘it just worked’. Which is pretty cool!

It turns out the answer was the 8086_C_19850820 file. As far as I can tell there was only one thing that didn’t patch correctly but I was able to build a libc in no time.. that didn’t work. In the patch it removes ulrem/auldiv from arith.a86 Not sure why, I haven’t messed with it. But that means I had to restructure to build with the non floating point n86c compiler as that’s the way PCIP is expected to be built. After rebuilding with this compiler and this seemingly properly patched library I finally had it working!

Ping my local gateway!

Instead of a garbled mess, I had something I could read!

telnetting to my test BBS

Now instead of a garbled mess, I can see it was trying to display the connected IP, and a clock.

Sadly it doesn’t work with SLiRP. I’m sure it’s either classful routing or it really doesn’t like how SLiRP handles ARP. I suspect it’s also trying to do old style classful routing as well, which means you can’t just load arbitrary subnet masks wherever you want, to try to squeeze the 4 billion IP’s out of the internet.

The updated telnet client connecting to a test BBS

Final thoughts

I suspect that although there were binaries in the above tar files, going through the effort to rebuild PCIP really wasn’t all that expected for most people to carry out. Sadly, there was no shared source ‘sites’ online, and we’re lucky enough someone kept a few tarballs lying around. I really can’t blame them for sticking with then current development tools, especially for what you’d need to build a C compiler back in the early 80’s. It’s a shame the QL or the Macintosh didn’t have the RAM or the DASD capacity to become that home cross compiler of the 80’s.

Most project just require you to work on that actual project, while this has been a substantially larger undertaking from anything normal, but I guess I’ve learned a bit along the way with all those “pointless” GCC port things I’d done, well it turns out they are incredibly useful! It’s been a fun archeological expedition for me, thankfully C is still a thing, I wonder what happened to all the ADA/Perl/Pascal/”Wave of the future” stuff that is always disappearing. At least more and more people work on full system emulation so there is always that!

For anyone that curious you can find all the code over on github:

https://github.com/neozeed/8086pcc

Against my better judgement, I’ve added a binary package on github.

So, I used to think Lan Manager Domains and early NT Domains were the same thing

Obviously, this was wrong.

And besides the NetBEUI being eaten on the network for no reason I can see, as I’m sure these machines should be able to talk to each-other this was the end result:

The OS/2 domain is not a Windows NT domain

And what about not trying to create a machine account?

The domain controller for this domain cannot be located.

I should have expected this not to work.

Having dumped NetBEUI for TCP/IP, I can see them talk, it’s not a name resolution issue or anything like that. On the flip side can LAN Manager join a Windows NT domain? I’m not sure on that one either.

And not too surprising using the LAN Manager DOS client I can log onto the OS/2 domain just fine.

Kind of verbose and annoying but yes, it works!

However, despite LAN Manager 2.2 providing a TCP/IP stack to connect to both OS/2 and NT servers, there is no winsock interoperability dll. Do I really have to load more than one NIC and stack at the same time?!

The one thing I had been hoping to build up to was using mailslots, a UDP like IPC/RPC mechanism from back in the old dark days of early LAN Manager 1.0 The can be broadcasted to all nodes on the network that are listening by writing to \\*\MAILSLOT\<YOUR LOCAL BOX>. On the surface these broadcast type things are modern day terrible, we prefer lookup services like DNS, but in the 80’s it’s not like people were going to put tens of thousands of machines on a single network…

The Mandelbrot example

I cannot thank my Patrons enough for this attempt at doing something multitenant as I really did need Microsoft C 6, and the Windows 3.0 SDK. The example from the Lan Manager 2.0 Programmer’s Toolkit (Why was this stuff never in the base SDK?!) shows an OS/2 LAN Manager service providing rendering services over the network to render the Mandelbrot to the Win16 client. It’s actually very neat. It really gives OS/2 that pre-Windows NT feel, with the services as they are not in your face, although at the same time I’ve found that I had to do an interactive logon to get things started, so maybe LAN Manager OS/2 servers were not “Lights out”? I guess I need to look more into it, as it just feels more and more how NTOS2/ clearly grew out of OS/2 + LAN Manager.

Obviously as soon as I see this, thanks to getting my hands on the OS/2 6.78 network client, I also see it’s not only obsolete but going to be removed. If anything, it’s impressive that an OS/2 feature has remained in NT for so long.

The LAN Manager 1.0 disk sets, actually include headers & libraries, but no examples.

I had wanted to do something with TCPIP and mailslots, and I had figured that Windows NT would be the best glue being in that perfect space of OS/2 compatibility and robust TCP/IP, but I wasted far too much time to basically see that if they are not part of the same domain, the mailslot’s just don’t work.

I haven’t given up, but I primarily used Netware for PC networking back in the 90’s so this is all kind of new to me. Looking through resource kits online there doesn’t seem to be a lot of material about integrating LAN Manager into a NT Domain.

I’ll have to re-think this.

The world vs NetBEUI

UPDATE* ME from the future, I was ind of right in that something was missing, nothing was eating the trafic, rather 86box only had the all MAC broadcast address set as a filter, and older protocols like NetBEUI instead use magical protocol-based multicast/broadcast addresses. Hopefully in an upcoming release of 86box it’ll be updated!

NetBEUI

I don’t know what is going on, other than it feels like something is filtering me on either Windows 10 or 11. I have a Windows 2000 server with NetBEUI running under Hyper-V, and trying to map to it from OS/2 using the Lan Manager 2.2 disks floating around (you have to merge them as some stuff is missing like #2 driver disks), lets you map directly but finding each-other seems to be difficult. Even trying to rebuild on the same machine using either a MS Loopback interface, or a VMware interface also has endless weirdness, or in the capture above they can be seen on Wireshark speaking but not hearing each-other. I thought it was LAN drivers, or the old friends, IO/IRQ/DMA channel issues, but after trying various configs, I tried TCP/IP and..

TCP/IP

They find each-other right away. Instantly, it just works. Have we hit the point where there is silent filters screwing up non TCP/IP protocols using pcap injection on Windows? Am I going insane?

WLO for OS/2

Speaking of OS/2 weirdness, I was doing some SQL installs, and I came across disk images for 4.1A, and in there is the graphical tools for Windows & OS/2! Although I didn’t think Microsoft ever did port the graphical tools to OS/2, and I’d be right. However on setup disk 2 is WLO/Porthole! Turns out this was used in a shipping product! Unlike the one for Excel 3.0, or the applettes, this installs into the C:\OS2\DLL directory! So, this is the real deal!

Admin 2000 from OS/2

I could logon to my Windows 2000 server from OS/2, but the opposite would never work. On the capture I just see it endlessly trying to find by name, but OS/2 is silent.

I must be missing something obvious somewhere.

I’m hoping to be building this towards something, so I’ll update later.

Setting up SNA networking with Hercules using DLSw

I finally broke down and made a quick video on setting up the DLSw ‘lab’ that I had uploaded on Internet Archive.

See it works!

Although I should have gone more in depth with the cisco part.

Namely showing how to check the interfaces, the ethernet l2 traffic, how the DLSw peers, and then the establishment of the circuit once the session is established.

But I wanted to be quick. I don’t think I can edit a video that is up so I’ll probably follow it up with another quick video.

simple network diagram

Trying to put it into words, the Windows 3.1 VM hosts Extra! 4.2 talking SNA to the Loop_SNA virtual Ethernet interface, which then is connected to the virtual cisco router, which also has a TCP/IP enabled interface, Loop_TCPIP, which the host Windows 10 machine can talk to, allowing it to communicate with the Hercules VM which I had compiled to run as a native Win64 EXE for Windows.

Basically, at it’s heart, this is the important part of the cisco config:

source-bridge ring-group 1
dlsw local-peer peer-id 192.168.146.5
dlsw remote-peer 0 tcp 192.168.146.1
dlsw mac-addr 4000.1020.0100 remote-peer ip-address 192.168.146.1
dlsw udp-disable
dlsw transparent switch-support

interface FastEthernet0/0
 ip address 192.168.146.5 255.255.255.0
 no shut

interface Ethernet1/0
 dlsw transparent redundancy-enable 5555.5555.5000
 dlsw transparent map local-mac 4000.1020.0100  remote-mac 4000.0999.0100
 no shut

I used a ‘clean’ VMware virtual machine to host this test, just to show how to do the setup from scratch, taking nothing for granted.

Hopefully this explains it a bit better for those who wanted to know, along with the video to show the high level of it working. Not sure anyone would want to watch me stumble through setting up MVS, along with going into more detail on the cisco config & workstation config.

As always, thanks to 9track for providing the patches, and showing that this whole setup was possible!

Installing NetManage Chameleon on Windows 3.0!

After seeing the spotlight on twitter from WinWorld, on NetManage Chameleon, an old TCP/IP stack that supported Windows 3.0! With more details over on the forum. I was inspired to set it up myself.

I did go a bit overboard showing how to install MS-DOS & Windows 3.0 on Qemu. Maybe it’ll help someone who wants to try to use Qemu, but is too scared? Maybe I moved too quickly.

One thing I did do differently in this run, is launching the monitor and a serial port as tcp servers so I could telnet into the VM, effectively having a way to share text like a clipboard back and forth. I’m kind of surprised I hadn’t really started using Qemu in this manner much earlier.

qemu.exe -L pc-bios ^
-m 16 ^
-hda apricot.vmdk ^
-net nic,model=pcnet -net user ^
-monitor telnet:127.0.0.1:4000,server,nowait ^
-serial telnet:127.0.0.1:4001,server,nowait ^
-fda yourdisk_here.vfd

Surprisingly it went surprisingly well, other than my goof of having the OS/2 driver instead of the MS-DOS driver for the nic.

Sadly, the tn3270 program bundled with Chameleon doesn’t work properly with Hercules.

As always I’ve uploaded it to archive.org: apricot-dos4-win3-chameleon3.7z

The Rise of Unix. The Seeds of its Fall. / A Chronicle of the Unix Wars

It’s not mine, rather it’s Asianometry‘s. It’s a nice overview of the rise of Unix. I’d recommend checking it out, it’s pretty good. And of course, as I’m referenced!

The Rise of Unix. The Seeds of its Fall.

And part 2: A Chronicle of the Unix Wars

A Chronicle of the Unix Wars (youtube.com)

Years ago I had tried to make these old OS’s accessible to the masses with a simple windows installer where you could click & run these ancient artifacts. Say 4.2BSD.

Download BSD4.2-install-0.3.exe (Ancient UNIX/BSD emulation on Windows) (sourceforge.net)

Installing should be pretty straight forward, I just put the license as a click through and accept defaults.

Starting BSD via ‘RUN BSD42’ and the emulator will fire up, and being up a console program (Tera Term) giving you the console access. Windows will probably warn you that it requested network access. This will allow you to access the VAX over the network, including being able to telnet into the VAX via ‘Attach a PTY’ which will spawn another Tera Term, prompting you to login.

telnettting into the VAX

You can login as root, there is no password, and now you are up and running your virtual VAX with 4.2BSD!

All the items

I converted many of the old documents into PDF’s so you may want to start with the Beginners guide to Unix. I thought this was a great way to bring a complex system to the masses, but I’m not sure if I succeded.

776 downloads

As it sits now, since 2007 it’s had 776 downloads. I’d never really gotten any feedback so I’d hoped it got at least a few people launched into the bewildering world of ancient Unix. Of course I tried to make many more packages but I’d been unsure if any of them went anywhere. It’s why I found these videos so interesting as at least the image artifacts got used for something!

But in the off hand, maybe this can encourage some Unix curious into a larger world.

Other downloads in the same scope are:

Enjoy!