Thanks to shadyjesse Philpem’s FreeBee can now run the C compiler!

I call it Freebee with C!

Again super thanks to shadyjesse for finding and fixing the larger issues, and philpem for his great emulator, freebee!

So 1970’s

I have to say, having never played with an AT&T Unix PC, it’s kind neat with this windowing non X11 UI. Although even in emulation it’s incredibly slow. But such was the Unix microprocessor revolution of the era, it’s crazy to think the mighty SUN-2 is also on the same level of performance, although SUN would at least go the way of the 68020 before giving up on the 68k for SPARC.

Even though the 68000 lacked the ability to recover from bus faults, allowing a better path to UNIX with the 68010, OEM’s still brought their own MMU technology to flesh it out, leading to divergent systems. Not that it mattered all that much for AT&T as they started to establish themselves as the new defacto go to UNIX vendor they quickly abandoned the market leaving the Unix PC, and 3B2’s to die off. While so many like to think that the ‘Unix’ business is booming, it really only boomed once AT&T exited the market until Linux had started to gain enough mindshare post 1.0… Which also included 68000 support, although aimed for the the stronger 68030/68040’s.

Anyways I’m sure you didn’t come here for my ramblings about the 68000 instead you want an easy to run package to click and GO!

So here, you are, freebee based on build d3c9486 of freebee.

There are two executables, for normies, tourists, and people only wanting to witness the fun it doesn’t matter which one you use. For anyone wanting to install the 3B1 Unix, you’ll want “freebee-10sec-O2.exe”. Since the 3B1 uses a non standard format, if you want to use FAT 360kb disks from a PC emulator then you’ll need “freebee-9sec-O2.exe”. Isn’t compatibility great?

emulator-sun-2

Since I was playing with the 68000 based GCC ’87 I know it was going to be more geared to SUN workstations, certainly of the early 80’s vintage as they would be the most ‘affordable/cheap/donated’ to FSF (Or so I’d imagine).

Naturally the go to emulator is TME, however this time while searching around for the install scripts and stuff I found lisper‘s (heeltoe.com) emulator-sun-2, a greatly cut down and SUN-2 focused emulator that emphasizes ease of use.

Wait, what? SUN-2, and ease of use? Why yes, not only that, as it uses SDL 1.2 it also means it’s much easier to compile. After an hour of messing around with it, I had it running on Windows. After a few minutes I had it running on my ARM based Acer NovaGO.

At it’s core is the m68k 68010 emulation from Karl Stenerud‘s Musashi core which is a great choice for the SUN-2 as it’s a 68010 based machine. Some fun notes from web.cuzuco.com/~cuzuco/sun2/ include:

  • CPU is a Motorola 68010 running at 10MHz
  • Maximum physical memory is 4 Megabytes
  • Maximum virtual memory is 16 Megabytes
  • All I/O is via a Multibus (an Intel design)
  • Main disk is a SMD, the largest size is 380Mbyte
  • Has a SCSI adapter, but the disk is slow and small (42Mbyte)
  • Sun was just finishing NFS
  • alludes to future AT&T UNIX System VI and VII
  • Display supported dual heads and a resolution of 1152×900
  • List price as tested: $44,900
  • Sun was still private, had 400 employees and sold 1500 units

You can read about the debut of the SUN-2 in the UNIX/WORLD Magazine, VOlume 1, Number 5 dated October 1984 in archive.org. It starts on page 86.

I started to integrate sigurbjornl’s patches for networking but I think I need to work through SunOS 2.0’s weird VAX 4.2BSD arp issues (anyone have the source code to SunOS 2.0?!). I’ll probably update it with UDP or some fixed ARP thing to remove that or just let the SUN-2 talk to a VAX with 4.2BSD so they can be weird, together.

I’m also pretty sure my old Cockatrice III sort of debugged SLiRP thing broke the packed structs to let it work properly when compiled with Microsoft C, so I’ll have to break down and either try to fix that, or update and borrow the vastly updated SLiRP from SIMH.

For Windows users who want to play along the bundle is on the terribly named page “Ancient UNIX/BSD emulation on Windows” as SUN2.zip.

Confessions of a paranoid DEC Engineer: Robert Supnik talks about the great Dungeon heist!

What an incredible adventure!

Apparently this was all recorded in 2017, and just now released.

It’s very long, but I would still highly recommend watching the full thing.

Bob goes into detail about the rise of the integrated circuit versions of the PDP-11 & VAX processors, the challenges of how Digital was spiraling out of control, and how he was the one that not only championed the Alpha, but had to make the difficult decisions that if the Alpha succeeded that many people were now out of a job, and many directions had to be closed off.

He goes into great detail how the Alpha was basically out maneuvered politically and how the PC business had not only dragged them down by management not embracing the Alpha but how trying to pull a quick one on Intel led to their demise.

Also of interest was his time in research witnessing the untapped possibilities of AltaVista, and how Compaq had bogged it down, and ceded the market to the upstart Google, the inability to launch a portable MP3 player (Although to be fair the iPod wasn’t first to market by a long shot, it was the best user experience by far).

What was also interesting was his last job, working at Unisys and getting them out of the legacy mainframe hardware business and into emulation on x86, along with the lesson that if you can run your engine in primary CPU cache it’s insanely fast (in GCC land -Os is better than -O9).

The most significant part towards the end of course is where he ‘rewinds’ his story to go into his interest in simulations, and of course how he started SIMH when he had some idle time in the early 90’s. SIMH of course has done an incredible amount of work to preserve computing history of many early computers. He also touches on working with the Warren’s TUHS to get Unix v0 up and running on a simulated PDP-7 and what would have been a challenge in the day using an obscure Burroughs disk & controller modified from the PDP-9.

Yes it’s 6 hours long! But really it’s great!

I’m not dead

It’s just been really busy with this move, unpacking and the usual losing things, finding things and breaking things.

In the middle of it all, I found something online, that I want to at least do some proper article thing about… As it’s been really exciting, and goes back to the first month I started this blog.

Outside of NeXTSTEP, the other i386 commercial version of Mach 2.5 surfaced, the Mt Xinu version!

Even better, a few years ago, I had stumbled onto the source code for 2.5 buried on the 4th disc of the CSRG set, and with a LOT of luck and persistence I can confirm that the sources are complete enough to build.

loading vmunix.sys
rearranging symbols
text	data	bss	dec	hex
389088	45564	101364	536016	82dd0
ln vmunix.sys vmunix; ln vmunix vmunix.I386x.STD+WS-afs-nfs

However, as luck always has it, start.s in the i386 code does something weird at the 3GB mark causing a triple fault on any kind of modern emulation/virtualization setup.

    / Fix up the 1st, 3 giga and last entries in the page directory
    mov     $EXT(kpde), %ebx
    and     $MASK, %ebx

    mov     $EXT(kpte), %eax
    and     $0xffff000, %eax
    or      $0x1, %eax

    mov     %eax, (%ebx)
    mov     %eax, 3072(%ebx)        / 3 giga -- C0000000

    mov     $EXT(kpde), %edx
    and     $MASK, %edx

Not all that sure why, but at least on Bochs, I can see the triple fault.

00036527018d[CPU0  ] page walk for address 0x0000000000101122
00036527018d[CPU0  ] page walk for address 0x00000000e0000011
00036527018d[CPU0  ] PDE: entry not present
00036527018d[CPU0  ] page fault for address 00000000e0000011 @ 0000000000101124
00036527018d[CPU0  ] exception(0x0e): error_code=0002
00036527018d[CPU0  ] interrupt(): vector = 0e, TYPE = 3, EXT = 1
00036527018d[CPU0  ] page walk for address 0x00000000c0161370
00036527018d[CPU0  ] PDE: entry not present
00036527018d[CPU0  ] page fault for address 00000000c0161370 @ 0000000000101122
00036527018d[CPU0  ] exception(0x0e): error_code=0000
00036527018d[CPU0  ] exception(0x08): error_code=0000
00036527018d[CPU0  ] interrupt(): vector = 08, TYPE = 3, EXT = 1
00036527018d[CPU0  ] page walk for address 0x00000000c0161340
00036527018d[CPU0  ] PDE: entry not present
00036527018d[CPU0  ] page fault for address 00000000c0161340 @ 0000000000101122
00036527018d[CPU0  ] exception(0x0e): error_code=0000
00036527018i[CPU0  ] CPU is in protected mode (active)
00036527018i[CPU0  ] CS.mode = 32 bit
00036527018i[CPU0  ] SS.mode = 32 bit
00036527018i[CPU0  ] EFER   = 0x00000000
00036527018i[CPU0  ] | EAX=e0000011  EBX=0015f000  ECX=00161dc1  EDX=0015f000
00036527018i[CPU0  ] | ESP=0000efbc  EBP=0000efbc  ESI=00193fb8  EDI=00009d84
00036527018i[CPU0  ] | IOPL=0 id vip vif ac vm RF nt of df if tf SF zf af PF cf
00036527018i[CPU0  ] | SEG sltr(index|ti|rpl)     base    limit G D
00036527018i[CPU0  ] |  CS:0028( 0005| 0|  0) 00000000 ffffffff 1 1
00036527018i[CPU0  ] |  DS:0020( 0004| 0|  0) 00000000 ffffffff 1 1
00036527018i[CPU0  ] |  SS:0010( 0002| 0|  0) 00001000 0000ffff 0 1
00036527018i[CPU0  ] |  ES:0020( 0004| 0|  0) 00000000 ffffffff 1 1
00036527018i[CPU0  ] |  FS:0000( 0005| 0|  0) 00000000 0000ffff 0 0
00036527018i[CPU0  ] |  GS:0000( 0005| 0|  0) 00000000 0000ffff 0 0
00036527018i[CPU0  ] | EIP=00101122 (00101122)
00036527018i[CPU0  ] | CR0=0xe0000011 CR2=0xc0161340
00036527018i[CPU0  ] | CR3=0x00000000 CR4=0x00000000
00036527018i[CPU0  ] 0x0000000000101122>> add byte ptr ds:[eax], al : 0000
00036527018d[SIM   ] searching for component 'cpu' in list 'bochs'
00036527018d[SIM   ] searching for component 'reset_on_triple_fault' in list 'cpu'
00036527018e[CPU0  ] exception(): 3rd (14) exception with no resolution, shutdown status is 00h, resetting

Mach 3.0 doesn’t do this, so I’ll have to dig far deeper into start.s which is kind of really beyond me.

Building a boot disk … is involved. 😐

rm -rf /usr/src/mach25-i386/obj
mkdir /usr/src/mach25-i386/obj
cd /usr/src/mach25-i386/standi386at/boot
make fdboot
/home/user/mkfs /dev/rfloppy 2880 18 2 4096 512 32 1
dd if=/usr/src/mach25-i386/obj/standi386at/boot/boot.fd of=/dev/rfd0d
/home/user/fsck -y /dev/rfloppy
cd /usr/src/mach25-i386/
make
mount /dev/floppy /mnt
cp /usr/src/mach25-i386/obj/STD+WS-afs-nfs/vmunix /mnt
sync
umount /mnt
/home/user/fsck -y /dev/rfloppy

So, I’m not all that dead. For anyone super impatient, you can download my VMDK here, which runs on Qemu & VMware, it includes a serial terminal on COM1 so you can use a real terminal, and if you are like me, uuencode/uudecode files in & out of the system. As always read the 404 page for the current username/password.

Why BSD/OS is the best candidate for being the only tested legally open UNIX.

This is an anonymous guest post. Disclaimer: Nothing in this post constitutes legal advice. The author is not a lawyer. Consult a legal professional for legal advice.

Introduction

The UNIX® system is an old operating system, possibly older than many of the readers of this post. However, despite its age, it still has not been open sourced completely. In this post, I will try to detail which parts of which UNIX systems have not yet been open sourced. I will focus on the legal situation in Germany in particular, taking it representative of European law in general – albeit that is a stretch, knowing the diversity of European jurisdictions. Please note that familiarity with basic terms of copyright law is assumed.

Ancient UNIX

The term “Ancient UNIX” refers to the versions of UNIX up to and including Seventh Edition UNIX (1979) including the 32V port to the VAX. Ancient UNIX was created at Bell Laboratories, a subsidiary of AT&T at the time. It was later transferred of the AT&T UNIX Support Group, then AT&T Information Systems and finally the AT&T subsidiary UNIX System Laboratories, Inc. (USL). The legal situation differs between the United States of America and Germany.

In a ruling as part of the UNIX System Laboratories, Inc. v. Berkeley Software Design, Inc. (USL v. BSDi) case, a U.S. court found that USL had no copyright to the Seventh Edition UNIX system and 32V – arguably, by extension, all earlier versions of Ancient UNIX as well – because USL/AT&T had failed to affix copyright notices and could not demonstrate a trade secret. Due to the obsessive tendency of U.S. courts to consider themselves bound to precedents (cf. the infamous Pierson v. Post case), it can be reasonably expected that this ruling would be honored and applied in subsequent cases. Thus under U.S. law, Ancient UNIX can be safely assumed to belong in the public domain.

The situation differs in Germany. Unlike the U.S., copyright never needed registration in order to exist. Computer programs are works in the sense of the German 1965 Act on Copyright and Related Rights (Copyright Act, henceforth CopyA) as per CopyA § 2(1) no. 1. Even prior to the amendment of CopyA § 2(1) to include computer programs, computer programs have been recognized as copyrightable works by the German Supreme Court (BGHZ 112, 264 Betriebssystem, no. 19); CopyA § 137d(1) rightly clarifies that. The copyright holder at 1979 would still have been USL via Bell Labs and AT&T. Copyright of computer programs is transferred to the employer upon creation under CopyA § 69(1).

Note that this does not affect expiry (Daniel Kaboth/Benjamin Spies, commentary on CopyA §§ 69a‒69g, in: Hartwig Ahlberg/Horst-Peter Götting (eds.), Urheberrecht: UrhG, KUG, VerlG, VGG, Kommentar, 4th ed., C. H. Beck, 2018, no. 16 ad CopyA § 69b; cf. Bundestag-Drucksache [BT-Drs.] 12/4022, p. 10). Expiry occurs 70 years after the death of the (co-)author that died most recently as per CopyA § 65(1) and 64; this has been the case since at least the 1960s, meaning there is no way for copyright to have expired already (old version, as per Bundesgesetzblatt Part I No. 51 of September 16, 1965, pp. 1273‒1294).

In Germany, private international law applies the so-called “Territorialitätsprinzip” for intellectual property rights. This means that the effect of an intellectual property right is limited to the territory of a state (Anne Lauber-Rönsberg, KollisionsR, in: Hartwig Ahlberg/Horst-Peter Götting (eds.), ibid., pp. 2241 et seqq., no. 4). Additionally, the “Schutzlandprinzip” applies; this means that protection of intellectual property follows the lex loci protectionis, i.e. the law of the country for which protection is sought (BGH GRUR 2015, 264 HiHotel II, no. 25; BGH GRUR 2003, 328 Sender Felsberg, no. 24), albeit this is criticized in parts of doctrine (Lauber-Rönsberg, ibid., no. 10). The “Schutzlandprinzip” requires that the existence of an intellectual property right be verified as well (BGH ZUM 2016, 522 Wagenfeld-Leuchte II, no. 19).

Thus, in Germany, copyright on Ancient UNIX is still alive and well. Who has it, though? A ruling by the U.S. Court of Appeals, Tenth Circuit, in the case of The SCO Group, Inc. v. Novell, Inc. (SCO v. Novell) in the U.S. made clear that Novell owns the rights to System V – thus presumably UNIX System III as well – and Ancient UNIX, though SCO acquired enough rights to develop UnixWare/OpenServer (Ruling 10-4122 [D.C. No. 2:04-CV-00139-TS], pp. 19 et seq.). Novell itself was purchased by the Attachmate Group, which was in turn acquired by the COBOL vendor Micro Focus. Therefore, the rights to SVRX and – outside the U.S. – are with Micro Focus right now. If all you care about is the U.S., you can stop reading about Ancient UNIX here.

So how does the Caldera license factor into all of this? For some context, the license was issued January 23, 2002 and covers Ancient UNIX (V1 through V7 including 32V), specifically excluding System III and System V. Caldera, Inc. was founded in 1994. The Santa Cruz Operation, Inc. sold its rights to UNIX to Caldera in 2001, renamed itself to Tarantella Inc. and Caldera renamed itself The SCO Group. Nemo plus iuris ad alium transferre potest quam ipse habet; no one can transfer more rights than he has. The question now becomes whether Caldera had the rights to issue the Caldera license.

I’ve noted it above but it needs restating: Foreign decisions are not necessarily accepted in Germany due to the “Territorialitätsprinzip” and “Schutzlandprinzip” – however, I will be citing a U.S. ruling for its assessment of the facts for the sake of simplicity. As per ruling 10-4122, “The district court found the parties intended for SCO to serve as Novell’s agent with respect to the old SVRX licenses and the only portion of the UNIX business transferred outright under the APA [asset purchase agreement] was the ability to exploit and further develop the newer UnixWare system. SCO was able to protect that business because it was able to copyright its own improvements to the system. The only reason to protect the earlier UNIX code would be to protect the existing SVRX licenses, and the court concluded Novell retained ultimate control over that portion of the business under the APA.” The relevant agreements consist of multiple pieces:

The APA dates September 19, 1995, from before the Caldera license. Caldera cannot possibly have acquired rights that The Santa Cruz Operation, Inc. itself never had. Furthermore, I’ve failed to find any mention of Ancient UNIX; all that is transferred is rights to SVRX. Overall, I believe that the U.S. courts’ assesment of the facts represents the situation accurately. Thus for all intents and purposes, UNIX up to and including System V remained with Novell/Attachmate/Micro Focus. Caldera therefore never had any rights to Ancient UNIX, which means it never had the rights to issue the Caldera license. The Caldera license is null and void – in the U.S. because the copyright has been lost due to formalities, everywhere else because Caldera never had the rights to issue it.

The first step to truly freeing UNIX would this be to get Micro Focus to re-issue the Caldera license for Ancient UNIX, ideally it would now also include System III and System V.

BSD/OS

Another operating system near UNIX is of interest. The USL v. BSDi lawsuit includes two parties: USL, which we have seen above, and Berkeley Software Design, Inc. BSDi sold BSD/386 (later BSD/OS), which was a derivative of 4.4BSD. The software parts of the BSDi company were acquired by Wind River Systems, whereas the hardware parts went to iXsystems. Copyright is not disputed there, though Wind River Systems ceased selling BSD/OS products 15 years ago, in 2003. In addition, Wind River System let their trademark on BSD expire, though this is without consequence for copyright.

BSD/OS is notable in the sense that it powered much of early internet infrastructure. Traces of its legacy can still be found on Richard Stevens’ FAQ.

To truly make UNIX history free, BSD/OS would arguably also need to see a source code release. BSD/OS at least in its earliest releases under BSDi would ship with source code, though under a non-free license, far from BSD or even GPL licensing.

System V

The fate of System V as a whole is difficult to determine. Various licenses have been granted to a number of vendors (Dell UNIX comes to mind; HP for HP-UX, IBM for AIX, SGI UNIX, etc.). Sun released OpenSolaris – notoriously, Oracle closed the source to Solaris again after its release –, which is a System V Release 4 descendant. However, this means nothing for the copyright or licensing status of System V itself. Presumably, the rights with System V still remain with Novell (now Micro Focus): SCO managed to sublicense rights to develop and sell UnixWare/OpenServer, themselves System V/III descendants, to unXis, Inc. (now known as Xinuos, Inc.), which implies that Xinuos is not the copyright holder of System V.

Obviously, to free UNIX, System V and its entire family of descendants would also need to be open sourced. However, I expect tremendous resistance on part of all the companies mentioned. As noted in the “Ancient UNIX” section, Micro Focus alone would probably be sufficient to release System V, though this would mean nothing for the other commercial System V derivatives.

Newer Research UNIX

The fate of Bell Labs would be a different one; it would go on to be purchased by Lucent, now part of Nokia. After commercial UNIX got separated out to USL, Research UNIX would continue to exist inside of Bell Labs. Research UNIX V8, V9 and V10 were not quite released by Alcatel-Lucent USA Inc. and Nokia in 2017.

However, this is merely a notice that the companies involved will not assert their copyrights only with respect to any non-commercial usage of the code. It is still not possible, over 30 years later, to freely use the V8 code.

Conclusion

In the U.S., Ancient UNIX is freely available. People located everywhere else, however, are unable to legally obtain UNIX code for any of the systems mentioned above. The exception being BSD/OS, assuming a purchase of a legitimate copy of the source code CD. This is deeply unsatisfying and I implore all involved companies to consider open sourcing (preferably under a BSD-style license) their code older than a decade, if nothing else, then at least for the sake of historical purposes. I would like to encourage everybody reading this to consider reaching out to Micro Focus and Wind River Systems about System V and BSD/OS, respectively. Perhaps the masses can change their minds.

A small note about patents: Some technologies used in newer iterations of the UNIX system (in particular the System V derivatives) may be encumbered with software patents. An open source license will not help against patent infringement claims. However, the patents on anything used in the historical operating systems will certainly have expired by now. In addition, European readers can ignore this entirely – software patents just aren’t a thing.


UNIX® is a registered trademark of The Open Group.

ttyplot – a real time plotting utility for the terminal

(This is a guest post from Antoni Sawicki aka Tenox)

I spend most of time in a day staring at a terminal window often running various performance monitoring tools and reading metrics.

Inspired by tools like gtop, vtop and gotop I wished for a more generic terminal based tool that would visualize data coming from unix pipeline directly on the terminal. For example graph some column or field from sar, iostat, vmstat, snmpget, etc. continuously in real time.

Yes gnuplot and several other utilities can plot on terminal already but none of them easily read data from stdin and plot continuously in real time.

In just couple of evenings ttyplot was born. The utility reads data from stdin and plots it on a terminal with curses. Simple as that. Here is a most trivial example:

To make it happen you take ping command and pipe the output via sed to extract the right column and remove unwanted characters:

ping 8.8.8.8 | sed -u 's/^.*time=//g; s/ ms//g' | ttyplot 

Ttyplot can also read two inputs and plot with two lines, the second being in reverse-video. This is useful when you want to plot in/out or read/write at the same time.

A lot of performance metrics are presented in as a “counter” type which needs to be converted in to a “rate”. Prometheus and Graphana have rate() or irate() function for that. I have added a simple -r option. The time difference is calculated automatically. This is an example using snmpget which is show in screenshot above:

{ while true; do snmpget -v 2c -c public 10.23.73.254 1.3.6.1.2.1.2.2.1.{10,16}.9 | gawk '{ print $NF/1000/1000 }'; sleep 10; done } | ttyplot -2 -r -u "MB/s"

I now find myself plotting all sorts of useful stuff which otherwise would be cumbersome. This includes a lot of metrics from Prometheus for which you normally need a web browser. And how do you plot metrics from Prometheus? With curl:

{ while true; do curl -s http://10.4.7.180:9100/metrics | grep "^node_load1 " | cut -d" " -f2; sleep 1; done } | ttyplot

If you need to plot a lot of different metrics ttyplot fits nicely in to panels in tmux, which also allows the graphs to run for longer time periods.

Of course in text mode the graphs are not very precise, but this is not the intent. I just want to be able to easily spot spikes here and there plus see some trends like up/down – which works exactly as intended.I do dig fancy braille line graphs and colors but this is not my priority at the moment. They may get added later, but most importantly I want the utility to work reliably on most operating systems and terminals. 

You can find compiled binaries here and source code and examples to get you started – here.

If you get to plot something cool that deserves to be listed as an example please send it on!

IOCCC 2018 Best of show, aka PDP-7 emulation in under 4kb!

The most incredible thing about this PDP-7 emulator is that it not only can run Unix v0 as recovered by TUHS, but for kicks there is also a PDP-11 emulator coded in PDP-7 assembly that is capable of running both Unix v6 & 2.9BSD!

So if I have a PDP-7 emulator, how do I run operating systems that expect a PDP-11? Simple… I emulate a PDP-11/40 on the PDP-7. I have written PDP-7 assembler code to emulate a PDP-11/40 with the following equipment:

PDP-11/40 (KD11-A)
EIS instruction set (KE11-E)
Memory management unit (KJ11-A)
Line time clock (KT11-D)
124 Kwords of memory (244 Kbytes)
RK05 fixed disk drive (RK11)
Console TTY (DL11)

And as you can see, here is the PDP-7 running 2.9BSD via the emulated PDP-11!

You read that right.  And yes, here it is running 2.9BSD.  This is nothing short of amazing!

Read about Christopher Mills’s entry here:

http://ioccc.org/2018/mills/hint.html

And yes, the source has been stylized like a paper tape…

It really works best on a ‘real’ Unix like system.  I’ve run it on Debian 9 & OS X 10.13.14 … It does consume 100% of a CPU core.  But it’ll run a PDP-7 with v0 Unix, PDP-11 with Unix v6, and the aforementioned 2.9BSD. The Linux subsystem for Windows 10 didn’t handle the terminal control so well.

So, yeah kill your CPU and bask in the 4k glory!

fread and fwrite demystified: stdio on UNIX V7

(This is a guest post by xorhash.)

1. Introduction

Did I say I’m done with UNIX Seventh Edition (V7)? How silly of me; of course I’m not. V7 is easy to study, after all.

Something that’s always bothered me about the stdio.h primitives fread() and fwrite() are their weak guarantees about what they actually do. Is a short read or write “normal” in the sense that I should normally expect it? While this makes no answer about modern-day operating systems, a look at V7 may enlighten me about what the historical precedent is.

As an aside: It’s worth noting that the stdio.h functions are some of the few that require a header. It was common historical practice not to declare functions in headers, just see crypt(3) as an example.

I will first display the man page, then ask the questions I want to answer, then look at the implementation and finally use that gained knowledge to answer the questions.

2. Into the Man Page

The man page for fread() and fwrite() is rather terse. Modern-day man pages for those functions are equally terse, though, so this is not exactly a novelty of age. Here’s what it reads:

NAME

fread, fwrite – buffered binary input/output

SYNOPSIS

#include <stdio.h>

fread(ptr, sizeof(*ptr), nitems, stream)
FILE
*stream;

fwrite(ptr, sizeof(*ptr), nitems, stream)
FILE
*stream;

DESCRIPTION

Fread reads, into a block beginning at ptr, nitems of data of the type of *ptr from the named input stream. It returns the number of items actually read.

Fwrite appends at most nitems of data of the type of *ptr beginning at ptr to the named output stream. It returns the number of items actually written.

SEE ALSO

read(2), write(2), fopen(3), getc(3), putc(3), gets(3), puts(3), printf(3), scanf(3)

DIAGNOSTICS

Fread and fwrite return 0 upon end of file or error.

So there are the following edge cases that are interesting:

  • In fread(): If sizeof(*ptr) is greater than the entire file, what happens?
  • If sizeof(*ptr) * nitems overflows, what happens?
  • Is the “number of items actually read/written” guaranteed to be the number of items that can be read/written (until either EOF or I/O error)?
  • Is the “number of items actually written” guaranteed to have written every item in its entirety?
  • What qualifies as error?

3. A Look at fread()

Note: All file paths for source code are relative to /usr/src/libc/stdio/ unless noted otherwise. You can read along at the TUHS website.

rdwr.c implements fread(). fread() is simple enough; it’s just a nested loop. The outer loop runs nitems times. The outer loop sets the number of bytes to read (sizeof(*ptr)) and runs the inner loop. The inner loop calls getc() on the input FILE *stream and writes each byte to *ptr until either getc() returns a value less < 0 or all bytes have been read.

/usr/include/stdio.h implements getc(FILE *p) as a C preprocessor macro. If there is still data in the buffer, it returns the next character and advances the buffer by one. Interestingly, *(p)->_ptr++&0377 is used to return the character, despite _ptr being a char *. I’m not sure why that &0377 (&0xFF is there. If there is no data in the buffer, it instead returns _filbuf(p).

filbuf.c implements _filbuf(). This function is a lot more complex than the other ones until now. It begins with a check for the _IORW flag and, if set, sets the _IOREAD flag as well. It then checks if _IOREAD is not set or if _IOSTRG is set and returns EOF (defined as -1 in stdio.h) if so. These all seem rather inconsequential to me. I can’t make heads or tails of _IOSTRG, however, but it seems irrelevant; _IOSTRG is only ever set internally in sprintf and sscanf for temporary internal FILE objects. After those two flag checks, _filbuf() allocates a buffer into iop-<_base, which seems to be the base pointer of the buffer. If flag _IONBF is set, which happens when setbuf() is used to switch to unbuffered I/O, a temporary, static buffer is used instead. Then read() is called, requesting either 1 bytes if unbuffered I/O is requested or BUFSIZ bytes. If read() returned 0, the FILE is flagged as end-of-file and EOF is returned by _filbuf(). If read() returned <0, the FILE is flagged as error and EOF is returned by _filbuf(). Otherwise, the first character that has been read is returned by _filbuf() and the buffer pointer incremented by one.

According to its man page, read() only returns 0 on end-of-file. It can also return -1 on “many conditions”, namely “physical I/O errors, bad buffer address, preposterous nbytes, file descriptor not that of an input file”

As an aside, BUFSIZ still exists today. ISO C11 § 7.21.2 no. 9 dictates that BUFSIZ must be at least 256. V7 defines it as 512 in stdio.h. One is inclined to note that on V7, a filesystem block was understood 512 bytes length, so this was presumably chosen for efficient I/O buffering.

4. A Look at fwrite()

rdwr.c also implements fwrite(). fwrite() is effectively the same as fread(), except the inner loop uses putc(). After every inner loop, a call to ferror() is made. If there was indeed an error, the outer loop is stopped.

/usr/include/stdio.h implements putc(int x, FILE *p) as a C preprocessor macro. If there is still room in the buffer, the write happens into the buffer. Otherwise, _flsbuf() is called.

flsbuf.c implements _flsbuf(int c, FILE *iop). This function, too, is more complex than the ones until now, but becomes more obvious after reading _filbuf(). It starts with a check if _IORW is set and if so, it’ll set _IOWRT and clear the EOF flag. Then it branches into two major branches: the _IONBF branch without buffering, which is a straight call to write(), and the other branch, which allocates a buffer if none exists already or otherwise calls write() if the buffer is full. If write() returned less than expected, the error flag is set and EOF returned. Otherwise, it returns the character that was written.

According to its man page, write() returns the number of characters (bytes) actually written; a non-zero value “should be regarded as an error”. With only a cursory glance over the code, this appears to happen for similar reasons as read(), which is either physical I/O error or bad parameters.

5. Conclusions

In fread(): If sizeof(*ptr) is greater than the entire file, what happens?
On this under-read, fread() will end up reading the entire file into the memory at ptr and still return 0. The I/O happens byte-wise via getc(), filling up the buffer until getc() returns EOF. However, it will not return EOF until a read() returns 0 on EOF or -1 on error. This result may be meaningful to the caller.

If sizeof(*ptr) * nitems overflows, what happens?
No overflow can happen because there is no multiplication. Instead, two loops are used, which avoids the overflow issue entirely. (If there are strict filesystem constraints, however, it may be de-facto impossible to read enough bytes that sizeof(*ptr) * nitems overflows. And of course, there’s no way you could have enough RAM on a PDP-11 for the result to actually fit into memory.)

Is the “number of items actually read/written” guaranteed to be the number of items that can be read/written (until either EOF or I/O error)?
Partially: Both fread() and fwrite() short-circuit on error. This causes the number of items that have actually been read or written successfully to be returned. The only relevant error condition is filesystem I/O error. Due to the byte-wise I/O, it’s possible that there was a partial read or write for the last element, however. Therefore, it would be more accurate to say that the “number of items actually read/written” is guaranteed to be the number of non-partial items that can be read/written. A short read or short write is an abnormal condition.

Is the “number of items actually written” guaranteed to have written every item in its entirety?
No, it isn’t. A partial write is possible. If a series of structs is written and then to be read out again, however, this is not a problem: fread() and fwrite() only return the count of full items read or written. Therefore, the partial write will not cause a partial read issue. If a set of bytes is written, this is an issue: There will be incomplete data – possibly to be parsed by the program. It is therefore to preferable to write (and especially read) arrays of structs than to write and read arrays of bytes. (From a modern-day perspective, this is horrendous design because this means data files are not portable across platforms.)

What qualifies as error?
Effectively, only a physical I/O error or a kernel bug. Short fread() or fwrite() return values are abnormal conditions. I’m not sure if there is the possibility that the process got a signal and the current read() or write() ends up writing nothing before the EINTR; this seems to be more of a modern-day problem than something V7 concerned itself.