Web Rendering Proxy v4.6 released

(this is a guest post by Antoni Sawicki aka Tenox)

Pleased to announce WRP version 4.6! After almost two years of no updates due to dependency issues I finally resolved everything. More frequent work should resume now.

The main improvement visible to users is the new GIF encoding. I have been struggling with poor GIF performance for quite a while. This was mostly manifested on lower end machines running WRP such as Raspberry PI or these micro instances in the cloud.

Thanks to invaluable work of Hill Ma we now have blazing fast GIFs. Probably order of 100x improvements! This comes at a cost of quality, especially of color palette and dithering. However worse quality of imagery has a surprise improvement of font/text quality which is what a lot of people wanted.

WRP 4.6 with default 216 color GIF

Note that by default WRP uses GIF with 216 “web safe” colors. We choose this not so much for number of colors but rather activation of the super fast GIF encoding.

WRP 4.6 in 256 color mode GIF

When switching to 256 color mode the image look much better, however it takes around 25x longer to encode (7ms vs 170ms).

When using PNG this is of course not a problem.

0 height mode, which renders tall images of full page length has also been improved and is now more stable. Be careful when using very old machines with little memory as the images can be pretty big.

I hope that WRP will help you use your vintage computers more 🙂

Please report bugs and issues on github!

Downloads here.

IBM AIX for IA64 (Itanium) aka Project Monterey runs again!

(This is a guest post by Antoni Sawicki aka Tenox)

Project Monterey was an attempt to unify the fragmented Unix market of the 90s in to a single, cross vendor Unix OS that would run on the upcoming Intel Itanium (and others) CPU. The main collaborators were: IBM, who brought its AIX, SCO brought UnixWare, HP was supposed to bring parts of HP-UX and Sequent DYNIX/ptx. Ironically the project shared fate of the Itanium CPU—it totally failed. In the end Linux took spot of the “single Unix OS”. IBM donated AIX pieces to Linux instead and the main legacy of Project Monterey was the famous SCO vs IBM lawsuit.

IBM did however produce AIX version for the Itanium architecture! According to Wikipedia, some 30+ licenses were sold in 2001-2002. For years a dedicated group of individuals was trying to locate a copy of the legendary OS. It seemed that the OS was lost forever…

…until some 21 years later friends of NCommander checked in with a set of AIX5L IA64 CDROMS! The CDs have now been dumped and you can download them here. Unfortunately downloading will not get you much closer to actually running this. As of today no publicly available virtualization or emulation platform can boot this. Yes we tried Simics, looked at QEMU IA64 and XEN/KVM for IA64, etc. The OS will not boot on modern Itanium 2 (McKinley) CPUs, only the early “pre-release” Itanium 1 aka Merced. The only emulator allegedly capable of doing so was the super elusive unobtanium called Intel SoftSDV.

It’s currently speculated that AIX5L IA64 will work on and only on so called “Intel Software Development Vehicle (SDV)” sometimes referred to as “Intel Engineering Sample”. It was an Intel made machine, later sold in several OEM branded version: IBM IntelliStation Z Pro 6894, HP i2000 Workstation, SGI 750, Dell Precision Workstation 730 and Fujitsu-Siemens Celsius 880.

Intel Itanium Software Development Vehicle Lineup

…yes, they all look alike because all of them were in fact produced by Intel with custom case badges and paints.

Luckily I was able to score a working HP i2000. AIX booted up and installed on a first try:

AIX 5L IA64 on HP i2000 Workstation – boot loader
AIX 5L IA64 on HP i2000 Workstation – logged in

Initially I was not able to get the onboard NIC working. Upon short investigation AIX5L IA64 supports only two network cards:

adapter 23100020 IBM 10/100 Mbps Ethernet PCI Adapter (23100020)
adapter ae120200 10/100/1000 Base-T Ethernet PCI Adapter (ae120200)

The AIX Itanium Early Adopters Release Notes mentions a few other cards but I do not see drivers for these in the OS. The doc mentions “Extended Hardware Drivers CD” which we don’t have.

Luckily again I was able to find a working NIC on eBay!

The system comes with X11 and CDE but so far I was not able to get any GPU working beyond basic text mode. I tried many different video cards from that era but there simply doesn’t appear to be any driver in the OS except for basic VGA / LFT. I think the key to getting video working is the previously mentioned extended hardware drivers cd.

Finally, if you want to read more I have found some interesting pieces on ibmfiles and various mirrors here and here.

Update: Thanks to efforts of TRN we now have a working GCC and ports of lots of apps!

Update 2: After going through a pile of video cards I now have local X11 and CDE!

AIX IA64 local X11 with CDE

This was the lucky winner:

How to fix rsync slowing down over time (SOLVED)

(This is a guest post by Antoni Sawicki aka Tenox)

I often make copies of large data archives, typically many TB in size. I found that rsync transfer speed slows down over time, typically after a few GB, especially when copying large files. Eventually reaching crawl speeds of just few KB/s. The internet is littered with people asking the same question or why rsync is slow in general. There really isn’t a good answer out there, so I hope this may help.

After doing some quick profiling I found out that the main culprit was rsync's advanced delta transfer algorithm. The algorithm is super awesome for incremental updates as it will only transfer changed parts of a file instead of the whole thing. However when performing initial copy it’s not only unnecessary but gets in the way and the CPU is spinning calculating CRC on chunks that never could have changed. As such…

Initial rsync copies should be performed with -W option, for example:

$ rsync -avPW <src> <dst>

The -W or --whole-file option instructs rsync to perform full file copies and do not use delta transfer algorithm. In result there is no CRC calculation involved and maximum transfer speeds can be easily achieved.

Long term, rsync could be patched to do a full file transfer if the file doesn’t exist in destination.

Also while copying jumbo archives of many TB I don’t want to see every individual file being copied. Instead I want a percentage of the total archive size and current transfer speed in MB/s. After some experiments I arrived at this weird combo:

$ rsync -aW --no-i-r --info=progress2 --info=name0 <src> <dst>

Ready to run OpenVMS VM – Student Kit from VSI

(This is a guest post by Antoni Sawicki aka Tenox)

I was recently registering a new OpenVMS Community License. In the process I learned that there is a ready to run, pre-installed and pre-configured VM with OpenVMS 8.4. Completely free for non-commercial purposes. You don’t even need to register or leave your details (WOW). Just download and run! Thank you VSI!

https://training.vmssoftware.com/student-license/

The student kit runs only on Windows as contains FreeAXP emulator. However it’s super easy to download, install and run.

VSI OpenVMS Student Kit

I’m hoping that in near future once x86 OpenVMS port is ready there will be images for x64 hypervisors like VMware, VirtualBox, Hyper-v and QEMU/KVM hopefully.

Cloud Functions Bucket File Editor

(This is a guest post by Antoni Sawicki aka Tenox)

The title is little complicated so let me explain! Ever since dawn of the web I been running some web server to host various websites. In the beginning it was a physical machine under my desk, then a colocated server, then a VPS Virtual Machine, then a container. Eventually all the websites that I look after or host for friends/family ended up in “the cloud”. Storage buckets to be precise, as they offer a super cheap and simple way of hosting static websites. I no longer have to maintain machines, look after updates, security or configuration. Just upload HTML and done.

However here lies a problem. You can’t just “edit” a file in a bucket, you can only download it, edit locally and re-upload. This typically is done via Storage Browser UI or gsutil cli utility (in case of GCP). Or some sophisticated IDE with storage API support or sometimes a FUSE client. In any case it’s hard and cumbersome. Why can’t there be a simple Edit function in the UI?

Why isn’t there a File Edit Option????

I have recently discovered that there even is a brand new Cloud Shell Editor which is a VScode instance bound to your “cloud shell instance”. It allows you to virtually edit files “in the cloud” but of course not in storage buckets! WHY NOT?

While this been bothering me for a while I also got interested in so called “cloud functions” which are small pieces of code that you can run “in the cloud” without need for a VM or even a container (although surely they run in one behind the scenes). Just paste the code and run… somewhere. Cloud Functions or AWS Lambdas have been notoriously abused by crypto miners. I wanted to play with these for legitimate reasons and finally decided to create a simple web based text editor that would allow to edit text files in storage buckets. So CFEdit was born.

The editor is super simple, no frills, just a file selector and textarea for editing files. You deploy it by creating a new function, either via the UI or CLI. You can restrict it just to a single bucket or allow editing in any bucket within a project. It doesn’t use IAM and thus doesn’t require Google accounts. I specifically wanted to avoid complexity of that and just went with own user database and HTTP Basic Auth.

Now I can create account for family members or friends and allow them to edit their web pages via simple web based editor rather than asking for uploading files or resorting to ftp/sftp GCS gateways or stuff like that.

In future I’m planning to add other function like create blank file, directory, upload/download etc. Let me know what’s needed.

CFEdit can be downloaded from github.com/tenox/cfedit, the readme describes in steps on how to deploy it.

Revisiting Windows NT 4.0 MIPS on QEMU

(This is a guest post by Antoni Sawicki aka Tenox)

This was previously well covered by Gunkies and Neozeed, however as almost a decade passed, some improvements could be made and annoyances fixed.

Firstly NT MIPS now works in 1280×1024 resolution under QEMU. It previously had issues with mouse tracking, but this is now fixed. So the new image has a higher resolution.

Secondly the old images were made with FAT filesystem which I didn’t like too much. The reason for that is the infamous RISC NT osloader needs to be placed on a FAT partition. Then, if NT is installed on a second NTFS partition the default drive will be D:\, C:\ being the just the osloader drive. This was super annoying in practice. So a common procedure was to just have one FAT partition for both osloader and winnt. I have fixed it by supplying a pre-partitioned disk and specified the second partition for osloader and the first for NT.

Also I only had just a bare/vanilla image with no additional software installed. The new image includes most of the available apps, including IE3, some editors, Reskit and Visual Studio.

Lastly I wanted to figure out all the right settings and flags for qemu as they were discrepancies between different sources and nothing seem to work smoothly. The correct flags seem to be:

qemu-system-mips64el -hda nt4.qcow2 -M magnum -global ds1225y.filename=nvram -L . -rtc "base=1995-07-08T11:12:13,clock=vm" -nic user,model=dp83932

The -rtc flag is not really needed if you are ok with having the current date in the guest.

Thanks to Neozeed for figuring out the network settings! Unfortunately the old/legacy -net nic -net user is no longer working while the new -device doesn’t like dp83932. The documentation was quite helpful.

Thanks to reader Mark for pointing out the correct NVRAM settings! See comments below.

The new image with all the apps preinstalled is here and a plain “vanilla” here.

Curiously this now works right out of the box on QEMU 6.1 and is pretty smooth and stable compared to what it was before. Good job QEMU team and thank you! Just in case I still keep the old binaries for Windows made by Neozeed here.

Update: I built Yori for NT MIPS! You can download here!

Fun with Nano Server

(This is a guest post by Antoni Sawicki aka Tenox)

While everybody is busy buzzing about Windows 11, I wanted to commemorate the finest operating system ever made by Microsoft – Nano Server.

For most of people Nano Server was esoteric, distant and unapproachable. It had a rather high entry barrier, requiring you to build it on a Windows Server 2016 host using PowerShell magic spells. You couldn’t just simply download and run it. Even if you managed to get it running, there wasn’t anything you could actually do with it for fun. People didn’t bother to even check it out. My goal is to demystify this a bit, lower the entry bar and made it easy for people to hack it.

Background info (you can skip it)

Nano Server was an interesting attempt at creating a datacenter grade OS that’s not managed via local GUI, keyboard and mouse, but rather full automation, remote tooling and code. It went one step further than Server Core or Windows PE by completely removing GUI components and local shell. Hence it’s not actually called “Windows” or “Windows Nano” but rather simply “Nano Server”. Rumor has it, it started as MinWin. The OS has a rudimentary text mode console with functionality similar of VMware ESXi console. However Nano was much more than a bare metal hypervisor. It was a fully fledged operating system. Unlike ESXi you can develop and install services/apps for it and hypervisor wasn’t even it’s default role.

Ever since I first saw a demo on Microsoft Ignite (previously known as TechEd) I wanted to run aclock on the text console. Much like the WinNT BSOD edition. This article started around my efforts to run (or port if needed) aclock to this platform. At the time of writing, the technology has been dead for several years now. However all the artifacts and documentation are still available on Microsoft’s website. Probably not for long, so a good moment to do it now, before everything gets deleted in to oblivion.

How to quickly deploy Nano Server and run command line apps on the console

The hard way: you need to download Windows 2016 Server (eval) and run a PowerShell command to produce a bootable VHD file.

Microsoft provides (soon to be deleted) Nano Server Quick Start. However the steps are trivial so you can totally skip that and just do this:

  • Launch PowerShell terminal window on the WS2016 host.
  • Run: Import-Module D:\NanoServer\NanoServerImageGenerator -Verbose
    (D:\ drive being where Windows Server CDROM is mounted)
  • Run: New-NanoServerImage -DeploymentType Guest -Edition Standard -MediaPath d:\ -BasePath c:\nano -TargetPath c:\nano.vhdx -ComputerName nano -Development
    (c:\nano folder and c:\nano.vhdx image will be created for you)

Done! This will build a .vhdx image that can be run under Hyper-V as “Gen-2” VM. For Gen-1 or to run it on any other hypervisor change .vhdx to .vhd in -TargetPath while running the PowerShell command.

The easy way: you can just download a pre-built VM image from here. There are VHD for Hyper-v Gen-1 and VHDX for Hyper-v Gen-2 and OVA for everything else.

First Boot

Once you boot it up you will be greeted with a PowerShell prompt. Just like that! You can type cmd to launch the good old cmd.exe shell. MS-DOS 2016?

Keep in mind, this is a developer mode (see -Development flag). Normally you would be greeted with a login prompt and a boring menu that allows to change some networking settings and not much beyond that. In production mode you need to resort to hacks (or this) to get stuff running, fortunately nothing like that needed here.

So what can you run on it?

Firstly in order to get some external utilities going, you can mount a SMB share using net use in cmd or New-SMBMapping in PS world. Nano being a server and all, you can also share out a folder via net share or use C$ (you may need to create a user by using either net user /add in cmd or New-LocalUser in PS). Alternatively you can install Posh-SSH and use SCP to transfer files. If you don’t have working network you can just shut it down, mount the vhd image on the host and copy stuff in to the image then detach the VHD.

Aclock worked on the first run, no issues, using standard win64 exe:

aclock running on Nano Server Console

Wow! So looks like Nano console does have basic terminal controls. That opens quite a lot of possibilities. But can you run more complex apps? Text editors? Web browsers? GAMES?

Well, yes…, but likely not, but it really depends – on dependencies (read: DLLs).

From all the editors I tried XVI is probably the best:

XVI Editor Running on Nano Server Console

Everything else has a variety of issues:

  • The font is lacking line drawing characters. Some editors like YEdit allow to use ASCII drawing characters fortunately.
  • There is no reverse video. This manifests mostly in menus, etc. however it also applies to the cursor.
  • There is no cursor, or rather the cursor is an underscore and not transparent cell. Moving arrow left in the CLI doesn’t actually move the cursor it erases characters. There is no line editing.
  • Also related to reverse video, it appears Nano console has some weird issues with colors.
  • Missing DLLs. Nano Server not being a “Windows” OS is missing a lot of Windows DLLs and it has its own nano DLL hell. This has actually been acknowledged in MinWin. As such a lot of apps will not launch due to dependencies.

For example YEdit works remarkably well except for the menus, which use reverse video:

YEdit running on Nano Server Console

Update: Malcolm has fixed it in latest version of YEdit! Thank you!

Update: thanks to Ron Yorston you can also run BusyBox on Nano! All you need to do is get the 64bit version and before you run it set an environmental variable to disable ANSI emulation. In CMD set BB_SKIP_ANSI_EMULATION=0 in PS $env:BB_SKIP_ANSI_EMULATION=0. Done!

BusyBox on Nano Server

You even get ls colors and vi editor works flawlessly! Unix shell on Nano, thats awesome!

So what about games?

Initially nothing worked as expected. Either due to line drawing, colors or previously mentioned DLL hell. There was one game that actually worked – PowerShell adaptation of snake:

PowerShell Snake running on Nano Server Console

But I wanted something better. I had high hopes for ascii-patrol, which is pure text mode and they build it for win64. Unfortunately the game requires a bunch of multimedia / sound DLLs from Windows which are not present in Nano.

Thankfully Neozeed has stepped in, took the source code, amputated all the multimedia stuff, borrowed the Unix clock code and gettimeofday, and used an older Visual Studio to build it. But he managed to produce a fully working and playable version!!!! Truly amazing stuff!

ASCII Patrol Running on Nano Server Console

The binary is available here. To play the game scroll down one screen to start a mission. If you enter profile customization simply press ESC to get out. Thanks again Neozeed!

I’m hoping readers can find more text mode/ascii apps and games that will work on the console. Please comment and send links!

In another dimension, having a working text editor, Yori shell, smb/scp, maybe with help of mingw64, sdk tools or borrowed compilers from Visual Studio, one could have a self hosted developer workstation with this.

For now please just download the pre-build image, or make one yourself and run it in your favorite hypervisor and have some fun with it!

With this, goodbye Nano Server! You will be always remembered. I know folks at Redmond tried really hard to make it such beautiful gem.

lsblk for Windows released

(This is a guest post by Antoni Sawicki aka Tenox)

Pleased to announce that lsblk utility for Windows is finally released. This is not entirely new, the original code was on github for a few years now, but it was lacking major features of printing drive letters, mount points and filesystem types.

Why would anyone even want lsblk for Windows? There are many other “native” ways of displaying disks and volumes. For example in PowerShell: Get-Disk or Get-PhysicalDisk or in cmd: wmic diskdrive list brief. Not to mention diskpart or disk management UI. There are a few answers to this.

Firstly the output format of lsblk on Linux is rather intuitive and useful, which can’t be said about previously mentioned utilities. People with Linux background find it more like at home.

PowerShell and wmic lack ability to combine disk and volume information, unless you want to write a larger script. This is now part of lsblk.

Lastly lsblk uses low level function to list objects directly from the kernel (think WinObj), rather than going through various high level services, management interfaces and relying on VDS (Virtual Disk Service). As such it’s super fast and you can use it even with VDS stopped or inoperable. Finally it’s yet another of these native Linux tools now also available on Windows.

Finally, some of the column names may sound cryptic, so here is an explanation:

ST – Status 1=healthy 0=unhealthy
TR – Trim / Unmap / Discard Capability
RM – Removable Media
MD – Media changed (for removable media)
RO – Read only

Running VMWare ESXi on Raspberry PI

(This is a guest post by Antoni Sawicki aka Tenox)

Just for fun with virtualization I wanted to try out VMWare ESXi for ARM64, most specifically Raspberry PI. ESXi for ARM has been around for a couple of years now. Since PI4 packs 8GB of RAM and has a reasonably fast CPU it can be a worthwhile experience. Also more OSes for Raspberry PI are now available in UEFI boot mode.

Not going to go through exact installation steps as these are all around the web and youtube. Just to summary you will need to download an image from VMWare website as well as bunch of UEFI firmware files from github and combine it all together on to a SD card. When you boot it you will go through an install process which is straightforward. You can overwrite install media and use it as the target so no need for multiple SD cards. Once it boots you will see familiar ESXi boot screen:

ESXi booting on Raspberry PI 4

In order to get it going you will obviously need to add some storage. You can use NFS, iSCSI or locally attached USB drive. For the latest you need to disable USB arbitrator.

# /etc/init.d/usbarbitrator stop
# chkconfig usbarbitrator off

What can it run?

ESXi ARM only officially supports only UEFI boot based OSes. Fortunately this is a default option for Ubuntu PI, Free/Net/OpenBSD also work and so does Windows. But what about OSes that use U-Boot? Since ESXi-ARM Fling 1.1 you can boot oses in a “direct” mode with no UEFI! This is a huge step, but unfortunately as of today it doesn’t support UEFI-less VGA, only a serial port. Hopefully this can be fixed in future. I would love to have a RISC OS and/or Plan 9 VM. On the other hand Plan 9 supports EFI boot so an image could be made.

Windows guest install was also much easier than I expected. Thanks to UUP dump you basically roll your own bootable ISO. I think it’s actually easier to get it going on ESXi than natively on RPI hardware or QEMU.

Windows 10 Guest VM on ESXi Fling Raspberry PI

NIC driver obviously did not work by default, but there is a VMXNET3 ARM64 driver in the wild:

VMXNET3 for Windows 10 ARM64 on ESXi Fling on Raspberry PI

What is it good for?

Right now probably just for fun. But I can easily see datacenters filled in with ARM servers running ESXi. Future is bright and free of Intel! Personally I will keep it around for development purposes if I need to make builds for ARM on various OSes.

Interestingly enough you can even run VMWare ESXi ARM on QEMU with nested virtualization!

Also this is the official VMWare ESXi ARM Blog worth checking for future updates.