Anglofying & running Japanese Lemmings for Windows

On one of my later trips I picked up this fun title, Lemmings!

And looking at the back of the box, what fun it contains!

Support PC-98, Epsons, IBM’s, IBM PC/AT.. Probably FM Towns as well

One interesting thing about 1995, is that with the rise of Windows 95, this marked the end of the specialized PC market in Japan. Just as WING/Direct X basically killed off the DIY driver/extender environment on MS-DOS, by being able to abstract the hardware it removed any meaningful difference between an EPSON PC vs a PC-98, FM Towns, or even the lowly IBM AT/386.

This being a Win32 includes both WING & Win32s. A perfect snapshot of an early Win32 commercial game circa 1995, as you needed to cater to that massive Windows 3.1 install base, although so many were rushing to Windows 95. Naturally this also means that the setup program is a Win16 app, once more again to preserve that bridge of the Windows 3.1 & Windows 95 world.

Well the obvious thing to do is just install it on a legacy 32bit OS, and what better than Windows XP?

Lemmings happily running under Windows XP via VMWare.

Now to run it on something like Windows 10, it’s just a matter of copying the WINLEMM.INI into %sysroot%, along with placing a copy of WING32.DLL into the %sysroot%\SysWOW64 directory and you are good to go!

Japanese Lemmings on English Windows 10

Sadly the character encoding in Windows is still really lacking and doesn’t render all that great. However that had me thinking as almost a decade ago I did find a demo of Lemmings for Windows. Could it be possible to just overlay the executables & DLL’s to produce an English commercial version?

Surprisingly the answer is yes. I wasn’t sure what to expect, but it’s as simple as that!

The game is mostly playable, some parts are just coded to run as fast as possible, as no doubt nobody was imagining 1+ Ghz machines. So the intro, warp & suicide are almost instant.

It’s something to keep the kids entertained for a day in recent events. It’s been a LONG CNY.

Living with ARM for a week

So after the crazed purchase I made a few weeks ago, I returned from Japan, and was able to unbox and use the machine I’d been wanting for a while, a non x86 Windows laptop!

The NovaGo has a Snapdragon 835, and my phone, the ASUS ROG phone has the 845. Yes for this week, my cellphone actually has the stronger processor than my computer. Honestly this is almost an unthinkable situation! Although I haven’t been using my phone as a desktop substitute this week. It’s amazing how MS screwed up 10 on the phones, and Continium.

By default it comes crippled with this ‘S’ mode Windows, which hearkens back to the Windows RT launch, with the difference that it’s a quick trip to the application store to unlock Windows 10 Professional. It’s a free download as it should be, and it doesn’t even require a reboot!

Build quality isn’t so bad, the screen folds all the way back to make the machine into a ‘tablet’ although I don’t like that mode so much, it just feels wrong to wrap a keyboard around a monitor. However if you have rambunctious young kids, it’s great as when someone went running by me flailing their arms around like a while animal, when they struck the laptop the screen could easily fold back 180 degrees. Yay.

My first thing to do after setting up Office and VMWare VDI was to install the Linux subsystem, and Ubuntu. it’s exactly the same as it is on x86_64, which is great. And this let’s me have the best of both worlds, just like x86_64. As much as I dislike stumbling around with that aborted child of Pascal & Fortran (Python) at least I can run it under (mostly) Linux to get something close to like the production environment.

The C/C++ compiler is actually all cross tools. I wanted CLI only stuff because I like torturing myself, and it required a few GB of downloads. The good news is that the latest Windows 10 SDK does support GDI/CLI apps, so no crazy SDK hacking required, unlike back in the Windows RT days. Oddly enough the Taskforce 87 interpreter runs fine, but nothing else does.

I did a horrible job at hacking up SDL 1.2 to at least run (kind of, the audio doesn’t work, and it’s all WinDB *EDIT I got it fixed!!!) I got a few things up and running, including DOSBox and FrontVM. One thing that greatly helps is that i386 binaries ‘just work’. Honestly you wouldn’t even know you are running them when you are. Which made hunting down the ARM64 version of Chromium Edge kind of difficult to find. There really needs to be a more apparent way to tell them apart, if anything for battery efficiency.

As pictured, SQL Server 4.21a runs just fine, again you wouldn’t even notice. Same for Visual C++ 1.0 & FORTRAN PowerStation 1.0.

Again the audio in my crap SDL build doesn’t work, so DOSBox is silent, and without Direct X, the text mode is tiny. Oh also, there is no OpenGL in this version of Windows dev kit for some reason. Running ssystem is ungodly slow. Also the default optimizations seems to be Os, optimize for space, and on this ASUS I have to say /Ox is way way faster. DooM is quite playable on DOSBox when build with /Ox, unlike /Os.

For me, I spend most of my day to day in Office, and VMWare VDI, connecting to secure networks. So I’m just one step above a terminal. Which I guess is kind of sad, but this machine more than fills that roll for me. The 120GB of storage is tight. This isn’t a development machine persay, nor is it something to download tonnes of data to, it’s a lightweight machine where it’s strength is the built in 4G modem, and when running ARM software the longer battery life. To me the biggest drawback is that the keyboard isn’t backlit. Even though I touch type, I didn’t realize how much I’d grown used to it for casual use.

I guess it’s a hard toss up from this and a PINEBOOK Pro, I think most readers here would prefer the Pinebook, for all it’s openness, although I still like the idea of being able to copy over the Win32s version of Lemmings, and it just running. For me I kind of like this thing, although once I switch back to an x86_64 with more memory, better GPU and disk options, maybe this just feels like some kids toy.

Neko98 ARM!

I don’t know how I didn’t think of this, but I also ported Neko98! Although the STL is having an issue with the ‘control panel’ so Neko is on autopilot.

This app can’t run on your PC

As for the emulation, it is 32bit only, so expect to see this stupid message quite a bit. The neckbeard is a nice touch though.

Also built into the thing is a cell modem. I guess it’s really not a surprise as the 835 really is a cellphone SOC. I have a ‘wifi egg’ as they are called here, a WiFi hotspot with unlimited internet from CLS, which is on the old 4G network. I popped the SIM in, and it picked up the APN settings on it’s own and I was connected in under a minute. I have to say that it’s about time that SIM cards have this stuff programmed into them for a plug & play experience. And thankfully the ASUS is unlocked, although from what I understand these were sold in the USA bundled with some cell service plans.

For anyone with one of these rare machines that cares to play along you can find my built stuff on my ‘vpsland’ archive:

https://vpsland.superglobalmegacorp.com/install/Windows10-ARM/

And the f2c/dungeon build along with the C/C++ compiler is in the aptly named dungeon-2.5.6-ARM(32bit)-win32.zip file.

Just made a drunk purchase… Not too mad

NovaGo!!

I’ve been a hidden long time fan of non x86 NT, I’ve owned Alphas and PowerPC (still sadly no MIPS), and when it came to the arm platform, ive since picked up the Surface RT and the Surface 2 RT. YouTube works fine on both, although the 2 is far faster and overall nicer user experience. I use the 1st Gen as a winamp player as it’s easier to jailbreak and cross compile to and mess with. But locked down Windows 8.0 for arm is insanely limited.

Enter Windows 10 and another botched shot at Windows on ARM for the general consumer. These ship with a S limited version of windows, which apparently can be easily unlocked to full 10 pro. I chose the Asus as it’s a laptop, and has more ram than the HP. Both however should be enough for casual day to day usage of office and edge chromium. I’ll have to see how it goes for either cross or native compiling.

Although the arm in these machines is 64bit, is there 32bit user land at all? Is it still possible to maintain a 32bit userland of gcc 1/2 and binutils for legacy compiles? How terrible is x86 qemu on arm emulation? DOSBox native? I guess SDL should be a simple rebuild like NT MIPS?

I’m also curious about WineVDM and MS-DOS player.

Oh well, I’m just waiting for a flight in the airport, going slowly insane.

UPDATE on the ASUS here:

My conclusion is really that the biggest problems with the physical machine is the lack of a backlit keyboard, and the tiny storage. Windows on ARM feels like a solution looking for a problem, but the obtuse problem is non x86 diversity. And in those regards it’s pretty much working fine.

But looking forward to a non x86 usable machine. I even have an unlimited chip for Hong Kong. It’ll be interesting if it can keep up for me, and if I’ve finally hit Ted Smith FORTRAN Maximum usage. Although this has no floppy drives.

VMware 12.5.9 on Windows 10 build 1903 / version 18362

So while I’m in Japan, I bought this tiny and borderline useless Fujitsu Esprimo B532, powered by an i5, and not very much else. I upped it to 8GB of RAM, and put in a SSD and upgraded to Windows 10 to make it slightly tolerable.

i5-3470T

The i5-3470T is ancient! And so old that newer versions of VMware and Hyper-V won’t run on it. The old solution was simply to use an older version of VMware. In my case the highest version that’ll run is 12.5.9, however when trying to launch it I got this fun message:

VMware Workstation Pro can’t run on Windows!

Well wasn’t that a big bust.

I guess there is something hidden somewhere, but I just renamed the executable, and set it to Windows 8.0 compatibility mode, and wow it works!

Windows NT 3.1 October 1991 pre-release

And there we go! Now the latest version of NT can run the first public pre-release of Windows NT. YAY.

WDF_VIOLATION on Apple Macintosh

So I bit the bullet and updated to Windows 10 build 1903. And then the fun started on my glorious 2006 MacPro. It finished the update, and on reboot I get the login screen, and then almost immediately a blue screen.

Naturally the QR code is useless as it doesn’t specify any stop codes, and the minidump… Well that requires gigabytes and gigabytes of crap to download to get a tool to read it. (I still haven’t finished that rabbit hole, like COME ON! why isn’t it included?!).

However after hitting F8 a million times, I found that safe mode & networking work just great. Searching online was basically useless as there was no specific stop code to go with this WDF_VIOLATION. Further looking around I did notice one thing, and that it was all Macintosh machines that crash out to this WDF_VIOLATION error. It must be something specific to the Apple hardware running Windows 10!?

Armed with this (dis)information, I went ahead and disabled all the Apple specific drivers & startup items.

From msconfig.exe:

From MSCONFIG.EXE I disabled the following services:

  • Apple OS Switch Manager
  • Apple Time Service

And in the task manager, I disabled the following startup items:

  • Realtek HD Audio Manager
  • Boot Camp Manager

I had the other VMWare serial & USB hook previously disabled, as I just don’t want them at all on my setup. The big upshot is that after rebooting out of safe mode, I’m now up and running on Windows build 1903.

Considering the BootCamp stuff was so woefully out of date, don’t expect Apple to fix this anytime soon. And since I’m on a MacPro 2006, I certainly won’t be getting any updates from Apple. But at least I can struggle to keep this thing up to date otherwise.

Now I can enjoy that ‘new command prompt’ everyone keeps telling me about.

***UPDATE***

I went through this on another Bootcamp Mac, and what I had to do was uninstall the “Boot Camp Services”. It’s startup component triggers the bluescreen as it’s doing some nonsensical inventory, banging around on the drivers in a not friendly way. I had version 4.0.4033 of the Boot Camp Services installed.

Uninstall Boot Camp Services version 4.0.4033

Removing this kept all the old drivers, which continue to work just fine.

Got some new processors!

Xeon E5-2667v2

Moved up from the old pair of E5-2620 v2’s to a pair of E5-2667 v2’s. What a big difference from a base clock of 2.1Ghz to 3.3Ghz. And yes, more cores!

2475 Cinebench score

And I can build DOSBox in 3 seconds using Visual C++ 6.0 Ultimate. I guess eventually I’ll get a modern machine, but for now this is pretty damned good. Which reminds me the newer processors for my 2006 Mac Pro should be arriving soon enough.

I couldn’t quite justify the more than double price for the E5-2697 v2 processor, although it has 50% more cores, but with a max clock of 3.5Ghz.

Oh well it’s as good as any update to the Huananzhi X79.

Huananzhi X79

So I picked up this board on AliExpress for about $200 USD. Natrually the x79 chipset is NOT a dual CPU chipset, so yeah it’s one of those ‘not exactly 100% legit’ Chinese motherboards.

One thing about Chinese companies that many don’t sell directly to consumers, instead they sell on Tao Bao, Alibaba, or to foreigners, AliExpress. The company’s site is http://www.huananzhi.com, as they had written on the box. Yes you need the www. portion of the name, as again many things are… well dated on the Chinese internet.

The product page is currently http://www.huananzhi.com/html/1/149/150/256.html, if anyone cares. It lists the technical specs as:

  • Support for Xeon E5 series processors
  • High-speed DDR3 four-channel memory slot: support 1333/1600/1866MHz, maximum capacity 128G memory slot
  • Desktop general purpose memory
  • X79 dual motherboard supports 5.1 channel ALC887 chip
  • 8-layer PCB backplane, 7+7 phase power supply
  • High-speed USB3.0, SATA3.0 interface transmission speed is increased
  • PCI-E expansion slot*4
  • RJ45 Gigabit LAN interface
  • North Korean heat sink with HUANAN logo

Yes, I don’t get the whole Korean heat sink thing either. Anyways I thought it’d be fun to try so I ordered the thing. It took 3 days to get to my office in China, and an additional week to get from China to Hong Kong. I hear these things can take upwards of a month to arrive in North America.

Also worth noting is that they will not ship with a CMOS battery, so you need to supply your own CR-2032 battery, otherwise the board will not operate correctly.

The contents of the box are VERY minimal, but they did include 2 SATA cables, some CPU thermal paste, a very bare and … well not very good manual, a CD which I haven’t even tried to read, along with an IO shield.

Oh and for visitors of the future, here is a poorly scanned copy of that manual. It really is 2 pages. For what it’s worth it makes as much sense in English as it does in Chinese.

I decided to pair this with a pair of E5-2620 v2‘s that I got for $40 USD shipped, as I didn’t want to initially spend a lot of money in case all of this just exploded or something. These were the ‘widest’ and cheapest processors I could find, I wanted a v2 E5 as they are faster then the first generation.

Also worth noting is that the board is only capable of driving v1 & v2 E5’s. And they need to be the E5-2 type, which support operating in pairs, unlike the E5-1 set. I have no idea if the E5-4’s aka 4-way part would work in a pair. Although it may be an interesting experiment to try.

The board apparently doesn’t support overclocking or anything that fancy.

Although it reports itself as an x79 based motherboard, it is in reality an Intel C602, based chipset. I don’t know if they are harvesting them off of recycled servers, or if they have located a giant cache of repair parts that have been pushed beyond 5 year warranties, so they are prime candidates for being re-purposed as end user motherboards. Nice things about these boards vs standard server boards is the inclusion of a Realtek HD Audio chip, VIA USB 3.0 controller, and even the nice spacing out of the slots so you could really use all the slots.

Since this is a dual processor board you really want a PSU with dual 8 pin power connectors, however as mentioned in the poorly translated manual, you can take a PCI-E 6 pin adapter, and place it into the 8 pin socket, just position it backwards so that the 12v+ pins are facing inwards.

It may look strange (well more so as I’m using an extension cable that is sadly more focused on aesthetics than function, but heh it was cheap), but rest assured it works!

Another thing to keep in mind is that since this board uses a server chipset, not a consumer one, just as it is using server processors, you will need server grade memory. In this case it’s REG ECC DDR3 based memory. I went with 1833Mhz parts, which are the fastest DDR3 parts they made. Although the processors I chose have a maximum frequency support of 1600Mhz, but the memory works fine when underclocked.

Another gotcha is the CPU fans. These need to fit the Intel Xeon 2011, but have support for the 2011 motherbards. Which unlike the consumer versions don’t have a separate plate to bolt to the underside, rather they screw in all from the top. I had purchased a pair of cheap heatsinks that were about the right size, but didn’t include any of the mounting hardware for a 2011 board. I picked up these GELID Phantom Black CPU’s for about $80 for the pair.

They are quite big, and include a pair of fans for each processor which will make the end build look a little crazy.

I didn’t want to spend a lot, and went with the cheapest PSU I could find to output more than 450 watts. Although it did turn on and run with the lower PSU the machine did shut off overnight for no apparent reason. I’ve been okay with the larger and cheap Antec NX 650 PSU.

Although, this is the older style ‘bundle o cables’ type of PSU which I’m not such a fan of.

If I had charged up a cordless screwdriver this would have taken a few minutes, but screwing in the heatsinks was a chore, and they really do dominate the boards real estate.

Almost completed build

I thought I had a case, but it turns out that it was for normal ATX sized boards, and this is an E-ATX board so it simply will not fit.

view from above

Another nice server like feature is that the board has an LED readout for early post codes, as booting this board will take some time. I think with 32GB of RAM it’s almost a minute.

I took the SSD & Hard disk out of my MacPro 2010 and put them into the new machine, and it booted up right away. Once connected to the internet Windows 10 picked up the new hardware and downloaded and installed the board drivers as needed. Interestingly enough Windows 10 also wanted a new activation code as the CPU/Motherboard was changed, although it didn’t complain about it.

When it comes to jobs that can run in parallel this is an incredible build. Obviously single core performance at 2Ghz is. well. terrible. I know going to a 4Ghz max E5-2667 v2 won’t be exactly magic either, but there is something nice about having 32 threads. Running stuff like parallel compiles, compression and video encoding is a dream on these massively parallel machines.

Games, are ‘okay’. I get 60fps with Fallout 76 on this current 2Ghz build on medium settings with the 1050 video card.

I do plan on getting faster CPU’s after the Chinese New Year, as right now basically everything is shut down (it sucks being the only person in the office building, literally), and shipments wont’ resume for at least another week.

Installing Windows 10 pro onto an older Mac Pro

So I have this 2006 Mac Pro 1,1 that I’ve had laying around and I wanted to put my old Nvidia 1030 into it, along with Windows 10 for a newer (stronger?) home machine.

So I burnt the downloadable ISO from Microsoft onto a DVD, tried to boot it up and got this:

Ignore the microwave…

I got stuck at this “Select CD-ROM Boot Type : ” prompt, which you can’t type anything into. Apparently it’s a common and known issue with 64bit boot code, as the older Intel Apple Mac’s are of course 32bit only. So there is a fix, you have to use something called “oscdimg” to rebuild the ISO with a 32bit friendly loader.

So first I just used 7zip to extract the downloaded ISO, and then create the new 32bit ISO with the following:

oscdimg -n -m -bc:\temp\Windows10\boot\etfsboot.com c:\temp\Windows10 c:\temp\LegacyBootWindows10dvd.iso

And yes, it boots!

Brigadier

Now what about those special ‘bootcamp’ drivers? Obviously under Windows 10 there is no nice way to get the ones you want. And what about which model/download package to get?

Then I found Brigadier!

It’ll probe your hardware type and grab the drivers for you!

https://github.com/timsutton/brigadier/releases

Visual Studio 2003 on Windows 10 & Errors with Desktop Compositing/Themes

While building the latest DOSBox SVN using Visual Studio 2003 I found something kind of annoying under Windows 10.  The first thing is that if I search through the source code base, the application locks up, hard.   It turns out that this has been an ongoing issue with Windows 8 (maybe Vista/7?) with Aero rendering of all things.  The fix is to disable Desktop Compositing & Desktop Themes, but the application comparability tab is hidden on many applications for Windows 10.

Broken Visual Studio

See how the application preview doesn’t render anything at all?  This is the hint that it’s broken.  I think it may be worth sharing this ‘fix’ as I’m sure that other applications that behave strangely have the same issue.

I found the solution to this over on stackoverflow in this discusstion:[https://stackoverflow.com/questions/2422581/visual-studio-net-2003-on-windows-7-hangs-on-search].  The fix is a registry entry in the “HKEY_CURRENT_USER\Software\Microsoft\Windows NT\CurrentVersion\AppCompatFlags\Layers” key.

The required settings to devenv.exe is:

“^ RUNASADMIN DISABLEDWM DISABLETHEMES”

Which, will run Visual Studio as Administrator allowing you to debug, and disable all the Aero assists for the application allowing things like search to work again.

I had gone further and enabled the Windows XP SP3 compatibility settings, however on doing a clean build I was presented with this error:

fatal error C1033: cannot open program database ''

Which I never could find any good source on what caused it, other than by guessing to remove the Windows XP flag, and now I’m able to build.

While debugging and re-building the debug database is held open on Windows 7 (maybe Vista?) and beyond on x64 based OS’s. You’ll get the annoying LNK1201 error.

There is a fix on bytepointer.com (local backup vs7.1_on_win7_pdb_handle_leak_unofficial_fix_win10.zip) that involves patching/replacing natdbgde.dll . All I can say is that it seems to be working for me.

In order to do a full build of DOSBox I had to re-build SDL, SDL-net, zLib, libPNG, and set them to a common C runtime linker setting to get a build where the final link didn’t complain.  However when it came to existing project files I did have to find some older Visual C++ 6.0 stuff for many of the components, but using those I was able to ‘upgrade’ them to the 2003 environment and produce a working set.

I’ve got to say, that the AVI capture in the newer branches (I’m using build r4177) is really great!

Running VMware ESX Server 2.5

One of my favorite things about VMware is that it can run itself.  This allows me to test & stage new setups, test API stuff on my desktop, allowing me to build a “micro data center” that I don’t need to ask & beg for permission to take down, or if I do something stupid, I’m just a quick revert away from putting it back, and more importantly not making other people mad.

This also let’s me step back in time, in this case to the dark & ancient world of 2005, where I’d first deployed VMware ESX 2.5.2 along with vCenter 1.3.1 .  I figured that I could use my ancient Dell P490, as I’d been using it as a desktop at home for casual use, but this seemed like a good thing to stress the system on.  Also handy to have is the installation guide, which VMware still has online.

I installed Windows 10 Pro, and VMware Player 12.5.9, The box has a single physical processor that is dual core, 8GB of RAM and a 1TB disk.  Not exactly a high end machine, but it’ll suffice.

The first thing to do was install ESX 2.5.2, I’d set it up as a Linux VM, with 1 CPU, 2GB of RAM, and 3 disks, one for the OS, another for SWAP, and a Data store / Data disk.

After the nice GUI setup is completed we are dumped to a console on reboot.  ESX is meant to be managed remotely.

Once the OS installed, edit the VMX file, and make the following changes, to allow VMware to setup the passthrough capabilities so the VM can run other VMs.

guestOS = “vmkernel”
monitor_control.vt32 = “TRUE”
monitor_control.restrict_backdoor = “TRUE”

Now the Version will report that it’s VMware ESX.  The other thing you’ll find out quickly is that you need a browser to manage the server (funny how things went back to this direction, later versions relied entirely on the ‘fat’ .NET client), and I found that FireFox 1.5 works the best.

The .NET client requires .NET 1.1 to operate correctly.  It will not install on Windows 10 by default, as the .NET 3.5 (which includes .NET 2.0 runtime) is not acceptable, it has to be the 1.1 runtime, along with the J# runtime, which it’ll install if needed.  I went through the installation steps in the aptly name ‘Installing .NET 1.1 on Windows 10‘. post.

Of course you’ll need a place to run the vCenter server, I just setup a Windows 2000 server, installed SQL 2000, .NET v1.0 & v1.1 and then the Virtual Center component.  VirtualCenter relies on a database backend, and I thought it’d be interesting to look at the tables through MSSQL, although Oracle, Access and some generic ODBC are also options for this ancient version of VirtualCenter.

For those who don’t know, VirtualCenter is the application that lets you build a ‘virtual datacenter’ join multiple ESX servers together, and more importantly orchestrate them together into a cluster, allowing you to vMotion VMs between servers,  which of course is the ‘killer feature’ of VMware ESX.  If you don’t have vCenter / VirtualCenter then you are missing out on so much of the products capabilities, which is sadly hidden away.

I setup a tiny Windows NT 4.0 domain, with a domain controller, and a terminal server.  My host machine is a bit weak to setup more ESX hosts, as there just isn’t enough punch in the box.  Although any modern machine will probably exhaust RAM before CPU running a mid 90’s workload.

Back in the day, I had moved our entire DC onto 4 ‘killer’ machines with fiber channel storage and had consolidated the entire DC to a single cabinet.  It was incredible that we were initially able to almost meet existing performance.  Of course the killer feature again is vMotion so a year later, I only needed 4 new servers which was an easy budget ask, and in the middle of the day I vmotioned from the old servers into the new servers, and things across the board were now faster.  Finally the bean counters saw the light that we didn’t have to buy faster gear for a single group, or that we no longer had the issues where we had ‘important enough’ to be in the data center but with no hardware maintenance, or proper backups.  Now everyone is on equal footing and all the boats raised with the tide so to speak.

In this quick re-visitation it would be fun to setup shared storage, multiple hosts and vMotion, but back in the days of ESX 2.5 there was no support for having VMFS over NFS or iSCSI.  As much as I’d love to use the Dr Dobbs iSCSI Target Emulator, it just sadly isn’t an option.  The ability to move beyond Fiber Channel shared storage (or other supported dedicated host bus adapters) was added in version 3, greatly expanding the capabilities of ESX.

 Obviously the career mistake here was to be a happy Admin, and concentrate on other things as now the infrastructure ‘just worked’ and it freed up an extraordinary amount of time.  The smarter people were either taking these types of experiences and turning it into a consultation gig (low effort) , or taking lessons learned in VMware space, and focusing them onto QEMU/KVM and building libre infrastructure (high effort).

Such is life, be careful riding those trendy waves, eventually you have to either lead, follow or just get out of the way.