Extracting warc files

well since the catastrauphic outage, I’ve been looking through my backups trying to see how much of my ‘vpsland’ archive I have. And it’s not so hot. The good news is the physical machine that has the last known good copy is fine. It’s just in a place I can’t get to on the other side of the world. And I’m still in exile so shipping it really isn’t an option at the moment.

On the plus side I found a warc archive, some 22GB of the 400GB worth of files. So its a start.

So what are WARC files? why do people gzip them to get maybe 1% compression? How do magnets work anyways?

web archives are single snapshots in time of a site. Sounds like a MHT but something more ‘portable’ and open standard-ish. Which means there is a million tools, none of which seem to do exactly what you want.

All I want to do is extract all my files from the WARC, but that seems to not be what most things are geared to, mostly displaying the WARC like a web page, which means clicking hundreds of thousands of files. –yikes

Thankfully warcat seems to be able to fit the bill

python3 -m warcat extract ../[email protected]015-10-04-fc233ad0-00000.warc.gz

I didn’t see any package on Ubuntu so did the pip install:

pip3 install warcat

And that seems to have done the trick.

Now to figure out how to setup some cheap storage on azure and copy this stuff up or extract over there.

spot pricing

I’m using the new ‘spot‘ pricing model, to try to keep costs down. Obviously it’s not as good as dedicated slices, but it’ll not make me broke either. And I have a lot more messing around with containers to do, trying to string together nonsense.

Windows scp to remote machines with spaces in the directories

Well one nice thing about Windows 10 is that it has a built in ssh/scp client! Although telnet is optional, I get that it’s insecure but jeez what is a retro user to do?

Anyways the subject at hand is copying files from somewhere that has spaces in the path. In this case I need a copy of OS X Snow Leopard from my Mac Pro cylinder to this junk Fujitsu Celsius. I’m still having USB issues, but I’d like to get my data off of an external disk formatted in HFS+. And for ‘reasons’ I wanted to use something “native” but I don’t feel like building a Hackintosh. While not a strict tutorial on getting Snow Leopard running, I did upload my old download of Empire EFI on archive.org as this kind of stuff is damned near impossible to find.

So back to the matter at hand, I have this VM setup on my Mac Pro, and I want it on this Windows machine. You’d think it would be something like this:

scp -C [email protected]:"/Users/neozeed/Virtual Machines.localized/OSX 10.6/*" .
scp: /Users/neozeed/Virtual: No such file or directory
scp: Machines.localized/OSX: No such file or directory
scp: 10.6/*: No such file or directory

Okay so double quotes didn’t work. How about a Unix style escape for spaces? I mean it *is* scp after all, maybe it doesn’t know it’s on Windows.

C:\osx>scp -C [email protected]:"/Users/neozeed/Virtual\ Machines.localized/OSX\ 10.6/*" .
scp: /Users/neozeed/Virtual: No such file or directory
scp: Machines.localized/OSX: No such file or directory
scp: 10.6/*: No such file or directory

Well maybe it parses it like C, so you need double backslash? NO that doesn’t work either. Talk about frustrating. So, in an act of insanity, I tried single quoting the interior spaces around double quotes, something idiotic like a bash variable:

C:\osx>scp -C [email protected]:"/Users/neozeed/Virtual' 'Machines.localized/OSX' '10.6/*" .
Mac OS Snow Leopard.vmdk                                                               69%   11GB  16.0MB/s   05:16 ETA

And yes, now it’s transferring. I’m just using a cheap 50zt 5 port 100Mbit dumb switch. It’s good enough and it’ll probably take some 30 minutes to transfer all the bits, but it’s working.

So there you go. You may not need it now, or tomorrow but it’ll save you the 20 minutes of frustration!

So there has been a problem

I’ve been on the road for a few months, basically in exile. I’ve downsized my once large online presence to a desktop in an office I’m still obligated to for rent and internet. So my i7 workstation had been up to the task, although it seems I didn’t have the right power settings to turn back on from a power failure.

So now I’m faced with a stale backup as i can’t find my currents. so yeah content has been lost. So i also don’t want to race back to a single point of failure, so once more again I’m going to try a cloud. Microsoft keeps sending me these $200 trial things so i thought I would try it. There is a large learning curve about their networks , and deployments. although there is a bunch of ”shake and bake” deployments to speed things along. As always the key is reading the documents.

I still have a lot of stuff to upload, so for the moment the database is restored but there is a lot of messing around needed for the the old layout.

So appologies for the mess.

Special thanks for Tenox for helping me with all kinds of issues, and support from my muse.

Its always darker before its bright, and its already getting dark in northern Europe.

IBM AIX for IA64 (Itanium) aka Project Monterey runs again!

(This is a guest post by Antoni Sawicki aka Tenox)

Project Monterey was an attempt to unify the fragmented Unix market of the 90s in to a single, cross vendor Unix OS that would run on the upcoming Intel Itanium (and others) CPU. The main collaborators were: IBM, who brought its AIX, SCO brought UnixWare, HP was supposed to bring parts of HP-UX and Sequent DYNIX/ptx. Ironically the project shared fate of the Itanium CPU—it totally failed. In the end Linux took spot of the “single Unix OS”. IBM donated AIX pieces to Linux instead and the main legacy of Project Monterey was the famous SCO vs IBM lawsuit.

IBM did however produce AIX version for the Itanium architecture! According to Wikipedia, some 30+ licenses were sold in 2001-2002. For years a dedicated group of individuals was trying to locate a copy of the legendary OS. It seemed that the OS was lost forever…

…until some 21 years later friends of NCommander checked in with a set of AIX5L IA64 CDROMS! The CDs have now been dumped and you can download them here. Unfortunately downloading will not get you much closer to actually running this. As of today no publicly available virtualization or emulation platform can boot this. Yes we tried Simics, looked at QEMU IA64 and XEN/KVM for IA64, etc. The OS will not boot on modern Itanium 2 (McKinley) CPUs, only the early “pre-release” Itanium 1 aka Merced. The only emulator allegedly capable of doing so was the super elusive unobtanium called Intel SoftSDV.

It’s currently speculated that AIX5L IA64 will work on and only on so called “Intel Software Development Vehicle (SDV)” sometimes referred to as “Intel Engineering Sample”. It was an Intel made machine, later sold in several OEM branded version: IBM IntelliStation Z Pro 6894, HP i2000 Workstation, SGI 750, Dell Precision Workstation 730 and Fujitsu-Siemens Celsius 880.

Intel Itanium Software Development Vehicle Lineup

…yes, they all look alike because all of them were in fact produced by Intel with custom case badges and paints.

Luckily I was able to score a working HP i2000. AIX booted up and installed on a first try:

AIX 5L IA64 on HP i2000 Workstation – boot loader
AIX 5L IA64 on HP i2000 Workstation – logged in

Initially I was not able to get the onboard NIC working. Upon short investigation AIX5L IA64 supports only two network cards:

adapter 23100020 IBM 10/100 Mbps Ethernet PCI Adapter (23100020)
adapter ae120200 10/100/1000 Base-T Ethernet PCI Adapter (ae120200)

The AIX Itanium Early Adopters Release Notes mentions a few other cards but I do not see drivers for these in the OS. The doc mentions “Extended Hardware Drivers CD” which we don’t have.

Luckily again I was able to find a working NIC on eBay!

The system comes with X11 and CDE but so far I was not able to get any GPU working beyond basic text mode. I tried many different video cards from that era but there simply doesn’t appear to be any driver in the OS except for basic VGA / LFT. I think the key to getting video working is the previously mentioned extended hardware drivers cd.

Finally, if you want to read more I have found some interesting pieces on ibmfiles and various mirrors here and here.

Update: Thanks to efforts of TRN we now have a working GCC and ports of lots of apps!

Update 2: After going through a pile of video cards I now have local X11 and CDE!

AIX IA64 local X11 with CDE

This was the lucky winner:

Happy 4th of July!

No kaboom!

Shockingly no explosions, I was recapping stuff to notice that the PSU I’m using is sliced. Of course a 35 year old PSU runs better. I need some transistors, and maybe some diodes, but I don’t have good access to any at the moment. So weird how 80’s DRAM could need +12/+5/-5v to operate. Oh well.

VOGONS.org turns 20 years old today!

Stiletto had dropped on by to share this amazing milestone!

Today is the 20th anniversary of http://VOGONS.org. I was there helping to brainstorm it into existence in June 2000 at @bravenet on vladr’s VDMSound forums before @zetafleet hosted it, I registered for it on July 1st, and I helped give it its name and “theme”!

Follow the twitter thread here!

For the longest time VOGONS was the place to get information about VDMsound the sound blaster emulator for NTVDM, allowing a far more rich gaming experience on NT, DOSBox, the ubiquitous PC/MS-DOS emulator that is simply everywhere, and of course where I was ‘discovered’ via ‘Quake1 with WATTCP built with DJGPP on DOSBox‘ some 10+ years ago!

So happy 20th to VOGONS!

NetHack now part of the Software Collection at MOMA

Asset 199863

It’s over online display here:

Jean-Christophe Collet made mention of it over on linked in of all places. Link to the article is here. Since I do not have a linked-in account (I was forced to open one at a prior employer, they were an early investor, back when absolutely *nobody* used the thing, before Microsoft had bought them), so I was unable to read the article. But before I had created a smurf account, I found that just switching to incognito mode got me the article in question.

It’s almost crazy to put it in perspective after all these years, but when NetHack became a thing in 1987, the GPL didn’t exist yet, instead opting for the BISON license. There was no NetBSD, nor was there a Net/1 release of BSD, meaning there was no wide-scale free Unix to the Masses.

I’ve tried numerous times to get into it, even playing it a fair bit when I got it running for the x68000. But I never got far, as I’m clearly out of my league. Which is a shame I’m clearly missing out on quite a bit.

Since it’s all news to me, there indeed is a software category over at MOMA. When I did live in New York, I was a member. I did all that artsy stuff, although I’m clearly not the artiste.