ttyplot – a real time plotting utility for the terminal

(This is a guest post from Antoni Sawicki aka Tenox)

I spend most of time in a day staring at a terminal window. Usually deeply nested ssh connections through several jump boxes, vpns or tunnels. Also commonly via WebEx, Zoom, GoToAssist or Teleconsole. I often run various performance monitoring tools and read tons of metrics.

Inspired by tools like gtop, vtop and gotop I found myself wishing for a more generic terminal based tool that would plot data coming from unix pipeline directly on the terminal. For example some column or field from sar, iostat, vmstat, snmpget, etc.

Yes gnuplot and several other utilities can plot on terminal already but none of them easily read data from stdin and plot in real time.

In just couple of evenings ttyplot was born. The utility reads data from stdin and plots it on a terminal with curses. Simple as that. Here is a most trivial example:

To make it happen you take ping command and pipe the output to gawk to extract the right column and remove unwanted characters and then pipe it to ttyplot:

ping 8.8.8.8 | gawk '{ gsub(/time=/,"",$(NF-1)); print $(NF-1); fflush(); }' | ttyplot -t "ping to 8.8.8.8" -u ms

Fflush() is needed because by default Unix pipeline is buffered and we don’t want that. This article will show you more ways of working around the problem.

Ttyplot can also read two inputs and plot them with two lines, the second being in reverse-video. This is useful when you want to plot input and output or read/write at the same time.

A lot of performance metrics are presented in these pesky “counter” types which need to be converted in to a “rate”. Prometheus and Graphana have rate() or irate() function for that. I have added a simple -r option. The time difference is calculated automatically from time difference between samples. This is an example using snmpget which is show in the screenshots above.

{ while true; do snmpget  -v 2c -c public  10.23.73.254  1.3.6.1.2.1.2.2.1.10.9   1.3.6.1.2.1.2.2.1.16.9 | gawk '{ print $NF/1000/1000; fflush(); }'; sleep 10; done } | ttyplot -2 -r -u "MB/s"

I now find myself plotting all sorts of useful stuff which otherwise would be cumbersome. This includes a lot of metrics from Prometheus for which you normally need a web browser.

If you need to plot a lot of different metrics ttyplot fits nicely in to panels in tmux, which also allows the graphs to run for longer time periods.

Of course in text mode the graphs are not very precise, but this is not the intent. I just want to be able to easily spot spikes here and there plus see some trends like up/down – which works exactly as intended.

I do dig fancy braille line graphs and colors but this is not my priority at the moment. They may get added later, but most importantly I want the utility to work reliably on most operating systems and terminals. 

You can find compiled binaries here and source code and examples to get you started – here.

If you get to plot something cool that deserves to be listed as an example please send it on!

Gopher kills the LC

Macintosh LC

The LC isn’t a strong Macintosh.  It is after all, a low cost model.  And what I’m doing isn’t even slightly fair to it.

Since it has a mere 68020 running at a blazing 16Mhz with no 68881 nor any MMU running something like A/UX is simply out of the question.  However MMU less Mac’s can run MachTen.

Although I did make a backup of the disk to find out that this thing had been in Harvard of all places, apparently once belonging to Mark Saroyan.

Although there was nothing even slightly academic or useful on the disk.  I wonder if the software was even pirated as the last owner sure enjoyed all the various SIM games (city/earth/life/ant) it seems more than anything else.

I formatted the massive 50MB SCSI disk, put on a fresh copy of MacOS 7.0.1 along with the network driver and MachTen 2.2.

System 7.0.1

And as far as LC’s go, this one isn’t too bad, it’s loaded up with the maximum 10MB of RAM, although it seems the VRAM is pretty sparse as it’ll only go to 16 colours.  But since we are playing UNIX here, I didn’t see any need for that, and set it to mono.

I thought it’d be fun to install a gopherd server onto this machine, and that is where the fun started.

Granted it’s been a long time since I used a machine with no real L2 cache, let alone running at a whopping 16Mhz, and using a compiler like GCC is just incredibly slow.

So I thought I could just ‘cheat’ the system by taking the source code to GCC-1.42 and tweaking the SUN3-Mach configuration into a SUN2-Mach configuration but keeping it targeting a BSD like OS, along with setting it to compile to a 68020 without a 68881.  Oddly enough getting a cross compiler wasn’t so difficult, but the assembler on the LC, a modified GAS wouldn’t assembler the files. So I went ahead and built a68 from GAS 1.38 and now I can cross assemble from Windows. However I couldn’t get the linker ld from binutils-1.9 working.  I guess it was an endian issue somewhere, but my attempt at byte swapping files it was reading just led to further confusion.  And I figured linking on the target host wouldn’t be the end of the world, as compiling sure feels like it is.

I can’t see like anyone would care, but here it is: 
MachTen-crossgcc-1.42-nolinker.7z

So fighting the source and in a matter of a 30 minutes of on/off work I had it compiled.  All I needed to do then was FTP the objects to the machine, link and run.   Surprisingly this proved to be pretty simple.

gopherd running!

I managed to get a few pages out of it, and suddenly my telnet sessions dropped.  Looking over at the console and MacOS was busy being MacOS.

error of type 3

And that was that.

I tried another program to cross compile and upload phoon!

phoon cross compiled, natively linked.

It took a while to set the clock to the right year, as my minimal System 7 install doesn’t have the time control panel, and advancing 1 year at a time from 1999 takes time, by advancing the date to New Years Eve every minute 19 times to get us to 2018 with the old date syntax:

date 12312359

Lessons learned?

Obviously if I want to do something like this, I’m going to need a better Macintosh.  Or just not do things like this….

I’m kind of on the fence as to whither 68k Unix is really all that useful in the age of Ghz x86.  

Source to the Apple-II MIT Logo recovered

;  LOGO Language Interpreter for the Apple-II-Plus Personal Microcomputer
;  Written and developed by Stephen L. Hain, Patrick G. Sobalvarro,
;  and the M.I.T. LOGO Group, at the Massachusetts Institute of
;  Technology.

It’s over on github: https://github.com/PDP-10/its/blob/master/src/aplogo/logo.958

So I picked up this Nokia 1020, and decided to downgrade it to Windows 8.0

When I’d bought this 1020 in question, I picked it up from an AT&T customer with the intention of basically using it as a camera.  Well it was kind of okay but it had been upgraded to Windows 10, which technically is unsupported on the 1020 hardware.  And no wonder as the 1020 specific camera applications aren’t available, nor is performance all that great on Windows 10.  I’m guessing there is a laundry list of reasons of why Windows 10 was not available for the 1020.

So I saw this website, lumiafirmware.com that not only has an incredible amount of firmware saved, but it sure makes the process of loading different firmware pretty simple.

  • Install “Windows Device Recovery Tool” http://go.microsoft.com/fwlink/?LinkID=525569
  • Download the firmware form this site only the ffu.
  • And now open CMD as administrator
  • Type “cd C:\Program Files (x86)\Microsoft Care Suite\Windows Device Recovery Tool”(32 bit pc) “cd C:\Program Files\Microsoft Care Suite\Windows Device Recovery Tool”
  • Connect now your phone to your pc
  • Type in cmd “thor2 -mode uefiflash -ffufile “C:\the location of the ffu\file name.ffu” -do_full_nvi_update -do_factory_reset”
  • Wait now until your phone gives a green screen
  • Type now in cmd “thor2 -mode rnd -bootnormalmode”

Really doesn’t that sound not that difficult at all?!

Except it wasn’t.

Looking at the incredibly extensive list, I see that the AT&T 1020 phone in yellow colour is the RM877.  So I download the appropriate ffu image and run:

thor2 -mode uefiflash -ffufile RM877_3051.50009.1451.1_RETAIL_nam_att_205_01_473928_prd_signed.ffu -do_full_nvi_update -do_factory_reset

A bunch of verbose output happens and then:

Start sending payload data V2Sbl in async mode...
Percents: 0
Device responded a message that has invalid size!
Expected: 16, Received: 0
Exception during programming: 262150
Safe write descriptor index reached: false
Payload data transfer speed (0.22 MB/s) Elapsed time 40.83 sec
Payload data size 9.000123 MB
[IN] programSecureFfuFile. Closing RM877_3051.50009.1451.1_RETAIL_nam_att_205_01_473928_prd_signed.ffu
programming operation failed!
Operation took about 1 minute, 6 seconds. Average transfer speed was 0.11 MB/s.

Unknown error code.

THOR2 1.8.2.18 exited with error code 262150 (0x40006)

Unknown error code!  What the hell does that mean?

Glancing over at the phone I’ll show you what it means.

Nokia Red screen, waiting to be programmed

So the phone was stuck at the red screen waiting to be programmed, but obviously that failed.  Re-running the programmer just gave me this:

Detecting UEFI responder
Send HELLO
Send HELLO
Send HELLO
Send HELLO
Send HELLO
Lumia UEFI Application did not respond to version info query
Device is not in Lumia UEFI mode
Reading device mode failed
Failed to detect UEFI responder. 84017

THOR2_ERROR_TO_COMMUNICATE_WITH_DEVICE

THOR2 1.8.2.18 exited with error code 84102 (0x14886)

So I thought I’d hold down the volume down button & power and do a reset.  Except now my device was nearly bricked.  The display wouldn’t turn on at all, and there was no haptic feedback on powering up and or down.

HOWEVER plugging in the USB would make the connection noise, and how me a “QHSUSB_DLOAD” device.  So I could try to talk to it in emergency mode.

Apparently on the same download page, there is a hex file and a mbn file.  Downloading those and running:

thor2 -mode emergency -hexfile FAST8960_EOS_NAM.hex -mbnfile RM877_msimage_v1.0.mbn -orig_gpt

And this looked like it was going to work… but then as always it failed.

Sending OPEN_MULTI_REQ
Received valid response to OPEN_MULTI_REQ
Programming image R
Image opened successfully for reading
SAFE hex file was used and unallowed memory address was being written.
Reset the device and use the correct HEX file.
ALPHA EMERGENCY FLASH END
Emergency messaging closed successfully
Operation took about 11.00 seconds.

Unknown error code.

THOR2 1.8.2.18 exited with error code 85034 (0x14C2A)

Use the correct HEX file? But it was the one that I downloaded from the right page!  Great now my phone is dead.  Unplugging it in and out however gives me a little hope as it’s still showing up as a QHSUB_DLOAD device.

After a lot of searching I see Joemar Serrato went down the same road, and they did something weird where they split out the mbnfile from the ffu file themselves.  So I ran the following command to split out the mbn files:

thor2 -mode ffureader -ffufile RM877_3045.0000.1325.0001_RETAIL_nam_att_205_01_223461_prd_signed.ffu -dump_gpt -filedir gpt

And now I had 2 files.

10/04/2018  06:53 PM            17,408 GPT0.bin
10/04/2018  06:53 PM            17,408 GPT1.bin

Now with all that drama I could re-run the emergency mode flash

thor2 -mode emergency -hexfile FAST8960_EOS_NAM.hex -mbnfile gpt\GPT0.bin -orig_gpt

Sending OPEN_MULTI_REQ
Received valid response to OPEN_MULTI_REQ
Programming image g
Image opened successfully for reading
Uploading MBN image 70
SAFE hex file was used and unallowed memory address was being written.
Reset the device and use the correct HEX file.
ALPHA EMERGENCY FLASH END
Emergency messaging closed successfully
Operation took about 11.00 seconds.

Unknown error code.

THOR2 1.8.2.15 exited with error code 85034 (0x14C2A)

And it appeared nothing changed at all. Great.  In disgust I unplug the phone and hold down the volume down & power button for 15 seconds, but this time it vibrated.  I couldn’t believe it!  Within a few seconds it powered back up to the red screen, ready to load new firmware!

thor2 -mode uefiflash -ffufile RM877_3045.0000.1325.0001_RETAIL_nam_att_205_01_223461_prd_signed.ffu -do_full_nvi_update -do_factory_reset

Size of system mem: 2097152 KB
Send backup to RAM req...
Clearing the backup GPT...SKIPPED!
Successfully parsed FFU file. Header size: 0x000c0000, Payload size: 0x00000000572e0000, Chunk size: 0x00020000, Header offset: 0x00000000, Payload offset: 0x00000000000c0000
RKH match between device and FFU file!
Option: Skip CRC32 check in use
Start sending header data…
Start sending payload data V2Sbl in async mode...
Percents: 0
Percents: 1
Percents: 2
Percents: 3
Percents: 4
Percents: 5

I was astonished!  The progress bar was slowly moving, and the phone was doing what it should have been doing about 2 hours ago.  I dared not move, so unfortunately I don’t have any pictures.

Payload data transfer speed (9.79 MB/s) Elapsed time 142.41 sec
Payload data size 1394.893921 MB
Read flashing status..
[IN] programSecureFfuFile. Closing RM877_3045.0000.1325.0001_RETAIL_nam_att_205_01_223461_prd_signed.ffu
Get EMMC write speed...
Get EMMC write speed, SKIPPED!
Get data verify speed...
Get data verify speed, SKIPPED!
Send restore backup from RAM req...
programming operation completed!
[THOR2_flash_state] Post programming operations
[THOR2_flash_state] Executing factory reset
[Factory reset result] 0
Factory reset done
[THOR2_flash_state] Executing Full NVI Update
Write flash options ( WriteNvi: Full )

Write parameter to device via UEFI Flash App
Write parameter Ok
Flash options set successfully.

[Full NVI update result] 0
Operation took about 2 minutes, 24 seconds. Average transfer speed was 10.16 MB/s.

Exited with success

And then just like that, the phone rebooted itself, and then brought itself up to the ‘green screen’

Nokia green screen (firmware loaded successfully)

All that remained to do was to tell the phone to boot up normally

thor2 -mode rnd -bootnormalmode

Version 8.0.10327.77

And that was it!

My 1020 is now running 8.0, which includes not only the special Nokia phone apps, but all the AT&T bloatware of 2012.  I can’t believe there was a yellow pages app.  How are they still a thing?