So what had started this little ‘adventure’ is that years ago there was this great sale over at Joe’s Datacenter, where I had picked up a physical server for the mere price of $20 USD a month! What a deal! No more quotas, CPU sharing issues and generally having to share. Awesome!
So, I have them install Debian, and load up the KVM modules, and away I go, and life is good. So foolishly years later, I jumped onto the whole container thing, installed docker, and that is where everything went south.
Every few seconds while doing a tcpdump on the 100% virtual bridge I’d see a massive influx of arp traffic. I tried static arp’s on the host & the guest and it was ‘better’ but now the network traffic would hang. Things like TCP would break after a minute and stuff like UDP game state would break so bad it’d end up unplayable. This basically broke maraakate.org and hosting stuff like Quake I/ Quake II/ Daikatana and other iD based games.
My existing virtual machines now had a major issue where they would stop responding to traffic. I never could find a fix, and it ended up with me moving my blog out to sloppy.io to keep running as a container-based service until it magically stopped working and I gave up and did a full dump/reload on a hosted WordPress over at ChicagoVPS What a nightmare!
But what about all those virtual machines?
Well even after apt-get purge of everything docker, upgrading and downgrading the kernel nothing helped. The VMs still dropped off the network periodically. So with some spare time I decided to just go ahead and backup the box, and wipe the machine.
Since the physical network was working fine I was able to rsync the 300GB+ worth of data over the span of a few days. That was fine, and considering how crap the server had been, I figured some straight downtime wouldn’t hurt anything too much. While looking for an OS to install onto the machine, I saw that Joe offers VMWare ESXi 6.5, so I thought I’d just go with that, as naturally VMWare runs both VMs and with Project Photon I could maybe mess with containers again at a later date.
Since rsync had worked so well for moving hundreds of gigabytes of data from the USA to Hong Kong, I figured it’d be trivial to just convert the existing RAW KVM/Qemu disk images back to the United States of America. And that is where the fun begins.
Let me tell you!
While reading this great post on virten.net they drop mention of XSIBackup, which requires registration (yuck) and worse their stupid registration system is broken:
LOL WHUT?
Rest assured the email does show up!
Dear Neo Zeed, thank you for visiting 33hops.com
This is your download url http://a.33hops.com/downloads/?key=bq7l5ptPB70MJj9dkftxFegr3xWoBZwpdFPQOUC3Cm10KPSIl3v1532877224253
But of course it doesn’t work
The key is invalid or has expired, only two downloads are allowed per key, download a new one at 33hops.com
What
The
Fuck
I know this is an ongoing issue at large when you provide executable binaries on the internet, as you will no doubt get flagged with some false positive by some virus crusading idiot who just sets up a tool and never reads anything but sends out threatening emails. I went through this with the need for the simple 404 redirect, all thanks to Gerhard W. Recher.
So since this wasn’t going to be an avenue to peruse, I dug a little deeper and ran across this post over at virtuallyGhetto. So, it turns out the userland for ESXi is a CentOS environment that uses busybox. And if you just download and install CentOS 3.9 into a VM, and build whatever you want, and ideally add in the -static flag, and copy it over. For those who want to look into more ‘internals’ of the userland, check out zemris.fer.hr.
Great!
Things like UID/GUID mappings are broken in the libc it seems among other things. So for my simple rsync config I just put the numbers in myself.
uid = 0
gid = 0
use chroot = no
max connections = 4
syslog facility = local5
pid file = /var/run/rsyncd.pid
hosts allow = a.b.c.d
[datastore1]
path = /vmfs/volumes/datastore1
comment = WDC_WD5000AAJS2D00A8B0
read only = no
I have read that you really ought to keep your binaries/config on the datastore, so they aren’t subject to upgrades overwriting them and other chaotic stuff. So editing the file “/etc/rc.local.d/local.sh” I just added the following before the exit 0:
/vmfs/volumes/datastore1/tools/rsync –daemon –log-file=/tmp/rsync.log –config=/vmfs/volumes/datastore1/tools/rsyncd.conf
And then ran it manually to kick it off.
So now I don’t have to rely on someone’s elses broken downoad system, and now we can build other fun native stuff.
And the best part is that after all of this fighting Maraakate’s site is back online and I get this message from him:
holy crap that new server setup so much better
its like playing it locally honest to god
played a whole unit not a single fuck up
no rubber banding lag effect or any of that
So now things are actually performing better on VMWare than we were getting on KVM. And yes, I had flattened out the disk images, loaded up the paravirtual disk & network drives on KVM, but VMWare does such a surprisingly better job.
I honestly wasn’t expecting that.
And as a bonus, I messed with qemu-0.9.0 (I didn’t feel the need to go through glibc2 hell), and qemu-img works great with a simple raw to vmdk
[root@jdc-user:/vmfs/volumes/5b5806fd-339444da-f897-003048d70598] ./tools/qemu-i
mg.static info win30.raw
image: win30.raw
file format: raw
virtual size: 32M (33554432 bytes)
disk size: 32M
[root@jdc-user:/vmfs/volumes/5b5806fd-339444da-f897-003048d70598] ./tools/qemu-i
mg.static convert -f raw -O vmdk win30.raw win30.vmdk
[root@jdc-user:/vmfs/volumes/5b5806fd-339444da-f897-003048d70598] ./tools/qemu-i
mg.static info win30.vmdk
image: win30.vmdk
file format: vmdk
virtual size: 32M (33554432 bytes)
disk size: 27M
And it boots!
Transcopied Windows 3.0 VM
So yes, wrapping up you can in fact run stuff on ESXi, copy data, and even convert disk images.
Oh yeah, and so people can deal with my 404 based download system (the password is on the 404 page)
Let the games begin!