Clockfort's Tech Blog

ISBN: 9781593271862
The Book of Xen: A Practical Guide for the System Administrator

Most Xen documentation on the Internet can be a tad focused on the single-computer, single-admin personal-use Xen administration case. This book, thankfully, is not. This is definitely the book to keep on your shelf if you require tips and tricks for setting up your own VPS hosting service, with its world full of  malevolent users needing to be kept in their place, quotas for bandwidth, disk I/O, CPU time, and memory usage, and allowing your users to configure their own instances without you having to step in every time they blow out their /boot partitions.

There are plenty of concepts covered in here for other use-cases (besides just hosting your own VPS provider) as well, including remote-mounting disks over NFS/iSCSI/AoE, migrating live Xen instances across a cluster of servers, and backing up disk images and machine states.

The Book of Xen provides a fair and balanced view of Xen management; that is to say, while it it does talk often about the many distro-specific ways of easily bootstrapping and configuring a new virtual server (like Debian’s debootstrap, Red Hat’s virt-install, or even creating images in Citrix XenServer) it also covers vendor and distro-neutral ways of performing all the required installation and management tasks. The Book of Xen is also fair in that it also goes on to describe the use and configuration of Microsoft, BSD, and Solaris Xen dom0 and domUs as well, with the caveat that support for Xen is weak and upcoming on such platforms as FreeBSD, and that HVM is required for many of these more exotic operating systems like “Microsoft Windows”, as there are no Xen hooks in the Windows kernel.

I particularly liked the Book of Xen’s first chapter, which, unrelated to the rest of the book’s sysadmin-oriented content, was a good overview of the technical underpinnings of the Xen hypervisor platform, and how it interacts with the hardware and virtualized machines from a very low-level perspective. As it is stated later in the book, and something that I agree with, the authors believe that one must know a technology, how it works, and its more basic manual and command line tools, before ever trusting a GUI or web interface to do the same. It will also surely aid debugging later when something goes wrong, as the administrator will have a good idea as to where the problem might lie.

All in all, I liked the book and would recommend it to anyone setting up their own Xen servers, however, I wished it would have had more information about Xen on the Intel Itanium (which is touched upon in the book as being a supported platform, but not talked about further) and I wish it had talked more about some of the topics they covered, like giving users access to their own Xen management consoles, in the common situation where there are many physical machines that a user’s instance could be on, a situation which completely broke their offered solutions for this situation and others.

There is like, zero documentation for doing this on the Internet. No one else wants to install Gentoo in a VServer guest on a Debian Itanium2 host? Lame.

This quick overview contains many things specific to my personal setup; therefore:

• You will want to check your local gentoo mirror to see what the current ia64 tarball is.
• You will want to change the name, hostname, and network address of your vserver
• You can use whatever directories you want; the actual files will go in /var/lib/vservers/<hostname>/ , (and /etc/vservers/<hostname>/ ) and you can delete the stage3 tarball after you’re done.
• So you don’t get confused, “jolt” is the name of my debian host machine, and “coffee” is the vserver guest I am creating.

#Install the Vserver kernel and utilities if you already have not.

jolt:/space/vserver# aptitude install linux-image-vserver-mckinley util-vserver vserver-debiantools

#Reboot to boot into new kernel.

jolt:/space/vserver# reboot

#Are you running the new kernel? It should say “vserver” in there somewhere.

jolt:/space/vserver# uname -a
Linux jolt 2.6.26-2-vserver-mckinley #1 SMP Thu Nov 5 07:44:36 UTC 2009 ia64 GNU/Linux

#Get the latest Stage3 tarball from a mirror close to you.

jolt:/space/vserver/base-images/# wget http://mirrors.rit.edu/gentoo/releases/ia64/current-stage3/stage3-ia64-20091229.tar.bz2

#Build the vserver.  –context is just a unique number you set from 1-49152, non-inclusive. Everything else is relatively self explanatory.

jolt:/space/vserver/base-images/# vserver coffee build –context 1253 –hostname coffee –interface eth2:129.21.50.66/24 –initstyle gentoo -m template — -d gentoo -t /space/vserver/base-images/stage3-ia64-20091229.tar.bz2

jolt:/space/vserver/base-images/# cd /var/lib/vservers/coffee
jolt:/var/lib/vservers/coffee# cp -L /etc/resolv.conf ./etc/resolv.conf
jolt:/var/lib/vservers/coffee# chroot ./ /bin/bash
jolt / # env-update
>>> Regenerating /etc/ld.so.cache…
jolt / # source /etc/profile
jolt / # export PS1=”(chroot) $PS1″
(chroot) jolt / # emerge –sync
(chroot) jolt / # eselect profile list
Available profile symlink targets:
[1] default/linux/ia64/10.0 *
[2] default/linux/ia64/10.0/desktop
[3] default/linux/ia64/10.0/developer
[4] default/linux/ia64/10.0/server
[5] hardened/linux/ia64/10.0
(chroot) jolt / # eselect profile set 4
(chroot) jolt / # cp /usr/share/zoneinfo/US/Eastern /etc/localtime
(chroot) jolt / # nano /etc/make.conf
(chroot) jolt / # cat /etc/make.conf
#Gentoo Itanium VServer make.conf Created: 2010-01-14 by Clockfort (devnull@remove_this_part_for_spam_reasons.csh.rit.edu)
CFLAGS=”-O2 -mtune=mckinley -pipe”
CXXFLAGS=”-O2 -pipe”
# WARNING: Changing your CHOST is not something that should be done lightly.
# Please consult http://www.gentoo.org/doc/en/change-chost.xml before changing.
CHOST=”ia64-unknown-linux-gnu”
MAKEOPTS=”-j5″

# Portage Options
EMERGE_DEFAULT_OPTS=”–ask –verbose –tree –jobs=5″
PORTAGE_ELOG_CLASSES=”info warn error log”
PORTAGE_ELOG_SYSTEM=”save”
AUTOCLEAN=”yes”
FEATURES=”parallel-fetch userfetch collision-protect buildpkg”
PORT_LOGDIR=/var/log/portage

# Portage Sync/Download Locations
GENTOO_MIRRORS=”http://mirrors.rit.edu/gentoo http://www.gtlib.gatech.edu/pub/gentoo ftp://ftp.wallawalla.edu/pub/mirrors/ftp.gentoo.org http://lug.mtu.edu/gentoo/”
SYNC=”rsync://rsync.namerica.gentoo.org/gentoo-portage”

# Use Flags
USE=”$USE symlink” # Kernel
USE=”$USE mmx sse sse2″ # CPU
USE=”$USE pam ssl” # Authentication
USE=”$USE bash-completion” # Completion
USE=”$USE X gtk svg qt3support” # X support
USE=”$USE xulrunner” # native browser integration
USE=”$USE jpeg png” #image support
#FEATURES=”${FEATURES} candy”

(chroot) jolt / # passwd
#Ignore the scanelf “unaligned access” error you get, it won’t hurt anyone, just slow things down a bit.
(chroot) jolt / # emerge syslog-ng
(chroot) jolt / # rc-update add syslog-ng default
#Remove reference to /proc/kmsg
(chroot) jolt / # nano /etc/syslog-ng/syslog-ng.conf
(chroot) jolt / # emerge vixie-cron
(chroot) jolt / # rc-update add vixie-cron default
(chroot) jolt / # exit
jolt:/var/lib/vservers/coffee# vserver coffee start
jolt:/var/lib/vservers/coffee# vserver coffee enter
See also:
http://linux-vserver.org/Installation_on_Debian
http://www.gentoo.org/proj/en/vps/vserver-howto.xml
http://linux-vserver.org/Frequently_Asked_Questions
http://www.gentoo.org/doc/en/handbook/

ISBN: 9781886411951
How Not to Program in C++: 111 Broken Programs and 3 Working Ones, or Why Does 2+2=5986

Do you enjoy puzzles? Do you enjoy debugging other people’s code? If so, you’ll enjoy this book.

If reference materials or traditional educational coding books were like newspapers, then this book would be the crossword puzzle page. Just like any crossword puzzle, some sections are harder than others, and the puzzles in this book are no exception.  If you can’t quite figure out the subtle differences between pointers, addresses, double pointers, etc, then this listing of the plethora of ways that you can get yourself into trouble while programming in C++ will likely prove too difficult to solve at parts. Without a doubt though, anyone who fully knows C++ (and C, which is also covered in this book) and all its intricacies will not find most of the puzzles exceptionally difficult.

That being said, the book still proves a good read for anyone of higher programming skill; any reader will be scratching his or her head to find the truly subtle ways that the all-too-simple-looking sample programs have been broken. Luckily, there are helpful, very optional hints and answers in the back of the book one can read in order to get thinking on the right track without spoiling too much of the fun. Additionally, throughout the book’s puzzles, there are amusing programmer-related debugging horror stories and funny programming quips that provide a brief smile, even when one is wracking one’s brains out to find the misplaced comma, subtly misspelled keyword, missing semicolon, or devious memory misallocation that is keeping the answer of the problem elusive.

ISBN: 9780596101510

This epically-titled O’Reilly book is a well-organized collection of network configuration tips, stories, and common “gotchas”, as told by a self-admitted grouchy old network admin to younger, wet-behind-the-ears network administrators.

The author, in a move uncommon to most networking manuals, just cuts to the chase and says what needs to be said. Everything is told from a Cisco perspective, with Cisco terminology, and the only hint of variation allowed for is the occasional explanation when something is radically (or subtly, in some way that would ruin everything when you least expect it) different between CatOS and IOS. This is undeniably a good thing; it keeps the book short, and realistically, Cisco is one of the forefront leaders in the enterprise network market.

A wide range of topics are talked about – possible problems that you could run into with auto-negotiation on your fast-ethernet network, how to configure spanning tree or etherchannel, getting QoS to work properly, and a whole host of topics one should know when creating one’s own medium-to-large sized network. Even for those that already know how to implement these features, the author explains exactly when someone would want to use these features and how they evolved, and how they ought to be properly used.

I would recommend this book to anyone interested in enterprise networking – ranging from the relatively professionally uninitiated like myself (I only do networking administration for Computer Science House at the Rochester Institute of Technology) to those who are just transitioning from networking classes to actual jobs, who will benefit perhaps the most from the book’s tips and tricks from a person in industry, or even the average middle-aged network administrator, who may find a large portion of the book a snooze, but likely still pick up a trick or two that was previously unknown.

I found an undocumented feature in Cisco IOS 11.2.
The command “show interface description” exists in newer versions, so I typed it in out of habit. Even though it doesn’t tab-complete it, or know what that option to the sh int command is, it still works… Albeit badly, and it gives terribly formatted output (All on one line, no spaces between names, no blank spaces where unlabeled ports are, etc)

It still works though! :-)

Makes me wonder how many undocumented features exist in versions of IOS earlier than they are supposedly “released”.


drpepper#sh ver
Cisco Internetwork Operating System Software
IOS (tm) C2900XL Software (C2900XL-HS-M), Version 11.2(8.10)SA6,
MAINTENANCE INTERIM SOFTWARE
Copyright (c) 1986-1902 by cisco Systems, Inc.
Compiled Fri 15-Feb-02 09:47 by devgoyal
Image text-base: 0x00003000, data-base: 0x0020E278
...(rest of output truncated for brevity)
drpepper#sh int ?
FastEthernet FastEthernet IEEE 802.3
Null Null interface
VLAN Switch VLAN Virtual Interface
accounting Show interface accounting
crb Show interface routing/bridging info
irb Show interface routing/bridging info
link-trap Show interface traps on no link


drpepper#sh int desc
Blacktea LinkCSH UplinkJolt ManagementRESNETRESNETRESNETRESNET


I got very bored in a far-too-easy networking class and ended up coding a very memory efficient version of Conway’s Game of Life.
Of note, it uses bit-twiddling/bit-packing to access individual bits on byte-accessible memory for the gameboard, and it uses a sort of buffer-thing to further reduce memory usage, rather than push temporary changes into an entire other temporary game board. I just made it up as I went along. The idea behind being this memory efficient is so that I can plop this on a microcontroller (possibly CSH’s Big Infosys?) and just let ‘er rip on a LED matrix :-)

Source is up at http://github.com/clockfort/Life/

I decided to redo the NFS setup I previously had on my ol’ Itanium2 machine.

How to test to see which NFS block size is right for your setup:

#Testing write speeds:
mount hostname:/remote_folder/ /mnt/local_folder/ -o rw,wsize=1024
time dd if=/dev/zero of=/mnt/local_folder/write_test bs=16k count=16k
umount /mnt/local_folder/
#Testing read speeds:
mount hostname:/remote_folder/ /mnt/local_folder/ -o ro,rsize=1024
time dd if=/mnt/local_folder/write_test of=/dev/null bs=16k
umount /mnt/local_folder/

Go through and change wsize and rsize to 1024, 2048, 4096, 8192, 16384, 32768. MAKE SURE to unmount after every test, as otherwise caching may cause you to see erroneously high speeds for subsequent tests.

Additional performance boosters:
Use “async” and “noatime” options when mounting your remote NFS directory. Async can be a little dangerous, as the data is not immediately flushed to disk on the server, but if the server is rather reliable, it shouldn’t be a problem. “noatime” prevents the access time for the files accessed over NFS from being updated. POSIX says you should update the access time every time you read a file, but doing a write operation to a disk every time you need to read from a file is time-costly and, frankly, rather silly from a performance perspective.

If you can, allow jumbo packets between the two machines you are using NFS between. Beware, however, that these oversized packets must be supported by the server, the client, and every network switch and router in-between in order for that to function properly. If you know for sure that is the case, then go for it, and you’ll probably see an increase in throughput, as the nfs packets (especially the larger ones) will be fragmented into far fewer IP packets. Also beware that other vendors implementations of NFS differ than Linux’s slightly, for instance Apple OS X clients seem to only be able to connect to NFS servers with “insecure” as an /etc/export option for that particular NFS directory.

Test Machine Specs:
2x 1.5GHz Itanium2 processors
10GB ECC DDR
3x 74GB 15k RPM U320 SCSI drives in RAID5
Gigabit ethernet

I am somewhat confused as to why my own NFS write performance is so appalling, while the read performance approaches the theoretical maximums of a gigabit ethernet connection.
Any advice on improving write performance over NFS would be much appreciated.

(Graph is here if you’re reading this post without images)

I propose that this is the best Ubuntu Netbook Remix Firefox setup ever.
Uses:

• Netbook Remix’s default setup of semi-tabbed gnome windows
• Hide Navigation Bar Firefox Plugin
• Hide Menu Bar Firefox Plugin

Simply binding both of these plugins to activate with the left alt key is working out deliciously well. It’s a small screen, but with some tweaks like these, the lack of real estate isn’t as much of a problem, and I can continue enjoying the crazy portability that is my EeePC 1000HE, and not have to scroll through every webpage to get down to the information I want to read. Plus, it looks pretty sexy with the new minimalist-themed Google :-)

(Screenshot is here if you’re reading this post without images)

Edit: As of 2010, Sun has corrected themselves and now state that OpenBSD is only supported with virtualization processor extensions enabled.

VirtualBox, while I like the interface and it has a few features that are lacking in VMWare and other virtualization products, has many issues.
First off, it doesn’t quite implement the x86 instruction architecture perfectly.
Secondly, this is terrible:
VirtualBox Trouble Ticket #192
VirtualBox Trouble Ticket #639

Basically, Sun has insisted for years that VirtualBox emulates OpenBSD perfectly and yet, it has been completely broken for years and they have not only not fixed the bugs in their own application, but have also not stopped insisting that it works just fine. Theo has posted a rant or two about how Virtualbox’s emulation sucks, and how he has wasted time tracking down non-existant bugs because of users repeatedly filing bug reports for segfaulting software when it is in fact just VirtualBox’s shitty emulation; I’m starting to see why he has this opinion.

Seriously, do not use VirtualBox to emulate OpenBSD (or !any! Unix other than Solaris). Everything is broken. Go try QEMU or VMWare.
But, if you want to try it anyway, here’s the settings you’ll NEED to have. Other settings are left up to your own options.:

General
OS type: OpenBSD
System
VT-x/AMD-V: Enabled*
PAE/NX: Enabled
Nested Paging: Disabled
Display
Video memory: 32MB
Hard Disks
IDE Controller Type: PIIX4
Network
Adapter: Intel PRO/1000 T Server

*obviously this is not possible if your host OS/CPU does not support it. OpenBSD segfaults less with it enabled. It runs(…ish) with it disabled.

tl;dr: VirtualBox is an excellent desktop virtual machine software, especially with its new features, like its 3D-rendering pass through technologies, and many-CPU SMP support. It is a terrible server virtual machine software, and is buggy, cannot run any unix distribution properly, except Sun’s own Solaris 10 kool-aid.

Here’s to you, blog viewers of this last month. I’m fairly sure this isn’t the normal OS distribution… You guys are awesome. :-)

Initially I thought the OpenBSD must have been me, as I run it on my router (yay pf!), but then I realized I don’t even have a browser installed on it… really can’t have been me, I guess. You know what that means? There are at least two users of OpenBSD! Alright! Party!