Mitigating factors are that the camera is really awesome, and also that I got root on it, voiding my warranty a mere fifteen minutes after unboxing it.

Being the kind of guy to plan things through before taking action (usually), I had previously made a test set-up with all the configs in vmware on my local workstation at home, before heading over and installing them. So I pop open my laptop, fire up the Cisco VPN Client, connect over there, leapfrog some routers in between and land on my workstation. Turns out I hadn’t left the vm running, so I can’t access it from the shell by SSH’ing in.

“Hey, no problem!” I thought. “I’ll just mount the disk images with vmware-mount.pl, fetch the configuration files, send them over with a convulted mess of ssh-within-ssh-within-ssh and unix pipes to my laptop! Piece of cake!”

But then something hits me. The Virtual Disk Image (vmdk) file had LVM Volumes as partitions. Which I can’t directly mount from my Ubuntu workstation. Usually, you can get away with issuing the following:

$ sudo /opt/vmware/bin/vmware-mount.pl /path/to/disk-image.vmdk patition_number -t ext3 /mnt/mountpoint

But to much of my dismay, partition 2 on this particular disk image is an LVM Volume, so it can’t be directly mounted. I has to be mapped and a bunch things has to be done before I can get to the data. I’m not even sure I have LVM support on the Ubuntu machine at the moment.

Curses. Unless I fuck around with it to make it work. Which I did. Read on for the details.

So, I light up a cigarette and get to work. I first issue the command described above, regardless of the outcome. vmware-mount.pl works by firing up part of vmware, accessing the disk, and then feeds the data through a Network Block Device (/dev/nb0), which, in turn, is then mounted wherever you specify.

The output is optimistic.

--------------------------------------------
VMware for Linux - Virtual Hard Disk Mounter
Version: 1.0 build-45731
Copyright 1998 VMware, Inc.  All rights reserved. -- VMware Confidential
--------------------------------------------

It has been reported that this program does not work correctly with 2.4+ Linux 
kernels in some cases, and you are currently running such a kernel. Do you 
really want to continue? [N] y

No Network Block Device detected.

There is no Network Block Device defined on this machine. This script is about 
to create the /dev/nb0 Network Block Device. Continue? [Y] 

Creating the /dev/nb0 Network Block Device

No Network Block Device driver detected.

Trying to load the Network Block Device driver kernel module... Success.

Client: The partition is now mapped on the /dev/nb0 Network Block Device.
mount: unknown filesystem type 'LVM2_member'

If you know the filesystem of the partition you want to mount, you can provide 
it using the -t command line option. Since you haven't done so, this script is 
going to try to determine the filesystem of the partition based on the partition
type and id.

Unable to retrieve the filesystem of the partition (the partition type is BIOS 
and the partition Id is 8E). Please file an incident with VMware at 
http://www.vmware.com/forms/Incident_Login.cfm by copying this error message.

So here I think, “I’m pretty fucked. I’ll have to hack vmware-mount.pl, and god knows vmware’s Perl scripts are complex.”

So, out of the blue, I try again. But then I notice the vmware disk image is locked. And /dev/nb0 is still present. So my vmware Disk Image is still attached to the network block device! Great!

So, I ignore the error messages, and install LVM support on my ubuntu box.

$ sudo apt-get install lvm2

Note: On Ubuntu Feisty, according to bug #96802, the lvm package has some packaging errors. You'll end up getting the error message "no program found for your current version of LVM" if you try to issue any of the LVM management commands. Long story short, issue the following to fix it:

$ sudo ln -s /lib/lvm-200 /lib/lvm-0

Next step, detect and mount the LVM volume you just added

Step 1: Load LVM kernel modules:

$ sudo modprobe dm-mod

Step 2: Detect new LVM Physical Volumes

$ sudo vgscan

It should appear as /dev/nb0. You may safely ignore errors about other devices. The last line should be: Found volume group "VolGroup00" using metadata type lvm2, or something similar.

Step 3: Activate the newly found vgroup

$ sudo vgchange -ay VolGroup00

(Substitute "VolGroup00" appropriately if you named your Volume Group differently)

Step 4: Display available volumes and mount them as necessary

$ sudo lvdisplay

Step 5: Mount logical Volume needed:

$ sudo mount -t ext3 /dev/VolGroup00/LogVol00 /mnt/lvm1 -o ro

Step 6: SUCCESS! (Profit)

I can now access my RHEL5 Disk from my Ubuntu machine. Now, time to go back to the convulted mess of ssh-within-ssh-within-ssh-and-unix-pipes, but that's a story for another day.

Also, an important note, I have no idea how to unmount that volume. Well, I know how to unmount the LVM volume, but I have no idea how to get rid of /dev/nb0 without a reboot. Also, do note that this may make your machine majorly unstable. I had to call a friend to walk over to my place and reboot my workstation because it hung while reading files. But in the end, it worked.

I hope this helps.

So, right now, I’m migrating my qmail, vpopmail, courier, clamav and friends set-up from Gentoo to Debian, on a new virtual machine. I followed the basic steps from here:

http://wiki.debian.iuculano.it/quick_howto

This gent was nice enough to provide the required qmail packages and patches. Only problem came when I tried to make courier work with vpopmail — turns out the official debian courier-imap packages don’t support vchkpw, which is the authentication mechanism for vpopmail. Fortunately, I managed to hack it in an almost clean way, that doesn’t involve building it yourself from tarball.

Read on for the jazz.

So, first of all, fetch the courier-authlib source. You should first make sure that your /etc/apt/sources.list file contains the “deb-src” lines for everything. apt-get update if necessary.

So, step one, create a folder to hold the sources and debian packages, and cd into it.

mkdir -p /usr/src/deb-source; cd  /usr/src/deb-source

Step two, fetch the sources.

apt-get source courier-authlib

Once that is done, you must install the tools and libraries courier might need to be built. This sounds scary, but is actually a breeze under Debian.

apt-get build-dep courier-authlib

This should bring in a few things like build-essential, if you don’t have a compiler on the machine yet.

Next, cd’ into the source folder, and into the debian package files.

cd courier-authlib-0.58/debian

Now you’re sitting where all the package specifications are configured, and the sweet thing is, you can hack them to your heart’s content. First of all, we need to edit the file named control.

vi control

Add the following at the end of the file to create a description for vchkpw.

Package: courier-authlib-vchkpw
Architecture: any
Depends: ${shlibs:Depends}, courier-authlib (>= ${RELUP})
Description: External authentication support for Vpopmail.
 This package contains vpopmail support for courier-authlib.

Next, we must create a list of files for the package.

echo "/usr/lib/courier-authlib/libauthvchkpw.so*" > courier-authlib-vchkpw.files

Failure to do this could lead to the following message during build:

/usr/src/deb-source/courier-authlib-0.58/debian/tmp/usr/lib/courier-authlib/libauthvchkpw.so.0.0.0 /usr/src/deb-source/courier-authlib-0.58/debian/tmp/usr/lib/courier-authlib/libauthvchkpw.so.0 /usr/src/deb-source/courier-authlib-0.58/debian/tmp/usr/lib/courier-authlib/libauthvchkpw.so File(s) found not belonging to any package, please contact maintainer make: *** [install] Error 1

Next step, altering the build parameters to include vpopmail support. That’s fairly easy. Just edit the file rules.

vi rules

Scroll down to the part where COMMON_CONFOPTS is defined, near line 35. Change the line where it says:

–without-authvchkpw \

so that it now reads:

–with-authvchkpw \

Once that is done, save the file, and you’re now ready to build the new and improved package:

# cd ..
# dpkg-buildpackage

Compiling authvchkpw.c
authvchkpw.c: In function 'callback_vchkpw':
authvchkpw.c:46: warning: value computed is not used
authvchkpw.c: In function 'auth_vchkpw_changepass':
authvchkpw.c:142: warning: passing argument 1 of 'parse_email' discards qualifiers from pointer target type
Compiling authvchkpwlib.c

This confirmed that support was indeed being included this time.

Once the build is done, simply install the required package.

# cd ..
# ls -lah *.deb
courier-authdaemon_0.58-4_amd64.deb     courier-authlib-pipe_0.58-4_amd64.deb
courier-authlib_0.58-4_amd64.deb        courier-authlib-postgresql_0.58-4_amd64.deb
courier-authlib-dev_0.58-4_amd64.deb    courier-authlib-userdb_0.58-4_amd64.deb
courier-authlib-ldap_0.58-4_amd64.deb   courier-authlib-vchkpw_0.58-4_amd64.deb
courier-authlib-mysql_0.58-4_amd64.deb
# dpkg -i courier-authlib-vchkpw_0.58-4_amd64.deb
Selecting previously deselected package courier-authlib-vchkpw.
(Reading database ...
27234 files and directories currently installed.)
Unpacking courier-authlib-vchkpw (from courier-authlib-vchkpw_0.58-4_amd64.deb) ...
Setting up courier-authlib-vchkpw (0.58-4) ...
#

And we're done.

Now all that is left is just to edit /etc/courier/authdaemonrc where it specifies the authentication modules to include our newly created one:

authmodulelist="authvchkpw"

And voilà. That wasn't so hard, was it?

I hope someone finds the following guide useful

Check it out.

/usr/libexec/gcc/darwin/ppc/3.3/cc1plus
/usr/libexec/gcc/darwin/ppc/3.3-fast/cc1plus
/usr/libexec/gcc/i686-apple-darwin8/4.0.1/cc1plus
/usr/libexec/gcc/powerpc-apple-darwin8/4.0.1/cc1plus

That explains why there’s a crapload of things I can’t build on my intel Mac.

C compiler cannot create executables, yeah well, that makes sense. I guess I could possibly cross compile, but that’s too much hassle. I’ll just have to wait for someone to update the ports.

So if you’re wondering why that occurs to you, it’s because there’s no gcc 3.3 for x86 on OS X 10.4. You have to use gcc 4.

I have no idea why, but Courier, which is the mail server suite I use, in conjunction with DJ Bernstein’s qmail and vpopmail, all this lovely stuff keeps breaking all the time on Gentoo. I’ve had courier hand mask’ed since they kept screwing up.

Now, the lastest baselayout prevents authdaemond from starting, and leaves no evidence in the logs. The Gentoo bugzilla is confusing, they say it’s Courier’s fault, and that they need maintainers, and that uh.. I don’t know, I’m not sure. They sound like they expect the problem to fix itself, or that it will magically occur at some time.

Gentoo is definitely not something you should run on a production server, sometimes.

Update: Well, updating to the lastest courier and removing /etc/init.d/authdaemond seems to have fixed it. The ones who are having problems are actually those who are running the “unstable” courier, with ~x86 flags. I wish you guys luck.

I ntp’ed it:

nailhead # ntpdate time.apple.com
15 Aug 23:54:57 ntpdate[22961]: step time server 17.254.0.27 offset -131984.081641 sec

131984.081641 seconds in the future, huh? Guess the flux capacitor still has hiccups there. I’m working on a fix, but all I can think of requires a reboot, which I’m not going to do right now. Stay tuned.

I decided it was a crime to shitcan such beautiful machines, so with the help of my friend Mike “I mangle french words” Le Blanc, we drove there, armed with a large truck and patience. Carrying them down the three stories with no elevator was an interesting experience. I scavenged lots and lots of interesting hardware there.

But i’m getting ahead of myself here. The point is, lots of screwy things start occuring when you have a Quad SMP machine, such a timers and clock drift. I started having a lot of problems with Apache recently… once in a blue moon, on restart, it would spawn a single process with no PID file, and hang there. Checking out error_log pointed out that apache would apparently hang while generating the Secret seed for Digest authentication (mod_digest). Disabling mod_digest would have worked, but sadly, I use it. This could be found in error_log:

[notice] Digest: generating secret for digest authentication ...
[notice] Digest: done
[notice] Apache configured -- resuming normal operations
[notice] caught SIGTERM, shutting down
[notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec2)
[notice] Digest: generating secret for digest authentication ...

Fortunately, I cobbled up a fix. Updated! Read on for a more elegant fix.

So it hangs while generating the secret for digest authentication. My gut feeling was that it had to do with the system running out of entropy (random data, that is). Probably waits for apr_generate_random_bytes() to complete but that never happens. I immediatly checked out the kernel for available entropy from /dev/random and friends.

# cat /proc/sys/kernel/random/entropy_avail
16

Now, I have no clue why, but entropy ran out (not quite, but close). Probably another whacky quad SMP issue. I’m starting to drive the Gentoo devs and maintainers crazy with my strange hardware.

Note: Before proceeding, please read the updated section below.

So, to remedy it, I fetched and installed rngd from the rng-tools package in Gentoo Portage. Check out your distro’s package repository for it or something similar. This tool allegedy gathers random data from I/O transactions and hardware stuff around. I compiled it, installed it and launched it. The Gentoo way:

# emerge -va rng-tools
# rc-update add rngd default && /etc/init.d/rngd start

Once that was taken care of, I checked the available entropy by querying the kernel once again.

# cat /proc/sys/kernel/random/entropy_avail
6854

That’s much better. To my surpise, Apache finished loading instantly the split second rngd started up. Schweet. Sounds like that fixes the problem. I have no idea why it occurs right now, but this works around it elegantly.

UPDATE! I figured out what the greasy poop was going on. Installing rngd will gather entropy using /dev/urandom if no hardware RNG is found by default. I didn’t pay much attention to that detail at the time. However, you will notice that when installing the Apache Portable Runtime on Gentoo (dev-libs/apr) the use flag urandom is available. Using euse (from gentoolkit) to get info about it returns the following:

valkyrie ~ # euse -i urandom
global use flags (searching: urandom)
************************************************************
no matching entries found

local use flags (searching: urandom)
************************************************************
[-    ] urandom (dev-libs/apr):
Use /dev/urandom instead of /dev/random

This will make apache fetch randomness from /dev/urandom directly, therefore giving the same result as rng-tools. The proper solution would be to just re-emerge dev-libs/apr with the urandom use flag enabled.

I tried it and it works beautifully, and looks a lot more elegant to me.

[ebuild  N    ] virtual/perl-Test-Simple-0.62  0 kB
[ebuild  N    ] virtual/perl-MIME-Base64-3.05  0 kB
[ebuild  N    ] virtual/perl-digest-base-1.13  0 kB
[ebuild  N    ] virtual/perl-Storable-2.15  0 kB
[ebuild  N    ] virtual/perl-net-ping-2.31  0 kB
[ebuild  N    ] virtual/perl-Digest-MD5-2.33  0 kB
[ebuild  N    ] virtual/perl-libnet-1.19  0 kB
[ebuild  N    ] virtual/perl-PodParser-1.30  0 kB
[ebuild  N    ] virtual/perl-Test-Harness-2.42  0 kB
[ebuild  N    ] virtual/perl-DB_File-1.814  0 kB
[ebuild  N    ] virtual/perl-Time-Local-1.11  0 kB 

Yeah. Virtuals, which means nothing is emerged at all, my world file is just changed. This is a refreshing change from the many baselayout updates during the past weeks which caused me to overwrite part of my customized /etc/profile on the server out of lazyness, and read diffs until ocular bleeding occured.

So amidst all of these happenings, I took the time to upgrade my web servers to reflect the recent PHP portage changes happening in Gentoo lately. It all went well, until I had to re-merge the 30 or so PEAR packages. One of them wanted to pull back the old dev-php/php-4* packages, creating a blocker.

Turns out three PEAR packages are now depreciated, according to the php managers for Gentoo.

CHTEKK: just emerge the new PEAR-PEAR, it includes XML_RPC, 
Archive_Tar and Console_Getopt PEAR packages
jakub: we should mention it in the guide
CHTEKK: yeah
mr_daemon: Okay... I have something like 30 PEAR packages... 
how could I find out which are depreciated?
CHTEKK: only those three
CHTEKK: all the others remained, and some (very few) changed category 
and are now in dev-php4/ or dev-php5/

So basically, the guide should be updated to reflect this fact. If you’re having problems, just discards PEAR-XML_RPC, PEAR-Archive_Tar and PEAR-Console_Getopt. They are all included in PEAR-PEAR.