I have a machine on my network that is very special. It’s a rather old Quad Xeon, an HP LH4 that I scavenged out of the proverbial dumpster of a buisness that didn’t want it anymore. In fact, they were about to trash six of them.
I decided it was a crime to shitcan such beautiful machines, so with the help of my friend Mike “I mangle french words” Le Blanc, we drove there, armed with a large truck and patience. Carrying them down the three stories with no elevator was an interesting experience. I scavenged lots and lots of interesting hardware there.
But i’m getting ahead of myself here. The point is, lots of screwy things start occuring when you have a Quad SMP machine, such a timers and clock drift. I started having a lot of problems with Apache recently… once in a blue moon, on restart, it would spawn a single process with no PID file, and hang there. Checking out error_log pointed out that apache would apparently hang while generating the Secret seed for Digest authentication (mod_digest). Disabling mod_digest would have worked, but sadly, I use it. This could be found in error_log:
[notice] Digest: generating secret for digest authentication ...
[notice] Digest: done
[notice] Apache configured -- resuming normal operations
[notice] caught SIGTERM, shutting down
[notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec2)
[notice] Digest: generating secret for digest authentication ...
Fortunately, I cobbled up a fix. Updated! Read on for a more elegant fix.
So it hangs while generating the secret for digest authentication. My gut feeling was that it had to do with the system running out of entropy (random data, that is). Probably waits for apr_generate_random_bytes() to complete but that never happens. I immediatly checked out the kernel for available entropy from /dev/random and friends.
# cat /proc/sys/kernel/random/entropy_avail
16
Now, I have no clue why, but entropy ran out (not quite, but close). Probably another whacky quad SMP issue. I’m starting to drive the Gentoo devs and maintainers crazy with my strange hardware.
Note: Before proceeding, please read the updated section below.
So, to remedy it, I fetched and installed rngd from the rng-tools package in Gentoo Portage. Check out your distro’s package repository for it or something similar. This tool allegedy gathers random data from I/O transactions and hardware stuff around. I compiled it, installed it and launched it. The Gentoo way:
# emerge -va rng-tools
# rc-update add rngd default && /etc/init.d/rngd start
Once that was taken care of, I checked the available entropy by querying the kernel once again.
# cat /proc/sys/kernel/random/entropy_avail
6854
That’s much better. To my surpise, Apache finished loading instantly the split second rngd started up. Schweet. Sounds like that fixes the problem. I have no idea why it occurs right now, but this works around it elegantly.
UPDATE! I figured out what the greasy poop was going on. Installing rngd will gather entropy using /dev/urandom if no hardware RNG is found by default. I didn’t pay much attention to that detail at the time. However, you will notice that when installing the Apache Portable Runtime on Gentoo (dev-libs/apr) the use flag urandom is available. Using euse (from gentoolkit) to get info about it returns the following:
valkyrie ~ # euse -i urandom
global use flags (searching: urandom)
************************************************************
no matching entries found
local use flags (searching: urandom)
************************************************************
[- ] urandom (dev-libs/apr):
Use /dev/urandom instead of /dev/random
This will make apache fetch randomness from /dev/urandom directly, therefore giving the same result as rng-tools. The proper solution would be to just re-emerge dev-libs/apr with the urandom use flag enabled.
I tried it and it works beautifully, and looks a lot more elegant to me.
Well, i don’t have any quad xeon machines around but i was having the same problem, apache would take almost 5 minutes to do a restart.
i installed that rngd (i’m on gentoo as well) and bang zoom, working nicely,
thanks,
eddyod
Hey, glad this could be useful to at least someone
I’ve seen a lot of posts on the subject around message boards but with no answer…
Thanks for the comment!
Same here. My apache takes 5 minutes and 30 secondes to restart (gentoo on a Celeron 1,7 with 512 MB DDR) and not it tooks 3s !!!
I searched a fix for a long time and thanks to you, it works fine now
Santé !
I had the same problem just now. I could strace the apache2 pid and it would be stuck reading file handle 23, and “ls -l /proc/[apache2pid]/fd” showed that fd 23 was /dev/random. Adding rngd immediately fixed that problem. Thanks!
Thanks, that seem to solve my hanging problem also.
You’re a god!! I’ve been having this issue for a while now with no luck. I’m also running a Gentoo machine but with very standard hardware. ! proc and 512MB of memory but when I cat’d /proc/sys/kernel/random/entropy_avail I only had 32, after the emerge I was up to 2200
.
Cheers,
Seth
Our single CPU p3 ML370 G2 with 1 gb has the same problem. I also run Gentoo. This solved it. Thanks for the help
Worked perfectly on my UML host. Apache startup time is down from 1.5 minutes to about 1 second. Thanks very much.
Nice Tipp
you made a happy man
Thk’s a lot Apache now startup take less than 1 second on our dual xeon. 
Thanks for the comments everyone
I’m glad I could help!
you’re the man!
damn!
Wow. This was just what I needed, thanx!!!
Thank you!
You have sorted about 8 server of mine out!
Yes you are the MAN!
Freakin’ awesome!
Thank you so much for this! I’m new to Apache, but I’m running LAMPP on a Gentoo box and it has been doing this. It’s running an AMD 64 3200+ with 2GB of RAM, but Apache just refused to start up with any kind of speed. Just like you said… as soon as the emerge finished, Apache kicked off all of my queued requests.
Thank you again! You rock!
Pingback: Kristaps Kaupe
You’re the man.. thank..
I gotta agree, you’re the man! Seems a lot of us have this problem after all? Wonder what it is? Kernel problem? Timer issue on an SMP machine? I’m just glad its working!!
Pingback: raptorized.com » Blog Archive » Update on the Apache-Hangs-On-Digest-Generation topic
Pingback: vbali blogja » Digest: generating secret for digest authentication…
same problem as everyone, glad that you ranked top 5 in google :p
(NB. http://forums.gentoo.org/ currently down for maintenance)
Now it seems that everyone that have this problem is running a Gentoo box (shouldn’t the “urandom” USE flag be turned on by default ?)
Thanks for this, the slow starts were driving me nuts!
Apache slow to start, same issue as everyone else. Re-emerged dev-libs/apr with urandom, works like a charm. Thanks for the writeup.
nice dude! thank you very much for sharing your experience
Hey
I am so glad i found this post
I have re emerged dev-libs/apr with urandom USEflag and now im doing a revdep-rebuild (its needed) and voilá
cat /proc/sys/kernel/random/entropy_avail shows 225 now (very old / slow machine) it had 8 before
Ill have to rephrase that entropy_avail is up to 1027 right now.
Thanx alot
you rock mate… Damm
Wow, what a strange thing to run out of. We ran into this problem; thanks for solving it and posting about it so I wouldn’t have to figure it out for myself.
Crazy, though. I’ve got enough things to worry about without having my computers run out of entropy, of all things…
you saved my weekend (on sunday evening
). all works fine now. THANKS!
OMFG! I want to carry your children!
Seriously though… call me…
Thank you for such a brief, clean and useful explanation.
I have been wondering on this low startup problems from time to time never being able to solve it.
I bow to the master.
Pingback: Reaktorblog
I was having the same problems, this worked for me, without using rng-tools.
Make random a symlink to urandom.
/etc/udev/rules.d/50-udev.rules
KERNEL=="random", NAME="oldrandom", MODE="0666"
KERNEL=="urandom", NAME="%k", symlink+="random", MODE="0444"
then,
$ udevstartHope this helps.
Mmm. I’m not too hot about that idea. I’m not sure of the implications of other software that might be relying on the more expansive random, so I’d rather do this on a case by case basis.
You are a genius. I was scratching my head for hours getting apache to run on a quad xeon I just reclaimed from the garage.
Juan
I hope this will be really helpful, but I would like to use the more elegant solution, and I have no idea what “re-emerge dev-libs/apr with the urandom use flag enabled” means. Do I need to recompile apr? Can I just change something in httpd.conf?
I’m on OSX 10.4 with MacPorts, so that means I REALLY don’t know anything. Any more help would be lovely.
i agree with mr_daemon. i have a small setup so making the symlink hasnt given me any problems thus far, but if you have stuff you know relies on random go with OP
Thank you, man, it’s extrimely useful!
Pingback: Slow Apache Starts on Ubuntu | What a n00b!
Worked like a champ, tx!