« And now for something completely different
Back to the Future Part 1 »

Apache hangs on Digest Secret generation

I have a machine on my network that is very special. It’s a rather old Quad Xeon, an HP LH4 that I scavenged out of the proverbial dumpster of a buisness that didn’t want it anymore. In fact, they were about to trash six of them.

I decided it was a crime to shitcan such beautiful machines, so with the help of my friend Mike “I mangle french words” Le Blanc, we drove there, armed with a large truck and patience. Carrying them down the three stories with no elevator was an interesting experience. I scavenged lots and lots of interesting hardware there.

But i’m getting ahead of myself here. The point is, lots of screwy things start occuring when you have a Quad SMP machine, such a timers and clock drift. I started having a lot of problems with Apache recently… once in a blue moon, on restart, it would spawn a single process with no PID file, and hang there. Checking out error_log pointed out that apache would apparently hang while generating the Secret seed for Digest authentication (mod_digest). Disabling mod_digest would have worked, but sadly, I use it. This could be found in error_log:


[notice] Digest: generating secret for digest authentication …
[notice] Digest: done
[notice] Apache configured — resuming normal operations
[notice] caught SIGTERM, shutting down
[notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec2)
[notice] Digest: generating secret for digest authentication …

Fortunately, I cobbled up a fix. Updated! Read on for a more elegant fix.

So it hangs while generating the secret for digest authentication. My gut feeling was that it had to do with the system running out of entropy (random data, that is). Probably waits for apr_generate_random_bytes() to complete but that never happens. I immediatly checked out the kernel for available entropy from /dev/random and friends.


# cat /proc/sys/kernel/random/entropy_avail
16

Now, I have no clue why, but entropy ran out (not quite, but close). Probably another whacky quad SMP issue. I’m starting to drive the Gentoo devs and maintainers crazy with my strange hardware.

Note: Before proceeding, please read the updated section below.

So, to remedy it, I fetched and installed rngd from the rng-tools package in Gentoo Portage. Check out your distro’s package repository for it or something similar. This tool allegedy gathers random data from I/O transactions and hardware stuff around. I compiled it, installed it and launched it. The Gentoo way:


# emerge -va rng-tools
# rc-update add rngd default && /etc/init.d/rngd start

Once that was taken care of, I checked the available entropy by querying the kernel once again.


# cat /proc/sys/kernel/random/entropy_avail
6854

That’s much better. To my surpise, Apache finished loading instantly the split second rngd started up. Schweet. Sounds like that fixes the problem. I have no idea why it occurs right now, but this works around it elegantly.

UPDATE! I figured out what the greasy poop was going on. Installing rngd will gather entropy using /dev/urandom if no hardware RNG is found by default. I didn’t pay much attention to that detail at the time. However, you will notice that when installing the Apache Portable Runtime on Gentoo (dev-libs/apr) the use flag urandom is available. Using euse (from gentoolkit) to get info about it returns the following:


valkyrie ~ # euse -i urandom
global use flags (searching: urandom)
************************************************************
no matching entries found

local use flags (searching: urandom)
************************************************************
[-    ] urandom (dev-libs/apr):
Use /dev/urandom instead of /dev/random

This will make apache fetch randomness from /dev/urandom directly, therefore giving the same result as rng-tools. The proper solution would be to just re-emerge dev-libs/apr with the urandom use flag enabled.

I tried it and it works beautifully, and looks a lot more elegant to me.

This entry was posted on Friday, August 11th, 2006 at 11:24 am and is filed under Apache, Computers, English, Tutorials/How-To, Unix/Linux. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.

38 Responses to “Apache hangs on Digest Secret generation”

  1. eddyod says:
    August 14th, 2006 at 1:30 pm

    Well, i don’t have any quad xeon machines around but i was having the same problem, apache would take almost 5 minutes to do a restart.
    i installed that rngd (i’m on gentoo as well) and bang zoom, working nicely,
    thanks,

    eddyod

  2. mr_daemon says:
    August 14th, 2006 at 2:10 pm

    Hey, glad this could be useful to at least someone :)

    I’ve seen a lot of posts on the subject around message boards but with no answer…

    Thanks for the comment!

  3. Shadow aok says:
    August 27th, 2006 at 12:01 pm

    Same here. My apache takes 5 minutes and 30 secondes to restart (gentoo on a Celeron 1,7 with 512 MB DDR) and not it tooks 3s !!!
    I searched a fix for a long time and thanks to you, it works fine now :) Santé !

  4. Paul says:
    September 8th, 2006 at 12:24 am

    I had the same problem just now. I could strace the apache2 pid and it would be stuck reading file handle 23, and “ls -l /proc/[apache2pid]/fd” showed that fd 23 was /dev/random. Adding rngd immediately fixed that problem. Thanks!

  5. Peter says:
    October 9th, 2006 at 7:14 pm

    Thanks, that seem to solve my hanging problem also.

  6. Seth says:
    October 11th, 2006 at 10:35 pm

    You’re a god!! I’ve been having this issue for a while now with no luck. I’m also running a Gentoo machine but with very standard hardware. ! proc and 512MB of memory but when I cat’d /proc/sys/kernel/random/entropy_avail I only had 32, after the emerge I was up to 2200 :).

    Cheers,

    Seth

  7. Bastian says:
    October 16th, 2006 at 5:42 am

    Our single CPU p3 ML370 G2 with 1 gb has the same problem. I also run Gentoo. This solved it. Thanks for the help :-)

  8. Graham King says:
    October 17th, 2006 at 2:24 pm

    Worked perfectly on my UML host. Apache startup time is down from 1.5 minutes to about 1 second. Thanks very much.

  9. adlerweb says:
    November 28th, 2006 at 4:24 am

    Nice Tipp

  10. the says:
    November 29th, 2006 at 11:55 am

    you made a happy man :D Thk’s a lot Apache now startup take less than 1 second on our dual xeon. :D

  11. mr_daemon says:
    November 29th, 2006 at 12:00 pm

    Thanks for the comments everyone :) I’m glad I could help!

  12. der_flo says:
    December 29th, 2006 at 9:36 am

    you’re the man!
    damn!

  13. Jim says:
    January 3rd, 2007 at 2:42 am

    Wow. This was just what I needed, thanx!!! :)

  14. Brett says:
    January 4th, 2007 at 6:17 am

    Thank you!

    You have sorted about 8 server of mine out!

    Yes you are the MAN!

  15. as0t0 says:
    January 6th, 2007 at 3:19 am

    Freakin’ awesome!
    Thank you so much for this! I’m new to Apache, but I’m running LAMPP on a Gentoo box and it has been doing this. It’s running an AMD 64 3200+ with 2GB of RAM, but Apache just refused to start up with any kind of speed. Just like you said… as soon as the emerge finished, Apache kicked off all of my queued requests.

    Thank you again! You rock!

  16. Kristaps Kaupe says:
    January 14th, 2007 at 2:35 pm

    Blogiem.lv 2 serveri…

    Taks, beidzot ir palaists Blogiem.lv atsevišķs MySQL serveris, bet uz vecā tīmekļa servera paralēli griežas MySQL slave, kurš gan šobrīd kalpo nu jau izmanto.

    Turpinājumā pāris publiskas piezīmes ar šovakar piefiksētām lietām……

  17. Peter says:
    January 16th, 2007 at 9:44 pm

    You’re the man.. thank..

  18. Mike Rosile says:
    January 21st, 2007 at 6:54 pm

    I gotta agree, you’re the man! Seems a lot of us have this problem after all? Wonder what it is? Kernel problem? Timer issue on an SMP machine? I’m just glad its working!!

  19. raptorized.com » Blog Archive » Update on the Apache-Hangs-On-Digest-Generation topic says:
    February 14th, 2007 at 12:39 am

    [...] Check it out. [...]

  20. vbali blogja »  Digest: generating secret for digest authentication… says:
    February 16th, 2007 at 8:52 am

    [...] Egészen mostanáig, mikoris belefutottam ugyanebbe a jelenségbe egy sokkal kisebb teljesítményű gépen, ahol a titok generálása néha még fél órába is beletelt. Ezt már nem tartottam megengedhetőnek így ismét keresgéltem a megoldás után, így találtam rá erre a postra: Apache hangs on Digest Secret generation. Örömmel nyugtáztam a hozzászólásokat olvasva, hogy nem csak nekem okozott fejfájást a hiba elhárítása. [...]

  21. Ash says:
    February 22nd, 2007 at 10:53 am

    same problem as everyone, glad that you ranked top 5 in google :p
    (NB. http://forums.gentoo.org/ currently down for maintenance)

    Now it seems that everyone that have this problem is running a Gentoo box (shouldn’t the “urandom” USE flag be turned on by default ?)

  22. Scott says:
    February 25th, 2007 at 2:04 pm

    Thanks for this, the slow starts were driving me nuts!

  23. Alan J Castonguay says:
    March 4th, 2007 at 7:17 pm

    Apache slow to start, same issue as everyone else. Re-emerged dev-libs/apr with urandom, works like a charm. Thanks for the writeup.

  24. dn says:
    March 12th, 2007 at 10:27 am

    nice dude! thank you very much for sharing your experience ;-)

  25. Aron Kovacs says:
    March 13th, 2007 at 9:47 am

    Hey
    I am so glad i found this post :) I have re emerged dev-libs/apr with urandom USEflag and now im doing a revdep-rebuild (its needed) and voilá
    cat /proc/sys/kernel/random/entropy_avail shows 225 now (very old / slow machine) it had 8 before :)

  26. Aron Kovacs says:
    March 13th, 2007 at 10:04 am

    Ill have to rephrase that entropy_avail is up to 1027 right now.
    Thanx alot

  27. Deep says:
    April 15th, 2007 at 11:59 pm

    you rock mate… Damm

  28. Moonwick says:
    May 23rd, 2007 at 11:09 am

    Wow, what a strange thing to run out of. We ran into this problem; thanks for solving it and posting about it so I wouldn’t have to figure it out for myself.

    Crazy, though. I’ve got enough things to worry about without having my computers run out of entropy, of all things…

  29. rsvanda says:
    August 5th, 2007 at 1:40 pm

    you saved my weekend (on sunday evening :-) ). all works fine now. THANKS!

  30. RL2000 says:
    September 27th, 2007 at 5:52 am

    OMFG! I want to carry your children!

    Seriously though… call me…

  31. Alberto says:
    September 27th, 2007 at 6:18 am

    Thank you for such a brief, clean and useful explanation.
    I have been wondering on this low startup problems from time to time never being able to solve it.

    I bow to the master.

  32. Reaktorblog says:
    October 14th, 2007 at 4:42 am

    Apache Digest generation on Gentoo…

    How to get it running ?
    ……

  33. Ryan Yoosefi says:
    October 18th, 2007 at 10:23 pm

    I was having the same problems, this worked for me, without using rng-tools.

    Make random a symlink to urandom.

    /etc/udev/rules.d/50-udev.rules

    KERNEL=="random", NAME="oldrandom", MODE="0666"
    KERNEL=="urandom", NAME="%k", symlink+="random", MODE="0444"

    then,
    $ udevstart

    Hope this helps.

  34. mr_daemon says:
    November 5th, 2007 at 11:58 am

    Make random a symlink to urandom.

    Mmm. I’m not too hot about that idea. I’m not sure of the implications of other software that might be relying on the more expansive random, so I’d rather do this on a case by case basis.

  35. Juan says:
    December 3rd, 2007 at 11:51 pm

    You are a genius. I was scratching my head for hours getting apache to run on a quad xeon I just reclaimed from the garage.

    Juan

  36. PP says:
    December 6th, 2007 at 1:28 pm

    I hope this will be really helpful, but I would like to use the more elegant solution, and I have no idea what “re-emerge dev-libs/apr with the urandom use flag enabled” means. Do I need to recompile apr? Can I just change something in httpd.conf?

    I’m on OSX 10.4 with MacPorts, so that means I REALLY don’t know anything. Any more help would be lovely.

  37. ikratus says:
    December 30th, 2007 at 11:28 am

    i agree with mr_daemon. i have a small setup so making the symlink hasnt given me any problems thus far, but if you have stuff you know relies on random go with OP

  38. Dyr says:
    June 19th, 2008 at 7:54 am

    Thank you, man, it’s extrimely useful!

Leave a Reply

Line and paragraph breaks automatic.
XHTML allowed: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>