Since I was stumped, I asked on darwinos-users, and got a reply from Shantonu Sen (who is smart and very active on the mailing lists).
I'm going to see if I can get a sample of 'securiytd' from when this happens next and that might point to something else that I might need to sample (lookupd, DirectoryServices ...)
Otherwise, I'll have to get the kernel debug kit installed and see if I can set up remote debugging and get it to work.
Fun fun.
Update:
Here's the sample from the stalled securityd -
Analysis of sampling pid 45 every 10.000000 milliseconds Call graph: 979 Thread_100f 979 0x25dcc 979 0x2ee4 979 0x38d4 979 0x8080 979 0x83e8 979 0x9054 979 0x9128 979 0x9acc 979 0x9c50 979 0xa094 979 0x3eac 979 pthread_mutex_lock 979 semaphore_wait_signal_trap 979 semaphore_wait_signal_trap 979 Thread_1103 979 _pthread_body 979 0x16a48 979 0x16ab8 979 0x83e8 979 0x9054 979 0x9128 979 0x136b0 979 0x13868 979 0x13dac 979 0x19354 979 0x19354 979 Thread_1203 979 _pthread_body 979 0x16a48 979 0x16ab8 979 0x83e8 979 0x9054 979 0x9128 979 0x9acc 979 0x9c50 979 0xa094 979 0x3eac 979 pthread_mutex_lock 979 semaphore_wait_signal_trap 979 semaphore_wait_signal_trapTotal number in stack (recursive counted multiple, when >=5):
Sort by top of stack, same collapsed (when >= 5):
semaphore_wait_signal_trap 1958
0x19354 979
And for no real reason at all, I'm starting to wonder if maybe the RAM on that machine is bad/going bad?