daemondo is not restarting processes that die
Daemondo is not restarting a process for me that dies. I am sure it is tracking the script's pid correctly (using startupitem.pidfile) and I have setup logging to verify all this. At first I did not have the startupitem pidfile configured properly and daemondo kept starting more instances of the daemon, which was a good sign to me that it was looking for the pidfile, using it, and might in fact restart a process that died. But after fixing the startupitem.pidfile properly, if I kill the script (to simulate the script dying abnormally) daemondo never restarts it. Or is kiling it using the Unix kill command a valid test? Is there a bug in daemondo? It isn't behaving as the main.c comments (or our documentation) say it should. I have never explicitly tested it before, and I'm not sure anyone else has either so it seems like a bug. Mark
Hi Mark, I'll have to look at the Daemondo code again. I'm not sure whether it will get a child death notice for a script that it didn't start directly, and this may be the problem in the case that you describe. Looking... James On Oct 19, 2007, at 2:52 PM, markd@macports.org wrote:
Daemondo is not restarting a process for me that dies. I am sure it is tracking the script's pid correctly (using startupitem.pidfile) and I have setup logging to verify all this. At first I did not have the startupitem pidfile configured properly and daemondo kept starting more instances of the daemon, which was a good sign to me that it was looking for the pidfile, using it, and might in fact restart a process that died.
But after fixing the startupitem.pidfile properly, if I kill the script (to simulate the script dying abnormally) daemondo never restarts it. Or is kiling it using the Unix kill command a valid test? Is there a bug in daemondo? It isn't behaving as the main.c comments (or our documentation) say it should. I have never explicitly tested it before, and I'm not sure anyone else has either so it seems like a bug.
Mark
James Berry <jberry@macports.org> writes:
I'll have to look at the Daemondo code again. I'm not sure whether it will get a child death notice for a script that it didn't start directly, and this may be the problem in the case that you describe.
Looking...
Thanks James, I appreciate it. If it can't do that the docs will need updated because I thought it would. But I guess I was supposing that daemondo monitored the process by looking at the pidfile; if it has to passively receive a death notice I can see how that might not work. Would actively monitoring daemons (for non-executable startupitems) be too expensive? It seems like it has most of the logic to do it based on the way it started multiple processes when the pidfile wasn't located correctly. It seems like having it check once in awhile after it is running wouldn't be to much of a stretch. But then you know what they say, the less you know about something, the easier it seems. :) Mark
Mark, and others: In r30313 I've modified daemondo to use kqueue/kevent to watch for the death of the targeted process. This should catch the situation where we're watching for a grandchild process, for which we don't receive child death notices. Let me know if this helps. James Checkin notes: Improve Daemondo to use kqueue/kevent to watch for the death of the target process. Previously Daemondo relied on child death signals, which are given only for direct offspring, and not grandchildren, etc. The use of the kevents now gives notice when the targetted process is not direct offspring. Note: these changes may introduce compatibility issues for Panther, or even Tiger, since I have not yet tested in those environments. They may also introduce compilation problems on platforms for which there is no kqueue support at all. Bug reports are welcome. On Oct 22, 2007, at 7:36 PM, markd@macports.org wrote:
James Berry <jberry@macports.org> writes:
I'll have to look at the Daemondo code again. I'm not sure whether it will get a child death notice for a script that it didn't start directly, and this may be the problem in the case that you describe.
Looking...
Thanks James, I appreciate it. If it can't do that the docs will need updated because I thought it would. But I guess I was supposing that daemondo monitored the process by looking at the pidfile; if it has to passively receive a death notice I can see how that might not work. Would actively monitoring daemons (for non-executable startupitems) be too expensive? It seems like it has most of the logic to do it based on the way it started multiple processes when the pidfile wasn't located correctly. It seems like having it check once in awhile after it is running wouldn't be to much of a stretch. But then you know what they say, the less you know about something, the easier it seems. :)
Mark
James Berry <jberry@macports.org> writes:
Mark, and others:
In r30313 I've modified daemondo to use kqueue/kevent to watch for the death of the targeted process. This should catch the situation where we're watching for a grandchild process, for which we don't receive child death notices.
Let me know if this helps.
James
James, It works! I tested it on 10.4. When I kill the process it starts again immediately. Thanks! It helps me out a lot, and it will others I'm sure as well. If it weren't hard, perhaps it would be valuable to have the notification of process death reported in the startupitem.logfile when present. Or at least if you agree with that. Right now the only indication a process died is by noting the "starting process" message in startupitem.logfile. Not a big deal though. Thanks again for your hard work! Mark
On Oct 24, 2007, at 12:30 PM, markd@macports.org wrote:
James Berry <jberry@macports.org> writes:
Mark, and others:
In r30313 I've modified daemondo to use kqueue/kevent to watch for the death of the targeted process. This should catch the situation where we're watching for a grandchild process, for which we don't receive child death notices.
Let me know if this helps.
James
James,
It works! I tested it on 10.4. When I kill the process it starts again immediately. Thanks! It helps me out a lot, and it will others I'm sure as well.
Great to hear.
If it weren't hard, perhaps it would be valuable to have the notification of process death reported in the startupitem.logfile when present. Or at least if you agree with that. Right now the only indication a process died is by noting the "starting process" message in startupitem.logfile. Not a big deal though.
By default, logs are requested at level 1 if you ask for the startupitem.logfile. So I've added some additional logging at level 1 that should make clear when the process dies, as well as a couple of other cases (receipt of SIGHUP and SIGTERM). Let me know if that helps: r30335. James
James Berry <jberry@macports.org> writes:
I've added some additional logging at level 1 that should make clear when the process dies, as well as a couple of other cases (receipt of SIGHUP and SIGTERM).
Let me know if that helps: r30335.
It looks great. Here is the logging that now happens when I kill my perl script daemon: 2007-10-25 11:44:51 nedimonitor: Starting process 2007-10-25 11:44:51 nedimonitor: Target process id is 18910 2007-10-25 11:45:21 nedimonitor: Target process pid 18910 has died. 2007-10-25 11:45:21 nedimonitor: Starting process 2007-10-25 11:45:21 nedimonitor: Target process id is 18925 I can't thank you enough for the work on daemondo. For a lightweight network monitoring script, it is the difference between something I can rely on and something I can't. Mark
participants (2)
-
James Berry
-
markd@macports.org