On 01 Sep, 2007, at 4:35, Ryan Schmidt wrote:

I have this problem: the lighttpd web server on my MacBook Pro unexpectedly dies rather frequently (much more frequently than it did on my PowerBook). When this happens, the daemondo process is still there, and I have to use launchctl to unload and then load the lighttpd plist to get the server running again.

The point of launchd is that it notices when processes unexpectedly quit, and restarts them automatically so close to no downtime is observed. But launchd is monitoring daemondo, which does not exit, so launchd does not restart anything. And daemondo doesn't seem to have the same process-death-noticing feature so daemondo is not automatically restarting the failed lighttpd.

Agreed. I try to avoid daemondo when possible. When it isn't possible, daemondo is an indispensable tool, but it turns out that for most cases, it's possible to avoid it. I've written many launchd scripts on my own, several of which I've put online [1]. It's possible to make launchd scripts similar to these in MacPorts via startupitem.executable, but currently, the startupitem.* commands in MacPorts do not support setting the "User" key. For lighttpd, that's not necessary; it starts as root so it can listen on the privileged port 80, then drops privileges. For mysql, though, it's pretty important. There's also no support for inetd, which I use for rsyncd and cups- lpd--but these are not things I expect MacPorts to install scripts for anyway. Chris [1] http://xml.sfiera.net/

On Sep 1, 2007, at 11:47, Chris Pickel wrote:

On 01 Sep, 2007, at 4:35, Ryan Schmidt wrote:

...

I have this problem: the lighttpd web server on my MacBook Pro unexpectedly dies rather frequently (much more frequently than it did on my PowerBook). When this happens, the daemondo process is still there, and I have to use launchctl to unload and then load the lighttpd plist to get the server running again.

The point of launchd is that it notices when processes unexpectedly quit, and restarts them automatically so close to no downtime is observed. But launchd is monitoring daemondo, which does not exit, so launchd does not restart anything. And daemondo doesn't seem to have the same process-death-noticing feature so daemondo is not automatically restarting the failed lighttpd.

Agreed. I try to avoid daemondo when possible. When it isn't possible, daemondo is an indispensable tool, but it turns out that for most cases, it's possible to avoid it. I've written many launchd scripts on my own, several of which I've put online [1].

It's possible to make launchd scripts similar to these in MacPorts via startupitem.executable, but currently, the startupitem.* commands in MacPorts do not support setting the "User" key. For lighttpd, that's not necessary; it starts as root so it can listen on the privileged port 80, then drops privileges. For mysql, though, it's pretty important.

Thanks, I didn't even know about startupitem.executable. According to the new guide, it's the preferred startupitem method, but only 10 ports currently use it! I'll have to look into it.

markd@macports.org wrote:

Ryan Schmidt <ryandesign@macports.org> writes:

...

...

Agreed. I try to avoid daemondo when possible. When it isn't possible, daemondo is an indispensable tool, but it turns out that for most cases, it's possible to avoid it. I've written many launchd scripts on my own, several of which I've put online [1].

Thanks, I didn't even know about startupitem.executable. According to the new guide, it's the preferred startupitem method, but only 10 ports currently use it! I'll have to look into it.

I didn't realize that daemondo couldn't keep daemons running that die. Anyway, I just updated the new guide's startupitems section and established some terms to distinguish between the different types. See what you think of it and if it is accurate.

http://geeklair.net/new_macports_guide/#reference.startupitems

Looks good Mark. It doesn't explain why one would use launchd or daemondo. Is this the appropriate place to put it, or is it described elsewhere? Also, for startupitem.pidfile, the default is shown as Default: none | ${prefix}/var/run/${name}.pid is one for launchd and the other for daemondo? Regards, Blair

Hi all, and sorry I haven't chimed in on this thread earlier. I'll answer a few questions below... On Sep 4, 2007, at 1:47 PM, markd@macports.org wrote:

Blair Zajac <blair@orcaware.com> writes:

...

Looks good Mark.

It doesn't explain why one would use launchd or daemondo. Is this the appropriate place to put it, or is it described elsewhere?

Hi Blair,

Yes you're right it doesn't say, and this would be the appropriate place. Fact is, I'm not sure the answer. James' original notes said that startupitem.executable was the preferred type, but it didn't say why. If daemondo doesn't monitor and restart daemons, as I guess it can't since it only knows about scripts, then perhaps that is the reason.

daemondo will indeed quit when it detects that the launched process has quit, and thus will "keep alive" the process, since launchd will then restart daemondo. In this way, daemondo acts as a shim or adapter between the scripts supported by the startupitem command, and the single process expected by launchd. The reason that startupitem.executable is preferred is that this gives the best possible chance that daemondo will be able to detect the death of the launched process: since daemondo can launch the process, it can also detect when it quits, stop it, etc. For those cases where startupitem.executable cannot be used, daemondo also supports the startupitem.pidfile commands that allow the process' pidfile to be monitored: daemondo will read the pidfile and watch for the death of that process. So daemondo, and thus launchd, will be aware of the daemon process death (and be able to restart the daemon process) only under two circumstances: (1) startupitem.executable was supplied (thus daemondo starts the process) (2) startupitem.pidfile was supplied (thus daemondo reads the process id) Under all other circumstances, daemondo will not know that the daemon process has died, and will not exit when the process does die, and thus launchd won't restart the process since it doesn't know it died. Put another way, if daemondo can know the process has died, then launchd will know too, but not otherwise.

But since so many apps just have startup scripts that don't monitor and restart their daemons, it seems odd to call it the "preferred type" since if that is to be done whenever possible (even when developers provide startup scripts), we'd be establishing a higher standard with ports than the developers who created the programs consider necessary. So I wonder if the "why" question doesn't come down to:

1) If a port author wants the daemon monitored and restarted if it dies, use an executable startupitem type.

Note that for simplicity, startupitem.executable is handled by daemondo at present. This has two purposes: - It keeps the startupitem generating code a little simpler. - It allows the potential support for higher value services to be provided by daemondo. In particular, note daemondo's --restart- netchange option, which can be quite useful, but for which there is no current support by the startupitem keys.

2) Otherwise, use a wrapper type (daemondo) startup script if the developer provided a startup script that works ok on the target platform and the deamon isn't unstable for some reason.

In most cases, the second step after using startupitem.executable should be to make sure that the pidfile is identified (if there is one) since this will allow daemondo to track the process and will lead in most cases to satisfaction.

3) If the developer didn't provide a startup script or one that works ok on the target platform, you may use either executable or wrapper startupitem types. Unless we can establish executable startupitems are *really* the preferred type for MacPorts, then the advice is: executable if possilble and wrapper if not.

Yes, executable really are the preferred means, with pidfile close behind.

If I can arrive at some answers on this I can clarify that section.

...

Also, for startupitem.pidfile, the default is shown as

Default: none | ${prefix}/var/run/${name}.pid

is one for launchd and the other for daemondo?

No and I struggled on that one because the startupitem.pidfile keyword has two separate values: one specifies the pid handling behavior, the other the pidfile path. So there are two defaults for the one keyword. Also, since the keyword is of a type "executable", it cannot be used with wrapper startupitems.

Hmm. I don't get your distinction between these "types". The pidfile keyword is likely used only if executable is not.

"Executable StartupItem keywords may not be used together with any of the wrapper StartupItem keywords."

Hmm. That may be too strongly put. I'll have to read your section, which I haven't yet.

But that should be stated clearly in the intro to the topic, but it isn't so it is easily missed. I can make it all more clear and I will do that. Hopefully others will know the answer to the first question and whether executable startupitem types are truly "preferred". Your feedback is helpful; thanks a lot.

Please let me know how I can clarify things any further. Thanks for the work you're putting into the documentation. James

Mark

James Berry <jberry@macports.org> writes:

Hi all, and sorry I haven't chimed in on this thread earlier. I'll answer a few questions below...

On Sep 4, 2007, at 1:47 PM, markd@macports.org wrote:

...

Blair Zajac <blair@orcaware.com> writes:

...

Looks good Mark.

It doesn't explain why one would use launchd or daemondo. Is this the appropriate place to put it, or is it described elsewhere?

Hi Blair,

Yes you're right it doesn't say, and this would be the appropriate place. Fact is, I'm not sure the answer. James' original notes said that startupitem.executable was the preferred type, but it didn't say why. If daemondo doesn't monitor and restart daemons, as I guess it can't since it only knows about scripts, then perhaps that is the reason.

daemondo will indeed quit when it detects that the launched process has quit, and thus will "keep alive" the process, since launchd will then restart daemondo. In this way, daemondo acts as a shim or adapter between the scripts supported by the startupitem command, and the single process expected by launchd.

The reason that startupitem.executable is preferred is that this gives the best possible chance that daemondo will be able to detect the death of the launched process: since daemondo can launch the process, it can also detect when it quits, stop it, etc.

For those cases where startupitem.executable cannot be used, daemondo also supports the startupitem.pidfile commands that allow the process' pidfile to be monitored: daemondo will read the pidfile and watch for the death of that process.

So daemondo, and thus launchd, will be aware of the daemon process death (and be able to restart the daemon process) only under two circumstances:

(1) startupitem.executable was supplied (thus daemondo starts the process) (2) startupitem.pidfile was supplied (thus daemondo reads the process id)

Under all other circumstances, daemondo will not know that the daemon process has died, and will not exit when the process does die, and thus launchd won't restart the process since it doesn't know it died. Put another way, if daemondo can know the process has died, then launchd will know too, but not otherwise.

...

But since so many apps just have startup scripts that don't monitor and restart their daemons, it seems odd to call it the "preferred type" since if that is to be done whenever possible (even when developers provide startup scripts), we'd be establishing a higher standard with ports than the developers who created the programs consider necessary. So I wonder if the "why" question doesn't come down to:

1) If a port author wants the daemon monitored and restarted if it dies, use an executable startupitem type.

Note that for simplicity, startupitem.executable is handled by daemondo at present. This has two purposes:

- It keeps the startupitem generating code a little simpler. - It allows the potential support for higher value services to be provided by daemondo. In particular, note daemondo's --restart- netchange option, which can be quite useful, but for which there is no current support by the startupitem keys.

...

2) Otherwise, use a wrapper type (daemondo) startup script if the developer provided a startup script that works ok on the target platform and the deamon isn't unstable for some reason.

In most cases, the second step after using startupitem.executable should be to make sure that the pidfile is identified (if there is one) since this will allow daemondo to track the process and will lead in most cases to satisfaction.

...

3) If the developer didn't provide a startup script or one that works ok on the target platform, you may use either executable or wrapper startupitem types. Unless we can establish executable startupitems are *really* the preferred type for MacPorts, then the advice is: executable if possilble and wrapper if not.

Yes, executable really are the preferred means, with pidfile close behind.

...

If I can arrive at some answers on this I can clarify that section.

...

Also, for startupitem.pidfile, the default is shown as

Default: none | ${prefix}/var/run/${name}.pid

is one for launchd and the other for daemondo?

No and I struggled on that one because the startupitem.pidfile keyword has two separate values: one specifies the pid handling behavior, the other the pidfile path. So there are two defaults for the one keyword. Also, since the keyword is of a type "executable", it cannot be used with wrapper startupitems.

Hi James, Thanks for the info. The startupitem behavior is not as I thought so I'll have to rework the guide section after reading your email carefully later tonight. But some initial questions:

Note that for simplicity, startupitem.executable is handled by daemondo at present. This has two purposes:

I didn't know that.

...

Also, since the keyword is of a type "executable", it cannot be used with wrapper startupitems.

Hmm. I don't get your distinction between these "types". The pidfile keyword is likely used only if executable is not.

I understand by your response that the startupitem.pidfile may be used without startupitem.executable, and with it as well. So I categorized the startupitem.pidfile wrong. But there needs to be some distinctions made in startupitem types. If the startupitem.executable is preferred; what are the alternative types over which it is preferred? There are 10 other startupitems keywords, so are there 9 other types? Obviously not. And the manpage says: "This (executable) is the preferred type of startup item rather than any of startupitem.init, startupitem.start, startupitem.stop, or startupitem.restart, and may not be used together with any of these options." To what type does this other collection of startupitems belong? "Non-executable", "wrapper", or some such name needs to be established for that type. So I see 3 categories of startupitems keywords: A) executable (only one of them), B) "non-executable" or "wrapper" or some other name, C) naming and logging (startupitems that don't have to do with a type). Type A and B can't be mixed together, and type C are attributes that may be used with either A or B. Maybe it could be classified better, but I think we really need some coherent way of categorizing these items and relating them to each other. Mark

On Sep 5, 2007, at 6:48 PM, markd@macports.org wrote:

James Berry <jberry@macports.org> writes:

...

For those cases where startupitem.executable cannot be used, daemondo also supports the startupitem.pidfile commands that allow the process' pidfile to be monitored: daemondo will read the pidfile and watch for the death of that process.

...

So daemondo, and thus launchd, will be aware of the daemon process death (and be able to restart the daemon process) only under two circumstances:

(1) startupitem.executable was supplied (thus daemondo starts the process) (2) startupitem.pidfile was supplied (thus daemondo reads the process id)

Under all other circumstances, daemondo will not know that the daemon process has died, and will not exit when the process does die, and thus launchd won't restart the process since it doesn't know it died. Put another way, if daemondo can know the process has died, then launchd will know too, but not otherwise.

These statements seem incompatible with ....

...

The pidfile keyword is likely used only if executable is not.

This one.

I don't see the incompatibility of those statements, but then I again I know what i meant, not necessarily what it means to others ;). The later sentence, btw, is missing a word on the end. It should read: "the pidfile keyword is likely useful only if the executable keyword is not specified." Does that help any?

Looks to me like startupitem.pidfile must be set for a deamon to be tracked whether it is executable startupitem or not.

No, daemondo will track an "executable" in an case (and it doesn't need to know where their pidfile is, generally, since it launches the code and thus knows the pid). In the case of script code (non "executable") daemondo doesn't know the pid, since it doesn't know what the script code did. In this case, it has to rely on reading a pidfile to get the process id, or else simply not know.

And the man page says startupitem.pidfile is "particularly useful" for startupitem.executable. Can you explain this?

That was either garbage to begin with, or else got messed up in creation of the man page. Off the top of my head I can't see any particular reason to use a pidfile keyword in conjunction with the executable keyword, unless it's to specify that it should delete a pidfile created by the executable, and I'm not sure that even works for that case. Hope that helps. James