Linux-Fu: Keeping Things Running
If you’ve used Linux from the early days (or, like me, started with Unix), you didn’t have to learn as much right away and as things have become more complex, you can kind of pick things up as you go. If you are only starting with Linux because you are using a Raspberry Pi, became unhappy with XP being orphaned, or you are running a cloud server for your latest Skynet-like IoT project, it can be daunting to pick it all up in one place.
Recently my son asked me how do you make something run on a Linux box even after you log off. I thought that was a pretty good question and not necessarily a simple answer, depending on what you want to accomplish.
There’s really four different cases I could think of:
- You want to launch something you know will take a long time.
- You run something, realize it is going to take a long time, and want to log off without stopping it.
- You want to write a script or other kind of program that detaches itself and keeps running (known as a daemon).
- You want some program to run all the time, even if you didn’t log in after a reboot.
One of the things that makes Linux tough is that there are lots of options and so what works on one system may not work on another. If you can assume a single distribution, you might have a better chance of finding that things work the same. To keep things manageable, I’m going to focus on the first two items and maybe catch up on the last two at a later date. I’m also going to assume we are talking about command line programs. If you need to run graphical programs after you log out, that opens a lot of strange questions — it’s certainly possible, but it is strange since your graphical user environment will go away when you log out (hint: use VNC or Nx to create a persistent desktop).
I will, however, give you clues for the last two use cases. First, a program that detaches itself is a daemon. The steps to do that can be involved, or you can outsource it. Making a program run all the time can be simple or complicated. Most Linux distributions will have a file /etc/rc.local that runs as root on start up (at least, normal start up). You can add things there if you are just doing a one-off. Otherwise, you need to know if you are using SystemV init, Upstart, OpenRC, or Systemd. There are probably some others to contend with, as well. But that’s a topic for another day. If you can’t wait, try your Portuguese (or Google’s) on this IBM paper.
A Tale of Two Cases
Back to the first two cases, though. Suppose you are going to run a program remote_backup and you know you will launch it (perhaps using an ssh session) and then you’ll want to disconnect or make sure it doesn’t stop running if you accidentally get disconnected for some reason. Either way, just launching it from the command line means when your session exits, the program will stop. Or will it?
This can be as easy as running programs in a way that they will be immune to hangups. If you use bash and your options are set right, the answer can be very simple. If you use the & to start a program in the background, or you can suspend a running program (Control+Z) and move it to the background with the bg command, it might run even after you exit your session. This is another case of Linux’s flexibility — many ways to accomplish similar results — getting in the way.
If you run bash, you can see your “shell options” which include the “hangup on exit” settings. Try running this at a shell prompt:
shopt | grep huponexit
If you see that huponexit is set to off, then simply pushing a program into the background will let it survive your session disconnecting. Of course, if you try the same thing on another system, it might not work, and you’ll wonder why.
If you know you are running bash with huponexit off, you can run your program in the background very simply by ending with a single ampersand:
remote_backup &
However, it is safer to expressly prevent the program from dying if you log off. If for no other reason, because you won’t see any output from the program after you log off if you use the above method. If you are thinking ahead you can just run the program with “no hangup”:
nohup remote_backup
If you need arguments, just put them on the end like usual. Nohup will do a few things:
- Redirect stderr to stdout
- Redirect stdout to nohup.out (depending on your version of nohup, that may be in ~/nohup.out)
- Redirect stdin to an unreadable file
- Run the program and return to the shell prompt
The end result is the program will run, can’t get any input, and will put any output into nohup.out. If there is any data already in nohup.out, the new data will go to the end.
The reason all of these redirections occur is because nohup detects that each stream is connected to a terminal. If you already redirect things to a file, nohup won’t disturb them. So you could say:
nohup remote_backup >/tmp/backupstatus.log &
or
nohup bash -c 'echo y | remote_backup >tmp/backupstatus.log' &
That last line will take input from the pipe so it won’t redirect. All output (stdout and stderr) will go to /tmp/backupstatus.log.
Prove It!
If you want to see the difference nohup makes, log into a Linux server with ssh and type the following at a shell prompt:
echo '#!/bin/bash' >~/huptest
echo sleep 60 >>~/huptest echo 'date >/tmp/test.txt' >>~/huptest chmod +x ~/huptest ~/huptest
Before the 60 seconds expire, press the tilde key (~). Then press a period. The tilde is the SSH escape character (if it is at the start of a line). Press ~? if you want to know about more things you do with it. Now, go have a cup of whatever you drink and come back way after a minute. Log in. You won’t find a file at /tmp/test.txt (unless there was one already there, but the contents should make that clear). Your session ending killed the waiting program.
Now try this:
~/huptest &
That tells the shell not to wait for the program to finish. If you are using bash, this will work as long as huponexit is set to off. You can experiment by using:
shopt -s huponexit
and
shopt -u huponexit
The first command will turn the flag on. The second one resets the flag to off.
Finally, you can run the same commands with nohup:
nohup ~/huptest &
Even nohup isn’t perfect. If the program you are running intercepts the nohup flag itself, nohup won’t help you. However, most programs you are going to worry about do not catch the flag and using nohup will work regardless of the shell and its configuration.
There are many other ways you could do this. For example, investigate the at command which will run things at a specified time (which could be a second after the current time). The programs run will use sh, not bash (without some work) but will run detached.
Lack of Planning
There are two cases where nohup doesn’t help, though. First, you may not have planned for this ahead of time. If you start running something and realize it is going to take awhile or that you suddenly have to leave, what do you do? You might also have the case where you need to start a program, give it some manual input, and then want it to run without you watching it.
You may know that you can suspend a running program using the Control+Z key. Bash will tell you that it has created a job and will give you a number for that job. You can force it into the background using bg. For example, if the job number is 3:
bg %3
If you have huponexit off, that’s all you need to do. But in the general case, you need to tell the shell to either stop sending HUP signals to it or to remove it from the job table completely. You can use disown (built into bash) to do both of these functions.
Using disown with no options will remove the named jobs from the job table. If you don’t want that extreme, use the -h option to just inhibit the HUP signal from the shell for that one job. You can also specify -a to hit all jobs or -r to affect all running jobs. This is a built-in command for bash (so man bash to read more) and so it is shell dependent. Once you’ve disowned a background task, you can log off with no fear
Persistent Sessions
As usual with Linux, there is more than one way to go for any given task. You could use screen or tmux to provide a session. This is similar to VNC where you can log in and find whatever you were working on still there. If you do a lot of work at the command line, you might want to try byobu which gives a nicer interface to screen or tmux (see right).
If you want to try that, log in as usual, run screen, tmux, or byobu. Execute the test script again (with or without the & at the end). And use the ~. trick to kill the ssh session. Then log back in and restart the same program (screen, tmux, or byobu). You’ll see the screen there just like you left it. That’s handy in general. Plus you can build multiple graphic-like windows easily, but that’s not germane to this post.
Options
Part of the power of Linux is you have plenty of options. Part of the problem with Linux is you have plenty of options. The real problem isn’t if you are configuring a single Raspberry Pi where you control everything. You make it work and that’s that. The real issue is when you try to deploy something to multiple boxes with different users and maybe even different distributions.
Of course, this isn’t the only thing that causes you trouble if you are going for portability. If you are trying to write scripts that can go to many different Posix systems, you might read the GNU documentation for autoconf which has a lot of good information about the problems and solutions.
Filed under: Hackaday Columns, linux hacks, Skills
</div>
Read More here.