May 19
More on keeping commands running after exiting the Linux terminal
I wrote a while back an article about the nohup command which allows a running Linux command to keep running even after exiting the terminal. The magic happens because nohup will make a process ignore the SIGHUP signal, which is broadcasted when exiting a terminal. Then in another article I was explaining how signals communicate with Linux processes and are able to send commands to them.
So the whole thing demystified means that Linux processes are handling SIGHUP signals by default by exiting, and this SIGHUP signal is sent automatically when exiting a terminal. Add the two together and you get the well known and sometimes annoying result: your long running copy command stops when exiting a SSH connection to your remote server.
So then the solution to all this comes naturally: why don’t we make it so that the copy command ignores the SIGHUP signal? That way the terminal would exit, the SIGHUP signal would be broadcasted but when it reaches our copy command, it would do nothing. It would be ignored. And that is exactly what the nohup command does, as explained before.
But things go a little deeper if you wish to know, which opens the magic box a bit more and gives another tool that we can use to achieve the same thing. You see, there was a carefully avoided unknown in the previous paragraphs: the SIGHUP event is broadcasted…where? Surely not all processes running are closed, that would mean a total system reboot. Maybe all processes started by the logged in user? No, because I close terminals all day long on my desktop session and the graphics system does not go away, nor do all the other started apps. So what is happening then?
The power of the Linux shell
Let’s consider an interactive Linux script, or a program that asks the user if they are certain they want to do an operation on a file. In the terminal, who allows the program to asks questions? And who allows the user to answer? Yes, the shell.
The shell reads input from the user and displays output from the program. The shell also allows a program to run in the background by using the bg command, or to come back in the foreground by using the fg command. The shell is also able to run programs straight in the background by adding an & at the end of the command line and is able to interrupt the execution of a program by pressing ctrl+z. The shell intermediates the communication between the user and the foreground program and it displays the output of foreground and background programs.
So the shell has a lot to say about running processes. It is only natural for it to keep track of them in a list. To know which are in the background, which is in the foreground, which are interrupted and which it needs to close when the terminal exits.
Linux shell jobs
So then, as we already know, each time you run a Linux application it will start in a process. The process itself has a lot going on and we will talk about it in another article, but there is one more thing happening when you start a program in a Linux shell: a job is created. The shell job is not a process, it’s a sort of identifier for a task that is currently running. It’s like a named pointer to the running process.
Also, the shell job has nothing to do with the kernel: it’s a shell construct. It exists in the shell and the Linux commands we use to inspect the shell jobs are shell commands, not Linux programs. We can see the Linux jobs by using the jobs command. Let’s see an example:
#ping linux.org > /dev/null &We are starting a long running task by pinging a random Internet target. We send the ping output to /dev/null because we are not interested in what it has to say. Finally, we start the command in the background by adding &. The result will be the following:
We know that 44191 is the process ID. But can you guess what [1] is? Yes, it’s the job ID. A job has been created by this Linux shell and it’s the first one so it gets ID 1. If we open another shell and start another ping, it will have the job ID 1. Jobs are a shell construct and they are scoped to it. Each opened shell will have its own job table.
Ok, let’s see what we have in the jobs table by using the jobs command:
We have one running job. We can see the process ID too by adding the -l parameter:
The plus beside the [1] job ID means it’s the last job that was started. Let’s start another one:
We can bring the last job to the foreground by running fg or we can bring the first job to the foreground by running fg %1:
Exiting a terminal and cleanup
When we exit a terminal, the shell running there is responsible for cleaning up: it will send the SIGHUP event to all processes in its jobs list. So then, yes we can use nohup to make the processes ignore the SIGHUP signal, but now that we know about the jobs list we opened up another possibility. Could we edit the jobs list?
And here comes the disown command. Again, not a program but a shell command because it deals with shell constructs. Disown allows us to remove jobs from the jobs table:
I started a long running task above, I checked to see it’s in the jobs table by using the jobs command: it’s there, and then I used disown on it and behold: the jobs table no longer knows anything about it. But is the process still running?
Yes it is, we can see it with the ps command, the process is definitely still running but it’s no longer tracked by the shell. We cannot bring it in the foreground using fg, we cannot communicate with it via the terminal: the shell lost all its power over it.
This makes things interesting because it means we can close the shell and it will know nothing about our running ping. But there’s more:
Look at that -h parameter. It’s as if disown was created specifically with this role in mind: allowing us to end a Linux terminal session without closing our running tasks. The -h parameter will make all jobs ignore the SIGHUP signal (or just the desired job if we specify it). So we have come full circle, back to the nohup command, but this time with a lot more knowledge about what’s really happening behind the scene, and done by a different command, disown.
I hope you had fun reading this article. I hope it shed some light on the way Linux works. For me, learning about the shell jobs, although such a small and uninteresting subject, brought a sense of control as I understood part of what is happening when a shell ends its life. See you in the comments and in the next article!
