Job control commands
| Command | Description |
|---|---|
Ctrl-Z | Suspends current process |
jobs | Shows suspended jobs |
bg | Send job to background |
fg | Bring background job to the foreground |
➤ btop
fish: Job 1, 'btop' has stopped
➤ jobs
Job Group CPU State Command
1 72513 0% stopped btopWe can use bg %<job number> to allow a process to continue running but keep it in the background, freeing the terminal for other commands. To bring it to the foreground again, use fg %<job number>
Background execution commands
We can use & to immediately send a command into the background. For example, we can run tar my_file.tar.gz &.
We can use nohup as well (which stands for “no hang-up”). This is used to ignore the hang-up signal when logging out; this is used to ensure that a specific command keeps running on a server even if SSH or terminal sessions end. For example, we might do something like:
ssh user@server 'nohup ./long_script.sh > script_output.log 2>&1 &'This will initiate an SSH connection to a server and run long_script.sh wrapped with nohup. This means that if the SSH connection is terminated, the script will continue to run; after that, we are appending the output of the script to a log file (including stderr via 2>&1). Finally, the command is immediately sent to the background with &.
Job scheduling
Note
Although
cron,at, andanacronmay be fairly prevalent tools, some systems do not have them installed by default. For example, ArchLinux uses SystemD timers instead, and allows to replace these programs using systemd. For example, theatcommand can be replaced withsystemd-run --on-calendar="YYYY-MM-DD HH:MM:SS" mycommand.sh. See the archwiki for for information.
crontab
crontab is used to schedule tasks with intervals. A cron job is defined using crontab files, which can be either system-wide or per-user.
at
This allows us to schedule a task to run at a specific time in the future. It is a lot like cron, except used for one-off events.
anacron
This are cron-like jobs, except it will manage job scheduling even when the computer is off.
Process ID
The Linux kernel keeps track of running tasks/programs with an assigned Process ID (PID), which is a unique non-repeating int. It is used for things like sending shutdown signals, checking CPU or memory usage, etc.
Additionally, the kernel keeps track of PPIDs, or Parent PID. This is because we might have something like a bash process running (say, PID 2487) that itself launches a new process when we run the ls command (which might have a PID of 2490).
This allows us to see running processes and to trace what is generating new processes.
There are a few ways to see PIDs:
ps aux, ps -eo pid, stat, and cmd can all be used to list every running process. On the other hand, stuff like top and htop will continually refresh PIDs in real-time.
With the PID in hand, we can cat /proc/<PID>/ to see kernel stats about the specific process, we can kill it with kill -9 <PID>, or we can lower a process’ priority with renice -n 10 -p <PID> (see Nice Renice)
To trace PPIDs we can use ps -e --forest -o pid,ppid,cmd or the much shorter pstree -p.
Note that we can use pkill -P <PPID> <signal> to send a signal to a process and all its decendants.
Orphans and Zombies
An orphan process a running task whose parent has exited, and is therefore adopted by PID 1.
A zombie process is one that has completed execution but remains in the process table because its parent has not yet reaped1 it.
Process states
A process state is used to signal whether a process is actively using the CPU, waiting, or hanging.
In Linux, there are five process states: Running, Sleeping, Blocked, Stopped, and Zombie.
| State | Letter State | Description |
|---|---|---|
Running | R | Actively processing on the CPU |
Sleeping | S / D | A process that is waiting for input, a timer, or another even from which it can be awakened |
Stopped | T | A process that has been manually paused (via Ctrl-Z or a debugger). Remains frozen until it has a continue signal |
Blocked | S / D | Blocked is a kind of uninterruptible sleep, but refers to waiting on a painfully slow process |
Zombie | Z | Placeholder after a process exits but before parent reaps exit status |
Note
SandDare both sleep signals; the later one is “uninterruptible sleep”. The difference is thatDprocesses are waiting for some other task to finish, be it a network connection or a read/write operation, and cannot be interrupted otherwise. Can be used to check for I/O bottlenecks, etc. rather than looking at the application process itself.
Note
Whenever we stop a process with Ctrl-Z the PID remains intact, but the process stops using the CPU and actually runing tasks. To restore it we can use
fgorkill -CONT <PID>
Process priority
We can adjust process priorities with nice and renice.
Process Monitoring
There are a few tools for monitoring processes in Linux:
| Tool | Description |
|---|---|
ps | Gives a quick, static list of running processes and their resource usage |
top | An updating view that highlights busiest tasks |
htop | Like top, but with colors and parent-child relationships |
atop | Shows live metrics and logs information overtime |
The ps command by default shows processes attached to the current terminal. To see every process, use the -e flag: |
➤ ps
PID TTY TIME CMD
1209 pts/0 00:00:00 fish
4611 pts/0 00:00:01 fish
59818 pts/0 00:00:00 ps
➤ htop
fish: Job 1, 'htop' has stopped
➤ ps
PID TTY TIME CMD
1209 pts/0 00:00:00 fish
4611 pts/0 00:00:01 fish
59875 pts/0 00:00:00 htop
59883 pts/0 00:00:00 psWe can also show processes from other user’s terminals with ps -a. Alternatively, we can use ps -u <username> to see ownership, CPU, and memory usage for processes owned by a specific user account.
| Flag | Description |
|---|---|
-e | Show every process |
-a | Show processes from other user terminals |
-u | Show processes associated with a user |
-f | Full format with PPIDs and start times |
-p | Focus on a specific task |
We can use commands like mpstat to see how busy each CPU core is, or pidstat to see CPU, memory, and I/O usage by PID2
mpstat provides a per-processor breakdown of CPU activity, showing how much time each core spends in user mode, system mode, I/O wait, idle, etc.
pidstat is used to target a specific PID.
Process limits
CPU time - max amount of time a process can use the CPU in seconds Memory usage - max virtual memory a process can run, in RAM or swap Stack size - controls how much memory can be allocated for function calls Open file descriptors - sets a ceiling on how many files, sockets, or pipes a process can open at once Core file size - governs whether a process will dump its memory image to disk, and how large the dump will be
Signals
Linux has multiple signals that can be used for perform specific actions, such as reloading, pausing, shutting down, etc. Three common signals are:
| Signal | Signal number | Description |
|---|---|---|
SIGHUP | 1 | Stands for hangup. |
SIGKILL | 9 | Forcefully stops a program without letting it clean up or finish |
SIGTERM | 15 | Asks a program to shutdown gracefully |
These signals are used with the kill command. For example, we can asks process 1234 to reload without stopping (for instance, a webserver) with kill -1 1234.
If we want to terminate multiple processes based on a name without involving every PID, we can use killall. For example, your computer might have many instances of steamwebhelper running even after closing Steam:
➤ ps -e | rg steam
64669 ? 00:00:10 steamwebhelper
64672 ? 00:00:00 steamwebhelper
64677 ? 00:00:00 steamwebhelper
64678 ? 00:00:00 steamwebhelper
64680 ? 00:00:00 steamwebhelper
64694 ? 00:00:06 steamwebhelper
64700 ? 00:00:00 steamwebhelper
64701 ? 00:00:00 steamwebhelper
64726 ? 00:00:06 steamwebhelper
65019 ? 00:00:01 steamwebhelper
65830 pts/0 00:00:00 steam-runtime-lTo kill all of those, we could do killall steam.
The last variation is pkill, or pattern kill. We can use it to terminate processes based on a named pattern, for example: pkill -9 backup will send SIGKILL to all processes with backup in their name.