Introduction to the shell#
The shell is for most Unix users the main interface to an Unix-based system like Linux or macOS. The shell is a command line interface to the system and comes in different flavors like bash, tcsh, ksh, zsh, etc. While multiple shells exist, the most popular shells used are bash for interactive use and sh for system scripting.
A lot can be configured in the shell, for example the prompt. The prompt is the string that is displayed before the command line and can be changed by modifying the PS1 variable. The prompt normally also shows the current working directory and the current user, but this is a convention. A second convention is that if the prompt ends with a #, then your in a shell with root privileges and when it ends with a $, then you are in a shell with normal user privileges.
# when the shell is in a root user context## whoami
root
$ when the shell is in a normal user context#$ whoami
user01
This convention is used in all examples on this site together with the assumption that the user is logged in as user01 and is by default located in the /home/user01 directory.
Using su#
Every shell is running under the account that was used to login with. This account also refered to as an user is member of a primary group that is required, and can also have up to 16 or 32 secondary groups depending if Network File System (NFS) is used or not.
With the command id you can get the primary group and the secondary groups of the user that is currently logged in.
id shows the primary group and secondary groups for the user that is logged in#$ id
uid=1000(user01) gid=1000(user01) groups=1000(user01)
The command can also be extended to get the groups of other users defined on the system as is show below for the user root.
id can also show the primary and secondary group for other users#$ id root
uid=0(root) gid=0(root) groups=0(root)
A shell runs as the user that is logged in. This is the user that is used to run the command line and to see the effective user that is used to run the command whoami.
# whoami
root
The effective user can be changed by using the command su. This command changes the user that the shell runs as, but to accomplish this the password of the target user is required.
root#$ su root
password:
By default only the effective is changed by the su command that stands for switching user or substitute user, but the environment is not changed. To change the environment as well, use the - option, which is an alias for -l and --login, and the environment is also changed when the user changes.
root#$ su - root
Password:
When we test this by defining VAR1 and run the command echo $VAR1 to see the variable under the user root, then no result is given as the environment variable has been cleared when the user was switched to the user root.
export VAR1="Hello World"
$ su - root -c 'echo $VAR1'
Password:
When we define the same variable and switching to root to run the same command the content of VAR1 is shown as the command su did not clear any environment variables as with the - option that simulates a new login.
export VAR1="Hello World"
$ su root -c 'echo $VAR1'
Password:
Hello World
Warning
Using the options -, -l, --login start the shell as a login shell with an environment similar to a real login:
clears all the environment variables except TERM and variables specified by
--whitelist-environmentinitializes the environment variables HOME, SHELL, USER, LOGNAME, and PATH
changes to the target user’s home directory
Passing environment variable from one user to another can introduce security risks as it can lead to data leakage or unexpected code execution based on set environment variables.
Using sudo#
With the command su one can switch to another user, but the password of the target must be known and also gives full access. Another command called sudo allows users to run commands as another user when this is defined with the option to do this passwordless or authenticating with their own password.
The command sudo works with both an allow and block list of commands that an user can execute as a certain user. With the command sudo -l an overview can be retrieved as is shown below.
$ sudo -l
Matching Defaults entries for user01 on server01:
!visiblepw, always_set_home, match_group_by_gid, always_query_group_plugin,
env_reset, env_keep="COLORS DISPLAY HOSTNAME HISTSIZE KDEDIR LS_COLORS",
env_keep+="MAIL QTDIR USERNAME LANG LC_ADDRESS LC_CTYPE",
env_keep+="LC_COLLATE LC_IDENTIFICATION LC_MEASUREMENT LC_MESSAGES",
env_keep+="LC_MONETARY LC_NAME LC_NUMERIC LC_PAPER LC_TELEPHONE",
env_keep+="LC_TIME LC_ALL LANGUAGE LINGUAS _XKB_CHARSET XAUTHORITY",
secure_path=/usr/local/sbin\:/usr/local/bin\:/usr/sbin\:/usr/bin\:/sbin\:/bin\:/var/lib/snapd/snap/bin
User user01 may run the following commands on server01:
(ALL) ALL
One of the drawbacks of the command su is that the password of the target password must be known as has been showed in section Using su. The command su doesn’t require the target password when executed as the user root, and with sudo we can run this command without knowning the target password as is show in th example below.
$ sudo su -
[sudo] password for user01:
#
This can also be used to run all kinds of commands like in the example below where the file /etc/sudoers can be read while only being readable by root.
/etc/sudoers with command cat#$ sudo cat /etc/sudoers
## Sudoers allows particular users to run various commands as
## the root user, without needing the root password.
##
...
By default the command sudo assumes that the target account will be root, but other accounts can be specified with the option -u as show in the following example where we view the content of a directory only accessable only by the user chrony or members of the group chrony.
-u#$ sudo -u chrony ls -la /var/log/chrony
total 0
drwxr-x---. 1 chrony chrony 0 Aug 31 15:54 .
drwxr-xr-x. 1 root root 854 Jan 15 00:03 ..
Editing files with sudo isn’t advised for security reasons, but with the command sudoedit a copy of the file will be made to be safely modified and that will be used to later replace the original.
/etc/hosts safely with command sudoedit#$ sudoedit /etc/hosts
Warning
Allowing commands like vi and nano to be used via sudo opens the possibility read and/or change other files then originally defined, and to start new commands that aren’t defined by sudo as allowed command.
Monitoring the system#
Linux is a multi-user system and as a result different users can be logged on at the same time, but also the same user can be logged on multiple times. knowning the basics of the system can be useful to quickly see how busy the system is and maybe also how long the system is running.
Showing system basics#
Most Linux systems that are being used by multiple users it can be useful to quickly see how busy the system is and maybe also how long the system is running. With command uptime some basic information can be quickly retrieved as show in the example below. The example below shows that the system is running for 18 hours and 49 minutes, has one user logged on, and an average number of processes running in the last minute, and five and fiveteen minutes.
$ uptime
15:58:26 up 18:49, 1 user, load average: 1.21, 1.66, 1.93
Linux is a multi-user system and as a result different users can be logged on at the same time, but also the same user can be logged on multiple times. With the command who an overview can be created to see which users are logged on, which terminal they’re using and when they started the session.
$ who
user01 tty2 2023-01-14 21:09 (tty2)
The second command that can be used to see who is logged on to the system is w. This command also shows which command is being executed and the processor time consumpted.
$ w
USER TTY LOGIN@ IDLE JCPU PCPU WHAT
user01 tty2 Sat21 18:50m 0.04s 0.04s /usr/libexec/gnome-session-binary
Showing processes#
Linux is a multi-user and multi-process operating system allowing multi-users to run multi-processes at the same time, but who is running which processes? With the command ps a report of all current processes can be shown to see what is running on the system. The command ps comes in two flavors for its syntax and reporting as it both supports the standard syntax and BSD syntax. For now we will only focus on the standard syntax.
$ ps -ef
UID PID PPID C STIME TTY TIME CMD
root 1 0 0 Mar07 ? 00:00:16 /usr/lib/systemd/systemd rhgb --switched-root --system --deserialize 36
root 2 0 0 Mar07 ? 00:00:00 [kthreadd]
root 3 2 0 Mar07 ? 00:00:00 [rcu_gp]
root 4 2 0 Mar07 ? 00:00:00 [rcu_par_gp]
root 5 2 0 Mar07 ? 00:00:00 [slub_flushwq]
root 6 2 0 Mar07 ? 00:00:00 [netns]
root 8 2 0 Mar07 ? 00:00:00 [kworker/0:0H-events_highpri]
...
Note
The command ps also supports the BSD syntax, but the output is different. The example below shows the same output as the example above, but in BSD syntax.
$ ps aux
Within Linux processes can be organized in a tree structure where each process has a parent process and can have multiple child processes. With the command pstree a tree of processes can be shown to see how the processes are organized. The command pstree can be used to show the processes of a specific user or of a specific process. The example below shows the tree of processes starting with process identifier 1.
$ pstree 1
systemd─┬─ModemManager───3*[{ModemManager}]
├─NetworkManager───2*[{NetworkManager}]
├─abrt-dbus───2*[{abrt-dbus}]
├─3*[abrt-dump-journ]
├─abrtd───2*[{abrtd}]
├─accounts-daemon───3*[{accounts-daemon}]
├─alsactl
├─auditd───{auditd}
├─avahi-daemon───avahi-daemon
├─bluetoothd
├─chronyd
...
Most users will use the command ps to see which processes are running on the system and then use grep to filter out the correct lines. With the command pgrep the filtering can be done directly on the command line and the output can be used to further process the data.
$ pgrep systemd
1
836
854
956
957
958
1034
1036
1754
27251
27282
27283
Note
The command pgrep can be used to find the process ID (PID) of a process based on its name, and the syntax can also be used for other commands like pkill and pidwait. The only exception is the option -l which is only supported by pgrep.
Process priority#
All processes in Linux have a priority which can be used to influence the scheduling of the process. The priority of a process can be set by the user or by the system. The priority of a process can be set between -20 and 19 where -20 is the highest priority and 19 is the lowest priority. The default priority of a process is 0. The priority of a process can be set with the command renice.
With the command ps the priority of a process can be shown if the option -l is used to use the long format output. The column PRI shows the priority of the process, but the column NI shows the priority of the process normalized to the range -20 to 19.
$ ps -efl
F S UID PID PPID C PRI NI ADDR SZ WCHAN STIME TTY TIME CMD
4 S root 1 0 0 80 0 - 43062 - Mar07 ? 00:00:16 /usr/lib/systemd/systemd rhgb --switched-root --system --deserialize 36
1 S root 2 0 0 80 0 - 0 - Mar07 ? 00:00:00 [kthreadd]
1 I root 3 2 0 60 -20 - 0 - Mar07 ? 00:00:00 [rcu_gp]
1 I root 4 2 0 60 -20 - 0 - Mar07 ? 00:00:00 [rcu_par_gp]
1 I root 5 2 0 60 -20 - 0 - Mar07 ? 00:00:00 [slub_flushwq]
1 I root 6 2 0 60 -20 - 0 - Mar07 ? 00:00:00 [netns]
1 I root 8 2 0 60 -20 - 0 - Mar07 ? 00:00:00 [kworker/0:0H-events_highpri]
...
By default the priority of a process is set to 0. The priority of a process can be changed with the command renice. The command renice can be used to change the priority of a process or to start a process with a different priority. The command renice can be used to change the priority of a process with the option -p and the process ID (PID) of the process. The command renice can be used to start a process with a different priority by specifying the command to run after the options.
$ renice -n 10 -p 1234
Using the command nice a process can be started with a different priority to already get less priority by the scheduler.
$ nice script.sh
By default a process has a niceness level of 0, but the command nice sets this to level 10 unless the option -n is used to specify a different niceness level.
$ nice -n 15 script.sh
While the process priority can be set to a value between -20 to 19, users are only allowed to select priorities from 0 to 19. The priorities -20 to -1 are reserved for the system. The command renice will fail if a user tries to set a priority level below 0 as shown in the example below.
-1 for a process as a non-root user#$ renice -n -1 7970
renice: failed to set priority for 7970 (process ID): Permission denied
The command renice can be used to change the priority of a set of processes. The command pgrep can be used to find the process IDs (PIDs) of a set of processes and the output of pgrep can be used as input for renice.
$ renice -n 15 $(pgrep <process name>)
Working with files withing the shell#
Linux is a files oriented operating system and understanding how to work with files is essential to work with Linux. This section will cover the basic commands to work with files and directories.
Basic commands for files and directories#
The command ls can be used to list the files and directories in a directory. The command ls can be used to list the files and directories in the current directory if no directory is specified. The command ls can be used to list the files and directories in a specific directory if the directory is specified as an argument to the command.
$ ls
file1 file2 file3
The command touch can be used to create an empty file. The command touch can be used to create an empty file with the name specified as an argument to the command.
$ touch file1
The command cp can be used to copy a file. The command cp can be used to copy a file with the name specified as the first argument to the command to a file with the name specified as the second argument to the command.
$ cp file1 file2
The command mv can be used to move a file. The command mv can be used to move a file with the name specified as the first argument to the command to a file with the name specified as the second argument to the command.
$ mv file1 file2
The command rm can be used to remove a file. The command rm can be used to remove a file with the name specified as an argument to the command.
$ rm file1
The command mkdir can be used to create a directory. The command mkdir can be used to create a directory with the name specified as an argument to the command.
$ mkdir dir1
The command rmdir can be used to remove a directory. The command rmdir can be used to remove a directory with the name specified as an argument to the command. The command rmdir can only be used to remove empty directories.
$ rmdir dir1
Archiving and compression#
The command tar can be used to create and extract archives. The command tar can be used to create archives with the option c and to extract archives with the option x. The option f can be used to specify the archive file to use. The option v can be used to show the files that are archived or extracted.
archive.tar#$ tar cf archive.tar file1 file2 file3
The command tar can also be used to create an archive for a directory by instead of specifying the files to archive, the directory to archive can be specified. The example below creates an archive for the directory /etc.
/etc into archive /root/etcbackup.tar## tar cf /root/etcbackup.tar /etc
Archives can be tested with the option t to show the files that are in the archive. The option f can be used to specify the archive file to use.
/root/etcbackup.tar can be read successfully.## tar tf /root/etcbackup.tar
Archives are by default not compressed, but with different options the archive can be compressed. The option z can be used to compress the archive with Gzip. The option j can be used to compress the archive with Bzip2. The option J can be used to compress the archive with xz.
Creating a compressed archive with tar and the option z for Gzip compression of the archive.
Creating a compressed archive with tar and the option j for Bzip2 compression of the archive. The option to create a compressed archive can also be used to extract the compressed archive as shown in the example.
# mkdir /root/etcbackup
# cd /root/etcbackup
# tar xjf /root/etcbackup.tar.bz2
Transfering files with SCP#
Copying a file to a remote host with the command scp which stands for secure copy and will assume you want to login with the same username as currently logged in.
/etc/passwd to directory /tmp on server02#$ scp /etc/passwd server02:/tmp
Copy a file to a remote host like in the previous example, but now with a specific user called root that will be used to login to server02.
$ scp /etc/groups root@server02:/tmp
Copy a file from a remote host as root to the current directory:
$ scp root@server02:/tmp/groups .
With the -r flag the scp command will copy the contents of the directory:
/etc locally to a remote host#$ scp -r /etc user01@server02:~
Transfering files with SFTP#
The command sftp is a program for secure file transfer between hosts and works similar like the ftp command. In the example below a connection to the remote host localhost is established with the user user01. And directly the remote host can be browsed with the ls command.
$ sftp user01@localhost
user01@localhost's password:
Connected to localhost.
sftp> ls
bin file.sh output
With the mkdir command a new directory can be created on the remote host and with the cd command the current directory can be changed.
sftp> mkdir test
sftp> cd test
With the put command a file can be uploaded to the remote host. The file will be uploaded to the current directory on the remote host.
sftp> put /etc/hosts
Uploading /etc/hosts to /home/user/test/hosts
/etc/hosts 100% 510 0.5KB/s 00:00
Using rsync to copy files#
Rsync is a fast, reliable, efficient, secure, and portable file transfer. The example below will copy the contents of the directory /etc to the directory /srv/rsync/etc/:
$ rsync -av /etc /srv/rsync
The example below will copy the contents of the directory /etc/ to the directory /srv/rsync/:
$ rsync -av /etc/ /srv/rsync
Copy log files from server01 to /tmp on the current host:
$ rsync -av server01:/var/log /tmp
Remote login with SSH keys#
By default you will be asked for a password when you login to a remote host with the command ssh. This can be avoided by using SSH keys. The following example will generate a SSH key pair and copy the public key to the remote host.
$ ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/home/user01/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /Users/username/.ssh/id_rsa.
Your public key has been saved in /Users/username/.ssh/id_rsa.pub.
The key fingerprint is:
01:0f:f4:3b:ca:85:d6:17:a1:7d:f0:68:9d:f0:a2:db [email protected]
The key's randomart image is:
+--[ RSA 2048]----+
| |
| |
| . E + |
| . o = . |
| . S = o |
| o.O . o |
| o .+ . |
| . o+.. |
| .+=o |
+-----------------+
Warning
The command ssh-keygen uses the default key name id_rsa. If you want to use a different name, you must specify it with the -f option.
The public key is stored in the file ~/.ssh/id_rsa.pub and the private key in the file ~/.ssh/id_rsa. The private key must be kept secret and should not be shared with anyone. The public key can be shared with anyone and must be added to the file ~/.ssh/authorized_keys on the remote host. In the example below the public key is copied to the remote host.
$ scp ~/.ssh/id_rsa.pub user@remotehost:~/.ssh/myhost.pub
On the remote host the public key must be added to the file ~/.ssh/authorized_keys:
$ cat myhost.pub >> .ssh/authorized_keys
With the command ssh-copy-id the public key can be copied to the remote host and added to the file ~/.ssh/authorized_keys automatically as is show in the example below:
$ ssh-copy-id [user@]remotehost
When connecting to the remote host we are automatically logged in without being asked for a password.