System Administration

EnvironmentFiles used by systemd

In a previous posting, Environment variables set by systemd, variables were directly set within the systemd unit file. This is fine for a small amount of modifications, but in some case these environment variables are provided by another package on the system or need to be the same for multiple services.

We have modified our Python application to print all environment variables that are set to make this example easier.

#!/usr/bin/env python3

import os

for param in os.environ.keys():
    print("%20s %s" % (param, os.environ[param]))

We create the first environment file /usr/local/env/file1 with the content as below to assign string values to variables. Just as in the systemd unit file no string interpolation is being done, only lines with an equal sign are processed and everything after a hash sign is ignored.


We also create a second environment file /usr/local/env/file2 with the content below. Directly we see that variable FVAR1 is also be declared in this environment file.


To use the environment files we need to declare them in the systemd unit file below. The line for file1 shows that we require the file to be present otherwise the service will fail, but for file2 the filename has been preceded by a hyphen to indicate to systemd that the file is optional and no error will be generated if it is absent.

Description=App Service

Environment="VAR0=hello world"

After restarting the application with systemd all the environment variables that were set are shown in the system journal. The most interesting variable shown is FVAR1 as we declared it in two files earlier and we see that the value set in file1 was replaced by the value set in file2 that was processed later.

$ sudo systemctl daemon-reload
$ sudo systemctl restart app.service
$ sudo journalctl -u app
Aug 12 09:43:35 systemd[1]: Started App Service.
Aug 12 09:43:35 app[4483]: LANG en_US.UTF-8
Aug 12 09:43:35 app[4483]: VAR0 hello world
Aug 12 09:43:35 app[4483]: FVAR2 test2
Aug 12 09:43:35 app[4483]: FVAR3 Test3
Aug 12 09:43:35 app[4483]: FVAR1 TEST1
Aug 12 09:43:35 app[4483]: PATH /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin

While the purpose isn’t directly clear for everyone in the beginning, the most common use-case for enterprises is to manage environment variables that need to be set for Oracle for example. As lot of developers and engineers struggle why ORACLE_HOME isn’t being set while /etc/profile.d/ is present and works fine on other platforms like IBM AIX.

System Administration

Environment variables set by systemd

Applications sometimes need environment variables to be set for triggering certain behavior like giving debug output or to route traffic via a HTTP-proxy for example. A common way is to modify the start-stop script, but with systemd on most Linux systems, like Debian and Red Hat based distributions, this can also directly set within the unit file and you don’t have to export the variables anymore.

The Python script below that we run via systemd checks if environment variable VAR1 has been set and will generate a different output based on that.

#!/usr/bin/env python3

import os

if os.environ.get('VAR1'):
    a = os.environ['VAR1']
    a = 'default'


Running the Python script also shows the output difference as the second command doesn’t print the string “default” anymore to the terminal, but the text “test” that we set via the environment variable.

$ ./ 
$ VAR1=test ./ 

Setting the environment variables via systemd is done by adding the attribute Environment to the Service section of the unit file for the service. After a systemctl daemon-load the environment variable will be set when you start or restart the service.


If more variables need to be set, then more Environment attributes can be added to the Service section.


While it may break some human workflows in the beginning, but in long term it is a good step for following the infrastructure as code path as Ansible could be used for managing these variables. Also storing these kind of variables in the same way makes both troubleshooting and collecting settings for an audit easier.

Internet, Unix en security

PAM bug hit Debian and others

It has been years since PAM was hit by a serious bug in PAM, but people who upgrade to libpam-systemd version 44-1 can find that sudo stops working. Reading the bugreport on Debian and it doesn’t look promising as it also effects other distributions. For now it may be wise put systemd on hold in case the package transfers from unstable to testing.

System Administration

A smoother transition in Debian

About a month ago a transition in Debian took a wrong turn for systemd users. This ended in systemd users who where unable to shutdown there system, but recently a newer version was uploaded to unstable solving this issue. So time to unlock the block packages and upgrading systemd to version 37-1.1 and blocked packages.

$ echo "bootlogd install" | sudo dpkg --set-selections
$ echo "initscripts install" | sudo dpkg --set-selections
$ echo "sysvinit install" | sudo dpkg --set-selections
$ echo "sysvinit-utils install" | sudo dpkg --set-selections
$ echo "sysv-rc install" | sudo dpkg --set-selections
$ sudo apt-get install -t unstable systemd libpam-systemd libsystemd-daemon0 libsystemd-login0 bootlogd initscripts sysv-rc sysvinit sysvinit-utils

Only remark after upgrade as for the last time a normal reboot isn’t possible since the system already looks at the new location, but a `halt -f` solves that.

Internet, Unix en security

No smooth transition in Debian

Bugreport 638019 appears to be very straight forward, until the code finally hit Debian Testing last weekend. A simple relocation of a FIFO-buffer from /dev to /run caused direct trouble for machines with systemd and a normal shutdown wasn’t possible anymore. Both bugs 657979 and 657990 are a results of the modification. Seeing the overview of effected files and made me go back to the previous working release of source package sysvinit with the following commands

$ cd `xdg-user-dir DOWNLOAD`
$ wget
$ wget
$ wget
$ wget
$ wget
$ dpkg -i bootlogd_2.88dsf-18_amd64.deb initscripts_2.88dsf-18_amd64.deb sysvinit_2.88dsf-18_amd64.deb sysvinit-utils_2.88dsf-18_amd64.deb sysv-rc_2.88dsf-18_all.deb

And as there is no solution for now except a dependency change for systemd the package are being placed on hold like the last time they broke systemd.

$ echo "bootlogd hold" | sudo dpkg --set-selections
$ echo "initscripts hold" | sudo dpkg --set-selections
$ echo "sysvinit hold" | sudo dpkg --set-selections
$ echo "sysvinit-utils hold" | sudo dpkg --set-selections
$ echo "sysv-rc hold" | sudo dpkg --set-selections

It sounds strange for Linux-people, but I really wished I had an alternative boot environment like Solaris has. Maybe this is the reason for me to invest more time in read-write within BtrFS.