System Administration

mount: unknown filesystem type ‘exfat’

exFAT has been chosen by the SD Card Association as the standard file system for SDXC cards with 32 GiB or more of storage. Sadly the Fedora Project has chosen not to bundle support for exFAT due to patent issues. A free implementation of exFAT has been made and is available via RPMFusion Free for RPM-based systems.

$ sudo dnf -y install$(rpm -E %fedora).noarch.rpm 
$ sudo dnf -y install fuse-exfat

If you now try to mount your SD-card in Nautilus for example it should mount your drive. The performance should also be better than with NTFS as there is less overhead.

System Administration

Percent Lifetime Used attribute for SSDs

Solid-state drives sound ideal as they have no spinning parts and are very quiet, but they have a limited lifespan as you can’t write a memory cell only an X amount of times. But how to check your SSD on Linux to see if it is still in good shape? S.M.A.R.T. has become the standard for disk health years ago and can be queried by smartctl. So if we query for the health status and show all available attributes we get a good overview.

$ sudo smartctl -AH /dev/sda
smartctl 6.5 2016-05-07 r4318 [x86_64-linux-4.14.8-300.fc27.x86_64] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke,

SMART overall-health self-assessment test result: PASSED

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
  1 Raw_Read_Error_Rate     0x002f   100   100   000    Pre-fail  Always       -       0
  5 Reallocate_NAND_Blk_Cnt 0x0033   100   100   000    Pre-fail  Always       -       0
  9 Power_On_Hours          0x0032   100   100   000    Old_age   Always       -       26329
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       164
171 Program_Fail_Count      0x0032   100   100   000    Old_age   Always       -       0
172 Erase_Fail_Count        0x0032   100   100   000    Old_age   Always       -       0
173 Ave_Block-Erase_Count   0x0032   010   010   000    Old_age   Always       -       2708
174 Unexpect_Power_Loss_Ct  0x0032   100   100   000    Old_age   Always       -       79
180 Unused_Reserve_NAND_Blk 0x0033   000   000   000    Pre-fail  Always       -       4403
183 SATA_Interfac_Downshift 0x0032   100   100   000    Old_age   Always       -       0
184 Error_Correction_Count  0x0032   100   100   000    Old_age   Always       -       0
187 Reported_Uncorrect      0x0032   100   100   000    Old_age   Always       -       0
194 Temperature_Celsius     0x0022   048   036   000    Old_age   Always       -       52 (Min/Max 21/64)
196 Reallocated_Event_Count 0x0032   100   100   000    Old_age   Always       -       16
197 Current_Pending_Sector  0x0032   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   100   100   000    Old_age   Always       -       1
202 Percent_Lifetime_Used   0x0031   010   010   000    Pre-fail  Offline      -       90
206 Write_Error_Rate        0x000e   100   100   000    Old_age   Always       -       0
210 Success_RAIN_Recov_Cnt  0x0032   100   100   000    Old_age   Always       -       0
246 Total_Host_Sector_Write 0x0032   100   100   000    Old_age   Always       -       115599678623
247 Host_Program_Page_Count 0x0032   100   100   000    Old_age   Always       -       2235969936
248 Bckgnd_Program_Page_Cnt 0x0032   100   100   000    Old_age   Always       -       3472676821

The most interesting attributes are 202 on how much lifetime is left, but also 5, 180, and 9 that show you the number of replaced storage cells and how many hours the disk has been running. If attributes 5 and 180 are changing it is most definitely time to replace this solid-state drive as memory cells have been worn out.

System Administration

Removing SPF Resource Records

With the creation of RFC 4408 also new a record type 99 for DNS was created to identify SPF Resource Records. It was advised to have both TXT and SPF records in DNS with the same content.  RFC 4408 was obsoleted by RFC 7208 in 2014 with paragraph 3.1 stating the following:

SPF records MUST be published as a DNS TXT (type 16) Resource Record (RR) [RFC1035] only. The character content of the record is encoded as [US-ASCII]. Use of alternative DNS RR types was supported in SPF’s experimental phase but has been discontinued.

RFC 7208, paragraph 3.1

Now that the SPF Resource Record has been discontinued for a while, the time has come to remove it from DNS (if not done already) and make sure it never comes back. Luckily most code libraries already preferred the TXT variant, but still, this is one to put on the maintenance checklist to remove it for any application code and/or infrastructure.

System Administration

Upgrading from CentOS 7.3 to 7.4

Last month CentOS 7.4 was announced and it was time to rebuild some servers from scratch to make sure all playbooks were still correct as it is always good to know you can quickly (re)build servers when needed. For some other servers, the impact would be big due to huge amounts of data that needed to be moved around and an in-place upgrade would be sufficient.

Upgrading is very straightforward as it the same as the update option with “–obsoletes” flag set which removes obsolete packages. So let start with CentOS 7.3.

$ cat /etc/redhat-release 
CentOS Linux release 7.3.1611 (Core)'

First, make sure all cached data is purged so we get the latest information. Secondly, run yum upgrade to download all required packages and install them. And thirdly, reboot your system.

$ sudo yum clean all
$ sudo yum upgrade
$ sudo systemctl reboot

When logging back into the system it presents itself as a server running CentOS 7.4.

$ cat /etc/redhat-release 
CentOS Linux release 7.4.1708 (Core)

A side note to this procedure is not to use it on a server that has running applications and/or has users connecting to it. They may experience errors/failures during the upgrade procedure as libraries may not match what an application expects to start correctly or an essential service that is not running during the upgrade.

System Administration

Table size in PostgreSQL

Disk space seems endless, until you run out and/or have to pay the bill. The question is how to find tables with a high disk storage usage and with the query below it shows the table and index size, but also the size of TOAST data for PostgreSQL.

SELECT schemaname, tablename,
  pg_size_pretty(tsize) AS size_table,
  pg_size_pretty(size) AS size_index,
  pg_size_pretty(total_size) AS size_total
        pg_table_size(schemaname||'.'||tablename) AS tsize,
        pg_relation_size(schemaname||'.'||tablename) AS size,
        pg_total_relation_size(schemaname||'.'||tablename) AS total_size
      FROM pg_tables) AS TABLES
WHERE schemaname='public'
ORDER BY total_size DESC;

After running this query on the development schema and exporting the results to CSV, we can see that a ManyToMany table consumes a total of 39 MB. With over 330.000 entries this seems numbers seem to be fine as the table size is in line with the amount of data stored in it.

public,domain_asset_domain_asset_group,12 MB,12 MB,39 MB
public,domain_account_domain_function,2960 kB,2936 kB,9720 kB
public,domain_account,1760 kB,1728 kB,4088 kB
public,domain_command,2016 kB,1992 kB,3528 kB
public,person,832 kB,792 kB,1736 kB
public,domain_command_collection,712 kB,688 kB,1248 kB
public,domain_asset_group,648 kB,624 kB,1160 kB
public,domain_asset,544 kB,520 kB,1088 kB
public,domain_function,440 kB,416 kB,784 kB
public,sessions,64 kB,32 kB,80 kB
public,asset_application,8192 bytes,8192 bytes,56 kB
public,domain_authority,8192 bytes,8192 bytes,40 kB
public,asset_function,8192 bytes,0 bytes,24 kB

Collecting this data and graphing it may help spot problems and predict storage needs. It may help DevOps teams to figure out if their databases are growing and with what speed.