https://dailystuff.nlDailystuff on the Internet - Posts tagged SQL2024-03-16T09:59:04.007581+00:00ABloghttps://dailystuff.nl/blog/2017/table-size-in-postgresql.htmlTable size in PostgreSQL2017-09-08T00:00:00+00:00Hans Spaans<section id="table-size-in-postgresql">
<p>Disk space seems endless, until you run out and/or have to pay the bill. The question is how to find tables with a high disk storage usage and with the query below it shows the <a class="reference external" href="https://www.postgresql.org/docs/current/functions-admin.html#FUNCTIONS-ADMIN-DBSIZE">table and index size</a>, but also the size of <a class="reference external" href="https://www.postgresql.org/docs/current/storage-toast.html">TOAST</a> data for PostgreSQL.</p>
<div class="highlight-PostgreSQL notranslate"><div class="highlight"><pre><span></span><span class="k">SELECT</span><span class="w"> </span><span class="n">schemaname</span><span class="p">,</span><span class="w"> </span><span class="n">tablename</span><span class="p">,</span>
<span class="w"> </span><span class="n">pg_size_pretty</span><span class="p">(</span><span class="n">tsize</span><span class="p">)</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="n">size_table</span><span class="p">,</span>
<span class="w"> </span><span class="n">pg_size_pretty</span><span class="p">(</span><span class="n">size</span><span class="p">)</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="n">size_index</span><span class="p">,</span>
<span class="w"> </span><span class="n">pg_size_pretty</span><span class="p">(</span><span class="n">total_size</span><span class="p">)</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="n">size_total</span>
<span class="k">FROM</span><span class="w"> </span><span class="p">(</span><span class="k">SELECT</span><span class="w"> </span><span class="o">*</span><span class="p">,</span>
<span class="w"> </span><span class="n">pg_table_size</span><span class="p">(</span><span class="n">schemaname</span><span class="o">||</span><span class="s1">'.'</span><span class="o">||</span><span class="n">tablename</span><span class="p">)</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="n">tsize</span><span class="p">,</span>
<span class="w"> </span><span class="n">pg_relation_size</span><span class="p">(</span><span class="n">schemaname</span><span class="o">||</span><span class="s1">'.'</span><span class="o">||</span><span class="n">tablename</span><span class="p">)</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="n">size</span><span class="p">,</span>
<span class="w"> </span><span class="n">pg_total_relation_size</span><span class="p">(</span><span class="n">schemaname</span><span class="o">||</span><span class="s1">'.'</span><span class="o">||</span><span class="n">tablename</span><span class="p">)</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="n">total_size</span>
<span class="w"> </span><span class="k">FROM</span><span class="w"> </span><span class="n">pg_tables</span><span class="p">)</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="k">TABLES</span>
<span class="k">WHERE</span><span class="w"> </span><span class="n">schemaname</span><span class="o">=</span><span class="s1">'public'</span>
<span class="k">ORDER</span><span class="w"> </span><span class="k">BY</span><span class="w"> </span><span class="n">total_size</span><span class="w"> </span><span class="k">DESC</span><span class="p">;</span>
</pre></div>
</div>
<p>After running this query on the development schema and exporting the results to CSV, we can see that a ManyToMany table consumes a total of 39 MB. With over 330.000 entries this seems numbers seem to be fine as the table size is in line with the amount of data stored in it.</p>
<div class="highlight-PostgreSQL notranslate"><div class="highlight"><pre><span></span><span class="n">schemaname</span><span class="p">,</span><span class="n">tablename</span><span class="p">,</span><span class="n">size_table</span><span class="p">,</span><span class="n">size_index</span><span class="p">,</span><span class="n">size_total</span>
<span class="n">public</span><span class="p">,</span><span class="n">domain_asset_domain_asset_group</span><span class="p">,</span><span class="mf">12</span><span class="w"> </span><span class="n">MB</span><span class="p">,</span><span class="mf">12</span><span class="w"> </span><span class="n">MB</span><span class="p">,</span><span class="mf">39</span><span class="w"> </span><span class="n">MB</span>
<span class="n">public</span><span class="p">,</span><span class="n">domain_account_domain_function</span><span class="p">,</span><span class="mf">2960</span><span class="w"> </span><span class="n">kB</span><span class="p">,</span><span class="mf">2936</span><span class="w"> </span><span class="n">kB</span><span class="p">,</span><span class="mf">9720</span><span class="w"> </span><span class="n">kB</span>
<span class="n">public</span><span class="p">,</span><span class="n">domain_account</span><span class="p">,</span><span class="mf">1760</span><span class="w"> </span><span class="n">kB</span><span class="p">,</span><span class="mf">1728</span><span class="w"> </span><span class="n">kB</span><span class="p">,</span><span class="mf">4088</span><span class="w"> </span><span class="n">kB</span>
<span class="n">public</span><span class="p">,</span><span class="n">domain_command</span><span class="p">,</span><span class="mf">2016</span><span class="w"> </span><span class="n">kB</span><span class="p">,</span><span class="mf">1992</span><span class="w"> </span><span class="n">kB</span><span class="p">,</span><span class="mf">3528</span><span class="w"> </span><span class="n">kB</span>
<span class="n">public</span><span class="p">,</span><span class="n">person</span><span class="p">,</span><span class="mf">832</span><span class="w"> </span><span class="n">kB</span><span class="p">,</span><span class="mf">792</span><span class="w"> </span><span class="n">kB</span><span class="p">,</span><span class="mf">1736</span><span class="w"> </span><span class="n">kB</span>
<span class="n">public</span><span class="p">,</span><span class="n">domain_command_collection</span><span class="p">,</span><span class="mf">712</span><span class="w"> </span><span class="n">kB</span><span class="p">,</span><span class="mf">688</span><span class="w"> </span><span class="n">kB</span><span class="p">,</span><span class="mf">1248</span><span class="w"> </span><span class="n">kB</span>
<span class="n">public</span><span class="p">,</span><span class="n">domain_asset_group</span><span class="p">,</span><span class="mf">648</span><span class="w"> </span><span class="n">kB</span><span class="p">,</span><span class="mf">624</span><span class="w"> </span><span class="n">kB</span><span class="p">,</span><span class="mf">1160</span><span class="w"> </span><span class="n">kB</span>
<span class="n">public</span><span class="p">,</span><span class="n">domain_asset</span><span class="p">,</span><span class="mf">544</span><span class="w"> </span><span class="n">kB</span><span class="p">,</span><span class="mf">520</span><span class="w"> </span><span class="n">kB</span><span class="p">,</span><span class="mf">1088</span><span class="w"> </span><span class="n">kB</span>
<span class="n">public</span><span class="p">,</span><span class="n">domain_function</span><span class="p">,</span><span class="mf">440</span><span class="w"> </span><span class="n">kB</span><span class="p">,</span><span class="mf">416</span><span class="w"> </span><span class="n">kB</span><span class="p">,</span><span class="mf">784</span><span class="w"> </span><span class="n">kB</span>
<span class="n">public</span><span class="p">,</span><span class="n">sessions</span><span class="p">,</span><span class="mf">64</span><span class="w"> </span><span class="n">kB</span><span class="p">,</span><span class="mf">32</span><span class="w"> </span><span class="n">kB</span><span class="p">,</span><span class="mf">80</span><span class="w"> </span><span class="n">kB</span>
<span class="n">public</span><span class="p">,</span><span class="n">asset_application</span><span class="p">,</span><span class="mf">8192</span><span class="w"> </span><span class="n">bytes</span><span class="p">,</span><span class="mf">8192</span><span class="w"> </span><span class="n">bytes</span><span class="p">,</span><span class="mf">56</span><span class="w"> </span><span class="n">kB</span>
<span class="n">public</span><span class="p">,</span><span class="n">domain_authority</span><span class="p">,</span><span class="mf">8192</span><span class="w"> </span><span class="n">bytes</span><span class="p">,</span><span class="mf">8192</span><span class="w"> </span><span class="n">bytes</span><span class="p">,</span><span class="mf">40</span><span class="w"> </span><span class="n">kB</span>
<span class="n">public</span><span class="p">,</span><span class="n">asset_function</span><span class="p">,</span><span class="mf">8192</span><span class="w"> </span><span class="n">bytes</span><span class="p">,</span><span class="mf">0</span><span class="w"> </span><span class="n">bytes</span><span class="p">,</span><span class="mf">24</span><span class="w"> </span><span class="n">kB</span>
</pre></div>
</div>
<p>Collecting this data and graphing it may help spot problems and predict storage needs. It may help DevOps teams to figure out if their databases are growing and with what speed.</p>
</section>
Disk space seems endless, until you run out and/or have to pay the bill. The question is how to find tables with a high disk storage usage and with the query below it shows the table and index size, but also the size of TOAST data for PostgreSQL.After running this query on the development schema and exporting the results to CSV, we can see that a ManyToMany table consumes a total of 39 MB. With over 330.000 entries this seems numbers seem to be fine as the table size is in line with the amount of data stored in it.2017-09-08T00:00:00+00:00https://dailystuff.nl/blog/2013/renaming-database-in-postgresql.htmlRenaming database in PostgreSQL2013-04-04T00:00:00+00:00Hans Spaans<section id="renaming-database-in-postgresql">
<p>Sometimes you have a system with legacy naming standards, but you really want to switch over to the new standard to keep all the scripting clean without some exceptions no one is going to remember in 12 months. Oracle had the command <strong>ALTER DATABASE</strong>, but since Oracle 10 you need to take the database offline and do some magic. MySQL got the <strong>RENAME DATABASE</strong> option with release 5.1.7 and lost the <a class="reference external" href="https://dev.mysql.com/doc/refman/5.1/en/rename-database.html">option</a> again with release 5.1.23 as it was eating data.</p>
<p>Luckily PostgreSQL still has the command <strong>ALTER DATABASE</strong> so let’s rename a database and the owner. Before we start we need the password and then we need to log in as the PostgreSQL superuser <code class="docutils literal notranslate"><span class="pre">postgres</span></code> or another account with similar privileges. So first we check the database name and owner.</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">postgres</span><span class="o">=</span><span class="c1"># \l</span>
<span class="n">List</span> <span class="n">of</span> <span class="n">databases</span>
<span class="n">Name</span> <span class="o">|</span> <span class="n">Owner</span> <span class="o">|</span> <span class="n">Encoding</span> <span class="o">|</span> <span class="n">Collation</span> <span class="o">|</span> <span class="n">Ctype</span> <span class="o">|</span> <span class="n">Access</span> <span class="n">privileges</span>
<span class="o">--------------+----------+-----------+-------------+-------------+-----------------------</span>
<span class="n">dbu0001</span> <span class="o">|</span> <span class="n">dbu0001</span> <span class="o">|</span> <span class="n">UTF8</span> <span class="o">|</span> <span class="n">en_US</span><span class="o">.</span><span class="n">UTF</span><span class="o">-</span><span class="mi">8</span> <span class="o">|</span> <span class="n">en_US</span><span class="o">.</span><span class="n">UTF</span><span class="o">-</span><span class="mi">8</span> <span class="o">|</span>
</pre></div>
</div>
<p>Now we rename the database owner as we made a typo and we need to set the password again.</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">postgres</span><span class="o">=</span><span class="c1"># alter user dbu0001 rename to dbo0001;</span>
<span class="n">NOTICE</span><span class="p">:</span> <span class="n">MD5</span> <span class="n">password</span> <span class="n">cleared</span> <span class="n">because</span> <span class="n">of</span> <span class="n">role</span> <span class="n">rename</span>
<span class="n">ALTER</span> <span class="n">ROLE</span>
<span class="n">postgres</span><span class="o">=</span><span class="c1"># alter user dbo0001 password 'yeaxaureiraeLohsh6deJ2ohngahpu9a';</span>
<span class="n">ALTER</span> <span class="n">ROLE</span>
</pre></div>
</div>
<p>The second task is to rename the database to the correct name.</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">postgres</span><span class="o">=</span><span class="c1"># alter database dbu0001 rename to dbs0001;</span>
<span class="n">ALTER</span> <span class="n">DATABASE</span>
</pre></div>
</div>
<p>And basically, we are now done as the ownership was already modified when we renamed the account with our first statement, but let’s check what PostgreSQL says it now has.</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">postgres</span><span class="o">=</span><span class="c1"># \l</span>
<span class="n">List</span> <span class="n">of</span> <span class="n">databases</span>
<span class="n">Name</span> <span class="o">|</span> <span class="n">Owner</span> <span class="o">|</span> <span class="n">Encoding</span> <span class="o">|</span> <span class="n">Collation</span> <span class="o">|</span> <span class="n">Ctype</span> <span class="o">|</span> <span class="n">Access</span> <span class="n">privileges</span>
<span class="o">--------------+----------+-----------+-------------+-------------+-----------------------</span>
<span class="n">dbs0001</span> <span class="o">|</span> <span class="n">dbo0001</span> <span class="o">|</span> <span class="n">UTF8</span> <span class="o">|</span> <span class="n">en_US</span><span class="o">.</span><span class="n">UTF</span><span class="o">-</span><span class="mi">8</span> <span class="o">|</span> <span class="n">en_US</span><span class="o">.</span><span class="n">UTF</span><span class="o">-</span><span class="mi">8</span> <span class="o">|</span>
</pre></div>
</div>
<div class="admonition warning">
<p class="admonition-title">Warning</p>
<p>Don’t forget to update the connection strings for applications using this database and maybe <strong>GRANTS</strong> that have been set and the <em>pg_hba.conf</em> file.</p>
</div>
</section>
Sometimes you have a system with legacy naming standards, but you really want to switch over to the new standard to keep all the scripting clean without some exceptions no one is going to remember in 12 months. Oracle had the command ALTER DATABASE, but since Oracle 10 you need to take the database offline and do some magic. MySQL got the RENAME DATABASE option with release 5.1.7 and lost the option again with release 5.1.23 as it was eating data.Luckily PostgreSQL still has the command ALTER DATABASE so let’s rename a database and the owner. Before we start we need the password and then we need to log in as the PostgreSQL superuser postgres or another account with similar privileges. So first we check the database name and owner.2013-04-04T00:00:00+00:00https://dailystuff.nl/blog/2013/starting-to-stop-sql-injections-part-2.htmlStarting to stop SQL injections, part 22013-02-11T00:00:00+00:00Hans Spaans<section id="starting-to-stop-sql-injections-part-2">
<p>In a previous <a class="reference internal" href="../2012/starting-to-stop-sql-injections.html#starting-to-stop-sql-injections"><span class="std std-ref">posting</span></a>, I gave an example of how to make database queries safer by using parameter binding and basically stopping SQL injections. The next step is to make the code more readable and maintainable. This doesn’t sound like a priority for secure software development, but readable code is also code that can be verified and maintained by other people. It gives you the edge to debug problems quickly and invites others to supply patches. So let’s take the example where the previous posting ended.</p>
<div class="highlight-php notranslate"><div class="highlight"><pre><span></span><span class="cp"><?php</span>
<span class="nv">$sth</span> <span class="o">=</span> <span class="nv">$dbh</span><span class="o">-></span><span class="na">prepare</span><span class="p">(</span><span class="s1">'select userid from accounts where username = ?'</span><span class="p">);</span>
<span class="nv">$sth</span><span class="o">-></span><span class="na">execute</span><span class="p">(</span><span class="k">array</span><span class="p">(</span><span class="nv">$form_username</span><span class="p">));</span>
</pre></div>
</div>
<p>For one or two parameters this may work, but when queries become bigger you need to start counting, and counting beyond three is a bad idea in most cases. Let’s change the question mark with a named variable called <em>‘:username’</em> in this example. One could then use the function <a class="reference external" href="https://www.php.net/manual/en/pdostatement.bindparam.php">bindParam()</a> to specify which named variable needs to be replaced and has additional features, but in this example, we use the standard binding during in execute phase.</p>
<div class="highlight-php notranslate"><div class="highlight"><pre><span></span><span class="cp"><?php</span>
<span class="nv">$sth</span> <span class="o">=</span> <span class="nv">$dbh</span><span class="o">-></span><span class="na">prepare</span><span class="p">(</span><span class="s1">'select userid from accounts where username = :username'</span><span class="p">);</span>
<span class="nv">$sth</span><span class="o">-></span><span class="na">execute</span><span class="p">(</span><span class="k">array</span><span class="p">(</span><span class="s1">':username'</span><span class="o">=></span><span class="nv">$form_username</span><span class="p">));</span>
</pre></div>
</div>
<p>Please remember to use a named variable only once in a SQL query as it will only be replaced one time and not multiple times.</p>
</section>
In a previous posting, I gave an example of how to make database queries safer by using parameter binding and basically stopping SQL injections. The next step is to make the code more readable and maintainable. This doesn’t sound like a priority for secure software development, but readable code is also code that can be verified and maintained by other people. It gives you the edge to debug problems quickly and invites others to supply patches. So let’s take the example where the previous posting ended.For one or two parameters this may work, but when queries become bigger you need to start counting, and counting beyond three is a bad idea in most cases. Let’s change the question mark with a named variable called ‘:username’ in this example. One could then use the function bindParam() to specify which named variable needs to be replaced and has additional features, but in this example, we use the standard binding during in execute phase.2013-02-11T00:00:00+00:00https://dailystuff.nl/blog/2012/starting-to-stop-sql-injections.htmlStarting to stop SQL injections2012-12-10T00:00:00+00:00Hans Spaans<section id="starting-to-stop-sql-injections">
<p>In a lot of examples about PHP, strings are concatenated before a database query is executed as below. Some examples advise to use PHP-functions <a class="reference external" href="https://www.php.net/mysql_real_escape_string">mysql_real_escape_string()</a> and/or <a class="reference external" href="https://www.php.net/addslashes">addslashes()</a> to make database query safe against SQL-injections. But this isn’t really a solution as when using addslashes() also requires the use of <a class="reference external" href="https://www.php.net/stripslashes">stripslashes()</a> after retrieving data from a database. Some sites show the lack of proper implementation and show the famous <strong>'</strong> string on a website.</p>
<div class="highlight-php notranslate"><div class="highlight"><pre><span></span><span class="cp"><?php</span>
<span class="nv">$sth</span> <span class="o">=</span> <span class="nv">$dbh</span><span class="o">-></span><span class="na">prepare</span><span class="p">(</span><span class="s1">'select userid from accounts where username = "'</span><span class="o">.</span><span class="nv">$form_username</span><span class="o">.</span><span class="s1">'"'</span><span class="p">);</span>
<span class="nv">$sth</span><span class="o">-></span><span class="na">execute</span><span class="p">();</span>
</pre></div>
</div>
<p>Like in Perl with DBI, also PHP has PDO that allows for variables to be parameterized while executing a query as in the example below. This removes the need for homemade solutions that don’t cover all use-cases and allows for a way to provide a stable and more secure interface for your applications when communicating with databases.</p>
<div class="highlight-php notranslate"><div class="highlight"><pre><span></span><span class="cp"><?php</span>
<span class="nv">$sth</span> <span class="o">=</span> <span class="nv">$dbh</span><span class="o">-></span><span class="na">prepare</span><span class="p">(</span><span class="s1">'select userid from accounts where username = ?'</span><span class="p">);</span>
<span class="nv">$sth</span><span class="o">-></span><span class="na">execute</span><span class="p">(</span><span class="k">array</span><span class="p">(</span><span class="nv">$form_username</span><span class="p">));</span>
</pre></div>
</div>
<p>This doesn’t stop the need for sanitizing variables like input from users.</p>
</section>
In a lot of examples about PHP, strings are concatenated before a database query is executed as below. Some examples advise to use PHP-functions mysql_real_escape_string() and/or addslashes() to make database query safe against SQL-injections. But this isn’t really a solution as when using addslashes() also requires the use of stripslashes() after retrieving data from a database. Some sites show the lack of proper implementation and show the famous ' string on a website.Like in Perl with DBI, also PHP has PDO that allows for variables to be parameterized while executing a query as in the example below. This removes the need for homemade solutions that don’t cover all use-cases and allows for a way to provide a stable and more secure interface for your applications when communicating with databases.2012-12-10T00:00:00+00:00