Extending GitHub Actions with Annotations#

With the introduction of GitHub Actions automated testing became more accessible and integrated into the pull requests to make it more clear what is being merged and if it checks off all requirements. This makes other services like Dependabot easy to use and keep your code up to date, but these small changes in dependencies for example. Reviewing code or documentation changes can be more difficult when a linter like yamllint or flake8 gives an error or warning as you have to dig into the logs to search for what is wrong.

GitHub Actions also support annotations that can be presented in the web interface to directly see which notifications there are including files and line numbers as shown below. This way feedback from a workflow executed by GitHub Actions is presented in the web interface.

Annotation in GitHub Actions

Using yamllint#

Some tools like yamllint have out-of-the-box support to generate output that GitHub can parse but it has to be enabled to generate the correct output. The example workflow below runs every time a push is done to the master branch or when a pull_request is made against the master branch. It installs the required tools like flake8 and yamllint, and then it executes yamllint.

Example of a basic CI workflow#
 1---
 2name: CI
 3
 4on:
 5  push:
 6    branches:
 7      - master
 8  pull_request:
 9    branches:
10      - master
11
12jobs:
13  lint:
14    name: Linting the Code Base
15    runs-on: ubuntu-latest
16
17    steps:
18      - name: Checkout Code
19        uses: actions/checkout@v3
20
21      - name: Install dependencies
22        run: |
23          python -m pip install --upgrade pip
24          pip install flake8 yamllint
25
26      - name: Lint with yamllint
27        run: |
28          yamllint .

The example workflow may fail, but all the output stays in the log file generated during the run and may get lost over time. Luckily yamllint supports different output formats and one of them is for GitHub. With the extra option --format github on line 34 this is enabled and GitHub will show all found annotations to the web interface.

Enabling GitHub formatting for yamllint#
32      - name: Lint with yamllint
33        run: |
34          yamllint . --format github

Using Flake8#

Another popular framework to perform styling checks for example is flake8 for Python. While this tool currently has formatting for a lot of CI solutions, GitHub Actions isn’t limited to any. Some GitHub Actions exist to enable annotations for GitHub Actions, but how do they work? The first step is to define a problem matcher file like in the example below that tells which lines should be marked as a warning or error, secondly, the regular expression to filter out the lines from the output, and the final step is mapping the found strings to the correct variables.

Example file .github/flake8-problem-matcher.json#
 1{
 2  "problemMatcher": [
 3    {
 4      "owner": "flake8-error",
 5      "severity": "error",
 6      "pattern": [
 7        {
 8          "regexp": "^([^:]+):(\\d+):(\\d+):\\s+(E\\d+\\s+.+)$",
 9          "file": 1,
10          "line": 2,
11          "column": 3,
12          "message": 4
13        }
14      ]
15    },
16    {
17      "owner": "flake8-warning",
18      "severity": "warning",
19      "pattern": [
20        {
21          "regexp": "^([^:]+):(\\d+):(\\d+):\\s+([CFNW]\\d+\\s+.+)$",
22          "file": 1,
23          "line": 2,
24          "column": 3,
25          "message": 4
26        }
27      ]
28    }
29  ]
30}

With the JSON file for the problem matcher created, the workflow can be extended by first telling GitHub Actions the include an additional problem matcher file as described on line 39. After this, the normal workflow for flake8 can be added to run certain tests. All the output generated by flake8 will be parsed by GitHub Actions and presented in the web interface.

Adding .github/flake8-problem-matcher.json to the CI-workflow#
38  - name: Add problem matcher
39    run: echo "::add-matcher::.github/flake8-problem-matcher.json"
40
41  - name: Lint with flake8
42    run: |
43      # stop the build if there are Python syntax errors or undefined names
44      flake8 . --count --select=E9,F63,F7,F82 --show-source --statistics
45      # exit-zero treats all errors as warnings. The GitHub editor is 127 chars wide
46      flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics

Using Sphinx#

Writing and generating documentation with Sphinx is another example of where annotations can help to review and troubleshoot failed builds. The JSON file is similar to the one in Using Flake8 with the only difference that no column variable will be set. The step for echo "::add-matcher::.github/sphinx-problem-matcher.json" has to be added to the workflow.

Example of .github/sphinx-problem-matcher.json#
 1{
 2  "problemMatcher": [
 3    {
 4      "owner": "sphinx-warning",
 5      "severity": "warning",
 6      "pattern": [
 7        {
 8          "regexp": "^(.*?):(\\d+): WARNING: (.*)$",
 9          "file": 1,
10          "line": 2,
11          "message": 3
12        }
13      ]
14    },
15    {
16      "owner": "sphinx-error",
17      "severity": "error",
18      "pattern": [
19        {
20          "regexp": "^(.*?):(\\d+): CRITICAL: (.*)$",
21          "file": 1,
22          "line": 2,
23          "message": 3
24        }
25      ]
26    }
27  ]
28}

Conclusions for GitHub Annotations#

Functionalities like annotations with GitHub Actions succeed or fail with support from the tools used by developers. CodeClimate tried a similar approach by capturing the output in a JSON format but doesn’t seem to have caught on. For GitHub, this may be different as there is a large community and other parts of the tools are also developed within the GitHub ecosystem.

Native support as comes with yamllint is still in the minority, but if the output is parseable like with flake8 and Sphinx one could make a separate filter or rely on the third-party action. But the latter comes with a risk that they may do more than set up your environment correctly to filter out the correct lines or become unmaintained. GitHub Actions can be extended very quickly to make it all happen.