Extending GitHub Actions with Annotations#
With the introduction of GitHub Actions automated testing became more accessible and integrated into the pull requests to make it more clear what is being merged and if it checks off all requirements. This makes other services like Dependabot easy to use and keep your code up to date, but these small changes in dependencies for example. Reviewing code or documentation changes can be more difficult when a linter like yamllint or flake8 gives an error or warning as you have to dig into the logs to search for what is wrong.
GitHub Actions also support annotations that can be presented in the web interface to directly see which notifications there are including files and line numbers as shown below. This way feedback from a workflow executed by GitHub Actions is presented in the web interface.

Using yamllint#
Some tools like yamllint have out-of-the-box support to generate output that GitHub can parse but it has to be enabled to generate the correct output. The example workflow below runs every time a push is done to the master branch or when a pull_request is made against the master branch. It installs the required tools like flake8 and yamllint, and then it executes yamllint.
1---
2name: CI
3
4on:
5 push:
6 branches:
7 - master
8 pull_request:
9 branches:
10 - master
11
12jobs:
13 lint:
14 name: Linting the Code Base
15 runs-on: ubuntu-latest
16
17 steps:
18 - name: Checkout Code
19 uses: actions/checkout@v3
20
21 - name: Install dependencies
22 run: |
23 python -m pip install --upgrade pip
24 pip install flake8 yamllint
25
26 - name: Lint with yamllint
27 run: |
28 yamllint .
The example workflow may fail, but all the output stays in the log file generated during the run and may get lost over time. Luckily yamllint supports different output formats and one of them is for GitHub. With the extra option --format github
on line 34 this is enabled and GitHub will show all found annotations to the web interface.
32 - name: Lint with yamllint
33 run: |
34 yamllint . --format github
Using Flake8#
Another popular framework to perform styling checks for example is flake8 for Python. While this tool currently has formatting for a lot of CI solutions, GitHub Actions isn’t limited to any. Some GitHub Actions exist to enable annotations for GitHub Actions, but how do they work? The first step is to define a problem matcher file like in the example below that tells which lines should be marked as a warning or error, secondly, the regular expression to filter out the lines from the output, and the final step is mapping the found strings to the correct variables.
.github/flake8-problem-matcher.json
# 1{
2 "problemMatcher": [
3 {
4 "owner": "flake8-error",
5 "severity": "error",
6 "pattern": [
7 {
8 "regexp": "^([^:]+):(\\d+):(\\d+):\\s+(E\\d+\\s+.+)$",
9 "file": 1,
10 "line": 2,
11 "column": 3,
12 "message": 4
13 }
14 ]
15 },
16 {
17 "owner": "flake8-warning",
18 "severity": "warning",
19 "pattern": [
20 {
21 "regexp": "^([^:]+):(\\d+):(\\d+):\\s+([CFNW]\\d+\\s+.+)$",
22 "file": 1,
23 "line": 2,
24 "column": 3,
25 "message": 4
26 }
27 ]
28 }
29 ]
30}
With the JSON file for the problem matcher created, the workflow can be extended by first telling GitHub Actions the include an additional problem matcher file as described on line 39. After this, the normal workflow for flake8 can be added to run certain tests. All the output generated by flake8 will be parsed by GitHub Actions and presented in the web interface.
.github/flake8-problem-matcher.json
to the CI-workflow#38 - name: Add problem matcher
39 run: echo "::add-matcher::.github/flake8-problem-matcher.json"
40
41 - name: Lint with flake8
42 run: |
43 # stop the build if there are Python syntax errors or undefined names
44 flake8 . --count --select=E9,F63,F7,F82 --show-source --statistics
45 # exit-zero treats all errors as warnings. The GitHub editor is 127 chars wide
46 flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics
Using Sphinx#
Writing and generating documentation with Sphinx is another example of where annotations can help to review and troubleshoot failed builds. The JSON file is similar to the one in Using Flake8 with the only difference that no column variable will be set. The step for echo "::add-matcher::.github/sphinx-problem-matcher.json"
has to be added to the workflow.
.github/sphinx-problem-matcher.json
# 1{
2 "problemMatcher": [
3 {
4 "owner": "sphinx-warning",
5 "severity": "warning",
6 "pattern": [
7 {
8 "regexp": "^(.*?):(\\d+): WARNING: (.*)$",
9 "file": 1,
10 "line": 2,
11 "message": 3
12 }
13 ]
14 },
15 {
16 "owner": "sphinx-error",
17 "severity": "error",
18 "pattern": [
19 {
20 "regexp": "^(.*?):(\\d+): CRITICAL: (.*)$",
21 "file": 1,
22 "line": 2,
23 "message": 3
24 }
25 ]
26 }
27 ]
28}
Conclusions for GitHub Annotations#
Functionalities like annotations with GitHub Actions succeed or fail with support from the tools used by developers. CodeClimate tried a similar approach by capturing the output in a JSON format but doesn’t seem to have caught on. For GitHub, this may be different as there is a large community and other parts of the tools are also developed within the GitHub ecosystem.
Native support as comes with yamllint is still in the minority, but if the output is parseable like with flake8 and Sphinx one could make a separate filter or rely on the third-party action. But the latter comes with a risk that they may do more than set up your environment correctly to filter out the correct lines or become unmaintained. GitHub Actions can be extended very quickly to make it all happen.