How to use black, flake8, isort, and pre-commit framework to format Python codes

black: The Uncompromising Code Formatter

With black you can format Python code from 2.7 all the way to 3.8 (as of version 20.8b1), which makes for a great replacement for YAPF which can only format code depending on the Python version being used to run it.

My preference is using PEP 8 as my style guide, and so, 79-characters per line of code is what I use. So it’s as simple as running the following code at the root of my project and all non-compliant files will be reformatted:

$ black --line-length 79 --target-version py27 . 

Let’s explain each option.

• -l or --line-length: How many characters per line to allow. [default: 88]
• -t or --target-version: Python versions that should be supported by Black’s output. [default: per-file auto-detection]

Fairly simple. Allow 79 characters per line, and use py27 as the targetted version.

isort: A Python library to sort imports.

And just as their slogan states: “isort your imports, so you don’t have to.”

Command:

$ isort --multi-line 3 --profile black --python-version 27 .

The options used are mainly to be compatible with black (see here):

• --multi-line: Multiline output (0-grid, 1-vertical, 2-hanging, 3-vert-hanging, 4-vert-grid, 5-vert-grid-grouped, 6-vert-grid-grouped-no-comma, 7-noqa, 8-vertical-hanging-indent-bracket, 9-vertical-prefix-from-module-import, 10-hanging-indent-with-parentheses).
  • 3-vert-hanging
• --profile: Base profile type to use for configuration. Profiles include: black, django, pycharm, google, open_stack, plone, attrs, hug. As well as any shared profiles.
  • black
• --python-version: Tells isort to set the known standard library based on the specified Python version. Default is to assume any Python 3 version could be the target, and use a union of all stdlib modules across versions. If auto is specified, the version of the interpreter used to run isort (currently: 39) will be used.
  • 27 for Python 2.7

But there’s still something missing. black does not care about comments or docstrings, and isort cares even less, for obvious reasons; enter flake8.

flake8: A python tool that glues together pep8, pyflakes, mccabe, and third-party plugins to check the style and quality of some python code

Anthony Sottile (@asottile) has mentioned that he plans to drop support for Python 2.7 in future releases, maybe in version 3.9 or 4.0.

Fortunately, I can still use it for Python 2 by running the following command:

$ flake8 --max-doc-length=72 --ignore=E211,E999,F401,F821,W503

PEP 8 recommends limiting docstrings or comments to 72 characters, which is exactly what I’m using for flake8.

So let’s explain each option used.

• --max-doc-length: Maximum allowed doc line length for the entirety of this run. (Default: None)
• --ignore: Comma-separated list of errors and warnings to ignore (or skip). For example, --ignore=E4,E51,W234. (Default: [‘E226’, ‘E123’, ‘W504’, ‘E121’, ‘W503’, ‘E126’, ‘E704’, ‘E24’])

In my case, I am using 72 as the maximum allowed characters for my docstrings, in accordance with PEP 8, and ignoring the following errors:

• E211: whitespace before ‘(‘
  • Since in Python 2 print is not a function, black adds a space between the print statement from Python 2, and the opening parenthesis
• E999: SyntaxError: invalid syntax
  • In my case, this occurs again with the print statement where I am printing just one argument like this print arg
• F401: module imported but unused
  • I do import some modules in order to get “Intellisense” when I peek into the details in PyCharm
• F821: undefined name name
  • In one of my libraries I am checking if an argument is a string, and in order to cover my bases with plain strings (str) and Unicode, I found that using basestring would work for all characters, including non-Latin characters
• W503: line break before binary operator
  • It doesn’t like when binary operators are broken into multi-line statements

pre-commit: A framework for managing and maintaining multi-language pre-commit hooks.

Finally, let’s put it all together with pre-commit.

So in order to use flake8 you’ll have to create a .flake8 file. Mine looks like this:

[flake8]
ignore = E211, E999, F401, F821, W503
max-doc-length = 72

A pyproject.toml file that in my case looks like this:

[tool.black]
line-length = 79
target-version = ['py27']
[tool.isort]
profile = "black"
multi_line_output = 3 
py_version = 27

And finally my .pre-commit-config.yaml file:

repos:
  - repo: https://github.com/psf/black
    rev: 20.8b1
    hooks:
      - id: black
  - repo: https://github.com/PyCQA/isort
    rev: 5.7.0
    hooks:
      - id: isort
  - repo: https://github.com/PyCQA/flake8
    rev: 3.8.4
    hooks:
      - id: flake8

After you’ve configured all of this for the first time, first run the install command for pre-commit and to run tests I use run with the --all-files option, just like this:

$ pre-commit install
pre-commit installed at .git/hooks/pre-commit
$ pre-commit run --all-files
black....................................................................Passed
isort....................................................................Passed
flake8...................................................................Passed

So every time you try to commit something to your Git repo, all tests should be marked as Passed, otherwise, the commit will fail.

At the moment of writing this post both black and isort do support the use of pyproject.toml, something that flake8 still hasn’t been implemented unlike flake9 or FlakeHell, which I have not integrated into my workflow; I’m still using flake8 because I’ve installed it via Homebrew.

While you have the option to “pip-install” all of these tools, currently, I decided to use Homebrew because I don’t usually check if my packages are outdated, something that Homebrew contributors actually do with each new release. See: black, flake8, and isort, which will install python@3.9 as they all depend on it.

But if you do use pip, I recommend adding an alias for updating all of your outdated packages that should run the following command:

$ python -m pip list --outdated --format=freeze | grep -v '^\-e' | cut -d = -f 1 | xargs -n1 python -m pip install --upgrade

Automate Python workflow using pre-commits: black and flake8

Before I commit my staged Python files, black formats my code and flake8 checks my compliance to PEP8. If everything passes, the commit is made. If not, then I the perform necessary edits and commit again. Less time is spent on code formatting so I can focus more on code logic.

Code reviews are fun! They enable me to learn from others’ codes while providing an opportunity to teach what I know. However, there are still some things I wish to improve when facilitating reviews in my open-source projects:

• Less time commenting on code format, and more time discussing code logic
• Less hassle spotting format errors (“can you really see that trailing whitespace on Line 76?”)
• Stop sounding nitpicky (“Please put two blank lines between function definitions”)

If I could automate the processes above and remove the human-in-the-loop, we can focus more on code logic and implementation. Good thing, I learned about Git hooks, specifically pre-commit hooks. It enables you to automatically run a short script before committing. This script can be a checking tool or a formatter. If the script passes, then the commit is made, else, the commit is denied.

In this section, I’ll describe how I created a pre-commit pipeline in PySwarms using the black code formatter, flake8 checker, and the pre-commit Python framework. The entire pipeline looks like this:

Figure: Pre-commit pipeline with black and flake8 for checking my .py files

I’ll first discuss the pre-commit framework, then add components one-by-one: first is black, and then flake8. I will show the dotfiles present in my project, so feel free to adapt them into your own!

The pre-commit Python framework

We can run shell files all we want to dictate how our pre-commit process goes, but this pre-commit framework written in Python got us covered. It even comes with a set of pre-commit hooks out of the box (batteries included!). To adopt pre-commit it into our system, we simply perform the following actions:

1. Install pre-commit: pip install pre-commit
2. Add pre-commit to requirements.txt (or requirements-dev.txt)
3. Define .pre-commit-config.yaml with the hooks you want to include.
4. Execute pre-commit install to install git hooks in your .git/ directory.

The YAML file configures the sources which the hooks will be taken from. In our case, flake8’s already been included in this framework so we just need to specify its id. On the other hand, we need to define where to source black using a few lines of code. Below is a sample .pre-commit-config.yaml file that I use in my project:

repos:
-   repo: https://github.com/ambv/black
    rev: stable
    hooks:
    - id: black
      language_version: python3.6
-   repo: https://github.com/pre-commit/pre-commit-hooks
    rev: v1.2.3
    hooks:
    - id: flake8

Update (03-04-2020) You can also add flake8 from its own repo like so:

repos:
-   repo: https://github.com/ambv/black
    rev: stable
    hooks:
    - id: black
      language_version: python3.6
-   repo: https://gitlab.com/pycqa/flake8
    rev: 3.7.9
    hooks:
    - id: flake8

In the next section, I will discuss my code formatter (black) and checker (flake8). As usual, I will provide my config files for each component.

Code Formatter: black

The black code formatter in Python is an opinionated tool that formats your code in the best way possible. You can check its design decisions in the repository itself. Some notable formatting decisions, in my opinion:

• Unlike in PEP8, code length is 88 characters, not 79.
• Use of double-quotes than single-quotes in strings.
• If there are many function args, each arg will be wrapped per line.

I’d rather maintain the recommended 79-character length. Good thing, they have an option to do so. I just need to configure my pyproject.toml to line-length=79 and everything is all set. Here’s my .toml file for configuring black:

[tool.black]
line-length = 79
include = '\.pyi?$'
exclude = '''
/(
    \.git
  | \.hg
  | \.mypy_cache
  | \.tox
  | \.venv
  | _build
  | buck-out
  | build
  | dist
)/
'''

If you are not a fan of black, there’s always autopep8— a formatter more faithful to PEP8. Good thing, the pre-commit framework already has a hook on this tool, so there’s no need to source from another repository.

Flake8 checker

Flake8 is a powerful tool that checks our code’s compliance with PEP8. In order for black to work nicely with flake8 (or prevent it from spewing out various errors and warnings), we need to list down some error codes to ignore. You can check my .flake8 configuration below:

[flake8]
ignore = E203, E266, E501, W503, F403, F401
max-line-length = 79
max-complexity = 18
select = B,C,E,F,W,T4,B9

Results

So what we have is a pipeline that safeguards my project against wrongly-formatted code. On CONTRIBUTING page, I explicitly mentioned using pre-commits (or run flake8 and black on their code manually) before submitting a Pull Request.

Figure: Pre-commit pipeline with black and flake8 for checking my .py files

Now that we have a pre-commit framework set up with black and flake8, let’s see it in action! Here we’ll see how black formats a Python file automagically:

Figure: Short demo on pre-commit hooks

pre-commit

pre-commit is a framework for managing pre-commit hooks in Git. Oh, but what is Git Hook?

Git Hook is a script that is run automatically every time a specific event occurs in the Git repository.

In this case, the event here is the commit code. We will use the pre-commit hook to check the changes in the code of each style convention automatically before commit and integrating it into the system. If something goes wrong, the commit will fail and we’ll get the associated error messages to fix. Commit is only successful when no errors occur.

So far we have just introduced and run the above tools manually. Now it’s time to combine them to run automatically!

WORKFLOW WITH PRE-COMMIT HOOKS

overview

Workflow with pre-commit (Image taken from the end of post link)

Before implementing Git commit, I will use isort and black to format the code automatically, then use flake8 to check again with standard PEP8 (all configured by pre-commit ). The commit will be successful if there is no error. If an error occurs, we will go back and fix it where necessary and commit again. This workflow helps to reduce the time to reformat the code manually, thereby focusing more on the logic. Team working together is also easier and more efficient.

Set-up step by step

(These are the configuration files that I am using. You can customize them to suit your own style or needs!)

Create virtualvenv if needed and install pre-commit:

pip3 install pre-commit

Don’t forget to add it to requirements.txt or Pipfile after installation.

Configure pre-commit by creating a .pre-commit-config.yaml in the root directory with the following content (you can adjust to the latest version at the time of reading this article):

repos:
-   repo: https://github.com/pre-commit/pre-commit-hooks
    rev: v2.3.0
    hooks:
    -   id: check-yaml
    -   id: end-of-file-fixer
    -   id: trailing-whitespace

-   repo: https://github.com/asottile/seed-isort-config
    rev: v1.9.3
    hooks:
    -   id: seed-isort-config

-   repo: https://github.com/pre-commit/mirrors-isort
    rev: v4.3.21
    hooks:
    -   id: isort

-   repo: https://github.com/psf/black
    rev: 19.10b0
    hooks:
    -    id: black
            language_version: python3.6

-   repo: https://gitlab.com/pycqa/flake8
    rev: 3.8.3
    hooks:
    -   id: flake8

Configure isort by creating a .isort.cfg file in the root directory with the following content:

[settings]
line_length = 79
multi_line_output = 3
include_trailing_comma = True

If you have noticed in the pre-commit configuration file, along with isort, we use add seed-isort-config to automatically add packages to known_third_party in the isort configuration (instead of doing it manually). Configure black by creating a pyproject.toml file in the root directory with the following content:

[tool.black]
line-length = 79
include = '.pyi?$'
exclude = '''
/(
    .git
    | .hg
    | .mypy_cache
    | .tox
    | .venv
    | _build
    | buck-out
    | build
    | dist
)/
'''

Configure flake8 by creating .flake8 file with the following content:

[flake8]
    ignore = E203, E266, E501, W503, F403, F401
    max-line-length = 79
    max-complexity = 18
    select = B,C,E,F,W,T4,B9

After configuration is complete, run the following command to complete the installation:

pre-commit install

Finally, before executing Git commit, run the following command:

pre-commit run --all-files

As you can see, installing the above workflow is very easy for the team because all the configuration files are already in the project, members just need to run the pre-commit install command.

Other useful pip packages

• pip-autoremove for removing packages and all of their dependencies.
• pylint, a static code analysis tool that looks for programming errors helps enforce a coding standard, sniffs for code smells and offers simple refactoring suggestions.

References:

https://thecesrom.dev/2021/03/06/how-i-use-black-flake8-and-isort-to-format-python2-code/

https://ljvmiranda921.github.io/notebook/2018/06/21/precommits-using-black-and-flake8/

https://levelup.gitconnected.com/raise-the-bar-of-code-quality-in-python-projects-7c49743f004f

https://pre-commit.com/