Django Test Runner plugin for Kiwi TCMS Installation Configuration and environment Usage Changelog v10.0 (02 Mar 2021) v9.0 (13 Jan 2021) v1.1.3 (28 October 2020) v1.1.2 (25 June 2020) v1.1.1 (25 June 2020). A test runner is a class defining a runtests method. Django ships with a DiscoverRunner class that defines the default Django testing behavior. This class defines the runtests entry point, plus a selection of other methods that are used by runtests. In this tutorial, you created a test suite in your Django project, added test cases to test model and view logic, learned how to run tests, and analyzed the test output. As a next step, you can create new test scripts for Python code not in models.py and views.py.

Released:

django test runner to use py.test tests

Project description

This project allows you to use py.test as a django test runner, instead of thedefault test runner.

To use it, add it to your python path and add django_pytest to your installedapps. Also set the TEST_RUNNER = ‘django_pytest.test_runner.run_tests’ setting.If you’re using Django 1.3 or newer setTEST_RUNNER = ‘django_pytest.test_runner.TestRunner’ or Django will printdeprecation warnings each time you run your tests.

Also create a conftest.py in your project directory and include:

from django_pytest.conftest import pytest_funcarg__client, pytest_funcarg__django_client

You can also use:

from django_pytest.auth_funcargs import pytest_funcarg__user, pytest_funcarg__groups

to import a user or some groups with users in them

Now anywhere in your project, you can create files calledtest_<something>.py. These are standard py.test test files. Use the funcargclient in every test to both instantiate a test database that is clearedafter each test and to provide you with a django test client object identicalto the one used in django’s test system. For example:

def test_filter(client):response = client.get(‘/browse/’, {‘filter’: ‘1’})assert response.status_code 200

Use ./manage.py test to run the py.test test runs (ie: it replaces thestandard django test runner). You can pass py.test options to the commandand they will be forwarded to py.test. (Technically, I haven’t got it passingall options, just the most common ones I use)

The management command has been set up so that syncdb will use the django coresyncdb if SOUTH_TESTS_MIGRATE is set to False, if south is installed. Thisprevents migrations from running when running unit tests. This speeds up testsetup significantly, but it means your test db may not be identical toproduction, if you have faulty migrations.

py.test automatically picks up any subclasses of unittest.TestCase, providedthey are in a module named test_<something>.py. Thus, all your existing djangounittests should work seemlessly with py.test, although you may have to renameyour test files if they do not conform to this convention. You can also writecustom py.test test collection hooks to pick up test modules that are named ina different directory structure.

This project differs from <http://github.com/bfirsh/pytest_django> in that itprovides a django test runner that calls py.test, rather than creating apy.test plugin to test django projects. I believe there is overlappingfunctionality from the two projects, and also that they can be integrated intoa single project, but I have not looked at the feasibility of this yet.

Release historyRelease notifications | RSS feed

0.2.0

0.1.5

0.1.4

0.1.3

0.1.2

0.1.1

0.1.0

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Hashes for django-pytest-0.2.0.tar.gz

This is a blog post version of the talk I gave at DjangoCon Australia 2020 today.The video is on YouTube and the slides are on GitHub (including full example code).

You run your tests with manage.py test.You know what happens inside your tests, since you write them.But how does the test runner work to execute them, and put the dots, E’s, and F’s on your screen?

When you learn how Django middleware works, you unlock a huge number of use cases, such as changing cookies, setting global headers, and logging requests.Similarly, learning how your tests run will help you customize the process, for example loading tests in a different order, configuring test settings without a separate file, or blocking outgoing HTTP requests.

In this post, we’ll make a vital customization of our test run’s output - we’ll swap the “dots and letters” default style to use emojis to represent test success, failure, etc:

But before we can write that, we need to deconstruct the testing process.

Test Output¶

Let’s investigate the output from a test run.We’ll use a project containing only this vacuous test:

When we run the tests, we get some familiar output:

To investigate what’s going on, we can ask for more detail by increasing the verbosity to maximum with -v 3:

Okay great, that’s plenty!Let’s take it apart.

The first line, “Creating test database…”, is Django reporting the creation of a test database.If your project uses multiple databases, you’ll see one line per database.

I’m using SQLite in this example project, so Django has automatically set mode=memory in the database address.This makes database operations about ten times faster.Other databases like PostgreSQL don’t have such modes, but there are other techniques to run them in-memory.

Django Test Teardown

The second “Operations to perform” line and several following lines are the output of the migrate command on our test databases.This output is identical to what we’d see running manage.py migrate on an empty database.Here I’m using a small project with no migrations - on a typical project you’d see one line for each migration.

After that, we have the line saying “System check identified no issues”.This output is from Django’s system check framework, which runs a number of “preflight checks” to ensure your project is well configured.You can run it alone with manage.py check, and it also runs automatically in most management commands.Typically it’s the first step before a management command, but for tests it’s deferred until after the test databases are ready, since some checks use database connections.

You can write your own checks to detect configuration bugs.Because they run first, they’re sometimes a better fit than writing a test.I’d love to go into more detail, but that would be another post’s worth of content.

The following lines cover our tests.By default the test runner only prints a single character per test, but with a higher verbosity Django prints a whole line per test.Here we only have one test, “test_one”, and as it finished running, the test runner appended its status “ok” to the line.

To signify the end of the run, there’s a divider made with many dashes: “-----”.If we had any failures or errors, their stack traces would appear before this divider.This is followed by a summary of the tests that ran, their total runtime, and “OK” to indicate the test run was successful.

The final line reports the destruction of our test database.

This gives us a rough order of steps in a test run:

1. Create the test databases.
2. Migrate the databases.
3. Run the system checks.
4. Run the tests.
5. Report on the test count and success/failure.
6. Destroy the test databases.

Let’s track down which components inside Django are responsible for these steps.

Django and unittest¶

As you may be aware, Django’s test framework extends the unittest framework from the Python standard library.Every component responsible for the above steps is either built in to unittest, or one of Django’s extensions.We can represent this with a basic diagram:

We can find the components on each side by tracing through the code.

The “test” Management Command¶

The first place to look is the test management command, which Django finds and executes when we run manage.py test.This lives in django.core.management.commands.test.

As management commands go, it’s quite short - under 100 lines.Its handle() method is mostly concerned with handing off to a a “Test Runner”.Simplifying it down to three key lines:

(Full source.)

So what’s this TestRunner class?It’s a Django component that coordinates the test process.It’s customizable, but the default class, and the only one in Django itself, is django.test.runner.DiscoverRunner.Let’s look at that next!

The DiscoverRunner Class¶

DiscoverRunner is the main coordinator of the test process.It handles adding extra arguments to the management command, creating and handing off to sub-components, and doing some environment setup.

It starts like this:

(Documentation, Full source.)

These class level attributes point to other classes that perform different steps in the test process.You can see most of them are unittest components.

Note that one of them is called test_runner, so we have two distinct concepts called “test runner” - Django’s DiscoverRunner and unittest’s TextTestRunner.DiscoverRunner does a lot more than TextTestRunner and has a different interface.Perhaps Django could have used a different name for DiscoverRunner, like TestCoordinator, but it’s probably too late to change that now.

The main flow in DiscoverRunner is in its run_tests() method.Stripping out a bunch of details, run_tests() looks something like this:

That’s quite a few steps!Many of the called methods correspond with steps on our above list:

• setup_databases() creates of the test databases.This only creates the databases necessary for the selected tests, as filtered by get_databases(), so if you run only database-free SimpleTestCases, Django doesn’t create any databases.Inside it both creates the databases and runs the migrate command.
• run_checks(), well, runs the checks.
• run_suite() runs the test suite, including all the output.
• teardown_databases() destroys the test databases.

A couple of other methods are things we may have expected:

• setup_test_environment() and teardown_test_environment() set and unset some settings, such as the local email backend.
• suite_result() returns the number of failures to the test management command.

All these methods are useful to investigate for customizing those parts of the test process.But they’re all part of Django itself.The other methods hand off to components in unittest - build_suite() and run_suite().Let’s investigate those in turn.

build_suite()¶

build_suite() is concerned with finding the tests to run, and putting them into a “suite” object.It’s again a long method, but simplified, it looks something like this:

This method uses three of the four classes that we saw DiscoverRunner refers to:

• test_suite - a unittest component that acts as a container for tests to run.
• parallel_test_suite - a wrapper around a test suite used with Django’s parallel testing feature.
• test_loader - a unittest component that can find test modules on disk, and load them into a suite.

run_suite()¶

The other DiscoverRunner method to look into is run_suite().We don’t need to simplify this one - its entire implementation looks like this:

Its only concern is constructing a test runner and telling it to run the constructed test suite.This the final one of the unittest components referred to by a class attribute.It uses unittest.TextTestRunner, which is the default runner for outputting results as text, as opposed to, for example, an XML file to communicate results to your CI system.

Let’s finish our investigation by looking inside that class.

The TextTestRunner Class¶

This component inside unittest takes a test case or suite, and executes it.It starts:

(Full source.)

Similarly to DiscoverRunner, it uses a class-level attribute to refer to another class.The default TextTestResult class is the thing that actually writes the text-based output.Unlike DiscoverRunner’s class references, we can override resultclass by passing an alternative to TextTestRunner.__init__().

We’re now ready to customize the test process.But first, let’s review our investigation.

A Map¶

We can now expand our map to show the classes we’ve found:

There’s certainly more detail we could add, such as the contents of several important methods in DiscoverRunner.But what we have is by far enough to implement many useful customizations.

How to Customize¶

Django offers us two ways to customize the test running process:

• Override the test command with a custom subclass.
• Override the DiscoverRunner class by pointing the TEST_RUNNER setting at a custom subclass.

Because the test command is so simple, most of the time we’ll customize by overriding DiscoverRunner.Since DiscoverRunner refers to the unittest components via class-level attributes, we can replace them by redefining the attributes in our custom subclass.

Super Fast Test Runner¶

For a basic example, imagine we want to skip all our tests and report success every time.We can do this by creating a DiscoverRunner subclass with a new run_tests() method that doesn’t call its super() method:

Then we use it in our settings file like so:

When we run manage.py test now, it completes in record time!

Great, that is very useful.

Let’s finally get on to our much more practical emoji example.

Django Test Runner

Emoji-Based Output¶

From our investigation we found that the unittest component TextTestResult is responsible for performing the output.We can replace it in DiscoverRunner, by having it pass a value for resultclass to TextTestRunner.

Django already has some options to swap resultclass, for example the --debug-sql option which prints executed queries for failing tests.

DiscoverRunner.run_suite() constructs TextTestRunner with arguments from the DiscoverRunner.get_test_runner_kwargs() method:

This in turn calls get_resultclass(), which returns a different class if one of two test command options (--debug-sql or --pdb) have been used:

If neither option is set, this method implicitly returns None, telling TextTestResult to use the default resultclass.We can detect this None in our custom subclass and replace it with our own TextTestResult subclass:

Our EmojiTestResult class extends TextTestResult and replaces the default “dots” output with emoji.It ends up being quite long since it has one method for each type of test result:

After pointing the TEST_RUNNER setting at EmojiTestRunner, we can run tests and see our emoji:

Yay! 👍

No Composition¶

After our spelunking, we’ve seen the unittest design is relatively straightforward.We can swap classes for subclasses to change any behaviour in the test process.

This works for some project-specific customizations, but it’s not very easy to combine others’ customizations.This is because the design uses inheritance and not composition. We have to use multiple inheritance across the web of classes to combine customizations, and whether this works depends very much on the implementation of the customizations.Because of this not really a plugin ecosystem for unittest.

I know of only two libraries providing custom DiscoverRunner subclasses:

• unittest-xml-reporting - provides XML output for your CI system to read.
• django-slow-tests - provides measurement of test execution time, to find your slowest tests.

I haven’t tried, but combining them may not work, since they both override parts of the output process.

In contrast, pytest has a flourishing ecosystem with over 700 plugins.This is because its design uses composition with hooks, which act similar to Django signal.Plugins register for only the hooks they need, and pytest calls every registered hook function at the corresponding point of the test processes.Many of pytest’s built-in features are even implemented as plugins.

If you’re interested in heavier customization of your test process, do check out pytest.

Fin¶

Django Run Tests

Thank you for joining me on this tour.I hope you’ve learned something about how Django runs your tests, and can make your own customizations,

—Adam

🎉 My book Speed Up Your Django Tests is now up to date for Django 3.2. 🎉
Buy now on Gumroad

One summary email a week, no spam, I pinky promise.

Related posts:

Tags:django

Django Test Suite Runner

© 2020 All rights reserved.