Testing

Today we're going to be talking about unit and integration tests. While we'll try to work in as many holiday jokes as we can muster, don't mistake the injection of humor as a reason to make light of automated tests. Testing is serious business.

Specifically we will be discussing:

• Why testing is important.
• Ways to improve how long your tests take to run
• Tools you should know about
• Techniques for writing tests more quickly

Every Dev Down in Dev-ville Liked testing a lot...
But the Grinch, Who lived just North of Management-ville, Did NOT!

The Grinch hated Testing! The whole Testing process!
Now, please don't ask why. No one quite knows the reason.

It could be that his head wasn't screwed on quite right.
It could be, perhaps, that his shoes were too tight.

But I think that the most likely reason of all
May have been that his thinking was two sizes too small. Dr. Suess on automated testing

Testing makes your team better and faster

If Santa had a cookie for every time one of his middle management Elves didn't take automated testing seriously, he would need twice as many reindeer and wouldn't fit in his sleigh.

He often hears these arguments for being on the testing naughty list:

• Writing tests takes too much time
• They don't move features/deliverables forward
• We'll do it later...

The Elves that embrace testing realize all three of these arguments are wrong and end up showered in gifts from Old Saint Nick.

Code without tests is broken as designed. Jacob Kaplan-Moss

Software is exacting. A small typo can have a large impact, which is why we have to test our software in the first place. If you aren't currently using automated tests now, then you're doing manual testing and wasting the precious resources of your team.

Resources that could be being used to actually move your software forward.

You will never entirely eliminate your QA team and/or all manual testing, but every hour dollar you spend in manual testing is leasing a problem that is better owned.

You're already spending the time, just not capturing so you don't have to spend it again next week. And quit fooling yourself that if you don't "have time for it" this week that you will magically find time for it next week.

Speed up test execution

The first rule of system performance is: do less work. This certainly applies to unit and integration tests.

Don't create 500 rows in your database, setting the stage for a test, when 50 or even 5 would suffice.

Mock, aka fake, calls to slower external services.

Don't save data to disk or a datastore when you really don't need to.

When you're actively developing in one area of your software, make sure you are just running the tests for that section and not the entire test suite. It's probably only seconds per test run, but seconds add up into minutes and hours before you know it.

If running your entire test suite takes more than a minute, run it asynchronously using a CI system like Jenkins or one of the fine SaaS products out there.

Your developers will still need to locally run sub-sets of your whole suite on a regular basis, but you can at least farm out the final check of everything to a computer rather than wasting their time.

If we run the full test suite before pushing each commit, and you should be, CI quickly pays for itself. Let's assume an average developer salary of $60k, a cost of $100/month to run a CI system, and a full test run time of 2 minutes. Your CI system breaks even in only 50 commits per month.

Testing Tools

Python has the great built in unittest library and Django extends this with its TestCase. You should become very comfortable with them and know all about what options they offer you.

Beyond those "built in tools" you should check out the other two major Python testing frameworks nose and pytest.

These offer more advanced features and plugins to help reduce the time it takes to write and run tests, but also makes writing them more pleasurable for your devs. Making it faster, easier, and more fun increases the likelyhood they'll actually do it.

Let's take pytest for example, here are some things you can take advantage of:

• Mark tests with arbitrary tags so you can include/exclude them from test runs. You can mark all tests that require network access with 'network' and slower tests as 'slow', skip them, and have them only run by your CI system
• Options to have less boilerplate test code
• Cache test results and re-run just the ones that failed on your last run with pytest-cache
• Split up and run your tests across multiple cores pytest-xdist or have it continously loop
• For Django projects, pytest-django provides great features, notably not destroying your test database and recreating it on every test run

Writing tests faster

If you aren't accustomed to testing writing them can feel annoying. Spending time making your tests easier and faster to write makes it more enjoyable.

One tip that many people don't think about is you aren't stuck with the features of unittest, nose, and/or pytest. Make your own base test classes and have your individual tests inherit common useful tools/features.

For example, we typically use a base test class that inherits from Django's TestCase and gives us features like:

# no need to import reverse everywhere
url = self.reverse(url_name)

# Assert a response has the right status code
self.response200(response)
self.response404(response)

# get and post methods that assume named urls and kwargs
response = self.get('blog-detail', slug='some-post')
response = self.post('user-create', data=user_data)

# Get a URL based on the named url passing in kwargs and return the response
response = self.get_check_200('blog-detail', slug='that-one-post')

# Assert that login is required
self.assertLoginRequired('dashboard')

# Handle login as a context manager
with self.login(username='admin', password='secret'):
    self.get_check_200('dashboard')

DRY still applies in tests. If you see lots of boilerplate, spend a few minutes to reduce/remove it. These make your test code more concise and readable, while also reducing the time it takes to write them.

To make writing your tests faster avoid using fixtures.

Fixtures quickly become brittle and cumbersome to manage over time. You either end up spending time keeping the fixtures up to date as your tests change or you quickly end up with 38,000 of them one for each particular test scenario.

Instead, use code to generate only the necessary data for that particular test. As you add/remove data in your code, you aren't necessarily forced to also manage it in a bunch of fixtures files. That change you made likely has a sensible default so most, if not all, of your generation code can continue unchanged.

For more complex data and especially for Django model data, check out Factory Boy and Model Mommy. These are great frameworks for crafting the relationships between your data and automating the creation of "chains" of data.

What do I mean by "chains of data?" One of the annoying parts of generating test data with code is to test out that one blog view you often write setup code that:

• Creates a User
• Creates an Author object associated to the User
• Creates a Site, Blog, and Blog Category objects
• And THEN create the BlogPost

Now do all that again for the next 12 tests you want to write.

Tools like Factory Boy allow you to define how these chains work and instead you write something like:

def test_blog_view(self):
    # I need a BlogPost, so build me all of the other stuff please
    post = BlogPostFactory.create(title='Test Title', active=True)
    # Your tests here ...

Hopefully today you've learned a few testing tricks or if nothing else some more ammo to use against your boss if they aren't keen on testing. Happy Holidays!