Difficulty: Beginner
Estimated Time: 10 minutes

Intermediate TDD

This scenario is a worked example creating a web scraping utility.

It creates tests at the acceptance test level and at the unit test level.

Acceptance tests are tests that run a higher level than unit tests and are used to validate correct the code behaves correctly. Because the tests are run at a higher level, it is more difficult to test all edge cases.

On the flip side, whilst unit tests are great for testing every edge case an acceptance test is still required to validate that the units are connected together in the correct way. Thus a good test suite will include both unit and acceptance tests.

When creating these tests, it is worth having a clear understanding of the boundaries of the system under test and what sub systems need to be mocked, in order to provide isolation from external systems outside of your control. Writing tests first will encourage thinking about this before creating the code, which might slow the initial development will allow overall development to go faster.


We built a web scraper using python.

Writing the tests first forced us to think about how we would test the program and structure the code to enable easier testing.

We have created code that is composed of small units that can be tested in isolation.

By developing the code in small increments and keeping all of the tests passing at each point, we could have been continually delivering the code to master knowing that we haven't broken anything.

Intermediate TDD

Step 1 of 14

Step 1

A web scraper

In this example, we're going to build a command line utility that will take a URL and return the sum of the integers found on the page.

Acceptance Tests first

The first thing to do is to write an acceptance test. We need to consider the test interfaces to use. An acceptance test should be closer to the actual environment.

run-acceptance-tests.sh is a script that will launch a webserver to serve files before running the tests. But before we can use it, we need to create a response for it to return.

Lorem ipsum 1000 dolor sit amet, 33consectetur adipiscing elit, sed do eiusmod tempor
incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud
exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute
iru12re dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat
nulla pariatur. Excepteur sint occaecat cupid789atat non proident, sunt in culpa
qui officia deserunt mollit anim id est laborum.

And now let's create the test to verify it

#! /usr/bin/env python3

"""This module contains the acceptance tests for this module"""

import unittest
import subprocess

class TestWebScraper(unittest.TestCase):
    def test_can_add_whole_numbers(self):
        result = subprocess.check_output('python3 webscraper.py http://localhost:8000/response1.html', shell=True)
        self.assertEqual(result, b'1834\n')

if __name__ == '__main__':

Finally, let's verify that we can run the test and that it fails:

bash run-acceptance-tests.sh