sexta-feira, 31 de março de 2017

Running tests with stubs containerized

We've been speaking about tests on other posts and I mentioned about using stubs to run tests with external dependencies.

A good approach is to use docker, I just did some examples of stub services and pushed the images to docker hub. To run my tests, I added a dependency on my docker-compose file for those images and use them to run my tests with docker compose.

It lead me to write this post and share this knowledge with you.

I've created a stub service with NodeJS (using hapi to startup a service and expose my routes), with this code in hands, we can start playing with docker, we'll do some simple steps to publish our image to docker hub and use it on our tests.

Register an account on docker hub

  • Visit Docker Hub and fill the form with dockerhubid, email and password
  • A confirmation email will be sent, and you just have to click a link to activate your account
  • On command line you can run docker login, it'll ask your credentials

Create your image

First of all, with our stub code, we need to create a local docker image, to be able to publish it later.

The first thing to do is to create a DockerFile, which looks like:

FROM node:6.0

RUN mkdir /usr/src/app
WORKDIR /usr/src/app

COPY package.json /usr/src/app
RUN npm install --dev

COPY . /usr/src/app


CMD npm start

Basically I'm using a node:6.0 image, copying my project and running some commands to install and expose my stub.

To create the image now, we need to execute this DockerFile, and it can be done easily with one command.

On the command line, do the steps below:
  • Navigate to the folder where your DockerFile is
  • Run docker build -t name .

It'll find a DockerFile on the current folder and will build the image with the name informed.

The command "docker build" has other options, in this case I just wanted to keep it simple.

Pushing my image to docker hub

Once I built my DockerFile, I need to push my image to docker hub, and for that we need two steps:

Tag the image

It's a very simple step, we need to tag our image id. This tagging process is to associate the image id you just generated with a name that will be pushed to docker hub.

docker tag imageId dockerHubId/imageName:version

Remember that dockerHubId must be the user logged in (your account).

Push the image

Once we tagged our image, we can now push it to docker hub.
The command to push is very simple.

docker push dockerHubId/imageName:version

Running tests with docker compose

Once we have our image published on docker hub, we can create the docker-compose.yml to run our tests using the stubs from the image just published.

In the example I'll show, it's a NodeJS project running tests with the command "npm test".

Here is the docker-compose.yml file with the dependency of a image with our stub.

  image: mcure/mystub:latest
    - "3002"

  build: .
  command: npm test
    EXTERNAL_URL: "http://mystub:3002"
    - mystub
My code has a dependency on a environment variable for referencing an external API, which I'm calling EXTERNAL_URL.

My project's config file uses this environment variable to set the URL of the external API.

When executing this docker-compose.yml, it'll build the container from the DockerFile of my project, and it will run the "command" specified replacing the environment variable set on the configuration.

For each test execution, it'll create a new container from that image, which means that my test can be ran without any external dependency, and I have now a test more isolated not depending on any real service.

With this post, I hope I helped you to understand a bit more about docker and the strategy of running tests with stubs.

Tweet me if you wanna discuss more about it.

segunda-feira, 16 de janeiro de 2017

Contract tests

Today's post is to talk about contract tests, the value of having them and strategies to have trustable test suites.

Everyone know the value of tests, they will ensure that the system behaves as expected, be using BDD or TDD or whatever technique you prefer. Of course any kind of system/problem may have a more appropriated kind of test.

In the last few years, I've been working with APIs. Each day is a new challenge, sometimes we have the need to version it, change contracts, make new integrations, include new rules, etc.

I've been developing an API with NodeJS, and it was chosen CucumberJS to write tests, so then the team was be able to do BDD. Of course there are other libraries that would allow us to do the same, anyway...

When writing APIs, one important thing to think about is to have contract tests, so then I believe BDD was a good take.

Testing a contract is not something straight forward, you need to think about many things and create strategies.

Treating external dependencies with contract tests

When you write an API, there is a big chance that you have external dependencies, be it a database, other APIs, messaging mechanisms, cache, etc. You don't want to trust real dependencies when running contract tests, tests must be 100% independent and should not trust a real external dependency.

What's the chance of an external dependency be unavailable at the time you want to test your contract? Do you want your contract tests to fail if a external dependency is not available? No, right?!

There are some good tips to avoid this kind of issue to happen, depending on the kind of dependency we can have different strategies.

External APIs

Having an external API as dependency may be a problem as mentioned before, the API may be down, or with some delay, it can make your tests to fail or take a long time. A solution for this problem is to create stubs (Martin Fowler post about mocks and stubs).

Stub is a simple interface which will give you standard responses of a giving API. Let's say you have a dependency on a Products's API and your API needs to retrieve a list of products. This stub would always return a list of products, this stub could be started up with your suite of contract tests, and be stopped at the end of its execution.


Many times we are dependent on data, so a database dependency may be a big problem.

You cannot just trust your QA or DEV environment database, not just because of the fact that the database may be down, under stress, a dataload is happening, it can even be empty, or it can have any kind of unavailability. But also because you may need some data on it, and you also want to make sure it will always, no matter when, return the data you expect.

You can easily start up a database with docker and include some fake data on it before your tests execution. Remember that your tests must be 100% independent, which means that a database change that is done on the test 1 will not impact the test 2, and the test 2 is not dependent on a state change made on test 1.

These are just 2 good tips to make your contract test faster and trustable.

Containerize your tests

As mentioned before, containers are a good solution for some external dependencies, they're immutable and can be executed on any environment. Why not run your tests with docker-compose? You can start containers from the scratch, seed some data, run the tests, and kill it when the execution is finished.

If you need to run the tests again, you will make sure the initial state is always the same, this way you have a good and trustable test suite. It even facilitates your life when you want to include the tests on a CI server, you just need docker installed and that's it.

Contract assertions

One very common mistake I see on some contract tests suites is that people only check if the request worked properly.

The contract is composed by many other things, here are some things you can check when doing contract tests:

HTTP Status code

The HTTP status code must be checked, different requests should return different status codes. If you do a GET on a specific endpoint, it should return 200 (ok). Or 404 (not found) if the given resource was not found. if you are creating a record (doing a POST), the status code should be 201 (created). These are just some examples.

Response body

The response body is also part of the contract, when you send a request, you expect a response body always with the same shape. Let's say you have a products's endpoint which returns the fields: id, description, unitPrice, available. Everytime you GET a product, you may expect to receive back these fields.

It just remembered me of Consumer Driven Contracts, you should definitely consider it when you are consuming APIs. (here is a great post from Martin Fowler about it)

I think that's all, I hope I helped you understanding how a contract work and how it can be tested, please feel free to tweet me and discuss about this subject. here is a great post about contract breaks, enjoy it.