How to Performance Test Your Http API Application

taken from https://gatling.io/

In the past I’ve written multiple articles regarding the testing of applications. The focus lies on automation of these tests. Every application version that is developed should be tested as much as possible automatically. I don’t want to do manual steps to verify that the application is behaving as expected. Until now I’ve talked primarily about verifying the correct behavior, that means the interaction with the application is leading to the correct results (e.g. Property-based Tests) according to the use-cases. Also that by mistake changed productive behavior (e.g. Mutation Testing) is recognized. There is an other dimension of correct behavior - I want to verify that the use-cases of the application are executed in acceptable time. Depending on the use-case there is a requirement that requesting a specified resource should in maximum take a fix amount of time.

The requirement for the request time leads to different questions:

• How long does it take for one single user to execute a request ?
• How long does it take for a specified number of parallel users to to execute requests?
• Can parallel requests lead to concurrency problems?

This questions can be answered by using performance testing.

In this first part of two articles I want to start giving an introduction to performance testing an application with http endpoints using the Gatling framework.

Gatling

According to the wikipedia article Gatling is described with the following:

Gatling Corp develops the open-source performance testing tool for web applications, Gatling, 
and its enterprise version, Gatling FrontLine. 
The Project's aims include;

- High performance
- Ready-to-present HTML reports
- Scenario recorder and developer-friendly DSL

Summing it up in my words Gatling provides an easy way of writing performance tests, that are executed with high performance and the result is printed in a good readable way, which allows comparison. It is a project that is open source developed in a community edition (you can find the repository on Github), but also is available as enterprise version, which includes e.g. a managment interface.

Gatling performance tests can be written in different languages including Kotlin (also Java and Scala are available). So in the following I will use the Kotlin syntax for writing the tests.

Setup

In the following section I will show how Gatling can be added to a project. The application I use for testing the performance is a Ktor application. You can find the repository on Github. It is a simple http API that allows to do the following operations:

• creation of user
• creation of books
• adding books to user

The application is just implemented for the purpose of showing how performance tests are written not represent a real world application.

To be able to separate the performance tests from the application code, I add an own submodule for it. With this it is also possible to later extract the performance test functionality to an own project.

For adding Gatling to the project I just need to add the corresponding plugin to the build.gradle.kts of the submodule:

plugins {
    id("io.gatling.gradle").version("3.9.2")
    kotlin("jvm") version "1.8.10"
}

Also I need to add the corresponding dependencies to the dependencies section:

dependencies {
    implementation("io.gatling:gatling-core:3.9.2")
    implementation("io.gatling:gatling-http-java:3.9.2")
}

Gatling provides a lot of modules, which can be added separately. For the basic functionality I only need the two ones from above.

The plugin configuration is very simple. I add a pattern for detection of the simulation classes that should be executed and also specify the base URL for executing the request against:

gatling {
    simulations = closureOf<PatternFilterable> {
        include("**/*Simulation*.kt")
    }

    systemProperties = mapOf(
        "baseurl" to "http://localhost:8080",
    )
}

There is a lot more configuration for the plugin possible. Just have a look in the documentation.

The default configuration of Gatling can be changed by adding a gatling.conf file in the resources directory. All available options, which can be configured can be found in the Github repository. I will explain some of them later when it comes to the reports.

For logging Gatling is using the logback framework, so placing a lockback.xml file to the resources directory enables to configure the output. To have a good overview what is happening during development I add a custom configuration which is logging the relevant parts on TRACE level.

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
    <property name="log_level" value="TRACE"/>
    <appender name="CONSOLE" class="ch.qos.logback.core.ConsoleAppender">
        <encoder>
            <pattern>%d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg%n</pattern>
            <immediateFlush>false</immediateFlush>
        </encoder>
    </appender>
    <logger name="io.gatling.http.ahc" level="${log_level}"/>
    <logger name="io.gatling.http.engine.response" level="${log_level}"/>
    <root level="TRACE">
        <appender-ref ref="CONSOLE"/>
    </root>
</configuration>

After this steps are finished I’m ready to start writing performance tests. A first version has the below structure:

class UserSimulation : Simulation() {

    private val baseUrl: String = System.getProperty("baseurl")

    private val httpProtocol = HttpDsl.http
        .baseUrl(baseUrl)

    private val addUserScenario = CoreDsl.scenario("Add new user")
        .exec(
            HttpDsl.http("add user").post("/api/v1/user")
                .body(
                    StringBody(
                        """
                        {
                            "firstName": "John",
                            "lastName": "Doe",
                            "birthDate" : "1999-01-01"
                        }
                    """.trimIndent()
                    )
                ).asJson()
                .check(
                    HttpDsl.status().`is`(201),
                )
                .check(
                    bodyString().exists()
                )
        )

    init {
        setUp(
            addUserScenario.injectOpen(CoreDsl.atOnceUsers(100)),
        ).protocols(httpProtocol)
    }
}

• The simulation for creation of user needs to extend the base Simulation class.
• I read the base url, defined in the Gradle configuration, and build a HttpProtocolBuilder.
• The scenario is created by specifying a name and call the exec() - function. This consistes of the definition of the request (http verb and the body) and a check that is executed against the response.
• The configuration of the scenario execution is done in the init-block of the simulation class. It is specified if an open or a closed workload model is used and the the amount of users for that the scenario is executed. The last configuration is the setup of the protocol.

This example is just a very basic one that has some limitations, that should be improved in order to have reliable results. The first problem is that only static data is send as request, so the same user is created again and again. This leads to problems as soon as there is an unique constraint on the name of the user.

So to have a more realistic behavior I need a way to provide random input data. Gatling provides multiple ways to accomplish this task. I will start with the so called feeder.

Feeder

A feeder provides random data for every request that is executed. A basic version for creation of random names for the user in order to not violate the unique constraint can look like below:

private val nameFeeder = generateSequence {
    val name = RandomStringUtils.randomAlphabetic(10)
    mapOf("name" to name)
}.iterator()

There is a sequence generated which consists of a map of entries where the key is the variable name, that can be later used in the scenario, and the random value.

I can use the feeder in the scenario by adding it in the builder function feed() before the exec - block. The defined variables can be accessed by wrapping it in a #{variable_name} block inside the body of the request.

private val addUserScenario = CoreDsl.scenario("Add new user")
    .feed(nameFeeder)
    .exec(
        HttpDsl.http("add user").post("/api/v1/user")
            .body(
                StringBody(
                    """
                    {
                        "firstName": "#{name}",
                        "lastName": "#{name}",
                        "birthDate" : "1999-01-01"
                    }
                """.trimIndent()
                )
            ).asJson()
            .check(
                HttpDsl.status().`is`(201),
            )
            .check(
                bodyString().exists()
            ),
    )

So now it is possible to create requests with random content. This is the easy part. In the same way I can create a scenario for creation of books.

Creating the input from a random function is only one option. I can also use a CSV file as input source. This can especially be interesting if I already have valid input in an external file source.

To be able to use a CSV feeder I create a subdirectory data in the resources directory and inside of this a file called names.csv. This file contains a header with the variable names I want to use. In my case it is firstname and lastname. Below the header I put all the data I want to use as random input for the scenario.

The Feeder can be created with the csv - function.:

val nameFeederCsv = csv(
    "data/names.csv"
).eager().random()

By specifying the eager() and the random() function the sample data will be loaded before starting the scenario and the entry for every request is choosen randomly. The created feeder can be used in the same way as the programatically one from the beginning of the section.

A last option for creation of input data for the scenarios I want to describe is the JDBC feeder.

If the data I want to use is already stored in a database I can use the jdbc feeder to load a specified table with columns.

All that is necessary for using this functionality is to add the database driver as dependency to the performance test module:

dependencies{
  ...
  gatlingRuntimeOnly("com.h2database:h2:$h2Version")
  ...
}

For loading the input I need to specify the jdbc url, username and password. The last information that is necessary in order to load the correct data is a select statement. With this I specify the columns I want to return. These need to match the variable names I use in the scenario configuration.

val nameFeederJdbc = jdbcFeeder("jdbc:h2:file:./db", "user", "password", "SELECT first_name, last_nameFROM user_input")

The feeder can be used in the same way as the previous ones described.

There are also further feeder available that can be used with Gatling. Just have a look in the documentation.

Session

The next level of writing performance tests is send requests, which require dependent data. In the following I will create a scenario for adding books to an user. For this the request needs the userId of an existing user and a bookId of an existing book. So how to do this?

I need to pass information from one request to the next one. First lets show you the flow of the scenario:

1. Creating a user
2. Creating a book
3. Add the book to the user

The first two requests are very basic, because they have no dependent input. I just need to store the response for it in the session, so I can access it by a following request. In Gatling this can be done with appending a check block to the existing checks (e.g. for status code). This block takes the response body, optionally transforms it and saves it with given key to the session.

.check(bodyString().transform { body-> body.replace("\"", "") }.saveAs("userId"))

In the case of the userId and the bookId I need to remove the quotes from the body string, so that it can be used as correct UUID for the following request.

The code for the creation of books is quite the same, so I omit showing it.

The request for adding the book to the user can use the both in the session stored variables to setup the request. With the #{key} syntax I can access the corresponding variables.

HttpDsl.http("add book")
    .post("/api/v1/user/#{userId}/book")
    .body(StringBody(
      """
      {
        "books": ["#{bookId}"]
      }    
      """.trimIndent()
      )).asJson()
    .check(
        HttpDsl.status().`is`(201),
    )
    .check(
        bodyString().exists()
    )
    .check(bodyString().saveAs("userId"))

With this I can put these parts together and get the scenario for adding a book to the user. To reuse the requests I extracted them to functions.

private val addBookScenario = CoreDsl.scenario("Add book to user")
    .feed(nameFeeder)
    .feed(bookFeeder)
    .exec(
        createCreateUserRequest(),
    )
    .exec(
        createCreateBookRequest()
    )
    .exec(
        createAddBookRequest()
    )

Now I’m create scenarios out of multiple requests. But still the scenario is not perfect because I only can one user and one book and add it to the user. A more realistic case is to create multiple books and add it to a single user.

So let’s try to update the existing scenario to be more flexible.

The steps I want to to in the scenario are the below ones:

- Execute n create user requests

- Execute n create book requests

- Choose a user of the existing ones and add n different books to it

For this it is necessary to store the information of the first 2 steps in the session so the following one can access it.

The create user and book requests are returning a single UUID for every request. I already store this value to the session. But until now there is only one userId and one bookId stored. To be able to keep multiple of them in a list, I need to change the way I setup the scenario.

I keep the configuration for each create user and book request:

private fun createCreateUserRequest() = HttpDsl.http("add user").post("/api/v1/user")
    .body(
        StringBody(
            """
             {
                 "firstName": "#{firstname}",
                 "lastName": "#{lastname}",
                 "birthDate" : "1999-01-01"
             }
            """.trimIndent()
        )
    ).asJson()
    .check(
        bodyString().transform { body -> body.replace("\"", "") }.saveAs("userId")
    )

This is creating a user and storing the userId to the session. This step I want to do multiple times before starting with the adding of books. Gatling provides an implemtaton of the repeat statement for this. I specify the amount of iterations I want to have and inside the on() — function I define the steps that should be executed multiple times:

CoreDsl.scenario("Add book to user")
        .repeat(AMOUNT_INITIAL_DATA)
        .on(
            feed(nameFeeder)
                .feed(bookFeeder)
                .exec(createCreateUserRequest())
                .exec(createCreateBookRequest())
        )

The problem that occurs when executing this scenario, in every round the variable for userId/bookId in the session is overwritten. So after the repeat function is finished only the values of the last round are available in the session.

So to have all values available I need to add additional exec blocks that are taking the value of every round and add it to a list, that is also stored in the session. This is necessary because until now I didn’t find a way to properly append the value directly in the list.

.exec { session ->
    val userIds: List<String> = session.getList("userIds")
    val userId: String = session.get("userId") ?: error("UserId not found")
    session.set("userIds", userIds + userId)
}

The same exec block I need for the create book request. With this I have a list of userIds and bookIds that I can use for the request to add books to users.

.exec { session ->
    val userIds: List<String> = session.getList("userIds")
    val userId = userIds.random()
    session.set("currentUserId", userId)
}
.exec { session ->
    val bookIds: List<String> = session.getList("bookIds")
    val bookIdsTaken = bookIds.shuffled().take(Random.nextInt(1, AMOUNT_INITIAL_DATA))
    session.set("currentBookIds", bookIdsTaken.map { "\"$it\"" })
}

To have a kind of randomness in the scenario I take a random userId and n bookIds of the existing bookIds. This data I inject into the request setup.

private fun createAddBookRequest() = HttpDsl.http("add user book")
    .post("/api/v1/user/#{currentUserId}/book")
    .body(
        StringBody(
            """
            {
                "books": #{currentBookIds}
            }
            """.trimIndent()
        )
    ).asJson()
    .check(
        HttpDsl.status().`is`(HTTP_STATUS_CREATED),
    )
    .check(
        bodyString().exists()
    )

Comparing this solution with the previous one the request is closer to the reality, where user add a random amount of books instead of always one.

With this the first part of the introduction to performance testing with Gatling is finished. I showed how the framework can be integrated in an existing application and how test simulations with different parts can be written, using feeder, session state or loops. Also I showed how the order of requests can be influenced.

In the second part of the article I will show how the configuration of the test execution can be done using open or closed models in order to check the performance for parallel requests. Also the topic of detection of concurrency will be part of it. Beside this I will show how to extract the best of the generated reports of Gatling.

The code of this article you can find in the Github repository.