GraphQL performance testing: Load testing with Grafana k6 and API Connect

For many companies, performance is the main reason to go with GraphQL, a query language for APIs. But is that a valid argument? Often, developers compare GraphQL to REST APIs and see the N+1 requests (or over-fetching) as an important reason to choose GraphQL. Let's put that to the test and explore whether GraphQL APIs can outperform existing REST APIs.

You'll take two GraphQL-formatted REST APIs (IP-API and Frankfurter) and load test the performance of GraphQL, and nested data in GraphQL, and then compare it with the original REST APIs. You'll use Grafana k6, one of the most popular open source load testing tools, to complete this performance test.

Explore a GraphQL API

Let's explore a GraphQL API created with (GraphQL API development)[https://www.ibm.com/products/api-connect/api-development], which has been converted from a REST API. In this case, the GraphQL API development CLI was used to convert the IP-API REST API to GraphQL, using the instructions from the docs.

This free REST API lets you search for a location based on its IP address. The IP-API REST API has been "GraphQL-ized" using GraphQL API development and explored with GraphiQL. The GraphiQL interface can be seen in the following image:

The operation you're sending to the GraphQL API appears on the left side of the screen, while the right side shows the response. The query is named GetLocation. Query names are recommended to help GraphQL APIs with, for example, caching. Note that the response has the same shape as the query you requested.

Now that you've explored this GraphQL API, you'll set up k6 so you can use it to test GraphQL.

Using k6 for GraphQL

To load test this GraphQL API, you're using k6, an open source load testing tool. You can run k6 on your local machine by installing it from the GitHub repository or by using the k6࣬ cloud offering. Using this tool, you can test any API that accepts HTTP requests. The most straightforward test you can run uses the http.post function from k6:

import http from 'k6/http';

const query = `
  query GetLocation {
    ipApi_location(ip: "8.8.8.8") {
      id
      city
      country
    }
  }
`;

const headers = {
  'Content-Type': 'application/json',
};

export default function () {
  http.post(
    'YOUR_GRAPHQL_ENDPOINT',
    JSON.stringify({ query }),
    { headers },
  );
}

This k6 script can now hit the GraphQL API using the query to load test its performance. Because the IP address is static, k6 will make the same request repeatedly in your performance test. Because the GraphQL API will cache the results, the tests will get less realistic as different users will use different IP addresses when hitting the GraphQL API. Therefore you should make use of dynamic variables in your GraphQL query.

The same query with a dynamic value for ip will look like the following:

query GetLocation($ip: String!) {
  ipApi_location(ip: $ip) {
    ip
    city
    country
  }
}

When sending the request, you need to append a JSON object containing a value for ip alongside your GraphQL query. On the GraphiQL interface, you can use the Variables tab to add the object:

Using the dynamic query parameters, you could use the http.batch function from k6 to send multiple requests with different values for ip to the GraphLQ API to simulate a more realistic testing scenario:

import http from 'k6/http';

const query = `
  query GetLocation($ip: String!) {
    ipApi_location(ip: $ip) {
      ip
      city
      country
    }
  }
`;


const headers = {
  'Content-Type': 'application/json',
};

const ENDPOINT = 'YOUR_GRAPHQL_ENDPOINT'

export default function () {
  http.batch([
    [
      'POST',
      ENDPOINT,
      JSON.stringify({ query, variables: { ip: '8.8.8.8' } }),
      { headers },
    ],
    [
      'POST',
      ENDPOINT,
      JSON.stringify({ query, variables: { ip: '12.0.1.3' } }),
      { headers },
    ],
    [
      'POST',
      ENDPOINT,
      JSON.stringify({ query, variables: { ip: '95.120.0.0' } }),
      { headers },
    ],
  ]);
}

This k6 script will send a batch of requests to the GraphQL API with different IP addresses. These requests are sent in parallel, giving you a more realistic scenario than sending just one. You could also create an array of IP addresses and loop over this to create a new array that you pass to the http.batch function.

Load testing a GraphQL query

With the k6 scripts set up, you can now test the performance of the GraphQL API. You can run two performance tests, one with a static IP address and another with batched requests and dynamic IP addresses.

To run these tests, you need to either have k6 downloaded and installed on your local machine, or you need a k6 cloud account.

To run the first test, save the script in a file called simple.js so you can run the test as follows:

k6 run --vus 10 --duration 30s simple.js

This command runs k6 with 10 VUs (virtual users) for 30 seconds. For more information, see Running k6.

The results show that the GraphQL API was hit almost 2500 times in 30 seconds, with an average duration of 122ms. This result is close to the average duration of hits for 95% of all requests, meaning there are no outliers.

Based on these results, you can also test the scalability of the GraphQL API that runs on GraphQL API development. You need to take a closer look at the number of iterations that the GraphQL API handled:

iterations.....................: 2472 82.082529/s

When you ran the k6 script for 30 seconds with 10 concurrent VUs, you can see that k6 hit the GraphQL API almost 2500 times, or 82 times per second. If the GraphQL API is perfectly scalable, it should be able to handle ten times more iterations when you increase the number of concurrent VUs to 100. Let's try this:

k6 run --vus 100 --duration 30s simple.js

This produces the following results:

For a perfectly scalable service, the number of iterations would be 820. Our result is 798, which is only a 3% difference. The GraphQL API isn't perfectly scalable, but it's quite close to being so.

In addition to testing the simple query with a static IP address, you can also run the script with the dynamic IP address by placing it in a file called batch.js:

k6 run --vus 10 --duration 30s batch.js

The iterations to the GraphQL API in this test are sent in a batch, meaning that every iteration results in three HTTP requests, the number of requests added to the http.batch function. As you learned previously, the GraphQL API is almost perfectly scalable.

The number of iterations the GraphQL API can handle in this test should be roughly the same, while the number of HTTP requests should be around three times larger. The test for the number of iterations produced the following result:

http_reqs......................: 7251   240.737555/s
iteration_duration.............: avg=124.43ms min=116.53ms med=121.91ms max=509.1ms  p(90)=126.39ms p(95)=129.13ms
iterations.....................: 2417   80.245852/

At 2417 requests, as opposed to the expected 2500, the number of iterations is comparable, and the number of HTTP requests is three times larger than the number of iterations.

Now that you know k6 can test the performance of GraphQL APIs and the GraphQL API is scalable, you can test a heavier GraphQL query.

Load testing different data sources

Determining the shape of the data isn't the only reason developers adopt GraphQL as their API query language. GraphQL APIs have just one endpoint, and the queries (or other operations) can also handle nested data. You can request data from different database tables, as with SQL joins, or even from various data sources in a single request. This functionality is different from REST APIs, where you typically have to hit multiple endpoints to get data from other sources.

In the GraphiQL interface for the GraphQL API that you're testing, you can explore other available queries. One of those queries will get data from the Frankfurter REST API, an open source API containing current and historical exchange rate data published by the European Central Bank. This REST API is converted to GraphQL in the same way as the IP-API. To get the current conversion rate from euros to US dollars, use the following query:

query GetConversion {
  frankfurter_latest_rates(from: "EUR", to: "USD") {
    amount
    base
    date
    rates
  }
}

This query gets the conversion rate from 1 EUR to USD on the current date. Because the schema for this API is a combination of the data from IP-API and Frankfurter, you can do more with it. Using this combination, you can get the current location based on the IP address and convert the local currency of that location to USD.

This currency conversion is available on the field priceInCountry. You can view the result using the GraphiQL interface, which will look similar to the following image:

Besides the IP address location, this query also lets the GraphQL API convert euros to the local currency of that location. In this case, it means converting euros to dollars again.

To get this data, the GraphQL API will do the following:

• Send a request to the underlying IP-API REST API to get the location and currency based on the IP address.
• Convert the currency of that location to euros using the Frankfurter REST API.

You can use this nested query in a k6 script to do another performance test of the GraphQL API. This query has a different depth than the query you've used in the previous section, since the data is now nested because it comes from different sources. You can put the following k6 script in a new file called nested.js:

import http from 'k6/http';

const query = `
  query GetConversion($ip: String!, $amount: Float!, $from: String!) {
    ipApi_location(ip: $ip) {
      ip
      city
      country
      currency
      priceInCountry(amount: $amount, from: $from)
    }
  }
`;

const headers = {
  'Content-Type': 'application/json',
};

export default function () {
  http.post(
    'YOUR_GRAPHQL_ENDPOINT',
    JSON.stringify({
      query,
      variables: {
        amount: 1,
        from: 'EUR',
        ip: '8.8.8.8',
      },
    }),
    { headers },
  );
}

Run this script using the same parameters as the previous tests:

k6 run --vus 10 --duration 30s nested.js

The results of this performance test are similar to the test with the batched requests. The batching isn't occurring in the k6 script but rather at the GraphQL API, which handles the two requests to the underlying REST APIs.

As the GraphQL API is built to be performant, the difference between the initial query, which gets only the location of the IP address, and this query, which gets both the IP address location and the currency conversion, is minimal. You can check these results in the output of the k6 load test below:

The test you've just run shows that GraphQL is able to get your data from different sources all in one request. Now, you'll break down the GraphQL query to the REST API requests to illustrate this point.

Compare GraphQL to REST performance

You've already learned how to test performance of GraphQL using k6, and you've seen how that test differs from testing a REST API. The last GraphQL query you tested is calling two different REST API endpoints.

There are two scenarios you can try to compare GraphQL and REST performance. Either set up two separate tests to compare the performance of the individual REST endpoints against their corresponding GraphQL queries, or recreate the complete behavior of the GraphQL API by directly calling the two REST API endpoints from a k6 test.

The second scenario is the most interesting to test, as both REST endpoints need to return data for the GraphQL query to resolve. The GraphQL query behavior you're testing looks like the following:

query GetConversion($ip: String!, $amount: Float!, $from: String!) {
  ipApi_location(ip: $ip) {
    ip
    cit
    country
    currency
    priceInCountry(amount: $amount, from: $from)
  }
}

This query sends requests to the REST endpoints:

• To get the location and currency for IP address 8.8.8.8 from IP-API: http://ip-api.com/json/8.8.8.8?fields=city,country,currency.
• To convert the currency from the IP address location to euros from Frankfurter: https://api.frankfurter.app/latest?amount=1&from=EUR&to=USD.

To test this using k6 you need to set up the following script in a new file. Name the script rest.js:

import http from 'k6/http';
import { check } from 'k6';

const headers = {
  'Content-Type': 'application/json',
};

export default function () {
  const response = http.get(
    'http://ip-api.com/json/8.8.8.8?fields=city,country,currency',
    null,
    { headers },
  );

  check(response, {
    'currency should be returned': (res) => res.json().currency === 'USD',
  });

  http.get(
    'https://api.frankfurter.app/latest?amount=1&from=EUR&to=USD',
    null,
    { headers },
  );
}

This k6 performance test will call the two REST endpoints. Additionally, in a real-world scenario, it would make no sense to reach the second endpoint to convert the locations' currency to euros if the first request did not return the local currency of the IP address.

Run the test above with the same conditions as for the other tests:

k6 run --vus 10 --duration 30s rest.js

The test produces the following results:

The first thing that stands out is that only 50 iterations with two HTTP requests per iteration are complete in 30 seconds. Only 3 VUs have been able to send requests in these tests, and 20% of all requests have failed. Compared to the GraphQL API, these test results are disappointing. The GraphQL versions of the two REST APIs can handle many more requests and resolve them faster.

Why do you get these results? The GraphQL API created using GraphQL API development applies caching, while the REST API itself doesn't seem to apply any caching at all. The requests to the IP-API endpoint seem to fail 20% of the time. Furthermore, the GraphQL API will also batch requests or optimize for N+1 requests when needed. For example, you need to request the same data from the currency REST API twice.

Summary

GraphQL usage has been on the rise, especially among front-end developers who want a more declarative way to handle data. In this article, you've explored how to test the performance of a GraphQL API using k6. This GraphQL API was created with the GraphQL API development CLI by converting the open source REST APIs from IP-API and Frankfurter.

The test results showed the GraphQL API was close to being perfectly scalable and performant in all scenarios. In contrast, the individual REST API endpoints had significant performance issues in the tests. The difference is due to the GraphQL API's performance optimizations, such as caching.

For a complete performance test script for GraphQL, see the GraphQL Benchmark tool.

Want to learn more about (GraphQL API development)[https://www.ibm.com/products/api-connect/api-development]? Try it out, have a look at the documentation, or join our Discord server to stay up to date with our community.