Introducing the Java SE 11 HTTP Client

Traditionally, when you wanted to do an HTTP call in Java, you had to use the HttpURLConnection class. Until very recently, Java provided only the HttpURLConnection API, a low-level painful API that is cumbersome to use and isn’t known for being feature-rich and user-friendly. If you’re wondering why that API was such a pain, find out more in this post on StackOverflow.

Many third-party libraries alternatives external to the JDK emerged to compensate for this API and some are still widely used, such as Apache HttpClient, Jetty, and Spring’s RestTemplate.

The JDK needed a modern and easy-to-use API.

The big addition to the standard library in Java 11 is the HTTP Client API, a reinvention of HttpURLConnection.

In general, the goal of the new HttpClient is to be easy to use in common cases, but also to be powerful enough for more complex cases.

The evolution of HttpClient and WebSocket API
The latest HttpClient and WebSocket APIs originally shipped with JDK 9 under the Java Enhancement Proposal JEP 110 as part of an incubator module named jdk.incubator.httpclient. They were re-incubated again in JDK 10, but finally became standardized in a module named java.net.http and brought into the mainstream in JDK 11 under JEP 321.

The HTTP Client can be used to request HTTP resources over the network. HttpClient supports both HTTP/1.1 and HTTP/2. By default, the client will send requests using HTTP/2.

Requests sent to servers that do not yet support HTTP/2 will automatically be downgraded to HTTP/1.1 for backward compatibility. Both synchronous and asynchronous programming models handle request and response bodies as reactive-streams and follow the familiar builder pattern.

The Reactive Streams API offers interfaces that manage asynchronous streams of data, including the notion of back pressure in which data consumers can slow down producers to get an optimal flow of data. And HttpClient integrates with the Reactive Streams APIs that were introduced in Java SE 9.

That means that you don’t need external libraries to do HTTP/2 calls or WebSocket communication in Java code.

Incubator modules
Incubator modules are a means of putting non-final APIs in the hands of developers while the APIs progress towards either finalization or removal in a future release. See JEP 11: Incubator Modules for more information.

Why does it use HTTP/2?

HTTP/2 is a new efficient protocol which is widely adopted by most of the browsers and servers and I’ve provided a summary of the major improvements the protocol enables:

  • Binary format: Makes it more compact
  • Multiplexing: Means multiple requests are allowed at the same time, on the same connection
  • Server Push: Means additional future-needed resources can be sent to a client
  • Single Connection to the Server: Reduces the number of roundtrips needed to set up multiple TCP connections
  • Header Compression: Uses HPACK compression to reduce overhead

The HTTP Client is well positioned for future use when HTTP/2 is widely deployed. HTTP/2 is the default preferred protocol, so the implementation seamlessly falls back to HTTP/1.1 when it is necessary.

Deeper look at the HTTPClient API

This API provides a high-level client interface to HTTP (versions 1.1 and 2) and low-level client interfaces for the WebSocket API. The main types defined in the API are:

The protocol-specific requirements are defined in the Hypertext Transfer Protocol Version 2 (HTTP/2), the Hypertext Transfer Protocol (HTTP/1.1), and The WebSocket Protocol.

In general, asynchronous tasks execute in either the thread invoking the operation (for example, when you send) an HTTP request) or by the threads supplied by the client’s executor).

Dependent tasks, those that are triggered by returned CompletionStages or CompletableFutures, that do not explicitly specify an executor will execute in the same default executor) as that of CompletableFuture or in the invoking thread if the operation completes before the dependent task is registered.

CompletableFutures returned by this API will throw an UnsupportedOperationException for their obtrudeValue) and obtrudeException) methods.

Invoking the cancel) method on a CompletableFuture returned by this API may not interrupt the underlying operation, but may be useful to complete exceptionally dependent stages that have not already completed.

Unless otherwise stated, null parameter values will cause methods of all classes in this package to throw a NullPointerException.

Goals for the HTTP Client API

The HTTP Client APIs has many goals designed to help you understand the important characteristics of this API and how you can use it in your programming:

  • Easy to use for common cases, including a simple blocking mode
  • A simple and concise API which caters to 80-90 percent of application needs
  • Supports standard and common authentication mechanisms
  • Easy to set up the WebSocket interface handshake
  • Friendly to embedded-system requirements; in particular, the avoidance of permanently running timer threads
  • Supports HTTPS/TLS
  • Must be very performant and its memory footprint less than older and third-party APIs
  • Provides non-blocking request and response semantics through CompletableFuture, which can be chained to trigger dependent actions
  • Provides back pressure and flow-control of request and response bodies via the platform’s reactive-streams support in the java.util.concurrent.Flow API

See JEP 110: HTTP/2 Client (Incubator) for more details and to read about the other goals.

Experiment with HTTP Client and WebSocket APIs

Let’s develop a modular application to run these examples that take advantage of the JPMS (Java Platform Module System). First, you have to import the module into your application to start using its packages and classes.

For more information on the Java Platform Module System, you can explore the Java Platform Module System (JSR 376).

You need to define your module using a module-info.java file which will indicate the required module you need to run the application:

Modulecom.tm.siriusxi.httpclient {
requiresjava.net.http;
   requires java.logging;
   requires com.fasterxml.jackson.databind;
   requires com.fasterxml.jackson.core;
}

You can download this tutorial source code from GitHub and follow along with the tutorial by importing the project into your favorite IDE. I am using IntelliJ IDEA Community Edition 2018.3+.

The full code classes HttpClientApp, EchoListener, and WebSocketApp reside in the package com.tm.siriusxi.http.ws.

Here are the tools you will require:

  1. The latest Java SE Development Kit 11 for JDK 11
  2. You can use IntelliJ 2018.3+ IDEA community Edition, Apache NetBeans 10, or Eclipse IDE, any of the latest versions that support JDK 11
  3. I will be using Postman Echo for RESTful services
  4. And I will use Kaazing WebSocket Echo Demoat

Now that you have all the tools ready, let’s deep dive into some action.

HTTP Client API

Unlike HttpURLConnection, the HTTP Client API provides synchronous and asynchronous request mechanisms.

The HTTP Client core classes

The API consists of the six following core classes:

  • HttpClient
  • HttpRequest
  • HttpRequest.BodyPublisher
  • HttpResponse
  • HttpResponse.BodyHandler
  • HttpResponse.BodySubscriber

Here’s what each of the classes is designed to do.

HttpClient

The main entry point of the API. This is the HTTP client that is used to send requests and receive responses. It supports sending requests both synchronously and asynchronously by invoking its methods send and sendAsync, respectively. To create an instance, a builder is provided. Once created, the instance is immutable.

HttpRequest

Encapsulates an HTTP request, including the target URI, the method (GET, POST, etc.), headers, and other information. A request is constructed using a builder, is immutable once created, and can be sent multiple times.

HttpRequest.BodyPublisher

If a request has a body (like in a POST request), this is the entity responsible for publishing the body content from a given source, such as from a string, a file, etc.

HttpResponse

Encapsulates an HTTP response, including headers and a message body, if any. This is what the client receives after sending an HttpRequest.

HttpResponse.BodyHandler

A functional interface that accepts some information about the response (status code and headers) and returns a BodySubscriber, which itself handles consuming the response body.

HttpResponse.BodySubscriber

Subscribes for the response body and consumes its bytes into some other form (such as a string, a file, or some other storage type).

BodyPublisher is a sub-interface of Flow.Publisher, introduced in Java 9. Similarly, BodySubscriber is a sub-interface of Flow.Subscriber. This means that these interfaces are aligned with the reactive streams approach, which is suitable for asynchronously sending requests using HTTP/2.

Implementations for common types of body publishers, handlers, and subscribers are pre-defined in factory classes BodyPublishers, BodyHandlers, and BodySubscribers. For example, to create a BodyHandler that processes the response body bytes (via an underlying BodySubscriber) as a string, the method BodyHandlers.ofString() can be used to create such an implementation. If the response body needs to be saved in a file, the method BodyHandlers.ofFile() can be used.

Following are a number of examples and recipes that can be used to perform common tasks using the Java HTTP Client.

Demonstrating how all three APIs work together

First, a streamlined example is required to demonstrate how all three APIs are chained into action to send a request and receive a response. Here is a GET request that prints the response body as a JSON string to the console:

var client = HttpClient.newHttpClient();

var request = HttpRequest.newBuilder()
            .uri(create("https://postman-echo.com/get/"))
            .build();

client.sendAsync(request, ofString())
      .thenApply(HttpResponse::body)
      .thenAccept(out::println)
      .join();

Now let’s look deeper at the HttpClient class which is responsible for sending requests and receiving responses.

Creating and configuring HttpClient

Before we can make HTTP calls, we need the HttpClient instance. All requests are sent using HttpClient which can be instantiated using:

  • HttpClient.newBuilder() method for more control of client configuration
  • By calling HttpClient.newHttpClient() which returns an instance with all default settings

The HttpClient instance provides important methods we can use to handle our requests and responses.

Setting a proxy

To define a proxy for the connection and add more control, merely call the proxy() method on a builder instance:

HttpClient.newBuilder()
.proxy(ProxySelector.of(new InetSocketAddress("www-proxy.com", 8080)));

Alternatively, the system-wide default proxy selector can be used, which is the default on MacOS:

HttpClient.newBuilder()
.proxy(ProxySelector.getDefault())
.build();

Setting connection authentication

An Authenticator is an object which negotiates credentials (HTTP authentication) for a connection.

It provides different authentication schemes (like basic or digest authentication). In most cases, authentication requires a username and password to connect to a server.

We can use the PasswordAuthentication class which is just a holder of these values:

HttpClient.newBuilder()
.authenticator(new Authenticator() {
    @Override
protectedPasswordAuthenticationgetPasswordAuthentication() {
return new PasswordAuthentication(
        "username",
        "password".toCharArray());
    }
}).build();

Note that not every request should use the same username and password. The Authenticator class provides a number of getXXX (for example, getRequestingSite()) methods that can be used to find out what values should be provided.

In this example, I passed a plaintext username and password values; of course, in a production scenario, this will have to be different, especially for passwords.

Setting the redirect policy

When calling a website page, sometimes the page you want to access has been moved to a different address. In this case, you’ll receive HTTP status code 3xx, usually with the information about new URI.

By setting an appropriate redirect policy, HttpClient can redirect the request to the new URI automatically. All redirect policies are defined and described in enum with a name HttpClient.Redirect.

Using the followRedirects() method, you can set the redirect policy:

HttpClient.newBuilder()
.followRedirects(HttpClient.Redirect.ALWAYS)
.build();

Using and handling cookies

It’s a straightforward task to set a cookie for your connection. Youcan use the builder method cookieHandler(CookieManagercookieManager) to define a client-specific CookieManager.

For example, define a CookieManager which isn’t allowed to accept cookies:

HttpClient.newBuilder()
.cookieHandler(new CookieManager(null, CookiePolicy.ACCEPT_NONE))
.build();

Once this is configured and built, an HttpClient can be used to send multiple requests.

So now let’s learn about the request creation and configuration process before we call the server and receive the response.

Creating the HttpRequest instance

HttpRequest, as the name suggests, is an object which represents the request we want to send.

To create a new instance, you use HttpRequest.Builder by calling HttpRequest.newBuilder().

The builder class provides a bunch of methods you can use to configure your request. Here are the most important ones.

Setting request URI

The first thing you have to do when creating a request is to provide the URL.

You can do that in two ways — by using the constructor for Builder with URI parameter or by calling method uri(URI) on the Builder instance:

HttpRequest.newBuilder(new URI("https://postman-echo.com/get"));
HttpRequest.newBuilder().uri(new URI("https://postman-echo.com/get"));

The last thing you have to configure to create a basic request is an HTTP method.

Specifying the HTTP method

You can define the HTTP method which will be used by the request from Builder:

  • GET()
  • POST(BodyPublisherbody)
  • PUT(BodyPublisherbody)
  • DELETE()

We’ll cover BodyPublisher in detail later. Now let’s just create a very simple GET request example:

HttpRequest request = HttpRequest.newBuilder()
.uri(new URI("https://postman-echo.com/get"))
.GET()
.build();

This request has all the parameters required by HttpClient; however, sometimes we need to add additional parameters to our request, such as these important ones:

  • the version of the HTTP protocol
  • headers
  • a timeout

Setting the HTTP protocol version

The API fully leverages the HTTP/2 protocol and uses it by default, but you can define which version of protocol you want to use and this code will override the HttpClient version. If you don’t set the HttpClient version, the default will be used.

HttpRequest.newBuilder()
.uri(new URI("https://postman-echo.com/get"))
.version(HttpClient.Version.HTTP_2)
.GET()
.build();

I want to remind you that the client will fallback to HTTP/1.1 if HTTP/2 isn’t supported.

Setting headers

In case you want to add additional headers to your request, use the provided builder methods. You can do that in one of two ways:

  • Passing all headers as key-value pairs to the headers() method
HttpRequest.newBuilder()
.uri(new URI("https://postman-echo.com/get"))
.headers("IBM", "DeveloperWorks", "Author", "Mohamed Taman")
.build();
  • Using the header() method for the single key-value header:
HttpRequest.newBuilder()
.uri(new URI("https://postman-echo.com/get"))
.header(("IBM", "DeveloperWorks")
.header("Author", "Mohamed Taman");

Setting a timeout

Let’s now define the amount of time we want to wait for a response. If the response is not received within the specified timeout then an HttpTimeoutException is thrown; infinity is the default timeout value.

The timeout can be set with the Duration object by calling method timeout() on the builder instance:

HttpRequest.newBuilder()
.timeout(Duration.of(10, SECONDS))
.build();

Setting a request body

We can add a body to a request by using the request builder methods POST(BodyPublisherbody) and PUT(BodyPublisherbody).

The new API provides a number of BodyPublisher implementations out-of-the-box as a set of factory methods ofXXX() which simplify passing the request body and is created from HttpRequest.BodyPublishers:

  • Create a body from a string with BodyPublishers.ofString()
  • Create a body from an InputStream withBodyPublishers.ofInputStream()
  • Create a body from a byte array with BodyPublishers.ofByteArray()
  • Create a body from a file at the given path with BodyPublishers.ofFile()

In case we don’t need a body, we can simply pass in a BodyPublishers.noBody():

HttpRequest.newBuilder()
.uri(new URI("https://postman-echo.com/post"))
.POST(HttpRequest.BodyPublishers.noBody())
.build();

Setting a request body with any BodyPublisher implementation is very simple and intuitive.

To create a string body to pass simple String as a body, you can use the HttpRequest.BodyPublishers.ofString() factory method. It takes a String object as an argument and creates a body from it:

HttpRequest.newBuilder()
.uri(new URI("https://postman-echo.com/post"))
.headers("Content-Type", "text/plain;charset=UTF-8")
.POST(ofString("Sample request body"))
.build();

To create an input stream body, the InputStream has to be passed as a Supplier (to make its creation lazy), so it’s a little bit different from the previous ofString() method, but it is still a straightforward task:

byte[] data = "Simple request body".getBytes();

HttpRequest.newBuilder()
  .uri(new URI("https://postman-echo.com/post"))
     .headers("Content-Type", "text/plain;charset=UTF-8")
     .POST(ofInputStream(() -> new ByteArrayInputStream(data)))
     .build();

Notice how I used a simple ByteArrayInputStream here; that can, of course, be any InputStream implementation.

You can also create a ByteArray body using a ByteArrayProcessor and pass an array of bytes as the parameter:

HttpRequest request = HttpRequest.newBuilder()
.uri(new URI("https://postman-echo.com/post"))
.headers("Content-Type", "text/plain;charset=UTF-8")
.POST(ofByteArray(data))
.build();

Finally, you can create a body from a file. To work with a file, you can make use of the provided FileProcessor; its factory method takes a path to the file as a parameter and creates a body from the content:

HttpRequest.newBuilder()
.uri(new URI("https://postman-echo.com/post"))
.headers("Content-Type", "text/plain;charset=UTF-8")
.POST(ofFile(Paths.get("src/test/resources/data.txt")))
.build();

In this section, I’ve shown you how to create an HttpRequest, how to configure it, and finally how to create a different request body. Now let’s see how you can use a new HttpClient to send requests to the server, synchronously or asynchronously, using the request you created.

Sending Sync or Async requests

HttpClient provides two possibilities for sending a request to a server:

  • send(…) synchronously (blocks until the response comes)
  • sendAsync(…) asynchronously (doesn’t wait for a response, non-blocking)

Up until now, the send(…) method naturally waits for a response:

HttpResponse<String> response = HttpClient.newBuilder()
.build()
.send(request, BodyHandlers.ofString());

This call returns an HttpResponse object and this means that the next instruction from your application flow will be executed only when the response is already returned.

This approach has a lot of drawbacks, especially when you are processing large amounts of data. To overcome this limitation, you can use the sendAsync(…) method, which returns a CompletableFeature<String> to process a request asynchronously:

CompletableFuture<String> response = HttpClient.newBuilder()
.build()
.sendAsync(request, BodyHandlers.ofString())
  .thenApply(HttpResponse::body);

The API can also deal with multiple responses and stream the request and response bodies:

List<URI>uris = Arrays.asList(
new URI("https://postman-echo.com/get?foo1=bar1"),
new URI("https://postman-echo.com/get?foo2=bar2"));

HttpClient client = HttpClient.newHttpClient();

List<HttpRequest> requests = uris.stream()
        .map(HttpRequest::newBuilder)
        .map(reqBuilder ->reqBuilder.build())
        .collect(Collectors.toList());

CompletableFuture.allOf(requests.stream()
        .map(request ->client.sendAsync(request, ofString()))
        .toArray(CompletableFuture<?>[]::new))
        .join();

So your main code will keep on executing, configuring the callback on the future, and thenAccept. But this callback will only be triggered once the server returns a response. The HTTP client will use a background thread to make the call.

Be aware, the server response will take a while. And in the meantime, your application will have ended. So what do you do to make this example work? Call the join method on the future.

This joins the application thread that your code runs on with the future. At this point in the code, the join method will wait until the future has been completed. And if it’s completed, that also means that your thenAccept callback will run. And indeed, when you run this example, you get back the expected results.

Setting an executor for asynchronous calls

You can also define an executor which provides threads to be used by asynchronous calls. This way you can perform such tasks as limiting the number of threads used for processing requests:

ExecutorService executor= Executors.newFixedThreadPool(6);

CompletableFuture<HttpResponse<String>> response1 = HttpClient.newBuilder()
.executor(executor)
.build()
.sendAsync(request,asString());

By default, the HttpClient uses executor java.util.concurrent.Executors.newCachedThreadPool().

The synchronous and blocking send API is easier to use, but the asynchronous API will help you create responsive and more scalable applications, so you’ll have to choose what best fits your use case.

Now let’s focus on the last class from the HTTP Client API &MDASH; HttpResponse.

Getting response information with an HttpResponse object

The HttpResponse class represents the response from the server. It provides a number of useful methods, the most important ones being:

  • statusCode() returns status code (type int) for a response (HttpURLConnection class contains possible values)
  • body() returns a body for a response (return type depends on the response BodyHandler parameter passed to the send() method)

The response object has other useful methods such as uri(), headers(), trailers(), and version().

URI of response object

The method uri() on the response object returns the URI from which we received the response.

Sometimes it can be different than the URI in the request object because a redirection may occur:

assertThat(request.uri()
.toString(), equalTo("http://stackoverflow.com"));

Headers from response

You can obtain headers from the response by calling method headers() on a response object:

HttpClient.newHttpClient()
      .send(request, BodyHandlers.ofString());

HttpHeadersresponseHeaders = response.headers();

It returns an HttpHeaders object as a return type. This is a new type defined in the java.net.http package which represents a read-only view of HTTP headers.

It has some useful methods which simplify searching for headers value.

Response version

You can use the request object version() method to define which version of HTTP protocol was employed to talk with the server. You will get a server version answer, but remember, even if you have defined that you want to use HTTP/2, the server may answer via HTTP/1.1.

HttpRequest.newBuilder()
.uri(new URI("https://postman-echo.com/get"))
.version(HttpClient.Version.HTTP_2)
.GET()
.build();

HttpClient.newHttpClient()
.send(request, BodyHandlers.ofString());

assertThat(response.version(), equalTo(HttpClient.Version.HTTP_1_1));

Now let’s explore the last piece of this article, the WebSocket API.

WebSocket Client API

The HTTP Client API also supports the WebSocket protocol, which is used in real-time web applications to provide client-server communication with low message overhead.

The java.net.http module also contains a client for WebSocket communication. WebSocket interface is the heart of this new addition which contains four other abstractions to build, represent close codes, listen for events and messages, and finally, handle partial messages.

The full code classes EchoListener and WebSocketApp reside in the com.tm.siriusxi.http.ws package.

For starters, you can implement the WebSocket.Listener interface, which, as its name suggests, is a listener for events and messages on a WebSocket:

public final class EchoListener implements WebSocket.Listener {

    ….
    @Override
    public void onOpen(WebSocket webSocket) {
        logger.info("CONNECTED");

webSocket.sendText("This is a message", true);

WebSocket.Listener.super.onOpen(webSocket);
    }

    @Override
    public CompletionStage<?>onText(WebSocket webSocket, CharSequence data, boolean last) {
        logger.info("Receiving Message -->");

        logger.info(String.format("onText received a data: %s", data));

        if (!webSocket.isOutputClosed()) {
webSocket.sendText("This message should echoed back..", true);
        }

        return WebSocket.Listener.super.onText(webSocket, data, last);
    }

    @Override
    public CompletionStage<?>onClose(WebSocket webSocket, intstatusCode, String reason) {
        logger.info("CLOSING");

        logger.info(String.format("Closed with status %d, and reason: %s", statusCode, reason));

executor.shutdown();

        return WebSocket.Listener.super.onClose(webSocket, statusCode, reason);
    }
}

You can use an HttpClient to create a WebSocket using the newWebSocketBuilder() factory method. It connects to a URI, sends messages for one second, and closes its output. The API also makes use of asynchronous calls that return CompletableFuture.

WebSocket webSocket = httpClient.newWebSocketBuilder()
        .buildAsync(URI.create("ws://demos.kaazing.com/echo"),
                new EchoListener(executor)).join();

webSocket.sendClose(WebSocket.NORMAL_CLOSURE, "ok").thenRun(() -> logger.info("Sent close")).join();

More examples and cases

In the project examples, under the HttpClientApp class, you will find more common examples which you can use in your day-to-day job tasks. These examples are divided into methods with JavaDoc explanations that cover the following scenarios:

  • Synchronous Get request
    • Response body as a String
    • Response body as a File
  • Asynchronous Get request
    • Response body as a String
    • Response body as a File
  • Post request
  • Concurrent requests
  • Get JSON response
  • Post JSON request
  • Setting a proxy

We have reached the end of this tutorial and have explored Java SE 11’s HttpClient API, which provides a lot of flexibility and powerful features in details.

Summary

I’ve provided the background of the HTTP Client and WebSocket APIs and discussed their evolution through the incubator modules in JDK 9 and JDK 10 and the standardized versions in Java SE 11.

You’ve explored the examples on how to use the API to perform common tasks, such as issuing an HTTP request and receiving the response asynchronously, how to chain asynchronous tasks depending on the response, and how the API leverages reactive streams and can interoperate with various reactive streams implementations.

You can find HTTP Client examples on GitHub.