Nonblocking and Streaming - Part 1

Overview

Modern versions of Catalyst allow you to take charge of how your write operations work. This enables better support for working inside common event loops such as AnyEvent and IO::Async to support non blocking writes to stream large files.

Lets explore how PSGI implements delayed and streaming responses and how we can use streaming response with an event loop to enable use of nonblocking IO. Then lets see how Catalyst exposes that API. This part one of a four part series.

Introduction

When Catalyst was ported to run as a native PSGI application, great pains were taken to make sure we could properly support delayed and streaming responses. However only recently was that ability properly exposed as Catalyst API. In order to better understand how Catalyst works, lets step back and refresh on the PSGI specification.

PSGI Delayed Response and Streaming

You are likely familiar with the basic PSGI 'hello world' application which goes something like this:

    my $psgi_app = sub {
      my $env = (@_);
      return [ 200,
        [ 'Content-Type' => 'text/plain' ],
        [ 'Hello World!'] ];
    };

If you put the above into a file hello-classic.psgi you could run it under a decent webserver very easily:

    $ plackup -s Starman bin/hello-classic.psgi

    013/11/27-09:18:04 Starman::Server (type Net::Server::PreFork) starting! pid(69860)
    Resolved [*]:5000 to [0.0.0.0]:5000, IPv4
    Binding to TCP port 5000 on host 0.0.0.0 with IPv4
    Setting gid to "20 20 20 12 61 79 80 81 98 33 100 204 398 399"
    Starman: Accepting connections at http://*:5000/

And now you can use a commandline tool like telnet to hit that server and execute HTTP requests:

    $ telnet 127.0.0.1 5000
    Trying 127.0.0.1...
    Connected to localhost.
    Escape character is '^]'.
    GET / HTTP/1.1
    Host: www.test.org

    HTTP/1.1 200 OK
    Content-Type: text/plain
    Transfer-Encoding: chunked
    Date: Wed, 27 Nov 2013 22:49:39 GMT
    Connection: keep-alive

    c
    Hello World!
    0

    Connection closed by foreign host.

Take note of the Response header 'Transfer-Encoding: chunked'. Since Starman understand HTTP/1.1 it can send chunked responses in the case when you fail to provide the content length (or, you give an expected length but specify 'chunked' anyway). In this case that 'c' that you see is the length in hexadecimal (12 bytes). That final '0' means there's no more chunks to send.

Now, PSGI offers a few more tricks for the case when you don't want to fully pre-generate your response. In this case instead of returning a tuple (Status, Headers, Body) from your PSGI application, you return a code reference:

    my $psgi_app = sub {
      my $env = shift;
      return my $delayed_response = sub {
        my $responder = shift;
        my $writer = $responder->(
          [200, [ 'Content-Type' => 'text/plain' ]]);

        $writer->write('Hello');
        $writer->write(' ');
        $writer->write('World');
        $writer->close;
      };
    };

In this case the $delayed_response coderef gets run by the underlying server, which is required to pass it a second coderef, the $responder. So basically your application is 'delayed' it doesn't actually run until the server runs it, which can allow you to take advantage of event loops like AnyEvent or IO::Async to schedule your response better. But its still useful even under a blocking server like Starman.

Lets look a bit more at the code example. The $responder coderef can accept the full $Status, \@Headers, \@Body tuple, but a more interesting use is to give it just the first two elements of the tuple, as we do in this example application. In this case the $responder returns the server's $writer object, which basically is like a filehandle that points at the client which originated the request (for example a web browser or the telnet commandline application). This object defines two methods, write and close. Each time you call write you stream some data out to the client. Generally the connection stays open until you call close, which means you can use this for long responses or for techniques like cometd where you keep a connection to the client open for a long time.

*Note: Even thought I say the $writer object can be thought of as a file handle to the client, the underlying architecture is both more complex and yet lacking in some of the features we expect from a filehandle. For example, in the current PSGI spec, there's no clear and common way to detect when you can't write due to some issue with the client or network. Also, there's nothing stopping the server from buffering your writes before sending them on to the client, if it is necessary to do so for scheduling purposes. For example, the server might buffer your write in memory, or to some other temporary store if needed.

Hopefully someday the plack-cabal can get together on a specification to expose and clarify some of these important missing bits.

Let's see what this application looks like under telnet. We start the psgi application with plackup as in the previous case.

    $ plackup -s Starman bin/hello-delayed.psgi 
    2013/11/27-09:52:15 Starman::Server (type Net::Server::PreFork) starting! pid(69989)
    Resolved [*]:5000 to [0.0.0.0]:5000, IPv4
    Binding to TCP port 5000 on host 0.0.0.0 with IPv4
    Setting gid to "20 20 20 12 61 79 80 81 98 33 100 204 398 399"
    Starman: Accepting connections at http://*:5000/

And hit it with telnet:

    $ telnet 127.0.0.1 5000
    Trying 127.0.0.1...
    Connected to localhost.
    Escape character is '^]'.
    GET / HTTP/1.1
    Host: www.test.org

    HTTP/1.1 200 OK
    Content-Type: text/plain
    Transfer-Encoding: chunked
    Date: Wed, 27 Nov 2013 22:57:11 GMT
    Connection: keep-alive

    5
    Hello
    1

    5
    World
    0

    Connection closed by foreign host.

You'll note this time each of the writes is a separate chunk, with a separate length. Now, we used the chunked encoding here to help illustrate how each call to ->write pushes a separate 'chunk', but to be clear, we could have used this streaming interface without chunked transfer encoding. For example, the same code with estimated length in the header:

    my $psgi_app = sub {
      my $env = shift;
      return sub {
        my $responder = shift;
        my $writer = $responder->(
          [200, [ 
            'Content-Type' => 'text/plain',
            'Content-Length' => '11', ]]);

        $writer->write('Hello');
        $writer->write(' ');
        $writer->write('World');
        $writer->close;
      };
    };

And the telent trace:

    $ telnet 127.0.0.1 5000
    Trying 127.0.0.1...
    Connected to localhost.
    Escape character is '^]'.
    GET / HTTP/1.1
    Host: www.test.org

    HTTP/1.1 200 OK
    Content-Type: text/plain
    Content-Length: 11
    Date: Wed, 27 Nov 2013 23:06:33 GMT
    Connection: keep-alive

    Hello WorldConnection closed by foreign host.

so we still streamed the information, just its not as easy to see from looking at the telent trace, as it is when using chunked transfer encoding. But its definitely still streaming, as you can see if you start Starman in DEBUG mode and examine the debug output for that request:

    >$ STARMAN_DEBUG=1 plackup -s Starman bin/hello-delayed-with-length.psgi 
    2013/11/27-17:09:44 Starman::Server (type Net::Server::PreFork) starting! pid(71887)
    Resolved [*]:5000 to [0.0.0.0]:5000, IPv4
    Binding to TCP port 5000 on host 0.0.0.0 with IPv4
    Setting gid to "20 20 20 12 61 79 80 81 98 33 100 204 398 399"
    Using no serialization
    Starman: Accepting connections at http://*:5000/
    Beginning prefork (5 processes)
    Starting "5" children
    Child Preforked (71888)
    Child Preforked (71889)
    Child Preforked (71890)
    Parent ready for children.
    Child Preforked (71891)
    Child Preforked (71892)

    2013/11/27-17:09:51 CONNECT TCP Peer: "[127.0.0.1]:61580" Local: "[127.0.0.1]:5000"
    [71888] Read 16 bytes: "GET / HTTP/1.1\r\n"
    [71888] Read 20 bytes: "Host: www.test.org\r\n"
    [71888] Read 2 bytes: "\r\n"
    127.0.0.1 - - [27/Nov/2013:17:09:52 -0600] "GET / HTTP/1.1" 200 - "-" "-"
    [71888] Wrote 126 bytes
    [71888] Wrote 5 bytes
    [71888] Wrote 1 byte
    [71888] Wrote 5 bytes
    [71888] Request done
    [71888] Waiting on previous connection for keep-alive request...
    [71888] Closing connection

Here you can see the output very clearly. Starman sends 126 bytes to the client (this is the HTTP header information) followed by three writes of 5, 1 and 5 bytes each.

BTW, if you actually go at run the code you'll actually see the 1 second 'keep alive' delay at the end of the response. Basically after the write calls ->close, it spends a bit of time waiting to see if that connection sends a a request to hold the connection open. For Starman this is 1 second, but you can configure it. This keep alive is a big part of HTTP 1.1 and all HTTP 1.1 connects are considered persistent unless declared otherwise, so that's why its important for Starman and other HTTP 1.1 servers to have that ability to timeout if the client isn't doing anything with the persistent connection.

You can actually see this keep alive in action with STARMAN_DEBUG and if you are speedy with the second request (or bump up the starman keepalive timeout). For example:

    $ telnet 127.0.0.1 5000
    Trying 127.0.0.1...
    Connected to localhost.
    Escape character is '^]'.
    GET / HTTP/1.1              (first request which initiates the connection)
    Host: www.test.org

    HTTP/1.1 200 OK
    Content-Type: text/plain
    Content-Length: 11
    Date: Wed, 27 Nov 2013 23:20:13 GMT
    Connection: keep-alive

    Hello World

    GET / HTTP/1.1              (second request over the same connection)
    Host: www.test.org

    HTTP/1.1 200 OK
    Content-Type: text/plain
    Content-Length: 11
    Date: Wed, 27 Nov 2013 23:20:14 GMT
    Connection: keep-alive

    Hello World

    Connection closed by foreign host.

Here's the Starman debugging output.

    2013/11/27-17:20:12 CONNECT TCP Peer: "[127.0.0.1]:61630" Local: "[127.0.0.1]:5000"
    [71889] Read 16 bytes: "GET / HTTP/1.1\r\n"
    [71889] Read 20 bytes: "Host: www.test.org\r\n"
    [71889] Read 2 bytes: "\r\n"
    127.0.0.1 - - [27/Nov/2013:17:20:13 -0600] "GET / HTTP/1.1" 200 - "-" "-"
    [71889] Wrote 126 bytes
    [71889] Wrote 5 bytes
    [71889] Wrote 1 byte
    [71889] Wrote 5 bytes
    [71889] Request done
    [71889] Waiting on previous connection for keep-alive request...
    [71889] Read 36 bytes: "GET / HTTP/1.1\r\nHost: www.test.org\r\n"
    [71889] Read 2 bytes: "\r\n"
    127.0.0.1 - - [27/Nov/2013:17:20:14 -0600] "GET / HTTP/1.1" 200 - "-" "-"
    [71889] Wrote 126 bytes
    [71889] Wrote 5 bytes
    [71889] Wrote 1 byte
    [71889] Wrote 5 bytes
    [71889] Request done
    [71889] Waiting on previous connection for keep-alive request...
    [71889] Closing connection

So you can see one connection, two requests. This keep alive forms the basis of many long polling and similar techniques (like cometd) for keeping a persistent connection between the client and the server, for the purposes of enabling speedy (semi - realtime) type interfaces. Although of course since Starman is a forked, blocking server, you'd be limited to the number of persistent connections by the number of forked children (in Starman the default is 5). If you want to scale such a thing, you probably need to switch to a non blocking server, which we will discuss later.

Again, keep alive and chunked responses are separate matters, just it makes it a bit easier to visually see what is going on. Some reasons for streaming (in my mind) are for when you have very large responses that you'd rather not buffer in memory or to a temp file, but would prefer to send in bits (sort of like when you have a very large SQL query and you use a cursor to iterate row by row rather than load all the rows into an array). On the other hand, chunked encoding is useful when you don't know upfront the length you are sending (such as when getting the length is computationally expensive, or can't be known initially, or when the intention is to send infinitely. The two concepts overlap in use case but the technology themselves is separate.

Great, so that's how PSGI does delayed response and streaming. In the following article, we'll take look at how this works under Catalyst.

For More Information

Summary

We've reviewed various parts of the PSGI specification, and set a good foundation of understanding to proceed with the remaining articles.

Author

John Napiorkowski jjnapiork@cpan.org