libapr provides APIs to support network programming. The APIs are based on traditional socket programming scheme. If you are familiar with bind(2), listen(2), accept(2) and connect(2), you can easily understand it. I show you how to use the APIs with three categories, server side progamming, client side progaramming, and multiplex process programming.
A typical server process opens a listen port, and listen to the port for any client process to connect. Then, it accepts the client's connection, and communicate with the client using their network protocol. Although the network protocol depends on your application, the basic structure of server code is almost same.
At first, we have to create a socket address object, apr_sockaddr_t. In traditional socket programming, socket address structure would cause confusion. In contrast, libapr's socket address structure is simple. It can hide chaos among various platforms and IPv4/IPv6 stacks. We can create the object by apr_sockaddr_info_get(). The prototype declaration is as follows:
/* excerpted from apr_network_io.h */
APR_DECLARE(apr_status_t) apr_sockaddr_info_get(apr_sockaddr_t **sa,
const char *hostname,
apr_int32_t family,
apr_port_t port,
apr_int32_t flags,
apr_pool_t *p);
The first argument is result argument. The second argument is a hostname, or an IP address. I'll describe it later. The third argument is address family. It is usually APR_INET. If you intend to use IPv6, please specify APR_INET6. The fourth argument is a port number. Server side program should specify the port number to listen. For example, if you're creating a Web server, you might have to specify number 80. As you will see, client side program specifies the port number of the remote server. So, if you're creating a Web browser, you might have to specify number 80, too. Note that you can set the port number to zero. If so, the system selects a port number, which is available. The fifth argument is flags. In most cases, it is 0. The final argument is memory pool to use.
We are back to the second argument. As you will see in client side programming below, client program generally specifies the server(remote) hostname or IP address. In contrast, server program should specifies its local address, and it binds the socket address object to a socket by apr_socket_bind().
What value is allowed as a local address? One choice is a solid address or hostname. Namely, if you are creating a yahoo server, you can specify "www.yahoo.com" or "66.94.230.38". In general, such values are supplied by configuration files. Second choice is loopback address, i.e. "127.0.0.1" or "localhost". It works and it's valid. However, in such a case, only a client process running on the same host can connect to the server. If your purpose is to allow only local processes to connect to your server, specifying loopback address is a right choice. The other choice is to specify NULL or APR_ANYADDR(="0.0.0.0"). They imply that the server will bind all network interfaces to a socket. Accordingly, in such a case, any client process can connect to the server via a solid address or loopback address. Internally, NULL is better than APR_ANYADDR. As a result, it's usually good to specify NULL as the second argument of apr_sockaddr_info_get(). There is one exception. When the server host is multihome host, i.e. it has multiple solid IP addresses, and you don't want some IP addresses available from remote hosts, you shouldn't specify NULL. You must bind solid IP addresses to the listening socket.
Next, we create a socket object, apr_socket_t. In traditional socket progamming, socket type is just integer, and it acts as a file descriptor. apr_socket_t is opaque type and it hides such the OS dependencies. We can create socket object by apr_socket_create(). The prototype declaration is as follows:
/* excerpted from apr_network_io.h */
APR_DECLARE(apr_status_t) apr_socket_create(apr_socket_t **new_sock,
int family, int type,
int protocol,
apr_pool_t *cont);
The first argument is result argument. The second argument is address family. It is same as one of apr_sockaddr_info_get()'s third argument. Later, we make a relation between socket address object and socket object. In which, if they have different address family, it fails. The third and fourth arguments are socket type and protocol type. In general, all you have to know is two combinations. One is SOCK_STREAM as type and APR_PROTO_TCP as protocol. The other is SOCK_DGRAM as type and APR_PROTO_UDP as protocol. The final argument is memory pool to use.
Now, let's take a look at server-sample.c. The following is a typical code of server side to create a TCP listening socket.
/* excerpted from server-sample.c, but I omitted error checks */
apr_sockaddr_t *sa;
apr_socket_t *s;
apr_sockaddr_info_get(&sa, NULL, APR_INET, DEF_LISTEN_PORT, 0, mp);
apr_socket_create(&s, sa->family, SOCK_STREAM, mp);
apr_socket_bind(s, sa);
apr_socket_listen(s, DEF_SOCKET_BACKLOG);
There are two application dependent constant numbers, DEF_LISTEN_PORT and DEF_SOCKET_BACKLOG. Don't care about them.
You can find two new APIs, apr_socket_bind() and apr_socket_listen(). By calling apr_socket_bind(), we can make a relation between socket address object(apr_sockaddr_t) and socket object(apr_socket_t). We call it binding the address to the socket. Then, by calling apr_socket_listen(), we change the socket's status to listening. It means we allow the socket to accept connections from remote clients. The second argument of apr_socket_listen() is length of the internal queue. The queue is a waiting queue of remote clients. They wait in the queue in (OS)kernel until the application accepts them. The length is historically called backlog. If you are not sure about proper value, I suggest you to use 'SOMAXCONN', which is a system default max value.
Next, we have to handle new connections from remote clients. At first, we have to call apr_socket_accept().
/* excerpted from server-sample.c, but I omit error checks */
apr_socket_t *ns;/* accepted socket */
apr_socket_accept(&ns, s, mp);
The second argument of apr_socket_accept() is the listening socket that we create above. Here, we get another socket object, named 'ns', accepted socket. From socket object creation's perspective, apr_socket_accept() is similar to apr_socket_create(). Namely, apr_socket_accept() also creates a new socket object. The first argument is result argument. Note that the newly created socket object is completely different from the listening socket object, which is passed as second argument to apr_socket_accept(). After returing from apr_socket_accept(), the listening socket is still just listening socket. It means we can continue to call apr_socket_accept(), and when the other remote client connects to the server, we're going to have a yet another socket object from apr_socket_accept(). After apr_socket_accept(), we have to handle two sockets independently. In general, we have the listening socket keep listening, and we have the accepted socket talk network protocol with the remote client. In server-sample.c, we send a simple HTTP request and receive the response. To send and receive using socket, we call apr_socket_send() and apr_socket_recv(). Their prototype declarations are as follows:
/* excerpted from apr_network_io.h */
APR_DECLARE(apr_status_t) apr_socket_send(apr_socket_t *sock, const char *buf,
apr_size_t *len);
APR_DECLARE(apr_status_t) apr_socket_recv(apr_socket_t *sock,
char *buf, apr_size_t *len);
They are similar to apr_file_read() and apr_file_write() described above. The first and second argumnts are needless to explain. The third argument of both is value-result argument. By which, we specify the input buffer's length on entry and get the output result's length on exit. apr_socket_send() returns the length of sent bytes. apr_socket_recv() returns the length of received bytes.
We will discuss about return values of those APIs later.
Comparing to server side programming, client side programming looks simpler. A new API to note is only one, apr_socket_connect().
/* excerpted from apr_network_io.h */
APR_DECLARE(apr_status_t) apr_socket_connect(apr_socket_t *sock,
apr_sockaddr_t *sa);
This is similar to apr_socket_bind() on those arguments list. Like apr_socket_bind(), apr_socket_connect() makes a relation between socket object and socket address object. The difference is that the socket address object is related to the remote server address. Let's take a look at client-sample.c. The following is a typical code of client side program.
/* excerpted from client-sample.c, but I omit error checks */
apr_sockaddr_t *sa;
apr_socket_t *s;
apr_sockaddr_info_get(&sa, DEF_REMOTE_HOST, APR_INET, DEF_REMOTE_PORT, 0, mp);
apr_socket_create(&s, sa->family, SOCK_STREAM, mp);
apr_socket_connect(s, sa);
The second and fourth arguments of apr_sockaddr_info_get() are remote server's IP address(or hostname) and port number. The others are needless to explain, because they are same as server side. apr_socket_create() is also needless to explain, because it's also same as server side. By calling apr_socket_connect(), we begin to connect to the remote server. I'll explain return values of these APIs.
After connecting to the server, the client program talks network protocol with the server through the socket. As you can see in client-sample.c, the code is same as server side. We can use apr_socket_send() and apr_socket_recv().
Go back to server-sample.c again. It has a loop to keep calling apr_socket_accept(), so that the server process can accept multiple clients. However, it doesn't call apr_socket_accept() until it completes the session between one client. Obviously, when multiple clients connect to the server simultaneously, the server handles only one client at a moment. Such a server is normally useless. We need concurrent server. In general, there are three programming models to develop concurrent server; multi-process model, multi-threaded model, and multiplexing model. Here, we focus on the final one.
Historically, multiplexing model relies on select(2) or poll(2) APIs, and the code is often based on event-driven style. The essential idea is that we check multiple sockets whether they are ready to read or write. Then, when we know some of the sockets are ready to read/write, we do actual read/write with them. In addition, we need non-blocking sockets for multiplex processing. I'll describe how to use non-blocking sockets in the next section.
Let's take a look at pollset-sample.c. The creation of the listening socket is almost same as server-sample.c. The different is that we make it non-blocking. Then, we create a pollset obejct by apr_pollset_create(). pollset is a set of sockets to watch. The prototype declaration is as follows:
/* excerted from apr_poll.h */
APR_DECLARE(apr_status_t) apr_pollset_create(apr_pollset_t **pollset,
apr_uint32_t size,
apr_pool_t *p,
apr_uint32_t flags);
The first argument is result argument. The second argument is size of pollset. Unfortunately, pollset is not a dynamic-sized set. We have to specify the size of set at creation. The third argument is memory pool to use. The fourth argument is flag, but it's a reserved argument now.
After creating pollset object, we can add sockets to the pollset by apr_pollset_add(). In the following example, we add the first and second sockets to the pollset to know whether the socket is ready to read. Then we remove a socket from the pollset because we are not interested in its readability. Finally, we add the third socket to know whether it is ready to write.
/* excerpted from pollset-sample.c */
/* we watch @lsock(listening socket) to know whether it is ready to read(APR_POLLIN) */
apr_pollfd_t pfd = { mp, APR_POLL_SOCKET, APR_POLLIN, 0, { NULL }, NULL };
pfd.desc.s = lsock;
apr_pollset_add(pollset, &pfd);
/* we watch @ns(connected socket) to know whether it is ready to read(APR_POLLIN) */
apr_pollfd_t pfd = { mp, APR_POLL_SOCKET, APR_POLLIN, 0, { NULL }, serv_ctx };
pfd.desc.s = ns;
apr_pollset_add(pollset, &pfd);
/* we are not interested in the socket's readability, so remove it from the pollset */
apr_pollfd_t pfd = { serv_ctx->mp, APR_POLL_SOCKET, APR_POLLIN, 0, { NULL }, serv_ctx };
pfd.desc.s = sock;
apr_pollset_remove(pollset, &pfd);
/* we watch @sock(connected socket) to know whether it is ready to write(APR_POLLOUT) */
pfd.reqevents = APR_POLLOUT;
apr_pollset_add(pollset, &pfd);
We need to look at apr_pollfd_t's definition. It is in apr_poll.h as follows:
/* excerpted from apr_poll.h */
/** Union of either an APR file or socket. */
typedef union {
apr_file_t *f; /**< file */
apr_socket_t *s; /**< socket */
} apr_descriptor;
/** Poll descriptor set. */
struct apr_pollfd_t {
apr_pool_t *p; /**< associated pool */
apr_datatype_e desc_type; /**< descriptor type */
apr_int16_t reqevents; /**< requested events */
apr_int16_t rtnevents; /**< returned events */
apr_descriptor desc; /**< @see apr_descriptor */
void *client_data; /**< allows app to associate context */
};
pollset is designed to work for files as same as for sockets, but we ignore files in this document. We set apr_pollfd_t::desc_type to APR_POLL_SOCKET, and we specify socket object as apr_pollfd_t::desc. apr_pollfd_t::reqevents and apr_pollfd_t::rtnevents have a relation, the former is input and the latter is output. The values are bit-wised flag of APR_POLLIN, APR_POLLOUT, etc. The following sample shows it.
/* if you want to know whether the socket is ready to read or write, you can specify the bit-wised flag as follows */
apr_pollfd_t pfd = { mp, APR_POLL_SOCKET, APR_POLLIN|APR_POLLOUT, 0, { sock }, app_ctx };
apr_pollset_add(pollset, &pfd);
apr_pollfd_t::rtnevents doesn't have any meaning when the apr_pollfd_t object is used for apr_pollset_add() or apr_pollset_remove(). It does have meaning when the object is returned from apr_pollset_poll(). apr_pollset_poll() prototype declaration is as follows:
/* excerpted from apr_poll.h */
APR_DECLARE(apr_status_t) apr_pollset_poll(apr_pollset_t *pollset,
apr_interval_time_t timeout,
apr_int32_t *num,
const apr_pollfd_t **descriptors);
The first argument is that we pass our pollset object. The second argument is timeout value. If the timeout is zero, it implies no timeout. The third and fourth argument are result arguments. By these result arguments, we get the number of active sockets, which are ready to read or write, and we get an array of the active socket objects. The array is apr_pollfd_t's array. We can know whether the sockets are ready to read/write by checking apr_pollfd_t::rtnevents's values. Here is a pseudo code.
/* pseudo code to show how apr_pollset_poll() works */
const apr_pollfd_t *ret_pfd; /* returned pollset */
rv = apr_pollset_poll(pollset, DEF_POLL_TIMEOUT, &num, &ret_pfd);
if (rv == APR_SUCCESS) {
/* num and ret_pfd are result-arguments.
* ret_pfd is an array of apr_pollfd_t, and the array size is num */
for (int i = 0; i < num; i++) {
if (ret_pfd[i].rtnevents & APR_POLLIN) {
printf("socket %p is ready to read\n", ret_pfd[i].desc.s);
} else if (ret_pfd[i].rtnevents & APR_POLLOUT) {
printf("socket %p is ready to write\n", ret_pfd[i].desc.s);
}
}
}
We have glanced network APIs in the previous sections. It doesn't seem so difficult. Nevertheless, real network programming is not so easy. In particular, it is hard to make your software efficient, secure and robust to various network errors. Here, I'll show you some hints for real programming.
There are mainly three modes with sockets, blocking-forever, blocking-with-timeout, and non-blocking. The mode is controlled by two APIs, apr_socket_opt_set() and apr_socket_timeout_set(). We have a portability issue on this. The table below shows it.
| APR_SO_NONBLOCK | timeout value to apr_socket_timeout_set() | mode |
| off(==0) | t == 0 | non-blocking |
| off(==0) | t < 0 | blocking-forever |
| off(==0) | t > 0 | blocking-with-timeout |
| on(==1) | t == 0 | non-blocking |
| on(==1) | t < 0 | blocking-forever |
| on(==1) | t > 0 | blocking-with-timeout |
| APR_SO_NONBLOCK | timeout value to apr_socket_timeout_set() | mode |
| off(==0) | t == 0 | blocking-forever |
| off(==0) | t < 0 | blocking-forever |
| off(==0) | t > 0 | blocking-with-timeout |
| on(==1) | t == 0 | non-blocking |
| on(==1) | t < 0 | non-blocking |
| on(==1) | t > 0 | non-blocking |
The default mode is APR_SO_NONBLOCK==0(off) and APR_SO_TIMEOUT==-1. Namely, default socket is blocking-forever on both Unix and Windows.
Conclusion (my recommendation): [a] When you want a non-blocking socket, set it to 'APR_SO_NONBLOCK==1(on) and timeout==0'.
[b] When you want a blocking-with-timeout socket, set it to 'APR_SO_NONBLOCK==0(off) and timeout>0'. Note that you must keep the order of calling the APIs. You must call apr_socket_opt_set(sock, APR_SO_NONBLOCK, 1) and then call apr_socket_timeout_set(sock, timeout). Otherwise, on Unix the socket becomes blocking-forever.
[c] When you want a blocking-forever socket, set it to 'APR_SO_NONBLOCK==0(off) and timeout<0'. In my opinion, we merely need blocking-forever sockets for real application.
We can control blocking/non-blocking mode for apr_socket_recv() and apr_socket_send() as I described above. How about other APIs?
We can ignore apr_socket_bind() and apr_socket_listen(). Because they never block. Simply, these APIs are always non-blocking mode. We have two more APIs to consider, apr_socket_accept() and apr_socket_connect().
Here, we consider apr_socket_accept(). It is almost same as apr_socket_recv()/apr_socket_send(). The mode follows the table in the previous section. Unlike apr_socket_recv()/apr_socket_send(), blocking-forever socket is useful for apr_socket_accept(). It is because that the program might be a server process and might have nothing to do until any client connects to.
If we write a mulplexing model code, we need non-blocking socket for listening socket. We check listening socket whether it is ready to read. Readiness to read indicates that any client has connected to the socket. After we know the readiness to read, we just call apr_socket_accept(). Please take a look at pollset-sample.c about this pattern. Note that you must not assume the newly connected socket, which is returned from apr_socket_accept(), inherits the mode from the listening socket. You should specify the mode for the connected socket explicitly. Please look at the following example.
/* excerpted from pollset-sample.c, but I omitted error checks */
apr_socket_accept(&ns, lsock, mp);
...SNIP...
/* non-blocking socket. We can't expect that @ns inherits non-blocking mode from @lsock */
apr_socket_opt_set(ns, APR_SO_NONBLOCK, 1);
apr_socket_timeout_set(ns, 0);
apr_socket_connect() is a bit different from other APIs on blocking/non-blocking mode. It has three modes, blocking-with-system-timeout, blocking-with-timeout, and non-blocking. Unlike other APIs, apr_socket_connect() never blocks forever. The default mode is blocking-with-system-timeout. The timeout value depends on OS, and it is relatively longer, e.g. over one minute. In my opinion, it is not good to use blocking-with-system-timeout mode for real applications, because it is uncontrollable. We have to set either blocking-with-timeout or non-blocking to the mode.
To make blocking-with-timeout sockets, we have to set it to 'APR_SO_NONBLOCK==1(on) and timeout>0'. As you see, this is not same as above. On Unix, we have no problem to specify 'APR_SO_NONBLOCK==0(off) and timeout>0'. Unfortunatelly, we have a problem on Windows. Setting the mode to 'APR_SO_NONBLOCK==0(off) and timeout>0' causes blocking-with-system-timeout sockets on Windows.
Conclusion: If we want blocking-with-timeout socket without portability issues, we should write code as follows:
/* pseudo code: blocking-with-timeout apr_socket_connect() */
apr_socket_opt_set(sock, APR_SO_NONBLOCK, 1);
apr_socket_timeout_set(sock, positive_timeout);
apr_socket_connect(sock, sa);
/* still blocking-with-timeout for other operations, apr_socket_send()/apr_socket_recv() */
apr_socket_opt_set(sock, APR_SO_NONBLOCK, 0);
apr_socket_timeout_set(sock, positive_timeout);
apr_socket_send(sock, ...);
apr_socket_recv(sock, ...);
Blocking-with-timeout apr_socket_connect() returns APR_SUCCESS if the connection has been established successfully. Otherwise, it returns an error value. For example, APR_TIMEUP, APR_ECONNREFUSED, or APR_EHOSTUNREACH. If error value is APR_ECONNREFUSED, the server process's listen(2) backlog is beyond the limit. Please see apr_socket_listen() description above.
For mulplexing model programming, we need non-blocking sockets with apr_socket_connect(). To make non-blocking socket for apr_socket_connect(), set it to 'APR_SO_NONBLOCK==1(on) and timeout==0' as usual. Non-blocking apr_socket_connect() returns APR_EINPROGRESS unless the connection is established. In general, apr_socket_connect() can't establish the connection immediately. Accordingly, non-blocking apr_socket_connect() usually returns APR_EINPROGRESS except that the connection is established between local processes, which run on the same host. In that case, apr_socket_connect() could return APR_SUCCESS even if it is non-blocking. For multiplex processing, we want to know when the connection is established. If we know that, we can call apr_socket_connect() and expect it to return APR_SUCCESS. Unfortunatelly, there is a portability issue on checking readiness to call apr_socket_connect().
My suggestion is following logic. I assume that most of applications will send some bytes after apr_socket_connect() has established the connection.
/* pseudo code: check readiness for non-blocking apr_socket_connect() by apr_pollset_poll() */
/* XXX this is not the best way, but works on almost all platforms */
/* set non-blocking to the socket */
apr_socket_opt_set(sock, APR_SO_NONBLOCK, 1);
apr_socket_timeout_set(sock, 0);
rv = apr_socket_connect(sock, sa); /* this returns immediately */
if (rv == APR_SUCCESS) { /* special case: the server might run on the same host */
/* ready to write */
} else if (APR_STATUS_IS_EINPROGRESS(rv)) { /* usual case */
/* add the socket to pollset to check APR_POLLOUT(writable) */
pfd.reqevents = APR_POLLOUT;
apr_pollset_add(pollset, &pfd);
/* we should specify timeout to apr_pollset_poll(). */
apr_pollset_poll(pollset, positive_timeout, &num, &ret_pfd);
/* go to [1] or [2] based on pollset's results */
}
/* [1] when the socket is ready to write */
call apr_socket_send(). /* you don't need to call apr_socket_connect(), but you can call it (no side-effects) */
/* [2] when the socket is not ready to write by the timeout
* we guess the socket can't establish the connection */
apr_socket_close(sock);
Detecting whether socket is unavailable is important for real applications. In general, there are three scenarios when we lost TCP session. The remote process has closed socket, the remote process has crashed(terminated), or the remote host has crashed. Except the last case, we maybe receive FIN packet and we know we've lost out TCP session. From API perspective, we can know it by that apr_socket_recv() returns APR_EOF. The following example shows how to detect whether the remote process has closed the session.
/* pseudo code to detect we lost TCP session, i.e. the remote process has closed the socket or process terminated */
apr_status_t rv;
rv = apr_socket_recv(sock, ..., &len); /* we assume @sock is blocking */
if (APR_STATUS_IS_TIMEUP(rv)) {
/* we might be here, if @sock is blocking-with-timeout */
} else if (APR_STATUS_IS_EOF(rv) || len == 0) {
/* we lost TCP session.
* XXX On Windows, rv would equal to APR_SUCCESS and len==0 in this case. So, we should check @len in addition to APR_EOF check */
}
When we use non-blocking socket, we can know we lost TCP session by that socket is readable and apr_socket_recv() returns APR_EOF.
/* pseudo code to detect we lost TCP session. We use non-blocking socket */
apr_status_t rv;
rv = apr_pollset_poll(pollset, DEF_POLL_TIMEOUT, &num, &ret_pfd);
if (rv == APR_SUCCESS) {
for (int i = 0; i < num; i++) {
if (ret_pfd[i].rtnevents & APR_POLLIN) {
rv = apr_socket_recv(ret_pfd[i].desc.s, ..., &len);
if (APR_STATUS_IS_EAGAIN(rv)) {
/* we have no data to read. we should keep polling the socket */
} else if (APR_STATUS_IS_EOF(rv) || len == 0) {
/* we lost TCP session.
* XXX On Windows, rv would equal to APR_SUCCESS and len==0 in this case. So, we should check @len in addition to APR_EOF check */
} else {
/* we got data */
}
}
}
}
We call apr_sockaddr_info_get() with the first argument, a target hostname, and we can get apr_sockaddr_t objet. Then, we call apr_sockaddr_ip_get() with the apr_sockaddr_t object. The following prototype declaration is excerpted from apr_network_io.h.
/* excerpted from apr_network_io.h */
APR_DECLARE(apr_status_t) apr_sockaddr_ip_get(char **addr,
apr_sockaddr_t *sockaddr);
We call apr_sockaddr_info_get() with the first argument, a target IP address, and we can get apr_sockaddr_t object. Then, we call apr_getnameinfo() with the apr_sockaddr_t object. Note that apr_sockaddr_t::hostname will be implicitly modified by apr_getnameinfo().
/* excerpted from apr_network_io.h */
APR_DECLARE(apr_status_t) apr_getnameinfo(char **hostname,
apr_sockaddr_t *sa,
apr_int32_t flags);