OpenQueue - Technical Overview

Last updated: July 16, 2000

Abstract

The OpenQueue protocol is an open standard for publish-and-subscribe message queuing. It is designed to provide simple, reliable, asynchronous messaging between processes, whether on the same machine or across the Internet.

Introduction

OpenQueue is designed to provide real-time, reliable, transactional, message queuing services based on the publish-and-subscribe model, and is used for notification and MIME message delivery. While connected to a server, a subscribing client receives published messages in real time. When a client reconnects after an absence, the server sends all messages queued for that client while it was offline. This protocol can be used for messaging between applications, as well as messaging from applications to users. Messages are published to "topics", or subject areas, and for each subscriber to a topic the server maintains a queue of messages in that topic.

Message queuing is a powerful tool used today for distributed computing (e.g., connecting databases and web servers, or performing financial transactions). It is especially useful where systems must be reliable even in the face of temporary machine failure. Message queuing can also be used for end-user applications involving real-time or queued communication, such as receiving news articles on a certain topic, but cost and platform limitations have not encouraged this yet.

While several vendors offer message queuing systems today, none offer truly open systems. Most will run only on limited platforms, although systems exist which run on cross-platform languages such as Java. However, none of the vendors have published their underlying protocols, and they use non-standard data formats. In addition, these systems tend to be top-heavy, and are inappropriate for many potential client platforms, including cellular phones and handheld computers. The OpenQueue protocol is designed to fill the niche of a completely open message queuing protocol.

Architecture

OpenQueue is focused on notification and MIME-based message delivery, and only those services central to this focus are specified in this protocol. For example, at this stage OpenQueue does not include commands to change user access rights to specific topics; the server would handle this in an implementation-specific way.

The architecture is based on clients and servers. A client must authenticate itself to each server to which it connects. (While plaintext passwords are currently allowed, to speed the development of code, they might be disallowed in the future.) Where this document describes a client as 'subscribed', it refers to its user's preference for receiving messages on a topic; it does not refer to the client's current state (logged in or offline). That is, a user remains subscribed to a topic even after her client program logs off; this allows the server to queue messages until the client's return. The client only receives messages for topics to which it has explicitly subscribed. This prevents "spamming" because there is no way to send messages to clients that have not requested them.

A client may be a subscriber to one or more topics on one or more servers, meaning it has permission to receive messages from those topics (using the GET or UPDATES commands). A client may also be considered a publisher for one or more topics, meaning that it has permission to publish (using the SEND command) messages to those topics. Any type of MIME-based data (HTML, XML, GIF, etc.) can be sent, and it is up to the publisher(s) of a topic to decide the form and meaning of the messages it uses.

For example, a topic named /weather/europe/satellite might contain JPEG images from a weather satellite over Europe. A separate topic, named /weather/europe/forecast, might contain weather forecasts for European cities, using some flavor of XML defined for describing weather forecasts. Both of these topics might live on the same OpenQueue server, since the server does not care about the contents of the messages (other than the topics to which they are addressed).

Read and/or write access for each topic can be assigned to each user. This is implementation-specific. The protocol specifies that a server can reject a client's request to get or send a message from or to a topic; if the server determines that the client lacks permission, it returns a response code to the client to indicate that.

Modes

If messages are published to a topic while a subscribed client is offline, then the server will deliver the messages when the client next logs in. This set of behaviors is referred to as the ALIVE mode. Other modes may also be defined, providing additional ways of connecting or levels of service. When a client connects to a server, the server advertises the modes available, and the client can choose the mode it wishes to use.

[One mode under investigation is called AWARE. Using this mode, the client tells the server that it is listening for messages on a certain IP address. The server disconnects, and then reconnects to the client (with an authenticating code) when a new message has arrived for that client. After delivering the messages, the server disconnects again. This mode allows clients to receive near real-time messages (with a few seconds of delay for establishing a connection) while also limiting the number of open connections the server needs to maintain. This allows the server to serve a much larger number of clients, especially clients who do not receive messages very often. Administrators could set up servers to offer the ALIVE (high-speed) mode for perhaps 50 preferred users, while only offering the AWARE (slightly delayed) mode for all other users. There are additional security issues to be addressed with the AWARE mode, however.]

States

A client is always in one of several states. When it first connects to a server, it is Unauthorized. When it has successfully logged in, it is Ready. When the server has initiated a transaction, the client is Receiving. When the client is publishing a message, it is Sending. A client that is receiving should not attempt to perform any operations until it acknowledges (or cancels) the current transaction with the ACK command. This is the only current restriction on activity in states; however, it is useful for implementers of OpenQueue to be aware of the state model.

Commands

Clients send tagged commands to the server in the following format:
!A1000 COMMAND option1 option2...

The first part, "!A1000" is the tag, which is unique for each time a client sends a command in a given session. The client may use any code it likes for the tag, except that it must begin with the exclamation point "!", and it must not contain any spaces. Tags are used to help clients see which requests are being answered when a server sends data. After the command line ends, additional information may be supplied in lines following the command, much as header information in an HTTP request can follow the initial HTTP request line. The command is considered complete when a blank line is sent.

Servers can send certain types of untagged data to the client asynchronously, without waiting for a request from a client. For example, the BYE command can be sent by the server at any idle time [?], which announces that the session is going to be terminated. In addition, if the user has sent the UPDATES ON command, the server will send transactions (blocks of messages) whenever it sees messages waiting to go to that client.

Transactions

The server delivers the topic messages to the client in the order in which they were published to the topic. The messages are sent as transactions, which are collections of messages that can be sent all at once. Each client can specify a maximum transaction size, which is the number of messages, or total memory footprint of data, which can be sent in one transaction. By default the server sends up to five messages per transaction. If a user has less than the maximum transaction size waiting, the server does not wait until the size is reached; the existing messages are sent right away. If delivery of all messages in the transaction succeeds (is "committed"), the server notes this for the client and attempts to send additional transactions, assuming messages remain to be sent. If delivery fails for any reason, the transaction is "rolled back", and the server will retry the transaction until it succeeds. This ensures that each message is guaranteed to be delivered once, only once, and in the order in which it was published to the topic.

There is a two-phase commit involved in each transaction. The steps are:

  1. The server sends the messages in the transaction (client receives UPDATE command).
  2. The client stores the messages in temporary memory and acknowledges receipt (client sends ACK command to server).
  3. The server tells client it accepts the ACK command.
  4. Both client and server note the transaction as committed, and the client process the messages (e.g., displays them to the user).

The server can use this two-phase commit process to implement transactional services, where OpenQueue transactions and other events either succeed together or fail together. For example, if a user is charged for each message received, an OpenQueue server could send the OpenQueue messages at the same time it prepares to charge the user's account in a database system. If either the database changes or the receipt of the OpenQueue messages fails, the server can back out of both. To back out of the OpenQueue transaction, the server would, in step 3 above, tell the client that it does not accept its ACK command. Then the client would throw away the messages received, and the transaction would be considered rolled-back.

Services Not Provided

OpenQueue explicitly avoids trying to dictate protocols or standards for anything beyond subscribing to topics, posting messages, and receiving messages. For example, the way(s) messages are structured or displayed is not OpenQueue's concern. Similarly, an OpenQueue server may consult any "Access Control List" system to see if a logged-in user has rights to subscribe to a topic, but we do not dictate the structure or the administration methods for the ACLs.

As another example, OpenQueue does not currently define ways to explore the existing topic space on a server, or to create new queues. An implementation might integrate an OpenQueue server with an LDAP directory service, where each topic (queue) in the server is identified as an item in the directory. The OpenQueue server itself would only know whether a given path (e.g., "/weather/europe/forecast") corresponds to an existing topic. A user could use LDAP browsing tools to find out what other topics exist in the same topic tree (such as "/weather/europe/satellite").

Topic and Message Organization

Each server maintains a set of topics, and each topic is described by a path. Topic paths on a server begin with a "/" (forward slash) character, which represents the root of the topic tree.  Complete references to a topic on a server can be made with a URL such as "oq://company.com/weather/europe/satellite". Currently OpenQueue operations must be on a specific topic path, but it is expected that in the future OpenQueue will support advanced operations based on wildcards, such as allowing a user to subscribe to "oq://company.com/weather/*". Each message in a topic is given a message ID number, in increasing order, starting with 1 for the first message in the topic. A reference can be made to the 95th message in the "/securitycameras/garage" topic as follows: "oq://company.com/securitycameras/garage#95".

Servers also support topic numbers. This is an optimization to allow quick references to topics with a minimum of bandwidth. The meaning of the number is opaque to clients, but a server implementer can assign a topic number that is efficient for it, such as a record number in a table of topics. For example, "oq://company.com/#1503" refers to a topic that the server identifies with topic number 1503. The topic should also have a path (e.g., "/users/fred/schedule"), and clients can then refer to a topic with either the path or the topic number:
    "oq://company.com/#1503"
    "oq://company.com/users/fred/schedule"

Individual messages can also be referenced either way, as the "#" at the root of the server address acts as a special identifier for a topic number:
    "oq://company.com/#1503#25"
    "oq://company.com/users/fred/schedule#25"

[Note that topic numbers are experimental, and might be dropped from future versions of the protocol.]

Example Session

In the example below, a client "C:" connects to a server "S:". When the client logs in, the server sends a server greeting identifying the version of the OpenQueue protocol it supports, the name and version of the server program, and the modes available. The server greeting also contains a nonce value, which is a random string that the client can use as a seed value for some methods of authentication (see the AUTH command). Ideally, the server will never send the same nonce value twice.

S: OPENQUEUE=0.50 SERVER=oqtane/0.1.8 MODES=ALIVE NONCE=24b6a83b
S: 
C: !a AUTH
C: AuthType: simple-md5
C: User: joe
C: UserNonce: 8b44fe9238f4b2bc
C: Hash: f1dec2ad149633205b1b9876b90de543
C: 
S: !a 200 - OK
S: Response: 0a92ab06beae939bcf1d83efb2263f62
S: 
C: !A1001 UPDATES ON
S: !A1001 200 - OK.
S: 
(... as soon as server is ready, it sends the first messages ...)
S: UPDATE TXN=1
S: START ITEM #1401 55
S: Subject: Server 'penguin' is out of memory.
S: Content-Length: 81
S: Content-Type: text/html
S: 
S: The 'penguin' server reports it is out of memory. 
S: There are 94 processes running.
S: END ITEM
S: START ITEM #1401 56
S: Subject: Multiple login failures on host 'petunia'.
S: Content-Length: 65
S: Content-Type: text/html
S: 
S: Host 'petunia' has seen 6 failed logins from the same IP address.
S: END ITEM
S: START ITEM #2688 131
S: Subject: Your tiffany lamp bid exceeded - current bid $65.
S: Content-Length: 163
S: Content-Type: text/html
S: 
S: Your last bid on the tiffany lamp, item <b>25-7786</b> 
S: has been exceeded by $10. Current bid is $65. To make a new bid 
S: of $70, click <A href="newbid.htm">here</A>. 
S: END ITEM 
S: ENDUPDATE 
S: 
C: !A1002 ACK 1 
S: !A1002 200 - OK 
S: (...time passes, 
and when a new message is ready the server sends a new transaction....) 
S: UPDATE TXN=2 ... 

Client Commands

AUTH
Example:
!a AUTH
AuthType: plaintext
User: joe
Password: Hop5grass


This is the simplest usage of the AUTH command, which authenticates a user to the server. AUTH should be the first command sent by the user after establishing a connection. However, the "plaintext" system above leaves passwords vulnerable to anyone sniffing the network packets, so it is highly suggested that a more advanced type of authentication be used instead, such as the "simple-md5" type shown below:

Example: (with a NONCE of "24b6a83b" in the server greeting)
C: !a AUTH
C: AuthType: simple-md5
C: User: joe
C: UserNonce: 8b44fe9238f4b2bc
C: Hash: f1dec2ad149633205b1b9876b90de543
C:
S: !a 200 - OK
S: Response: 0a92ab06beae939bcf1d83efb2263f62
S:


The simple-md5 system allows client and server to authenticate each other without sending a password directly over the wire. It assumes the server already knows a user's password, and it assumes both client and server can calculate the MD5 one-way hash of a given string.

The client creates a unique value called UserNonce, perhaps from a random number generator and a time stamp. Ideally, a user will never send the same UserNonce value twice. Then the client calculates the Hash value as Hash = MD5(NONCE:User:UserNonce:Password). In the above example, Hash = MD5("24b6a83b:joe:8b44fe9238f4b2bc:Hop5grass"). The parameters can appear in any order after the AUTH line.
When the server receives the parameters from the client, it performs the same computation that the client did, and checks to see if the Hash value sent equals MD5(NONCE:User:UserNonce:Password). If so, the user is authenticated, and the server responds with "200 - OK". Then the server sends a response so that the client can authenticate the server, if desired. The response is calculated as MD5(NONCE:Usernonce:Password). In the example above, Response = MD5(24b6a83b:8b44fe9238f4b2bc:Hop5grass).

The Mode string (e.g., "Mode: ALIVE") allows the client to specify one or more modes that it can understand and is interested in using, in order of preference. For example, the mode string "Mode: ALIVE AWARE" means "connect me with alive mode if possible; if not possible, connect me with aware mode; if neither is possible, don't connect me." If the client does not send the Mode parameter, the server's default mode will be assumed; for now servers should use ALIVE as the default. If no requested mode is available, the server should respond to the AUTH command with response code 307 (No mode is available which matches your request).


LOGIN
Example: LOGIN myusername mypassword mode=mode1,mode2

LOGIN identifies the client's user to the server. The username, password, and mode string elements are all required. The LOGIN command is the only command note prefaced with a command tag (i.e., use "LOGIN ..." rather than "!a001 LOGIN ..."). NOTE: The LOGIN command sends the password in the clear, and should only be used for development purposes, or in situations where it is guaranteed that no third party can listen into the conversation with the server. Instead, the AUTH command should be used for production work. The LOGIN command is deprecated and will likely be discontinued in future versions of the protocol.

SEND
Example: SEND /weather/europe/forecast
Example: SEND #1234

SEND begins the process of sending a message to a topic. If the server allows the operation at that point (considering who has access to the topic, whether resource limits have been reached, etc.), it returns response code 200 (OK). Otherwise, it returns an appropriate response code (e.g., 306 - No permission for this topic, or 305 - Topic not recognized). If 200 - OK is returned, the client may then start sending the content of the message, starting with any headers. A message can consist of only headers, or headers and a body (the body is separated by a blank line). The server knows the message is completely sent when either a) the client sends a "." on a blank line, or b) one of the headers is of the form "Content-Length: 1234", which indicates how many bytes to read after the first blank line after the headers. When the message is fully sent, the server sends another response code (200 - OK or an error) indicating whether the message was accepted or not. If the message is not accepted, the client may decide to attempt to send the message at a later time.

UPDATES
Example: UPDATES ON
Example: UPDATES OFF

UPDATES indicates whether the server should send messages to the client. By default updating is off when the client connects to the server. Currently two commands are recognized, UPDATES ON and UPDATES OFF. In the future, updating on a topic-by-topic level will be supported. If a client sends the UPDATES OFF command, it is possible that one last group of messages may be sent in the time it takes the client to send the UPDATES OFF command. It is the client's responsibility to deal with these messages (through the ACK command).

ACK
Example: ACK 1

ACK acknowledges receipt of the messages in a transaction. When the server responds with a 200 - OK response code, the two-phase commit is complete and the server and client can consider the messages sent.   The number after "ACK" is the number of the transaction, originally sent by the server in the "UPDATE TXN= 1" message.  The next transaction number from the server would likely be 2, but the client should just be concerned with matching the number that was sent, whatever it was.

[To be documented] GET, SUBSCRIBE, UNSUBSCRIBE

Server Commands

[To be documented] BYE, UPDATE

Status Response Codes

These are status reports from the server and indicate the response to the last command received from the client. Status response lines begin with a 3 digit numeric code that is sufficient to distinguish all responses. Some of these may herald the subsequent transmission of text. The first digit of the response broadly indicates the success, failure, or progress of the previous command.
200 - OK. (Command completed without problems)
202 - Ready for message to be sent; end with '.' on its own line.
203 - Received.
300 - Client Error.
301 - User name or password not recognized.
302 - Unknown command.
304 - Error logging out.
305 - Topic ID not recognized.
310 - Item ID not recognized.
306 - No permission to send to this topic.
307 - No mode is available which matches your request.
308 - Syntax error.
309 - No permission to read from this topic.
311 - Already logged in on a different connection.
320 - Cannot subscribe to the topic.
321 - Cannot unsubscribe from the topic.
322 - Cannot set cursor position.
505 - Server too busy. Not accepting more connections now.