Tech Spotlight: Advanced Message Queuing Protocol (AMQP)

Introduction

AMQP (Advanced Message Queuing Protocol) is a binary-wire, application layer (OSI Model) protocol used primarily by business organizations to carry out inter-application communication and high-speed data transfer.

The following use-cases can take advantage of AMQP:

• Real-time feed of constantly updating information.
• Encrypted assured transaction requirement.
• To connect different systems to talks to each other.
• Increasing reliability and uptime of application deployments.
• When messages need to be delivered as soon as the destination comes online.
• Send an enormous message while still receiving status updates over the same network connection.
• Work on all popular operating systems and languages.

Main Features

• Ubiquity: It is Open Internet Protocol standard and easy to understand, use and implement. It easily fits into existing messaging applications platforms.
• Safety: Secure and trusted global transaction network.
• Fidelity: Well stated message queuing and delivery semantics with at-most-once, at-least-once, and once-and-only-once.  Well stated failure semantics to manage all the exceptions
• Unity:  AMQP client initiates and communicates over TCP/IP or any other transport protocol. Provides the core set of messaging patterns via a single manageable protocol, asynchronous directed messaging, request/reply, publish/subscribe, store and forward. It supports the Hub & Spoke messaging topology  and Peer to Peer messaging across any network
• Interoperability: All AMQP clients interoperate with all AMQP servers.  Diverse programming languages can communicate easily.  Legacy message brokers can be retrofitted to remove proprietary protocols from the network.  Messaging can be enabled as a cloud service.
• Manageability: It is ubiquitous, fast, embedded (XML can be layered on top), enabling management to be provided by encapsulating systems (e.g., O/S, middleware, and phone) and scalable.

Architecture

AMQP implements layered architecture so that features & network transports can be independently extended by communities. It also enables business integration with other systems. Different layers are:

• Transport Layer
• Messaging Layer
• Transactions Layer
• Security Layers

Transport Layer: AMQP network consists of nodes which store and deliver messages safely. A message originates and ends in Nodes only. A Node can be Producer, Consumer or Queue depending upon whether message originated, terminated or relayed by it. Client Application or Broker acts as Containers having one or multiple Nodes. Client application contains producers or consumers, whereas Broker contains queue nodes which store and forward messages.

Figure 1:

Before the communication starts, a Connection is first established between Containers on top of any transport stream and required security implementation like TLS and SASL (Defined in Security Layer). Containers Communicate using Connections, and consists of Full-Duplex reliably ordered sequence of Frames. Connections manage the transfer capacity by defining maximum Frame size. Container acting as a client initiates the negotiation for maximum frame size. After the establishment of a connection and exchange of protocol header between the peers, each peer relays an open frame in the channel with info like capabilities and limitations of the peer and maximum frame size to open a Connection. After the exchange of open frame, peers communicate in mutually acceptable limitations. An AMQP connection consists of a negotiated number of multiple unidirectional Channels in either direction. Frames are marked with channel number indicating its parent Channel, and the Frame sequence for each Channel is multiplexed into a single Frame sequence for the Connection. The connection is closed with the transfer of a close frame. The close frame indicates the reason for closing the connection and is the last frame in any Connection. The close Frame is usually exchanged in Channel 0 but not restricted.

Sessions correlate two unidirectional channels to form a bidirectional sequential conversation between containers. A single Connection may have multiple independent Sessions simultaneously. Multiple sessions can be used for different activities like data transfer, control messages, etc.

An AMQP network is consist of Nodes connected by unidirectional routes namely Link. Any peer communicating can create links in any direction. Links are named by applications and carry payloads only in one direction, but the communication is bidirectional as there also communication of flow control like error message also happens from receiver to sender. To have a full-duplex bidirectional data transfer, nodes can create two links in the opposite direction. Links are connected to nodes at Terminuses which are either Source or Target. Source and target terminus tracks outgoing and incoming messages respectively. The primary responsibility of a Terminus is to maintain a record of the status of each active delivery (unsettled deliveries) attempt until it is safe to forget.

A single Session may be simultaneously associated with any number of Links, but a Link is attached to at most one Session at a time. Message transferred on a Link are sequentially identified within a Session. Sessions are established by creating a Session Endpoint and assigning any unused channel number. A begin frame announces the association of the Session Endpoint with an outgoing channel. A Session ends automatically when the Connection is closed or interrupted.

Both Connections and Sessions are modeled by each peer as endpoints that store local and last known remote state regarding the Connection or Session in question. Sessions provide the context for communication between Sources and Targets. A Link Endpoint associates a Terminus with a Session Endpoint. Within a Session, the Link Protocol is used to establish Links between Sources and Targets and to transfer Messages across them.

A Connection Endpoint contains either incoming or outgoing Channel Endpoint. Connection Endpoints map Frames (Except open and close) to an incoming Channel endpoint based on the incoming Channel number and relays Frames produced by Outgoing Channel endpoints by marking them with the associated outgoing channel number.  As Channels are unidirectional, at each Connection endpoint the incoming and outgoing Channels are entirely distinct.

Frame Structure

A Frame is the unit of work carried on the wire. Connections have a negotiated maximum frame size allowing byte streams to be easily defragmented into complete frame bodies representing the independently parsable units. Frames are consisting of Frame header (8Bytes), Extended header (Variable Size reserved for future extensions), and Frame Body. The Frame header contains information like Frame Size, Extended Header size, and Frame Body sizes. The 4th byte of the header shows the data offset (DOFF), it helps in determining the position of the Frame body. The 5th byte of the Frame header determines the format and purpose of the frame.  Byte 6 and 7 of the Frame contains the channel number.

Before sending any Frames on a Connection, each peer sends a protocol header indicating the protocol version used in the Connection. Peers trying to communicate must agree upon the usage of same protocol version and usually is the smallest of the highest protocol versions supported by the peers.

Messaging Layer: The Messaging Layer specifies a standardized use of transport layer extensions for different messaging applications to provide interoperability. It also defines the source and target to be used with a link.

The receiver often receives an immutable payload from the sender along with various annotations provided by the messaging infrastructure nodes. Normally the immutable payload is defined as a bare message. A bare message with added annotations at the head and tail is known as annotated message. Annotations either travel with the message indefinitely or gets consumed by the immediate node.

A bare message consists of properties, application-properties, and application-data. Annotated Message includes a header, delivery-annotations, message-annotations, properties, application-properties, application-data, and footer. Except for the bare message content, all other fields may or may not be present. The header defines standard delivery details about the transfer of the message in the network. The Delivery annotations hold delivery specific non-standard properties. The message annotations are used for properties of the message aimed at the infrastructure and propagate across every delivery steps. Properties filed of the bare message holds the immutable properties of the message whereas application-properties hold the structured application data. Data contains opaque binary data and footer section is used for message hashes or HMACs or any other encryption details.

The messaging layer defines a set of delivery states to indicate the state of the message at the receiver. Delivery states are either terminal or non-terminal, and terminal delivery-state is not changeable, referred to as an outcome. The outcomes are of types accepted (Successfully processed by the receiver), rejected (Invalid or nonprocessable message), released (Message not processed), or modified (message modified but not processed). The non-terminal delivery-state is received (Partial message data seen by the receiver.)

.

Transactions Layer: Transactional Messaging allows for the coordinated outcome of independent transfers spread across distinct links in either direction. For each transactional interaction, one container acts as the transactional resource and the other container work as a transactional controller, and they communicate over a control link established by the transactional controller. The transactional resource performs transactional work as requested by the transactional controller. Declare and discharge messages are used by the transactional controller over the control link to allocate and complete transactions respectively.

The transactional resource defines a special target that functions as a coordinator. The transactional controller establishes a control link to the coordinator. To begin transactional work, the transaction controller obtains a transaction identifier from the resource by sending a message to the coordinator and sent as unsettled. The coordinator responds with a disposition outcome of declared which carries the identifier or transaction-error in case if it is unable to perform declare specified by the controller. Transactional work is described regarding message states and composed of posting message, acquiring message and retiring message.

The controller concludes the transactional work by sending a discharge message to the coordinator and as an unsettled message.

Security Layers: Security Layers are used to have authenticated and encrypted communication over which AMQP traffic is tunneled. Security Layers can also be tunneled over one another. There is support for TLS (Transport Layer Security) and SASL (Simple Authentication and Security Layer) by AMQP. TLS is defined externally and is not part of AMQP specification, but SASL requires framing to be done by AMQP itself. To establish a TLS communication each peer first sends the protocol header, but the protocol id of normal AMQP communication (0) is not used. Instead, 2 is used to intimate that TLS is used for encryption. For SASL similarly protocol id 3 is used.

Implementations

Some commercially available AMQP products include:

• Apache Qpid, an Apache project
• IIT Software's SwiftMQ is an enterprise-grade JMS messaging product with full support for AMQP 1.0
• Kaazing's AMQP Web Client
• Microsoft's Windows Azure Service Bus
• AMQ by Red Hat built from Qpid
• StormMQ a cloud-hosted messaging service based on AMQP
• RabbitMQ by VMware Inc.
• MQlight by IBM