Application Protocols

The host layers protocols include session (SES), presentation (PRES) and application (APP) levels. In particular, the APP (application) layer in regular Internet communication is dominated by the HTTP protocol and XML-related derivatives, e.g. SoAP. Also, the FTP protocol for file transfer is ubiquitous; it has existed since the beginning of the Internet. Most of them are somehow related to the textual presentation of the information. They're referenced as “WEB” protocols.
Although advanced and more powerful IoT devices frequently use these protocols, this is problematic to implement in the constrained IoT devices world: even the simplest HTTP header occupies at least 24 + 8 + 8 + 31 bytes without payload!
There is also a problem of crossing firewall boundaries when communication between subnetworks of IoT devices is expected to occur.

Some IoT-designed protocols are reviewed below.

MQTT protocol ^[1] was invented especially for constrained IoT devices and low bandwidth networks. It is located in APP layer 7 of the ISO/OSI stack but covers all layers 5–7. It is a text-based protocol yet very compact and efficient. Protocol stack implementation requires about 10 kB of RAM/flash only.

MQTT uses a TCP connection, so it requires an open connection channel (this is opposite to UDP connections, where communications work in a way: “send and forget”). It is considered a drawback of the original MQTT protocol, but MQTT variations exist for non-TCP networks, e.g. MQTT-SN. Protocol definition provides reliability and delivery ensure mechanisms.

The standard MQTT Message header is composed of just two bytes only (table 1)! There are 16 MQTT message types. Some message types require variable-length headers.

MQTT requires a centralised MQTT Broker located outside firewalls and NATs, where all clients connect, send and receive messages via the publish/subscribe model. The client can act as publisher and subscriber simultaneously. Figure 1 presents a publish-subscribe model idea.

MQTT Message
MQTT is a text-based protocol and is data-agnostic. A message comprises a Topic (text) and a Payload (data). The Topic is a directory-like string with a slash (“/”) delimiter. Thus, all Topics constitute (or may represent) a tree-like folder structure similar to the file system's. The subscriber can subscribe to a specific, single Topic or a variety of Topics using wildcards, where:

• # stands for the entire branch,
• + stands for the single level.

Example Scenario

Publishers deliver some air quality information data in separate MQTT messages and for various rooms at the department Inf of the Universities (SUT, RTU, ITMO) to the Broker:

Subscriber 1 is willing to get all sensor data for SUT University, Inf (informatics) department only, for any space:

Subscriber 2 is willing to get only Temperature data virtually from any sensor and in any location in TalTech University:

Subscriber 3 is willing to get any information from the sensors, but only for the RTU

The message's payload (data) is also text, so if binary data is to be sent, it must be encoded (e.g., using Base64 encoding).

MQTT Broker
MQTT Broker is a server for both publishers and subscribers. The connection is initiated from the client to the Broker, so assuming the Broker is located outside a firewall, it breaks its boundaries.
The Broker provides QoS (Quality of Service) and can retain message payload. MQTT Broker QoS (supplied at the message level) has three levels.

• Unacknowledged service: Ensures that MQTT message is delivered at most once to each subscriber.
• Acknowledged service: Ensures message delivery to every subscriber at least once. The Broker expects acknowledgement to be sent from the subscriber. Otherwise, it retransmits data.
• Assured service: This two-step message delivery ensures the transmission is delivered exactly once to every subscriber.

Providing unique packet IDs in the MQTT frame is vital for Acknowledged and Assured services.

The DUP flag (byte 1, bit 3) represents information sent by the publisher, indicating whether the message is a “first try” (0) or a retransmitted one (1). This flag is primarily for internal purposes and is never propagated to the subscribers.

MQTT offers a limited set of features (options):

• clean session flag for durable connections:
  • if set TRUE, the Broker removes all of the client subscriptions on client disconnect,
  • otherwise, the Broker collects messages (QoS depending) and delivers them on client reconnecting; thus, connections remain idle.
• MQTT “will” - on connection loss, the Broker will automatically “simulate” publishing of the predefined MQTT message (Topic and payload). All clients subscribing to this message (whether directly or via a wildcard) will be notified immediately. It is an excellent feature for failure/disaster discovery.
• message retaining: it is a feature for regular messages. Any message can be set as retaining; in such case, the Broker will keep the last one. Once a new client subscribes to a topic, they will receive a retained message immediately, even if the publisher is not publishing any message. This feature is last known good value. It is good to present the publisher's state, e.g. the publisher sends a retained message meaning “I'm going offline” and then disconnects. Any client connecting will be notified immediately about the device (client) state.
  

Interestingly, MQTT is a protocol used by Facebook Messenger ^[2]. It is also implemented natively in Microsoft Azure and Amazon Web Services (among many others).

MQTT security is relatively weak. The MQTT Broker can offer user and password verification sent in plain text. However, all communication between the client and Broker may be encapsulated in an SSL-encrypted stream.

A short comparison of MQTT and HTTP protocols is presented in table 2.

CoAP protocol (RFC7252) originates from the REST (Representational State Transfer). CoAP does not use a centralised server as MQTT does, but every single device “hosts” a server on its own to provide available resources to the clients asking for service offering distributed resources. CoAP uses UDP (compared to MQTT, which uses TCP) and is stateless; thus, it does not require memory to track the state. The CoAP implementation assumes every IoT device has a unique ID, and things can have multiple representations. It is intended to link “things” together using existing standard methods. It is resource-oriented (not document-oriented like HTTP/HTML) and designed for slow IoT networks with high packet loss. It also supports devices that are periodically offline.
CoAP uses URIs to address services:

• coap://<host>[:<port>]/<path>[?<query>] to access a service/resource,
• a secure, encrypted version uses “coaps” instead of “coap”.

It supports various content types, can work with proxy and can be cached.
The protocol is designed to be compact and straightforward to implement. The stack implementation takes only about 10 kB of RAM and 100 kB of storage. The header is only 4 bytes.

CoAP protocol has a binary header to decrease overhead, but the payload depends on the content type. The initial, non-exclusive list of the payload types includes:

• text/plain (charset=utf-8) (ID=0, RFC2046, RFC3676, RFC5147),
• application/link-format (ID=40, RFC6690),
• application/xml (ID=41 RFC3023),
• application/octet-stream (ID=42, RFC2045, RFC2046),
• application/json (ID=50, RFC7159).

CoAP endpoint services are identified by unique IP and port number. However, they operate on the UDP instead of TCP (like, e.g. HTML does). The transfer in CoAP is made using a non-reliable UDP network, so a message can appear duplicated, disappear, or be delivered in another order than initially sent. Because of the nature of datagram communication, messages are exchanged asynchronously between two endpoints, transporting Requests and Responses. CoAP messages can be (non-exhaustive list):

• CON - Confirmable, those requiring ACK Acknowledge,
• NON - Non-Confirmable, those that do not need ACK,
• ACK - an acknowledgement message,
• RESET - sent if CON or NON was received, but the receiver cannot understand the context, e.g. part of the communication is missing because of device restart, messages memory loss, etc.

Empty RESET messages can be used to “ping” the device.

Because of the UDP network characteristics, CoAP provides an efficient yet straightforward reliability mechanism to ensure successful delivery of messages:

• stop and wait for retransmission with exponential back-off for CON messages,
• duplicate message detection for CON and NON-messages.

The request-response pair is identified by a unique “Token”. Sample request-response scenarios are presented in the images below. Sample CoAP message exchange scenarios between client and server are presented (two per image) in figure 2 and figure 3.

The scenario in figure 2 (left, with token 0 × 70) is executed in a situation when a CoAP server device (a node) needs some time to prepare data and cannot deliver information right away. The scenario in figure 2 (right, with token 0 × 71) is used when a CoAP server can provide information to the client immediately. The scenario in figure 3 (left, with token 0 × 72) appears when a CoAP server cannot understand the request. The scenario in figure 3 (right, with token 0 × 73) presents the situation where the request to the CoAP server was made with a non-confirmable request.

In its principles, the AMQP (Advanced Message Queuing Protocol) somehow recalls MQTT: it is message-oriented and uses a central broker. There are data publishers and consumers (that, in the case of the MQTT, are called subscribers). Messages are routed from publishers to the Broker, where they hit so-called exchanges, and then they are copied to the queues (0, 1 or more) from which the consumer can later read. A diagram of the message's flow is present in the figure 4.

AMQP uses TCP/IP. AMQP is intended to work in non-reliable networks; thus, the protocol has a message acknowledgement mechanism to ensure delivery. A message is removed from the queue only if it has been acknowledged. Besides acknowledged delivery, it is also possible to use an unacknowledged one that does not involve acknowledgements. If a message cannot be routed (for any reason), it can be returned to the publisher, dropped or placed in the “dead letter queue”. The behaviour is defined along with a message. Opposite to MQTT, in AMQP protocol, the connection status is unknown; thus, there is no mechanism to let other devices know that some node has disconnected, such as the last will in MQTT.

Queues
AMQP is a programmable protocol, so bindings are not defined by the Broker but rather by the publisher. Queues are also created on-demand via external actors (mostly consumers). Routing via bindings is provided with a message, and the Broker analyses it to provide correct message handling and delivery.
Consumers can subscribe to the exchange and define a queue. Bindings then act as filters so they receive only selected messages. A single queue is intended to handle one consumer, but there is a possibility of letting many consumers use a single queue in the round-robin model. As in the protocol version 0.9, queues have the following properties:

• Name,
• Durable flag - the queue and its contents are persistent across broker restarts,
• Exclusive flag - used exclusively by one consumer only,
• Auto-Delete flag - the queue is removed if the last consumer unsubscribed,
• Arguments - optional.

A queue name can be selected explicitly, or a broker may deliver one on demand. A queue has to be expressly defined. An existing queue can be silently redeclared with an exact attributes set. A declaration of the existing queue with different attributes set throws an exception code 406 (PRECONDITION_FAILED).

Exchanges
Specification 0.9 of the AMQP protocol creates 4 exchanges (exchange types):

• Direct Exchange (its name is empty string or amq.direct); The default, Direct Exchange, has a special feature that automatically creates and binds new queues, where the queue's name is the same as a routing key; thus, it is ideal for unicast communication. Assuming the queue's name is K (there can be more than one) and there comes a message with routing key N, the message is routed only to those queues, where K=N (figure 5).

• Fanout Exchange (amq.fanout); In the Fanout Exchange, all messages are routed to all queues bound to the Fanout Exchange, regardless of their routing key. They help broadcast the information (figure 6).

• Topic Exchange (amq.topic); Topic Exchange works similarly to MQTT topic subscriptions: an AMQP queue bound to the Topic Exchange defines a pattern rather than the fixed name, and messages with the matching routing key are copied to the queue (7). It is great for multicasting.

• Headers Exchange (amq.match); This exchange uses message headers for routing instead of routing keys. The routing key is ignored, and it is possible to bind a queue to the Headers Exchange using more than one header. It is also possible to specify whether it is enough to find a single matching among many conditions or all must be satisfied. Sample routing is present in the figure 8: Q1 requires one of the conditions to be satisfied, while Q2 requires all of the conditions to be satisfied to execute a binding and route a message.

Bindings
Bindings are rules defining how to route from exchange to queues. Depending on the exchange type the publisher interacts with, the messages are routed using algorithms that consider both exchange type, arguments and bindings. The publisher is responsible for providing information on which queue will receive the messages from the exchange.

Consumers
Consumers subscribe to the queue to “consume” messages. There are two ways to let consumers receive them: push API and pull API. For performance reasons, pull API should be avoided. Once subscribed to the queue, the consumer receives a unique “consumer tag” that is a string (text) and is later necessary to unsubscribe.

Messages
The essential, immutable AMQP frame size is 8 bytes, and the payload is up to 2GB. Besides the header frame, messages have several virtually freely definable attributes, but the Broker uses some predefined ones. Common attributes cover:

• content type,
• content encoding,
• routing key,
• persistence flag,
• message priority,
• message publishing timestamp,
• content expiration period,
• publisher app id.

Header attributes (used, e.g. by the Headers Exchanges) are optional and similar to X-Headers in HTTP ^[3].
A payload in the AMQP message is a byte array. Broker does not process or review the content, and it can be even zero length.

The address of the Broker is referenced with URI, similar to the CoAP (e.g.):

• amqp://<user>:<password>@<host>[:<port>]/<path> for raw connections,
• “amqps” for TLS/SSL secure connections.