Pub/Sub Messaging in Cache: An Overview
This Publish/Subscribe (Pub/Sub) messaging paradigm provides an intermediary channel (called a topic) to exchange messages between multiple applications without any knowledge of the sender (publisher) and the receiver (subscriber). A publisher application sends messages to a subscriber application through topics.
Since all Pub/Sub models require a channel for communication, NCache acts as a medium for topics so that the publisher publishes the message to the topic. The subscribers receive the message via the topic as a notification. Using NCache as a medium for topics ensures loose coupling within the model and allows increased abstraction, providing additional benefits to the distributed topics.
Besides a distributed cache, NCache also provides a dedicated Pub/Sub Messaging cache.
We recommend a dedicated Pub/Sub cache for the following reasons:
Eviction: If the cache contains messages and cache items, frequent eviction for cache items may also remove the messages before forwarding them to the subscriber.
State Transfer: State transfer adds a cost to each cache item operation. Since messages are frequently published, relayed, and expired, this recurrent activity can prove costly because it triggers state transfer.
Why Pub/Sub Messaging is Important
Real-time events require notification sharing among different applications in event-driven distributed architectures. Pub/Sub patterns enable publishers to share events with subscribers so that any desired processing takes place on the occurrence of an event of interest.
For example, a group of subscribers can be interested in information about order shipment details so that they can process it to track order delivery. Hence, they will subscribe to a topic that relays messages for order details. Once the publisher publishes a message to the topic, the subscribers will be notified and receive the message containing order details for further processing on their end.
The basic components of the NCache Publisher Subscriber Model are listed below and discussed in detail later:
Topic: A place where messages are published.
Publisher: An application (web application, desktop applications, microservices) that publishes messages to topic/topics
Subscriber: An application interested in receiving messages from topic/topics.
Subscription: An entity of interest created for a subscriber to receive intended messages.
Message: The actual data object the publisher sends and the subscribers receive via the topic.
The publisher publishes messages on the topic. Subscribers subscribe to the topic to receive the message. This topic exists in a distributed form in NCache. Hence, its creation occurs on all the cluster nodes. It contains the message store, which stores the actual data objects the publisher publishes in a queue. It also internally maintains a list of all subscribers subscribing to it and a list of publishers.
ITopic/Topic interface facilitates topic creation, topic getting, and topic deletion. You can also delete a topic asynchronously to avoid the wait for a topic to be deleted.
The distributed nature of topics and underlying messages enhances scalability.
Once a message is published to the topic, it fires an event, which the topic relays to subscribers based upon the preference of message delivery option so that they can get the message as needed. The following figure illustrates the role of a topic as an intermediary channel for publishers and subscribers:
In case of temporary subscriber disconnection and then auto re-connection, all information related to a topic such as subscriptions and failure event notifications is re-registered to the topic without any interruption on subscriber end.
NCache introduces topic-level priority that helps you prioritize the lowest and high-priority topics based on their importance. You can create Topics with higher priority if they need to publish critical messages. So, those messages are delivered first. Similarly, you can create topics with a lower priority if they publish unimportant messages. The cache will evict these messages first.
The priority of a topic can be specified at the time of topic creation only and cannot be modified afterward.
You can simply set the topic-level priority at the time of topic creation in order to prioritize the delivery of critical messages.
Topic Deletion Notification
All messages and related meta-info from the cache is deleted upon topic deletion. Hence, the subscriber and publisher receive notifications of this deletion because of the following reasons:
- The subscriber might be waiting for incoming messages from the registered topic. Once the topic does not exist, the subscribers can then handle their execution accordingly through event notification and prevent an infinite waiting state.
- The publisher can prevent sending messages to a non-existing topic and handle any pending messages and future execution accordingly.
To receive topic deletion notification, your application registers for the OnTopicDeleted event.
Publish Messages to a Topic
A publisher can publish messages to a topic by specifying topic name. Messages can be published asynchronously and in bulk to enhance the application performance. A partitioned cache distributes messages across the cluster. For message routing, every message has a unique id called
MessageID at the publisher end, and the hash code of
MessageID determines the cluster node to store the message.
NCache allows publishing a bulk of messages in a single call to improve performance and memory usage.
NCache allows a publisher to use the following properties while publishing messages.
Message delivery option: A publisher can decide if a message will be delivered to a single subscriber or broadcasted to all subscribers at the time of publishing using message delivery option.
Message expiration: The publisher can set expiration of a message to control the life span of a message in the cache. We cover expiration in more detail in the Messages section.
Message delivery failure notification: A publisher can register for MessageDeliveryFailure to get notified if a particular message has failed to deliver. These delivery failure scenarios are elaborated upon in further detail in the Message Assignment and Delivery to Subscription section.
Message delivery failure notification is only for the messages with expiration.
- Topic deletion notification: A publisher can register for topic deletion notification to prevent undue publishing of corresponding messages once a topic no longer exists.
Subscribe to a Topic
A subscriber application can subscribe to topic messages by registering itself to a topic of interest through a subscription. In Pub/Sub, a subscription represents an interest of a subscriber(s) shown in a particular topic.
In case a new node is added to the cache cluster and state transfer is triggered, all subscriptions along with the messages and cache data are replicated to the new node.
NCache allows to subscribe for a topic by providing topic name or a pattern. For further details, refer to Methods of Subscription. Moreover, you can also set message delivery mode as synchronous or asynchronous while creating a subscription.
NCache provides multiple types of Pub/Sub subscriptions which are discussed as follows.
Subscription and its Types
Subscriptions in NCache can be classified into following types:
A durable subscription is a named subscription.
In a durable subscription, the cache ensures that the subscriber does not miss out on any message in case of disconnectivity due to application/machine shutdown, application restart or network failure. Hence, durable subscriptions don't suffer due to subscriber disconnection and re-connection.
Durable subscriptions are not deleted unless the subscriber properly unsubscribes.
If a subscriber is disconnected, the messages intended for this subscriber are stored on a server until the subscriber rejoins or the messages are expired. Durable subscriptions are not deleted automatically on subscriber disconnection unless the subscriber has properly unsubscribed.
A durable subscription is further classified as:
Shared: A shared, durable subscription means that multiple subscribers share a named subscription. The messages assigned to a shared subscription are then load-balanced among the subscribers in a round-robin fashion. Even if any subscriber leaves the network, the messages continue their delivery to active subscribers. So, if a subscriber leaves gracefully or abruptly after an assignment, its assigned messages are reassigned to other active subscribers.
A shared subscription is only supported by a Durable Subscription.
In a shared subscription, the subscription will remain on the topic and cannot be unsubscribed until all subscribers have unsubscribed. This means as long as there is even a single active subscriber, the subscription will stay active.
- Exclusive: An exclusive durable subscription means that there is only one active subscriber registered against a subscription at a time. If a subscriber gracefully unsubscribes, the exclusive subscription can be assigned to a new subscriber. If a subscriber leaves abruptly, the new subscription request is accepted after waiting for an idle time. The assigned messages always stay there even if there is no subscriber.
In a non-durable subscription, the subscribers only receive messages intended for it for as long as it stays connected. If the subscriber leaves the network, it will receive no messages published during its disconnection period. A non-durable subscription is exclusive by default.
Subscribers will lose messages if the subscriber restarts.
Also, non-durable subscriptions are automatically deleted when the subscriber leaves the network. This means that if that subscriber rejoins or establishes the connection again, it will be considered as a new subscription.
Expiration of the Durable Subscription
You can set expiration for a durable subscription. For instance, if there is no active subscriber for a significant time. Here subscriptions will expire after a specific period of inactivity. When a durable subscription is expired, messages assigned to this subscription are reassigned based on the
DeliveryOption of these messages.
Each time the subscriber polls or performs any other activity, the expiration time of subscription is reset.
Handling Inactive Subscriptions
Only Non-Durable subscriptions are marked inactive and this does not apply to durable subscriptions.
In case there is no active subscriber against a subscription after waiting for a certain inactive period, the subscription is considered to be expired. If a subscriber rejoins after disconnection during the inactive period, it will be able to receive its assigned messages.
Inactive subscriptions should be handled to avoid memory overload on server side.
After waiting for the inactive period, the subscription is expired and its assigned messages will be reassigned to other subscriptions. The reassignment of messages depends on the delivery option of messages.
If the message assigned to a subscription A had delivery option set to ALL and it is already assigned to another subscription B before the subscription A expires, the reassignment will not be required.
If the message assigned to a subscription had delivery option set to ANY and the subscription expires, the assigned messages will always be reassigned to any other subscription.
A message contains the actual data object which is sent by the publisher and delivered to the subscribers through the topic. Once the publisher publishes the message to the topic, the registered subscribers are notified that a message pertaining to their interest has been published. In case of multiple messages, they are stored in a sequence inside the queue of a particular topic.
The same message can be assigned to multiple topics. This is uniquely identified through an auto-generated ID.
Here, we further discuss message assignment to the subscription, message delivery and acknowledgement mechanism.
Message Assignment and Delivery to Subscription
Initially, the messages are unassigned on the server side. All subscriptions get messages according to the
DeliveryOption. Once a subscription assignment occurs, the subscriber receives a notification. Then the subscriber implements a poll mechanism to bulk-fetch multiple messages - to reduce server-side overhead. After receiving the messages, the subscriber sends the acknowledgment, and the messages are considered delivered.
Messages are stored in the topic if there is no subscriber to receive the messages. Messages are assigned and delivered to the first subscriber as soon as it subscribes to the topic.
Message Delivery Option
A publisher needs to specify the message delivery option to decide if a message will be delivered to a single subscriber or broadcasted to all subscribers at the time of publishing. It is important to note that, the definition of successful delivery depends on the specified delivery option.
The two delivery options and the corresponding successful delivery criteria are discussed as follows:
All: All registered subscribers receive messages. A message deletes when all the subscribers acknowledge it.
Any: Any single registered subscriber receives the message. If the assigned subscriber doesn't send acknowledgment, the messages get reassigned to the next subscriber. If they do send acknowledgment, the messages are considered successfully delivered.
If a message is successfully delivered according to the specified delivery option, it is removed from the cache.
Message Storage and Distribution
Following are the important aspects of storage and distribution of messages in cache:
Messages are distributed among nodes based upon topologies.
For the Partition of Replica and Partitioned topologies, hash-based distribution is used.
For Replicated topology, messages are replicated to all nodes of the clustered cache. However, the coordinator node is responsible for message manipulation.
For Mirror topology, messages are published to the active node and then replicated to the passive node accordingly.
If the message store is near eviction, an event is logged indicating a full message store and the start of eviction.
Messages have an overhead on cache memory. Hence, message size must be accounted for while calculating the cache size.
Here we discuss the expected behavior of messages in the following cases:
On clear cache: Messages are removed along with cache items once the cache is cleared.
On cache restart: Similar to clear cache, cache contents are cleared when the cache is restarted. This also includes all topics and the messages contained within them.
Eviction: If eviction is enabled on a Pub/Sub cache, the data will be evicted first and then messages will be evicted.
Encryption and compression: If encryption and compression is configured at cache level, it also applies to topic message payload.
State transfer: In case of state transfer, when messages move to another node in the cluster, the node where the message is finally stored is responsible for delivery.
Message Delivery Failure Notification
Subscribers recieve a Delivery failure notification when a message having expiration expires before delivery to any of them.
In case a message does not get assigned or delivered to any of the subscriber, it is considered as failure. It may happen when the subscriber does not exist or is inactive due to network glitch. In such a scenario, a publisher can register message delivery failure notification. It is important to note that delivery failure notification is only for messages with expiration. Delivery of a message is considered to be failed when a message expires before delivery to any of the subscribers
If there are multiple publishers against a topic, the failure notification will be delivered to any of the active publishers which has registered for the failure notification.
Similar to cache items, a publisher can set expiration for messages. The messages will be expired from the cache as soon as the expiration interval has passed and uses the same cleaning interval mechanism.
The expiration of a message determines the time it persists in the cache. Regardless of delivery, if a message is not delivered, it will be removed after the expiration time. If a message is successfully delivered before expiration, it is removed from the cache without waiting for the expiration time.
NCache now supports ordered messages where the sequence of the messages is maintained on the client-side. A publisher can publish ordered messages by specifying a sequence name for a chunk of messages. Then the ordered messages are delivered to the subscribers in the exact same order they are published in. The sequence string should be the same for a chain of ordered messages. Using the sequence string, all the messages reside on the same node using the Location Affinity mechanism.
Synchronous delivery mode can be used for ordered messages.
Following are the important characteristics of ordered messages:
Messages from a publisher with the same sequence reside on a single cache node.
DeliveryOptionis set to Any, all the ordered messages of the same sequence are delivered to the same subscriber. In case the specific subscriber loses connection or becomes unavailable, a new subscriber is reassigned for this purpose. However, if the
DeliveryOptionis set to All, then all ordered messages of the same sequence are delivered to all the subscribers.
In case of state transfer, the ordered messages might lose their sequence and get published without keeping the order.
Ordered messages can only be published using the Sync mode in the Publish API. Bulk and Async API calls are not supported.
NCache provides you with the ability to monitor Pub/Sub Topic statistics and observe various performance counters in this regard. Pub/Sub topic activity and status can be monitored through the Windows PerfMon Counters and Windows PowerShell Cmdlet.
Reliability and High Availability
NCache implements the acknowledge mechanism for message delivery. Messages are kept in memory until they are delivered successfully based on the at least once delivery criteria. Hence, NCache ensures message delivery reliability for Pub/Sub Messaging in distributed architectures.
Moreover, fault tolerance of up to one node in partition-replica topology makes the Pub/Sub Messaging store highly available. If a node leaves the cluster for any reason, the replica has the message backup.
In This Section
Explains how to create, get and delete topics for Pub/Sub model in NCache.
Publish Messages to a Topic
Provides sample code that creates a topic and publishes messages to it.
Subscribe to a Topic
Provides sample code to subscribe against a topic and receive messages of interest.
Explains Pub/Sub events to notify the publisher and subscriber about various events happening across the cache and applications.
Monitoring Pub/Sub Topics
Describes the ways Pub/Sub statistics can be monitored through NCache Monitor, PerfMon and PowerShell cmdlets.