Queue

Queue connector is Tray.io's message broker that allows you to process messages at high scale while maintaining the message order, allowing you to deduplicate messages.

Overview

Queue connector is Tray.io's message broker that allows you to process messages at high scale while maintaining the message order, allowing you to deduplicate messages. Here are some common use cases for this connector:

• Throttle the execution of certain workflows so your account step concurrency limit is not reached.
• You want to preserve the order of incoming requests so they can be processed sequentially.
• Throttle the API calls to underlying services so API rate limits are not breached.

Setup & Authentication

To implement Queue, you will need a sender and a consumer.

1. Sender - Queue connector that will publish "fire and forget" messages to be processed by the consumer asynchronously in a separate workflow.
2. Consumer - Queue trigger that will listen for the messages in real time from the sender

Sender

You can add a Queue connector in the workflow from where you want to send messages. The above screenshot shows the values you need to enter in the 'Sender'.

• Workflow ID: Dropdown to select the 'Consumer' workflow
• Message group ID: Messages can be given a unique ID. Every message group ID offers a concurrency range of 1 to 1000.

This means that you can process up to 1000 messages concurrently in the consumer with let's say group ID: contacts and another 1000 messages concurrently with a group ID: accounts.

Note that this is a string which means alphanumeric characters are allowed (no spaces in between)

• Message body: Actual content of the message you want to process
• Attributes: Metadata for message body
• Message deduplication ID: This is optional, You only need to pass this ID if you are not doing content based deduplication (explained in the consumer below).

Messages with same deduplication ID will be filtered out i.e. if you fire two messages with same deduplication ID, only one will be processed by the 'Consumer'

• Token: This will show up on opening 'advanced properties' from the props panel. This is an added security feature.

You can check for the presence of this token in the consumer workflow and reject messages that do not have the same token value that was sent by the Sender.

Consumer

You can add a Queue trigger in the workflow where you want to consume messages. The above screenshot shows the values you need to enter in the 'Sender':

• Max concurrency: This must be a number between 1 to 1000.

Note: Max concurrency applies at the message group level, that means if the value is set to 10, the consumer will consume 10 messages simultaneously from all groups that have 10 or more messages in them. (provided same auth is used by the 'Consumer' and 'Sender')

• Content-based deduplication: If True, the messages are filtered on the value of the message body sent by the sender.

E.g. If two messages belonging to the same group, have same message body, only one is processed by the consumer.

If False, the messages must have a 'Message deduplication ID' set up on them from the sender. The deduplication ID will decide which messages to process and which one's to filter out.