Throttling and Thresholds

HyperSync Panel uses throttling and thresholds to allow control over rate limiting, and provide a safety net around managing high volume changes.

Throttling

Rules in HyperSync Panel use throttling to limit the frequency the rule can be applied. For example, when provisioning a new account to a directory, it may take a little time for the workflow to be processed, the result to be scanned back, etc. Alternatively, if a particular connected system unreliably accepts updates, it may be desirable to wait a minimum amount time before reattempting the action.

Throttling addresses both scenarios by specifying the minimum amount of time that must elapse before same rule executes again for the same identity graph.

Most provisioning and event rules should be configured with a throttle time, and the default time should consider whether the rule is a "one and done" type of provisioning action, or a recurring action, as well as the expected time for the changes to be fully processed and scanned back into Identity Panel (NOTE: Some providers like AD and Microsoft Graph will generally scan changes back right away).

Although a throttle time (if defined), must be a time span, it is also possible to use a rule expression.

One recommended approach is to use one or more environment variables for a throttle time. This allows many rules to use a shared throttle time which can easily be adjusted for implementation in DEV, testing in QA, and operations in PROD.

Thresholds

Thresholds are used to put a safety catch on how many times a rule can be triggered in a single sync process. For triggering thresholds, there is no difference between a full and a delta sync. When a threshold is invoked, it prevents any instances of the rule from that sync from executing until the actions are approved or the timeout expires. 

If subsequent syncs trigger the rule for different identities they will process normally, but for a given identity the rule will remain in a suspended state until either a decision is made on the suspended action, or the expiry time limit is reached.

Thresholds have a configurable expiration time. By default, the suspended action is cleared when the expiry is reached, but it is also possible to default approve (meaning the action will run unless it is manually rejected).

When a threshold is reached, HyperSync Panel sends a configurable email listing the pending actions. The default email formatting rule includes an HTML header and footer, and a separate table row for each named action. The header includes Approve All and Reject All links, and the individual rows include Approve and Reject links.

Typically, the default messages will be appropriate for most scenarios. In some cases it may be necessary to tailor the copy around the rule formatted properties while preserving the markup for approve and reject links, etc.

Releasing Threshold Actions

If actions are individually approved or if all actions of a particular type for a run are approved, the actions do not process immediately. Instead, they are processed on the next sync of any kind (full sync, delta sync, or individual object sync - Note: must be a commit not a preview). This allows the triggered actions to be tracked in the overall context of a sync history record.

Approving or rejecting "review needed" actions may be done by activating emailed links, generating a report with approve/reject links, or directly constructing links with the correct format:

// Approve Action
https://(identitypanelhost)/hypersync/approve?action=(HyperSync Queue object.Id)

// Reject Action
https://(identitypanelhost)/hypersync/reject?action=(HyperSync Queue object.Id)

// Approve for run
https://(identitypanelhost)/hypersync/run=(SyncHistory.Id)&rule=(HyperSync Rule.Id)

// Reject for run
https://(identitypanelhost)/hypersync/run=(SyncHistory.Id)&rule=(HyperSync Rule.Id)

Version 7.3.1 Updates

As of version 7.3.1, thresholds may be expressed as a raw number, or as a percentage (defined as the percent of the object type within the silo targeted by a given sync rule). E.g. For an export rule to AD user, with the silo containing 100,000 user objects in the graph, a threshold of 5% would be equivalent to a raw threshold of 5,000.

On newly initialized HyperSync installations, the default threshold is 5%, and the maximum selectable threshold is 5%. If it is desired to set a threshold higher than 5% a free license key must be requested via a support ticket. This key is available in a Permanent version, one week, and one day.

Silos with less than 10,000 objects are permitted to have a threshold over 5% even without the license key, since they are commonly used for bulk-update / template type objects.

If a raw number greater than 5% for a large silo is entered, it will be reduced to 5% of the silo/object size, and a sync error will be logged.

Delta synchronizations process in batches derived from number of objects changing in scans as they are saved. If more objects are queued while a delta sync is running, they will typically enroll into the same sync. To ensure that small batches don't fail to trigger thresholds when appropriate, delta synchronizations include a 1 hour time-based lookback at recent action processing history. This means that a set of changes on one delta sync may process but still cause thresholds to take effect on a subsequent delta sync, even if individually the two syncs would have been below the threshold limit.

VIP Thresholds

In addition to volume based thresholds, VIP thresholds make it possible to apply the threshold functionality to individual accounts. VIP thresholds send a regular threshold notification, but they always fire for accounts that meet the VIP Identification rule criteria.

Typically VIP rules will be triggered based on level in the organization, and they provide an opportunity to always manually review major account modifications to high profile users.

Rate Limited Execution

Because Thresholds operate in a binary fashion, either all actions need approval or none of them do, a different strategy must be used if the goal is rate limiting. For example, a downstream system (or service desk capacity) might be unable to accept more than 100 new accounts per sync cycle, and so there may be a need to allow certain actions to process, but only up to a certain point.

Rate Limiting can be achieved by using the Count() function which incrementally returns the number of times its been called in a given time window. This function stores its state in-process, which means it can only be used for rate limiting within a single sync (it won't work e.g. on a daily or weekly basis).

The Count function accepts a Boolean expression (it only increments when the expression is true), a named context (allowing different rules to have their own context), and a time period to reset the counter after (although it will reset at the end of the sync execution regardless).

In the above example, the condition rule for an AD provision rule has been modified to provision at most 100 new AD accounts in a single processing pass.