2024

May 7 2024

Incident reported

Description:

A user editing HyperSync settings in a shared non-prod environment reported seeing another customer's HyperSync settings. On page refresh settings reverted to logged-in tenant.

Scope of affected systems:
Only the non-prod shared host Eagle-EU-Preview was affected

There was no cross-tenant write access, read access was ephemeral and limited to the affected settings object

No user or PII data was exposed

No secret/encrypted data was exposed

No production data was exposed

Ongoing operations were not affected for any customers

Dedicated instances and instances hosting a single customer were not affected (includes all production instances)

Analysis:

On investigation, we determined that this incident was caused by an interaction between two software bugs, each of which occurs very rarely, and not reproducibly, but which in combination returned a cross-tenant response to a GET query of /api/settings.

Bug 1:

In rare cases it is possible for settings to fail to save in a way that results in the setting being written to a shared logging and tenant coordination database. The mis-saved setting is persistent in this database. As of the investigation of this incident, this has occurred twice in the lifetime of the affected non-prod environment and has not occurred in any other environments.

Bug 2:

In rare cases it is possible for settings read failure to result in attempting to read a setting from the shared logging database. In the normal case this would simply result in an error and a null read, but when Bug 1 has previously occurred, it may return the previously saved setting, even if that setting originated from a different tenancy.

Mitigation and Prevention:

Following review of all environments, incorrectly saved settings objects in Eagle-EU-Preview have been purged. No incorrectly saved settings were found in other environments. This mitigates the possibility of bug 2 incorrectly returning data.

Eagle-EU-Preview has been updated with an emergency patch to version 6.5.4, which refactors the database communications layer to use immutable connection settings, requiring new object instances to be initialized for all shared logging activity. Using immutable connections fixes bug 1 by removing the possibility of re-use of a logging connection for normal settings interaction.

Note: instances on versions prior to 6.5.4 are not affected if the environment is at the dedicated hosting tier.

2023