Managing Data Retention Compliance

Identity Panel provides a range of features for managing data retention compliance. These features replace ad-hoc scripted solutions for managing GDPR compliance within and beyond Identity Panel.

New compliance features fall into three categories:

1. Data retention policies
2. Data erasure request handling (Web UI and REST)
3. Detection of un-erased data in external systems

By default data retention policy is unlimited.  Be sure to understand your organization's own data retention requirements before proceeding to change this default, particularly if no external SIEM ingestion is configured and Identity Panel alone provides access to your historical data.

IMPORTANT! For self-hosted implementations of Identity Panel, be sure to create, maintain and verify database backups, particularly prior to your initial implementation of a retention policy.

Data Retention Policy

Use Identity Panel Security Settings (~/settings/security) to configure data retention.

Data retention is split into four separate polices which may be configured independently. If one is defined, another can be blank, or have a different setting. By default, all data is retained indefinitely and a retention policy may be specified to remove data after an elapsed period of time.

Notes

• Default retention is unlimited. The values above in light grey are hints only and should not be confused as default values.
• Once set, values can subsequently be deleted, and the hint values will again appear in light grey.
• A value of 0 (zero) will equate to no data retention.

Data Retention

The general data retention policy applies to all Identity Panel data, including object change history. It revises Identity Panel data to appear as if the product was installed on the elapsed date.

Delete Retention

The delete retention policy allows a separate time range to be configured for deleted identity objects stored in Identity Panel. This is useful when organizations want to retain multiple years of identity history, but prefer a more aggressive cleanup policy for objects that no longer exist in the external systems.

History Retention

The history retention policy applies a separate time range to run history entries, workflow execution history, health check and HyperSync action history.

The primary benefit of the history retention policy is to improve overall performance and data usage by eliminating low-utility data.  For example, setting a value according to the hint (365) or perhaps initially something larger (e.g. 2 years = 730 days) is advisable if there is evidence of report performance degradation.

Request Retention

The request retention policy allows a separate time range to be configured for user request submissions stored in Identity Panel, such as Access Requests (Access Panel) or MIM Portal requests and approvals. This is useful when organizations want to retain multiple years of identity history, but prefer a more aggressive cleanup policy for what could be considered more ephemeral record types.

As with history retention itself, implementing a request retention policy is recommended for improving overall performance and data usage.

Retention Processing

All data Retention policies are processed automatically by a weekly web job, or (for version 7 sites under specific conditions) via a step in a Schedule.

Note: The Data Disposal form on the Service Desk site is a way for customers to log requests to remove entire databases, e.g. in case of moves between deployment models post transition.

Data Erasure Requests

Data erasure is configured and invoked in Security settings. The data erasure feature allows submission of unique/identifying attributes of a user, which will result in the immediate deletion of all objects matching the submission.

The submitted unique value is NOT retained. Instead, after initial object deletion, it is hashed with SHA256 and the hash is used to evaluate objects if needed.

Erasure requests can be used to delete current matching objects, and may also be configured to prevent scanned objects from being recorded if they match the erasure request.

Configuring Erasure Requests

Before submitting erasure requests, it is necessary to configure Identity Panel with mappings of silo name, object type, and attribute list that may be matched by the erasure request. The filter value will be evaluated against each configured silo/object type/attributes configuration set.

When configuring erasure requests for GDPR compliance it is critical that matching criteria be specified for all data silos that are subject to erasure.

Submitting Erasure Requests

Requests may be submitted at the bottom of the security settings page. A data erasure request has three options:

• Apply to Application Users – indicates the request value applies to an Identity Panel user profile instead of stored identity data. Must be an objectSid for on-premises and an Azure identity guid for SaaS.
• Persist for Scans – indicates that future objects being saved by scans should be filtered off if they match the value specified in the request.
• Filter Value – The actual search value to filter.

REST Submissions

The following PowerShell sample illustrates how to submit an Erasure Request via the JSON REST API:

# Sample script to create a Data Erasure Request

Param(
   [string]$filterValue,
   [boolean]$userProfileOnly,
   [boolean]$persistForScan,
   # Credential argument. Alternatively remove the argument and switch Invoke-RestMethod to use -UseDefaultCredential 
   [Parameter(Mandatory=$False)]
   [PSCredential]$credential=(Get-Credential),
   [Parameter(Mandatory=$False)]
   [string]$uri="https://identitypanel.softwareidm.demo/api/erasure",
   [Parameter(Mandatory=$False)]
   [string]$apiKey="Q8Oyh2mT2_XJfQ6RQGLlp_cxXnRO6jV4JMnb0qAdHJk~"
)

$json = @{
   FilterValue=$filterValue;
   User=$userProfileOnly;
   Persist=$persistForScan
} | ConvertTo-Json

$headers = @{}
$headers["X-Api-Key"] = $apiKey
$response = Invoke-RestMethod $uri -Method Post -Credential $credential -Body $json -ContentType "application/json" -Headers $headers

if ($response.Completed -eq $null){
  Write-Host $response.Errors
} else {
  Write-Host "Success"
}

Reporting 

The following report definition may be copied to a text file and saved with a .json extension. If upload to Identity Panel reports, it will give a basic summary report of Data Erasure Request history.

Definition

{
  "Data": [
    {
      "$type": "SoftwareIDM.ReportingModule.Models.Report, SoftwareIDM.ReportingModule",
      "Id": "063772a8-4904-4f6e-9120-bfda57e3c791",
      "Name": "Data Erasure Requests",
      "UserId": null,
      "UserProfile": false,
      "Tags": null,
      "Roles": null,
      "CacheExpiration": "10:00:00",
      "Description": "Identity data erasure requests",
      "Parameters": [],
      "DataSets": [
        {
          "$type": "SoftwareIDM.ReportingModule.Models.QueryData, SoftwareIDM.ReportingModule",
          "Clauses": [],
          "SubQueries": [],
          "MapRule": "",
          "MemoRules": [],
          "Aliases": null,
          "Name": "Requests",
          "Type": "SoftwareIDM.PanelModule.Models.ErasureRequest, SoftwareIDM.PanelModule",
          "Limit": null
        }
      ],
      "ReportSets": [],
      "Relations": [],
      "FieldProjections": [
        {
          "$type": "SoftwareIDM.ReportingModule.Models.Projection, SoftwareIDM.ReportingModule",
          "Name": "Created",
          "RelationName": "",
          "RelationSide": null,
          "ValueRule": "DateTimeFormat(Created, \"yyyy-MM-dd\")"
        },
        {
          "$type": "SoftwareIDM.ReportingModule.Models.Projection, SoftwareIDM.ReportingModule",
          "Name": "Last Applied",
          "RelationName": "",
          "RelationSide": null,
          "ValueRule": "DateTimeFormat(Applied, \"yyyy-MM-dd\")"
        },
        {
          "$type": "SoftwareIDM.ReportingModule.Models.Projection, SoftwareIDM.ReportingModule",
          "Name": "Applied Count",
          "RelationName": "",
          "RelationSide": null,
          "ValueRule": "AppliedCount"
        },
        {
          "$type": "SoftwareIDM.ReportingModule.Models.Projection, SoftwareIDM.ReportingModule",
          "Name": "Requestor",
          "RelationName": "",
          "RelationSide": null,
          "ValueRule": "Requestor"
        },
        {
          "$type": "SoftwareIDM.ReportingModule.Models.Projection, SoftwareIDM.ReportingModule",
          "Name": "Data Hash",
          "RelationName": "",
          "RelationSide": null,
          "ValueRule": "CoerceString(FilterHash)"
        }
      ],
      "Transformations": [
        {
          "$type": "SoftwareIDM.ReportingModule.Models.Transform, SoftwareIDM.ReportingModule",
          "FilterRule": "Data.Applied Count < 1000",
          "GroupingRule": "",
          "TransformRules": []
        }
      ],
      "Styles": [],
      "Sort": {
        "Field": "",
        "Direction": "Ascending"
      }
    }
  ],
  "Count": 1
}

Detection and Notification

The Erasure request feature may also be used to detect and provide notification when an object is discovered in a system scanned by Identity Panel which matches existing erasure requests. This is done by configuring a workflow to trigger based on the "Erasure Invocation" object type.

Erasure Invocation objects are ephemeral and not actually saved in the database, but they can be used to trigger a workflow for notification, scripting, logging, or some other purpose. An Erasure Invocation includes a reference to the matched Erasure Request, as well as the object being saved. This means the notification workflow may include details to help administrators identify the system and account matching the erasure request.

Workflows for Erasure Invocation will be evaluated for both persistent and non-persistent erasure requests, meaning the decision to filter off the object data is independent of the ability to trigger notifications and other workflows.