Managing Data Retention Compliance

Identity Panel version 4.1 introduces a range of new features for managing data retention compliance. These features replace ad-hoc scripted solutions for managing GDPR compliance within and beyond Identity Panel.

New compliance features fall into three categories:

1. Data retention policies
2. Data erasure request handling (Web UI and REST)
3. Detection of un-erased data in external systems

Data Retention Policy

Use Identity Panel security settings to configure data retention.

Data retention is split into three separate polices which may be configured independently. If one is defined, another can be blank, or have a different setting.

Data Retention

The general data retention policy applies to all Identity Panel data, including object change history. It revises Identity Panel data to appear as if the product was installed on the elapsed date.

Delete Retention

The delete retention policy allows a separate time range to be configured for deleted identity objects stored in Identity Panel. This is useful when organizations want to retain multiple years of identity history, but prefer a more aggressive cleanup policy for objects that no longer exist in the external systems.

History Retention

The history retention policy applies a separate time range to run history entries, workflow execution history, and health check history. The primary benefit of the history retention policy is to improve overall performance and data usage by eliminating low-utility data.

Data Erasure Requests

Data erasure is configured and invoked in Security settings. The data erasure feature allows submission of unique/identifying attributes of a user, which will result in the immediate deletion of all objects matching the submission.

The submitted unique value is NOT retained. Instead, after initial object deletion, it is hashed with SHA256 and the hash is used to evaluate objects if needed.

Erasure requests can be used to delete current matching objects, and may also be configured to prevent scanned objects from being recorded if they match the erasure request.

Configuring Erasure Requests

Before submitting erasure requests, it is necessary to configure Identity Panel with mappings of silo name, object type, and attribute list that may be matched by the erasure request. The filter value will be evaluated against each configured silo/object type/attributes configuration set.

When configuring erasure requests for GDPR compliance it is critical that matching criteria be specified for all data silos that are subject to erasure.

Submitting Erasure Requests

Requests may be submitted at the bottom of the security settings page. A data erasure request has three options:

• Apply to Application Users – indicates the request value applies to an Identity Panel user profile instead of stored identity data. Must be an objectSid for on-premise and an Azure identity uuid for SaaS.
• Persist for Scans – indicates that future objects being saved by scans should be filtered off if they match the value specified in the request.
• Filter Value – The actual search value to filter.

REST Submissions

The following PowerShell sample illustrates how to submit an Erasure Request via the JSON REST API:

# Sample script to create a Data Erasure Request

Param(
   [string]$filterValue,
   [boolean]$userProfileOnly,
   [boolean]$persistForScan,
   # Credential argument. Alternatively remove the argument and switch Invoke-RestMethod to use -UseDefaultCredential 
   [Parameter(Mandatory=$False)]
   [PSCredential]$credential=(Get-Credential),
   [Parameter(Mandatory=$False)]
   [string]$uri="https://identitypanel.softwareidm.demo/api/erasure",
   [Parameter(Mandatory=$False)]
   [string]$apiKey="Q8Oyh2mT2_XJfQ6RQGLlp_cxXnRO6jV4JMnb0qAdHJk~"
)

$json = @{
   FilterValue=$filterValue;
   User=$userProfileOnly;
   Persist=$persistForScan
} | ConvertTo-Json

$headers = @{}
$headers["X-Api-Key"] = $apiKey
$response = Invoke-RestMethod $uri -Method Post -Credential $credential -Body $json -ContentType "application/json" -Headers $headers

if ($response.Completed -eq $null){
  Write-Host $response.Errors
} else {
  Write-Host "Success"
}

Reporting 

The following report definition may be copied to a text file and saved with a .json extension. If upload to Identity Panel reports, it will give a basic summary report of Data Erasure Request history.

Definition

{
  "Data": [
    {
      "$type": "SoftwareIDM.ReportingModule.Models.Report, SoftwareIDM.ReportingModule",
      "Id": "063772a8-4904-4f6e-9120-bfda57e3c791",
      "Name": "Data Erasure Requests",
      "UserId": null,
      "UserProfile": false,
      "Tags": null,
      "Roles": null,
      "CacheExpiration": "10:00:00",
      "Description": "Identity data erasure requests",
      "Parameters": [],
      "DataSets": [
        {
          "$type": "SoftwareIDM.ReportingModule.Models.QueryData, SoftwareIDM.ReportingModule",
          "Clauses": [],
          "SubQueries": [],
          "MapRule": "",
          "MemoRules": [],
          "Aliases": null,
          "Name": "Requests",
          "Type": "SoftwareIDM.PanelModule.Models.ErasureRequest, SoftwareIDM.PanelModule",
          "Limit": null
        }
      ],
      "ReportSets": [],
      "Relations": [],
      "FieldProjections": [
        {
          "$type": "SoftwareIDM.ReportingModule.Models.Projection, SoftwareIDM.ReportingModule",
          "Name": "Created",
          "RelationName": "",
          "RelationSide": null,
          "ValueRule": "DateTimeFormat(Created, \"yyyy-MM-dd\")"
        },
        {
          "$type": "SoftwareIDM.ReportingModule.Models.Projection, SoftwareIDM.ReportingModule",
          "Name": "Last Applied",
          "RelationName": "",
          "RelationSide": null,
          "ValueRule": "DateTimeFormat(Applied, \"yyyy-MM-dd\")"
        },
        {
          "$type": "SoftwareIDM.ReportingModule.Models.Projection, SoftwareIDM.ReportingModule",
          "Name": "Applied Count",
          "RelationName": "",
          "RelationSide": null,
          "ValueRule": "AppliedCount"
        },
        {
          "$type": "SoftwareIDM.ReportingModule.Models.Projection, SoftwareIDM.ReportingModule",
          "Name": "Requestor",
          "RelationName": "",
          "RelationSide": null,
          "ValueRule": "Requestor"
        },
        {
          "$type": "SoftwareIDM.ReportingModule.Models.Projection, SoftwareIDM.ReportingModule",
          "Name": "Data Hash",
          "RelationName": "",
          "RelationSide": null,
          "ValueRule": "CoerceString(FilterHash)"
        }
      ],
      "Transformations": [
        {
          "$type": "SoftwareIDM.ReportingModule.Models.Transform, SoftwareIDM.ReportingModule",
          "FilterRule": "Data.Applied Count < 1000",
          "GroupingRule": "",
          "TransformRules": []
        }
      ],
      "Styles": [],
      "Sort": {
        "Field": "",
        "Direction": "Ascending"
      }
    }
  ],
  "Count": 1
}

Detection and Notification

The Erasure request feature may also be used to detect and provide notification when an object is discovered in a system scanned by Identity Panel which matches existing erasure requests. This is done by configuring a workflow to trigger based on the "Erasure Invocation" object type.

Erasure Invocation objects are ephemeral and not actually saved in the database, but they can be used to trigger a workflow for notification, scripting, logging, or some other purpose. An Erasure Invocation includes a reference to the matched Erasure Request, as well as the object being saved. This means the notification workflow may include details to help administrators identify the system and account matching the erasure request.

Workflows for Erasure Invocation will be evaluated for both persistent and non-persistent erasure requests, meaning the decision to filter off the object data is independent of the ability to trigger notifications and other workflows.