REST API Overview

Overview

Identity Panel, Service Panel, and AccessPanel are single-page-applications which use a internal API to interact with the web application.

Request Handling

The REST API uses .NET Core Services and Middleware to apply security settings, content options, etc. to request pipelines universally.

Requests use content-type negotiation to determine the data type to accept and return. Most endpoints support JSON and BSON (BSON is primarily used by Panel Service as it consumes less bandwidth and has a faster serializer).

Some requests use file disposition headers and query parameters to support return the response as a file download.

Endpoints allow multipart form data as a file upload in lieu of the request stream. Files are not uploaded and stored on the server, instead they are copied to the request stream and passed through the usual content negotiation, deserialization, parsing, etc.

Request Services

• Data Protection: Standard data-protection API, uses default key-rollover, but with a custom keystore to persist key data in MongoDB (shared Identity Panel database, not tenant-specific) instead of filesystem. The keys themselves are protected via Wrap/Unwrap key operations using an Azure Storage Vault with HSM
• Web Encoders: common encoding, decoding operations
• Antiforgery: Standard .NET Core anti-forgery protection, cookie properties set to HttpOnly, with HTTPS SecurePolicy of Always (on-premise installations set to Same-As-Request)
• Type registration: Application startup uses attribute reflection to generate a white-list of de-serializable types for JSON.NET library
• Authorization: Standard .NET Core attribute based authorization service. Registered with custom policies to enforce authentication, license activation (via string slugs), and data permissions (via string formatted as Operation|DataObject). The authorization services is applied via attributes on controllers and actions, and via integration into serialization/de-serialization handlers in the database ORM
  • AuthLicense (licensing policy authorization handler)
  • AuthResource (data permission authorization handler). Data permissions are associated to user roles. User roles are assigned based on group membership auth_token claims.
  • NOTE: ORM authorization is disabled in some AccessPanel and ServicePanel requests. See relevant API sections.
• .NET Core MVC (MVC framework and handler)
  • JSON.NET serializer/de-serializer with white-list based type contract resolver
  • MongoDB.Bson library based input/output formatted with white-list based type resolver
• MongoEngine: ORM wrapper for MongoDB, tenant and authorization aware
• ElasticFullText: ORM for full text indexing service
• GetSettings: helper for reading tenant settings
• GetSecurity: helper for evaluating security assertions based on licensing and role assignment
AzureGroupClaims: helper for resolving group claims via Microsoft Graph API when the user has more groups than fit in security token

Request Pipeline

1. Decode forwarded headers: (if needed, obtain the original hostname from X-Forwarded-Host set by the Azure Frontdoof service). The original hostname is needed to validate the issuer of an OIDC token and to avoid rejecting cookies with Same-Origin policies.
2. Exception Handler: (wrap the remainder of the request pipeline in an exception handler that logs errors and redirects to a generic error page). In some cases user-friendly customer 500 error messages are produced by throwing an exception of type MessageError()
3. Gzip Files: subclass of .NET Core static files middleware, if a user requests a static file path with a permitted extension (based on default .NET Core static files handler rules), and a file exists with name.gz, and the request is GET/HEAD and has Accept-Encoding gzip, returns the *.gz file with the Content-Encoding gzip header. Static files are rooted to serve out of a sub-folder for static files.
4. Api Result: Makes adjustments to the request stream if needed for handling:
   1. If multipart/form-data is present, and content type is application/octet-stream, checks uploaded file extension and sets request Content-Type to one of: application/json, application/bson, text/xml, text/html, application/vnd.ms-excel, application/png, application/jpeg, application/gif. Replaces Body stream with file stream of first file.
   2. Decompresses and replaces request stream if content-encoding is gzip
   Sets internal request Content-Type if a mediaType query parameter is present (for working with clients that don't set content-type)
Authentication: default .NET Core authentication wrapper, uses cookie sessionsUser Auth middleware:
Checks authentication stateInitializes database connection to correct tenant based on authed userChecks for X-Tenant-Override header or tenantId query parameter, switches request tenant if user is a Service Admin for that tenant (as designated by membership in a group id claim value on the indicated tenant record)Applies anti-forgery validation by checking the X-RequestVerificationToken header or verificationtoken query parameter (front-end UI uses header in most scenarios)Adds Tenant and User objects to request state for controllers to accessLogs the request in the tenant specific AccessLog collection
Set X-Frame-Options DENY headerRouting and AuthorizationREQUEST PROCESSING

Default Roles with Permission

Admin: Read/Write/Execute on all data permissions, access to all UI featuresWriter: Read/Write/Execute on all data permissions, no access to UI features (built-in role for Panel Service)User: Read on all data permissions Write on ReportRecord (cache object), Write on WorkflowRecord with an authorized link, access to Object, Search, Dashboard, History, ReportsEveryone: no accessSystem Permissions (available in all requests, but not necessarily user returnable objects):
Read
AttributeNameMap (part of ORM)FieldMap (part of ORM)JsonSettingsDataLockImpl (contention control)HistoryFilters (performance caching)User (needed for login process)Tenant (needed for login process)
Write
AttributeNameMapLogMessage (error logging)AccessLog (request logging)DataLockImplHistoryFiltersUserTenantSearchQueue (full-text indexing)DataSignature (system non-repudiation records)

User Login

User interactions with the back end occur predominantly over JSON web API requests from the browser. These requests happen in an authenticated session (Azure OpenIDConnect for SaaS, Windows Integrated Auth for on-premise).

Login session is managed through the default .NET Core 3.1 login session cookie mechanism. All requests (both GET and POST) require an API Key or a Verification token for CSRF protection.

Authentication and session establishment is automatic for integrated authentication, and through the /account controller for OIDC.

For OIDC connections, the tenant of the logged in user is established by the TenantId JWT claim of the authentication token. On the backend, a shared Tenant database lists the issuer and domain name for each tenant, and contains connection strings to dedicated elasticsearch indices and tenant databases. Each tenant receives its own database with the name: idp_<tenant>_db.

The common database also contains shared error logging, an AuthState collection (for SCRAM-SHA state), and KeyData storage for persisting and managing key rollover for the .NET data protection API.

Daemon Login

Panel Service (for data collection, workflows, schedule steps, etc.) logs in as a service, and so can't use interactive OIDC authentication. For on-premise, Panel Service just uses the normal windows authentication. But for SaaS it uses SCRAM-SHA512 wrapped in a JSON/BSON payload for the messages (see RFC 5802).

A Panel Service must be paired with the web application by a user entering details about the service on the Install Service page, then pressing the "Create or Reset Application Password" button. This generates a user/password pair in the form:

User name: tenant|server|domain\serviceaccount

Password: 512 byte crypto-random password

When the password is pasted into the Panel Service UI it is encrypted with DPAPI in CurrentUser mode and written to a local config file.

Panel Service authentication credentials can only be created by requests to /api/tenant/scramregister by a user in the Admin or Writer roles. A new User record is created in the Tenant Database with the StoredKey and ServerKey parts specified by the SCRAM algorithm.

All Panel Service requests must also have an X-API-Key header to pass the anti-forgery validation. The following link illustrates how to obtain an authenticated web session in PowerShell.

https://bitbucket.org/softwareidm/mimtest/src/master/scripts/ScramAuth.ps1

Accessing the API

If interacting directly with the REST API is desired, it may be performed using the following helpers:

Any logged in session in Identity Panel can access helpers.rest via the javascript console. This helper handles reading the CSRF token input, prefixing /api, including credentials, and calling error handling:

helpers.rest.get(
  uri, 
  result => callback, 
  error => callback, 
  overrideOpts
)

helpers.rest.post(
  uri, 
  { payload },
  result => callback,
  error => callback,
  overrideOpts
)

JQuery API Call Implementation:

var restRoot = location.protocol + "//" + location.host + "/api/";

helpers.rest = {
    url: restRoot,
    defaultError: function (result, status) {
        if (result && result.status && result.status === 403) {
            helpers.alert('Access forbidden, contact an administrator to check permissions', 'Access Forbidden', 'danger');
        }
			
        if (result && result.status && result.status === 500 && result.responseText && result.responseText.match('Error":')) {
            helpers.alert($.parseJSON(result.responseText).Error, 'Error Loading', 'danger');
        } else if (result.Error) {
            helpers.alert(i18n.rest.errLoading + result.Error, "Error Loading", "danger");
        } else if (status) {
            helpers.alert(i18n.rest.errLoading + status, "Error Loading", "danger");
        }
    }
};

helpers.rest.get = function (url, callback, errorCallback, opts) {
    if (!errorCallback) {
        errorCallback = helpers.rest.defaultError;
    }

    var self = this,
        ps = {
            type: "GET",
            url: restRoot + url,
            cache: false,
            dataType: 'json',
            contentType: 'application/json; charset=utf-8',
            timeout: 30000,
            headers: {
                "X-RequestVerificationToken": $("#csrf input").val()
            },
            beforeSend: function (xhr) {
                xhr.withCredentials = true;
            },
            success: function (result) {
                if (!result || result.Error) {
                    errorCallback.call(self, result);
                } else {
                    callback.call(self, result);
                }
            },
            error: function (xhr) {
                if (window.isHosted && xhr && xhr.status && xhr.status === 401) {
                    location.reload();
                }
                errorCallback.apply(this, arguments);
            }
        };

    if (opts) {
        $.extend(ps, opts);
    }

    if (window.isHosted && access.IsSuper) {
        if (helpers.rest.xoverride) {
            ps.headers["X-Tenant-Override"] = helpers.rest.xoverride;
        } else if (ps.xoverride) {
            ps.headers["X-Tenant-Override"] = ps.xoverride;
        }
    }

    delete ps.xoverride;

    $.ajax(ps);
};
helpers.rest.post = function (url, data, callback, errorCallback, opts) {
    if (!errorCallback) {
        errorCallback = helpers.rest.defaultError;
    }

    var self = this,
        ps = {
            type: "POST",
            url: restRoot + url,
            cache: false,
            data: JSON.stringify(data),
            dataType: 'json',
            contentType: 'application/json; charset=utf-8',
            timeout: 30000,
            headers: {
                "X-RequestVerificationToken": $("#csrf input").val()
            },
            beforeSend: function (xhr) {
                xhr.withCredentials = true;
            },
            success: function (result) {
                if (result && result.Error) {
                    errorCallback.call(self, result);
                } else {
                    callback.call(self, result);
                }
            },
            error: function (xhr) {
                if (window.isHosted && xhr && xhr.status && xhr.status === 401) {
                    location.reload();
                }
                errorCallback.apply(this, arguments);
            }
        };

    if (opts) {
        $.extend(ps, opts);
    }

    if (window.isHosted && access.IsSuper) {
        if (helpers.rest.xoverride) {
            ps.headers["X-Tenant-Override"] = helpers.rest.xoverride;
        } else if (ps.xoverride) {
            ps.headers["X-Tenant-Override"] = ps.xoverride;
        }
    }

    delete ps.xoverride;

    $.ajax(ps);
};

TypeScript API Call Implementation:

export let headers: { [id: string]: string; } = {
    'Content-Type': 'application/json',
    'pragma': 'no-cache',
    'cache-control': 'no-cache',
    'Accept-Language': navigator.language,
};

const csrf = document.getElementById('csrf');
if (csrf !== null) {
    headers['X-RequestVerificationToken'] = csrf.getElementsByTagName('input')[0].value;
}

fetch(
    `api/${uri}`, 
    {
        method: 'GET',
        credentials: 'same-origin',
        headers: headers,
    }
).then((response) => response.json() as Promise<expected type>)
.then((data) => {
 // handle response
});