Designing a Test Case

There are a number of requirements that go into a good test case, and many of them have implications on the design of your identity environment. This article lists out what makes a good test case, and how you can tailor your environment and MIM Test solution to meet those requirements.

• Single (or limited) Concerns
• Performance (quick to run)
• Automated
• Repeatable
• Maintainable

Some additional considerations go into designing a suite of cases:

• Positive and Negative (edge case testing)
• Reasonable coverage

Limited Concerns

To the extent feasible, it's a good idea to have a single test case cover a fairly small surface area of business requirements. The larger and more elaborate a test case the trickier it is to debug the case itself, and it can distract from the actual project being validated.

Sometimes it is worth having several cases that are quite similar to each other if it simplifies the individual cases. Another benefit of splitting cases is it makes it easier to get appropriate coverage without over-complicating things.

It is even possible to have two separate cases perform essentially the same operation but do different validations on the results.

Performance

To get the most benefit out of automated testing you need to be able to run your suite in a reasonable amount of time. The prime implication of this, is MIM Test should own the environment. It is typical for DEV or TEST environments to have an extract or partial extract of production data to allow individual accounts to be selected and tested with. This is NOT the correct approach to take with MIM Test, as having thousands of accounts in the solution will severely impact sync times for testing.

The majority of accounts in the test environment should be created, synced and deleted by the automated testing platform itself.

Automated

To run test cases regularly as part of a regression suite they have to be fully automated. Any case that requires manual setup or cleanup will be a burden on regular use of the system. Often minor changes to the Identity system design will be needed to make it more testable.

Repeatable

To allow proper regression testing a test suite must be repeatable. Whenever you create a test case thing about how it will be automatically reversed. Here are some possible strategies:

• MIM Sync/Portal can be reversed with a database restore
• SQL tables can be truncated or have delete queries
• AD can have OUs emptied, or LDAP filters can be used to delete multiple accounts.
• You may use MIM to synchronize a searchable value through, then delete all accounts with that value
• Delimited files can be truncated or have rows deleted. Alternatively, a PowerShell task can copy back an older version of the file

One implication of repeatability is the need to own and have delete permissions on all MAs. In some cases it may be necessary to add a staging table or API shim layer to your sync solution if you have a connected system (such as SAP HR or PeopleSoft) that policy prohibits free manipulation of.

Maintainable

It is important to budget resources for maintaining a test solution over time as part of your application lifecycle. One pitfall is to use test cases up to the initial deployment, then fail to maintain them through subsequent break fixes and functionality iterations. When creating test cases give thought to how they can be made more flexible and less brittle. This will reduce cost and effort of maintaining the case later.

Some techniques to improve maintainability:

• Create re-usable fixtures and assertions. By memoizing constant values and piping those values into re-usable actions you can make updates in one place and affect multiple cases at once.
• Avoid testing too many things at once in a single case
• Re-run your cases with every iteration of changes to the solution
• Prune cases that are no longer relevant as requirements change

Positive and Negative Testing

MIM Test cases can be written to test both positive results and failure conditions. Generally it is easier to anticipate positive case requirements. It is often helpful to add additional negative test cases over time when issues crop up with bad data.

Coverage

Deciding what test case coverage is needed is a balance between effort to create and maintain the cases, with their utility. By adding business requirements to the MIM Test project you can map them on to cases to make sure that there are case for all the highlights of the solution. MIM Test will add green boxes next to requirements as they are associated to cases, making it easy to see which requirements have cases already.

Test Case Flow

Typically test cases will be grouped into suites. A couple good reasons for relating cases in a suite are:

1. Tests that deal with related functionality may be grouped together, e.g. a suite of provisioning cases
2. Tests may be grouped if they have to run in a particular order. For example a certain provisioning case may need to run to prepare for an update case.

Internally, test cases are made of fixtures, schedules, and assertions, most commonly in that order, since a natural flow is to make changes, synchronize and check results. Often other orders will be used, for example, a more complicated case might have the following sequence:

• Assertion to memoize values
• Fixtures to prep pre-conditions
• Sync cycle
• More fixtures to stage the actual test cases
• Assertion to memoize new values
• Another sync cycle
• Assertions to check results