Anatomy of a Test Suite

What is a Test Suite?

Measuring the performance of a system is complicated as well as quite an intensive operation. To make this process easier, we have Test Suites. As the name suggests, Test Suites are nothing but your Test Cases. It contains a set of queries. You might have different Test Suites for different set/type of queries. You can also measure the Quality of your search engine against a particular Test Suite by benchmarking that Test Suite.

What are the different types of Test Suites?

I. Public Test Suite (System Scope)

Public Test Suites have a set of Search Queries which are visible to all users. The set of operations performed on a query in Public Test suite will be visible to all users.

User-created Public Test Suites
These are the Public Test Suites created by the users. The queries and Snapshots inside these Test Suites are visible to all users.

II. Private Test Suite

Private Test Suites have a set of search queries which are visible only to the owner of that Test Suite. Everything you can do with a Public Test Suite can be done with a Private Test Suite; the only difference is that an operation performed on a query in Private Test Suite will be visible only to the owner of that Test Suite.

User-created Private Test Suite
These are the Private Test Suites created by the users. These can be created in the same way as Public Test Suites but under the Private section. The visibility of these Test Suites is limited to the user who created it.

III. Inbox

When you save any query from the Search Application, it first comes to the Inbox. To perform any operation on a query, you have to move it from the Inbox to any other Test Suite.

IV. Archive

As the name suggests, the Archive contains archived queries. You can move a query from a Test Suite to the Archive.

How to create a Test Suite?

There are two types of Test Suites:

  1. Public Test Suite
  2. Private Test Suite

You can create any type of Test suite depending on the visibility you want for your search queries. Both of them require you to follow the same procedure. The following steps will create a Public Test Suite.

What can I do with my Test Suite?

The Test Suite is your playground. You can add a query to a Test Suite, move the query from one Test Suite to another, archive a Query, or Benchmark to see the quality of your search.

How to add a query in a Test Suite?

Adding a query to a Test Suite is quite simple. Go to the Query view by clicking on the View Test Suite icon. Use the Add Query option that is present in the Tools menu on the top right corner of the grid, as shown below.

You also have the option to Import queries from .csv files. To import queries, you need to have rows in the format, mentioned below:

Column1, Column2, Column3

Query, 'Comma separated Ids(Optional)', 'Comma separated Value of field'

If you don’t provide IDs and want to Populate IDs based on the field value then you have to select the Field from the Field List and Provide values for that field in the .csv file.

How to move a query from one Test Suite to another?

Moving a query is as simple as creating one. This option is present for each query inside the Query Grid. Click on More Actions and from the menu, select the Move Query icon.

How to archive a query from a Test Suite?

This option is present for each query inside the Query Grid. When you archive a query, it moves to the Archive Test Suite. It can be again moved from the Archive to any other Test Suite.

How to move a Test Suite from Private to Public?

When your private Test Suite is fetching good results, you can make it public for other users. Once you make your Test Suite public. it will be tracked for all future benchmarks which have the System scope.

How to delete a Test Suite?

You can delete a Test Suite by clicking on the Delete icon.

What is a Snapshot?

A Snapshot is nothing but the expected results with ratings (Relevant or Irrelevant). You can execute a query and rate each document as Relevant or Irrelevant. Whenever you benchmark a Test Suite, it executes all the queries in that test suite. In order to find out how well your executed queries have performed you need rated documents of a query before it was benchmarked for comparison.

How do I take a Snapshot?

You must have a query in your Test Suite to take its snapshot. You can take a snapshot in the following 3 ways:

1. Run Query

2. Execute Test Suite

3. Save Ratings from Search App - Optionally if you have integrated the Save Ratings functionality in the Search application, you can save ratings from the Search app as well.

Where can I view/edit my Snapshot?

You will see the View/Edit option under each query. You need to click on the View/Edit icon and you will be able to modify the ratings for the snapshot.

The user can manually add documents if they are not part of the executed query. Functionality of searching documents in particular fields is also provided to assist the user while manually adding the documents to the snapshot.

How are effective ratings calculated if the query contains more than one snapshot?

Effective ratings are nothing but non-conflicting ratings from all the available snapshots. If the user has marked any snapshot as the baseline snapshot, then the ratings from that particular snapshot will be considered as effective ratings. The user can see effective ratings from any query by clicking on 'the View Rated Documents' icon present in front of the query.

How can I compare Snapshots?

To compare Snapshots, you must have at least two snapshots for a given query. You need to select two Snapshots which you want to compare and then click on the Compare icon.