All eazyBI for Jira eazyBI for Confluence Private eazyBI

Xray Test Management
eazyBI for Jira

Xray for Jira is a complete, full-featured Atlassian verified test management add-on for Jira. Xray supports both manual and automated tests and a complete testing life cycle: test planning, test designing, and test execution to help developers improve the quality of their systems.

Xray for Jira is an add-on developed by Xpand Add-ons, an Atlassian Solution Partner.

On this page:

Xray Data Import

Data Import in Server version

If you have installed Xray add-on for Jira Server then in Jira import options, section Add-ons, you will see an options Xray. Select the checkbox to import Xray data and after the next data import, Xray specific measures and dimensions will be available for your reports. You can also check or uncheck some of Xray specific custom fields for data import.

Data Import in Cloud version

eazyBI uses API keys to get data from Xray Cloud. 

  1. In Jira Cloud, go to section Apps → Xray -> API Keys and create API key. For the API key select user with Jira administrator rights.
  2. In eazyBI, Jira import options section Add-ons you will see an option Xray which is available when you have installed Xray Test Management app for Jira. Enter generated Client Id and Client secret.

After the next data import, Xray specific measures and dimensions will be available for your reports.

Data import in Private eazyBI.

You can import Xray custom fields also when using Private eazyBI. Add the following settings to the configuration file config/eazybi.toml to enable Xray data import. There are more details on the needed configuration for data from Jira apps.

Limit imported test runs

Option to limit imported testing data is available on Cloud and since eazyBI version 6.2.

To reduce import time and the testing data in the cube, you can import only the most recent test runs.

Set a relative time limit for test runs like 3 months to import test runs from the last three months or a particular date like Jun 14 2020  to set a date range from that date until now. Test runs are imported if they are created (test is added to the test execution) within the time limit (for scheduled test executions) or with started on date within the time limit (for completed or in progress test runs). If there is no limit set, then all test runs are imported.

You can set a limit to all eazyBI accounts through advanced settings

[jira.xray]
test_executions_limit = "3 months"

Another option is setting the limit for an individual eazyBI account. In the import options, add the period in the field "Imported Test Run limit". 

If import limits are applied for the whole eazyBI instance using advanced settings, then the limit in import options won't exceed the limit in advanced settings. In such a way, the overall limits can be set on the eazyBI account, and, in an individual account, this time can be reduced more.

Dimensions

After the import the following Xray specific dimension will be added:

Xray Requirement UPDATED

Requirement issues with issue types selected in Xray configuration "Requirement Issue Types"; for example, story, feature, etc. The dimension is organized into Project and Requirement levels.

Use this dimension with the measure Xray Requirements created count to see all covered and not covered requirements.

Xray Test Plan

Test plan issues that have assigned at least one test or executions with tests. The dimension is organized by default hierarchy (in Project and Plan levels) or by Test Plan boar hierarchy (in Project, Plan, Folder, and Subfolder).

Xray TestTests issues; all created issues with Xray-specific issue type "Test". The dimension is organized by default hierarchy (in Project and Test levels) or by Test repository hierarchy (in Project, Folder, Subfolder, and Test).
Xray Test Set

Test set issues that have at least one test. This dimension is organized into Project and Set levels.

Xray Test Pre-condition

Test pre-condition issues. This dimension is organized into Project and Pre-condition levels.

Xray Test Execution

Test execution issues that have assigned at least one test. This dimension is organized into Project and Execution levels.

Xray Defect

Defect issues that are found during testing and linked through a test execution screen. This dimension is organized in Project and Defect levels.

The Xray Defect dimension works well with execution-related measures, for example, Xray Test Run, Xray defect count.

Xray Defect Priority NEW

Current defect priority of defects that are found during testing and linked through a test execution screen. This dimension is organized in Project and Defect levels.

The Xray Defect Priority dimension works well with execution-related measures, for example, Xray Test Run, Xray defect count.

Xray Defect Priority dimension available on Cloud and starting from the eazyBI version 6.6.

Xray Defect Status NEW

Current defect statuses of defects that are found during testing and linked through a test execution screen. This dimension has a secondary hierarchy that groups statuses by Category.

The Xray Defect Status dimension works well with execution-related measures, for example, Xray Test Run, Xray defect count.

Xray Defect Status dimension available on Cloud and starting from the eazyBI version 6.6.

Xray Test Run Assignee

Users who are assigned to test runs.
Xray Test Executed byUsers who executed tests.

Xray Test Run Status UPDATED

Test execution statuses – by default, there are TODO, PASS, EXECUTING, FAIL, and ABORTED statuses, but also there can be other user-defined statuses.

The Xray Test Run Status dimension works well with execution-related measures, for example, Xray Test Run.

The former dimension name til version 6.5 is Xray Test Execution Status.

Requirement Status NEW

Testing status of requirement issue (feature, story, etc.) based on Xray calculated custom field "Requirement status".

If you change the configuration for calculating requirement status in Jira, ensure to re-import data in eazyBI to get and refresh all updated values.

The Requirement Status dimension is available since eazyBI version 6.6.

Dimension is not available for the Jira Cloud version. 

TestRunStatus NEW

Testing status of Test issue based on Xray calculated custom field "Requirement status".

The TestRunStatus dimension is available since eazyBI version 6.6.

Dimension is not available for the Jira Cloud version. 

Xray Pre-condition TypeTypes of pre-conditions - Manual, Cucumber.
Xray Test TypeTypes of tests (Manual, Cucumber, Generic, etc.). The dimension is organized into Type and Scenario Type levels.

Xray Test Execution Test Environment

Test Environments

Test environment where tests are executed or scheduled for execution.

The dimension name on Cloud is Xray Test Execution Test Environment. The dimension name on the Data Center and Server match the issue custom field name Test Environment.

Revision

System revision for test execution.

Dimension is not available for the Jira Cloud version. 

Custom Dimensions

If you do not find Xray custom field Test Environments in import options, you may import Xray custom field Test Environments using Advanced settings for custom fields. Use the following custom field definition example (replacing NNNNN with a corresponding field ID in Jira):

[jira.customfield_NNNNN]
data_type = "string"
dimension = true
multiple_values = true
name = "Xray Revision"
dimension_annotations = {group = "Xray"}

Measures

The following Xray specific measures will be imported. These measures can be analyzed by the mentioned Xray specific dimensions as well as by Issue dimension (where individual tests executions are registered as issues with "Test Execution" type and tests as issues with "Test" type) and by other standard dimension members which correspond to the issue. Note that, measures used in the report defines the context of standard dimensions. For example:

  • When using execution related measures, like, Xray Tests executed, Xray Test Runs, Xray Tests defect count, then standard dimension Status is applied to Xray Test Execution as execution is the core to gather information about executed tests, execution outcomes, fix versions, found defects, executors, etc.
  • When using test related measures, like, Xray Tests created, then standard dimension Status is related to Xray Test.
Xray Tests created

Count of issues with type "Test".

On the Time dimension, data are grouped by the issue creation date of the test. Used with Jira standard dimensions, like Project, Label, Status, Priority, etc., groups data by test issue fields.

Xray Test Executions created

Count of issues with type "Test Execution".

On the Time dimension, data are grouped by the test execution issue creation date. Used with Jira standard dimensions, like Project, Label, Status, Priority, etc., groups data by test execution issue fields.

Xray Requirements created  NEW

Count of requirement issues that match issue types selected in Xray configuration "Requirement Issue Types".

On the Time dimension, data are grouped by the issue creation date of the requirement issue. Used with Jira standard dimensions, like Project, Label, Status, Priority, etc., groups data by requirement issue fields.

Measure Xray requirements created available on Cloud and starting eazyBI version 6.6.

Xray Tests defect count

How many defects are registered for test execution (including individual test steps). A unique count of different defects is returned for each test run and aggregated at higher levels.

A defect should be linked through the Xray test execution screen. On the Time dimension, data are grouped by the test run date.

Xray Tests scheduled

How many tests are scheduled within selected dimensions but not yet executed (test has no dates for execution "Started" and "Finished"). If a test is scheduled for two separate test executions, then eazyBI count it as one test.

On the Time dimension, data are grouped by the date when test is added to the execution.

Xray Tests executing

How many tests are executing within selected dimensions (test has execution "Started" date and no "Finished" date). If a test is executing in two separate test executions, then eazyBI count it as one test.

On the Time dimension, data are grouped by the date when test execution (run) is started.

Xray Tests executed

How many tests are executed within selected dimensions (test has both execution dates "Started" and "Finished"). If a tests is executed in several test executions, then eazyBI count it as one test.

On the Time dimension, data are grouped by the date when test execution (run) is finished.

Xray Tests with executions

How many tests have at least one test execution (scheduled, in progress or completed) within selected dimensions. If a tests has several test executions, then eazyBI count it as one test.

On the Time dimension data are grouped by:

  • for completed test runs (has both execution dates "Started" and "Finished"), a date when a test run is Finished,
  • for in-progress test runs (has "Started" date and no "Finished" date), a date when a test run is Started,
  • for scheduled test runs (test run has no dates for execution "Started" and "Finished".), a date when a test is added to a test execution.
Xray Test Runs

How many times have tests been executed, are executing, or are scheduled for execution.

On the Time dimension data are grouped by:

  • for completed test runs (has both execution dates "Started" and "Finished"), a date when a test run is Finished,
  • for in-progress test runs (has  "Started" date and no "Finished" date), a date when a test run is Started,
  • for scheduled test runs (test run has no dates for execution "Started" and "Finished".), a date when a test is added to a test execution.
Xray Tests executed first date

Date and time for the first executed test within selected dimensions. 

  • If a test run has Finished date, then return the timestamp when it is finished.
  • If a test run does not have Finished date but has Started date, then return the timestamp when it is started.
Xray Tests executed last date

Date and time for the last executed test within selected dimensions. 

  • If a test run has Finished date, then return the timestamp when it is finished.
  • If a test run does not have Finished date but has Started date, then return the timestamp when it is started.
Xray average Test Run durationHow much time (hours and minutes) it took to execute a test on average.
Xray Tests execution summary

Test execution statuses within selected dimensions.

If a test has several test executions with different results, then eazyBI counts it as one for each resulting status.

Xray Total Test Run duration

Hidden measures that could be used in the calculated measures.

Total duration (hours and minutes) it took to run tests.

Properties

Also following properties will be created.

Xray Test last executed date

The last execution date for a test issue. It is issue property and value does not change according to selected Test Execution, Test Plan or other dimension members.

Used together with Xray Test dimension.

Xray Test last executed by

The last executor for a test issue. It is issue property and value does not change according to selected Test Execution, Test Plan or other dimension members.

Used together with Xray Test dimension.

Xray Test last execution status

The last execution status for a test issue. It is issue property and value does not change according to selected Test Execution, Test Plan or other dimension members.

Used together with Xray Test dimension.

Xray Test Execution Begin Date

Begin date of a test execution issue.

This is an execution property and should be used together with Xray Test Execution dimension.

Property is not available for Jira Cloud version. You can import Begin Date as issue property "Xray Begin date" and access it through the calculated measure.

Xray Test Execution End Date

End date of a test execution issue.

This is an execution property and should be used together with Xray Test Execution dimension.

Property is not available for Jira Cloud version. You can import End Date as issue property "Xray End Date" and access it through the calculated measure.

Xray Test Execution Revision

Current revision assigned to a tests execution issue.

This is an execution property and should be used together with Xray Test Execution dimension.

Property is not available for Jira Cloud version. You can import field as issue property "Xray Revision" and access it through the calculated measure. 

Xray Test Execution Test Environments

Current test environment assigned to a tests execution issue.

This is an execution property and should be used together with Xray Test Execution dimension.

Property is not available for Jira Cloud version. 

Xray Execution Test Plans

Key of test plan to which a test execution issue is assigned.

This is an execution property and should be used together with Xray Test Execution dimension.

Xray Test Test Plans

List of test plans in which test is executed or scheduled for execution.

This is a test property and should be used together with Xray Test dimension.

Xray Test Test Type

Current test type of a test issue.

This is a test property and should be used together with Xray Test dimension.

Property is not available for Jira Cloud version. 

Xray Test Cucumber Test Type

Current Cucumber tests type of a test issue if it has a test type "Cucumber".

This is a test property and should be used together with Xray Test dimension.

Property is not available for Jira Cloud version. 

Xray Test Plan Begin Date

Begin date of a test plan issue.

This is a test plan property and should be used together with Xray Test Plan dimension.

Property is not available for Jira Cloud version. You can import Begin Date as issue property "Xray Begin Date" and access it through the calculated measure.

Xray Test Plan End Date

End date of a tests plan issue.

This is a test plan property and should be used together with Xray Test Plan dimension.

Property is not available for Jira Cloud version. You can import End Date as issue property "Xray End Date" and access it through the calculated measure.

Xray Defect status

Current status of a defect issue.

This is a defect property and should be used together with the Xray Defect dimension.

Xray Defect priority NEW

Current priority of a defect issue.

This is a defect property and should be used together with the Xray Defect dimension.

The Xray Defect priority is available on the Jira Cloud and since eazyBI version 6.6 (soon to be released).

Xray Defect created

Creation date and time of a defect issue.

This is a defect property and should be used together with the Xray Defect dimension.

Xray Defect assignee

Current status of a defect issue.

This is a defect property and should be used together with the Xray Defect dimension.

Xray Requirement Testing Status NEW

Testing status of requirement issue based on Xray calculated custom field "Requirement Status".

This is a requirement issue property and should be used together with Xray Requriement dimension.

The Xray Requirement Testing Status is available since eazyBI version 6.6.

Property is not available for the Jira Cloud version. 

Xray Test Execution Overall Execution Status NEW

The aggregated execution status for the tests in the test execution based on Xray calculated custom field "Test Execution Status" (or "Overall Execution Status").

This is a test execution property and should be used together with the  Xray Test Execution dimension.

The Xray Test Execution Overall Execution Status is available since eazyBI version 6.6.

Property is not available for the Jira Cloud version. 

Xray Test Plan Overall Execution Status NEW

The aggregated execution status for the tests in the test plan based on Xray calculated custom field "Test Plan Status" (or "Overall Execution Status").

This is a test plan property and should be used together with the Xray Test Plan dimension.

The Xray Test Plan Overall Execution Status  is available since eazyBI version 6.6.

Property is not available for the Jira Cloud version. 

Xray Test Set Overall Execution Status NEW

The aggregated execution status for the tests in the test set based on Xray calculated custom field "Test Set Status" (or "Overall Execution Status").

This is a test plan property and should be used together with the Xray Test Set dimension.

The Xray Test Set Overall Execution Status is available since eazyBI version 6.6.

Property is not available for the Jira Cloud version. 

Issue properties

Xray tests, test executions, as well as defects, requirements, and test plans are Jira issues and are also represented in the Issue dimension. You can define calculated measure to get any issue property that is imported in eazyBI also for Xray-specific dimensions. To access issue property, use function get().

For example, to get priority for Xray Requirement, copy a formula of issue property Issue Priority and replace dimension name [Issue] with [Xray Requirement] like this:

[Priority].[Priority].getMemberNameByKey(
  [Xray Requirement].CurrentHierarchyMember.get('Priority ID')
)

Another example, to get Begin Date date for Xray Test Plan, copy a formula of the issue property Issue Xray Begin Date and replace dimension name [Issue] with [Xray Test Plan] like this:

[Xray Test Plan].CurrentHierarchyMember.get('Xray Begin Date')

Properties representing Xray calculated fields

Xray custom fields "Requirement Status", "Test Execution Status", "Test Plan Status", "Test Set Status" are Xray-calculated fields, and their value changes according to related test execution results.

During scheduled data import or manual import, eazyBI refresh data only for issues that were updated since the previous data import. But in the case of Xray calculated fields, it might be the execution status is recalculated but the updated date for the issue itself is not changed. As the result, the re-calculated value is not updated in eazyBI. If you would like to check on the latest value for the Xray-calculated field "Requirement Status", "Test Execution Status", "Test Plan Status", or "Test Set Status" on every import you may add the parameter check_calculated_value = true to the custom field settings.

Sample reports

If you selected the Import sample reports option, then additional sample reports are created in the Samples Xray folder. Please try these reports to see which measures would be useful for your specific needs. If you would like to customize some reports, then save them and change the name (as the default sample reports are overwritten during each Jira import). 

See also detailed description on how to build non-standard reports: