Xray Test Management
eazyBI for Jira
Xray for Jira is a complete, full-featured Atlassian-verified test management add-on for Jira. Xray supports both manual and automated tests and a complete testing life cycle: test planning, test designing, and test execution to help developers improve the quality of their systems.
Xray for Jira is an add-on developed by Xblend, an Atlassian Platinum Marketplace Partner.
On this page:
Xray Data Import
Data Import in Data Center
If you have installed Xray Test Management for Jira then in Jira import options, section Add-ons, you will see an options Xray. Select the checkbox to import Xray data and after the next data import, Xray-specific measures and dimensions will be available for your reports. You can also check or uncheck some of Xray-specific custom fields for data import.
Data Import in Cloud version
eazyBI uses API keys to get data from Xray Cloud.
- In Jira Cloud, go to section Apps → Xray -> API Keys and create API key. For the API key, select a user with Jira administrator rights.
- In eazyBI, Jira import options section Add-ons you will see an option Xray which is available when you have installed Xray Test Management app for Jira. Enter generated Client Id and Client secret.
After the next data import, Xray specific measures and dimensions will be available for your reports.
Limit imported test runs
To reduce import time and the testing data in the cube, you can import only the most recent test runs.
Set a relative time limit for test runs like 3 months to import test runs from the last three months or a particular date like Jun 14 2020 to set a date range from that date until now. Test runs are imported if they are created (test is added to the test execution) within the time limit (for scheduled test executions) or with started on date within the time limit (for completed or in progress test runs). If there is no limit set, then all test runs are imported.
You can set a limit to all eazyBI accounts through advanced settings.
[jira.xray] test_executions_limit = "3 months"
Another option is setting the limit for an individual eazyBI account. In the import options, add the period in the field "Imported Test Run limit".
If import limits are applied for the whole eazyBI instance using advanced settings, then the limit in import options won't exceed the limit in advanced settings. In such a way, the overall limits can be set on the eazyBI account, and, in an individual account, this time can be reduced more.
Dimensions
After the import, the following Xray specific dimension will be added:
Xray Requirement | Requirement issues with issue types selected in Xray configuration "Requirement Issue Types"; for example, story, feature, etc. The dimension is organized into Project and Requirement levels. In addition, you can add Issue hierarchy to group requirement issues; this option is available on Cloud and starting the version 7.1. Use this dimension with the measure Xray Requirements created count to see all covered and not covered requirements. |
|---|---|
Xray Test Plan | Test plan issues that have assigned at least one test or executions with tests. The dimension is organized by default hierarchy (in Project and Plan levels) or by Test Plan boar hierarchy (in Project, Plan, Folder, and Subfolder). In addition, you can add add Issue hierarchy to group Test Plan issues; this option is available on Cloud and starting the version 7.1. |
Xray Test | Tests issues; all created issues with Xray-specific issue type "Test". The dimension is organized by default hierarchy (in Project and Test levels) or by Test repository hierarchy (in Project, Folder, Subfolder, and Test). In addition, you can add Issue hierarchy to group Test issues; this option is available on Cloud and starting the version 7.1. |
Xray Test Set | Test set issues that have at least one test. This dimension is organized into Project and Set levels. In additiona, you can add Issue hierarchy to group Test Set issues; this option is available on Cloud and starting the version 7.1. |
Xray Test Pre-condition | Test pre-condition issues. This dimension is organized into Project and Pre-condition levels. |
Xray Test Execution | Test execution issues that have assigned at least one test. This dimension is organized into Project and Execution levels. In addition, you can add Issue hierarchy to group Test Execution issues; this option is available on Cloud and starting the version 7.1. |
Xray Defect | Defect issues that are found during testing and linked through a test execution screen. This dimension is organized in Project and Defect levels. In addition, you can add Issue hierarchy to group defect issues; this option is available on Cloud and starting the version 7.1. The Xray Defect dimension works well with execution-related measures, for example, Xray Test Run, Xray defect count. |
Xray Defect Priority | Current defect priority of defects that are found during testing and linked through a test execution screen. This dimension is organized in Project and Defect levels. The Xray Defect Priority dimension works well with execution-related measures, for example, Xray Test Run, Xray defect count. |
Xray Defect Status | Current defect statuses of defects that are found during testing and linked through a test execution screen. This dimension has a secondary hierarchy that groups statuses by Category. The Xray Defect Status dimension works well with execution-related measures, for example, Xray Test Run, Xray defect count. |
Xray Test Run Assignee | Users who are assigned to test runs. |
Xray Test Executed by | Users who executed tests. |
Xray Test Run Status | Test execution statuses – by default, there are TODO, PASS, EXECUTING, FAIL, and ABORTED statuses, but also there can be other user-defined statuses. The Xray Test Run Status dimension works well with execution-related measures, for example, Xray Test Run. The former dimension name in old versions was Xray Test Execution Status. |
Requirement Status | Testing status of requirement issue (feature, story, etc.) based on Xray calculated custom field "Requirement status". If you change the configuration for calculating requirement status in Jira, ensure to re-import data in eazyBI to get and refresh all updated values. |
TestRunStatus | Testing status of Test issue based on Xray calculated custom field "Requirement status". |
Xray Pre-condition Type | Types of pre-conditions - Manual, Cucumber. |
Xray Test Type | Types of tests (Manual, Cucumber, Generic, etc.). The dimension is organized into Type and Scenario Type levels. |
Xray Test Execution Test EnvironmentTest Environments | Test environment where tests are executed or scheduled for execution. The dimension name on Cloud is Xray Test Execution Test Environment. The dimension name on the Data Center matches the issue custom field name Test Environment. |
Revision | System revision for test execution. Field is not available for Jira Cloud version. You can import the custom field Revision as issue property "Revision" and access it through the calculated measure. |
Test field dimensions
The option to import Test field dimensions is available on Cloud and since eazyBI version 7.2.
Dimensions represent Test Execution issue fields when used with Test execution measures like Xray Tests executed, Xray Test Runs, Xray Tests defect count, etc. For example, the "Label" dimension with the "Xray Test Runs" measure would show the count of test runs by labels associated with Test Execution issues.
You can import new dimensions to analyze test execution results by Test issue fields. In the "Import additional Test dimensions" field list the Test issue fields you want to use for the test execution reports. In the list are available issue standard fields and singe-select custom fields you have imported in the data cube. During the next data import, eazyBI will create a new dimension with the prefix Xray Tests. For example, specify Label for import as additional Test dimension and eazyBI creates "Xray Test label" dimension you can use with Xray specific measures and group test executions results by labels assigned to Tests.
Custom dimensions
If you do not find Xray custom field Test Environments in import options, you may import Xray custom field Test Environments using Advanced settings for custom fields. Use the following custom field definition example (replacing NNNNN with a corresponding field ID in Jira):
[jira.customfield_NNNNN]
data_type = "string"
dimension = true
multiple_values = true
name = "Xray Revision"
dimension_annotations = {group = "Xray"}
Measures
The following Xray specific measures will be imported. These measures can be analyzed by the mentioned Xray-specific dimensions as well as by Issue dimension (where individual test executions are registered as issues with "Test Execution" type and tests as issues with "Test" type) and by other standard dimension members that correspond to the issue. Note that measures used in the report define the context of standard dimensions. For example:
- When using execution related measures, like, Xray Tests executed, Xray Test Runs, Xray Tests defect count, then standard dimension Status is applied to Xray Test Execution as execution is the core to gather information about executed tests, execution outcomes, fix versions, found defects, executors, etc.
- When using test related measures, like, Xray Tests created, then standard dimension Status is related to Xray Test.
- When using requirement related measures, like, Xray Requirements created, then standard dimension Status is related to the requirement issue (story, bugs, or others you have marked as requirements).
Xray Tests created | Count of issues with type "Test". On the Time dimension, data are grouped by the issue creation date of the test. Used with Jira standard dimensions, like Project, Label, Status, Priority, etc., groups data by test issue fields. |
|---|---|
Xray Test Executions created | Count of issues with type "Test Execution". On the Time dimension, data are grouped by the test execution issue creation date. Used with Jira standard dimensions, like Project, Label, Status, Priority, etc., groups data by test execution issue fields. |
Xray Requirements created | Count of requirement issues that match issue types selected in Xray configuration "Requirement Issue Types". On the Time dimension, data are grouped by the issue creation date of the requirement issue. Used with Jira standard dimensions, like Project, Label, Status, Priority, etc., groups data by requirement issue fields. |
Xray Tests defect count | How many defects are registered for test execution (including individual test steps). A unique count of different defects is returned for each test run and aggregated at higher levels. A defect should be linked through the Xray test execution screen. On the Time dimension, data are grouped by the test run date. |
Xray Tests scheduled | How many tests are scheduled within selected dimensions but not yet executed (test has no dates for execution "Started" and "Finished"). If a test is scheduled for two separate test executions, then eazyBI count it as one test. On the Time dimension, data are grouped by the date when test is added to the execution. |
Xray Tests executing | How many tests are executing within selected dimensions (test has execution "Started" date and no "Finished" date). If a test is executing in two separate test executions, then eazyBI count it as one test. On the Time dimension, data are grouped by the date when test execution (run) is started. |
Xray Tests executed | How many tests are executed within selected dimensions (test has both execution dates "Started" and "Finished"). If a tests is executed in several test executions, then eazyBI count it as one test. On the Time dimension, data are grouped by the date when test execution (run) is finished. |
Xray Tests with executions | How many tests have at least one test execution (scheduled, in progress or completed) within selected dimensions. If a tests has several test executions, then eazyBI count it as one test. On the Time dimension data are grouped by:
|
Xray Test Runs | How many times have tests been executed, are executing, or are scheduled for execution. On the Time dimension data are grouped by:
|
Xray Test Runs created first date NEW | Date and time when a Test was assigned to Test Execution for the first time. New measures Xray Test Runs created first/last date are available on Jira Cloud and starting from eazyBI version 7.2. To get those measures add eazyBI advanced settings and measures will be created automatically during the next data import for accounts with Xray data. Note the first data import after changes in advanced settings will be full Xray data reimport; if you have a significant testing data amount, consider adding the settings in the evening or before the weekend. [jira.xray] import_test_runs_created_timestamp = true |
Xray Test Runs created last date NEW | Date and time when a Test was assigned to Test Execution for the last time. New measures Xray Test Runs created first/last date are available on Jira Cloud and starting from eazyBI version 7.2. To get those measures add eazyBI advanced settings and measures will be created automatically during the next data import for accounts with Xray data. Note the first data import after changes in advanced settings will be full Xray data reimport; if you have a significant testing data amount, consider adding the settings in the evening or before the weekend. [jira.xray] import_test_runs_created_timestamp = true |
Xray Tests executed first date | Date and time for the first executed test within selected dimensions.
|
Xray Tests executed last date | Date and time for the last executed test within selected dimensions.
|
Xray average Test Run duration | How much time it took to execute a test on average. The values are represented in huors and minutes but the measure accuracy is up to seconds. The duration is based on the test run fields "Started On" and "Finished On". |
Xray Tests execution summary | Test execution statuses within selected dimensions. If a test has several test executions with different results, then eazyBI counts it as one for each resulting status. |
Xray Total Test Run duration | Hidden measures that could be used in the calculated measures. Total duration (hours and minutes) it took to run tests. The duration is based on the test run fields "Started On" and "Finished On" and accuracy is up to one minute. |
Properties
Also, the following properties will be created.
Xray Test last executed date | The last execution date for a test issue. It is issue property and value does not change according to selected Test Execution, Test Plan or other dimension members. Used together with Xray Test dimension. |
|---|---|
Xray Test last executed by | The last executor for a test issue. It is issue property and value does not change according to selected Test Execution, Test Plan or other dimension members. Used together with Xray Test dimension. |
Xray Test last execution status | The last execution status for a test issue. It is issue property and value does not change according to selected Test Execution, Test Plan or other dimension members. Used together with Xray Test dimension. |
Xray Test Execution Begin Date | Begin date of a test execution issue. This is an execution property and should be used together with Xray Test Execution dimension. Property is not available for Jira Cloud version. You can import Begin Date as issue property "Xray Begin date" and access it through the calculated measure. |
Xray Test Execution End Date | End date of a test execution issue. This is an execution property and should be used together with Xray Test Execution dimension. Property is not available for Jira Cloud version. You can import End Date as issue property "Xray End Date" and access it through the calculated measure. |
Xray Test Execution Revision | Current revision assigned to a tests execution issue. This is an execution property and should be used together with Xray Test Execution dimension. Property is not available for Jira Cloud version. You can import field as issue property "Xray Revision" and access it through the calculated measure. |
Xray Test Execution Test Environments | Current test environment assigned to a tests execution issue. This is an execution property and should be used together with Xray Test Execution dimension. Property is not available for Jira Cloud version. |
Xray Execution Test Plans | Key of test plan to which a test execution issue is assigned. This is an execution property and should be used together with Xray Test Execution dimension. |
Xray Test Test Plans | List of test plans in which test is executed or scheduled for execution. This is a test property and should be used together with Xray Test dimension. |
Xray Test Test Type | Current test type of a test issue. This is a test property and should be used together with Xray Test dimension. Property is not available for Jira Cloud version. |
Xray Test Cucumber Test Type | Current Cucumber tests type of a test issue if it has a test type "Cucumber". This is a test property and should be used together with Xray Test dimension. Property is not available for Jira Cloud version. |
Xray Test Plan Begin Date | Begin date of a test plan issue. This is a test plan property and should be used together with Xray Test Plan dimension. Property is not available for Jira Cloud version. You can import the custom field Begin Date as issue property "Xray Begin Date" and access it through the calculated measure. |
Xray Test Plan End Date | End date of a tests plan issue. This is a test plan property and should be used together with Xray Test Plan dimension. Property is not available for Jira Cloud version. You can import the custom field End Date as issue property "Xray End Date" and access it through the calculated measure. |
Xray Defect status | Current status of a defect issue. This is a defect property and should be used together with the Xray Defect dimension. |
Xray Defect priority | Current priority of a defect issue. This is a defect property and should be used together with the Xray Defect dimension. |
Xray Defect created | Creation date and time of a defect issue. This is a defect property and should be used together with the Xray Defect dimension. |
Xray Defect assignee | Current status of a defect issue. This is a defect property and should be used together with the Xray Defect dimension. |
Xray Requirement Testing Status | Testing status of requirement issue based on Xray calculated custom field "Requirement Status". This is a requirement issue property and should be used together with Xray Requriement dimension. Property is not available for Jira Cloud version. |
Xray Test Execution Overall Execution Status | The aggregated execution status for the tests in the test execution based on Xray calculated custom field "Test Execution Status" (or "Overall Execution Status"). This is a test execution property and should be used together with the Xray Test Execution dimension. Property is not available for Jira Cloud version. |
Xray Test Plan Overall Execution Status | The aggregated execution status for the tests in the test plan based on Xray calculated custom field "Test Plan Status" (or "Overall Execution Status"). This is a test plan property and should be used together with the Xray Test Plan dimension. Property is not available for Jira Cloud version. |
Xray Test Set Overall Execution Status | The aggregated execution status for the tests in the test set based on Xray calculated custom field "Test Set Status" (or "Overall Execution Status"). This is a test plan property and should be used together with the Xray Test Set dimension. Property is not available for Jira Cloud version. |
Issue properties
Xray tests, test executions, as well as defects, requirements, and test plans are Jira issues and are also represented in the Issue dimension. You can define a calculated measure to get any issue property that is imported in eazyBI also for Xray-specific dimensions. To access the issue property, use function get().
For example, to get priority for Xray Requirement, copy a formula of issue property Issue Priority and replace dimension name [Issue] with [Xray Requirement] like this:
[Priority].[Priority].getMemberNameByKey(
[Xray Requirement].CurrentHierarchyMember.get('Priority ID')
)
Another example, to get Begin Date date for Xray Test Plan, copy a formula of the issue property Issue Xray Begin Date and replace dimension name [Issue] with [Xray Test Plan] like this:
[Xray Test Plan].CurrentHierarchyMember.get('Xray Begin Date')
Add Issue hierarchy
The option to add issue hierarchy to Defect, Requirement, Test, Test Execution, and Test Plan dimensions is available on Cloud and since eazyBI version 7.1.
You can group issues representing testing entities by Epic hierarchy or any other hierarchy you have built for the Issue dimension.
Additional hierarchies for Xray dimensions are defined in eazyBI advanced settings (or site settings for Cloud) and are marked with the setting [[jira.xray.custom_hierarchies]]. Provide the dimension name for which to enable the hierarchy, the hierarchy name, and the top-level member name. List all hierarchy levels where the bottom level is a requirement/defect issue (sub-task level is not supported).
For example, enable Epic hierarchy for the "Xray Defect" dimension with three hierarchy levels: project of defect issue, Epic related to defect issue, and defect issue.
[[jira.xray.custom_hierarchies]]
dimension = "Xray Defect"
hierarchy = "Epic"
all_member_name = "All Defects by Epics"
levels = [
{name="Project",key_column = "project_key"},
{name="Epic",key_column="epic_key"},
{name="Defect"}
]
If you already have constructed an additional Issue hierarchy, you can enable it also for Xray dimensions. Suppose you have a Features by Stories hierarchy as described in this example Separate hierarchy level for Story between parent and epic. Specify the dimension name for which to make the hierarchy, the hierarchy name, and the top-level member name. The levels should match the additional issue hierarchy description except for the Sub-task level and no parameters for the lowest (requirement or defect) level.
[[jira.xray.issue_hierarchies]]
dimension = "Xray Requirement"
hierarchy = "Features by Stories"
all_member_name = "All Issues by Features"
levels = [
{name="Feature",key_column="customfield_featuree",issue_type="Feature"},
{name="Epic and risk",key_column="customfield_epice",issue_type="Epic"},
{name="Story",key_column="customfield_storye",issue_type="Story"},
{name="Requirement"},
]
You can enable the Plans hierarchy as well. In that case, list all hierarchy levels and parent keys matching the hierarchy configuration in Jira. The example below describes the Plans hierarchy for Initiative → Epic → Story and enables it to "Xray Requirement" dimension.
[[jira.xray.custom_hierarchies]]
dimension = "Xray Requirement"
hierarchy = "Plans"
all_member_name = "All requirements by Initiatives"
levels = [
{name="Initiative",key_column = "jpoh_parent_3"},
{name="Epic",key_column="jpoh_parent_2"},
{name="Story"}
]
Sample reports
If you selected the Import sample reports option, then additional sample reports are created in the Samples Xray folder. Please try these reports to see which measures would be useful for your specific needs. If you would like to customize some reports, then save them and change the name (as the default sample reports are overwritten during each Jira import).
See also detailed descriptions on how to build non-standard reports:
- Xray Requirement coverage report
- Xray Test burnup for Test Plan report