If you have prepared your tests with MSTest v2, NUnit, and xUnit frameworks with Visual Studio version 15.9 or newer, you can associate your tests with the test cases you have prepared in Azure DevOps Test Plan.
So if we have prepared our test as a Java project with Junit or TestNG, how will we integrate the test results with the test cases in the test plan? The first information I found in my research to find a solution for this was the information in Microsoft’s own documentation that if you are not using MSTest v2, NUnit, and xUnit, you cannot associate test cases with your automation tests. “What types of tests are supported?”
Although at first, I thought about moving the project to .Net, then I decided to do some more research and found the API documentation of Azure DevOps “Test REST API for Azure DevOps". By making use of these documents, I was able to unilaterally associate test cases that are in the Java projects and test cases in the automation project. By one-sided I mean; After the tests are run on the project side, they can update the execute status of the test cases associated with the Azure Test Plan (Passed, Failed, etc.). But you cannot trigger test automation through the test plan. I think even this problem will be solved in the future. Maybe there is a solution but I couldn’t find it.
Let’s come to the most important part, how can we establish this relationship.
First of all, we need an access token that you will create via your Azur account so that you can update it with the test plan API service. Use personal access tokens
Unfortunately, there is no endpoint in Azure rest API where we can give the test case id and get all information about the test case. Although, you can only pull case information as a work item with id. However, “test point” information is not returned in the response, and we will need this information when establishing the relationship between the run test and the test case. Click the below link to learn what the test points are
I proceeded with the test plan and test suite structure in the test plan. Within the scope of the test plan, there is a separate test suite for each service in the project. This way each class represents an API service. To summarize, every suite in Azure Test Plans will represent an API service, and every class in the test project will also represent an API service. So I assigned test suite IDs to each class with an annotation I created.
I assigned test case ids with a separate annotation before each test method.
Now I have the knowledge of which cases in which suites are running during the test is running. Now let’s get to the main part. First of all, I checked whether the test case IDs specified at the beginning of the test methods really exist in that suite. To be able to do this check, I get the suite id and azure DevOps suite information and the test case information in it and performed a check before each test run to see if they match the ids specified in the annotation.
You can get the all test cases id by giving the suit id by the below request.
We will get the id and name from the workItem object, System.Rev from workItemFields and id from pointAssignments to use when associating test results and test cases from the returned response. Now we have the id, name, testCaseRevision, and testPoint information of all the cases under the suite.
First, we can check whether the test case id specified in the test method really exists under this suite. If I don’t I’m throwing an error. We need to keep this information we receive on a class basis. I attitude these results in a HasMap object. Thus, when I give the class name, I can access the test case information of the suite specified in that class when necessary.
We need to create a test run in Azure Test Plans before all tests start so that we can assign the results when the test is completed. When creating my test run, I override the BeforeAllCallback beforeAll method because I need the class information on which the test is running. Although, this could be done directly with AfterAll or TestInfo
So how do we create the test run? Here is the answer below;
Post: https://dev.azure.com/{organization}/{project}/_apis/test/runs?api-version=6.0
In the name part, I gave the test class name I got from the ExtensionContext and a run id number that I created. Then, to be able to send the test results into this test run and update its status when all the tests are completed, we need to keep the id and URL information of the run as a result of the test again based on the test class. I used the same HashMap object for this. You can see this request under Test Plan > Runs as soon as you submit it.
When each test method is completed, we will now send the results into the test run we have created. For this, we need to know the test result, if you want, you can use the testDisabled, testSuccessful, testAborted, testFailed methods in the TestWatcher class. I used TestExecutionExceptionHandler in order not to handle each method separately. If TestExecutionExceptionHandler contains an exception, the test result will be “Failed” otherwise “Pass”.
POST https://dev.azure.com/{organization}/{project}/_apis/test/Runs/{runId}/results?api-version=6.0
We have all the information to post this, all we have to do is post them. When you post this information, you will be able to see this information in the test results tab in the Azure Test Plans test run, and at the same time, the test case execution status will be updated.
PATCH https://dev.azure.com/{organization}/{project}/_apis/test/runs/{runId}?api-version=6.0
Yes, now the results of the tests running in our test automation project are associated with the Azure DevOps Test Plan and will be updated with each run. As a result, even if we develop a test project in a different language other than .Net, we will be able to use the reports and graphics in the Test Plan in a healthy way.
I hope it was useful sharing, have a good day.
Click for the project sample