The Brand-new Smart XPath Generator from Katalon Studio
Tags: xpath, katalonstudio
Published: 2018-09-10 10:51:28

---
title: The Brand-new Smart XPath Generator from Katalon Studio
published: true
description: This article will introduce a brand-new XPath generator & intelligent method in Katalon Studio to automatically generate robust locators during the test recording phase.
tags: XPath, Katalon Studio
---
 ![The Brand-new Smart XPath Generator from Katalon Studio](https://d1h3p5fzmizjvp.cloudfront.net/wp-content/uploads/2018/09/06095757/Smart-XPath-generator1.png)
<h2><strong>Introduction</strong></h2>
In Web UI automated testing, a primary concern for testers is high maintenance costs of test scripts, which are usually caused by unstable and broken locators. A locator is a mechanism used by Selenium to identify an element during test execution. The popular locator mechanism is XPath. Unfortunately, locators are easily broken or unable to identify the target element when the application under test (AUT) changes. <a href="https://ieeexplore.ieee.org/document/7515470/" target="_blank" rel="noopener">An existing study</a> showed that locators account for more than 70% of breakages in Web UI test scripts.

Hence creating and using correct and resilient locators are crucial to the success of Web UI test automation.

In this article, we describe a brand-new and intelligent method introduced in Katalon Studio to automatically generate robust locators during the test recording phase. This advanced method produces new locators that are more maintainable and less vulnerable to changes in AUT than do Selenium and other test automation tools.
<h2><strong>Cases where common locators are inadequate</strong></h2>
Let’s look at ted.com, a website built using the EmberJS framework. The screenshot below shows the website’s cards with each contain different items (text, image, link, etc.).

<img class="init-size alignnone wp-image-5094 size-full" src="https://d1h3p5fzmizjvp.cloudfront.net/wp-content/uploads/2018/09/05140247/common-locators-are-inadequate.png" alt="common locators are inadequate" width="975" height="458" />

Suppose you want to record a simple action of clicking on a thumbnail of a video, the one with “How AI Can save our humanity”, for example, you would get the following XPaths generated by Selenium:
<table style="width: 762px; border: 1px solid black; height: 50px;">
<tbody>
<tr style="height: 9.90625px; border: 1px solid black;">
<td style="width: 752px; border: 1px solid black; height: 9.90625px;">
<p style="text-align: center;"><strong>Selenium’s XPaths</strong></p>
</td>
</tr>
<tr style="height: 24px; border: 1px solid black;">
<td style="width: 752px; height: 24px; border: 1px solid black;">//div[@id='react-container1']/div/div/div/div/div/div[2]/div/div[2]/div/div/div/a/div/div[2]</td>
</tr>
<tr style="height: 24px; border: 1px solid black;">
<td style="width: 752px; height: 24px; border: 1px solid black;">//div[2]/div/div[2]/div/div/div/a/div/div[2]</td>
</tr>
</tbody>
</table>
Although these XPaths can locate the element, they are too long and complex. But the main problem is that they are easily broken as they only work if the content is not changed. TED is a fast-growing website with new contents being produced every day. Therefore, it is no surprise that the first video today may be the third video tomorrow. If you look at these XPaths, or just at the last 3 characters, you will notice that they rely entirely on the assumption that the ordering of the video will not change. Such an assumption contradicts reality, as described. In short, these locators are fragile and hard to read and maintain.

Take another example, Microsoft profile on Linkedin.

<img class="init-size alignnone wp-image-5096 size-full" src="https://d1h3p5fzmizjvp.cloudfront.net/wp-content/uploads/2018/09/05140753/Microsoft-profile-on-Linkedin.png" alt="Microsoft-profile-on-Linkedin" width="975" height="608" />

Suppose you want to verify Microsoft’s profile information, and you use Selenium to record by clicking on each of the fields to obtain the XPaths, here’s what you will get:
<table style="width: 718px; border: 1px solid black;">
<tbody>
<tr>
<td style="width: 189px; border: 1px solid black; text-align: center;"><strong>Fields</strong></td>
<td style="width: 519px; border: 1px solid black; text-align: center;"><strong>Selenium’s XPaths</strong></td>
</tr>
<tr>
<td style="width: 189px; border: 1px solid black;">Website</td>
<td style="width: 519px; border: 1px solid black;">id=ember1801</td>
</tr>
<tr>
<td style="width: 189px; border: 1px solid black;">Headquarters</td>
<td style="width: 519px; border: 1px solid black;">//div[@id='org-about-company-module']/div/div[2]/div/p</td>
</tr>
<tr>
<td style="width: 189px; border: 1px solid black;">Year founded</td>
<td style="width: 519px; border: 1px solid black;">//div[@id='org-about-company-module']/div/div[2]/div/p[2]</td>
</tr>
<tr>
<td style="width: 189px; border: 1px solid black;">Company Type</td>
<td style="width: 519px; border: 1px solid black;">//div[@id='org-about-company-module']/div/div[2]/div/p[3]</td>
</tr>
<tr>
<td style="width: 189px; border: 1px solid black;">Company Size</td>
<td style="width: 519px; border: 1px solid black;">//div[@id='org-about-company-module']/div/div[2]/div/p[4]</td>
</tr>
<tr>
<td style="width: 189px; border: 1px solid black;">Specialties</td>
<td style="width: 519px; border: 1px solid black;">//div[@id='org-about-company-module']/div/div[2]/div/p[5]</td>
</tr>
</tbody>
</table>
Let’s examine the table above row by row. The website field is located by its ID, which seems to be a reasonable choice. You now can say that at least your test case will successfully locate this field. However, you will later find out that it won’t because the ID is different every time you visit this page. In other words, this ID is dynamic. So using ID as a locator is not a good choice in this situation.

The headquarters field is located by a relatively concise XPath. Unfortunately, this locator identifies five Web elements instead of a unique element as expected. (You can check this by opening Chrome and pressing Ctrl + Shift + I to open the Inspection Tool and search the XPath in the document - shown below).

<img class="init-size alignnone wp-image-5097" src="https://d1h3p5fzmizjvp.cloudfront.net/wp-content/uploads/2018/09/05140944/Inspection-Tool-and-search-the-XPath-in-the-document-300x220.png" alt="Inspection-Tool-and-search-the-XPath-in-the-document" width="600" height="439" />

The result is not surprising because indeed there are five Web elements with the tag &lt;<strong><em>p&gt;</em> </strong>in the company information section. Some playback engine will automatically select the first element among those five, but some will not.

The remaining XPaths suffer the same problems we already discussed with the TED website.
<h2><strong>Neighbor XPath: An adequate form of XPaths for the above situations</strong></h2>
If an element cannot be consistently located using its direct attributes, then identifying it using its more robust neighbors. That is a simple but powerful idea behind the new locator method introduced in Katalon Studio. This method is visually intuitive as it reflects the way users often identify a visible element on the user interface.

An XPath (also called neighbor XPath) generated by the method follows the following form:
<table style="width: 762px; border: 1px solid black;">
<tbody>
<tr>
<td style="width: 292px; border: 1px solid black; text-align: center;">1</td>
<td style="width: 76px; border: 1px solid black; text-align: center;">2</td>
<td style="width: 130px; border: 1px solid black; text-align: center;">3</td>
<td style="width: 102px; border: 1px solid black; text-align: center;">4</td>
<td style="width: 142px; border: 1px solid black; text-align: center;">5</td>
</tr>
<tr>
<td style="width: 292px; border: 1px solid black; text-align: center;" rowspan="2">&nbsp;

xpath=(.//*[normalize-space(text()) and normalize-space(.)='<strong>neighbor_text</strong>'])</td>
<td style="width: 76px; text-align: center;" rowspan="2">&nbsp;

[n]/</td>
<td style="width: 130px; border: 1px solid black; text-align: center;">preceding::</td>
<td style="width: 102px; border: 1px solid black; text-align: center;" rowspan="2">&nbsp;

tag_name</td>
<td style="width: 142px; border: 1px solid black; text-align: center;" rowspan="2">&nbsp;

[m]</td>
</tr>
<tr>
<td style="width: 130px; border: 1px solid black; text-align: center;">following::</td>
</tr>
</tbody>
</table>
This form assumes that a neighbor of the target element is robust and is identified by its text. Each column in this form is explained as follows:
<table style="width: 595px; border: 1px solid black;">
<tbody>
<tr>
<td style="width: 102px; border: 1px solid black; text-align: center;"><strong>Column No.</strong></td>
<td style="width: 483px; border: 1px solid black; text-align: center;"><strong>Discrete description</strong></td>
</tr>
<tr>
<td style="width: 102px; border: 1px solid black; text-align: center;">1</td>
<td style="width: 483px; border: 1px solid black;">Retrieve the element using the text of its neighbor</td>
</tr>
<tr>
<td style="width: 102px; border: 1px solid black; text-align: center;">2</td>
<td style="width: 483px; border: 1px solid black;">Retrieve the neighbor among those retrieved in step 1 using the neighbor’s index.</td>
</tr>
<tr>
<td style="width: 102px; border: 1px solid black; text-align: center;">3</td>
<td style="width: 483px; border: 1px solid black;">From the neighbor, going forward or backward</td>
</tr>
<tr>
<td style="width: 102px; border: 1px solid black; text-align: center;">4</td>
<td style="width: 483px; border: 1px solid black;">Retrieve all elements with the same tag name as the target element, then</td>
</tr>
<tr>
<td style="width: 102px; border: 1px solid black; text-align: center;">5</td>
<td style="width: 483px; border: 1px solid black;">Retrieve the target element</td>
</tr>
</tbody>
</table>
Let’s get back to the example with TED:

<img class="init-size alignnone wp-image-5098 size-full" src="https://d1h3p5fzmizjvp.cloudfront.net/wp-content/uploads/2018/09/05141315/the-example-with-TED.png" alt="the example with TED" width="975" height="458" />

Here are a few XPaths that take on the above form to identify the thumbnail preceding “How AI can save our humanity”.
<table style="width: 754px; border: 1px solid black;">
<tbody>
<tr>
<td style="width: 752px; border: 1px solid black; text-align: center;"><strong>Neighbor XPath</strong></td>
</tr>
<tr>
<td style="width: 752px; border: 1px solid black;">xpath=(.//*[normalize-space(text()) and normalize-space(.)='How AI can save our humanity'])[2]/preceding::div[3]</td>
</tr>
<tr>
<td style="width: 752px; border: 1px solid black;">xpath=(.//*[normalize-space(text()) and normalize-space(.)='Newest Talks'])[1]/following::div[11]</td>
</tr>
</tbody>
</table>
The <em>“xpath=(.//*[normalize-space(text()) and normalize-space(.)”</em> part is fixed, so we can ignore it when maintaining. We instead focus on the rest, which refers to the labels shown in the UI and so it is readable.

The first XPath is the most intuitive because it aligns with how users process and interact with the thumbnail - the label representing the thumbnail. This locator is easy to maintain as when the label on the UI changes, you just have to update the corresponding text in the script. Moreover, the overall template of a video is often more stable than the content (for example YouTube hasn’t changed it templates for years). Therefore the relationship between the target element and its neighbor within a composite item or a card is preserved despite the dynamic orderings <em>between items. </em>

For the Microsoft profile example:

<img class="init-size alignnone wp-image-5100 size-full" src="https://d1h3p5fzmizjvp.cloudfront.net/wp-content/uploads/2018/09/05141613/Microsoft-profile-example-2.png" alt="Microsoft profile example" width="975" height="608" />

Here are the corresponding neighbor XPaths:
<table style="width: 754px; border: 1px solid black;">
<tbody>
<tr>
<td style="width: 148px; text-align: center; border: 1px solid black;"><strong>Fields</strong></td>
<td style="width: 598px; text-align: center; border: 1px solid black;"><strong>Neighbor XPaths</strong></td>
</tr>
<tr>
<td style="width: 148px; border: 1px solid black;">Website</td>
<td style="width: 598px; border: 1px solid black;">xpath=(.//*[normalize-space(text()) and normalize-space(.)='Website'])[1]/following::a[1]</td>
</tr>
<tr>
<td style="width: 148px; border: 1px solid black;">Headquarters</td>
<td style="width: 598px; border: 1px solid black;">xpath=(.//*[normalize-space(text()) and normalize-space(.)='Headquarters'])[1]/following::p[1]</td>
</tr>
<tr>
<td style="width: 148px; border: 1px solid black;">Year founded</td>
<td style="width: 598px; border: 1px solid black;">xpath=(.//*[normalize-space(text()) and normalize-space(.)='Year founded'])[1]/following::p[1]</td>
</tr>
<tr>
<td style="width: 148px; border: 1px solid black;">Company Type</td>
<td style="width: 598px; border: 1px solid black;">xpath=(.//*[normalize-space(text()) and normalize-space(.)='Company type'])[1]/following::p[1]</td>
</tr>
<tr>
<td style="width: 148px; border: 1px solid black;">Company Size</td>
<td style="width: 598px; border: 1px solid black;">xpath=(.//*[normalize-space(text()) and normalize-space(.)='Company size'])[1]/following::p[1]</td>
</tr>
<tr>
<td style="width: 148px; border: 1px solid black;">Specialties</td>
<td style="width: 598px; border: 1px solid black;">xpath=(.//*[normalize-space(text()) and normalize-space(.)=’Specialties’])[1]/following::p[1]</td>
</tr>
</tbody>
</table>
Again, ignoring the fixed part of the XPaths above, we can see that the fields of Microsoft’s profile are embedded in the XPaths. During test execution, if a field is missing then the execution is stopped to indicate exactly what is wrong. With Selenium, however, the incorrect failure item is shown as the order is changed due to an item being removed.
<h2><strong>Using Smart XPath Generator in Katalon Studio</strong></h2>
Of course, you do not have to worry much about how to create neighbor XPaths like above. Katalon Studio automatically does the job. Neighbor XPaths are generated during recording. Katalon Studio tries to choose appropriate XPaths, while you can still manually select the most relevant ones as shown below.

<img class="init-size alignnone wp-image-5101 size-full" src="https://d1h3p5fzmizjvp.cloudfront.net/wp-content/uploads/2018/09/05141902/Using-Neighbor-XPaths-in-Katalon-Studio.png" alt="Using Neighbor XPath generator in Katalon Studio" width="963" height="447" />

With the custom locators strategy setting, you can choose the default strategy for smart XPaths for all test elements.

<img class="init-size alignnone wp-image-5102 size-full" src="https://d1h3p5fzmizjvp.cloudfront.net/wp-content/uploads/2018/09/05150515/Automatic-XPaths.png" alt="Automatic XPaths" width="975" height="421" />
<h2><strong>Conclusion</strong></h2>
We showed several examples demonstrating that Selenium’s XPaths are fragile and hard to maintain. The common cause is Selenium’s reliance on an element’s index and ordering.

This article described Katalon Studio’s advanced and intelligent method to automatically generate locators or neighbor XPaths in automation test scripts. The XPaths are more resilient to changes in the AUT and more maintainable than those generated by Selenium. This advanced method is available in the version 5.7 of Katalon Studio. <a href="https://www.katalon.com/resources-center/blog/smart-xpath-bdd-testing-katalon-v5-7/" target="_blank" rel="noopener">Learn more about version 5.7</a>

Read more:
<ul>
 	<li><a href="https://www.katalon.com/resources-center/tutorials/detect-elements-xpath/" target="_blank" rel="noopener">Detecting elements with XPath in Katalon Studio</a></li>
 	<li><a href="https://www.katalon.com/resources-center/tutorials/xpath-katalon-studio-tips/" target="_blank" rel="noopener">Common Tips For Using XPath In Katalon Studio</a></li>
</ul>

Original source: <a href="https://www.katalon.com/resources-center/blog/smart-xpath-generator/">Explore the Brand-new Katalon Studio’s Smart XPath Generator</a>