[Android Software/Hacking General [Developers Only]] cathor01:
Synopsis:

This article mainly introduces SoloPi, which is a wireless, non-intrusive and root-free Android testing tool implemented by Alipay on Android devices. The automatic, performance, compatibility and stability test of the mobile phone can be realized by directly controlling the mobile phone.

1. Introduction:

In recent years, mobile testing technology has made great progress with the rapid development of mobile internet. Test efficiency has been improved again and again from the single device automation based on testing scripts in the early time, to recording and playback, image recognition, cloud testing platform and other testing technologies, fits the needs of actual business, in-depth application and innovation.

This article mainly introduces a wireless, non-intrusive and Root-free Android testing tool SoloPi implemented by Alipay on the mobile terminal. The automatic , performance, compatibility and stability test of the mobile phone can be realized by directly controlling the mobile phone.

1.1. Mobile Testing 1.0 Era

Mobile testing 1.0 era, can also be called the exploration period. As tired of day-to-day manual operation, how to improve the test efficiency has become the most important topic in the field of mobile testing. During this period, in addition to Monkey, Uiautomator, Instruments and other official tools, a number of excellent open-source automatic testing tools / frameworks have emerged in the industry. Based on the automatic driven ability, it can not only realize the verification of basic functions, but also realize the automation of various targeted type-based tests by combining performance data gathering scheme and traversal algorithm. In this stage, the common form of automatic testing is to deploy the testing environment on a single PC or a few local PCs, and then use Jenkins and other tools to achieve continuous integration.

1.2. Mobile Testing 2.0 era

With the continuous development of testing technology, and benefits from the open source of STF, the concept of cloud testing platform began to emerge in the industry, which packaged real device equipment, task management, automatic framework and targeted type-based testing scheme into the platform and provided them as services, bringing one-step testing experience to users. On the other hand, the introduction of remote debugging, equipment scheduling and other technologies greatly improves the utilization rate of equipment, and testers no longer need to worry about the lack of testing equipment or the time-consuming queuing of testing tasks. For users of cloud testing platform, the common testing pattern at this stage is to develop testing scripts on local PC, upload them to the cloud testing platform for execution, and finally view the testing report on the platform. The testing process is simple and clear.

1.3. Mobile testing 2.0+

In keeping with the above "cloud testing", testing technology in the era of mobile test 2.0+ usually provides a set of complete solutions rather than an independent small tools, for example, to provide users with a set of customized IDE environment, combined with the recording playback, image recognition technology, the user may only need to do some simple checkbox select, drag and drop to complete the testing scripts development. On the other hand, due to the limitations of office environment, hardware conditions and other factors, more and more testers hope to initiate testing directly on the mobile terminal and make mobile testing with random mobile-state. Of course, whether it is the cloud, IDE, or mobile, all of them should be able to communicate with each other, test easily in multi-device and variable system, so that the testing scheme can be more flexible and applicable to more scenarios.

2. Wireless driven Android testing tool: SoloPi

The concept of "test easily in multi-device and variable system" is broad, which may not be clearly explained by only one article, so the following will focus on a set of wireless, non-intrusive, root-free Android testing scheme SoloPi that we have implemented on the mobile terminal in order to welcome the era of "mobile 2.0+". Automatic, performance, compatibility, stability testing and other work can be realized by directly controlling the phone.

2.1 Overall structure

<a href="https://gw.alipayobjects.com/mdn/rms_e29b5f/afts/img/A*z_65SqVz0A4AAAAAAAAAAAAAARQnAQ">Replay Video</a>

In this scheme, the underlying dependencies are mainly "wireless ADB, system assist, Chrome debugging, and image recognition technology", which will be described later in the article. At the same time, on the basis of the underlying dependency, we encapsulate a set of core capabilities, consisting of "widget positioning, event-driven, performance data gathering and dependency injection", and realize common service capabilities such as recording, playback and data processing in the service layer. At the top of the architecture, the interface interaction logic encapsulates the entrance of each function.

2.2 Wireless ADB

As is known to all, for Android automation, ADB Shell's executive ability is the foundation of everything.

On PC, ADB Client provided by Android SDK communicates with ADB Server also running on PC, and then ADB Server communicates with Adbd located on the device via USB. To implement a set of wireless scheme, it is necessary to get rid of the dependence on USB cable. Fortunately, Android also provides a socket-based ADB connection mode. In this case, you only need to communicate with the native 5555 port on the end according to ADB communication protocol to obtain the executive ability of ADB shell.

There are already Java open source projects that implement the ADB communication protocol, such as AdbLib, they encapsulate an ADB debugging communication service that replaces the role of ADB Server on a PC. We have integrated the AdbLib open source library in SoloPi applications and packaged it as a set of ADB command execution tools, laying a solid foundation for SoloPi's subsequent implementation of various testing capabilities. The following will introduce you SoloPi's core features.

2.3 Recording and playback

The recording and playback based on AccessibilityService, ChromeDevToolsProtocol and image recognition achieved the accurate search. It can be played back locally on the device or converted into scripts of Appium/Macaca and other automatic frameworks to connect to the cloud testing platform. In addition, in order to reduce user maintenance cost, we also provide functions of testcase editing and process control.

2.3.1 Implementation

During the recording process, SoloPi will intercept the user's operation, identify the location of the user's operation, highlight the control for the current operation and record the current operation type of the user. After each step of operation, all kinds of information about the operation type and target control will be recorded. Control information here includes basic information such as the control's ID and text, as well as relative layout, screenshot information, and so on.

During playback, SoloPi will parse the previously recorded data item by item, and locate the target control through intelligent search algorithm, comprehensive various properties, and perform appropriate operations, such as clicking and sliding, after finding the control. After all the steps are performed, the results of the playback, including logs, screenshots and other information, will be displayed as a summary of the playback.

2.3.2 Widget lookup capability

For traditional Native applications, the properties obtained through UiAutomator dump are sufficient for automation. However, with the steady development of mobile dynamic capability, more and more applications adopt the hybrid development scheme of "Native + H5/Tinyapp". In addition, considering the rapid development of mobile game industry in recent years, the demand for automatic testing of mobile games is also increasing. In order to fit as many scenarios as possible, SoloPi provides three lookup modes:

Don't need to say more, the core of the first scheme is to generate the current control view tree based on AccessbilityService and record the attributes such as ID and text, which is suitable for Native scenario.

The second scheme is based on the Debugging protocol of Chrome. The page layout and the attributes of each element can be obtained by injecting JS. The positioning idea of the control is consistent with the scheme of Accessbility. Suitable for H5/ Tinyapp scenarios.

The third scheme is the image matching scheme. SoloPi implements a set of image matching capability on the device, combines template matching, feature matching and other algorithms, and makes certain adaptation and tuning. Suitable for game automatic scenarios. In addition, in the current SoloPi scheme, the image matching ability will also be used as the first two positioning methods works for all the details, further improve the accuracy rate of control search.

2.3.3 Playback capability

The testcase recorded by SoloPi is stored as JSON, so that the testcase can be directly played back on the device as demonstrated in the above video. Moreover, the testcase can be converted into scripts of the current mainstream automatic testing framework such as Appium and Macaca through SoloPi's parser, and the cloud testing platform can be easily accessed. In addition, benefits from the ability of text capture and image recognition, SoloPi also realized the ability to record the testcases once on the Android side, and the generated scripts can be played back on both Android and iOS sides at the same time.

2.3.4 More functions

SoloPi also provides testcase editing functions such as insert, delete, and modify of testcase steps, which can effectively reduce the maintenance cost of testcase. In addition, SoloPi introduces flow control capabilities such as loops, conditions, and so on. If the test cases are properly choreographed, tool scripts that require repeated operations or stability testing scripts that require violent playback can be easily implemented.

Additional recording and playback capabilities include the ability to combine data-mock to improve testcase playback instability, to get through performance tests, and so on.

2.4 Multi-control By One Device

Among all kinds of targeted type-based tests, compatibility testing is the most time-consuming and laborious one. Testers need to pay attention to various system versions, various mobile phone manufacturers, various types of screens and so on. It is very expensive to guarantee the quality of compatibility testing through pure manual test.

SoloPi implements a compatibility testing scheme based on the recording and playback capability. In the playback scenario, we first record the user's actions on one device, and then implement the playback of the actions on any device. If the scene is extended to more than one device, multiple devices can be controlled by one device. We call this function "Multi-control By One Device (MBOD)". More concretely, a Socket connection is established between the host and the slave, and then send the user's operation to each slave in real time on the host, and complete the playback of the operation on the slave.

The setup of MBOD environment is flexible. After installing SoloPi on the mobile phone at hand, deployment can be completed through simple joint operation. MBOD is adapted to the current mainstream models and ROM in the market, and encapsulates some quick functions to improve the testing efficiency, such as application installation, data cleaning, equipment information viewing and so on.

2.5 Performance Testing

When it comes to targeted type-based tests, performance testings have to be mentioned. In recent years, mobile applications have become an indispensable part of people's daily life, which also puts higher requirements on the application experience. In order to give users a easy and smooth experience, it is not enough to just implement functions, and performance testing is an indispensable part of creating a good application. However, it is not easy to carry out performance testing. On the one hand, performance testing has certain thresholds, and many times it is necessary to develop scripts to implement it and deal with various compatibility issues. On the other hand, most performance testing scenarios capture basic indicators that make it difficult to detect deeper issues. SoloPi implements a set of performance testing tools to solve the above problems. It includes three functions of acquiring conventional performance indicators, response time calculation and mobile Lighthouse.

2.5.1 Regular performance indicators acquisition

SoloPi supports the real-time acquisition of regular indicators such as CPU, memory, FPS and network traffic, as well as the recording of performance data, which can be stored locally and displayed in the form of reports. SoloPi also provides the ability to upload data that can be sent to the server for further processing. The complete set of performance tools supports manual and broadcast trigger, which can be easily combined with automatic testing.

2.5.2 Response time calculation

In addition to capturing regular performance indicators, SoloPi provides the ability to calculate response time. One of the most commonly used methods for calculate response time is based on code embedding point or system logs (e.g. ActivityDisplayed Time), but the results of this method are differ from users' experience for interfaces that are more asynchronously loading.

SoloPi realized a scheme to calculate the response time close to the user experience based on the ability of recording screen and framing. Specifically, after opening to record screen, SoloPi will monitor screen click event based on the command of ADB shell's get event , as a starting point of the response time, when after recording screen, SoloPi would compare the video from back to front in reverse order to find out the point when the interface tends to be stable in time, and as the end of the calculation. The subtraction is response time.

2.5.3 Mobile Lighthouse

As technologies such as H5/ Tinyapp account for an increasing proportion of mobile applications, how to test the performance of such applications has become a new topic. As anyone who has been exposed to front-end performance knows, Lighthouse is a powerful tool for front-end performance testing, but it cannot be directly used on a mobile phone. And SoloPi is the performance testing tool which implements a Lighthouse in the front-end based on CDP, it can obtain the H5 / Tinyapp startup performance of the page, the flow of resources consumption, request quality, quality of JS, call situation and page information of JSAPI, and built more than 30 best practices of front-end development , aims to find fine-grained performance problems.

The concrete implementation plan is that SoloPi connects to target app with websocket based on Chrome dev tool protocol to listen for callbacks of events such as page initiated request, data receiving, start loading, etc., and to collect data such as error reporting and Trace. Then the data is classified and sorted according to the six dimensions of startup performance, resource flow consumption, request quality, JS quality, JSAPI call situation and page information. Then the collected results are judged by the built-in rules, and the report is finally generated and displayed in the interface.

3. More

As a complete set of targeted type-based testing scheme, besides recording and playback, Multi-control By One Device and performance testing previously mentioned，SoloPi also provides data-mock, performance compression, network simulation, intelligent Monkey, and other functions . Part of the source code of SoloPi is open source currently (Please search SoloPi in Github).

<a href="https://github.com/alipay/SoloPi">SoloPi</a>

Welcome to join us in developing better testing tools!