Using Provar With Amazon Web Services (AWS) Device Farm
If you develop new mobile-optimized applications, you know that the testing process throughout the development lifecycle is exponentially more challenging given the wide range of mobile devices that your end-users may be operating to access your app.
That’s why more leading organizations are turning to the AWS Device Farm to help streamline the testing process by helping you test your application on a shared fleet of 2500+ devices remotely in the cloud. (We get it. The cube next to you probably isn’t big enough to charge that many phones.)
With Provar, you can create a wide range of testing scenarios and then rapidly reapply those tests by integrating those tests with the AWS Device Farm. (Side note, we also support testing via a number of different remote web and mobile testing platforms such as Sauce Labs, Perfecto, BrowserStack too.)
If you’d like to use Provar in conjunction with the AWS Device Farm, you can use the following information to run Provar test cases on a variety of Android devices (that use a Chrome browser) within the AWS Device Farm.
To learn more, you can also check out the AWS Device Farm documentation that is available on the Salesforce website.
Prerequisites (a.k.a. Things You Need)
- An AWS console account.
- A Provar ANT zip file.
- A Provar project with at least one test case.
Hey, Let’s Get Started. Setting Up Your Folder Structure to Execute Test Cases in the Device Farm
Since the AWS Device Farm is designed to conceal the webdriver, you will need to upload the project in an AWS Device Farm in a zip format.
For that, you will need to create a folder structure using the steps below:
(1) Create a folder (e.g. ProvarProject).
(2) Add the project folder, the Provar_ANT folder and the license file in the ProvarProject folder. Rename the license file with the name “ProvarLicense.properties”.
(3) In the project folder, there are test cases in the “tests” folder, build.xml, “ANT” folder and the .testproject file containing settings for Appium.
(4) Using ANT build.xml, execute your test cases.
(5) The Provar_ANT folder contains all of your required jar files.
(6) Create a zip file of ProvarProject.
Up Next. Configuring Your Settings
You can configure the settings in a couple of different ways. You can either use the Provar UI or update the build.xml file and .testproject file.
(1) In the build.xml, a few changes are required:
Change provar.home where the Provar_ANT directory is available.
<property name="provar.home" value="../../Provar_ANT"/>
Change ProviderName (same as mentioned in .testproject) and DeviceName (any name) in the Run-Test-Case tag.
webBrowserProviderName="SampleProviderForDevice" webBrowserDeviceName="Google Pixel 2"
(2) In the .testproject file, select the browserProvider (same as ProviderName in build.xml). The values mentioned in PINK are mandatory fields and should have the same value as shown below.
<browserProvider name="SampleProviderForDevice" type="Appium"> <devices> <browserProviderDevice name="Google Pixel 2" platform="ANDROID"> <properties> <browserPlatformProperty name="deviceName" type="DesiredCapability" value="Google Pixel 2"/> <browserPlatformProperty name="platformName" type="DesiredCapability" value="ANDROID"/> <browserPlatformProperty name="platformVersion" type="DesiredCapability" value="9"/> <browserPlatformProperty name="udid" type="DesiredCapability" value="deviceUDID"/> </properties> </browserProviderDevice> </devices> <properties> <browserPlatformProperty name="gridUrl" value="http://127.0.0.1:4723/wd/hub"/> <browserPlatformProperty name="newCommandTimeout" type="DesiredCapability" value="100"/> </properties> </browserProvider>
Note: deviceUDID parameter will be replaced by actual device UDID at run time.
How to Execute Test Cases (What You Really Want to Know)
(1) Sign into your AWS console and navigate to the AWS Device Farm.
(2) Create a project in the AWS Device Farm console.
(3) In automated tests, select “Create a new run” button.
(4) Choose the web application and provide a name and click “Next Step”.
(5) In Configure Steps, select the Appium Node.js test type.
(6) Click on the upload button and upload the ProvarProject.zip file.
(7) Once the project is uploaded, specify spec file. See the spec file below.
version: 0.1 # Phases are collection of commands that get executed on Device Farm. phases: # The install phase includes commands that install dependencies that your tests use. # Default dependencies for testing frameworks supported on Device Farm are already installed. install: commands: # By default, Appium server version used is 1.7.2. # You can switch to an alternate supported version from 1.6.5, 1.7.1, 1.7.2, 1.8.0 , 1.8.1, 1.9.1 by using a command like "avm 1.7.1" # OR # To install a newer version of Appium use the following commands: - export APPIUM_VERSION=1.9.1 - avm $APPIUM_VERSION - ln -s /usr/local/avm/versions/$APPIUM_VERSION/node_modules/.bin/appium /usr/local/avm/versions/$APPIUM_VERSION/node_modules/appium/bin/appium.js - java -version - ant -version - export PROVAR_HOME=$DEVICEFARM_TEST_PACKAGE_PATH - cd $DEVICEFARM_TEST_PACKAGE_PATH - mkdir -p /home/device-farm/Provar/.licenses - mv $DEVICEFARM_TEST_PACKAGE_PATH/ProvarProject/ProvarLicense.properties /home/device-farm/Provar/.licenses # The pre-test phase includes commands that setup your test environment. pre_test: commands: # We recommend starting appium server process in the background using the command below. # Appium server log will go to $DEVICEFARM_LOG_DIR directory. # The environment variables below will be auto-populated during run time. - echo "Start appium server" - >- appium --log-timestamp --browser-name Chrome --device-name $DEVICEFARM_DEVICE_NAME --platform-name $DEVICEFARM_DEVICE_PLATFORM_NAME --udid $DEVICEFARM_DEVICE_UDID --chromedriver-executable $DEVICEFARM_CHROMEDRIVER_EXECUTABLE >> $DEVICEFARM_LOG_DIR/appiumlog.txt 2>&1 & - >- start_appium_timeout=0; while [ true ]; do if [ $start_appium_timeout -gt 60 ]; then echo "appium server never started in 60 seconds. Exiting"; exit 1; fi; grep -i "Appium REST http interface listener started on 0.0.0.0:4723" $DEVICEFARM_LOG_DIR/appiumlog.txt >> /dev/null 2>&1; if [ $? -eq 0 ]; then echo "Appium REST http interface listener started on 0.0.0.0:4723"; break; else echo "Waiting for appium server to start. Sleeping for 1 second"; sleep 1; start_appium_timeout=$((start_appium_timeout+1)); fi; done; # The test phase includes commands that run your test suite execution. test: commands: # Go into the root folder containing your source code - echo "Navigate to test source code" - cd $DEVICEFARM_TEST_PACKAGE_PATH - echo "Start Provar/Appium test" - sed -i "s/deviceUDID/$DEVICEFARM_DEVICE_UDID/g" ProvarProject/Project/.testproject # Enter the command below to start the tests . The comamnd should be similar to what you use to run the tests locally. - cd ProvarProject/Project/ANT - ant # The post test phase includes are commands that are run after your tests are executed. post_test: commands: # The artifacts phase lets you specify the location where your tests logs, device logs will be stored. # And also let you specify the location of your test logs and artifacts which you want to be collected by Device Farm. # These logs and artifacts will be available through ListArtifacts API in Device Farm. artifacts: # By default, Device Farm will collect your artifacts from following directories - $DEVICEFARM_LOG_DIR - $DEVICEFARM_TEST_PACKAGE_PATH/ProvarProject/Project/ANT/
(8) Create a new device pool and add devices in it.
(9) After selecting Device Pool, click the Next Step button.
(10) Click the Confirm and Start Run button.
(11) Check the status of the test execution using the links shown below.
Getting Those Reports You Need
The AWS Device Farm provides many options (like Video, Test Specification log, Files, Screenshots) to verify your test execution status. And, it stores your reports for up to 400 days. These reports include metadata, logs, screenshots, and performance data.
(1) Video – It records the execution session on each device.
(2) Suites – It provides different types of status. Also, you can get the execution timings at suite level.
(3) Log files.