Aaron Train

Firefox for Android UI Testing (Robocop) Update

2013-02-18T00:00:00-08:00

For more information on Robocop Automation, see this Mozilla Wiki entry

Short update of where we (Mozilla QA) are at with our recent automation efforts in converting our smoke-tests to automated tests. As of this month there are now approximately forty-five available Mochitest-Robotium (Robocop) tests with approximately thirty currently enabled and running per-checkin (see example run here). Each individual test is ran (see Firefox - Tinderbox Pushlog) in-part of a growing suite of automated user-interface tests that exercise in probing and injecting various native events to the Java front-end of Firefox for Android. In-turn, these provide automation for our list of identified browser smoke-tests (manual testing). While we have steady progress henceforth, we are always looking for more interested to-be test-authors who wish to help us continue to grow our automation efforts.

Short summary on some of the blocking situations we have recently ran into and some details on their solutions:

Due to a number of user-interface differences between various Android devices, it was deemed necessary to create a Device and Navigation support class in BaseTest. Further investigation revealed that was it also necessary to map navigation (Back, Forward and Reload) due to different types of Android devices (i.e, tablets vs phones and varying Android platform versions) see bug 747835.
Test stablization on the Panda and Tegra boards has been tricky. After further investigation we found out that the Tegra boards run Android 2.3 (API Level 11), and Firefox for Android serves the small-screen mobile user-interface for phones running Android 2.3. The Panda boards run Android 4.0.4 (API Level 15) and Firefox for Android serve the larger-screen mobile user-intreface optimized for 7-inch tablets running Android 4.0+. This split environment was rather sub-optimal.
On Android 3.0+ (API Level 12+), the system Cancel and Ok buttons were switched around (gee, thanks Google!) so it was deemed necessary to use a clickOnButton(name); function instead of clickOnButton(index);. This was causing some failures early on.
Again on Android 3.0+ (API Level 12+) pop-up notification buttons were not accessible at times due in part to the to an invocation of the system virtual keyboard, which would overlay cover the testable view. This was a result of having prior, entering text via the sendKeys() function. Thus this requires dismissing the virtual keyboard first and foremost before trying to click any buttons see ongoing investigation in bug 838596.
Depending on the Android device, platform and screen-size we have varying user-interfaces thus it was required to check for proper navigation depending on the environemtn (e.g, we have a function to check for the “More” and “Tools” submenus in order to access items), see bug 830755.
We had to hack around finding a solution for traversing the number of views in Firefox for Android’s settings user-interface. In detail, we had to dynamically fetch the number of checkboxes available to ensure that if further items are to be added, these will not affect the running test. This requires disecting the settings menu and asserting if each component is a checkbox, see ongoing investigation in bug 830834.

Current work and areas we need help in

Expanding our set of tests, see existing coverage here
Stabilize set of problematic tests (intermittent failures and timeouts), see mozilla-central’s test manifest; specifically, bug 825152, bug 746876, bug 824067, 813107, bug 757475, bug 738890

How you can help

Run our manual set of smoke-tests against Nightly builds of Firefox on your Android phone or tablet
Contribute towards writing additional tests (see: writing Robocop tests); individual tests are tagged as either automatable or not in Moztrap
Contribute towards improving the Robocop framework (see: Auto-tools); see open Robocop related bugs, source code

Dabbling with Marionette and Gaia UI layer testing

2012-11-26T23:00:00-08:00

Over the last few weeks in-between Firefox for Android release cycle work, I have been bestowed on the interests of contributing to the Gaia UI automation efforts established from the Mozilla Web QA and Automation & Tools teams by assisting in writing, extending and building upon the available tests and APIs. Preliminary development of an extensive test suite of basic functional smoke-tests tests is underway.

Reviewing the available Gaia smoke-tests, the feasible test-cases can be written into Python based manifest-driven tests that are executed by Mozilla’s own automation driver, Marionette. Sharing much of the same API as Selenium and Web Driver, it makes writing tests rather easy. For example, a first test I have written recently is able to launch, interact and assert content behaviors of HTML elements in the bundled Gaia music application with relative ease — afterall, each application is merely HTML.

Each test that is built upon the Marionette foundation utilizes a Python package named Gaiatest (available on PyPi here) that is in-fact based on Marionette. It is used specifically for writing tests against Gaia and has a growing variety of atomic Gaia related APIs such as: unlocking the device home-screen, toggling WiFi/cellular data, setting volume and interactions with other device settings, etc. For example, here is the current data_layer.js file where we can write functions that interact with accessible public interfaces from the variety of available WebAPI. In turn we can we abstract and create APIs for use in our tests (e.g, preliminary tests such as test_dialer.py disables device volume and soon to be test_browser.py where one can toggle WiFi and or celluar data).

I would encourage all those who are currently writing Gaia applications to try out Marionette and Gaiatest and to contribute to our UI automation efforts by writing well-tested tests to expand our current test-suite. The Github repository is where all activity is at with chatter taking place on #appsqa on Mozilla IRC.

Further information on running Marionette tests can be read here alongside a README for installing and setting up Gaiatest.

Testing H.264/AAC/MP3 decoding in Firefox Beta for Android

2012-10-13T14:00:00-07:00

My last post on modifying an Android Stagefright H.264/AAC/MP3 decoder preference in Gecko, which toggles software and hardware decoding of those formats in Firefox for Android briefly talked about how that override can be usful for testing. As part of our Firefox Beta testing cycle we want to extend our efforts in widening the degree of testing in leveraging the community for assistance in areas of testing different Android 4.0 (Ice Cream Sandwich) and Android 4.1 (Jelly Bean) based devics through different video based websites.

We could use your help testing but not limited to the following video and audio centric websites such as:

We are also looking for testing around different types of popular online video platforms where playback issues may exist. Recently, we have seen an issue where Brightcove was identifying Firefox as unsupported due to a mismatch in the naming of profiles used in the list of supported video formats returned from the video.canPlayType() JS API. Their player test-page now currently identifies Firefox for Android as supported.

We are also seeking out any potential playback issues with Android devices.

The following are example bugs related to devices of issues of which we have seen thus far:

Bug 785275 - Galaxy S III hardware decoder shows green bars when playing non-720p videos (OMX_COLOR_FormatYUV420SemiPlanar)
Bug 797364 - Stagefright: Galaxy SII & Galaxy Note hardware decoder video yields green layer over video
Bug 784329 - No video playback on the Galaxy Nexus
Bug 783830 - Playback on Samsung Galaxy SIII intercepted by AwesomePlayer

Currently, support is restricted to Ice Cream Sandwich and Jellybean devices with Gingerbread in development. The following devices are known to work and have been tested at Mozilla:

HTC One X (Android 4.0)
HTC One V (Android 4.0)
Samsung Galaxy Note (Android 4.0)
Samsung Galaxy Nexus (Android 4.0, 4.1)
Samsung Galaxy SII (Android 4.0)
Samsung Galaxy SIII (Android 4.0)
Samsung Nexus S (Android 4.0, 4.1)

How to Help

First, we would like for you to download the new Firefox on Android (currently on the Beta channel) available here and install it onto your Android device
Try out different video-based websites that serve HTML5 video and audio, identify playback (player, audio, and video) issues
Create a Bugzilla account in our bug tracker so that you can file bugs you find against Firefox on Android in Bugzilla. Reminder, effective bug reports are likely to be fixed. There are excellent bug writing guidelines for your pleasure to read here
Join and chat with us on Mozilla IRC to talk with Mobile QA and mobile developers to let us know of any issues. We are particularily interested in the state of playback on other Android mobile devices and which sites work and which sites dont work

Mozilla Mobile QA (Firefox for Android) and AppThwack

2012-10-02T15:28:00-07:00

Here at Mozilla, we’re always looking and investing in different ways that our mobile related testing efforts and strategies can evolve and expand into new avenues as Firefox for Android mature as a product. We’re always engaged in seeking out different yet worthwhile services and tools that can integrate into our regular QA process.

Whilst we continue to operate as a small team (Mobile QA for Firefox on Android is about two-full time employees); we’re seeking all the help we can get whether it be from community or from those who have created services and utilities to assist mobile QA efforts.

We react in seeking solutions by continuing to identify and respond to problems. In identifying our areas of contention, we still see Android device fragmentation and access on-demand to Android devices as major paint-points of Android testing. To administer any type of soluton would involve boasting and utilizing the services and assistance from others. Currently we use DeviceAnywhere for one-off direct device testing; but we desired a service that targets automation.

A couple months ago we stumbled upon AppThwack, a service that manages mobile application and fragmentation issues by deploying uploaded Android Package (APK) files and deploying (installation) on realAndroid based phones and tablets. The devices are hosted and interfaced through web front-end that makes it easy to view and extract results ran from automated tests (currently self-written and provided MonkeyRunner tests are available, and on-track towards deploying Robotium and possibly in the near future Mozilla based Robocop tests). Alongside, screen-shots are provided from multiple device orientations which make it usful for identifying any user-interface issues. A test-run takes a couple minutes and any data collected may be bundled and downloaded together to deploy towards any new Bugzilla bug for track-keeping.

Currently, we are utilizing their service for testing Firefox for Android builds during our rapid release Beta cycle and are looking at different strategies of integration for the other channels within our cycle.

If you would like to see what a test-run looks like click here.

Site Compatibility Testing for Firefox for Android

2012-09-27T14:47:00-07:00

The Mozilla Developer Network has recently accepted an excellent informational article written by Jason Smith, Lawrence Mandel, and Janet Swisher essentially entailing the largely problematic issue of browser and content compatibility with the mobile web. If you are interested in site testing for Firefox for Android, please give the article a read; and help us improve the web. Let us know if you have any questions about compatibility testing or want to become involved in Mozilla’s mobile web compatibility effort, please sign up for the compatibility@lists.mozilla.org mailing list. https://developer.mozilla.org/en-US/docs/Mozilla/Firefox_for_Android/Compatibility_Testing

Debug preference for testing H.264/MPEG-4 AVC in Firefox for Android

2012-09-12T15:14:00-07:00

Nightly for Android has recently adapted a new about:config preference for using and more-so testing Android H.264/MPEG-4 AVC decoding. The preference is media.stagefright.omxcodec.flags

Utimately, this preference allows one to override whether Firefox uses Android’s H.264/MPEG-4 AVC software or hardware decoder. As Chris Peterson mentions in the same announcement, this is useful for debugging different configurations and conveinent for testing.

Flags values to note:

0 - Android will use either hardware or software decoding (default)
8 - Force software decoding
16 - Force hardware decoding

Reference: OMXCodec::CreationFlags

Firefox for Android - Website Compatibility Testing

2012-08-19T15:37:00-07:00

Firefox for Android Testers,

Website compatibility testing will be a covered and focused area as part of the release testing process for Firefox for Android 15.0, and we want you to help us during our testing cycle.

We want you to use Firefox on Android (on the Aurora) channel to assist us in identifying any noticeably major issues found in websites. Issues can range from pages that do not look right due to layout and positioning problems to interactivity and font related issues. We are also looking for testing around top-sites to identify areas where content served to Firefox differs from other mobile browsers.

How you can help us?

Download Aurora for Android
Run the website compatiblity testing tool
File issues found on Bugzilla
Join our website compatiblity testing and evangelism mailing list
Join us on IRC in the #mobile and #qa channels

Bringing a better Firefox to Android will guide us all towards fulfilling Mozilla’s mission of providing access to one Web that everyone can access, regardless of device or location.

Discover and Help Test Web Apps in Firefox for Android on Friday August 24th, 2012

2012-08-19T15:37:00-07:00

This upcoming Friday, we would love for you to solicit your support towards testing and contributing to the process of qualifying a new alpha-release version of Firefox for Android on the Aurora channel as part of our rapid release testing cycle. We will be focusing on the process of discovering, installing, managing and using web applicationsinside of Firefox for Android.

This event is open to all those interested: newcomers, experienced testers, developers, and anyone interested in testing and using Firefox on Android.

We would like for you to use the new version of Firefox on your Android phone, and take a close look at the latest Aurora product in order to assist us in identifying any noticeably major issues found, and ensure that all feature functionality that is included in this upcoming release is on its way to a feature and testing complete state with the added emphasis on discovering and using web applications.

To cover the work throughout the whole day we have created an EtherPad test-plan. Please feel free to read and use it as a companion during the testday event. There will be moderators at hand in the IRC channel: #testday, to answer any questions you have.

Together with your help we want to make this testday a success and ensure the high quality of Firefox for Android for mobile devices for all of our users world-wide. If you have time on Friday August 24th, 2012, please join us on IRC, we will have Mozilla community and testers on hand to help answer any of your questions.

Your testing and feedback is highly valuable, and we hope to see you attend our test day event.

Destroy all the activities!

2012-06-30T19:46:00-07:00

These past few days, an influx of reports have arrived with users reporting that the new Firefox on Android will ’ultimately fail to properly load web-pages’as strongly convened from investigating the multiple reports.

Turns out for whatever reason folks seem to be enabling a tucked away Android 4.0 (Ice Cream Sandwhich) developer option — ’Don’t keep activities’ with the description: ’Destroy every activity as soon as the user leaves it’, which means that Gecko is likely killed when a user tries to visit a URL from the browser’s ‘Awesome Screen’.

As Kevin Brosnan points out in our tracking bug 769269,

~~~~ Firefox requires a background activity because of its hybrid Java/C++ design ~~~~

It is a nice find and an interesting case. Moral of the story please keep this developer option disabled to prevent destruction of all the activities!

Android MonkeyRunner Automation

2012-04-21T20:42:00-07:00

A few folks have asked me what’s involved with my simple automation script that ultimately produces screenshots of running browsers on Android. It’s rather uncomplicated, I am using the monkeyrunner tool that is bundled with the Android SDK. As the Android Developers guide indicates:

The monkeyrunner tool provides an API for writing programs that control an Android device or emulator from outside of Android code. With monkeyrunner, you can write a Python program that installs an Android application or test package, runs it, sends keystrokes to it, takes screenshots of its user interface, and stores screenshots on the workstation. The monkeyrunner tool is primarily designed to test applications and devices at the functional/framework level and for running unit test suites, but you are free to use it for other purposes.

Intended mainly as a tool for synthesizing input on Android, one can also easily get a capture of the Android screen with its Python/Jython API. The code below listens and connects to an Android device, loops through a list of provided Android browser package/activities, visits a site (provided by a CSV), waits a couple seconds for page-load (Hack! How can I go about better doing this?), snaps a picture and writes it out to a directory provided by argument.

from com.android.monkeyrunner import MonkeyRunner, MonkeyDevice
import sys, csv

browsers = ['com.android.chrome/.Main',
                'org.mozilla.fennec/.App']

# Connects to the current device, returning a MonkeyDevice object
device = MonkeyRunner.waitForConnection()

#Start each browser activity with the provided URL
for run, browser in enumerate(browsers):

    # Visit each site
    for visit, site in enumerate(csv.reader(open(sys.argv[1]).readlines())):

        # Start the activity with the provided site
        device.startActivity(component=browser, uri=site[0])

        # Wait for page load timeout        MonkeyRunner.sleep(20)

        # Snap a screenshot of the running activity
        device.takeSnapshot().writeToFile("%s%s-%s.png" % (sys.argv[2], visit, run), 'png')

Call for Help - Testing Firefox on your Android Device

2012-04-20T18:15:00-07:00

As part of Mozilla Mobile QA’s release criteria for continually testing Firefox on Android, we want to again extend our efforts in leveraging the community for assistance in areas of device.

We want you to test and provide feedback in regards to how Firefox operates on your Android device.

A problematic challenge we face is the ability to truly test and qualify the ever increasing and vast assortment of Android based devices (phone, tablet and soon to be more) out there in the wild; the nature of which the carriers and combination of an open platform presents us with. With so many devices available, it is difficult to to ultimately attest and standby the truth in acknowledging if a device will be truly compatible with Firefox or not based on a range of criteria. Compatibility on each device can also very well be a subjective opinion based on overall experience, fulfillment and value. As each device has different hardware and different processes and configurations, it is challenging to produce an appropriate label and or level of compatibility for a device. This is where we want your help.

To gather information from community based testing, we have created a simple survey that can assist in defining and painting the overall picture of a device at hand. We currently have a backlog of untested devices that we not have access to, and we could use your help in testing!

How to Help

First, we would like for you to download the new Firefox on Android (currently on the Nightly channel) available here and install it onto your Android device
Create a Bugzilla account in our bug tracker so that you can file bugs you find against Firefox on Android in Bugzilla. Reminder, effective bug reports are likely to be fixed. There are excellent bug writing guidelines for your pleasure to read here. Detail the issues you find to significantly increase the chances that a developer can act on the bug thus increasing the likelihood of positive resolution — provide in-browser Android screenshots, website URL, browser build information, and steps to reproduce the issues

Join and chat with us on IRC on the server irc.mozilla.org in the rooms #mobile and #qa to talk with Mobile QA and mobile developers.

A few minutes of your time in completing the survey will be largely valuable, and you will be contributing towards the testing efforts of Firefox.