FAQs - Performance

Have a Question?  Chances are, you’ll find the answer you are looking for here:


Q:  What do you mean by, “The Real Mobile User Experience”?

A:  Mobile users are mobile.  Connection speeds and page downloads vary by location and combination of device, operating system and network.  Knowing the impact these variables have on your mobile Web performance lets you optimize appropriately for YOUR users, not some database of expected network, device or OS performance.  The real world is simply not that precise.

 

Q: How are my test results presented?

A: We present a quick summary of results on your smartphone. If you’ve signed up for a 3PMobile® Performance Measurement account, you will also see your results displayed graphically and in data reports. Data visualizations include waterfall charts, pie, bar and line charts, maps and more.

 

Q: I don’t have an Android phone. Can I still test your software?

A: Yes.

  • Download the Android SDK which contains  an emulator
  • Notes on how to run the emulator
  • Please be aware that there are “inconsistencies” between the various revs of the SDK and the emulators.  We recommend either:
    • Android 2.1 – update1 API Level 7
    • Android 2.2 – API Level 8


Q: I can’t see any performance reports when I login into My Tests page

A: There are several possible issues:

  • Did you enable Performance measurement in the Privacy section of the Choice® testing browser?

After launching Choice®, click on Menu l Privacy l Owner l Preferences l Performance Measuring.  Click on the first check box – enable Performance Reporting.  In addition, you must add the Username and Password with which you used to sign up.

the screen shot below the three arrows point to the relevant sections inside the Performance Measurement panel. After entering the data you can either press the Menu Key and “Save and Exit” or press the back button and you will be presented with a dialog box asking if you would like to save your changes. Say “Yes”.


3P_Enable_Settings

 


  • Did you then go to a Web site and generate a performance report?

Once the Web page has loaded press the Menu Key and look for the Performance Stopwatch icon (Screenshot 1 below). Click on the Performance Stop watch and you’ll see the Performance Overview panel (Screenshot 2). If a report is available there will be a number there e.g. 1 You can then “View Summary” and or “Send a Report”.

3P_Stopwatch 3P_Performance_Panel 3P_Report_Summary

After sending a report you will see a dialog box indicating that the report was successfully sent. If you don’t see that dialog box it means that either there wasn’t a report to send or the phone could not establish a connection to the server.

To view your performance reports, go to http://www.3pmobile.com/mytests/ and sign in with your username and password.

 

Q: Can I test my business processes, like how long it takes to signup for a white paper, login or check-out?

A: We are currently working on this feature and will add it as soon as possible.

 

Q: Do I have to download a mobile application?

A: Yes. The only accurate way to measure the mobile user’s experience is to test how the page loads in a browser running on an actual mobile device.

 

Q: If I don’t have testers everywhere, how can get a representative sample of my mobile users’ experiences?

A. It depends upon your business model:

  • Mobile Workforce: We recommend installing the software on your employee’s devices so you can not only test, but monitor ongoing mobile Web performance.
  • Consumer Websites/Services: Customers like helping out. It provides an opportunity to engage your customers in improving their experience. Engaged customers lead to loyal customers.  Make ongoing performance testing part of your loyalty/retention plans and share the results to show you care.

 

Q: Who sees my data?

A. Only you. Each user has a password-protected account. 5o9 only uses aggregate data from all testers to provide you with benchmarks for comparative purposes. Please read our Privacy Policy.

 

Q: How much does your service cost?

A: You may sign up for a free, 7-day account to test the program. Pricing varies by usage model.  Please contact us for more information.

 

Q: Can I install your service on my servers?

A. If you are interested in an OEM license or local installation, please contact us directly.

 

Q: Do I need to clear my cache before each test?

A. No.

To accurately measure the initial user experience on your mobile website, our software automatically clears the cache before each test.  We appreciate the value of testing both ways, so you have the option to change these settings.   To change your caching preferences in the 3PMobile® Performance App:  Select Privacy Settings from the browser and then select Owner Preferences, followed by Performance Measurement. Under Cache Options, check or uncheck Clear the cache before a test. Be sure to Save and Exit when you are done.

When reviewing your 3PMobile® performance tests, look for the disk icon below your test URL on your My Tests page to see which tests were conducted after clearing the cache and which were not.  Compare the same page, tested both ways, to understand how caching (or not caching) can impact page performance.

 

Q: Every so often my screen goes black and I can’t correct it with the Back button or Home key.  I’m stuck.  What should I do?

A. This is a known Android bug.  To get “unstuck”, depress your main power button until the Power Off panel appears (Phone Options:  Silent Mode, Airplane Mode Power Off). When this appears, the the black screen should be replaced by a blurred image of your home page wallpaper showing behind the Power Off panel.  DO NOT POWER OFF.  Just hit the BACK key to cancel the Power Off menu and the black screen should be cleared.

 

Q:  What impact does using GPS have on my battery?

A: Like any location based service, the frequency with which you change location and access the GPS feature of your phone can accelerate battery drain.  Since the Choice® testing browser periodically updates your GPS coordinates, if you will not be testing for awhile, simply press the browser Menu key, select More and then choose Terminate this Program.

 

Q:  Do you track my location?

A:  No.   Respecting your privacy is very important to 5o9®.   Specific test location data is only visible to you, within your password protected account.  The mobile testing app periodically updates your GPS coordinates, but your location data is only updated on your device.  No data of any kind is ever sent to the 3PMobile® Performance Measurement service unless you specifically choose to send a performance report. You may disable location sharing by changing your device GPS settings, but you will loose the ability to measure the effect of location upon mobile Web performance. For more information, please see our Privacy Policy.

 

Q: Some people have suggested that resources are being loaded in some other timeframe than they’re being reported by the WebKit “OnLoadResource”.

A: When you get the “OnLoadResource” event from WebKit you can still return false, and cancel the loading of that resource.  That means that regardless of what might be happening at the transport layer, the resource has not been added to the page load until after the WebKit “OnLoadResource” event.  This happens because, if you cancel that resource at that moment, it never ever shows up on your Web page.

 

Q: Why is there a difference between onLoad time as reported by the HAR Viewer and the onPageFinished time reported by WebKit (especially when generating Screenshots)?

A: The HAR Viewer and WebKit reporting are two different functions.

  • The HAR Viewer is a Web application (PHP + JavaScript) that allows you to visualize HTTP tracing logs based on the HTTP Archive format (HAR). These files contain recorded information about HTTP traffic performed by Web pages.
  • The HAR format has no information on WebKit and vice versa.  The HAR format simply expects the overall time value (onLoad) to be the sum of all the timing values in the “Entries” array.
  • There can be a delay between the finish of the last resource load and the appearance of the “onPageFinished” or “onProgressChanged (100 percent)” event, both of which are the signal that WebKit is finished with the page.  Hence onLoad time can be different from onPageFinished. Both are correct for their respective functions.
  • Sometimes the “onProgressChanged (100 percent)” message can arrive after the “onPageFinished” event”.  It’s rare but it can happen (especially when there are “deferred loads” taking place).

Screenshots

There are three conditions that will cause a Screenshot procedure to fire…

  1. Your “Interim Screenshots” checkbox is on and an “onNewPicture” event fires
  2. Your overall “Include Screenshots” checkbox is on and we reach the end of either the “onPageFinished” or the ‘onProgressChange – 100 percent’ event processing.
  3. The HTML rendering engine and/or JavaScript engine encounters a specific ‘js5o9_screenshot()’ command embedded in the document itself.

We cannot control how long the Android operating system takes to take a screenshot. After we’ve taken the screenshot we then need to add it to the HAR file to be sent to the server.  The time you see is the time it was actually added to the report.  This is why it can appear after either “onPageFinished” or “onProgressChanged – 100 percent”

 

Q: Does your application support scripted testing?

A: Yes.  We are currently in beta with this feature and are actively building in Selenium scripting compatibility.

Didn’t find your answer?  Please contact 3PMobile® Support