LeOS-Ice-browser

15 KiB

Raw Permalink Blame History

layout	title	date	author
post	☀️ Google Summer of Code: Fretboard - A/B Testing Framework for Android	2018-08-10 14:48:00 +0200	fernando

Overview

Firefox for Android (codenamed Fennec) started using KeepSafe's Switchboard library and server component in order to run A/B testing or staged rollout of features, but since then the code changed significantly. It used a special switchboard server that decided which experiments a client is part of and returned a simplified list for the client to consume. However, this required the client to send data (including a unique id) to the server.

To avoid this Firefox moved to using Kinto as storage and server of the experiment configuration. Clients now download the whole list of experiments and decide locally what experiments they are enrolled in (the configuration for Fennec looks like this).

The purpose of this Google Summer of Code project, which is called Fretboard, was to develop an A/B testing framework written in Kotlin based on the existing code from Fennec, but making it independent of both the server used and the local storage mechanism, allowing it to be used on other apps, such as Firefox Focus, which started to need to do some A/B tests.

The source code implemented as part of this project is located at the android-components GitHub repo, more specifically here

Features

This is a basic and non-exhaustive list of features, for all details you can view the README here

Query if a device is part of a specific experiment
Get experiment associated metadata
Override a specific experiment (force activate / deactivate it)
Override device values (such as the appId, version, etc)
Update experiment list from the server manually or using JobScheduler
Update experiment list using WorkManager (blocked waiting for a non-alpha version of WorkManager to be released by Google)
Default source implementation for Kinto
- Uses diff requests to reduce bandwidth usage
- Support for certificate pinning (pending security review)
- Support for validating the signature of the downloaded experiment collection (pending security review)
Default storage implementation using a flat JSON file

Filters

Fretboard allows you to specify the following filters:

Buckets: Every user is in one of 100 buckets (0-99). For every experiment you can set up a min and max value (0 <= min <= max <= 100). The bounds are [min, max).
- Both max and min are optional. For example, specifying only min = 0 or only max = 100 includes all users
- 0-100 includes all users (as opposed to 0-99)
- 0-0 includes no users (as opposed to just bucket 0)
- 0-1 includes just bucket 0
- Users will always stay in the same bucket. An experiment targeting 0-25 will always target the same 25% of users
appId (regex): The app ID (package name)
version (regex): The app version
country (regex): country, pulled from the default locale
lang (regex): language, pulled from the default locale
device (regex): Android device name
manufacturer (regex): Android device manufacturer
region: custom region, different from the one from the default locale (like a GeoIP, or something similar).
release channel: release channel of the app (alpha, beta, etc)

Merged commits

Open pull requests

There are two open pull requests. The first one is open pending a review from the security team and the last one is waiting for a non-alpha version of WorkManager to be released by Google:

Project progress and difficulties faced

Prior to starting the project, I became familiar with Kinto and the diff response format, as well as with the existing code of the Switchboard fork from Fennec. I also thought it was a good idea to send a pull request to Firefox Focus because this library was going to be integrated into it, and also to become familiar with a code review process at Mozilla. I looked at the issue list and discovered a problem with display cutouts, so I sent two pull requests to address the issue: the first one and the second one.

At the beginning of the project it became necessary to familiarize myself with tools like Taskcluster, which I had never used (although I used similar tools like Travis CI).

The most difficult pull requests for me were the ones related to certificate pinning and experiment collection signature verification. For the first one I had to broaden my knowledge about it, as well as research how to properly implement it on Android, avoiding common mistakes. For the second one the most difficult part was to understand what algorithm Mozilla was using to validate the signatures, and how it worked. I discovered from the Kinto collection mode field that it was p384ecdsa, and then I had to research how to properly implement it in Kotlin. For this later I needed the help of Julien Vehent and Franziskus Kiefer, which pointed me to a great talk and also a Go and C++ implementation. After seeing the two implementations I realized my solution wasn't working because I didn't know that the signature actually contained two values concatenated (r and s), which then needed to be encoded using DER syntax

Overall I think I learned a lot doing this project and I really loved working with Mozilla.

Extending Fretboard

Right now there is an ongoing discussion about enhancing Fretboard with an expression language (being that JEXL/CEL/etc) for the matchers values instead of regular expressions like it's using now.

Thanks to

I would like to thank my mentor Sebastian Kaspari for all the help and guidance, for being so friendly and available to talk at any moment I needed, as well as reviewing my pull requests quickly.

I would also like to thank Franziskus Kiefer and Julien Vehent for helping me understand the signature validation system used by Kinto.

15 KiB Raw Permalink Blame History