Mojito framework overview

Mojito is a modular, source-controlled split testing framework that lets you build, launch and analyse experiments via Git/CI.

It is comprised of 3 core modules that can be used independently of each other or in unison:

1. Mojito JS Delivery: Run experiments on your site through JS. (Github)
2. Mojito Snowplow Storage: Events & data models for tracking into Snowplow Analytics. (Github)
3. Mojito R Analytics: Templatable RMarkdown experiment reports. (Github)

Features

• ~5.5kb minified & gzipped
• Define experiments with simple JS or YAML
• Self-hosted & git-controlled for familiar code review / merging
• Expressive trigger system & utilities
• Variant code (JS/CSS) minification & linting
• Track and handle JS errors caused by your variant code

Mojito vs. [vendor]

Differentiating features between popular vendors' tools and Mojito out of the box:

Feature	Optimizely X	Google Optimize	Mojito
Open-source license	❌	❌	✅ BSD3
Light front-end codebase	❌~80kb*	❗~25kb*	✅~5.5kb^
Git source control & CI	❌	❌	✅
Variant error-tracking/handling	❌	❌	✅
Auto CSS/JS minification	❗(not custom code)	❓	✅
Self-hosted	❗ (for a fee)	❗(via API)	✅
Data ownership	❗(via S3 export)	❗(via 360/BigQuery)	✅
Retroactively add new metrics	✅	❗(360 only)	✅
Server-side/App testing	✅	❗(via API)	❗(via Storage)
WYSIWYG test editor	✅	✅	❌

* Tested 2019-07-05, ^ Tested 2020-05-18

Getting started

Mojito consists of three components, which are often switched out in the course of Mint Metrics' client services:

1. Delivery: Front-end libraries to reliably control which treatments users are exposed to. e.g. Mojito JS Delivery
2. Storage: Data collection modules and data modelling steps to power your reports. e.g. Mojito Snowplow Storage
3. Analytics: Tools to measure & report on the effects caused by your treatments. e.g. Mojito R Analytics

Get up and running quickly with the README files inside each section.

Example experiment

Using Mojito's CI tools, you can set up experiments in YAML & JS:

id: ex1
name: Example test 1
state: live
sampleRate: 0.75
trigger: trigger.js
recipes:
  0:
    name: Original
  1:
    name: Variant
    js: variant.js
    css: variant.css

Where trigger.js activates the experiment when a condition is met and a callback to activate is fired:

function trigger(test) {
    if (document.location.pathname === '/') test.activate();
}

Upon activation, the will include 75% of traffic (sampleRate: 0.75) and split it 50-50 between "Original" and "Variant" groups.

For users assigned to the "Variant" group, we execute a) variant.js and b) variant.css files to transform the page through a a) JS function and b) CSS stylesheet respectively.

After you've defined an experiment YAML...

Run the deploy command to build & publish your container.

1. Install the necessary NPM packages: npm install
2. Build & publish your testing container: npm run deploy

Example analytics reports

If you use our Snowplow/Redshift & R Analytics component for reporting, all your metrics can be reported on with a simple array of metrics.

wave_params <- list(
  client_id = "mintmetrics",
  wave_id = "ex1",
  start_date = "2019-05-15 09:19:45",
  stop_date = "2019-06-05 14:29:00",
  time_grain = "hours",
  subject = "usercookie",
  recipes = c("Original", "Variant")
)

goalList <- list(
  list(
    title = "Transactions",
    goal = "purchase",
    operand = "="
  ),
  list(
    title = "Thankyou page views",
    goal = "page_view /contact/thank-you%",
    operand = "like"
  )
)
goalList <- mojitoFullKnit(wave_params, goal_list = goalList)

For this experiment, we'll report on transactions and page views:

Support for other analytics back-ends

You don't exactly need Snowplow Analytics to use Mojito. You can also track experiments to wherever you like, via a custom storage adapter. E.g. To Google Tag Manager, Adobe etc.

You can even hook Mojito Delivery up to Google Optimize's reports for free.

Server-side and app split testing libraries

Currently we support only front-end JavaScript experimentation through our Delivery module. Until we build out SDKs for apps and server-side code, our support for experiments on these platforms is through our Storage and Analytics components.

Using modules independently

The modular structure makes it easy to use components of Mojito independently of one another. We often use Mojito components individually when:

1. Existing SaaS tool like Optimizely, VWO or Convert.com is implemented as a Delivery mechanism: Track experiments into Storage and Analytics reports
2. Existing Storage target and Analytics reports, such as Google Optimize are being used: Build and launch experiments from Git/CI with Mojito's JS split testing library

Credits

Our Delivery JS library is a heavily modified fork of the excellent jamesyu/cohorts lib. Meanwhile we employ heavy use of the Snowplow Analytics event pipeline for our Storage component and RStudio/Knitr for our Analytics reports.

Getting involved

We would love to see PRs! We're able to assist if you hit any snags getting set up.

Reach out to us via:

• Mint Metrics' website
• Open an issue on Github