« Back to Index

[Rollout Strategies with A/B logic via Varnish and VCL]

View original Gist on GitHub

Tags: #rollout #vcl #cdn #varnish #fastly #ab

Rollout Strategies via Varnish.md

Rollout Strategies via Varnish

We can utilise A/B testing as a rollout strategy in VCL with Fastly CDN using either:

Percentage based rollouts

Note: see also https://www.fastly.com/blog/ab-testing-edge/

These can be a bit more complicated than the ‘regional rollout’ approach, so let’s look at them first.

Effectively you utilise cookies to help persist the various buckets you place users into, and then you need your application (or proxy) to process/inspect the HTTP request header that is then set and react accordingly (e.g. if the header exists and it has a value of "true", then your proxy can pass the request onto a different origin or if there’s only one origin you can have the application itself change the type of page that’s rendered).

vcl_recv {
  # if there is no cookie found then this is the first time this user has made a request
  # so we'll determine if they can have access to the new feature or not
  if (!req.http.Cookie:NewFeature) {
    # 10% of users will get access to the new feature
    if (randombool(10,100)) {
      set req.http.X-NewFeature = "true";
    } else {
      set req.http.X-NewFeature = "false";
    }
  } 
  # otherwise if there is a cookie, then we've already made the 'access' decision
  # so we'll use whatever value we determined previously for them by taking the value from the cookie
  else {
    # set the value of the header to whatever decision was previously made
    set req.http.X-NewFeature = req.http.Cookie:NewFeature; 
  }
}

vcl_fetch {
  # the following Vary header logic can be added by your application if necessary
  # we need to use Vary in order to ensure we serve the same content version as before for returning users
  if (beresp.http.Vary) {
      set beresp.http.Vary = beresp.http.Vary ", X-NewFeature";
  } else {
      set beresp.http.Vary = "X-NewFeature";
  }
}

vcl_deliver {
  # if the cookie doesn't already exist, set it.
  if (!req.http.Cookie:NewFeature){
    add resp.http.Set-Cookie = "NewFeature=" req.http.X-NewFeature ";";
  }
}

Note: be careful with setting the Vary header via the CDN/VCL as you might not necessarily want every origin response to be cached with a Vary header. If you do this then make sure you always set X-NewFeature (doesn’t matter the value, but it must at least be set) because otherwise your cache HIT ratio could end up being zero! as no matches might be found unless the header was provided (with at least some value)

Regional rollouts

These rollouts are simpler as they don’t rely on the Vary header.

The reason you can get away with not using a Vary header to change the content looked up in the cache is because users will obviously only be in one specific region at any given time (let’s say a user is based in Australia, in one session their requests are unlikely to suddenly be coming from America) and so they will always go to the same set of POPs (as that’s how a CDN’s locality based routing works - users are routed to specific POPs that are nearest to them).

Note: the downside of this approach is that it’s difficult to test because of needing tools such as a proxy to mimic you being in a different locality (which might not be as straight forward as you think, when for example, your source device isn’t a desktop browser but a native mobile application)

vcl_recv {
  # by default no one gets the new feature
  set req.http.X-NewFeature = "false";
  
  # everywhere except the US will get the new feature
  if (!client.geo.country_code == "US") {
    set req.http.X-NewFeature = "true";
  }
}

Alternative bucketing logic

There’s two alternative approaches.

The first…

set req.http.X-Unique-Id = regsuball("39059176dab142e19321c3255e01e56e", "\-", "");
set req.http.X-Val = std.strtol(req.http.X-Unique-Id, 16);
set req.http.X-Bucket = randomint_seeded(0, 99, std.strtol(req.http.X-Unique-Id, 16));

The second uses multiple ‘less than’ comparisons to identify specific percentages…

sub abtest_us_lift_feedranker_recv {
    if (req.url.path == "/" && req.http.X-User-Edition == "en-us") {
        if (req.http.Cookie:ab_cookie) {
            set req.http.X-AB-Test = req.http.Cookie:ab_cookie;
        } else {
            declare local var.bucketing_index INTEGER;
            
            set var.bucketing_index = randomint(0,99);
            
            if (var.bucketing_index < 5) {
                # var.bucketing_index between [0-4] - 5% exposure 
                set req.http.X-AB-Test = "control";
            } elseif (var.bucketing_index < 6) {
                # var.bucketing_index 5 - 1% exposure
                set req.http.X-AB-Test = "foo";
            } elseif (var.bucketing_index < 11) {
                # var.bucketing_index between [6-11] 5% exposure
                set req.http.X-AB-Test = "bar";
            } elseif (var.bucketing_index < 16) {
                # var.bucketing_index between [11-16] 5% exposure
                set req.http.X-AB-Test = "baz";
            } else {
                # not exposed to the experiment
                set req.http.X-AB-Test = "not-exposed";
            }
        }
    }
}

sub abtest_us_lift_feedranker_deliver {
    if (!req.http.Cookie:ab_cookie && req.url.path == "/" && req.http.X-User-Edition == "en-us") {
        add resp.http.Set-Cookie = "ab_cookie=" req.http.X-AB-Test ";expires=" time.add(now,14d) ";";
    }
}

For example, if a user randomly gets a number < 5 (let’s say 3) then because we’ve picked a number out of a 100 (to represent 100%) that means 3 is equal to 3%. Where as if the random number they got was 5, then that’s not going to match < 5 so we move to the next comparison which is < 6, so that matches the number 5 and it means the percentage for those users will be 1% because there is 1 number difference between < 5 and < 6.