Experimentation with Network Effects

The Challenge of Network Effects

One common challenge engineering teams encounter when running experiments on large-scale social products is that of network effects. Network effects occur when a user's treatment assignment alters the experience of other users not in the treatment group.

To take a practical example, imagine your team is planning to launch a group voice chat feature on the app Clubhouse. When designing your experiment, you'll need to answer:

1. How would you determine which users in the test group receive the group voice chat feature?
2. Would the test group users only have access to the feature when chatting with other users also in the test group?

Fundamentally, there are two challenges that arise when attempting to answer these questions:

• Dilution: if you select that both users must be in the test group in order to use the group voice chat feature, there will be very few interactions of interest (i.e very few instances where both users looking to chat are in the test group due to random assignment of test/control users). This will ultimately lead to dilution, as a small fraction of the test group uses the treatment.

• Spillover: if you choose that test users can use the group voice chat feature with users in the control group, then this causes contamination because the control group receives a “dosage” of the treatment. This will also shrink the treatment effect towards zero.

Designing Cluster Experiments

To solve this issue, data scientists often design cluster experiments. Clusters represents groups of users that share similar characteristics (ex: they are friends with each other, have similar interests, have messaged each other in the L30d, etc.). We then randomize users to treatment/control groups based on which cluster they belong to, so that users in the same cluster end up in the same group. This reduces spillover by decreasing the probability that users in the treatment group use the feature with another user in the control group.

Clusters for experiments are typically generated using a graph-partitioning algorithm. We can measure the efficacy of the algorithm using:

• Purity: the fraction (or percentage) of graph edges (or edge weight/metric value) that lie entirely within clusters rather than crossing between clusters. Purity helps us determine how many within-treatment vs between-treatment interactions will occur.
• Intra-cluster correlation coefficient (ICC): how correlated are metric values for users within the same cluster?
  • A non-zero ICC effectively shrinks the experiment’s sample size. For a cluster, the shrinkage is quantified by $1 + (clustersize - 1) * ICC$
  • When ICC = 1, then the units in the cluster are treated as a single unit. Therefore, the effective sample size of a test could be as low as the number of clusters (all ICC = 1.0) or as high as the number of users exposed (all ICC = 0.0).
  • ICC therefore helps us understand statistical power for our experiment.

Considerations

When designing cluster-based experiments there are often three considerations to be mindful of:

1. Clusters grow weaker over time and need to be periodically refreshed.
2. New users aren't assigned to clusters since we lack prior data about their interactions to assign them to one. Depending on the experiment type, we may need to exclude these users or manually assign them to a cluster.
3. Cluster algorithms sometimes result in the creation of a few "whale" clusters. The best practice is to exclude these clusters or to apply post-processing to break them down further.