Constructing an A/B testing evaluation framework for cell gaming on Databricks

Introduction

Mobile sport studios rely on steady experimentation to refine gameplay, monetisation, and stay operations. As experimentation scales, evaluation usually turns into the limiting issue. Results are sometimes stitched collectively manually, statistical approaches range by analyst, and insights arrive days after key indicators emerge. Over time, this creates friction: slower iteration, inconsistent conclusions, and declining confidence in A/B testing as a dependable determination software. 

The Challenge

At HARDlight, the problem was not simply pace, however belief. Different approaches led to totally different interpretations, making alignment tougher and weakening confidence in experimentation as a scientific determination software. Some stakeholders wanted a easy day by day standing, others wished to grasp participant behaviour or enterprise affect, and a smaller group required deep validation of particular sport levers. The current dashboards and reviews struggled to serve this full spectrum of wants successfully. For experimentation to scale, HARDlight wanted a method to standardise inference, make outcomes accessible at totally different ranges of depth, and rebuild belief in A/B testing as a shared, scientific determination course of. 

To handle this, HARDlight constructed a Databricks-native A/B testing evaluation framework that automates the trail from experiment knowledge to decision-ready perception. Statistical evaluation was carried out upstream in a repeatable, clear means, and Databricks AI/BI surfaced the outcomes by way of a daily-refresh expertise that started with an LLM-generated abstract and permits deeper exploration with progressively granular views. At the top of every experiment, outcomes have been frozen and preserved, guaranteeing choices, context, and learnings stay accessible lengthy after the check concludes.

The Solution: Automated A/B Testing on Databricks

HARDlight’s framework automates experimentation from ingestion by way of to determination help. Within Databricks, experiment definitions and telemetry are standardised, statistical modelling is utilized persistently, and outcomes are printed to a layered dashboard that refreshes day by day through the run window. An LLM abstract on the prime gives an accessible view of experiment standing, whereas deeper sections expose KPIs, diagnostics, and really helpful actions for knowledgeable customers.

The alternative of Databricks allows governance and repeatability throughout groups. Unity Catalog gives a single management aircraft for permissions and lineage of experiment property; Spark Declarative Pipelines orchestrates dependable pipelines for experiment ingestion and transformations; and MLflow helps experiment monitoring and mannequin packaging for reproducible evaluation. Together, these capabilities preserve knowledge and analytics ruled, constant and simple to function within the Lakehouse.

A key innovation is the “frozen dashboard” on the finish of the run. Instead of rolling on to the subsequent refresh, the framework preserves the ultimate snapshot and the choices taken, together with really helpful actions. This institutionalises learnings from previous experiments and permits stakeholders to revisit outcomes with out ambiguity.

Technical Architecture

The experimentation framework is constructed as a Databricks-native system that separates knowledge processing, statistical inference, and consumption, whereas maintaining all outputs ruled and reproducible by default. This design ensures analytical rigor scales with out growing operational overhead or fragmenting interpretation throughout groups.

Data Ingestion & Modelling

Experiment definitions, participant telemetry, and end result metrics are ingested from inside pipelines and curated into ruled tables with constant schemas. This standardisation permits analysts and product groups to motive about experiments persistently, no matter check design or length. Notebooks are used to compute statistical fashions that calculate impact estimates, uncertainties and section stage impacts over time. Rather than embedding logic in dashboards or reviews, all analytical outputs are materialised right into a unified experiment analytics mannequin. This creates a steady semantic layer that downstream customers can depend on with out re-running evaluation or reinterpreting outcomes.

AI/BI-powered Insight Delivery

On prime of this ruled analytics layer, Databricks AI/BI gives an accessible interface for consuming experiment outcomes. Each day by day refresh generates a succinct LLM abstract aimed toward non-technical stakeholders, translating validated statistical outputs into pure language. The dashboard makes use of progressive disclosure: customers can cease on the abstract when happy, or discover deeper layers of metrics, diagnostics, and section evaluation as their curiosity will increase. This layered expertise allows speedy scanning whereas maintaining analytical depth accessible for knowledgeable validation.

Experiment Lifecycle and Persistence

During the stay section, the dashboard refreshes day by day so groups can observe trajectory and react to indicators. At the conclusion, the dashboard freezes to protect outcomes, choices and really helpful actions. This lifecycle creates an auditable report that accelerates onboarding and reduces duplicated evaluation throughout future experiments.

Dashboard Layers Explained

The dashboard is designed to information customers by way of an experiment’s leads to a transparent, deliberate sequence. It begins with simplicity and steadily unveils extra element for these excited about exploring additional. Each part addresses a distinct query, and it’s completely acceptable to cease as soon as the reader has obtained the required info.

LLM-generated experiment abstract: At the highest of the dashboard is an LLM-generated abstract. While an experiment is stay, this provides a easy, high-level view of how issues are going, highlighting early indicators with out drawing untimely conclusions.

Once the experiment concludes, the abstract modifications function. It turns into a transparent rationalization of what occurred, calling out the metrics that moved with excessive confidence, in precedence order, and in plain language. The purpose is to assist groups shortly perceive the result and why it issues.

Confirmed outcomes and statistical affect: For extra technical audiences, the subsequent part presents a structured view of statistically important outcomes. Key metrics reminiscent of participant lifetime worth (LTV) and retention are listed alongside impact sizes and confidence ranges, making it straightforward to validate conclusions with out digging into uncooked evaluation.

Predicted lifetime worth affect: The dashboard then exhibits the estimated affect on participant lifetime worth for management and variant teams. Uncertainty and error margins are proven explicitly, reinforcing that these are knowledgeable estimates, not absolute forecasts.

Revenue affect by supply: Results are damaged down by income stream, together with advertisements, in-app purchases, and complete income. This helps groups perceive whether or not modifications are broad-based or pushed by particular monetisation channels.

Player engagement and behavior: Beyond income, engagement metrics reminiscent of retention and session behaviour are surfaced to make sure enterprise positive aspects are thought-about alongside participant expertise and long-term well being.

Segment-level evaluation: Segmentation is central to how HARDlight designs and evaluates experiments. This part exhibits how totally different participant segments reply to a change, whether or not outlined by retention, development, or different behavioural traits. It helps groups verify that focused experiences work as supposed, with out harming different elements of the participant base.

Monetisation mechanics and sport economic system: Deeper layers discover how experiments have an effect on in-game techniques, together with advert efficiency by placement, In App Purchase efficiency by product class, and modifications to arduous and tender forex flows throughout sources and sinks.

Core gameplay loops and appendices: At the deepest stage, detailed charts and tables cowl gameplay mechanics reminiscent of races, characters, and gadgets, together with supporting statistical visuals. This layer is meant for knowledgeable customers who need full transparency or have to reuse insights in future work.

Together, these layers let perception unfold naturally. Teams can transfer shortly when the reply is obvious, or go deeper when questions come up, all whereas working from the identical ruled, trusted supply of information. 

This construction is made potential by Databricks AI/BI, which permits complicated analytical outputs to be surfaced cleanly with out embedding customized code or analyst-only workflows into dashboards. Statistical outcomes, projections, and segment-level analyses are computed upstream in notebooks and materialised into ruled tables, whereas AI/BI gives a versatile presentation layer on prime. This removes the necessity to run Python inside dashboards, simplifies upkeep, and makes it possible for a lean workforce to iterate on and evolve the system over time.

Just as importantly, AI/BI makes it potential to serve very totally different audiences from the identical underlying knowledge. Narrative summaries, tabular outcomes, charts, and deep diagnostics can coexist with out duplicating logic or fragmenting interpretation. This was a key shift from earlier approaches, the place tooling constraints pressured trade-offs between analytical depth, accessibility, and sustainability.

Impact & Results

The framework has basically modified how experimentation operates at HARDlight. By automating evaluation and standardising statistical inference, the info workforce has lowered guide effort by greater than eight hours per week. By standardising experiment runs with Databricks Workflows, the workforce eradicated a lot of the guide setup work beforehand required for every evaluation. This saves roughly someday per experiment and has enabled a focused two-times enhance in month-to-month A/B testing capability with out growing headcount.

Manual Experiment Analysis Workflow:

Automated Experiment Insight Delivery on Databricks:

Beyond effectivity positive aspects, the system has improved consistency and confidence in outcomes. The frozen dashboard archive now acts as a sturdy supply of reality for accomplished experiments, decreasing repeated evaluation and making it simpler for groups to revisit previous choices with full context. This has considerably lowered the overhead of sustaining historic data throughout groups.

Perhaps most significantly, the framework has modified how insights are consumed throughout the studio. With a number of experiments operating in parallel, groups now obtain day by day, AI/BI-enabled updates that substitute multi-day guide aggregation and interpretation. Genie can be enabled instantly on the dashboard, permitting customers to ask questions on what they’re seeing and discover leads to their very own phrases, with no need to grasp the underlying knowledge mannequin. Together, clear summaries, ruled metrics, clear statistical outputs, and conversational entry have helped construct belief throughout product, LiveOps, and engineering groups, reinforcing experimentation as a shared, scientific means of working.

What’s Next

HARDlight plans to increase the framework with a forecasting software, extending the framework from descriptive and inferential analytics into forward-looking steerage. The broader imaginative and prescient is predictive experimentation and closed-loop optimisation — utilizing the Lakehouse to automate extra of the cycle from speculation to deployment, whereas preserving governance and consistency with Unity Catalog, Spark Declarative Pipelines and MLflow. This dashboard-first strategy can have important affect for different studios with comparable wants, layering LLM summaries over ruled metrics and diagnostics to scale experimentation with confidence on Databricks.