Jagex's choice-based conjoint analysis nightmare

Published by Craig F. Kolb in Implementing theory · 9 February 2026
Tags: Jagex, conjoint, analysis, better, practice

In early 2025 Jagex, a video game software developer, met with angry backlash from customers in response to a choice-based conjoint analysis (CBC) survey evaluating different subscription bundles.

Themes on forums - such as Reddit - included perceptions of exploitation, poor value, absurdity, confusion and corporate insincerity in response to the backlash.

After evaluating user feedback and comments, it became apparent that this situation could have been avoided. What can we learn from Jagex's misfortune and what could we do to avoid similar situations? My suggestions would be:

Use metric conjoint analysis instead.

Metric conjoint’s use of a purchase probability for each profile, allows respondents to select ‘zero chance’ / ‘zero probability’ or something similar as opposed to being forced to choose between bad options. Although they could eventually reject the alternatives, they were first forced to select one. As one player stated:

“...after selecting an option you also had the ability to say you would cancel instead if given these options.” Here is one of the screenshots provided by the user:

You will notice the large amount of text survey respondents were required to read through just to provide one data point - a choice. This was repeated again and again across multiple choice sets; typical of a choice-based conjoint design. In contrast, metric conjoint analysis would be far less burdensome, as a datapoint is collected for each profile, meaning less profiles are needed to be able to estimate attribute-level parameters.

Keep price ranges reasonable, instead of going to extremes that your customers are not used to. This is especially true of the main-effects designs common with this method - as the average effect of prices across the various profiles is estimated, and ignores price interactions with the features offered. That’s fine if you focus on a specific product segment, where the range of prices shown is a realistic possibility in combination with the available attribute levels. However in the Jagex case, some options went over $30 a month (as of today, the real prices don’t exceed $13.99 a month). While it is generally good practice to go a little outside of the expected price range, in case of inflation or new features, going so far above is going to seem unrealistic.
Avoid exploitative combinations. While experimental design generators are happy to generate combinations that could be seen as predatory (because statistically they keep correlations / dependence low) humans won't be so happy. Extremely high prices combined with base feature combinations led to a feeling that management was attempting to exploit customers. One noteable example was the inclusion of ads with paid options. As one Reddit user stated “The fact they're even considering ADS for any of these pricing models is ridiculous”. Many others seemed to share similar sentiments.
Consider not maximizing revenues. While the textbook approach is 'maximization' of one or more criteria such as share, revenue or profit; lately I've begun to realize that (in particular) revenue / profit maximization doesn't follow the principle of "Do to others as you would have them do to you" especially in situations of inelastic or near inelastic demand - e.g. markets where private equity 'roll ups' have led to oligopolies / monopolies. While a company with inelastic demand could raise prices and gain revenue initially - I believe it may make the company more vulnerable to future competitor actions, especially if price changes were perceived as exploitative for the specific market context.
Make the currency clear. One user expressed confusion over whether prices were denominated in Canadian dollars or not.
Make it clear upfront that just because there may be more expensive options for some players, it doesn’t mean more affordable options disappear. You could make it clear you are considering offering a ‘portfolio’ or ‘range’ of plans, each designed to suit different customers, including an entirely free ads-based option. You don’t want respondents in such a case feeling as one option will win and that’s the end of it.
Ditch the decorations / graphics, especially since they don’t improve clarity or make it easier to process information on the different levels. If you do add graphics to represent levels, you need to be careful not to cause an artificial attribute-importance inflation by only illustrating levels on one of the attributes.
Too many attributes and levels. Focus on the attributes you actually think you are able to change, don’t throw in every possible attribute ‘just in case’. In this way you keep the research focused, not only for respondents but management who must wade through the results afterwards.
Be especially careful with a networked customer base. In this case, the customer base was networked via forums and so any dissatisfaction or misunderstanding from survey participants is going spread to customers who had not participated.