Red Dead Redemption 2 is on track to be one of the highest rated games of all time. It’s a masterpiece of open world storytelling, and the largest weekend entertainment release ever. It was also a gruelling project to work on, according to recent reports. And while we can separate the art from the controversy of the artist, it’s worrying when a product can be held up as positive evidence for a broken process.
What seems to unfortunately be taken as accepted wisdom in this industry-wide discussion is that crunch hours are necessary when making a game of Red Dead Redemption 2’s quality. But the most exhaustive study done on industry crunch refutes this notion.
The Game Outcomes Project, conducted in 2015, is required reading for anyone interested in productivity in game development. The five-part series looked at qualitative and quantitative measures of success, surveying hundreds of game developers with 120 questions. Part 4 of the findings weighed project outcomes against how much their development teams relied on crunch. Given the media attention currently given to industry crunch, and calls for a game developer union, the study deserves a lot more attention than it’s received.
Crunch is a misleading term when talking about Red Dead Redemption 2, however. As Kotaku noted when talking to current and former employees, many teams at Rockstar have been “crunching” since August 2017. For some, this has meant 60-80 hour weeks.
So let’s put the Game Outcomes lens over the production of the biggest game of 2018.
Crunch Correlates Negatively With Success Metrics
We reached out to Paul Tozour, the lead on the Game Outcomes Project, to learn what the most up-to-date research says about Rockstar’s practices. On behalf of the Project, he provided a statement (read it in full at the bottom of this post).
“Studios that rely on crunch often claim that in the long run, those sacrifices pay off in terms of increased productivity and improved product quality,” according to Tozour and the Project. “However, the results of our 2015 study called that into question, and showed that the studios that crunched the most generally ended up with products that earned less revenue, got lower MetaCritic review scores, shipped later, and were generally lower-quality games than you would otherwise expect.
“This reinforces the findings of management science that fatigued, overworked employees are less productive and more likely to make bad decisions and expensive mistakes — particularly in a very challenging creative and technical field such as game development.”
This aligns with management studies done in the past. After a few weeks of 50 hours per week, cumulative output goes below what it would have been working only 40 hours per week.
The study linked above observes that crunch used in small doses can briefly increase productivity, with an equal recovery period afterwards. But the Game Outcomes Project, specifically surveying game developers, observed no beneficial “hormesis,” as it calls it.
We’re talking about a different kind of work to the physical exertion of Henry Ford’s experiments that resulted in the 40-hour work week. There may not be manual labour involved, but the detrimental effects of crunch – especially that of sleep deprivation – actually have a greater effect on game development, which is heavy on creativity and problem-solving.
“There’s incontrovertible evidence that extended overtime has damaging effects on the mental and physical health and morale of individual employees, increasing the likelihood of depression, relationship problems, substance abuse, and much more,” says Tozour.
“On top of that, managers who focus on simplistic solutions like crunch are much less likely to properly diagnose and correct much more fundamental problems of workflow, morale, team structure, vision, or other cultural issues that fundamentally impede their teams’ productivity.”
Players and companies alike have accepted this mythological pairing of unhealthy hours and game quality. It’s an incorrect but easily transmitted idea that there’s a “human cost” to making a game as good as Red Dead Redemption 2. But the data shows the opposite. Red Dead Redemption 2 is the exception to the rule — which is why the success of Red Dead Redemption 2 is dangerous.
The Importance of Seeming Earnest
Rockstar apologists will talk about the unique nature of Red Dead Redemption 2. Here’s a game that’s not just good, it’s uniquely good. Surely the attention to detail in Rockstar’s Western comes at a cost?
There’s an industry-wide fallacy that you can draw a straight line from more effort to more reward. The data shows that more “effort” (defined as working more) is actually counterproductive. Recall that a few 50-hour weeks results in a net productivity loss. A Rockstar document shown to Kotaku had marked employees in one team as “under” if they hadn’t met 60 hours for the week. The policy doesn’t match the science.
Even when we humans willingly work overtime, it’s detrimental. We’re biased in a way that makes us think we’re producing more, when it’s in fact less. That’s assuming people strive for effectiveness, which isn’t always the case…
While our peeks into how Rockstar’s sausage is made show passionate, enthusiastic developers, reports from inside the various Rockstar studios also describe a culture of meaningless busywork.
Current and former employees report that presence had priority over productivity — it was important to simply “be there,” whether there was work to be done or not, to secure a yearly bonus.
Of course it’s a huge company with more than a thousand employees worldwide, and any individual’s experience will come down to the local culture and direct manager. Many spoke about effective immediate managers who genuinely cared. But there are enough independent accounts in agreement, describing a company that cares about hours spent more than anything, to think there’s nothing necessary about this process.
Sealed With A Letterbox
While the long hours are more of a consistent culture problem than any last-minute dash to the finish line, the final stages of crunch were reportedly due to the decision to include black bars on the top and bottom of all cutscenes. This letterboxing lets the player know they’re in watching mode, not playing mode. It’s a feature that adds to the game. But it also added to the stress of those who made it.
With crunch such a prevalent problem in the industry, it’s entirely possible – though hard to reconcile – that two things might simultaneously be true:
- Red Dead Redemption 2 is a uniquely amazing game. An achievement that many will come to call one of their favourite games ever.
- Red Dead Redemption 2, as a project, was mismanaged and poorly planned.
There are many unknowns when developing a game. Rare is the project in which an auteur can slap a coherent production schedule onto their vision. What actually happens is developers prototype, find the fun, and go through multiple iterations and improvements. But unknowns can be planned for — especially in the case of a sequel.
— Game Outcomes (@GameOutcomes) December 27, 2015
Rockstar is a company famous for throwing things out, starting again, and achieving its idea of perfection before release. The development of RDR2 took eight years. Other projects were increasingly fraught — at one point a majority of Sydney developers had gone through Team Bondi’s revolving door, as it repeatedly threw out and remade L.A. Noire.
These luxuries are available when you own some of the most lucrative games of all time. One resource – the employee – becomes less important with the abundance of other resources. High turnover, burnout, health risks… These aren’t the worries of a studio that can snap its fingers to fill a role.
Any kind of enthusiast media suffers from the idea that we should feel “lucky” to participate. There’s a long line of people waiting to take your place. It might even be true, but it doesn’t take a genius to see how high turnover can affect a project negatively.
Rockstar, like many, have been coasting along with this idea that crunch culture is okay because this is the done thing in game development. Seemingly unaware of the data, it’s only hurting itself. How much better, or how much quicker, could Red Dead Redemption 2 have been developed if modern management science had been observed?
And how will Rockstar cope in a world with game developer unions and enforced limits to hours? Because we can already hear the beating of the drums, and every release like this takes us one step closer.
A Statement From The Game Outcomes Project
Below is the full statement from the Game Outcomes Project:
On most professional game engineering teams, if a programmer says, “this particular part of the code is slow and we need to optimize it,” your engineering lead is likely to say, “OK, show me the profiler results that prove that.” Experienced developers know that intuition is not a good guide to optimization, and you need to let your code optimization efforts be guided by empirical data.
The surprising thing is that on the management side, we don’t always apply a similar level of discipline to optimizing our teams and our work processes that we apply to our code. Managers in game development are often guided by intuition and often end up optimizing the wrong things. We frequently apply the wrong interventions without really understanding the problems on our teams, or attribute our past successes to the wrong things based on limited data, causing us to repeat those mistakes on the next project.
Although there’s a huge variety of teams in the industry, and some teams are very well-managed, there are also many that don’t adequately measure their productivity and act as if the results of the last several decades of management science do not exist or do not apply to them.
There’s incontrovertible evidence that extended overtime has damaging effects on the mental and physical health and morale of individual employees, increasing the likelihood of depression, relationship problems, substance abuse, and much more.
Studios that rely on crunch often claim that in the long run, those sacrifices pay off in terms of increased productivity and improved product quality. However, the results of our 2015 study called that into question, and showed that the studios that crunched the most generally ended up with products that earned less revenue, got lower MetaCritic review scores, shipped later, and were generally lower-quality games than you would otherwise expect.
This reinforces the findings of management science that fatigued, overworked employees are less productive and more likely to make bad decisions and expensive mistakes — particularly in a very challenging creative and technical field such as game development.
On top of that, managers who focus on simplistic solutions like crunch are much less likely to properly diagnose and correct much more fundamental problems of workflow, morale, team structure, vision, or other cultural issues that fundamentally impede their teams’ productivity.
We are planning a new, improved version of the Game Outcomes Project study in late 2018. Our hope is that in the long run, we can help nudge the industry toward a more empirical approach to management, where we continue to mine deep insights by comparing the results of hundreds of teams across the industry, and continue to build on the results of decades of management science to help make the game industry a healthier place that makes happier customers with less friction, less chaos, and less burnout.
See more about the Game Outcomes methodology here.
Round 2 of the Game Outcomes Project survey launches later this year! The 2018 version will also support French, German, Spanish, Japanese, and Chinese.
Sign up for our new mailing list to be notified when the survey or its results are available –> https://t.co/mHAyRXTj04
— Game Outcomes (@GameOutcomes) October 12, 2018