Topic: Wargame Design Musings
Playtesting...
Introduction: This discussion began over in the ATO folder, but to be fair to those nice gentlemen it was appropriate to move this somewhere else. This discussion was started by a question by David Dockter. The question on the table is many games when they come out often get tagged with having had insufficient playtesting. This was my reply to some of those views.
I am always amused by all the pundits when they wax elogquently on how to produce a really well developed game. At least one person, which soon grows into a chorus, cries that a game did not have enough playtesting, its broken, its ahistorical...etc. As they say in the military, amateurs talk about strategy and professional talk about logistics. Playtesting is a major logistic issue in the production of a game where no one gets paid, there is no reward except for a rules credit, and a free game (I always make sure my testers get a free game...oh boy).
I think characterizing the wargame community as a business is actually the wrong model. The real model is that it is a large Coop. Some folks design, some develop, many playtest, and the farms turn out the goods that people have pre-paid for with others stopping by the stand to pick up surplus produce. I believe that this is a better view of what the industry has become. Proof of this is that there are very few people who make a full time living in this business except at the production end along the lines of graphics, administration, fulfillment of orders, etc. with the rare freelance designer (aka Berg, Raicer) thrown into the mix. In effect what you pay for is only a fraction of the real production cost of the good as the biggest labor costs are done for free or very low pay (e.g., designer/developer). On EoTS I made less than $1 per hour, which is why I am keeping my day job.
When it comes to playtesting, how much is enough. All of my games get extensive playtesting. My last design had 12 teams who played the game around 100 times or so. I got a message from one team that they could not play that weekend because one of the testers house caught fire one night, he was hurt (but not permanently...thank god), and had to walk wounded through the snow a mile or so to a neighbors house to get to a hospital. Many playtesters have their personal lives intrude, but I thought this was the most novel reason for missing a weekend playtest. The footnote was priceless..."but luckily the playtest kit was undamaged, so we should be able to play next weekend." The playtesters worked out the kinks in the mechanics and find all of the obvious issues, gamey tricks etc. There are always a couple of things that remain undiscovered, particularly around balance, which for me on an historical simulation is not the overriding issue...historicity is.
So, what if we had played it another one hundred times...I'm sure we would have found more stuff...but not enough to justify the time and energy from the free labor. The reason that I say this is if one looks at the various ways of calculating 'confidence' in a simulation, which Operations Research types need to understand in my world, it would take, as an estimate several thousand playings before you would achieve a 10% confidence on whether a game had a bias. So 10 groups playing once per weekend (with a two week vacation) would generate 500 playthroughs a year (obviously unobtainable) and in 5-6 years would reach a 10% confidence on whether a game had a bias or not. To achieve a 50% confidence level would take longer than the years I have left in my life.
The way we achieve this in the professional community is we are using computer simulations and we can run them day and night for weeks to achieve the appropriate statistics. In our world, a CDG has much higher combinatorial complexity than most of those models and there is no easy quantitative way of factoriing in player skill. In essence, for all practical purposes, it is not possible.
What happens of course is when a game is released, more post publication playthroughs occur in one week than occured in the entire production cycle. Things crop up...questions get asked...people freak out...and the pundits wax eloquently on how they would have done it better.
This begs the question of what is playtesting for...in essence to wring out the major muscle movements of a design. Now if a design is part of a series, such as OCS, over time the design evolves to improve its mechanics and usually by the third iteration it stabilizes. A more traditional design has less moving combinatorial parts (no cards) so certain issues do not arise making the task a bit more manageable through the traditional processes, but even then I have never seen a design that I do not have questions about (all dimensions), but since I rarely play any game more than twice it is not an issue.
CDGs bring in another layer of probabilities that make determining outcome bias near impossible, so you play the game alot to eliminate system issues. What you want to get right is the mechanics and ensuring that you are driving the history into the design. In essence the game has to work well and be compelling to play. This is not to say that a product gets a by for incorrect set ups (my personal biggest heartburn), poorly proofread components, mistakes on charts etc. A CDG needs to meet the same high level of design and production excellence we all strive for. However, once the public gets a hold of a CDG or any game that gets played alot new unanticipated tactics will arise that may require tweaks to the balance. CDGs are very focused on head to head play vice solo play. Therefore balance has become a major critieria for CDGs due to tournament and internet play. The majority of the games on the market have never had their balance put through the intensity of the CDG experience, there are exceptions, but not many. Consequently it is very difficult to effectively validate a CDGs bias, let alone any game, from normal playtesting alone.
It should become clear that I do not view balance as a design issue, but something that has to evolve over time as people become more adept at playing a particular design, something not seen since the halcyon days of S&T. I have also found that many balance issues are ones of strategy and not mechanics, a debate that is ongoing in regard to my latest design. It appears that the Allies are having difficulties making amphibious invasions while the Japanese still have naval superiority. My answer is first get naval superority...some light bulbs are going on. All in all a very interesting set of dynamics.
Those are some of my thoughts...I'm sure many 'experts' will disagree, but after 40 designs and 30 years this is my current view.
Mark
Postscript: This note started an interesting yet controversial thread. I would note that the responses varied, so if those authors come over, I'm sure they will post their views again. I will note that Twilight Struggle a new system, with only 8 pages of rules, got years of playtesting including numerous conventions. If you take the current view in that folder, the Soviets are viewed, currently, as having a decisive advantage. Again, the nuances of the new system were basically known, numerous strategies were tested, but if you just take the conservative idea that each post represents 1 playthrough, that represents 3500 games that have revealed a new 'truth'. Was the playtesting insufficient, even after years of development? My personal view is it is a very well crafted professional design that is exciting and fun to play yet experiencing exactly what I wrote above. Others will disagree...I'm sure we will hear from them shortly.
Introduction: This discussion began over in the ATO folder, but to be fair to those nice gentlemen it was appropriate to move this somewhere else. This discussion was started by a question by David Dockter. The question on the table is many games when they come out often get tagged with having had insufficient playtesting. This was my reply to some of those views.
I am always amused by all the pundits when they wax elogquently on how to produce a really well developed game. At least one person, which soon grows into a chorus, cries that a game did not have enough playtesting, its broken, its ahistorical...etc. As they say in the military, amateurs talk about strategy and professional talk about logistics. Playtesting is a major logistic issue in the production of a game where no one gets paid, there is no reward except for a rules credit, and a free game (I always make sure my testers get a free game...oh boy).
I think characterizing the wargame community as a business is actually the wrong model. The real model is that it is a large Coop. Some folks design, some develop, many playtest, and the farms turn out the goods that people have pre-paid for with others stopping by the stand to pick up surplus produce. I believe that this is a better view of what the industry has become. Proof of this is that there are very few people who make a full time living in this business except at the production end along the lines of graphics, administration, fulfillment of orders, etc. with the rare freelance designer (aka Berg, Raicer) thrown into the mix. In effect what you pay for is only a fraction of the real production cost of the good as the biggest labor costs are done for free or very low pay (e.g., designer/developer). On EoTS I made less than $1 per hour, which is why I am keeping my day job.
When it comes to playtesting, how much is enough. All of my games get extensive playtesting. My last design had 12 teams who played the game around 100 times or so. I got a message from one team that they could not play that weekend because one of the testers house caught fire one night, he was hurt (but not permanently...thank god), and had to walk wounded through the snow a mile or so to a neighbors house to get to a hospital. Many playtesters have their personal lives intrude, but I thought this was the most novel reason for missing a weekend playtest. The footnote was priceless..."but luckily the playtest kit was undamaged, so we should be able to play next weekend." The playtesters worked out the kinks in the mechanics and find all of the obvious issues, gamey tricks etc. There are always a couple of things that remain undiscovered, particularly around balance, which for me on an historical simulation is not the overriding issue...historicity is.
So, what if we had played it another one hundred times...I'm sure we would have found more stuff...but not enough to justify the time and energy from the free labor. The reason that I say this is if one looks at the various ways of calculating 'confidence' in a simulation, which Operations Research types need to understand in my world, it would take, as an estimate several thousand playings before you would achieve a 10% confidence on whether a game had a bias. So 10 groups playing once per weekend (with a two week vacation) would generate 500 playthroughs a year (obviously unobtainable) and in 5-6 years would reach a 10% confidence on whether a game had a bias or not. To achieve a 50% confidence level would take longer than the years I have left in my life.
The way we achieve this in the professional community is we are using computer simulations and we can run them day and night for weeks to achieve the appropriate statistics. In our world, a CDG has much higher combinatorial complexity than most of those models and there is no easy quantitative way of factoriing in player skill. In essence, for all practical purposes, it is not possible.
What happens of course is when a game is released, more post publication playthroughs occur in one week than occured in the entire production cycle. Things crop up...questions get asked...people freak out...and the pundits wax eloquently on how they would have done it better.
This begs the question of what is playtesting for...in essence to wring out the major muscle movements of a design. Now if a design is part of a series, such as OCS, over time the design evolves to improve its mechanics and usually by the third iteration it stabilizes. A more traditional design has less moving combinatorial parts (no cards) so certain issues do not arise making the task a bit more manageable through the traditional processes, but even then I have never seen a design that I do not have questions about (all dimensions), but since I rarely play any game more than twice it is not an issue.
CDGs bring in another layer of probabilities that make determining outcome bias near impossible, so you play the game alot to eliminate system issues. What you want to get right is the mechanics and ensuring that you are driving the history into the design. In essence the game has to work well and be compelling to play. This is not to say that a product gets a by for incorrect set ups (my personal biggest heartburn), poorly proofread components, mistakes on charts etc. A CDG needs to meet the same high level of design and production excellence we all strive for. However, once the public gets a hold of a CDG or any game that gets played alot new unanticipated tactics will arise that may require tweaks to the balance. CDGs are very focused on head to head play vice solo play. Therefore balance has become a major critieria for CDGs due to tournament and internet play. The majority of the games on the market have never had their balance put through the intensity of the CDG experience, there are exceptions, but not many. Consequently it is very difficult to effectively validate a CDGs bias, let alone any game, from normal playtesting alone.
It should become clear that I do not view balance as a design issue, but something that has to evolve over time as people become more adept at playing a particular design, something not seen since the halcyon days of S&T. I have also found that many balance issues are ones of strategy and not mechanics, a debate that is ongoing in regard to my latest design. It appears that the Allies are having difficulties making amphibious invasions while the Japanese still have naval superiority. My answer is first get naval superority...some light bulbs are going on. All in all a very interesting set of dynamics.
Those are some of my thoughts...I'm sure many 'experts' will disagree, but after 40 designs and 30 years this is my current view.
Mark
Postscript: This note started an interesting yet controversial thread. I would note that the responses varied, so if those authors come over, I'm sure they will post their views again. I will note that Twilight Struggle a new system, with only 8 pages of rules, got years of playtesting including numerous conventions. If you take the current view in that folder, the Soviets are viewed, currently, as having a decisive advantage. Again, the nuances of the new system were basically known, numerous strategies were tested, but if you just take the conservative idea that each post represents 1 playthrough, that represents 3500 games that have revealed a new 'truth'. Was the playtesting insufficient, even after years of development? My personal view is it is a very well crafted professional design that is exciting and fun to play yet experiencing exactly what I wrote above. Others will disagree...I'm sure we will hear from them shortly.
Posted by markherman
at 1:57 AM EDT
