Advice from the QA Manager, Crytek-testing games that require regular updates. The longer lives project, the more critical accidental bugs. Sudden errors can alienate a significant part of the audience.
QA Manager, Crytek Evgeny Skachkov shared at the conference Games for the 2016 Gathering experience in quality control of the project Warface, developed by the Kiev branch of the company. Edition DTF published a transcript of the speech. Warface is an online free-to-play shooter.
Almost five years, the game is at the stage of operation and more than eight years in development. It is the largest free-to-play shooter in the CIS. Three years ago we set the officially registered Guinness record — 145 thousand users online on a single server. If you count all the territories PCU (peak concurrent users) more — about 200 thousand people.
Every day in the game go about 700 thousand unique users. Every year we issue about 12 major updates. How does the development of conventional, “boxed” product. Programmers and designers create something, come up with creative, prototypers and at some point decide that the game is ready.
It is necessary to test. Something theyre checking on their own, but its not enough. Going the QA team, and begins the “zerg rush”. Resorts crowd of QA-professionals “breaks” their wonderful product to shreds.
Panic sets in. It turns out that the game is impossible to beat, thats it, not working. The developers working on half of the blocking bugs, sell the game, releasing patches and earn some money. But what if your game is a service.
First, this approach does not work. After the first such updates will be less than half the audience. And releasing two unsuccessful updates, the project can close. What characterized the game as a service.
Here you always first create the design before you start developing. Typically used iterative development to follow the market demands, faster to respond to them. At each stage you test the project. Do you regularly (once a month) go out content updates.
And this is quite often. Every month need to give a good product. You have a very high cost of the blocking bugs. If they get in the final version, you are guaranteed to lose part of the audience. At the same time you cant afford to just constantly working nights and weekends for long periods in this mode, your team just “burn”.
We use a three-level “defense”. Early detection of errors. The sooner a bug is found the cheaper it is to repair.
This is a highly specialised professional testers. They can check and related areas, but clearly sharpened only under a single and integrated a team of developers at the earliest stages. The discussion of the design, it is already necessarily involved the QA technicians. They are testing the first build, always plan.
We call it plan draft — small lists of what you need to check for every key features. The execution stage includes writing test cases, testing. Cross-review — the process in which it is desirable that for each element, test case and test process followed people from related teams.
Holistic testing. We have very strict criteria to confirm the changes. Not having a check-list, no change will be in code. Test cases is an important part of the project.
Previously, you had something to test without them, but now its impossible. We have more than ten thousand test cases. If you have something youve tested and think think about it three years later — most likely, it is not. Strict actualization.
If you add some feature, it must be included in the test cases. Excellent documentation — without it anywhere. It is updated in the same way as test cases. For new features can be created from scratch, to update old — actualizarea.
Checklists bullet proof. There is no sense each time to describe the content of a test case. He is versatile, works equally. You just need to check whether the regular “cannon” a number of criteria.
Postintegration regression is the most important element. After adding the features you need to check whether it is integrated as intended, and whether the components that it could affect. We have iterative development, and at the end of each iteration were going with the guys from all teams and they will present their features.
This allows you to respond more effectively to bugs. Prior to that, there are situations when a person for several releases didnt know that something has changed. In order to make it work, you need a high-quality QA is a process that I briefly described. In the diagram two phases — development of features and its stabilization.
In the second phase, the QA specialist checks all new items, all that they could affect, and some critical things that you shouldnt check. Code lock means that the stabilization time nothing is added, except bug fixes. The criterion of completion of stabilization is the lack of a blocking error.
All this is tested. Not so scary, unless you consider that the territories where few. And all bugs appear that need to be corrected. A lot of work here.
The second team should enter a very hard-working QA testers. The main selection criterion is the love of strangers passing test cases (because they most likely never will not write). They also must love to investigate complex bugs. A user wrote “my game is not working” — we need to understand why.
This part of the “defense” is a link between the development and operation of. All the bugs and panicked shouting about them on the forums they need to turn into normal problems and add them to JIRA. Have Integration specialists a ton of tasks for regression testing. Its a pain, because they always do the same thing, at the same time as the final element before the release of the project in operation.
How to help them?. We try to automate everything you can. When you have to play 200 types of firearms, it is impossible to check every time, not broken anything. And it will break.
Therefore, there are automated scripts that verify file and other aspects. Testing tools. The more (useful, of course) — the better. Log analysis, telemetry, and console commands which help to speed up the process.
Also important testability the ability to test the item. If your programmer tells you that something is impossible to check, let them first prove. Stress testing is also automatiseret. You will never gain command on MMO free-to-play-shooter make for stress-testing office.
Performance testing automate the most tedious part of the job. QA since all the companies are financed by a residual principle, the fewer specialists, the cheaper and better. Therefore it is necessary to increase their effectiveness and strive for the ideal. Every employee should be a perfect QA.
Your lid should raise professionals and not just monitor their work. To help, to suggest, to always listen to the questions, be polite and kind. Regular meetings.
On them the QA technicians present their area of responsibility. As to improve the efficiency of the employee with the right specialization can be a situation where he will be absent, and nobody will be able to do the work for him. So that such meetings not only increase the General expertise and willingness, but also slightly reduce the risk that the QA will be no one to replace. Two more rules.
If you have QA-specialsit who loves to test the backend — dont force it to check the levels. And conduct more training and educational activities. The main KPI is the number of blocking bugs caught in the final version. All the rest is secondary.
It is necessary to analyze the blocking bugs, “broken” in the public version at all levels of “defense”. Improve test cases, test plans, process and skills. Hearts on the scheme — it features.
There are releases in which their lot or a little. In the first case, our departments of sales and marketing happy — a lot of features easy to promote and sell. In the second case, things are worse. However, in the first embodiment, there is less time for testing.
And somewhere you compromise. Most likely, some features not working, which often leads to the fact that stops working and the release. Business departments are not so happy and are pulling their hair out. This is the last rule.
Smaller is better. If you want to write the material for the rubric “the Market” tell us about the development of your game or in case of its growth to send the material on [email protected]