Burn Your Analytics

I daily read articles on online promotion, marketing promotions, A/B testing and other similar pieces. And regularly run into a complete lack of understanding the authors influence on the results is small, but very important part called “dispersion”. Meanwhile, it can completely destroy the entire logic of the study and even to produce results directly the opposite of what happened actually. In order to seriously talk about the variance, will have to give it definition. This will be the only difficult paragraph in the article, I promise.

In the online service castlots.org we “planted” five hundred times. The result. 54/46 in favor of the “eagle.”.

What is intuitive with a coin, for some reason ceases to be perceived, when it comes to other processes where the randomness is also present. Let me now describe typical and probably the most frequent error of this type, and then try to apply it to your business. Imagine entrepreneur Ivan, who gives advertising on Google to attract buyers in the online store. He set up the ad, paid for his. The campaign works.

One click on the ad to Ivan on the average from 1 to 2 dollars, and the profit from each sale is $ 150. We were very hungry when filled in this column on Spark.ru. 10 shows.

Sales yet. 300 impressions. 500 impressions. By this time two sales.

Ivan has spent $ 750. 500 impressions. Five armor. The conclusion – uh, this campaign two and a half times more efficient than the previous. Fill there more money.

100 views, 500. 1000 impressions. 4 armor. The result suddenly fell, to the last campaign. Damn, it stopped working.

Where the logical error of Ivan. Ivan took a decision about the effectiveness of advertising on the basis of a too small sample. And dispersion, which is definitely present in the process, deceived him. Here is an example of “potential trees”, and our business.

Our partners in the US advertise their quests on Google. Click price for them is about 30 cents, and the average sale brings $ 25. The conversion rate from visitor to customer is about 3%, i.e. Each visitor to the site with a probability of 3% becomes a buyer. We buy traffic in 100 people and lead them to the site.

The is as it suggests about our international expansion. Might look like timeline for this process. Will look like our actual profits (or losses). I did not postpone it indefinitely and with the help of Excel to visualize the results of this experiment.

I conducted four experiments and such, I turned the graphics. Variability in random processes may exceed ten times.

Once again I draw your attention – this is the same process. All parameters are of equal. The trouble. Whats wrong.

The problem in the system. Maybe 3% conversion rate is a little. Immediately warn – theoretically everything is fine. Weve built the system has a positive expectation (I will not explain what it is and why it is so, just take it on faith). The problem is that we used too small a sample.

A hundred tests, the influence of dispersion is so great that we can get dramatically different results. We again went to the simulator and threw five rubles a hundred times. The result.

49/51 in favor.. Tails. Lets increase the distance. So, four of the experiment, this time for 1000 trials each:

Seemingly, all is not so bad. The best and worst results differ by only twice. Lets go to the limit and spend 10,000 tests.

Well, finally. In four experiments we obtained a scatter of only 10%. 10,000 tests. The same random process.

And the spread is 10%. I tell you a secret that for perfect coincidence of the results to approximately 100,000 tests. When you evaluate the result of a certain process with a low probability of success, for example, design changes on site with conversion of the 3%, it is necessary to use a LARGE enough sample of tests to trust the final result.

As can be seen from the examples above, the sample must be in the region of 100,000 tests. If you are trying to understand how a text or an image on the landing page affect sales, it is necessary to conduct a sufficient number of trials. It is important that this rule works also outside of the Internet. If you distribute flyers in the city or glue posters, they should be distributed, and to glue a lot.

I suspect that the conversion of the flyers in the sale will be much less than 3%, and therefore the amount of 10 – 30 thousand pieces – the minimum threshold at which you can evaluate the effectiveness of the channel. Everything else is random. Dispersion.

Photo. Snob.ru. The higher the probability of your desired result, the smaller the sample needed to evaluate the result. For example, if you are testing on the website buttons in two colors, and one gives the conversion is 40% more sampling of 10,000 trials will, I think, quite.

Now think of all the thousands of studies and examples that you have seen online where this rule fails. And safely send them to the trash. You see, make space for something really useful. Artem Kramin, founder of the network “to Leave the room”. Cover photo.

Entrepreneurs underestimated the power of dispersion, protesting in the streets of Asuncion, Paraguay. (C) www.thequint.com.

Leave a Reply