Significance of Statistically Significant Results in A/B Testing

You are running an A/B test (or multivariate test) but are in a hurry to make a decision to pick the winning page. Should you pick a winner based on initial few days of the data before your tool has actually declared a clear winner? This question comes up quite often during the conversations with the clients.

I always warn against such an approach and advice to be patient and get statistically significant results before pulling a plug on an underperforming variation o declaring a winner. I wanted to share few graphs with you to illustrate my point.

The following example is from a test that I am currently running.
First Two Weeks: If you make a decision without completely getting through the test, you will pick �Yellow� as the winner while declaring �Blue� a loser.

Month Later: There does not seem to be a clear winner.

Few More Days Later: Seems like �Blue� is trending higher. Should you pick �Blue� now?

Finally: We don�t have a clear winner yet.

As you can see, picking a winner or dropping a loser in the early stages of test, without having a statistical significant result, would have been a wrong decision.

Comments? Questions?

---

Looking to fill your Web Analytics or Online Marketing position?

Post your open jobs on Web Analytics Job Board
--------------------------------------------------------------------------

Are Form Validations Invalidating Your Conversions?

Form validations are a great way to ensure that the data you collect from your users is clean and in the right format. However form validations can also frequently lead to lost conversions. CAPTCHA is one such type of validation.

Some validation put unnecessary formatting burden on your customer/visitor leading them to abandon your forms and go to your competitors. A lot of data formatting can be done via client side JavaScript or backend processing without putting the customer through a lot of pain. I am not suggesting to remove all required data and data formatting requirement, all I am saying is that be if you can handled formatting with scripts/code then do not make your customer do it. Let the users finish the form.

Your analytics tool will show you the abandonment rate of your forms. What it won�t tell you is how many of those abandonments were a result of form validations. Most of the analyst/optimization specialist will suggest reducing the number of fields and/or conducting A/B testing with a different layout to decrease abandonment. But if you validations are a problem then A/B testing will not help you. You need to get your validations correct first before your jump into changing the number of fields, layouts etc.

Below are some examples of the fields that might require special formatting and are generally the cause of data validation errors:

• Social Security Number � If you do collect social security then do not force the user to add the data in XXX-XX-XXXX format, i.e. do not force them to enter �-� between the numbers. If you do provide one box for entry then either let them enter the numbers in free format or provide 3 input boxes with character limit of 3,2, and 4 receptively.
• Phone Number � Do not force the users to enter �(� , �)�, �-�, or spaces between the numbers just because that�s the format you want to store the data in. You can do all the formatting in the backend or front end code before the data gets submitted to your database.
• Account Numbers � I recently came across a site that manages my new 401K plan. The form asked me to enter the company account number, which was provided to me by our payroll department. I entered the account number into the appropriate field, clicked on the �Submit� button and got an error that something was wrong with my social security, last name or account number. It did not tell me which exact field was the problem field. I checked everything and tried it again and then again. Finally I got frustrated with and called their toll free number. It turns out that I was required to enter �-� after the 2nd digit. What a waste of time and cause of frustration that was. Since it was my 401K plan I had no choice but to call the phone number, if this were a shopping cart form I would have quit immediately.
• Email Addresses � Email address format is universal and most of the customers/visitors know that they have to enter an �@� and a �.� Enforcing such a formatting rule in the form validation is expected. However one validation that I recently came across was totally unexpected. My business email address has a �.us� extension and not a �.com�, but a not so �wisemarketer.com� rejected the �.us� domain. According to them �There are some e-mail domains and patterns that, in our experience, are more difficult to communicate with than others.� Apparently �.us� domain falls under one of those blocked emails domains. Really??? You got to be kidding me. �Wisemarkter� lost a valid conversion.
  
  
  
  
  

Showing the Error Messages

People do miss required fields or enter data in the wrong formats leading to errors. If an error occurs it is the site owner�s responsibility to make sure that the proper error messages are shown to the users. Two things that you need to keep in mind when showing error messages to the users (or visitors or customers):

1. Clearly state the error message � Make sure to highlight the fields in error and let the visitors know exactly what is required. Cryptic or generic error message is not going to help. Be sure to provide �prescriptive guidance�. Wisemarketer does not make the error clear, users have to click on the �Why� link to see the description of the error message.
2. Show the error message where the customers can see them � What�s the point of showing an error message if the visitors can�t see them. Many sites forget this simple principal. If an error occurs automatically scroll to the error message so that the visitor can see the error message. One of the offenders of this is mighty �Google�. The following form on �Google Website Optimizer� has an error but it is hard to see that error message on the screen.

The error occurs when you miss filling data in one of the fields and then click on the �Continue� button. The error occurs but the page does not scroll to where the error message is displayed. To the user it looks like that nothing happened, no error and no form submission. The user has to scroll to the top of the page to see the error message.

Have you come across a form validation that drove you nuts? Send me the link. Comments?

--------------------------------------------------------------------------
Looking to fill your Web Analytics or Online Marketing position?

Post your open jobs on Web Analytics Job Board
--------------------------------------------------------------------------

Is CAPTCHA Eating Up Your Conversions?

CAPTCHA acronym for �Completely Automated Public Turing test to tell Computers and Humans Apart� is a way for sites to block spam. According to Wikipedia it is

�A CAPTCHA or Captcha (pronounced /'k�pt??/) is a type of challenge-response test used in computing to ensure that the response is not generated by a computer. The process usually involves one computer (a server) asking a user to complete a simple test which the computer is able to generate and grade. Because other computers are unable to solve the CAPTCHA, any user entering a correct solution is presumed to be human. Thus, it is sometimes described as a reverse Turing test, because it is administered by a machine and targeted to a human, in contrast to the standard Turing test that is typically administered by a human and targeted to a machine. A common type of CAPTCHA requires that the user type letters or digits from a distorted image that appears on the screen.�

Below is an example of a CAPTCHA on craigslist.



Though CAPTCHA is a great tool for blocking spam it could be coming in the way of user experience and resulting in a lower conversion than you would have had without it.

I have come across many sites where CAPTCHAs are totally illegible. Such CAPTCHAs not only deter SPAM but also valid human visitors. Even if the CAPTCHA is totally legible it adds one extra step between a visitor and the conversion. Sometimes it takes few refreshes of the CAPTCHA before a visitors gets it right, resulting in a very frustrating experience.

You can spend all your time doing A/B and Multivariate testing the form layout, images, text etc. but probably won�t move the needle if your CAPTCHA is the culprit.

If you have CAPTCHA on your site then I suggest following 5 checks to ensure you have a good CAPTCHA on your site.

• Clear - Is it clear to your visitors that you have to enter the CAPTCHA before the form can be submitted? Some sites don�t make it clear and leave visitors wondering why their form is not getting submitted. Make sure there is help available on CAPTCH if the visitors get stuck. Also make sure that there is a refresh button to refresh the CAPTCHA image incase visitors can't read it.


• Readbility - Check all your CAPTCHA images. Can you read them? Will you visitors be able to figure out what your CAPTCHA reads?


• Accessibility - Visually impaired visitors should be able to fill the form else you will loose them at CAPTCHA.


• Time � How fast is your CAPTCHA? If it is slow to load or validate you might be loosing conversions.


• Protection - Do you have a huge SPAM problem (that you need a CAPTCHA or did you put it because everybody else is putting them too? Keep in mind that event a CAPTCHA might not completely protect you from SPAM.

Is CAPTCHA hurting your conversions?

I suggest you conduct A/B testing to understand how CAPTCHA might be affecting your conversions. Create a version of the page that does not have CAPTCHA and test it against the control version (your current version with CAPTCHA).

Analyze the results. You should analyze the conversions you get from each version. Deduct any SPAM when calculating the conversion. Calculate true conversions per month/year from both versions. Make sure your results are statistically significant. Most likely you will see lower conversion from the version with CAPTCHA and higher SPAM from the version without CAPTCHA. Considering the impact of SPAM on your form, determine if the efficiency (clean data) gained by having CAPTCHA on your form outweighs the gain of extra conversion when you remove CAPTCHA. If CAPTCHA is doing more harm than good then remove it.

Examples of CAPTCHA


This CAPTCHA is hard to read but has help and accessibility built into it


This CAPTCHA is hard to read and does not have accessibility



This CAPTCHA is easy to read and has help and accessibility built into it

Have you seen a bad CAPTCHA that caused you to leave the site? Send me the link.

----------------------------------------------------------------------------
Looking to fill your Web Analytics or Online Marketing position?
Post your open jobs on http://www.web-analytics-jobs.com/
----------------------------------------------------------------------------

Widemile Optimize: A/B and Multivariate Tool

In my post titled �A/B and Multivariate Testing Landscape�, I reported on a variety of tools that marketers and analytics professionals were using for A/B and multivariate testing. As I�ve predicted the interest in testing and targeting/segmentation continues to grow. This was also validated by the Web Analytics Association�s �Outlook 2009: Survey Report�.



I just got through a good discussion and demo of the newest release of Widemile Optimize with Bob Garcia, VP Business Development for Widemile. I�m impressed with the new release of Widemile Optimization product that he demoed; this is the first Widemile release for corporate users. Widemile had released an Agency Edition of the solution last fall and the new release includes advanced segmentation and a refined user interface.

You might be wondering who Widemile is as they don�t have the same brand recognition as Optimost or Omniture Test&Target. Widemile has been around for several years and was one of the vendors listed in �A/B and Multivariate Testing Landscape� survey at eMetrics. I started working with them last year as they were rolling out their platform for agencies.

Widemile�s new release should help more marketers get started with testing as they�ve focused on simplifying the process while still offering more advanced users a lot of flexibility. Their new segmentation wizard makes it easy to create and manage visitor segments based on visitors browser information, Geo information etc. Bob showed me how you can clone a test run and set up a new test against specific segments in less than 5 minutes without any tagging changes.

Widemile uses fractional factorial methodology which allows for tests that use a subset of the combinations used in full factorial tests. Using a fractional factorial methodology allows the tool such as Widemile to reduce the amount of time required to get statistically significant test results.

Widemile must have read my A/B and Multivariate Testing Landscape post I mentioned earlier, which cited lack of budget as an impediment to doing testing, as they also announced an aggressive promotion (up to 50% off) for the first 20 customers that sign on. In closing, if you�re investigating your options for a professional testing solution Widemile should be on your short list.

Feel free to email me if you need help to figure out which testing solution will work best for you or need help in getting your testing going.

----------------------------------------------------------------------------
Looking to fill your Web Analytics or Online Marketing position?
Post your open jobs on http://www.web-analytics-jobs.com/
Latest Job: Senior Web Analyst at Vml (Kansas City, MO)
----------------------------------------------------------------------------
Site: AnilBatra.com
Twitter: http://twitter.com/anilbatra
----------------------------------------------------------------------------
Have you used Clop.in a URL shortner with real time analytics? http://clop.in/C6ZWAL

Market and Optimize Responsibly

We, the people in the business of marketing and optimization, know (I hope) that certain images, text, ad copy etc. work better than others in driving user to convert on our sites. We continuously test (A/B and Mult-variate) to see what works and what does not work for our visitors/customer on our sites and then optimize our sites/banners/emails etc. accordingly. In order to make users click we test headlines, text, testimonials etc. making certain claims about our services, and products in those headlines, ad copies, emails etc.

In making those claims we need to think responsibly. We need to make sure that we do not go over broad and make false claims, promises or exaggerate the results. Yes, those claims might give you higher click through rates and higher conversions but there is one more KPI that you need to keep in mind i.e. the KPI that measures responsibility (both ethical and legal). We need to all think ethically and legally about all the claims we make. Think about how we will feel if another company made such claims and we fall for it, will we feel cheated or not? Think about the potential of lawsuits.

Classmates.com is being sued for allegedly making false claims in their email. According to Media Post

�The plaintiff, Anthony Michaels of San Diego county, alleges that he signed up for a free membership to the site last Christmas Eve, but then upgraded to a paid one after receiving e-mail ads stating that other schoolmates were trying to contact him. Those statements turned out to be false, according to the lawsuit.
Michaels' lawyer, Brian Kabateck, said his client had no way of verifying whether his former schoolmates were actually seeking to contact him on the site, short of signing up for a one-year membership.
"The e-mail said: 'So and so's trying to find you, and in order to hook up with him you have to join and become a gold member,'"

Media Post further reports that Reunion.com also faces a lawsuit by members complaining about the site's marketing efforts. In that case, filed in federal district court in San Francisco, the members allege that Reunion.com sent e-mails that appeared to have come from specific friends, but were actually sent by the site.

We might (or might now) find out if the claims by classmates were false or not. Either way this lawsuit provides a great lesson for the marketers - Market and Optimize Responsibly.

----------------------------------------------------------------------------
Site: AnilBatra.com
Twitter: http://www.twitter.com/anilbatra
----------------------------------------------------------------------------
Looking to fill your Web Analytics or Online Marketing position? Post your open jobs on http://www.web-analytics-jobs.com/

Latest Postion
Technical Consultant for Internet Marketing and Web Analytics at Unica (Waltham, MA)

A/B and Multivariate Testing Landscape

In our most recent whitepaper we investigated the current state of A/B and Multivariate testing, focusing on how the practice is currently used and the obstacles companies face in implementing this type of an online strategy.

Our analysis, taken from surveys given at the eMetrics Marking Optimization Summit, found that 52% of online marketing managers are currently engaged in A/B or Multivariate testing and that an even greater percentage plan to begin testing within the next year.

A high-level overview of the findings:

• A variety of testing platforms are considered before making a final decision


• Experiments vary in frequency, page type and page element


• Practitioners often use more than one platform to meet their needs


• Pretest hypothesis often fail


• Lack of Budget was cited as the main reason by those who currently do not do any testing


• Complexity of Tools and lack of best practices were two top challenges faced by those engaged in Testing


• A/B and Multivariate testing is �worth it�


• Behavioral targeting is a small but growing practice

Surprisingly lack of budget was one of the top reasons why companies were not involved in A/B or Multivariate Testing. But if you look at the chart below you will see that companies that do conduct A/B and Multivariate testing find it that it�s worth it. Majority of the respondents who engage in testing said that either it is worth it or can�t live without it. So if you are one of those who think because of lack of budget you can�t do A/B MVT, think again? Your Return will make it worthwhile to do testing. You will be able to move budget from other areas such as Ad spend to A/B testing. Do your ROI calculations. You will be surprised with the positive results you will get with developing a culture of continuous testing.

You can download the whitepaper from ZeroDash1 website.


--------------------------------------------------------------------------------

Looking to fill your Web Analytics or Online Marketing position? Try WebAnalytics Job Board

New Position

Web Data Analyst at Alzheimer's Association (Chicago, IL)

---------------------------------------------------------------------------------

Omniture Acquires Touch Clarity and WebTrends Announces Partner Integration

Yesterday Omniture announced that it will acquire Touch Clarity, a Behavioral Targeting Company. Within few minutes a WebTrends announced partner integrations with leading digital marketing vendors, including email marketing firms ExactTarget, Responsys and Silverpop; on-site behavioral targeting firms Kefta and Touch Clarity; and consumer opinion and customer voice firms ForeSee Results and OpinionLab.

This promoted me to write this article because this is exactly what I predicted.

You can read the fill press release at
� Omniture Acquires Behavioral Targeting Company Touch Clarity
� WebTrends Announces Marketing Lab Partners to Deliver More Measurable, Automated Relationship Marketing

Anybody who read my predictions for 2007 (http://webanalysis.blogspot.com/2007/01/my-predictions-for-2007.html) might have already seen this coming.

Here were my 3 predictions for 2007, and this acquisition proves all 3 of them true. My new comments are in Bold

1. Web Analytics won't be standing alone - Marketers will want 360 degree view of the customers. Integration of various data sources and tools will be expected from web analytics and other supporting tool vendors. Omniture started the trend with Omniture Genesis, and this will continue, we will see more acquisitions and partnerships similar to Omnitures.

- Both the above press releases confirm that this prediction has come true. Customers are demanding this stronger integration. I wrote a follow-up on this same predictions last week, you can read it here http://webanalysis.blogspot.com/2007/02/follow-up-on-my-web-analytics.html

2. Web Analytics will be about taking actions � More and more marketers would like to take actions and not just report the findings. It just won't be about what happened, it will be about taking action to drive sales, user satisfaction, lead generation etc. Incentives and bonuses will be tied to the online KPIs. Optimization and Behavioral Targeting will become a common term used by marketers.

- Omniture acquisition of Touch Clarity, a Behavioral Targeting company confirms this prediction has come true too. WebTrends announcing partner integration with Kefta (Optimization and Behavioral Targeting) and Touch Clarity (Behavioral Targeting) confirms that marketers are demanding Optimization and Behavioral Targeting. It is not just about reporting, it is about taking actions. In near future I am expecting to see a lot of case studies about what actions customers took and how it affected their bottom line.

3. Behavioral Targeting � Only few main behavioral network players will be left and some of the existing ones with poor networks will either go out of business or be sold. See my previous article on why size of network matter. Behavioral Targeting won't exist in isolation. Web Analytics tool will have to support behavioral targeting and visa versa. Also, on-site behavioral targeting will become very common.

- Part of this prediction has come true. I have yet to see any consolidation in behavioral network players. But as the above two press releases confirm �On-Site Behavioral Targeting� is becoming common.

Web Analytics is maturing. Gone are the days when marketers were just satisfied by learning like
�Traffic went up�, �Traffic went down� etc. It is about deeper understanding of the visitors behavior (Web Analytics), segmenting them (Web Analytics and Behavioral Targeting), taking actions based on this deeper understanding e.g. targeting relevant content, products etc. (Email, Behavioral Targeting, Optimization).


So I am 2.5 on my 3 predictions. I have already declared 1 more prediction come true (http://webanalysis.blogspot.com/2007/02/follow-up-on-my-web-analytics_09.html). So for the year I am 3.5/5 so far and the year has just started. I feel good about my predictions.

Questions, Comments???