A/B Testing

Over the last couple of days I have looked into A/B testing to prepare myself for some work on the Choice Skills website. Though A/B testing in theory does not seem like an incredibly difficult concept, I am realizing that the method with which we will implement it is of some concern.

The first decision that needs to be made is what exactly are we going to change? Before we test, we need to have an idea of some of the aspects of the website that need improvement (which must be based off of current SiteCatalyst data). Testing for the sake of testing will be worthless unless we have successfully found some key data in SiteCatalyst that compels us to tinker around with some of the layout or design of the Choice Skills website. I think we have found a couple of things, we'll just need to decide where to focus our energies at the beginning.

I suppose that brings me to the next point - we need to be very specific with what we are changing. Now that I think about it, a total overhaul on www.choiceskills.com may not be a terrible way to start (we've already found a number of things to improve), but typically an A/B test is implemented to test specific changes that can be measured individually. If you are changing a whole bunch of things, you never really know what modifications are causing the increase or decrease in conversions, and that can be a serious problem. If we're looking for complete optimization, changing individual pieces of the site will more effectively lead us towards success.

Considering that it is in our best interests to modify a single aspect of the site at a time, it is also extremely important that we designate a time frame for the testing we will be doing. In order to successfully complete a test, Omniture suggests about 500 conversions as an accurate measurement. With a budding site like www.choiceskills.com, we don't have the opportunity to effectively test 500 conversions at this point. Finding a valid period of time in which we can test how effective the modifications are is going to be extremely difficult.

Not only is the duration of the test a challenge, but deciding how we want to implement an A/B test is another important decision to make. The easiest way to do it would be to set up a Consecutive Test, where we run site A for a given period of time and then site B thereafter for a given period of time, sending all site traffic to only one location. The more effective test, however, is a Synchronous Test, where we run both sites simultaneously and send only a portion of our site traffic to the newer design. Deciding between these two testing methods will be important. Our choice will either be based on ease of implementation or on the quality of data we want to extract.

If we decide to run a Synchronous Test we will also need to decide how exactly we want to implement the test on SiteCatalyst. My thought is that we should use Custom eVars to track the information coming from the variations of the site. The Custom eVars may not be the most elegant and detailed implementation, but considering the short period of time with which we have to work, and the fact that choosing a Synchronous Test is a more difficult implementation to begin with, I vote that we try and keep it relatively simple at this stage in the game. The Custom eVars will simply allow us to track conversions coming from the specific pages (i.e. Homepage A, Homepage B).

Though I don't completely understand the implementation of A/B testing at this point, I think I have the right frame of mind in making sure that we are making correct decisions on how exactly we should design our test before worrying about the specific implementation. I don't want to spend a ton of time worrying about the data coming out of our testing unless I know I can trust it.

Choice Skills

So, in a relatively high level look at some of the Omniture data for Choice Skills I found a few things that I think are worth looking into. Obviously with a more thorough dig through the data I'm sure I could find a lot more, but I decided we have to start somewhere so why not start with some of the more obvious things... Anyway, let's take a look:

• Implementation is not tracking paid search keywords. This is the one that I feel is pretty important since the #1 expense for the site right now is keywords. I would be pretty concerned about seeing what the successful keywords are and how they are driving some of the more important KPI (such as conversions leading to revenue). This is an implementation issue more than anything else, so it should be pretty easy to iron out.
• Switching the homepage HTML file. Currently, the homepage is pretty blank and just serves as a short portal to direct the user to more important pages. Why not make the "teaching_social_skills.html" file the homepage? This is the page people are spending the most time on, is leading to the most revenue and is the top entry page. The current homepage is almost worthless in that there isn't anything there that cannot be added to the more useful Teaching Social Skills page.
• Increase the number of "Order Now" links and improve their placement. Currently the links to the order pages are not in very good places (at the bottom of the Teaching Social Skills page or by clicking "store" at the top). These can be place in several more locations, including the homepage, and in more strategic locations. I don't have specific recommendations, but this is something we can specifically design after some dialog on the topic of getting more people to the homepage.
• Standardize file download names: OK, I admit, this one is picky and little, but since SiteCatalyst is tracking the information on downloads, I am guessing it is a metric that Choice Skills is interested in tracking. If that information is important to track (and I think it's definitely worthwhile considering the goals for the website and the product) then it would be best to standardize the names of the files being downloaded. There are files abc.pdf and ABC.pdf with different data for each one. It's possible that the names were recently changed, but it doesn't look like it (but I've learned I'm not always correct when it comes to interpreting SiteCatalyst data).

I'm sure there is tons more that I'll be able to find as I become more familiar with the site, but I think these are a couple of things that can be attacked and worked on quickly. I was going to dig into some of the conversion information, but with little data to work with I did not think I could come away with many quality recommendations at this point.

I'm really excited for this project! I think it's fabulous that we have something real to work with, giving us a fabulous experience to work on our skills and to learn about implementation. With the generosity of Omniture and the opportunity to take this class, I've definitely been given another reason to be glad I'm at BYU.

Omniture's SiteCatalyst

It's been a while since I've had some time to get a post up, and one of the reasons why is because I have been spending a lot of time preparing for the Omniture Web Analytics Competition at BYU. This was my second semester participating in the competition, but unfortunately we worked with the same website's data. Though you'd think I had an advantage, being familiar with www.costumecraze.com, the Omniture people made it clear at the opening meeting that we needed to come in with new information this time around.

I can't share any specific information due to my signing a non-disclosure agreement, but I can talk a bit about the SiteCatalyst product! To be completely honest, SiteCatalyst is an absolutely fabulous tool that still needs a little work. Currently, you have great flexibility and customization available to you (though sometimes you end up paying quite a bit more for some of those options) as an Omniture customer. The task of keeping up with all of your key performance indicators is pretty simple with the advanced dashboards you can create. From my experience, which is admittedly little, SiteCatalyst blows the competition away in sophistication and the level of customization available. Granted, those who use Omniture's tools are paying out the nose, so SiteCatalyst better be good.

I do, however, have a few complaints. Omniture has broken up their reports into three main sections: Commerce, Traffic, and Paths. This is really helpful in some regards because it breaks up the many reports available into three different categories, yet I found myself extremely frustrated on several occasions. Because these three categories are very rigidly constructed, getting reports on data from both traffic and commerce is impossible (or at least hard enough to seem so). For example, should I want to find a report that gave me the percentage of site traffic that came from a specific domain and the revenue from that domain I would be out of luck. For some questions you can look up two different reports within their respective categories, but this is not always the case.

Without giving out specific information, I was trying to do research on CostumeCraze's global presence (where commerce and traffic were coming from geographically) and I could not find reports that really helped me drill down on the issue. SiteCatalyst provides traffic maps, showing which countries around the world are providing site traffic, but I could not find anything that would help me see the revenue that matched up with that traffic. The data already being tracked, so why can't I get to the information I need? There were ways to work around the problem, sort of, but I'd rather just have interoperability between the three categories.

Omniture does offer other services, and it's entirely possible that Discover or another piece of software solves some of the problems I see in SiteCatalyst. Unfortunately, I haven't had access to any of the more advanced tools, but I can probably research it a little bit.