Design First, Analyze Second

James Hart Blog Leave a Comment

Design matters.  A plan is necessary.  Design is always the best first step in creation.  In engineering or invention, we should always start with a problem, develop potential solution ideas, determine optimality, and create a plan for building and putting those pieces together.  Good ol’ Merriam Webster aptly defines design as “to plan and make decisions about (something that is being built or created) : to create the plans, drawings, etc., that show how (something) will be made.”  Below I will highlight the approach I use to designing my analyses.

Problem Decomposition

In order to determine a proper solution for a problem you must understand the variables that are involved. A solution is simply a change to the environment in which the problem lives that produces the desired effect.  Analytics is odd. It’s odd because the problem you’re solving for is how to solve problems.

Oftentimes, the problem presented by the decision-maker is not specific enough to truly analyze. An illustrative and typical problem presented to a marketing analyst may be: Increase traffic to the website. The goal is stated, but the first question an analyst should ask is “why.” There are a lot of assumptions being made in that problem statement, such as “website traffic = more $” that need to be identified and accounted for in the analytical design.

In asking these questions you may reveal that the initial problem statement is not the problem after all.  Your problem statement might become:  Increase traffic to the website that results in a conversion.

Problem decomposition tells you a lot about the bounds of the solution.  Our example problem tells us that the decision-maker understands that there are multiple methods to increasing web-traffic to the site.  The analyst should first research the available methods to increasing traffic.   Further research might reveal that there is a budget associated with this initiative that should be considered in the solution set.  The problem now becomes: increase traffic to the website that results in the conversion by spending $X on paid search, Search Engine Optimization and banner ads.

Measurement Space

Full disclosure: Measurement Space is an idea developed by the U.S. Army’s TRADOC Analysis Center (my previous employer) that provides structure to the analytic planning process. Hopefully, my short explanation does it justice.

Measurement space is the set of differentiating data points in the intersection of the problem and solution. The question we are trying to answer when working through the measurement space process is: What is the set of accessible measures that will help us understand the differences between potential solutions? A very effective way to develop this set is to have a group of interested parties sit around a table and talk through the problem with a note taker logging each measure that is discussed.

For our problem, the discussion might include words like impressions, clicks and page rank. Other important metrics to consider would be cost per impression, cost per click or page rank lift.  The important part of this process is to collect all measures.

Next, go through a vetting process to determine if the measures are accessible. A great metric could be the exact portion of conversion revenue that can be attributed to each channel. This may be inaccessible due to a lack of understanding, lack of data or insufficient time to develop the data set. That metric will be set aside because it does not reside in the available measurement space.

The final step in the process is removing any measures that will not highlight a difference between the different solutions. For our problem, time to deployment (of the solution) may seem like an important measure. However, after initial research, we may find that the difference in time to deployment for the set of possible solutions is negligible and therefore need not be considered in the analysis.

Analysis Design

Finally we get into the fun part, deciding what tools or models are needed to highlight the differences in the solution set that best contribute to solving the problem. Note that multiple tools and models may be necessary to analyze different metrics. Each metric should be considered in tool selection to ensure the full range we decided on in the previous step is covered. The hardest part of this step is having to accept that the appropriate model is inaccessible due to lack of resources. This is where the optimization idea comes from in my introduction. Given a set of constraints what is the best method to do this analysis?

Final Thoughts

Note that many definitions leave out one of the most important parts: create plans, drawings, etc. Writing it down is as productive as thinking through it.

In this process, it is easy to lose sight of the initial problem because we learn so much.  Take note of what seems to be the “correct” question and save it for follow on analysis but ALWAYS provide the best solution possible to the decision maker’s problem:  Increase web traffic.

I believe that you can learn as much in the design process as you do in the statistical analysis/mathematical modeling portion of analysis.

While this is process may seem (and be) arduous, it is well worth it.  Design saves rework by answering the right question with feasible solutions and does so in the most efficient manner by avoiding time wasted on comparing or trying to develop unusable metrics.

Leave a Reply