This post originally appeared on the MITX blog.
In this era of data and analytics, consumer marketers are really lucky: they are selling to markets so large that they can constantly be testing and optimizing their physical and digital marketing. Whether you’re American Express, Amazon, Pottery Barn or whoever, unless you’re a consumer marketer in a narrow niche, you are marketing to tens of millions, and possibly hundreds of millions of people. Direct mail, email, online forms, TV, pay-per-click ads, offers – the opportunities to optimize are endless.
The head of digital marketing at a large insurance company one time told me that in their tests they saw significant differences in response to PPC ads if they used the term “auto insurance” versus “car insurance” in terms of both demographics and actions.
Companies that serve SMBs, such as Constant Contact, Vistaprint and HubSpot, are selling to hundreds of thousands, if not millions, of small businesses, so they have the opportunity to test and optimize, too.
And some B2B companies have markets of thousands, or tens of thousands, of companies.
But what if you make software for supermarket chains? There are only a few hundred chains in the country of any size. Same problem if you’re selling to large healthcare networks. And same problem for service companies targeting companies in the greater Boston area with, say, over $10 million in annual revenue. Some of these markets are very large in potential dollar revenue, either in an absolute sense or for the companies selling to them, but many have so few companies (n ≤ 500) that they defy statistically significant testing.
Let’s say that you are marketing to 50,000,000 people and can accept a margin of error of 5%. To have 95% confidence in the results of your testing, your sample only needs to be about 385 people. But what the hell, with that much at stake, you could even go for a 1% margin of error. Then you would need a sample size of 9,602 people. How many seconds does it take Amazon or Google to run a test on 9,602 people? Not many. For physical marketing, or TV, a consumer company might use one market as a test before doing a national rollout.
And you could run many 9,602 person tests, optimize your program, and still have over 99% of your market left to run the campaign on.
But if you’re selling to 5,000 companies and want a 1% margin of error, you’d need a sample size of 3,289 – over half your market. No way. If you drop your acceptable margin of error to the more common 5%, for 95% confidence you’d need to sample 357 – over five percent of your market. You might do that, in some situations.
But if you’re a B2B company selling to 500 companies, like all major supermarket chains, the sample size for a 1% margin of error is 476 out of those 500. Forget about it. Even if you drop your margin of error to 5%, you’d need to send your test out to a sample of 218. Again, that’s impractically large. You’re practically running your campaign to test it.
So what’s a marketer to do?
Source: http://xkcd.com/539/
There is no scientifically valid answer to this problem. The best that most B2B companies in this situation can do is combine what data they can get , including customer focus groups and other Voice of the Customer programs, with their market experience and the feedback of people in sales who are regularly meeting face to face with prospects. (Yes, marketing can talk to people in sales. Really.) Granted, any one focus group or account executive’s experience will be very limited, but talking to many customers and account executives can help flesh out marketing’s understanding of the buyers, their motivators, and what may work in marketing to them.
Even when you can do statistically significant testing, talking to your customers is a good idea. For companies in markets with a small number of potential customers, it’s critical.
And you can do some data-based testing. Just don’t expect to find out the difference between using “auto” or “car”.
Did you think this was interesting? Then sign up to get our daily email marketing tip that people call “very impressive and helpful”
