Scores or Stories

While I was out walking yesterday (and trying to distract myself from the sensation of my leg muscles tearing due to a nasty flare) I was thinking about how to go about gathering information at scale.  The most popular mechanism for this is to survey a population asking them for to score their level of agreement with statements like:

On a scale of 1 to 10 how happy were you with the service you received from …

I’ve designed many of these surveys over the years and I’m sad to say that I’ve never learned as much as I’d hoped to.  I’ve found that it’s easy to collect data and produce some nice graphs, but the information is rarely that insightful or actionable.  Normally it just reinforces impressions that I had when I wrote the questions.

In part the problem is that the respondents know what answer you want and so they have to make a choice to either tell you what you want to hear, to be honest, or to give voice their general frustration.  Unfortunately you never really know which motivation is in play. 

So I’ve been considering a different approach, in fact two approaches.  First I’ve been considering questions with two right answers, forcing respondents to pick.  For example we might ask:

In your experience are customers generally more interested in our completeness of vision or our ability to execute.

Respondents would answer by picking a point on a sliding ten point scale with execution and vision at opposite ends.  Another example might be:

What is more important to you, attending more training courses, or having all the facts and figures easily available to you on your laptop/iPad so that you can easily answer customer questions

The second approach I’ve been thinking about is asking respondents to write short stories, rather than answer questions, for example a respondent might be asked:

Write a short story about one of your most challenging meetings with a customer and what we could have done to better prepare and/or support you

I might throw in a few traditional scoring questions as well, just for old times sake.

As I’ve been thinking on this topic and struggling with a flare I’ve also been reminiscing.  About 15 years ago when I first started with this auto-immune condition I was off sick for two weeks and for some reason, that’s now lost to me, I designed a domain specific language with which to write survey questions. I also wrote a ‘compiler’ that took the surveys written in this language and created a web application, database and set of reports in a couple of seconds.  The language was extremely simple to use, but allowed many different types of questions to be written and had the benefit that it was just a text file that could be authored by several people, worked on offline, reviewed just like any other document, re-used and modified over many years. I wrote this just before Survey Monkey came into existence and using it was about 5 times faster, supported more question types and had much better reporting.  My little app was also self service and was used hundreds, maybe thousands of times by my company over the years.  I’m not sure what eventually happened to it as it never needed any support and consumed almost no resources on the server it shared.  When I retire I might resurrect this project.

The photo is of Brockholes Nature Reserve where I was walking and thinking about this post, it’s been a few years since I last visited and it’s matured nicely.  I will definitely be going again, the walking’s good, it has a great cafe and it’s on the way to the office.