Odd & Annoying Survey Methodology

Our municipality has contracted a survey firm to evaluate the after-school activities for children that it supports. Circus school, piano lessons etc. Questionnaires have been sent to (some? all?) enrolled children.

My kid is in three of these activities. I got three almost identical questionnaires, interpreted them as a mail-merge glitch, responded to one and threw two away unexamined. Then I was nagged about those two.

If I were running the survey I would purposely avoid collecting data on the same kid for more than one activity.

Author: Martin R

Dr. Martin Rundkvist is a Swedish archaeologist, journal editor, skeptic, atheist, lefty liberal, bookworm, boardgamer, geocacher and father of two.

12 thoughts on “Odd & Annoying Survey Methodology”

  1. -Are you still in Saltsjöbaden? If the municipality is small enough, they might actually pay attention if you give them advice about survey methods. In larger places, your complaints will just drown in white noise.


  2. Does the survey firm have any way of knowing that John Smith the piano student and John Smith the football player are the same person?

    I am in a small municipality (population 1X,XXX), and my name is not common in this part of the US. However, there was a period of several years when somebody sharing my first and last name was also on the local voter rolls. Every ten years the state requires municipalities to purge their voter rolls of non-voters (people who move away, especially out of state, do not always inform the town clerk in their former municipality). I had been voting (and still do vote) in every election, but my namesake had not. The town clerk sent me the note saying that I had to confirm my registration in order to stay on the voter rolls.

    I don’t know how common name duplication is in Sweden, but if it is in any way similar to English I expect there is a non-negligible chance in a municipality the size of Nacka that there are two people with the same name.


  3. This can be based on the fact that different databases do not work together and cannot see each child as uniquie, in database model terms. Nor can the system see the relation betwwen the child and the parents. We have our system of unique person identification numbers here in Sweden but sometimes you are not allowed to use that across different database systems except for taxing or crime investigations.

    It is based on different databases not being able to talk with each other or badly designed ones.


  4. If they care to do that check, Obviously the did not! Information is not connected or consolidated in their systems.

    Dead people get offers sent to them!


  5. Are they allowed to use citizen numbers that Martin? In Canada, for very good reason it is illegal to use people’s government-issued ID numbers as a general-purpose identifier. Allowing that, as they do south of the border, would make it too easy for companies to surveil people and combine databases without people’s consent; it is also dangerous because people tend to assume that such a number must be strong ID, and if they give it out widely it is easy to steal. So while a Canadian city might have such data for taxpayers (I am not sure), it would not have it for everyone and would not be able to give that data to a private marketing company.


  6. You can blame the research firm (after all, they are the ones responsible for administering the survey and advising the customer on how to do it right); however, having had some experience with market research, I would guess:

    a) The survey company was given three separate lists for the surveys and wasn’t paid extra for de-dupping the lists
    b) The three surveys may be tabulated separately
    c) The customer may have insisted on doing it this way

    And unfortunately I am not a survey methodologist nor am I a statistician by education or training I cannot comment on whether or not it is actually valid to survey you once for each activity. It probably depends on the questions asked; though, one would think it would be better to ask them all in one survey rather than three nigh duplicate surveys.

    However, think about a survey asking about soft drinks. They are going to ask you a series of questions about coca cola, then pepsi, then RC Cola, then whatever else is next. The questions are going to be identical but the product is going to be different. This is probably analogous to sending you one survey for each activity.


    1. Trouble is, polling a person changes that person. My replies to a questionnaire will be different if it’s the third identical one that day than if there’s only one.


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: