How does Bespoke Data Collection work?

There are two types of bespoke data collection that GMT+ offers: experiments and surveys. Both of these broad methodologies have venerable histories in the social sciences. Experiments are designed to test possible actions. Well-designed experiments will laser in on the answer to questions that usually start “what would happen if…”. Surveys exist to collect data from people that you do not otherwise have access to. Both methods have characteristics which recommend them as an approach to research. We offer a brief description of both approaches in this post, beginning with experiments.

All experiments will test a hypothesis, this is the “what will happen if” question. When working with people this means generating two groups: a ‘control’ and a ‘treatment’ group. Both groups offer the same context and are statistically identical in terms of the relevant characteristics of the people within each group. The two groups only differ in terms of one main characteristic – the presence or absence of a treatment. The treatment is the action whose effect you are wishing to test. If the treatment is the only difference between the two groups, then any difference between the groups in terms of the outcome measure is almost certainly due to the treatment.

Random allocation plays an important role in the experimental method. Ideally, people should be allocated randomly to each group. Random allocation helps reduce the chance that there will be bias between groups in terms of observable characteristics that may influence the experiment. Perhaps of more practical use is that random allocation reduces the chance of there being a difference between groups in terms of unobservable characteristics, such as a sense of conservatism or adventure, outcome by removing people’s choice of which group to join (assuming they have consented to be a part of the experiment itself). If one group (perhaps the control group) attracts conservative, risk-averse people while the treatment group attracts more adventurous, risk-loving people then the outcome of the experiment may not be driven by the treatment itself, but rather by the difference in the conservative/adventurous nature of the respective groups. Randomly allocating people, however, should ensure a more balanced distribution of personalities across than groups than allowing people to choose.

There are two broad approaches to conducting experiments: lab experiments and field experiments.

Lab experiments are experiments that do not necessarily take place in a laboratory. A lab experiment may take place online, in a park, in a university computer lab or almost any other place. The distinguishing feature of a lab experiment is that it takes place in more or less artificial circumstances. In a lab experiment the circumstances are artificial because the researchers are controlling most aspects of the context. This gives the lab experiment its greatest strength: precision – and weakness: applicability to the broader world.

Lab experiments allow you to test very precise questions which it would be too difficult, time-consuming or expensive to test in the course of life in the real world ( eg. should I make this red or blue, do I let people see previous “likes” or not). However, since the context is so artificial, great care needs to be taken when designing a lab experiment that the result’s relevance to the real world is not jeopardised. Fortunately lab experiments have been increasingly used by social scientists – especially by economists – over the last 30 years, so we are at a stage where clear guidelines exist for good lab experimental design.

Field experiments were initially called this because of their agricultural origins, however today we use the term to describe experiments that happen generally in the real world rather than a mainly contrived context. The disadvantage of a field experiment compared to a lab experiment is that the researcher has much less control of the context. As a result a field experiments allow much less precision in terms of the question that can be asked. Their huge strength however, is that they give a very accurate idea of what will happen in the real world. Good field experimental design will try to minimise the loss of precision while retaining application to the real world.

Obviously experiments do not necessarily mix neatly into these two broad approaches. Rather we find that lab and field characteristics may mix with each other as the question demands.

Sometimes you cannot get the information you need through administrative data or experimental methods and it just makes sense to ask people questions about themselves. This a job for surveys. Surveys are the tools researchers use to get data that cannot be obtained in easier ways. Given that we can usually observe behaviour, this frequently means that the main job of any survey is not mainly to ask a person what they did, but to get inside of their head and try and find out why. Given the long history of surveys, there are numerous approaches and many, many articles and books on each expressing a plethora of different views.

For us at GMT+, given our broad experience with surveys, we have become convinced of the following simple principles of survey design:

1. Do not waste your respondent’s time: ask them only what you cannot figure out through administrative data and other data.
2. Your respondent’s time is short and their focus will wander quickly: do not take more than 10 minutes of their time if you want focused, truthful answers.
3. Test your survey on real people before you take it live.

We hope that we can help you fulfil your requirements for Bespoke Data Collection!

Return to: What we do