The clue is in the name – a pilot study is essential in getting your research ‘off the ground’! A pilot study is a small-scale “trial run” conducted before launching the main research project. Its purpose is to test different elements involved in the project early on, in order to uncover potential flaws and make necessary adjustments before the full-scale research begins.
The objective is to generate feedback, whether good or bad, regardless of whether any issues are uncovered (or not). Pilot studies are conducted in an endless number of scenarios and are a vital part of a launch, whether that be in healthcare, education, the restaurant and food industry, the gaming industry, just to name a few. The end-goal is to improve customer experience by ensuring that the product launches smoothly.
So let’s take a look at why we should run pilot studies in market research, how to run a successful pilot study, and the benefits and challenges one might encounter during this process.
Pilot studies are essential. In the majority of cases, research will not be carried out without a pilot study first. In a practical sense, it tests procedures, timings, logistics, and ensures that everything is ‘working’ as expected in real-world conditions. If we apply this to other industries, just about all will run some form of test run, pre-launch, soft-launch, beta test.
Each pilot study has the same result in mind – risk aversion, minimising costly mistakes, and avoiding wasted time and resource. In healthcare, pilot studies are crucial from both an ethical and huge risk aversion point of view. They ensure that potential errors can be averted before a large number of people are involved. This ensures brand reputation stays in-tact, promotes trust and loyalty. In a general sense, pilot studies are important to ensure that customers feel valued, particularly if they have been chosen to be a part of a pilot study out of a large number of customers. They will feel ‘special’ which may lead to increased engagement from them, too.
What’s more, customers may be able to provide valuable feedback and insight into areas that may not have been thought about previously. Knowing and being able to promote the fact that customers had an initial say on the research community from the beginning and how it had an impact in the longer term means that you can confidently say ‘this is tried and tested’.
Starting off with the basics, keep the timeframe of the pilot short - around no longer than 4 weeks. Within these 4 weeks, the customers involvement in the pilot should be around 1-2 weeks. During this time period, customers should be invited to take part in a set of example community activities such as surveys, quick polls, and other fun and engaging activities (a taste of what being a member of the community will really be like). The experience should be as authentic as possible.
To achieve this, customers should be sent example communications from the community such as task invites, example newsletters, push notifications (if there is a mobile app involved) and more. It’s really important for researchers to closely monitor customer activity during this time. For instance, how often customers are logging in, and what prompts them to login (e.g. do they login organically, or following a notification or prompt). Following this, the feedback from the pilot will be collected. It’s also really significant for researchers to have means of close communication with customers during this time, by having regular touch points (such as providing a direct support email address, online group or one on one discussions). Furthermore, by running feedback led example activities, you are combining both collecting feedback and running example activities into one, saving you lots of precious time! For example, running a survey which collects what they think about the design and UX of the community, featuring interactive scale questions.
Or, running an interactive heat map-based Smartboard where users can pin point their exact feedback on a screenshot of the community landing page, for instance. The pilot study findings will then be shared with key people in the business during the third week, including stakeholders. This will give chance for the researchers leading the pilot to make any changes based on the feedback during the final week, ahead of the full launch. This timeframe will give ample time to capture how customers settle into the community, how they find the overall UX, such as welcome activities, community feel, and more. Essentially, pilot studies should be used to identify what researchers can feasibly refine in the given timescale rather than unrealistic major changes, to avoid delaying the full launch of the community.
Undoubtably, there are far more benefits over challenges to running a pilot study. Pilot studies are the key to minimising mistakes (sometimes costly and time-consuming), plus they avoid wasted resources. Thinking from the perspective of a customer joining an online community, being included in a pilot study is a way of saying ‘we value your opinions’.
The close interaction between researchers and customers during this phase is significant from a customer salience point of view (the practice of ensuring the customer is front-of-mind for everyone within an organization when making decisions, thereby closing the "say-do gap" between stated customer-centricity and actual actions) because of the customer being in the front-of-mind for everyone involved.
On the other hand, not so much of a challenge but something to consider, is that a pilot study can take away from the all-important initial first impressions, that the wider customer base may have otherwise experienced. This is because the smaller-scale pilot group of customers will have seen the initial community, provided feedback, thus changes may have resulted from this. Furthermore, another point to consider is that opinions on design & look and feel can be subjective, and so it is important to get a group consensus on feedback before making decisions on changes especially if they are noticeable changes. It is also important to consider timescales when running a pilot study, as conducting a pilot study can cause unexpected delays to the main launch of a community.
What may have started as a few weeks for a pilot study can turn into something that lasts much longer. Indecisiveness can come into play, which creates uncertainty. Therefore, it is essential to stick to a timeline and predict any time constraints ahead of time.
Another consideration for pilot studies is cost... will there be any cost implications? At the end of the day, the cost of a pilot study is likely minimal when comparing it to the potential costs of fixing multiple issues as a result of not conducting a pilot study. Not only a physical cost, but a figurative ‘cost’ of what a lack of testing can lead to. With all of these factors in mind, the benefits however far outweigh the challenges.
Pilot studies will most definitely improve the community experience for the wider customer base, and will shape the rest of the research community’s success. They promote confidence, reassurance, and close connection, for all parties involved (researchers, stakeholders, and customers). There is an added element of transparency and trust that comes with pilot studies, where researchers are being fully open with customers by setting out ‘This is what we want to do’ with the power then lying in the hands of the customer to provide their feedback.