--- title: "Man vs. Machine: An Insight Community Analysis Challenge (IIeX Europe 2018)" description: Winner of the IIex Europe 2018 Best New Speaker Award, Charlotte Pearson shares the findings from the insight community analysis challenge on which her presentation was based. Fascinating reading for qual researchers, both agency and client side alike. image: https://blog.flexmr.net/hubfs/4841069-6-man-vs-machine-an-insight-community-analysis-challenge-iiex-europe-2018.png --- close ## Ready to level up your insights? Get ready to streamline, scale and supercharge your research. Fill out this form to request a demo of the InsightHub platform and discover the difference insights empowerment can make. A member of our team will reach out within two working days. close ## Cost effective insights that scale Quality insight doesn't need to cost the earth. Our flexible approach helps you make the most of research budgets and build an agile solution that works for you. Fill out this form to request a call back from our team to explore our pricing options. [![FlexMR logo](https://www.flexmr.net/skin/images/images-custom/2b221b67-b091-4584-9017-872ee2ddf960-logo.svg)](https://www.flexmr.net/) Book a Demo Request Pricing [Contact Us](https://www.flexmr.net/contact/) Menu - [Home](https://www.flexmr.net/) - [InsightHub](https://www.flexmr.net/insighthub/) - [What is InsightHub?](https://www.flexmr.net/insighthub/) - [Data Collection](https://www.flexmr.net/insighthub/data-collection/) - [Data Analysis](https://www.flexmr.net/insighthub/data-analysis/) - [Data Activation](https://www.flexmr.net/insighthub/data-activation/) - [Research Templates](https://blog.flexmr.net/topic/survey-templates/) - [Information Security](https://www.flexmr.net/insighthub/information-security/) - [Services](https://www.flexmr.net/services/) - [Our Expert Services](https://www.flexmr.net/services/) - [Support & Education](https://www.flexmr.net/services/support-and-education/) - [Consultative Services](https://www.flexmr.net/services/consultative-services/) - [Insight Delivery](https://www.flexmr.net/services/insight-delivery-services/) - [Research Methods](https://www.flexmr.net/methods/) - [Sectors We Work With](https://www.flexmr.net/sectors/) - [About](https://www.flexmr.net/about/) - [About Us](https://www.flexmr.net/about/) - [Leadership](https://www.flexmr.net/about/leadership/) - [Meet the team](https://www.flexmr.net/about/meet-the-team/) - [Advisory Board](https://www.flexmr.net/about/advisory-board/) - [Press & Media](https://resources.flexmr.net/tag/press) - [Accolades](https://www.flexmr.net/about/accolades/) - [Careers](https://www.flexmr.net/about/careers/) - [Blog](https://blog.flexmr.net/) - [Resources](https://resources.flexmr.net) - [Contact](https://www.flexmr.net/contact/) - [Book a Demo](https://blog.flexmr.net/) - [Request Pricing](https://blog.flexmr.net/) - [Home](https://www.flexmr.net/) - [InsightHub](https://www.flexmr.net/insighthub/) - - [What is InsightHub?](https://www.flexmr.net/insighthub/) - [Data Collection](https://www.flexmr.net/insighthub/data-collection/) - [Data Analysis](https://www.flexmr.net/insighthub/data-analysis/) - [Data Activation](https://www.flexmr.net/insighthub/data-activation/) - [Research Templates](https://blog.flexmr.net/topic/survey-templates/) - [Information Security](https://www.flexmr.net/insighthub/information-security/) - ![Camp InsightHub](https://blog.flexmr.net/hubfs/Website_Files/Images/Navigation/camp_insighthub_nav.png) Embark on a new adventure. Join Camp InsightHub, our free demo platform, to discover the future of research. [Sign Up Today](https://insighthub.flexmr.net/users/new) - ![FlexMR InsightHub](https://blog.flexmr.net/hubfs/Website_Files/Images/Navigation/insighthub_screenshot.png) Read a brief overview of the agile research platform enabling brands to inform decisions at speed in this PDF. [Download Brochure](https://blog.flexmr.net/hubfs/Public_Files/InsightHub_Brochure.pdf) - ### InsightHub on the Blog - [Surveys, Video and the Changing Face of Agile Research](https://blog.flexmr.net/surveys-video-changing-agile-research) - [Building a Research Technology Stack for Better Insights](https://blog.flexmr.net/research-technology-stack) - [The Importance of Delegation in Managing Insight Activities](https://blog.flexmr.net/delegation-importance-insight-activities) - [Common Insight Platform Pitfalls (and How to Avoid Them)](https://blog.flexmr.net/platform-pitfalls-how-to-avoid-them) - [Services](https://www.flexmr.net/services/) - - [Our Expert Services](https://www.flexmr.net/services/) - [Support and Education](https://www.flexmr.net/services/support-and-education/) - [Consultative Services](https://www.flexmr.net/services/consultative-services/) - [Insight Delivery Services](https://www.flexmr.net/services/insight-delivery-services/) - [Research Methods](https://www.flexmr.net/methods/) - [Sectors We Work With](https://www.flexmr.net/sectors/) - ![FlexMR Services Team](https://blog.flexmr.net/hubfs/Website_Files/Images/Navigation/services.jpg) Our services drive operational and strategic success in challenging environments. Find out how. [Download Brochure](https://blog.flexmr.net/hubfs/Public_Files/Brochures/FlexMR_Services_Brochure.pdf) - ![Video Close Connection Programme](https://blog.flexmr.net/hubfs/Website_Files/Images/Navigation/video_services.jpg) Close Connections bring stakeholders and customers together for candid, human conversations. [Read Press Release](https://resources.flexmr.net/close-connections-launch) - ### Explore our Practices - [Communities Practice](https://www.flexmr.net/insighthub/) - [Closeness Practice](https://www.flexmr.net/practices/customer-closeness/) - [CX Practice](https://www.flexmr.net/practices/customer-experience/) - [Insight Communications Practice](https://www.flexmr.net/practices/insight-communications/) - [About](https://www.flexmr.net/about/) - - [About Us](https://www.flexmr.net/about/) - [Leadership](https://www.flexmr.net/about/leadership/) - [Meet the Team](https://www.flexmr.net/about/meet-the-team/) - [Advisory Board](https://www.flexmr.net/about/advisory-board/) - [Press & Media](https://resources.flexmr.net/tag/press) - [Accolades](https://www.flexmr.net/about/accolades/) - [Careers](https://www.flexmr.net/about/careers/) - ![FlexMR Credentials Deck](https://blog.flexmr.net/hubfs/Website_Files/Images/Navigation/flexmr_credentials.png) The FlexMR credentials deck provides a brief introduction to the team, our approach to research and previous work. [Download Credentials](https://blog.flexmr.net/hubfs/Public_Files/Brochures/FlexMR_Credentials.pdf) - ![FlexMR Insights Empowerment](https://blog.flexmr.net/hubfs/Website_Files/Images/Navigation/insights_empowerment.png) We are the insights empowerment company. Our framework addresses the major pressures insight teams face. [Download Brochure](https://blog.flexmr.net/hubfs/Public_Files/Brochures/Insights_Empowerment_Brochure.pdf) - ### Latest News - [Insight as Art Shortlisted for AURA Innovation Award](https://resources.flexmr.net/aura-innovation-award-2021) - [FlexMR Launch Video Close Connection Programme](https://resources.flexmr.net/close-connections-launch) - [VideoMR Analysis Tool Added to InsightHub](https://resources.flexmr.net/videomr-launch) - [FlexMR Makes Shortlist for Quirks Research Supplier Award](https://resources.flexmr.net/quirksmr-mr-supplier-award-2020) - [Blog](https://blog.flexmr.net) - - [Latest Posts](https://blog.flexmr.net) - [Strategic Thinking](https://blog.flexmr.net/topic/strategic-thinking) - [Technology & Trends](https://blog.flexmr.net/topic/tech-trends) - [Research Methods](https://blog.flexmr.net/topic/research-methods) - [Practical Application](https://blog.flexmr.net/topic/practical-application) - [Insights Empowerment](https://blog.flexmr.net/topic/insights-empowerment) - [View Full Blog Archives](https://blog.flexmr.net/all) - ![FlexMR Close Connection Programme](https://blog.flexmr.net/hubfs/Website_Files/Images/Navigation/close_connection.png) Discover how to build close customer connections to better support real-time decision making. [Build Connections](https://blog.flexmr.net/practical-support-real-time-decisions) - ![Market Research Playbook](https://blog.flexmr.net/hubfs/Website_Files/Images/Navigation/market_research_playbook.png) What is a market research and insights playbook, plus discover why should your team consider building one. [Discover Playbooks](https://blog.flexmr.net/market-research-playbook) - ### Featured Posts - [Five Strategies for Turning Insight into Action](https://blog.flexmr.net/5-strategies-turn-insights-into-action) - [How to Design Surveys that Ask the Right Questions](https://blog.flexmr.net/effective-survey-design-guide) - [Scaling Creative Qual for Rich Customer Insight](https://blog.flexmr.net/scale-creative-qual-rich-understanding) - [How to Measure Brand Awareness: The Complete Guide](https://blog.flexmr.net/measure-brand-awareness) - [Resources](https://resources.flexmr.net) - - [All Resources](https://resources.flexmr.net) - [Client Stories](https://resources.flexmr.net/tag/case-studies) - [Whitepapers](https://resources.flexmr.net/tag/whitepapers) - [Events & Webinars](https://resources.flexmr.net/tag/events) - [The Open Ideas Panel](https://ideas.flexmr.net/auth/sign_in) - [InsightHub Help Centre](https://help.flexmr.net/en) - [FlexMR Client Network](https://www.linkedin.com/groups/9165407/) - ![Insights Empowerment Readiness Calculator](https://blog.flexmr.net/hubfs/Website_Files/Images/Navigation/insights_calculator.png) The insights empowerment readiness calculator measures your progress in building an insight-led culture. [Get Your Score](https://resources.flexmr.net/insights-empowerment-calculator) - ![MRX Lab Podcast](https://blog.flexmr.net/hubfs/Website_Files/Images/Navigation/flexmr_podcast.png) The MRX Lab podcast explores new and novel ideas from the insights industry in 10 minutes or less. [Listen Now](https://resources.flexmr.net/mrx-lab-podcast) - ### Featured Stories - [Specsavers Informs Key Marketing Decisions with InsightHub](https://resources.flexmr.net/specsavers) - [The Coventry Panel Helps Maintain Award Winning CX](https://resources.flexmr.net/coventry-building-society) - [Isagenix Customer Community Steers New Product Launch](https://resources.flexmr.net/isa-insights) - [Curo Engage Residents with InsightHub Community](https://resources.flexmr.net/curo) - [Tech & Trends](https://blog.flexmr.net/topic/tech-trends) / - [Research Methods](https://blog.flexmr.net/topic/research-methods) / 10 MIN READ ## Man vs. Machine: An Insight Community Analysis Challenge (IIeX Europe 2018) #### Charlotte Duff # [Workshops and Beacons: A Guide to Building Custome...](https://blog.flexmr.net/workshops-beacons-customer-salience) Most organisations are not short of data, insight or decks.Their biggest challenge is not whether cu... [Read more](https://blog.flexmr.net/workshops-beacons-customer-salience) 9 MIN READ #### Ed Hill - [Events (1)](https://blog.flexmr.net/topic/events) - [Insights Empowerment (38)](https://blog.flexmr.net/topic/insights-empowerment) - [Practical Application (178)](https://blog.flexmr.net/topic/practical-application) - [Research Methods (285)](https://blog.flexmr.net/topic/research-methods) - [Strategic Thinking (232)](https://blog.flexmr.net/topic/strategic-thinking) - [Survey Templates (7)](https://blog.flexmr.net/topic/survey-templates) - [Tech & Trends (397)](https://blog.flexmr.net/topic/tech-trends) Those of you who attended IIeX Europe earlier this month may have had the opportunity to hear me present the outcomes of our recent qualitative analysis exploration in ‘Man Vs Machine’. At FlexMR we very much believe in innovation and education so I was delighted to win the Best New Speaker Award for this presentation in particular. For those of you who weren’t able to make it to Amsterdam, I have condensed my 20-minute presentation into the blog that follows. It includes some fascinating learning for qual researchers, both agency and client side, that I am keen to share widely. ### Man Vs Machine: The Insight Community Analysis Challenge If you have read any of my previous blogs you’ll know that I *have *sat very much on the ‘man’ side of the debate – the side that believes that qual (or certain types of qual at least) need a human to analyse them accurately and effectively and that AI cannot completely replace the human touch. But what if? What if now, as we sit here today, I'm wrong… And the challenge was born... **Methodology** Rather than using an open text question in a survey (which we know works for text and sentiment analysis software), we wanted to see whether text and sentiment analysis can work for true scaled qual. By true scaled qual, we mean an online insight community where there is collaboration between participants, where the moderator gets involved, prompting and probing to unearth feelings and emotions. Here, participants aren’t just responding to the initial research question, they are discussing the wider issues surrounding the topic and building the context. Answers are reflective and considered. Our insight community centred round the home and living category, from home styles, to product advertising, inspiration, browsing and purchasing. It ran over 5 days with 50 participants and in those 5 days alone our participants generated 62,000 words. **Analysis** On completion of the insight community, we conducted two analyses, one via text and sentiment technology and one in the ‘traditional way’, via a qual researcher. The text and sentiment analysis output was then passed on to me to review and conduct a third combined analysis independent of any others. All three approaches were timed and the outcomes cross examined. To give you some context, the text and sentiment analysis software we used for this experiment was a very basic, entry level software. Our aim - to see whether the machined analysis of this rich data did offer any added value in principle (before taking it further). ### Outcome #1 - Qual Analysis Speed So, let’s get to it – the first question you are probably all asking, who was faster? Compared to the human only approach, the machine was a very clear winner. However, speed isn’t everything, despite the ever increasing responsive need at the forefront of everyone’s minds, there’s no point in being quicker if you’re not providing quality insight. Insight must remain accurate and above all actionable. The main positive that came from the combined approach - whilst it was slower than the machine it wasn’t anywhere near as slow as without. ### Outcome #2 - Qual Analysis Focus One of the main contributors to their analysis speed is the structure associated with each of the three approaches. The human analyst has no structure initially, they literally have pages and pages of transcript to read through and make sense of. This reading and sifting however, does ensure an understanding of the context. The machine with its pre-programmed algorithms provides structure in its output but it doesn’t provide the context behind the charts – severely detrimental to insight quality. The machine output is an excellent starting point for the combined approach. It provides an immediate focus for in depth exploration and chart contextualisation. Without doubt this is one of the biggest time saving aspects of text and sentiment analysis for insight community outcomes. |   | ***Tweet This*** | | --- | --- | | [Text and sentiment software is an excellent starting point for insight community analysis. It provides an immediate focus for in depth exploration and chart contextualisation](https://ctt.ec/HO1_z) | | ### Outcome #3 - Qual Analysis Objectivity Even the most neutral researcher will ‘go in’ with some element of confirmation bias. Because they are expecting (however subconsciously) to see something or not see something, they may sift through the data with a focus on proving/disproving their expectations. The home and living ‘Amazon surprise’ revealed by the machine provided us with a great example of this. The qual researcher didn’t expect Amazon to play a significant part in vintage shopping experiences. Their analysis did highlight Amazon as being heavily featured in generic home and living shopping and purchasing but it didn’t pick up on vintage specifics. The text and sentiment analysis highlighted Amazon’s role (4th most mentioned brand) in vintage shopping. It didn’t explain why however. The machine has no confirmation bias but as previously mentioned it doesn’t give any context at all. With the benefit of the machined charts, I examined the vintage Amazon association in a human transcript deep dive, pulling this and the reasons for it out as key learning. For manufacturers and retailers dealing in handmade, vintage or bespoke home and living items this finding as well as a true understanding of its basis would be fundamental. ### Outcome #4 – Qual Analysis Sentiment Text and sentiment analysis software can produce a sentiment analysis for any and every community discussion output you choose to run through it. Again though, there is no context to explain sentiment chart reasoning so it’s impossible to understand what’s driving it. Indeed, my human transcript deep dive demonstrated that on many occasions the machine based sentiment analysis was totally incorrect. |   | ***Tweet This*** | | --- | --- | | [Text and sentiment analysis software can produce a sentiment analysis for any and every insight community output you choose to run through it... But that doesn't mean it's correct - Use with caution](https://ctt.ec/ded5R) | | The big learning here – when using automated sentiment analysis on insight community qual use it with caution! Results should never be thrown around as definitive sentiment until you have done a human sense check. Why doesn’t the machine always provide accurate sentiment? Quite simply, insight community discussions often expand into and flow between different topic subjects and as sentiment analysis doesn’t read context it doesn’t know which subject the sentiment relates to. It will also conclude a negative sentiment based on a high number of negative words within the comment stream where the contextual word meaning it is much more light-hearted and positive. Because of this, I found that sentiment analysis worked best with topics that are more concept orientated, or discussions that are ring fenced by their nature. In these circumstances it was very effective. ### Outcome #5 - Qual Analysis Empathy Having no statistical validation to back up their interpretations, qual researchers have a tendency to sit on the fence and not commit to a sentiment in their analysis unless it is especially clear cut. Perfectly highlighting this point, our traditional qual analyst didn’t commit to an overall sentiment for the community discussions surrounding brand charitable promotions. This on the other hand, was one of the few machined sentiment charts that I did pull out in the combined approach. Having sense checked the sentiment for accuracy my deep dive unearthed some fundamental home and living brand associations that would take our insights next level. My analysis established the ‘why’ and I was then in a position to run my own text analysis to statistically validate or otherwise. Without doubt the machine added value here. It took me down a path our traditional qual analyst didn’t go down for a deeper understanding. The combined approach established empathy supported by statistics. On the client side the additional insight provided would have incredible weight. ### Man Vs Machine: The Insight Community Analysis Conclusion As has likely become apparent when reading our outcomes above, a traditional insight community analysis approach will give you ‘the what’ and ‘the why’ but it lacks the statistical validation that many clients and stakeholders look for before making a business decision. The machine, gives you ‘the what’ and the statistical validation but there’s no why. In reality you may be misguided by relying solely on the machine analysis. The combined approach, using text and sentiment analysis followed by a human community qual deep dive, ticks every box - the ‘what’, the ‘why’ and statistical validation. It takes the empathic positives of the traditional approach and builds on this with the focus, objectivity, sentiment and statistical validation provided by the machine. Collectively they elicit a deeper and more confident interpretation. |   | ***Tweet This*** | | --- | --- | | [Combining text and sentiment software with a human qual deep dive ticks every insight community analysis box - the 'what'. the 'why' and statistical validation](https://ctt.ec/c2a70) | | ### The Insight Community Analyst 2.0 So, we proved that including a machined analysis element does in fact aid scaled qual insight. To help those of you interested in following in our footsteps we’ve put together a 6-step ‘man and machine’ analysis process for success. It is naturally slightly different to the challenge process. Based on my experience as the combination insight community analyst the real world process that follows is the optimum in terms of agility. **Step 1.** Assuming the analyst is different to the moderator, the first step is to immerse them in the topic. They need to familiarise themselves with this, the discussion guide and research objectives. **Step 2.** The analyst should sanitise the transcript making sure they eliminate moderator comments – vitally important as these will bias the machine analysis leading to inaccurate interpretation. **Step 3.** Run the text and sentiment analysis software. **Step 4.** The analyst should then review the text and sentiment output, identify emerging themes and run any additional charts required based on their knowledge of the research objectives. **Step 5.** At this point the analyst and moderator should confer. They moderator will know if there any themes that haven’t been picked up by the machine analysis. If there are the analyst can run extra charts as necessary. **Step 6.** Finally the analyst deep dive, using the text and sentiment analysis charts as a guide for focus and objectivity. By doing this, not only will you see a huge time saving compared to that of traditional qual analysis (we saved roughly 50% analysis time on a 5 day community), you will also add value client side with improved insight quality. On top of that, your clients or stakeholders will have the statistical validation required to be able to confidently make a business decision off the back of your insight community rather than needing to run any follow up quant. Now, that’s impact! #### About FlexMR We are The Insights Empowerment Company. We help research, product and marketing teams drive informed decisions with efficient, scalable & impactful insight. #### About Charlotte Duff Winner of the IIeX Europe Best New Speaker Award, Charlotte’s research and communication experience is invaluable to our clients. #### Stay up to date ## You might also like... ![Blog Featured Image Header](https://blog.flexmr.net/hubfs/Marketing_Files/Blogs/Headers/retaining_human_touch_customer_salience_cx_ai.png) ### [Retaining a Human Touch with Custom...](https://blog.flexmr.net/customer-salience-cx-ai) In recent years, companies have embraced artificial intelligence (AI) at pace, swept up in a technological momentum that promises efficiency and cost savings. However, in their rush to pursue quickly ... [Continue reading](https://blog.flexmr.net/customer-salience-cx-ai) 5 MIN READ ![Blog Featured Image Header](https://blog.flexmr.net/hubfs/Marketing_Files/Blogs/Headers/casual_conversation_iosetta_santini.png) ### [A Casual Conversation... with Ioset...](https://blog.flexmr.net/casual-conversation-iosetta-santini) Most research projects still currently end with a powerpoint report and a debrief. This has become deeply entrenched as the industry standard, and it is what stakeholders have come to expect when they... [Continue reading](https://blog.flexmr.net/casual-conversation-iosetta-santini) 4 MIN READ ![Blog Featured Image Header](https://blog.flexmr.net/hubfs/Marketing_Files/Blogs/Headers/synthetic_data_richer_market_insights.png) ### [Synthetic Data: The Key to Richer M...](https://blog.flexmr.net/synthetic-data-richer-market-insights) In the dynamic world of market research, the need for robust and accurate data is critical. Whilst the increased prevalence of generative AI has provided a range of benefits to market research process... [Continue reading](https://blog.flexmr.net/synthetic-data-richer-market-insights) 7 MIN READ ![](https://www.flexmr.net/skin/images/images-custom/48f118ee-cef9-42d6-b60b-5575d5ac26e3-flexmr_footer.svg) - [Home](https://www.flexmr.net/) - [Contact Us](https://www.flexmr.net/contact/) - [Privacy Policy](https://www.flexmr.net/legal/privacy/) - [Cookie Policy](https://www.flexmr.net/legal/cookies/) - [Terms of Use](https://www.flexmr.net/legal/terms-of-use/) [Follow us on Facebook](https://www.facebook.com/flexmr/) [Follow us on Twitter](https://twitter.com/_flexmr) [Follow us on LinkedIn](https://www.linkedin.com/company/flexmr) [Follow us on YouTube](https://www.youtube.com/c/FlexmrNetOnlineResearch) [![Grit Top 50 Logo](https://www.flexmr.net/uploads/photo-gallery-include-images/34/44258f4d-7452-40d6-bb23-0e220c8ba21f.png)](https://www.greenbook.org/mr/grit/business-innovation-edition) [![ISO 27001 Alcumus](https://www.flexmr.net/uploads/photo-gallery-include-images/34/fa89f9b3-d866-43b8-8de0-4a052a9053d8.png)](https://www.alcumus.com/en-gb/certification/iso-27001/) [![Cyber Essentials Plus Logo](https://www.flexmr.net/uploads/photo-gallery-include-images/34/cba57df6-75a2-4544-9599-dcc0c2f14b79.png)](https://blog.flexmr.net/hubfs/Public_Files/Other/FlexMR_Cyber_Essentials_Certificate.pdf?__hstc=163229475.dde2c3d212762f49b059c9f62e7537a0.1624542718433.1626339489304.1626701624468.7&__hssc=163229475.2.1626701624468&__hsfp=3142549060) © FlexMR 2021. All rights reserved. ```json { "@context" : "https://schema.org", "@type" : "BlogPosting", "author" : { "@type" : "Person", "name" : "Charlotte Duff", "url" : "https://blog.flexmr.net/author/charlotte-pearson-duff" }, "dateModified" : "2021-07-21T22:37:31.509Z", "datePublished" : "2018-08-31T14:33:45.000Z", "headline" : "Man vs. Machine: An Insight Community Analysis Challenge (IIeX Europe 2018)", "image" : [ "https://blog.flexmr.net/hubfs/4841069-6-man-vs-machine-an-insight-community-analysis-challenge-iiex-europe-2018.png" ], "mainEntityOfPage" : { "@id" : "https://blog.flexmr.net/insight-community-analysis-challenge-iiex-europe-2018", "@type" : "WebPage" }, "publisher" : { "@type" : "Organization", "logo" : { "@type" : "ImageObject", "url" : "https://blog.flexmr.net/hubfs/Website_Files/Images/Logos/flexmr_logo_amp.png" }, "name" : "FlexMR" } } ```