Picture this: In the vibrant atmosphere at our recent conference, a question danced on everyone’s lips:
Our answer to this is a mix of “yes” and “no.” At Caplena, we primarily use our custom AI model for core tasks and ChatGPT for the summarization feature. Drawing a parallel between Caplena and ChatGPT in text analysis might seem like a riddle wrapped in a mystery. But it’s more like contrasting a powerful motor with a cutting-edge vehicle. Each comes with a unique prowess and purpose, aptly suiting distinct scenarios. But are they identical twins in the AI family? A resounding ‘no’ rings clear. Let’s delve into the reasons why๐.
ChatGPT, along with its GPT4 version, is a powerful tool, providing human-like responses to a broad array of questions. In contrast, Caplena, a feedback analysis platform, uses a specific AI model optimized for the task of categorizing large amounts of text data into distinct topics. Naturally, the question arises as to which tool offers a more effective analysis of customer feedback. In this study, we aim to compare the task of accurately categorizing sentences into relevant topics.
To effectively compare Caplena and ChatGPT, we first need to define the task we’re evaluating. Our focus here is the process of topic assignment. Given a predefined set of topics, our goal is to determine their frequency of appearance in user feedback. To accomplish this, the AI needs to categorize each text into one or more relevant topics.
The fundamental difference between Caplena and ChatGPT in topic assignment is that Caplena takes a direct approach, focusing specifically on topic assignment as its core task.
On the other hand, ChatGPT operates in a chat-like format, designed for engaging and interactive conversations. While ChatGPT can understand and respond to various questions, its primary objective is not that of topic assignment. Instead, ChatGPT aims to generate human-like responses, making it indirectly capable of addressing the task of topic assignment in the conversation.
When comparing the user interfaces of ChatGPT and Caplena, ChatGPT utilizes a chat interface for receiving and generating conversational responses. Categorizing topics within sentences requires constructing a prompt in the form of a conversation, rather than directly inputting the text. Crafting a comprehensive prompt becomes crucial in ChatGPT’s approach. Let’s take a look at an example of how a prompt similar to this one might appear:
In our experiment, ChatGPT and Caplena went head-to-head, pitted against each other using a test dataset of 22 surveys. We employed the F1 score as our primary evaluation metric to assess the model’s accuracy on a scale of 0% to 100%. This metric measures how well the model aligns with a human-annotated dataset considering both precision and recall, which helps handle the uneven class distribution.
Our evaluation revealed that, overall, Caplena’s AI delivered superior accuracy. However, there were four instances where ChatGPT took the lead as illustrated below. Calculating the average F1 score across all 22 surveys, ChatGPT clocked in at 44%. Caplena’s AI scored a higher average of 56%. For reference, a naive model that randomly guesses will typically achieve an F1 score of approximately 5%. This difference is noteworthy despite a high degree of variance within the dataset. Although the numbers may seem similar at first glance, Caplena takes an additional step that significantly increases the percentage from 56%.
Caplena’s standout feature lies in its remarkable fine-tuning capability. This enables a significant improvement from an initial F1 score of 56% to an ambitious target of 70% or higher. As mentioned earlier, a naive model that randomly guesses will typically achieve an F1 score of approximately 5%. A study by Ishita, Oard, and Fleischmann found that a fully manual human analysis only achieved an F1 score of 62.7%. This proves that humans are not infallible coders. Achieving a score of 70% places Caplena’s precision analysis on par with – or surpasses – a human-level performance. While the initial score of 56% shows good performance in many scenarios, it falls short of human accuracy. Especially in more complex scenarios like lengthy reviews or topics with semantic overlaps.
To overcome these challenges, we have implemented a streamlined process that actively involves users. In this process, users validate model outputs for a small subset of the dataset. Specifically, users review whether Caplena’s categorization is correct or incorrect for a select number of test cases. Working with the AI usually involves a couple of dozen rows. The manually validated data is then used to fine-tune the model to the specific task of the user, improving the model’s score on average from 56% to 66% in our setting, ensuring that it aligns with the user’s requirements.
To maintain transparency in our performance metrics, we use a portion of the human-validated data as a test set to estimate the F1 score. This evaluation method allows users to gauge the accuracy of the Caplena analysis. Users can monitor the evolution of their F1 score in real-time, providing insights into the model’s performance. We recommend considering an F1 score of 65 – 70 to indicate satisfactory accuracy. For a fair comparison, we used manually validated samples to also enhance ChatGPT. In particular, we employed a technique called few-shot learning. This involves including some examples with solutions in the prompt. However, ChatGPT struggled to benefit from this extra information in the prompts. This resulted in a decrease in ChatGPTs overall score from 44% to 41%.
Our study encountered challenges due to ChatGPT’s prompt size constraint. This limited the number of verified samples that could be included in the few-shots learning prompt and hindered a higher F1 score. Thus, we had to cap Caplena’s F1 score at 66%, while ChatGPT achieved 41%, ensuring fairness in the experiment. However, it should be noted that Caplena can achieve scores of 70 and above in real-life applications.
Even without few-shot learning, we observed that ChatGPT struggles with very long prompts. While the hard limit of 3000 words in ChatGPT was sufficient to analyze all the surveys in our sample, we noticed a decline in performance when analyzing questions covering a wide range of topics. In contrast, Caplena’s model, specifically trained for this task, does not suffer from this issue in surveys with many diverse topics.
Fortunately, the successor model, GPT4, is capable of handling much longer sequences, up to 20,000 words. This development has us excited to test it out. However, at present, the API for GPT4 is slow, and frequent lockouts occur due to the high volume of requests. We eagerly anticipate analyzing the model in the future.
Also, there were instances where ChatGPT struggled to interpret our prompts correctly. This resulted in responses like:
โCannot classify the 35th review as it seems to be incomplete or unrelated to the topic list.โ
โLittle confused by this review, it doesn't seem to provide enough information to assign a topic. Can you provide more context or information to help me understand the review better?โ
โ1,2,3 all have [TOPIC]โ (when multiple reviews are included in a prompt).
In sum: Caplena outperforms ChatGPT in the precisely defined text analysis tasks of topic assignment. Now, let’s unpack other elements contributing to Caplena’s unrivaled performance in the task of assigning topics to text.
ChatGPT excels in text summarization, understanding and responding in a human-like manner to various prompts. Still, it is crucial to acknowledge its limitations in analyzing quantitative customer feedback topics. ChatGPT’s summarization feature is impressive, but it should not replace a comprehensive and precise textual analysis for optimal results.
During this ongoing beta testing phase, we are effectively combining the best of both worlds. Caplena carries out the precise analysis, and ChatGPT provides a concise summary of the analyzed data. With this combination, one can quickly obtain a data overview in a few sentences, accessible with a single click. This ensures efficiency and a user-friendly experience, streamlining the data analysis process.
To wrap up, Caplena stands out as the superior choice for customer feedback analysis over ChatGPT for several key reasons:
ChatGPT is an innovative AI tool that excels in generating human-like responses and offering contextual information. However, it is important to recognize both the strengths and limitations of ChatGPT. In the specific task of determining topics and their frequency of appearance in user feedback, Caplena clearly excelled. Caplena achieved an impressive overall score of 66%, surpassing ChatGPT’s 41% F1 score. Without the limitations imposed by this study, Caplena has the potential to achieve scores well beyond 70%, reaching above-average human-level accuracy. This is not surprising, given its purposeful design for the task, while ChatGPT serves a different use case.
Having said that, Caplena and ChatGPT are not in opposition; they complement each other. By integrating ChatGPT’s summarization feature into Caplena’s dashboard, users can leverage the strengths of both tools. This allows for valuable insights, utilizing powerful text analytics and visualization tools. This collaboration bridges the gap between sophisticated language models and advanced text analysis, expanding the boundaries of what can be achieved in this field.
So, is it really an “us vs them” question?
No. It’s an “us and them” opportunity for innovation and progress in the realm of text analytics.
At Caplena, we believe in the power of collaboration and teamwork. That’s why when it came to defining our core values, we knew we had to involve everyone in the company. So, we packed our bags and headed off-site for a fun-filled journey of discovery. We took a hands-on approach, holding workshops and brainstorming sessions, allowing everyone to share their thoughts and ideas. It was amazing to see how each person brought their unique perspectives and helped shape the values that define us.
After much discussion and collaboration, we emerged with a clear understanding of who we are as a team. These values not only guide us on our journey towards success – they also make the process of recruiting and onboarding new employees a breeze.
So, buckle up, and get ready to dive into the three core values that set us apart at Caplena!
At Caplena, each team member is different, has a strong desire to solve problems and is considered as an individual contributor. In other words, a healthy self-driven and passionate human-being, bringing their very-own (and different) value, contributing to our mission in their very-own way.
As a company, we are not process-driven. We encourage everyone to take ownership and be their better self, so everybody can learn a little bit more from each other every day, and finally create solutions.
Of course, we are not the only ones on the market. But we are ambitious, full of energy, proud of our tech and of our product. We want to be the best-in-class text analysis SaaS on the market! As a team, we strive for quality and not quantity, in everything we do.
Thanks to our powerful AI solution, we make our customers shine by helping them realize the potential of their business. And thanks to our international and globally distributed team of more than 10 different nationalities, we accompany them in their journey to customer satisfaction.
As a company we care about each other. Our team has low ego, and honest and direct feedback always goes in all directions. This is how we grow as a team.
We hire wholesome people who are nice and kind to each other, humble and happy to share moments together. Be it when going for a run or pizza (or both!) along the lake or when gathering for a larger team event, there is always space for a pinch of fun.
In conclusion, defining our core values was a journey that brought us closer together as a team and solidified who we are as a company. We’re beyond excited to share them with the world and use them as a guiding light in all of our decision-making. Of course, finding your company’s values isn’t a one-and-done task, but it’s a crucial first step in building a strong, inclusive culture that will help you attract and retain top talent. So, don’t be afraid to jump on the values discovery bandwagon and let the journey begin!
Don’t spend hours researching โ we’ve done the work for you. Explore the top five customer feedback tools of 2022, whether you’re a consultant, insight team member, or market researcher.
Caplena is one of the most modern customer feedback tools in 2022. Based in Zรผrich, Switzerland, Caplena is a fast-growing company trusted by top product-led organizations, including Swisscom, eBay, DHL, and Gymshark, to name a few. This customer feedback tool provides its users with translation for 31 languages utilizing two integrated translation engines. Caplena provides its user with fast, high-quality analysis of human sentiment autonomously. Want to ensure you are GDPR compliant when conducting research? Remove personal identifiable information (PII) from feedback by using the brand-new anonymization feature. One of the best things about this customer feedback tool is that it provides its user with a fun and easy-to-use interface. The dashboard is quite intuitive and user-friendly with a clean interface. After data analysis, Caplena offers different options to present your findings, for example, with a bar chart, driver chart, or a treemap.
Overall, with a free 2-week trial and the option to book a free demo, Caplena offers one of the best opportunities to test out the customer feedback tool and find out whether it’s the appropriate solution for you!
Thematic is a customer feedback tool that is well-known among the CX and insights community. As the name suggests, Thematic offers thematic analysis on its platform. Thematic was created specifically as an automated feedback analysis customer feedback tool. Thematic discovers themes in any feedback source, automatically and without training. With its core AI text analysis, Thematic offer a ‘1-click analysis’ feature for its users. The dashboard interface is quite smooth and user-friendly making data analysis a breeze! Thematic allows one to edit the data within the ‘themes editor’ section of the platform. All-in-all, this customer feedback tool is best suited for those who collect feedback from various sources regularly, such as surveys, live chat, and complaint reviews.
Launching its powerful text analysis tool, Text iQ, Qualtrics offers a customer feedback tool that provides topic flexibility, adaptability, and detection. Text iQ offers customers two packages: Basic and Advanced. The basic package for customer feedback is quite limited for large organizations and only allows a maximum of 20,000 responses per survey, excluding a summary mode. Further, recommended topics, translation, and sentiment analysis are only available in the advanced package, restricting the user to a basic analysis without true insight into your customer’s feedback. However, if you can splash the cash on the advanced package, you get access to a customer feedback tool that will help you in your feedback analysis. One issue we’ve found with Text iQ is the user experience for data visualization. Unlike other platforms, the Text iQ platform is not the most intuitive and for the everyday user, the dashboard design could be improved. Nevertheless, if you can ignore its straightforward visual appearance, Text iQ is a useful customer feedback tool to uncover deep insights.
Founded in 2009, Gavagai has been designed, built, and trained to enable customer experience teams to learn more about their customers! For those dealing with customer feedback in various languages regularly, this multilingual customer feedback tool may be the solution for you. Unlike many text analysis tools, Gavagai provides a comprehensive solution for those analyzing the Nordic tongue, a niche market that Gavagai seems to get right, as well as 46 other languages. As with Text iQ, Gavagai’s user experience could be improved by becoming more visually appealing. The platform’s font choice is also an area for improvement. However, it does provide an easy step-by-step process to turn your data collection into true insights. If you want a straightforward customer feedback tool and do not mind the look and feel of the user experience then Gavagai is a useful tool that will complement your customer experience team’s work.
Originally founded as Language Logic LLC, Ascribe processes hundreds of millions of open-ended text comments annually for its business partners in over 100 countries worldwide. For over two decades, Ascribe has served its clients, with old-school market researchers using this text analysis tool as the default. Unlike other customer feedback tools, Ascribe involves a lot more manual analysis, something we found quite frustrating. However, after the initial data analysis, Ascribe makes up for time wasted by offering vast options to present findings. For example, Ascribe offers users bar charts, pie charts, and relationship charts, as well as word clouds, trend lines, star charts, and a correlation matrix. While Ascribe offers a wealth of information in its visualization dashboard, it may feel congested and lacking in user-friendliness.
In conclusion, the top five customer feedback tools of 2022 are all excellent options for businesses looking to collect and utilize customer feedback to improve their products and services. These tools provide a range of features, including the ability to collect feedback through multiple channels, analyze and report on feedback, and even integrate with other business systems. Ultimately, by choosing one of these top five tools, businesses can gain valuable insights from their customers and use those insights to drive continuous improvement and maintain a competitive edge in the marketplace.
Two words: Black. Friday. For those in the US, the day after Thanksgiving kicks off the holiday season with huge discounts on products in-store and online. In recent years, the tradition has grown in popularity overseas with countries such as the UK, Ireland, France, and Germany participating in the retail phenomenon. Not sure what to buy this year? We’ve decided to put together a list of the top 10 practical Black Friday products you didn’t know you needed based on their AMAZING reviews (and ones to watch in the upcoming sales)!
We’ve decided to help you out and do the tedious process of evaluating these products’ reviews – only at a much larger scale. For each product, we took thousands of reviews and churned them through Caplena to see what people are really talking about. Let’s begin!
Avoid unnecessary pressure on the wrist and improve your posture with an ergonomic mouse this Black Friday! Ergonomic products are designed to help you work or play more naturally. Using Caplena’s integration feature for Amazon, the Caplena team discovered a few key insights into Lekvey’s ergonomic mouse. Firstly, the wireless mouse “is exactly as described”, with 68% of customers complimenting its ergonomic design. Some customers commented on the comfort it brings to them as it “fits [your] hand so perfectly,” while others enjoyed the product’s ‘lightweight’ and ‘quality’ feel. If you do have issues with the product, Lekvey’s customer support is very useful, with 26% of positive reviews praising their “polite and responsive customer service”. As well as that, one of its unique selling points is the fact that you can recharge the mouse rather than having to replace its batteries. In summary, if you are more than a casual user of computer mice, adding an ergonomic mouse to your Black Friday shopping list is a no-brainer.
During the Summer, it’s quite easy to go for walks or runs after work in the evening. However, by Winter, there’s no way you can continue doing the same sporting activities in the dark. Well, now you can! Get yourself a headlamp for every night activity. Need to go for a run? Want to walk the dog? Have to collect surveys and conduct feedback analysis? The night headlamp is a practical Black Friday solution for you.
Are you sitting comfortably for the next product on the Black Friday list? Think again! Investing in a Herman Miller office chair, such as the Ergon or the Aeron Chair, is one way you can invest in your back posture this Black Friday. Herman Miller was founded in 1905 and is a pioneer in modern design, working with the very best designers to create elegant ergonomic seating. As Herman Miller office chairs are normally quite expensive, keeping an eye out for a good Black Friday offer may help save you a few quid!
Since COVID-19, flexible working has become more commonplace. Taking standing desks to the next level, THERUN 2-in-1 Under Desk Treadmill offers hybrid and remote workers the opportunity to work and exercise simultaneously. Its 2-in-1 capabilities enable one to run in full treadmill mode as well as walk when folded. It has 8 shock-absorbing cushions built-in to provide cushioning for the knee, muscle, and joints. The treadmill has a LED touch screen plus it’s quite compact for easy storage.
Using Caplena for customer feedback analysis, we discovered that the treadmill’s usability features are the product’s key driver with 75% of customers complimenting the “excellent compact treadmill”. A unique insight we discovered considered the delivery of the product: For 11% of customers, the treadmill arrived broken or damaged. Using the driver chart, it’s clear that delivery is the product’s biggest weakness and that the supplier needs to reassess its shipping methods. However, if you think the odds are in your favor and you want to become more active during office hours, this treadmill may be a wise investment this Black Friday.
It doesn’t matter whether you are a full-time student or a CEO, if you don’t have an air fryer, then you need to invest in one! It’s a quicker, more energy-efficient, and healthier way to cook. Although it’s recommended to use a small spoonful of cooking oil, you can get away with using almost no oil and still enjoy delicious fried cod, crispy chicken thighs, and roasted vegetables to name a few. Air fryers take up little space on the kitchen countertop and are relatively quiet. The advantages are clear: It’s a delicious cost-saving Black Friday bargain.
Some people throw instant coffee into a cup of hot water and think they know what coffee is. These people haven’t lived! If you are a real coffee lover (like the Caplena team) then investing in a real coffee machine this Black Friday is CRUCIAL. The Sage Barista all-in-one express coffee machine has everything: Grind your coffee beans, get your expresso shot and use the steaming function to make the perfect cappuccino. Is it past morning time already? Have an Americano instead. Having a real coffee machine at home is a fun way to learn and appreciate the barista craft. As the saying goes: There’s no rush in perfection.
Looking for the perfect Black Friday gift for someone? We analyzed 8,434 customer reviews and discovered that over 97% of reviews were of positive sentiment. Customers shared how the reliable and practical Swiss Army knife was a “useful starter knife for teens” and the “perfect gift” for a family member or friend. Whether it’s for simply enjoying Swiss cheese or surviving in the Swiss alps for a few weeks, the original Victorinox Swiss army huntsman pocket knife is the perfect travel companion. To be extra authentic, opt for the red pocket knife to really impress your friends when you talk about your mountaineering adventures.
Let’s be honest, having multiple adapters to bring on your travels is a pain. Having an adapter for an adapter is an even bigger pain. Let us solve this issue once and for all: just buy the all-in-one worldwide travel adapter this Black Friday. This travel adapter won’t let you down in any country and is the sustainable option in a world full of travel adapters.
Initially, we believed that buying the iRobot Roomba s9+ robot vacuum would be a very practical Black Friday product to invest in. Who wouldn’t want a robot to do their chores? However, upon closer analysis of open-ended feedback, we realized that we were deceived by the robot’s 4-star rating. Using Caplena, we analyzed 5,159 reviews and skipped those that had empty responses as they contributed little to the overall analysis. We discovered that the iRobot Roomba s9+ should really have a 3-star average rating, with 52% of customers being unsatisfied based on its Customer Satisfaction (CSAT) score. We also found that 50% (2,822 responses) of negative sentiment concerned the usability of the robot. The customer’s biggest pain points considered its usability, product quality, pricing, and the lack of customer support. If you plan on spending nearly โฌ1k on a robot vacuum this Black Friday, we recommend avoiding this product and not becoming the babysitter to the robot that “constantly gets stuck”.
After de-stressing with a notable well-being app, it’s time to put your feet up and open a nice bottle of wine. But what if you only want one glass this Black Friday holiday? You don’t want the bottle to spoil. Introducing the world’s first wine condoms! This quirky gift is fun and practical. Never worry about spillages again! Airtight wine condoms also help preserve the wine’s freshness. Plus, the protection doesn’t add any extra height to your bottle of wine, making it easier to put your wine back into the fridge. They’re also very portable and slip right into your pocket or purse for dinner and house parties. If we haven’t convinced you yet, wine condoms, (unlike other condoms) are reusable, proving to be a very practical Black Friday purchase!
Black Friday 2022 is taking place on Friday, the 25th of November 2022, with Cyber Monday following it on Monday, the 28th of November 2022. Black Friday sales usually begin two weeks before the holiday and end around the second week of December just before the Christmas holidays. Whether it’s buying gifts for friends and family or investing in useful tools for the home or everyday use, take advantage of the special offers this coming Black Friday!
Delicious hot chocolates โ, Christmas markets, and joyful caroling ๐ถ are in order! Despite enjoying this emotionally charged time of year, Caplena cannot help but flaunt our true nature – after all, we are artificial intelligence that categorizes emotions and sentiments better than it feels them – we leave that to the humans ๐. We are aching to categorize something, and what could be more Christmassy than jolly Christmas markets? ๐ โจ
What are the most common things people mention when they happily talk about their most recent Christmas market visit? To find out, we studied three highly rated – but different – Christmas markets to analyze their Google reviews.
Perfecting the Christmas Markets’ Spirit is a scientific process.๐ฉโ๐ฌ Have you ever been to an amazing Christmas market that only had one good thing? Us neither. At Caplena, we contend that the key to that jolly feeling of love and togetherness is a blend of various topics. A mix alone, however, isn’t enough. The more the ingredients are balanced, the better! Using the Treemap, we can see the derived topics and their balance within our scientific recipe.
Our treemap of Christmas topics appears to be relatively balanced. No one topic dominates half of the space. Let’s examine the topics in more detail.
A dash of this and that…
Taking the top two spots are aesthetic-related codes. An ideal location (i.e., the general atmosphere of the city or market) is key. Smaller details follow; think fairy lights, fragrances, a shiny garland entwining pine boughs, glass ornaments, and all the other little touches bringing the season to life.
*Bonus: if the place has a historical vibe to it ๐
๐ Food & Mulled Wine
Food is the way to the โค๏ธ
The treemap proves this point clearly. All foods from different countries and cultures were combined into one code. Mulled wine is just like wine, but with a holiday spirit. In fact, it had such a high volume of mentions that it deserved its very own code. Because wine not?
The food code ties into every positively associated code, binding all the good elements of a Christmas market together. In fact, there is barely a code involving food not combined with one another. Have a look.๐
๐ Fun Activities
People like to eat, but (after) they have filled their bellies, some like to participate in light activity to digest all the delicious food they just consumed. In activities, ice skating takes first place by far. Also on the list are other attractions like bumper rides ๐ (although you might want to wait a while after eating for those)๐.
๐ Christmas Music
๐ต Santa Baby, so hurry down the chimney tonight ๐ต Christmas isn’t Christmas without music. Researchers examined people’s ability to recall memories after hearing a clip from a song in a 1999 study. During that time, they were able to recall specific emotions and events. Listening to some Christmas tunes might make you wistful for your childhood or simply make you feel festive and happy ๐.
๐ Family Friendly & Jolly Staff
It might seem like a minor point – but nothing destroys a wonderful Christmas market more than rude staff or people complaining about kids being there. Yes – not all of us are huge fans of children – but they represent the innocence and pure joy that Christmas tries to instill in us all ๐ช.
๐ Shopping
Christmas Markets wouldn’t be complete without… the market! Ironically the very thing it is named after comes in last on the list of Christmas market staples.
Whilst most of the reviews are quite positive, BEWARE of the few things that very much bothered people. You can best measure satisfaction vs impact using the driver chart.๐
Christmas Markets only suffer from three major negative factors. If the market is too expensive, for instance, the negative NPS score is quite high. A small or crowded market comes second. Finally, although ice skating is the third most important code, it carries the highest risk of disappointment due to paying a lot, rude staff, and long wait times.
Having proven what makes an amazing Christmas market – and what to avoid – let’s turn our attention to you, our customers! Thanks for being a part of our success. We wish you a happy holiday season. Now that you have all the ingredients that make Christmas markets so magically Christmassy – you are sure to have a beautiful Christmas holiday (backed by facts) ๐๐
