Pilot 3: what we learned from the last round of user testing
/As part of the third pilot, news consumers tested the CPN app for a period of four weeks, providing us with valuable feedback on the performance of our recommender.
During the third pilot, which took place in January and February this year, the news audiences of VRT, DIAS and Deutsche Welle, the three media partners in the project, were able to test the CPN recommender by accessing the publishers’ news content through the CPN software.
Through monitoring the usage of the app, we were able to measure user engagement, article diversity and viewing of long-tail articles. Furthermore, test users provided us feedback through surveys, which allowed us to measure feelings of missing out / being informed, and much more.
The third pilot concluded the user testing part of the CPN project. (You can read more about the previous rounds here: First Pilot, Second Pilot.)
The CPN recommender with Deutsche Welle content
Key findings
Analysing the data and feedback collected from the test users, we have identified four major learnings from the pilot.
Different recommendation engines have different effects. We noticed that content-based approaches provided more engagement. In other words, we saw longer read times and higher scroll depth when readers were presented with articles that were recommended to them based on content. Moreover, long-tail articles, which typically get less views, were read more often thanks to content-based recommendation.
Personalisation did not lead to a perceived filter bubble. News professionals and audiences sometimes fear that personalisation can put them in a filter bubble (something we discussed in depth here). However, in our evaluation, we saw no difference among users in the control group vs. the personalised group, as both groups reported similar experiences.
Different recommenders provide different results. The hybrid recommender provided very good results in terms of article diversity, whereas the content-based recommender decreased diversity. (The hybrid recommender uses a mix of different techniques in order to overcome the weaknesses of a single recommender system, whereas the content-based recommender provides recommendations based purely on the consumed content. Read more about the CPN recommender engine here.)
Feeling of informedness. With regard to feeling more informed, there was no difference between the two groups, but both the personalised and the control group felt equally informed.
Interested to know more?
The full report about the pilot process and findings will be made available on the website here once ready. For any questions, you can contact us here.
