CPN pilots: what we’ve learned so far, and how you can join the next round of user testing
/
The project members of the Content Personalisation Network (CPN) have worked tremendously to develop a new approach to personalise the daily news offer, allowing both large and small media companies to better target their content to media consumers. The CPN project’s ambition is to offer you as a media consumer more relevant news, in the right format, at the right time, and in a suitable context.
Today, we proudly present the fruits of our hard work: the CPN news personalisation application. We invite you to an open pilot test to discover and experience the CPN app. This test will be the third and final pilot of the application. The test will start on 3 February and end on 9 March. You can freely explore and test the application within this time period. The CPN app is available in English, Greek and Dutch.
Background
The CPN project is developing a news recommender system that is iteratively tested and validated in operational real-life environments throughout the project in three different pilot countries. These are Belgium (coordinated by VRT), Germany (coordinated by Deutsche Welle) and Cyprus (coordinated by Dias). Three main pilot tests are organised at different stages of the project, each adding a larger number of users as the developed proof of concept becomes more mature. We have currently finished the first and second pilot. Their main takeaways are presented below.
What did we do and learn during the first and second pilot?
In 2018, almost 100 end-users tested the first prototype of the CPN news recommender platform for a period of 10 days. Specifically, the participating end-users tested the web interface of the recommender. In each pilot country, the recommender system contained news content from the local media partner (VRT, Deutsche Welle and DIAS).
The first CPN prototype consisted of three news sections, presented in separate tabs:
Most popular - the most-read content among the users of the particular news outlet
Latest news - all articles in chronological order, with the latest on top
Personalised - the most relevant content for the user
The ‘personalised’ tab was the most read section in all three pilot countries. While news personalisation itself was evaluated as positive, there was a big fear of missing out (FOMO) when receiving personalised news articles: the participants were afraid that through personalisation they might only get news content based on their interests, which could cause them to miss other relevant or important news. The CPN recommender was not available as a mobile application during the first pilot, which was a negative point among the testers. On the other hand, the participants generally thought the web interface was straightforward and easy to use, although some thought that more important articles could be made more prominent.
We tested the second prototype which was an app in 2019. We evaluated whether users liked the CPN personalisation algorithm by asking them how informed they felt after using our prototype. VRT, Dias and Deutsche Welle, the media partners in the project, invited users to download the (Android) app onto their mobile phones and follow the news through CPN for a test period of four weeks. For example, the project team built the web app VRT MyNWS, resembling the general news website of VRT NWS.
News articles in the new mobile app could be found under three different tabs:
My news: personalised articles
Headlines: articles selected by the news department
Just in: most recently published articles
We monitored and compared how users experienced both a personalised and non-personalised offer throughout the whole test. The users were split into two halves: one group started out with just a random selection of articles, while the other received real recommendations. We switched the groups weekly and checked whether users could tell the difference between the two versions and which one they liked better.
People did appreciate the recommended results over the random results, felt more informed and gave us positive feedback on the application - but overall, statistically the differences were too slim to clearly say the CPN fully convinced testers. That’s why it’s important to increase our number of testers in the third and final pilot!
What’s next?
We will address three main issues in the third pilot:
We need a larger number of test users and offer people the news in a familiar environment. We will make it obvious that the news is coming from people’s trusted news sources, VRT, Dias and DW.
We will improve the CPN app, so it will be more attractive for people to use the app.
We will apply several rounds of quality checks to ensure we run the news recommender system on a good level. We will also evaluate which extra features could make the recommendations even better and more fitting for users to include them into the third CPN prototype. For example, approaches for dealing with breaking news and to mitigate the possible effects of the ‘filter bubble’.
We have created a mobile news application that creates a personalized news offer based on what you read in the past. Are you curious to see what your personalized news offer would look like? Test our news app, let us know what you think of it, and help us to improve the application! Pilot 3 will be fully open, so anyone interested in CPN can participate in the last pilot phase.
What do we expect of you?
The pilot test starts on 29 January 2020 and will continue until 28 February. During this time you can choose when you start and end testing the app. You can install the application on your mobile phone and check the news whenever you want. During the test period, we might send you from time to time a very short online questionnaire with a couple of questions on the application. When you finish testing the app, we will send you a final online questionnaire in which you can then evaluate our app. Your feedback is very welcome and will help us improve our app!
Thanks to the feedback from our testers, we will be able to improve and update the CPN app and its recommendation software. As we also welcome media organisations to use the CPN software to personalise their own content, the feedback from our readers is crucial in building a better personalised news experience.
Who are we looking for?
Everybody who sometimes reads news online! Our news application is an Android application, so you need an Android smartphone to participate in our test.
Do you want to discover and test the platform? Click on the link below to read more about user testing and download the CPN app!
Questions about the pilot? Contact us here!
Thank you for your interest!
The CPN team.
(Photo by Jon Tyson on Stock Snap)
