What we learned from VRT MyNWS: a pilot on news personalisation at the Flemish public broadcaster

/
image (3).png

At the end of March, the innovation and news department of the Flemish public broadcaster VRT experimented for four weeks with VRT MyNWS. This web application was built and tested as part of CPN and aimed to bring the VRT readers articles based on their interests. After the launch of VRT MyNWS, 949 testers subscribed to give feedback on their experience of personalisation, the usage of the web app and the display of articles. Read on to find out what we learned.

VRT MyNWS: a web app customised to readers

In order to measure whether people are better informed when they receive recommendations or not, the project team built the web app VRT MyNWS, resembling to the general news website of VRT NWS. News articles in the new web app could be found under three different tabs:

  • My news: personalised articles

  • Headlines: articles selected by the news department

  • Just in: most recently published articles

During one month, testers could give feedback via a button in the app, participate in surveys and were kept up to date via a weekly mailing. Overall, the team received over 200 emails with suggestions for improving the app. These mostly concerned the user interface and the recommendation algorithm. On a daily basis, a team of developers processed the feedback.

“It was extremely informative to gather input from end users, as well as the news department, and link those two into practice. Only by working together and experimenting on the production floor, we are able to innovate.”- Ilke Lemmelijn, CPN project coordinator.

Personalisation based on artificial intelligence and popularity

The testing with VRT MyNWS took place in two phases. In the first phase of the experiment, the tab ‘My news’ was loaded with articles that were selected according to three algorithms: collaborative filtering, content-based recommending and a random selection of recent articles. By collaborative filtering, readers received news articles that other readers with similar interests also found interesting. By content-based recommending, articles were selected and offered on the basis of the content and metadata of the articles. Lastly, by including a random selection of articles, the project team aimed to avoid readers receiving too many similar articles and ending up in a so-called filter bubble. The testers were divided into two groups that each received their own composition of the three algorithms.

In a second phase of the testing period, the project team chose to recommend articles based on their popularity during a certain timeframe. The team wanted to find out what the ideal timeframe would be. Testers were once again divided into three groups, each receiving popular articles from the last hour, the last 12 hours or the last two days. The results showed that testers appreciated it more when they received articles that were popular during the last hour, rather than for a longer period.

Feedback on the user interface and the algorithm

Based on the click behaviour, testers seemed to appreciate the simple algorithm from the second phase more than the complex system from the first phase. As such, the tab ‘My news’ was used 68% as opposed to other tabs in the second phase, while it was used 38% in the first phase. During the second phase, testers also used the 'My news' tab more when they were shown popular articles from the last hour old. Overall, testers said that they felt better informed in the second phase.

During the first phase, the user feedback mostly concerned the user interface, such as the width of articles, and the recommendation algorithm, such as irrelevant news. In both phases, the publication date of the articles appeared to be an important factor in personalisation. Apart from the small amount of relevant, older articles, the majority of testers preferred to read articles that are no more than two days old.

“Unique for recommendations in the news area is that the age of the articles is crucial in maintaining the interest of readers.” - Joris Mattheijssens, Data scientist at CPN and VRT Innovation

The next step for news personalisation at VRT

The insights and results of the VRT MyNWS test are further processed within CPN. The CPN team also calls out to other European news organisations to personalise their news stories with the software from CPN. In a new pilot, the Cypriot news organisation Dias and Deutsche Welle are also experimenting with the personalisation of their news articles. The insights and next steps will be published here.

Interested in piloting the CPN software? Contact us here!