CPN pilots: what we’ve learned so far, and how you can join the next round of user testing

The project members of the Content Personalisation Network (CPN) have worked tremendously to develop a new approach to personalise the daily news offer, allowing both large and small media companies to better target their content to media consumers. The CPN project’s ambition is to offer you as a media consumer more relevant news, in the right format, at the right time, and in a suitable context.

Today, we proudly present the fruits of our hard work: the CPN news personalisation application. We invite you to an open pilot test to discover and experience the CPN app. This test will be the third and final pilot of the application. The test will start on 3 February and end on 9 March. You can freely explore and test the application within this time period. The CPN app is available in English, Greek and Dutch.

Background

The CPN project is developing a news recommender system that is iteratively tested and validated in operational real-life environments throughout the project in three different pilot countries. These are Belgium (coordinated by VRT), Germany (coordinated by Deutsche Welle) and Cyprus (coordinated by Dias). Three main pilot tests are organised at different stages of the project, each adding a larger number of users as the developed proof of concept becomes more mature. We have currently finished the first and second pilot. Their main takeaways are presented below.

What did we do and learn during the first and second pilot?

In 2018, almost 100 end-users tested the first prototype of the CPN news recommender platform for a period of 10 days. Specifically, the participating end-users tested the web interface of the recommender. In each pilot country, the recommender system contained news content from the local media partner (VRT, Deutsche Welle and DIAS).

The first CPN prototype consisted of three news sections, presented in separate tabs:

• Most popular - the most-read content among the users of the particular news outlet

• Latest news - all articles in chronological order, with the latest on top

• Personalised - the most relevant content for the user

The ‘personalised’ tab was the most read section in all three pilot countries. While news personalisation itself was evaluated as positive, there was a big fear of missing out (FOMO) when receiving personalised news articles: the participants were afraid that through personalisation they might only get news content based on their interests, which could cause them to miss other relevant or important news. The CPN recommender was not available as a mobile application during the first pilot, which was a negative point among the testers. On the other hand, the participants generally thought the web interface was straightforward and easy to use, although some thought that more important articles could be made more prominent.

We tested the second prototype which was an app in 2019. We evaluated whether users liked the CPN personalisation algorithm by asking them how informed they felt after using our prototype. VRT, Dias and Deutsche Welle, the media partners in the project, invited users to download the (Android) app onto their mobile phones and follow the news through CPN for a test period of four weeks. For example, the project team built the web app VRT MyNWS, resembling the general news website of VRT NWS.

News articles in the new mobile app could be found under three different tabs:

• My news: personalised articles

• Headlines: articles selected by the news department

• Just in: most recently published articles

We monitored and compared how users experienced both a personalised and non-personalised offer throughout the whole test. The users were split into two halves: one group started out with just a random selection of articles, while the other received real recommendations. We switched the groups weekly and checked whether users could tell the difference between the two versions and which one they liked better.

People did appreciate the recommended results over the random results, felt more informed and gave us positive feedback on the application - but overall, statistically the differences were too slim to clearly say the CPN fully convinced testers. That’s why it’s important to increase our number of testers in the third and final pilot!

What’s next?

We will address three main issues in the third pilot:

1. We need a larger number of test users and offer people the news in a familiar environment. We will make it obvious that the news is coming from people’s trusted news sources, VRT, Dias and DW.

2. We will improve the CPN app, so it will be more attractive for people to use the app.

3. We will apply several rounds of quality checks to ensure we run the news recommender system on a good level. We will also evaluate which extra features could make the recommendations even better and more fitting for users to include them into the third CPN prototype. For example, approaches for dealing with breaking news and to mitigate the possible effects of the ‘filter bubble’.

We have created a mobile news application that creates a personalized news offer based on what you read in the past. Are you curious to see what your personalized news offer would look like? Test our news app, let us know what you think of it, and help us to improve the application! Pilot 3 will be fully open, so anyone interested in CPN can participate in the last pilot phase.

What do we expect of you?

The pilot test starts on 29 January 2020 and will continue until 28 February. During this time you can choose when you start and end testing the app. You can install the application on your mobile phone and check the news whenever you want. During the test period, we might send you from time to time a very short online questionnaire with a couple of questions on the application. When you finish testing the app, we will send you a final online questionnaire in which you can then evaluate our app. Your feedback is very welcome and will help us improve our app!

Thanks to the feedback from our testers, we will be able to improve and update the CPN app and its recommendation software. As we also welcome media organisations to use the CPN software to personalise their own content, the feedback from our readers is crucial in building a better personalised news experience.

Who are we looking for?

Everybody who sometimes reads news online! Our news application is an Android application, so you need an Android smartphone to participate in our test.

Do you want to discover and test the platform? Click on the link below to read more about user testing and download the CPN app!

Questions about the pilot? Contact us here!

Thank you for your interest!

The CPN team.

(Photo by Jon Tyson on Stock Snap)

Discover the final version of the CPN technology bricks!

Our latest report details the final iteration of the technical infrastructure that provides the features and services of the CPN platform.

The third and final iteration of the “technology bricks” set-up of the CPN platform is now ready, and the details about it are available in our latest report. The technology bricks consist of the components and APIs included in the CPN platform, and have been designed and developed in an iterative manner, gradually adding new features and services that satisfy the user and technical requirements.

The technology bricks developed by the CPN project are classified into three categories: Content, Users and Mapping. Read on to learn more about the different technology bricks.

Content technology bricks

Fine Grained Entity Recognition Module: The goal of the fine-grained entity recognition module is information extraction on news articles. Within the CPN project, the extracted facts (fine-grained named entities, as well as relations among them) are available as structured features for enhancing the content-based part of the recommendation engine. In addition, they can support a better navigation through the data or be used for visualization purposes.

Topic Extractor: This module extracts topics from articles in order to have a concise representation of the content of the article. It performs a language detection in order to apply the most suitable Natural Language Processing techniques to extract terminological candidates (topics). Topics are filtered and finally ranked according to a TF/IDF score, and the article, enriched with these topics, is saved into its final storage (Apache SOLR) to be retrieved later during the recommendation phase. This component is not exposed through the API gateway, but it helps the recommender module answer the question: What topics is the user interested in?

Recommender AB-Testing: The AB-Testing interacts with two modules of the CPN platform: the Recommender module and the API Gateway that is needed to publicly expose this internal API to publishers. It allows every publisher to create recommenders and user groups. It is a “configuration module”, and recommenders and groups can be created and modified in any moment by publisher/administrators of the platform. This module allows news publishers to experiment with different recommendation approaches, collect feedback and formulate insights.

User technology bricks

User modelling: The purpose of this module is to create, maintain and continuously update features associated to users and news items, thus building a profile of the users of the CPN Reader’s App. It also lays down the basis for the recommendation module to work. The module is able to extract metadata from three different source languages, English, Dutch and Greek.

Reader’s app: This is essentially the CPN mobile application which can be used for entering the news environment.

In the main layout, the “Your News” stream includes the personalized news information provided by the Recommender. The “Headlines” stream includes the most important news, characterized as such by the specific source media. The “Popular” stream includes the most read articles from among the media source’s available list of articles.

Within the registration layout, the user enters among others, the preferred media source (non-configurable) and the location of interests. The user can also connect the CPN ID to their Twitter account. This information, together with the locations of interest are sent via CPN API to the Recommender. Moreover, the user has the control of their permissions (location, preferences, time usage). The permissions can be updated at any time, and the user is informed about any updates by means of Personal Data Receipts Brick. For the registration/login process, users can also use their Facebook or Google accounts.

The third version of this module also includes a prototype of a smart speaker application! In Article Details, the user can say:

• “read” or “read article” in order to have the title and the content of the article read by the device and its text-to-speech engine.

• “interesting” or “mark interesting” to mark the article interesting. After that, the article can be found in the corresponding list of interesting articles.

• “irrelevant” or “mark irrelevant” to mark the article irrelevant. After that the article can be found in the corresponding list of irrelevant articles.

• At the Article Lists section, the user can say: “top your news” or “top headlines” or “top most read” to get the first five articles of the corresponding stream.

Personal Data Receipt: This module serves as a confirmation to the user that their data is being handled correctly and according to their permissions. It sends them an email receipt (the Personal Data Receipt) whenever they make changes to the permissions they have granted to the system. The module uses blockchain technology to manage transactions on personal user data, and serves to guarantee the rights of the users, as prescribed in the GDPR. (Read more about this feature here.)

Mapping technology bricks

Distribution framework: Current licensing and ownership models in journalism require lengthy negotiations to access, redistribute, and remix content. In the age of the 24h news cycle this can make it difficult to source quality content from freelancers in a timely and secure manner, while also ensuring provenance.

The Distribution Framework aims to simplify the contractual negotiations between creators and editors by creating a pool of licensed content collaboratively managed by multiple trusted journalistic organizations without a single overarching authority. This is achieved by using Distributed Ledger Technology, in particular Hyperledger Fabric, to manage the article licenses.

The Framework sits between all participants in the system, preventing disagreement and simplifying the process of sourcing and distributing news content. In the future this could be extended with pricing and payments to create a trustless distributed content marketplace.

Producer’s App: The Producer’s Dashboard UI, providing analytics on the data collected, and allowing for an easy integration into the producer’s workflow.

The third version of this prototype also allows to set an article as “breaking news”. It also provides contract templates to allow freelancers to easily work together and with editors, to define and track the scope of individual contributions and expected revenues.

Recommender: A core module that computes the most suitable news recommendations for CPN users. It analyses the users’ profiles and collected news to find the most “interesting” news items to be proposed by the app.

This module is built with a hybrid approach that uses variable proportions of content based and collaborative filtering techniques for learning from explicit and implicit feedback given by the users themselves: clicks, ratings, sharing, etc. The system is customizable for including content-delivery strategies’ optimization: multichannel and date/time optimization (predicting the probability of interests at a given time on a given channel) and includes mechanisms for fostering “serendipitous” discoveries.

Pilot 3

The technology bricks, with all their exciting new features, will be tested in a pilot environment starting soon. Follow our website and Twitter account for the latest developments, and stay tuned for the results!

Distribution Framework: licensing for the 21st century

We presented our licensing solution, which will provide freelance journalists and media companies with new opportunities to distribute and sell their work, at the recent IPCT event.

The International Press Telecommunications Council (IPTC), a London-based consortium of the world’s major media organisations, is the global standards body of the news media. It is composed of more than 50 companies and organisations from the industry, including global players such as AP, AFP, DPA, BBC, Getty Images, PA, Reuters and the New York Times.

The mission of IPTC is to simplify the distribution of information by building technical standards to improve management and exchange of information between content providers, consumers and intermediaries. Committed to open standards, the organisation makes its standards available for free to its members and the wider community.

CPN at IPTC

As we aim to disseminate and talk about our project and its achievements to relevant people and communities, we happily agreed to take part in the IPTC Autumn Meeting that took place in Ljubljana, Slovenia on 14–16 October. After providing a general description of the current status of the CPN project, we highlighted the elements of our initiative that are the most relevant to the IPTC’s membership.

First, we discussed how personalisation is already changing the news industry, and how we are tackling this challenging matter with the CPN project. Personalisation is an increasingly important issue for news publishers, and we think that facing this problem while prioritising privacy is the right way.

The main part of our presentation discussed how our initiative will help democratise the way freelancers and media organisations make news content together in the future. Our distribution framework will allow for easier distribution, utilisation and remixing of news articles and, in the future, pictures and videos.

Distribution Framework

By accessing our Distribution Framework web app, freelance journalists will be able to license any piece of content they own and set the terms and conditions for anyone else in the world to use their material. The licensing system will produce a hash of the content and write it with the attached license(s) on a private distributed ledger, with nodes currently owned by the media organisations participating in CPN.

This system will then produce a universally shareable link with open access. Anyone with the link will see the title and an excerpt of the article and will be able to read the full license and T&Cs and, if they agree, accept those terms. This is also recorded on the ledger, making the acceptance of the terms tamper-proof.

The way the system is designed will make access to articles and content easier, shortening the access negotiation time to a few minutes, and reducing the risk of potential legal battles over content ownership and contract obligations compliancy.

Why is this a part of CPN

We built the Distribution Framework on top of the CPN Platform to enable easier remixing of news pieces, removing most of the legal and negotiation hassle from the mix. With this system freelancers will be able to resell their content to a much wider audience in much less time, and media organisations will be able to share their content amongst themselves without having to renegotiate terms and conditions each time.

What we are looking for from IPTC

As a standards organisation, IPTC is crucial for the future of our project. We are looking forward to working with them on integrating their standards in our JSON and on building a standard together, with the option of open sourcing our code so we can increase its adoption through major media organisations.

Second Evaluation round - what did we do and what did we learn?

The second year of the CPN project has come to an end and we just handed in our second evaluation report. As the name says, it was all about getting feedback on our latest version of the CPN prototype. But we also undertook a close inspection of some of the extra features we’ve been thinking about to improve recommendations. Here is what we learned:

Can you tell the difference?

Of course we wanted to know from our test users whether they liked the CPN personalisation algorithm - by asking them how informed they felt, after using our prototype. But how do you get a proper answer for something so abstract? We tried it with a dual approach, by looking at the numbers as well through qualitative surveys.

So VRT, DIAS and DW, the media partners in the project, reached out to their audiences to form test groups (one per language offer). The users were invited to download the (Android) app onto their mobile phones and follow the news through CPN for a test period of four weeks.

The trick however was to have users experience both a personalised and a non-personalised offer throughout the whole test. So the users were split into two halves: one group (in each language) started out with just a random selection of articles, while the other received the real recommendations. The usage of the app was monitored and compared, and users were given a short feedback questionnaire at the end of every week. We switched the groups weekly - and checked whether users could tell the difference between the two versions and which one they liked better.

What did we learn?

Getting the right amount of users for and keeping them motivated throughout such a test is important - and somewhat difficult. We learned this the hard way. While we did have great responses from users in the first year on our surveys and also through individual tests e.g. with VRT’s experiment to integrate the CPN recommender in its own app], engagement in this evaluation round was much lower.

How often did users click on articles during the evaluation?

The core group that did the test from beginning through end gave us some indications as to the strengths and weaknesses of the CPN app. But the actual user click evaluation, while having a positive tendency, ended a bit inconclusive, statistically. This means people did appreciate the recommended results over the random results, felt more informed and gave us positive feedback on the application - but overall, the numbers were so low that statistically the differences were too slim to clearly say the CPN app fully convinced the testers.

How informed did users feel? More on the personalised side.

Of course this isn’t what we wanted to hear. A clear “yes” or even a clear “no” would have been much more instructive. But we are also not completely lost. The results from the three language groups all had a tendency towards acceptance of the application and some individual comments clearly pointed out where our work convinced (“results did fit my profile”) and where we clearly still need to invest time and work (“I found it confusing to always get new results”).

How do we move on?

After analysing the evaluation process in more detail, we identified three main issues that we have to address in the third pilot next year:

1. Clearly, we need a larger number of test-users - and one approach is to offer people the news in a known environment. People trust news from VRT, DIAS and DW as they know the brands. CPN however is completely new to them. So we need to make it more obvious where the news are coming from, by integrating the CPN recommendation in native apps and/or make the sources in the CPN app more obvious to the users.

2. Testing an application remotely over four weeks and keeping people’s motivation high is a challenge when you’re not a well known app company. So we are thinking about improving our approach (and our app), so it will be more attractive for people to help us.

3. The quality of the recommendation system is a key element in testing it’s acceptance. So we have started several rounds of quality checks to ensure we run the system on a good level. Furthermore we are evaluating which extra features could make the recommendations even better and more fitting for users to include them into the third version of the prototype.

The whole consortium is now working on those issues in preparation for the next year, both to run a third, more successful evaluation round with more tangible results. But also in order to have the system ready to test it with other media companies that are interested in using such a system as CPN. At the beginning of 2020, we will bring it all together and put it to the test - hopefully with a clearer picture afterwards.

About those extra features

Recommendation is based on prioritizing interests and matching them with the available content. So far, so easy. But this concept gets more complicated as you go along. How do you deal with breaking news for example? Are they subject to your interests, hence should only be shown when the topic is high enough up your personal profile list? Or should they always overwrite the algorithm, as probably many newsrooms and enemies of the filter bubble think? We tried to find an approach here through expert interviews and experiments, but have yet to put it to the (user) test.

Possible approach to integrate breaking news in a personalised news offer

Another big issue with recommendations is said ‘filter bubble’ as defined by Eli Pariser in 2011 - which is an even more complex topic. We worked our way through scientific papers and articles and started drafting several possible approaches to mitigate possible effects like this. The most promising approach we see as of now for CPN is to empower the users in getting a better understanding of what they read and what they don’t read - to eventually judge themselves, whether they are moving themselves into a dangerous direction. But this as well needs further testing and we are aiming to have a better understanding in the third pilot.

Interested in more?

Has this article sparked your interest in our evaluation? If you want to read more, feel free to access the full report, available on the website here. For any questions regarding the results or an interest in testing the application yourself, please go here to contact us.

Holding platforms accountable: fines alone won’t do. What does this mean for CPN and other EU platforms?

Major online platforms are facing intensifying calls for transparency on how they govern their users’ data. Smaller EU-based platforms such as CPN may have an advantage but need to make sure their privacy measures are up to the task.

On 24 July 2019, Facebook was fined 5 billion dollars by the U.S. Federal Trade Commission (FTC) for violation of their 2012 decree on obtaining consent from users. What is more, the FTC also required Facebook to change its governance structure: the company was ordered to establish an independent privacy committee and compliance officers. The aim of this enforcement action is for Facebook’s data practices to become more transparent and accountable.

Facilitating or shaping?

The accountability of platforms has grown as a theme in public debates about online services. Over the past decades, as platforms became ubiquitous, they have been presenting themselves as mere impartial facilitators - neutral intermediaries providing users, businesses, advertisers, and other parties with the means to interact. We have grown as accommodated to their environments as fish are to water. The composition and architecture of platforms are invisible to us, and yet, this architecture steers us, the way a door steers us to enter a room from a certain angle. Choices in software and interface design shape our interactions and influence our responses, and we need to be able to understand how this is done, where this may lead us and who benefits from this.

Governing platforms

Accountability is the main principle behind the E.U.’s General Data Protection Regulation (GDPR), but also outside circles concerned with the protection of privacy, calls are growing for platforms to explain and justify their governance of data. Discussions are ongoing on regulations that aim to curtail the distribution of disinformation via social media sites; platforms’ responsibilities to protect intellectual property rights are being reviewed; and competition authorities the world over are looking into the monopolistic effects of imbalances in data collections and data-handling capacities that are making such tech giants as Alphabet, Amazon, and Microsoft invincible in multiple markets.

We should distinguish ‘governance of platforms’ from ‘governance by platforms’, says communication researcher Tarleton Gillespie (2017). While governance of platforms refers to such regulatory interventions as mentioned above, it is the governance by platforms we should investigate more closely. While platforms generally don’t create content, they host, store, organize and distribute content of others, and they make important decisions about that content. For example, they decide on the distribution of risks and rewards for users, by ordering and presenting content along algorithmically optimized lines and by playing regulatory arbitrage. Choices in the architecture of platforms currently turn mostly on corporate strategies and profit motives, even if some platforms express a sense of public obligation.

Translating accountability into practice

While there has been criticism of the ‘leniency’ of the FTC’s enforcement action, it should be commended for paying attention to Facebook’s governance as well: the order to install a privacy committee is aimed at improving governance by the platform, introducing independent oversight and more transparency on the choice architecture. What is as yet unclear in this case is what ‘independent’ and ‘transparent’ mean in practice; who selects the committee members and compliance officers, will reviews and evaluations be published, and what will they include? Nevertheless, this type of enforcement shows promise: we need to start thinking about practical translations of accountability and transparency in order to introduce values beyond profit motives.

In the European Union, the GDPR already requires independent Data Protection Officers at board level to advise on and safeguard the protection of personal data. The European Data Protection Supervisor has called repeatedly for closer collaboration with other supervisory authorities to address issues that surpass personal data protection. The FTC enforcement action may be a baby step, but it is quite possibly leading in the right way.

What does this mean for CPN and smaller EU-based platforms?

In a sense, smaller EU-based media organizations already have a unique selling point in being less adept at re-using personal data. Moreover, as they are local, they know their communities and truly care. The following recommendations may help boost the privacy-friendliness of their data governance:

1. Ensure that transparency and privacy become part of the platform DNA.

2. Smaller platforms will fall under the same regulations, so find ways to conduct automated IP checks and content moderation.

3. Cut loose from providing the big platforms with even more personal data.

4. Invest in data governance by appointing a chief data officer, mapping data flows and assigning data governance responsibilities throughout organizational units.

By Jaco van der Bank and Ine van Zeeland

(Photo by Devon Rogers on Unsplash)