Our new developer pattern Create a cognitive news search app makes use of the Watson Discovery News data set, a public data set of news articles that are analyzed by Watson on daily basis. Using the Watson Discovery Service, we can query the News data set and extract and aggregate articles to get a list of the highest trending topics in the news. By applying a time filter we can get the most recently viewed topics, and for each of those topics we can find the most relevant news article and convert it into an RSS feed so that it can be consumed by anyone, in any way, since RSS is a standard XML format for exchanging data.

In the pattern, we explored how to query the Discovery service using the Node.js SDK. Using that same SDK, we can query the Discovery Service to aggregate the topics in all of the data set’s news articles to get the top 20 topics using the following query:

{
  "aggregations": [
    "term(enrichedTitle.entities.text,count:20)"
  ]
}

If we also want to include one news article for each topic we can include top_hits(1) in the aggregation:

{
  "aggregations": [
    "term(enrichedTitle.entities.text,count:20).top_hits(1)"
  ]
}


Finally, to make the Watson Discovery Service return the most recent topics, you can apply a filter to get topics for the past 24 hours. Here, we are using a library called moment to subtract 24 hours from the current time:

{
  "aggregations": [
    "term(enrichedTitle.entities.text,count:20).top_hits(1)"
  ],
  "filter": `blekko.chrondate>${moment().subtract(24,'h').unix()}`
}

Now that we have the data, we can generate the RSS news feed using a Node module called rss. It’s simple to use and supports a lot of options for constructing a comprehensive RSS feed with custom elements. For our news feed, we’ll stick to the standard tags.

To construct the feed, we simply need to initialize it:

const feed = new RSS({
  title: `Trending Topics in News`,
  description: 'RSS feed for Trending Topics found using Watson Discovery Service'
});

We then iterate over the topics, extract the news story, and create an item in the feed:

topics.forEach(item => {
  const story = topicStory(item);
  const categories = story.enrichedTitle.taxonomy.reduce((result, categories) =>
    result.concat(categories.label.split('/').slice(1)), []);
  feed.item({
    guid: story.id,
    title: item.key,
    url: story.url,
    description: story.enrichedTitle.text,
    author: story.author,
    categories
  });
});

When the news stories for each of the topics have been added to the feed, we get the XML string of the feed by calling feed.xml(). We then pass the XML string as our response in our Node.js server:

res.set('Content-Type', 'text/xml').send(feed.xml());

And that’s it! We now have an RSS feed that we can subscribe to and get push notifications from using an RSS news reader. I hope you’ll check out Create a cognitive news search app and try the steps I’ve outlined. Leave me a comment below and let me know how it goes.

Join The Discussion

Your email address will not be published. Required fields are marked *