The News Explorer has arrived. News Explorer uses the Alchemy API to automatically construct a news information network and present large volumes of news results in an understandable fashion. Now that we are able to extract semantic data from the news with advanced Watson NLP technology, we can provide new ways of visualizing and understanding news as it occurs.

The News Explorer is a cognitive web application that is available on Bluemix. News Explorer was developed only recently, but in the few weeks since the first version became operational, it has received tremendous attention in IBM communities. Now we are pleased to release the beta version to all our customers, and partners.

Click here to watch this video in HD.

Try out the application in your browser now: Launch News Explorer




The News Explorer approach for reading and discovery of news

Huge volumes of news are published every minute. No single human being could hope to keep up with the hundreds of thousand of news stories being generated daily.

With Watson we can now observe and understand the richness of information that is emerging as it’s obtained right from the source. From the freely available text of news reports, we are using natural language processing to extract Topics and essential entities such as Locations, Organizations, Companies, People, and Time. These are the key components required in order to build and navigate the gigantic network of interconnected, heterogeneous information that is implicit in the news.

For example, imagine you are interested in a specific announcement made by IBM about the News Explorer at an event in Rio de Janeiro yesterday. Which articles talk about this announcement? Is this news related to other IBM products, or other events, or companies? What news articles were written about these related entities? The News Explorer tool can be used to investigate these questions and more.

Network science is a new discipline that examines complex systems and processes that exhibit network behavior. With a network visualization representation of the news and the entities mentioned within it, we can arrive at a deeper understanding of what is being reported and how it is interconnected.

Figure 1: Decomposition of the news results about IBM along the dimensions of People, Companies, Organizations, Locations, and Time.

If a client wants to know more about IBM, for example (Fig.1), we can go way beyond providing a list of relevant news for them. We can show the top People, Companies, Organizations, Locations, and Topics occurring in the news. In the News Explorer we aggregate the relevant documents on every dimension, and show a map for locations encountered, a timeline representation of the news, and a network visualization for the remaining dimensions (Fig.2).

Figure 2: the News Explorer application viewed in full-screen mode



Getting Started with News Explorer

Click here to watch this video in HD.

In this tutorial we will learn how to work with the News Explorer, a web application powered by Bluemix, designed to gain insights from the latest news articles. The application is composed of a number of areas that work together to help you get an overview of what is going on and focus in on articles of interest.

The News Explorer is composed of a number of areas that work together to help you get an overview of what is going on and focus in on articles of interest. These areas expose relationships between entities that appear in the news: companies, organizations, people, articles, locations, concepts, and time. Relationships are expressed in data visualizations, and can be explored through a series of interactions custom-tailored for those entity types.

To make the best use of the application, we recommend using full-screen mode in Chrome or Firefox browser at a resolution of 1024px or wider.

Try out the application in your browser now: Launch News Explorer

Using the Application Areas

The Investigation Bar is where you will make queries. To start out, enter the name of an entity (e.g. “IBM”) in the first text field. Next choose the entity type from the drop-down (e.g. “company”). In the field after this, you can enter the time frame for your query. Finally hit “investigate” to make the query. Upon doing so, the Watson icon will pulse, signifying that a query is taking place. The News API is accessed, topics are identified, entities are extracted, relationships established, and seconds later the visualization areas begin to populate with data. After your are done exploring the results, you can always go back and perform a new query at the top or you can double-click on any entity from within the News Network area, whereupon that entity will become the subject of a new query.

Another option for exploration, called Connections, involves entering two different entities (e.g. Stephen Hawking and NASA), and visualizing the relationships between them. To access this feature, simply click the “connections” toggle to right of the “investigate” button. After clicking the “investigate” button, a network will be rendered showing paths of articles that connect the two specified entities. These can either consist of single articles that mention both entities explicitly, or multiple articles that refer to common entities that form a path between the two specified entities.

Finally it’s worth noting that the parameters of an exploration are automatically displayed in the URL (e.g. http://news-explorer.mybluemix.net/?query=IBM&type=company), so that you can share it or save it into a browser bookmark for future access. Remember, though, that the news is constantly changing, so returning to a saved URL will not return the same results that were seen earlier.

The Details area is the first area to populate with data. By default it will display all article titles returned by the query, rank ordered by their strength. Scroll through the article titles until you find one of interest to you. Clicking on the title will provide you with more information, such as the news category (if available), date, article clipping (if available), and finally any entities that show up in that article. These entities are also clickable. Clicking one will pull up more information at the top of the Details area. Another result that you may notice when clicking an article title or entity is that other areas in the application will change and shift accordingly, as appropriate.

The News Network at the center of the aplication is one visualization area that will respond to clicking article titles from Details. It does this by automatically zooming and panning to that article where it occurs in the network and selecting it. From there you can pan around by dragging the mouse or zoom in and out of the area manually via mouse-wheel, two-finger scroll or the zoom buttons built into the area, to get a bigger picture. Upon doing so, you will see a broader network or entities, namely more articles, people, organizations and companies. The network is built around the articles and the entities which they refer to. For example, a company entity might appear in two different articles, so its entity has a link or line connected to each of those articles. In any case, you may click on an entity to highlight all the links in its local network. You can then mouse-over each connected article to quickly come to know what they represent.

The Locations area is at the top-right of the application, and lists all locations found in the articles, and also attempts to display any that it can onto the world map. For a location to be on the map, it has to have a recognized country name or be a city with known geo coordinates (cities are represented by dots). In either case, a location’s name or map element can be clicked to pull up more information on it in Details, as well as highlight it on the News Network and along the Timeline (for the articles in which that location occurs) with a green outline. Throughout the application, green color always signifies location.

The Topics area at the bottom-right of application includes a word cloud that list all the main themes from articles within the query. Topics are returned by Watson from the articles, based on what it thinks articles are about in a broader sense.. The Topic cloud is also clickable, and upon doing so, you will notice that entities in the News Network and articles in the Timeline containing the selected concept are outlined in Magenta and the network visualization will zoom in on the region containing articles related to that topic. An additional feature of the Topics area is the ability to see related concepts within certain selections. These will show up in violet and can be clicked as well for further filtering. When you’re finished using Topics, you can always click on the empty white space in the Topics area to clear the topic selection entirely.

Last, but not least, is the Timeline area, which organizes articles temporally along the bottom of the application. Each column is a window in time, and scanning left to right, you will notice that that frequency at which query results were found varies over time. Thus far we’ve only discussed how the timeline responds to interactions from other areas, but the timeline is also interactive. To use it, mouse over any of the colorized cells to reveal a title and date of an article. Note that the darker teal that the cell is, the more entities are linked to it—this will help you target articles that are more hooked into the network. Note that if there are more than 10 articles in a given time window, a dark gray cell will be rendered on top, which can be moused over to reveal the total in that time window. You can also click on a cell to select any article, highlighting it in black. You will notice that Details and the News Network will automatically travel to that article in their respective areas. Furthermore, if the article includes recognized locations, you will see them highlighted in dark green in the Locations area.

Conclusion

You should now have a foundational understanding of how to use the News Explorer. Feel free to make as many queries as you like. Each one will return a very different result. Remember to come back and re-query too, as news will change over time, thus exposing more insights. We look forward to incorporating more exciting features in the future, as they become available. Should you have any further questions about the features of the News Explorer, feel free to contact us.

Team

Manager: Mauro Martino Lead Developer: Steve Ross Designer / Developer: Tim Stutts

Updated 2016: Half a year after the launch of News Explorer, we published a second blog post announcing exciting updates.

The News Explorer is built by the Cognitive Visualization Lab on the IBM Watson Developer Cloud.

44 comments on"Presenting the News Explorer"

  1. Allow the sections to be minimized and pulled out of the browser window to be used across monitors. Looking for a larger visualization section.
    thank you very much for this

    • Tim Stutts July 21, 2015

      Hi Steve. You’re welcome. This feature request is definitely one that’s on our radar, in addition to having control of where the various visualization areas appear in the layout. Thanks for your feedback.

  2. Do you have any plans to open source News Explorer? I would be very interested to see how this was done.

  3. A wonderful job. Thanks.

  4. Could you re-sort the visualisation data – so it was sorted by group and looked like fig.1 in your guide?

    • Tim Stutts July 27, 2015

      Hi James, Figure 1 is only something we use to help explain the existence of different entities conceptually. Putting all the entity types into one graph would create something far too dense to properly use. Remember that in order to show relationships between entities, we need to form connections and affect their position (i.e. distance to fellow entities). So the affect would never re-sort them the way that Figure 1 depicts, unless we ignore the connections, which wouldn’t offer us much value. To get around this, we sort the entities into separate graphs, which we believe to be more applicable to that entity type.

  5. Sharad Khandelwal July 27, 2015

    Is there a plan to expose this as an api to power other apps ?

  6. Fernando Balbachan July 31, 2015

    Hi Tim,

    Great job, involving NERC, NLU, and other NLP tasks!
    I would like to show you my Spanish NLU engine, called SentiLecto

    http://dev.natural.do/sentilecto

    You will find that the project integrates syntax, semantics, fact mining, NERC, anaphora resolution, modality treatment and other capabilities beyond aspect-based sentiment analysis.

    SentiLecto is being used to automatically generate this news blog
    http://entretenimientobit.com
    with more than 100 high quality news posted on a daily basis, rewriting and enriching content and, more interestingly, merging news covering the same facts.

    I think IBM News Explorer and SentiLecto’s current show case could be similar in essence.

    Congrats again!
    Fernando Balbachan

  7. Any estimated ETA for the news explorer code?

    • Hi Arturo. We do not not have an ETA at this time, but it will be available in the near future.

  8. It would be good to have the api suppport where we can filter results only by blog sources

  9. Charles Tchuinkwa November 05, 2015

    So when you guys release the code, we one we able to use to build own product such as Google Newsstand? Also, because you don’t have a clear estimate, can one use the current News API to build such product? I intend to add this to complement my project.

    • Hi Charles, when the code is released, you will be able to see how we leverage Watson’s Alchemy News API and d3 to create the experience that we’ve shown here. Google Newsstand is very different experience from our app, so I can’t make any claims there. However, you can begin to use Alchemy News API right away: http://www.alchemyapi.com/products/alchemydata-news

  10. Marcus Gross November 18, 2015

    The launch explorer button does not work on an ipad! Is it possible to show watson news explorer on an ipad? How?

    • Hey Marcus, though it is possible to use some of the functionality on an iPad, as all of the libraries do run on happen to run in Safari iOS, we are not planning to support iPad at this time. It would involve a re-design to accommodate touch gestures, smaller form factor, etc, resulting in a very different interface. It’s a great idea though!

  11. I want to see what stories I get back based on a keyword search. I obtained an API Key. I have run a query and generated a JSON feed. Is there a way to have that feed display in an RSS Reader like Feedly?

    • Robert, it is certainly possible, but it is not a view that our News Explorer or Alchemy News API offers as default at this time. This is a situation where you might want to work with a front-end developer to help put together such an interface.

  12. Brett Forbes December 07, 2015

    Hi Tim,

    Really nice work. Congrats to your team!
    Have you considered using a multi-scale visualisation approach on the graph? As you know most graph visualisation approaches reveal no semantic structure (i.e. shape-based meaning) when viewed from a distance, which is due to a weakness in the property graph modelling approach (i.e. modelling is too discretised).
    But I note that you already have defined a series of objects for each node type, and thereby may be able to achieve a better visualisation approach by laying the graph out so that a different z-value is assigned for each node type, to form a series of layers. This will create less visual clutter, since most of the links will go between layers, but would require a complete rework of the layout algorithms.
    I would be most interested on your thoughts on this, and have some other multi-scale visualisation ideas if you are interested.

    • Brett, here’s a reply from researcher Cody Dunne: “Thank you for the suggestion. We haven’t seriously considered moving away from a 2-D visualization.

      The layered approach you are suggesting could be accomplished in 2.5-D or 3-D. The resulting visualization would definitely be eye-catching, but would require substantial user interaction to make sense of. The default view would likely be equivalent to the 2-D visualization we have now, as picking any default angle aside from looking down the z-axis would make the spatial layout more misleading and introduce occlusion.

      Instead, we are looking at more advanced 2-D layout visualization techniques that expose the inherent structure better than D3 does.”

      • Brett Forbes December 08, 2015

        Hi Cody, thanks for your explanation, nice one. You raise some interesting points about occlusion, and I am most interested in your 2D advanced layouts, as you would be aware that force balance-type layouts produce no occlusion, but create a visual structure independent of your data structure (i.e. object types). So the problem then becomes how to arrange the nodes so the visual structure, maps the data structure, so one can effectively surf the visualisation and choose points of interest (i.e. a generic problem for all property graphs). I must say though, that the way you have linked the parts of the screen to provide multi-scale visualisations, including various indexes (i.e. timeline, word cloud, geographic etc) is really well thought out, so well done here. You guys are really stretching the boundaries of high-dimensional visualisation in an impressive manner.
        I note that on your overview video, the presentation on the connections sub-menu shows something very similar to my proposal, except that you layer them horizontally rather than vertically. My space of interest lies in visualising very high dimensional, multi-scale engineering/physics design optimisation problems, and the thought behind my proposal is that a series of layers can add up to higher-level meta-concepts, such as the flow of a process (i.e. how a thing is built), or causality (i.e. this + this produces that) etc. There are some rather interesting implications for these meta-concepts in the news space. The other thing I like is that you could group similar objects in 2D on each layer, creating visual structure at multiple scales. Of course, as you point out this requires a new approach to zooming, or surfing, and I haven’t had time to pull this off yet, so in summary my suggestions represented more of an hypothesis, rather than a well-defined approach.
        In essence, I am tremendously impressed with your achievements, and await with great interest your future developments. Thanks once again for your comments

  13. Name *Mike Borgman January 06, 2016

    Tim,

    Is there going to be an opportunity to host News Explorer on a site that’s not an IBM site or a white label application?

    Thank You,

    Mike Borgman

  14. Vincent Malarme March 21, 2016

    Great work ! Can you share any information about which other languages are (will be) supported than English ?
    Alternatively, which Watson component needs other language support than English for this to be supported. thanks

  15. David G Terrell April 19, 2016

    I recommend feeding academic papers to the News Explorer software/system. The same entities (persons, organizations, concepts, documents) apply and the result would be marketable. I would love to see this idea applied to individual papers exploited for the authors and their institutions, the concepts mentioned, and the referenced “first degree of separation” papers in the footnotes/endnotes. Also, if I could move extracted data to IBM i2 Analyst’s Notebook for further link and Social Network analysis, I’d be most happy.

  16. Hello. The IBM Watson News Explorer works in Spanish?

    • Hello! We get this question quite frequently. At present the Watson News Explorer works in English only. We can’t support other languages until the AlchemyData News API does. At that point it would become possible to support versions in other languages.

  17. Hi! I was testing this spectacular resource, but I found it limited yet. If the abrangence of search can be all internet, why when I do a search about one organization from Brazil, for example, the results show only English articles? Or maybe I am something wrong? So, how I do to get the articles in any language that the organization is mentioned? Thanks for your reply.

    • At this point, the Watson News Explorer is available for English language articles only. When the AlchemyData News Service expands to support other languages, the News Explorer can be extended to support other languages as well.

  18. It would be useful to see the API calls that it generates so that we can use to build our own AlchmeyAPI calls.

    Cheers
    John

    • John,

      A typical AlchemyData news query made by the News Explorer might look like this:

      https://access.alchemyapi.com/calls/data/GetNews?return=enriched.url.cleanedTitle,enriched.url.url,enriched.url.author,enriched.url.publicationDate,enriched.url.text,enriched.url.entities,enriched.url.concepts,enriched.url.taxonomy&start=now-7d&end=now&q.enriched.url.enrichedTitle.entities.entity=|text=IBM,type=company|&count=100&outputMode=json&apikey=[YOUR API KEY HERE]

      This is not an inexpensive query! In fact each query typically costs over 1000 transactions, but the data we get back is invaluable.

      Note that in order to get more targeted news articles, the search is done on the enrichedTitle rather than on the article bodies. If we don’t get enough results, we automatically reissue the query with the looser criteria of q.enriched.url.entities.entity=|text= instead of &q.enriched.url.enrichedTitle.entities.entity=|text= to find more articles based on a search of the article bodies. We also are willing to extend the time horizon of the query and/or remove the type restriction on the search term, since we found that people sometimes misclassify or fail to update the type information when submitting new queries.

      Once we get our results back, we have to do a lot of processing to get the data into an appropriate format for our visualizations. Our internal representation is optimized for the network visualization, so we create node objects for every article, and then add additional nodes, each with the appropriate type and associated information, for each person, company, organization, location, and concept mentioned. Here it becomes necessary to aggregate all the mentions of the same entity in different articles, which can be challenging because of inconsistent naming, etc. We get a great deal of help from the alchemy API results, which often provide a “disambiguated” description, which can be used to aggregate entities with different names. If two entities are disambiguated to the same dpbedia URL, for example, we consider them identical, and record both names as being associated with that entity as well, to help aggregation in cases where the entity is not successfully disambiguated. It isn’t perfect, but it works quite well. Recording the dbpedia reference for each disambiguated entity also allows us to do dbpedia lookups to provide additional information like a photo and summary when it is selected.

  19. Is there a way to include boolean logic? For example “not like *Trump*”

    • Dan,

      There is some limited support for boolean logic in News Explorer queries. You can do up to one level of or’s over one level of and’s. The and’s have higher precedence. So you can do queries like “IBM and Cognitive Computing” or like “Cognitive Computing or Artificial Intelligence” or like “IBM and Cognitive Computing or NASA and SpaceX and Elon Musk” Any entity type provided is ignored for boolean queries. If there is a single Or in the query, you’ll find that the “river of news” display splits the results into two separate streams, top and bottom.

  20. Are there any plans to allow the user to search by date range in the news explorer and/or to be able to search archives, say going back a year or so? For instance, if I wanted to search the term “IBM” for a solid year of news to see what the heat map data and sentiment looked like, would that be possible?

    • Pat, That would be an interesting feature. Unfortunately, due to the sheer volume of news being processed, the underlying Alchemy news service only maintains a backlog of 2 months of news data. Depending on the rate at which the news for the entity that we are querying is coming in, we provide a different time window. So in some cases you may get a collection of articles spanning hours or weeks. The service responds with the most recent articles in the time period requested, so in order to do a longer-term query for an entity that has a high news rate we would have to make multiple queries, effectively sampling the news from different time intervals. That’s possible in principle, but would involve many more requests to the service, which would really slow down the process, and we would still be limited to the last two months, which is all the data that is available.

Join The Discussion

Your email address will not be published. Required fields are marked *