Contents


Overview

Skill Level: Intermediate

Information Governance Catalog, REST, JSON, XML

InfoSphere Information Governance Catalog provides REST APIs to embed catalog queries or metadata creation into applications. With cURL you can leverage these APIs from command line (e.g., for scripting or easy testing). The recipe explains how.

Ingredients

InfoSphere Information Governance Catalog Version 11.5

InfoSphere Information Governance Catalog (IGC) is a module of IBM's InfoSphere Information Server suite. It provides comprehensive capabilities for the management of business, technical, and operational metadata. This includes an authoring workflow for business metadata, import and management of technical metadata (e.g., for DBMS, BI, ETL, and modelling tools), and a framework to define own metadata categories. One of the key features is the automatic analysis and presentation of data lineage graphs and lineage reports so that the user can easily understand the relationships between metadata objects.

IGC REST APIs

IGC REST API allows client applications to access and create content. With the REST API the catalog content can be integrated into other software tools and portals. IGC REST API enforces security roles, workflow, and viewing permissions just like the IBM InfoSphere Information Governance Catalog user interface. To test REST API calls IGC provides a REST Exporer interface: https://<igc_server>:<port>/ibm/iis/igc-rest-explorer

cURL

curl is an open source command line tool and library for transferring data with URL syntax. curl supports SSL certificates, HTTP GET, HTTP POST, HTTP PUT, and much more.

All together

curl is an appropriate utility to interact with Information Governance Catalog REST API via Command Line. It can facilitate IGC REST API scripting for interactions such as the creation of glossary terms or the upload of bundle assets and flows, or it can just be used to test REST calls before using them in applications.

Step-by-step

  1. Verification of Information Governance Catalog

    First verify that you can access Information Governance Catalog at https://<igc_server>:<port>/ibm/iis/igc/

    The version of Information Server should be 11.5 or higher. The default port is 9443 but Information Governance catalog can be configured with a different port.

    Ensure that there is at least some content in the catalog. The following example will use glossary categories and terms. The sample categories and terms look like this:

    Samples categories and terms in IGC

  2. Download / install cURL

    Verify if curl is already available as command on your client or server system. To do so, enter “curl” at the command prompt:

    > curl

    If curl is not already avaliable on your system, download curl for example from here: https://curl.haxx.se/download.html

  3. Set up cURL to use SSL and Information Server's certificate

    curl performs peer SSL certificate verification by default. This is done by using a CA certificate store that the SSL library can use to make sure the peer's server certificate is valid. Alternatively the certificate can be passed as command parameter. If you communicate with HTTPS, as it is the case with Information Governance Catalog using certificates, you can be sure that the remote server really is the one it claims to be.

    Alternatively, if the remote server uses a self-signed certificate, you can tell curl to not verify the peer. This is also an option when interacting with IGC via REST API.

    1) Invoke IGC REST API without verification

    In this case invoke curl with option --insecure, like this:

    > curl --insecure -H "Accept:application/json" 
    -u igcuid:pwd https://<igc_server>:<port>/ibm/iis/igc-rest/v1/types

    2) Invoke IGC REST API with verification

    In this case invoke curl with a certificate, like this:

    > curl --cacert certdata -H "Accept:application/json" 
    -u igcuid:pwd https://<igc_server>:<port>/ibm/iis/igc-rest/v1/types 

    To create the PEM certificate “certdata”, use the openssl command as follows:

    1. > openssl s_client -connect <igc_server>:<port> | tee logfile
    2. Edit logfile:
      only keep lines from -----BEGIN CERTIFICATE----- to -----END CERTIFICATE-----
      Remove all other lines
      Save as cert_from_logfile
    3. > openssl x509 -inform PEM -in cert_from_logfile -text -out certdata

    Then use --cacert certdata in your curl commands.

    The verbose output of the curl command with ssl handshake will document success:

    ...
    * successfully set certificate verify locations:
    * CAfile: certdata
    CApath: none
    * Server certificate:
    * subject: C=US; O=IBM; OU=Software Group; CN=<igc_server>
    * start date: Feb 26 21:57:45 2015 GMT
    * expire date: Feb 24 21:57:45 2023 GMT
    * common name: <igc_server> (matched)
    * issuer: C=US; O=IBM; OU=Software Group; CN=ishost
    * SSL certificate verify ok.
    ...
  4. Try a GET command

    The Search API is an easy starter for testing IGC REST calls with GET command. Before trying to get it going with curl, be sure the call syntax works in the IGC REST Explorer: https://<igc_server>:<port>/ibm/iis/igc-rest-explorer/

    The following sample will search for the details of a glossary category. With the parameters specified the call searches all objects of type “category”, with name exactly (%22) matching “B Subcategory”. The response is in JSON format. -v indicates verbose output of the curl command. If you had problems with setting up the CA certificate, use --insecure instead of --cacert certdata.

    > curl -v --cacert certdata -H "Accept:application/json" 
    -u igcuid:pwd "https://<igc_server>:<port>/ibm/iis/igc-rest/v1/search/?types=category&text=%22B%20Subcategory%22&search-properties=name"

    Note that the url of the call is enclosed in double quotes (“https://…”), which is necessary because of the ampersands (&) in the call syntax.

    The command will produce the following JSON output:

    {
    "items": [
    {
    "_type": "category",
    "_id": "6662c0f2.ee6a64fe.9ma5da20f.8ojoq24.qskf6f.ufc3ls6bga66kuig0ttar",
    "_context": [
    {
    "_type": "category",
    "_id": "6662c0f2.ee6a64fe.9ma5da1uu.i4onhaq.s7frld.tbukp4mgjqa0v4s6crrt0",
    "_url": "https://chrisisvm1:9443/ibm/iis/igc-rest/v1/assets/6662c0f2.ee6a64fe.9ma5da1uu.i4onhaq.s7frld.tbukp4mgjqa0v4s6crrt0",
    "_name": "A Test Category"
    }
    ],
    "_url": "https://chrisisvm1:9443/ibm/iis/igc-rest/v1/assets/6662c0f2.ee6a64fe.9ma5da20f.8ojoq24.qskf6f.ufc3ls6bga66kuig0ttar",
    "_name": "B Subcategory"
    }
    ],
    "paging": {
    "numTotal": 1,
    "pageSize": 10,
    "end": 0,
    "begin": 0
    }
    }

    Remarks:

    The above REST call searches for objects in the published glossary. If you are using the glossary workflow and want to search for objects in the development glossary, just add workflowMode=draft to the call like this:

    > curl -v --cacert certdata -H "Accept:application/json" 
    -u igcuid:pwd "https://<igc_server>:<port>/ibm/iis/igc-rest/v1/search/?types=category&text=%22B%20Subcategory%22&search-properties=name&workflowMode=draft"

    Note that the GET command is inherited from the API and does not need to be specified with -X GET. To add -X GET will not cause an error, although an informational message is displayed in the verbose output of the curl command. Same with the POST commands in the subsequent steps.

  5. POST command with JSON content

    This step demonstrates how to create metadata in the catalog using the IGC REST API. A simple starter is to create a glossary term in a pre-existing category. The example will add the term “RestTerm” to the category “B Subcategory”.

    Note that glossary objects are being created in the development glossary if the glossary workflow is enabled. This means, that the call to create a term in a category needs to reference the ID of the category in the development glossary.

    To be more flexible the API payload (JSON in this case) can be swapped out to a file, in the following example to the file term.json:

    > curl.exe -v --cacert certdata -H "Content-Type:application/json" 
    -H "Accept-Encoding:identity" --data "@term.json"
    -u igcuid:pwd https://<igc_server>:<port>/ibm/iis/igc-rest/v1/assets

    With term.json:

    {
    "_type" : "term",
    "short_description" : "Term added via REST API",
    "status" : "ACCEPTED",
    "parent_category" : "6662c0f2.ee6a64fe.9ma5da1vr.smtm5l5.pj727l.upe0269pntbo8t5cbn2gp",
    "name" : "RestTerm"
    }

    The JSON payload references the ID of the parent category “B Subcategory” within the development glossary.

    Output with return code 201 (success):

    ...
    * upload completely sent off: 208 out of 208 bytes
    < HTTP/1.1 201 Created
    ...

    The new term is immediately visible in the development glossary in IGC:

    newTerm2

  6. POST command with XML content

    The revious step called an IGC REST API with JSON payload. Other IGC REST calls require XML payload instead of JSON. An example for an IGC REST call with XML payload is to upload bundle assets within the Open IGC framework.

    Background:

    Open IGC is a framework to extend the standard IGC metadata categories (databases, tables, files, ETL processes, XML schemas, BI models, data models, and many more) with own categories. The APIs include the definition of own 'asset bundles', the upload of 'bundle assets', and the upload of 'flows' to link own assets with standard or other own assets in data lineage.

    Again, the API payload (XML in this case) can be swapped out to a file, in the following example to the file xmlForAssetPublishing.xml:

    curl.exe -v --cacert certdata -H "Content-Type:application/xml" 
    -H "Accept-Encoding:identity" --data "@xmlForAssetPublishing.xml"
    -u igcuid:pwd
    https://<igc_server>:<port>/ibm/iis/igc-rest/v1/bundles/assets

    Output with return code 200 (success):

    ...
    * We are completely uploaded and fine
    < HTTP/1.1 200 OK
    ...

    As response of this call a list of all successfully loaded assets (with ID and RID) is returned in JSON format:

    {
    "Top_1": "5b818a0c.187019e4.9ma5d9lth.imo2a32.vfuqjr.n8u7lijccd9gfkjhadlrq",
    "Map_1_1": "5b818a0c.187019e4.9ma5d9lth.ivpkk63.8u6adg.7glab71tna754sdtn8rq6",
    "Sub_1_1_1": "5b818a0c.187019e4.9ma5d9lth.jc1dk7j.8160dm.e98q73ajk3v4k0rida236",
    "Sub_1_1_2": "5b818a0c.187019e4.9ma5d9lth.jf6bjme.mjjep9.gi270e4c5rijt39ea8k2m",
    ...
    }

    The uploaded assets are visible immediately in the catalog:

    qsubs

  7. Wrapping it up

    Information Covernance Catalog REST API is a powerful interface to interact with the governance catalog and to embed catalog functionality in applications. When it is required to embed those interactions in scripts (e.g., to load Open IGC bundle assets after parsing external metadata), curl is an easy to use utility to call IGC REST APIs by command. curl can be used besides or in addition to the REST Explorer which is included in IGC to test catalog interactions. It offers additional capabilities to develop and test IGC REST calls before embedding them in applications.

    If you are interested in an end-to-end example of how using cURL to publish Open IGC metadata look at this Recipe:

    Extend your Information Governance Catalog with Q Replication Metadata
    https://developer.ibm.com/recipes/tutorials/extend-your-information-governance-catalog-with-q-replication-metadata/

2 Comments on "Interact with your governance metadata in IGC - using REST APIs with cURL"

  1. This is really helpful.. Is there anyways to add comment using API calls ? Eg – I have a IGC rule or a term, for which I just need to add a comment under the development glossary.. Possible ?

Join The Discussion