Standalone Tooling for creating IBM GDM Maps programatically from meta-data

The Input to the tooling is data transformation information captured in:
1) an Excel Spreadsheet (CSV) which is discussed first this article OR
2) a json format file OR
3) an XML format file (a WebSphere ESB Business Object Map was used in the prototype)

Two and three above are recently added alternatives discussed later in the article.

Click on the image for full screen

Example CSV input

A DFDL schema is used by the tooling to parse the CSV data. You’ll see the first row is a header row containing information about the source and target objects and parsers. After that we have repeating rows of element mapping definitions.
The DFDL schema for the CSV data.
Click on the image for full screen

DFDL Schema definition

The Output is an IBM GDM .MAP file ready to be deployed.
Click on the image for full screen

IBM GDM image output

The tool uses the standalone Java DFDL Parser classes to parse a comma delimited (CSV) file. It then leverages JAXB against the published MSL.XSD schema file for the IBM GDM.

Currently, the tool supports simple single input schema, single output schema maps. It concentrates on building the element relationships. It currently supports MOVE, ASSIGN and FN:CONCAT( for two input fields to one output field).

The collateral supporting MOVE, ASSIGN and FN:CONCAT is available from GitHub:

https://github.com/DAVEXACOM/IBMGraphicalDataMapGenerator

Possible uses for this tooling

1) Automated Integration Delivery in conjunction with Patterns based development.
2) Converting transformation logic from other integration tools (from other vendors) to an IBM GDM deployable to IBM Integration Bus and potentially IBM Datapower.

Running the tool – using the CSV as input meta-data

The Java class ParseMetaSerialMSL contains both the DFDL CSV Parser code and the JaxB serialize to MSL (XML) code. (It should be noted that as far as Java development goes…. I was a pretty average COBOL programmer back in the early 90s).
The ParseMetaSerialMSL.java class is the executable code and provides demonstration on how to use the standalone DFDL Java parser and leverage JaxB Java code against JaxB Objects generated by the IIB Toolkit from the IBM GDM MSL.XSD XML schema.
ParseMetaSerialMSL.java reads the IOMapModelMetaCSV.xsd (DFDL model) and IOMapModelMeta.csv the source file containing the definitions of the mapping to be performed from the GDMGenerator project. It writes out a inmodel_outmodel.map file. The .map file and the XSDs for the input and output schemas are in the GeneratedMapTargetApp application project.

From the IIB Toolkit (Java Perspective) open the ParseMetaSerialMSL.java source and “Run as Java Application”.

ParseMetaSerialMSL class

Check the console to review the XML output and a successful run.

Switch to the Integration perspective and you will see the

Generated Map Target App project before execution

Refresh the GeneratedMapTargetApp project and open the newly created inmodel_outmodel.map

Generated Map Target App project after execution

Running the tool – using the JSON as input meta-data

The ParseJSONSerialMSL.java class is the executable code and provides demonstration on how to convert a json file to java object and leverage JaxB Java code against JaxB Objects generated by the IIB Toolkit from the IBM GDM MSL.XSD XML schema.

ParseJSONSerialMSL.java reads the IOMapModelMetaCSV.json source file containing the definitions of the mapping to be performed from the GDMGenerator project. It writes out a inmodel_outmodelfrom_JSON.map file. The .map file and the XSDs for the input and output schemas are in the GeneratedMapTargetApp application project.

From the IIB Toolkit (Java Perspective) open the ParseJSONSerialMSL.java source and “Run as Java Application”.

Check the console to review the XML output and a successful run.

Switch to the Integration perspective

Refresh the GeneratedMapTargetApp project and open the newly created inmodel_outmodel_fromJSON.map

Running the tool – using the XML (WESB BO Map) as input meta-data

The ParseBOXMLSerialMSL.java class is the executable code and provides demonstration on how to leverage JaxB to unmarshal an XML file (WESB BO Map) to Java Objects generated in the IIB Toolkit from an XSD generated from an example BO Map (not the definitive BO Map schema). It then leverages JaxB Java code against JaxB Objects generated by the IIB Toolkit from the IBM GDM MSL.XSD XML schema and marshals the Java to produce the GDM map.

ParseBOXMLSerialMSL.java reads the WESB_mesBlastBlockMessageType_To_Blastblock_Submap.map source file containing the definitions of the mapping to be performed from the GDMGenerator project. It writes out a WESB_mesBlastBlockMessageType_To_Blastblock_Submap.map file in the GeneratedMapTargetApp application project. The the XSDs for the input and output schemas are in the GeneratedMapTargetApp application project.

ParseBOXMLSerialMSL.java supports Move and CustomJava. To support CustomJava I have created a “placeholder” Java project and called GeneratedMapTargetJava and class called placeholder. This allows me to configure the IBM GDM CustomJava mapping with a target class and method. The Java code from the BO Map is copied into the Documentation field for that mapping. So this would have to be refactored by hand post conversion.

From the IIB Toolkit (Java Perspective) open the ParseBOXMLSerialMSL.java source and “Run as Java Application”.

Check the console to review the XML output and a successful run.

Switch to the Integration perspective

Refresh the GeneratedMapTargetApp project and open the newly created WESB_mesBlastBlockMessageType_To_Blastblock_Submap.map file.

Wrap Up

As mentioned, this is prototype tooling and demonstrates how to use:
1) the standalone DFDL Java parser OR
2) a Json to Java converter OR
3) a JaxB unmarshaller
and JaxB code against the published Map Specification Language schema (MSL.XSD) to create IBM GDM Maps based on some meta-data. I chose to capture my mapping meta-data in:
1) an Excel Spreadsheet OR
2) a json format file
3) an XML file (WESB BO Map)
out of convenience but you may have Business Analysts or Data Architects that use other tools to capture data transformation requirements. Meta-data from those tools could be used in a similar way. Or perhaps harvest meta-data from an existing non-IBM data mapping tool.

11 comments on"IBM Graphical Data Map (GDM) Generator Tool and WESB BO Map conversion prototype"

  1. Hi,

    Is there a similar tool that can run the GDM in the graphical mode?
    today, in order to create a mapping graphically, the only way is to install the IBM integration toolkit and create it through there.
    I’m looking for a tool that could run as a stand alone application, that its only purpose is to create and edit the map files.

    Thanks.

    • Hi, I used the Java Perspective in the IIB Toolkit. An Eclipse based or other Java development environment could have been used. The prototype can be run standalone or embedded elsewhere. It was convenient for me to use the IIB Toolkit, when developing the prototypes so I could quickly switch to the IIB perspective and open the graphical data map to ensure my code was building the map directly.
      The prototype uses DFDL Standalone java classes and JAXB, no IIB.
      Cheers

      • hey, thanks for the quick reply!
        What I meant is that we need an app that can graphically map between XSDs that does not require an IDE (as the ibm toolkit).
        We would have wanted something like “MapForce” that after graphically map, creates a file that could be a “.map” file or could be converted easily to one

        • hey, is there an answer ? @Dave_Arnold

          • Dave Arnold October 11, 2017

            Hi. the IBM GDM is actually a standalone tool but delivered as an Eclipse Plug-in. It is added to the IIB Toolkit by the the IIB product team. It is also added to the IBM zOS Connect, IBM MDM Server and some of the IBM Rational Toolset. I don’t believe you there is a separate download for where you can add it to a base Eclipse install. Cheers

  2. Dave_Arnold April 26, 2017

    I have updated the article and materials on GitHub with a prototype that uses a XML (in the form of a WESB BO Map) as input meta-data.

  3. Dave_Arnold April 05, 2017

    I have updated the Materials on Github and this article has been updated to add JSON files as an alternative to CSV files as input meta-data to generate GDM maps.

  4. Does this work for a mediation module in BPM Advanced? The mapping looks the same.

    • Dave Arnold March 27, 2017

      Hi MKuzak. The target output for the tool is the IBM Graphical Data Mapper so it should work for any tool that uses that the GDM. In the tool I hard code the mapping engine to “xquery” for IIB.

      MyMapGen.setEngine(“xquery”);

      You might need to change this depending on the target, for use with IID for Datapower this would need to be changed to “XSLT 1.0” or “XSLT 2.0”.

      Or open the map as an XML->Data Map after generation and it can be changed by hand in the GUI.

      Cheers

  5. Hello David,

    Thanks a lot for this blog and i was looking for something like this. Can you also give the code for conversion from ESQL to GDM…

    • Dave Arnold March 27, 2017

      Hi Hari. The problem with using a coded language such as ESQL (or Java) as the source as input to the IBM GDM Generator is that different programmers use different coding styles and formatting. It would be really difficult to reliably interpret ESQL code and derive the mapping relationships.
      It may be possible to determine the basic map information, i.e. the information that is in the top line of CSV file, the input and output schema types and message types from a .ESQL file but often even these are captured in the ESQL because the inputs are set on the Input Node.
      Cheers

Join The Discussion

Your email address will not be published. Required fields are marked *