Currently with dynamic mapping in IBM API Connect Analytics, if a user has payload logging enabled or they pass headers with their API request, we receive that information from the Gateway. (Up to 10% of Bluemix orgs actively have payload logging.) When that data is entered into Elasticsearch, it is dynamically mapped to all fields in order to be searchable. Then in Kibana, when a user creates a visualization to illustrate their data, the fields can be used to create the visualization on their payload data.
Our customers have generally given positive feedback on this feature. However, there are a couple problems users face when everything is dynamically mapped. With dynamic mapping, users aren’t able to specify data types. A major problem occurs with users in the public cloud who have a huge number of fields (i.e. thousands and thousands) on their payload. All of these fields are stored in memory on the Elasticsearch master nodes, thus putting a lot of load on the master nodes and resulting in frequent performance issues. A second problem occurs with Elasticsearch, which will choose the best fit for dynamic mapping. For example, Elasticsearch will analyze your text-based fields. If it finds a string with spaces in it, Elasticsearch will divide it into separate strings.
The Analytics team has worked on the ability for API Connect users to easily set desired metrics they want to capture and then send into storage without needing to store the entire API call log. The user can now define API header and payload data to be transmitted. We created a way not to have dynamic mapping enabled without sacrificing the ability for users to create visualizations off their payload and specify how that payload data is stored in Elasticsearch.
Now you can have greater control of the dynamic mapping capability to create better custom visualizations. You can also count on stronger performance of analytics visualizations when utilizing the dynamic mapping feature.
Defining the Values in Analytics
Let’s see how this change looks in the Bluemix UI. We’ve added a new section titled Analytics in the Design tab, which you can navigate to using the side navigation bar on the left. By clicking the Source tab, you can see it under ‘x-ibm-configuration.’ You will also see the new Analytics section. Back in the Design tab, users can add a new field using the blue Add Field + button at the top, which would map the field. You can add a field in two ways:
a) You can add a basic field where you can define the four sections that appear below. (Note: You can edit the the Swagger file as well, but we recommend entering the values manually in the interface.)
● NAME: Define the name. Let’s say we want to add a new location field: ‘MyLocation.’
● LOCATED IN: Define it to be located in either the header of payload. In this example, we’ll leave it as ‘Header.’
● ELASTICSEARCH TYPE: Define the Elasticsearch type. Since this is a location, we’ll use ‘geo_point.’
● FROM: Define it as a header request or response. We’ll use ‘response.’
b) A second way to add a field is to use the Parameters section below. In the screenshot, you will see a few that have already been defined. Click Add Field + > api_parameter.
● The ‘NAME’ and ‘LOCATED IN’ sections are disabled so that they cannot be changed since they are directly linked to the parameters in the Parameters section.
● Define ‘ELASTICSEARCH TYPE’ as ‘integer.’
● Define ‘FROM’ as ‘response.’
Then if you navigate to the Source tab, you can see a difference between the two kinds of field mappings from a schema perspective. One will store the name and location and the other will store a reference to the parameter. The backend will be able to use this parameter to get the same information.
This creates content in the Swagger.yaml and is saved.
Creating the Visualization
After a basic API is created with different fields and you create the visualization in the Swagger, you can create the visualization in Analytics. This is available for Bluemix and on-prem.
1. Navigate to the API Connect Sandbox and click the Design tab.
2. Choose the visualization type. In this example, we’ve created a heatmap.
3. Save the visualization using the disk icon next to the search bar at the top.
Try it out Today!
Try out this new feature today by signing up or logging into Bluemix.