Adding logical Big SQL workers should be done in non-peak hours when there are no Big SQL applications running on the system. Big SQL maybe restarted one or more times so that the underlying memory can be shuffled correctly within each compute node during logical worker operations. An introduction to this feature is presented in Logical Big SQL Workers Boost Performance. Important: Add and reconfigure logical Big SQL workers when all compute nodes (or Big SQL Workers from Ambari) are active. If any worker nodes are down then restart them.

In ‘Add Services Wizard’->‘Customize Services’-> ‘Advanced bigsql-env’ drop down menu there are 2 parameters related to Logical Big SQL Workers. These parameters specify the total number of Logical Big SQL Workers that can be added to each compute node during the Install. Note that the default is 1 for both parameters.

Configuring Logical Big SQL Workers during install

During install the default number of Logical Big SQL Workers that can be added can be changed by increasing the ‘Logical Big SQL Workers per host’ parameter. Use the guidelines in Logical Big SQL Best Practices to determine how many logical Big SQL workers should be added on the cluster.

Configuring Logical Big SQL Workers after Install

In the ‘Configs’ tab of the BigInsights – Big SQL service, there is a parameter called ‘Logical Big SQL Worker Add/Remove Count’ as shown in the figure above. This parameter is used during the addition and removal of Logical Big SQL Workers after install. The value of this parameter can be changed during or after the Big SQL install.

Logical Big SQL Worker Operations

In the Big SQL ‘Service Actions’ menu, there are Logical Big SQL Workers operations which when executed affect ALL of the Big SQL Compute nodes. Options include ‘Add Logical Big Sql Worker’, ‘Check Logical Big Sql Worker’, ‘Balance Logical Big Sql Worker’ and ‘Remove Logical Big Sql Worker’ as shown in the Figure below:

From the ‘BigInsights- Big SQL’ Summary tab there are also links to ‘Big SQL Head’ and Big SQL Workers’ also shown in the figure above. If you click on the ‘Big SQL Head’ link the same ‘Logical Big Sql Worker’ options are displayed.

Add Logical Big SQL Worker

To add Logical Big SQL Workers across all the Big SQL compute nodes, from the Big SQL ‘Service Actions’ menu, click on the option ‘Add Logical Bigsql Worker’. Logical Big SQL Workers are added to ALL Big SQL Compute Nodes. This is an OFFLINE operation because the resources need to be reconfigured after this operation. Big SQL can be restarted one or more time during this operation. Note that prior to adding a Logical Big SQL Worker there is one already so adding a Logical Big SQL Worker for the first time will result in two Logical Big SQL Workers in total.

The maximum number of logical big sql workers that can be added to each compute node is 16. The maximum number of logical workers that can be added across the cluster is 999.

The figure below shows what the addition of the logical Big SQL worker looks like from Ambari, wait for the operation to be 100% completed. It may take a few minutes for the operation to complete as Big SQL needs to be restarted one or more times for the operation to complete successfully. Check that the logical Big SQL worker was added successfully by selecting the ‘Check Logical Big SQL Worker’ operation.

Check Logical Big SQL Worker

This option reports how many Logical Big SQL Workers are configured across the compute nodes. The figure below shows that the check operation is complete, you can get more information on the number of logical Big SQL workers configured on each compute node by clicking on the arrow beside the 100% status of the Check_Logical_BigSQL_Worker operation.

Keep clicking the arrows beside each of the tasks and you should eventually come to a screen tells you the number of logical workers on each compute node. In this example there are 4 logical Big SQL Workers across each of the compute nodes:

Ideally the number of logical Big SQL workers should be consistent across the compute nodes. If the counts are not consistent, then balance the number of logical Big SQL workers by using the Balance option. One reason why the configuration is not balanced could be that there was an error during addition of the logical Big SQL worker operation on a specific compute node.

Balance Logical Big SQL Worker

For performance reasons, the number of logical Big SQL workers should be consistent across the Big SQL compute nodes. If the counts are not consistent, then you can use the ‘Balance’ options to ensure that there is the same number of logical Big SQL workers per compute node.
The ‘Balance Logical Bigsql Worker’ option when chosen from the ‘Service Actions’ menu will determine the maximum logical Big SQL workers configured on each of the compute nodes and if any compute node has fewer logical Big SQL workers, logical Big SQL workers will be added to those compute nodes so that all compute nodes have the same number of logical workers. If logical Big SQL workers need to be added, this operation is an OFFLINE operation as a Big SQL restart is needed to reconfigure the memory on the cluster.

Remove Logical Big SQL Worker

To remove Logical Big SQL Workers across all of the compute nodes, from the Big SQL ‘Service Actions’ menu, click on the option ‘Remove Logical Bigsql Worker’. This is an OFFLINE operation as the system memory needs to be reconfigured during the remove operation.

Thanks to the following folks for contributing to this work: Metin Kalayci, Hebert Pereyra, Abhayan Sundararajan