Controlling AIX kernel extensions through user application

Introduction

Sometimes, you may come across situations where you need to use a kernel extension’s system call in your user application. For this, the kernel extension needs to be started before starting the user application. But it becomes complex and tricky when the user application is already running.

A deadlock scenario is created when the user application must be running to start the kernel extension, and without the kernel extension running, the user applications cannot be active as it is using the kernel extension’s system call. To overcome this peculiar scenario, dynamic linking of kernel extension needs to be done.

This tutorial is helpful in writing, building, dynamically loading, and running a kernel extension. Along with general kernel extension’s loading and unloading behaviors, this tutorial focuses on how to control the kernel extension through user-level applications and control user applications through kernel extension. Thus, this tutorial provides a two-way mechanism of controlling and communicating between kernel extensions and user applications. It also covers remote topics such as establishing an Ethernet-level socket communication with user-level applications residing on another server, detecting and controlling operating system shutdown and reboot operations, and managing kernel-level logs.

This tutorial is useful for IBM® AIX® kernel and application developers as it has sample programs which are generic and can be modified to meet specific requirements.

Prerequisites to write a kernel extension

  • Knowledge of C programming language with basic AIX operating system.
  • XLC compiler to compile the sample programs.

Overview

This tutorial briefs the use of kernel extension, how to control a kernel extension from user level, and how to load or unload a kernel extension through user-level programming.

The following figure shows the major components that are part of user application and kernel extension interactions. Al interactions between the user application and the kernel extension happen through wrapper libraries.

Figure 1. User-application and kernel extension interaction through wrapper alt

In Figure 1:

  • Application: Refers to any application running at the user-level and uses APIs of the wrapper library to control kernel extensions.
  • Wrapper library: Contains the system calls to load, unload, initialize, terminate, and query kernel extensions and dynamically load the symbols of the extension.
  • Kernel extension: Contains the definition of the system calls that can be used by the applications.

About kernel extension

An AIX kernel provides an interface to dynamically extend its functions using kernel extensions. Kernel extensions are routines that are added to extend the functions of an AIX kernel. Kernel extensions can customize the kernel using kernel services to load, unload, initialize, and terminate dynamically loaded kernel routines. A process running in user mode can customize the kernel using the sysconfig subroutine.

Sysconfig kernel service

The sysconfig subroutine is used to customize the AIX operating system by providing services such as loading, unloading, and configuring kernel extensions.

Syntax of sysconfig routine

#include <sys/types.h>
#include <sys/sysconfig.h>
int sysconfig (Cmd, Parmp, Parmlen)
int Cmd; /* command for loading, initializing and query the kernel extension */
void *Parmp; /* need to pass the address of the proper structure with the command */
int Parmlen; /* length of the information */

/* Structure for loading a kernel extension */
struct cfg_load
{
    caddr_t path;       /* ptr to object module pathname    */
    caddr_t libpath;    /* ptr to a substitute libpath      */
    mid_t   kmid;        /* kernel module id (returned)      */
};

/* Structure for calling a kernel extension's module entry point */
struct cfg_kmod
{
    mid_t   kmid;/* module ID of module to call      */
    int cmd; /* command parameter for module     */
    caddr_t mdiptr;/* pointer to module dependent info */
    int mdilen;/* length of module dependent info  */
};

structure for querying kernel extension

struct cfg_load queryCmd;
bzero (&queryCmd, sizeof (queryCmd));
queryCmd.path = kernExtPath; /* path of kernel extension binary */
queryCmd.libpath = NULL;
sysconfig (SYS_QUERYLOAD, &queryCmd, sizeof (queryCmd));

structure for loading and initializing kernel extension

To load:

struct cfg_load     loadCmd;
struct cfg_kmod     configCmd;
loadCmd.path    = kernExtPath;
loadCmd.libpath = NULL;
sysconfig (SYS_KLOAD|SYS_64BIT, &loadCmd, sizeof(loadCmd ) );

To initialize:

configCmd.kmid              = kmid;
configCmd.cmd               = CFG_INIT;
configCmd.mdiptr            = NULL;
configCmd.mdilen            = 0;
sysconfig (SYS_CFGKMOD|SYS_64BIT, &configCmd, sizeof(configCmd));

Structure for terminating and unloading kernel extension

To terminate:

struct cfg_load     loadCmd;
struct cfg_kmod     configCmd;
configCmd.kmid      = kmid;
configCmd.cmd       = CFG_TERM;
configCmd.mdiptr    = NULL;
configCmd.mdilen    = 0;
sysconfig (SYS_CFGKMOD|SYS_64BIT, &configCmd, sizeof(configCmd));

To unload:

bzero (&loadCmd, sizeof(loadCmd));
loadCmd.kmid = kmid;
rc = sysconfig (SYS_KULOAD|SYS_64BIT, &loadCmd, sizeof (loadCmd));

Basic kernel services

This section lists some of the frequently used kernel APIs that are used in almost all kernel extensions.

  • pincode(): Routine used to pin code and data of the loaded object module
  • unpincode(): Routine used to unpin code and data of the loaded object module
  • creatp(): Routine used to create kernel process and sets the state to IDLE
  • initp(): Routine used to change the state of the process from IDLE to READY
  • thread_create(): Routine used to create a kernel-only thread and set the state to IDLE
  • kthread_start(): Routine used to start the kernel-only thread and change the state to running
  • setpinit(): Routine used to set init process as parent for the current kernel process

Exporting kernel system calls

A kernel extension provides additional kernel services and system calls by specifying an export file. An export file contains a list of symbols to be added to the kernel namespace. In an export file, symbols are listed one per line. These system calls are available to both 32- and 64-bit processes.

System calls are identified by using one of the syscall32, syscall64, or syscall3264 keywords after the symbol name. The kernel provides a set of base kernel services to be used by kernel extensions. Kernel extensions can export new kernel services, which can then be used by subsequently loaded kernel extensions. Base kernel services, which are described in the services documentation, are made available to a kernel extension by specifying the /usr/lib/kernex.imp import file while linking the extension.

Basic kernel extension program

Now, we will write some basic kernel extension program with a system call defined and we will write an application program which uses the system call.

kernext.c

#include <strings.h>
#include <sys/errno.h>
#include <sys/types.h>
#include <sys/syslog.h>
#include <sys/device.h>
#include <sys/pin.h>
#include <sys/thread.h>
#include <sys/signal.h>
#include <unistd.h>

int ke_entry_point(int const cmd, struct uio* const uio_buf)
{
    int rc = 0;
    bsdlog(LOG_DEBUG|LOG_KERN,"Enter ke_entry_point:: command = 0x%d
           \n",cmd);
    if( cmd == CFG_INIT )
    {
        bsdlog(LOG_DEBUG|LOG_KERN,"Initializing ke_entry_point KernExt \n");
        rc = pincode( (int (*)())ke_entry_point );
        if( rc )
        {
           bsdlog(LOG_DEBUG|LOG_KERN,"pincode failed with rc=%d\n",rc);
           return rc;
        }
    }
    else if( cmd == CFG_TERM )
    {
        bsdlog(LOG_DEBUG|LOG_KERN,"Terminating ke_entry_point KernExt \n");
        rc = unpincode(ke_entry_point);
        if( rc )
        {
bsdlog(LOG_DEBUG|LOG_KERN,"Failed to unpin the code. rc = %d\n", rc );
        }
    }
    return 0;
}

int testke_syscall( int arg )
{
 bsdlog(LOG_DEBUG|LOG_KERN,"In testke_syscall: recevied arg %d.....\n",arg);
 return 0;
}

Before building the kernel extension, create an export file (.exp) to store the system call information.

Export file

The content of the export file must include all the system calls that we are going to use in our kernel extension. Content shown below is the export file for the sample program kernext.c

# cat test_ke.exp
#!/unix
testke_syscall syscall3264

Using the bsdlog feature to write to a log file from kernel extension

Kernel extensions make use of bsdlog (a feature provided by the AIX operating system to write kernel level logs) for logging mechanism. The following procedure explains how to create a syslog entry and rotation of the kernel extension log file.

  1. Create the testke.log file.
    # touch /tmp/testke.log
    testke.log will be the log file if user wants to add logs from the kernel extension.
  2. Add the following line in /etc/syslog.config
    kern.debug /tmp/testke.log rotate size 100K files 4
    This line makes sure that testke.log rotates once it reaches 100 KB and allows maximum of 4 log file rotations.
  3. Refresh the syslog service to include new changes.
    refresh -s syslogd
    Run the above command so that the changes added in the syslog.conf file gets reflected.

A kernel extension has no main() routine. Instead, it has an entry point, which is invoked by the controller application when the CFG_INIT command is issued using SYS_CFGKMOD with sysconfig(). By default, the name of the entry point needs to be __start( ). If you specify a different name, as in the above case, you need to specify that while building your kernel extension.

Controller application

We need an intermediate controller application to load, initialize, terminate and unload the kernel extension separately. Here is a sample controller application program.

kctrl.c

#include <stdio.h>
#include <errno.h>
#include <string.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <sys/sysconfig.h>
#include <sys/device.h>

int main(int argc, char *argv[])
{
struct cfg_kmod opt_kmod;
struct cfg_load opt_load, query_load;
struct stat statbuf;
char szKernExt[256], c;
/* Check if the user has appropriate privileges */
if (getuid() != 0)
{
fprintf(stderr, " Not SuperUser.\n");
exit(EACCES);
}
/* Check arguments */
if (argc != 2)
{
printf ("Usage: %s <kernel_extension>\n", argv[0]);
exit(EINVAL);
}
strcpy(szKernExt,argv[1]);
/* Check existence of file */
if (stat(szKernExt,&statbuf) != 0)
{
perror("stat");
exit(errno);
}
/* Fill up the cfg_load structure */
opt_load.path = szKernExt; /* file name */
opt_load.libpath = NULL; /* no library */
opt_load.kmid = 0;
/* Perform various operations on the kernel extension */
while(1)
{
fprintf (stderr, "\n Enter choice, (l)oad, (u)nload, (i)nit, (t)erm, (q)uery or (e)nd\n");
while((c = getchar()) < 'a' && c > 'z') ;
/* discard garbage */
switch(c)
{
case 'l': /* load a kernel extension */
/* load kernel extension request */
if( opt_load.kmid != 0 )
{
printf("Extension already loaded, kmid is %u\n", opt_load.kmid);
break;
}
if (sysconfig(SYS_KLOAD|SYS_64BIT,&opt_load,sizeof(struct cfg_load)))
perror("sysconfig(SYS_KLOAD)"); /* print error message */
else /* otherwise */
printf("Extension Successfully loaded, kmid is %u\n", opt_load.kmid);
break;

case 'i': /* Initialize a KernExt */
/* Initialize the kernel extension */
if( opt_load.kmid == 0 )
{
printf("kernel Extension not loaded\n");
break;
}
opt_kmod.kmid = opt_load.kmid;
opt_kmod.cmd = CFG_INIT;
opt_kmod.mdiptr = NULL;
opt_kmod.mdilen = 0;
if (sysconfig(SYS_CFGKMOD|SYS_64BIT,&opt_kmod,sizeof(opt_kmod)))
perror("sysconfig(SYS_CFGKMOD)"); /* print error message */
else
printf(" Extension Initialized \n");
break;

case 'u': /* Unload kernel extension */
/* Check if KernExt is loaded */
if (opt_load.kmid == 0)
printf("kernel Extension not loaded\n");
else
{
if (sysconfig(SYS_KULOAD|SYS_64BIT,&opt_load,sizeof(opt_load)))
perror("sysconfig(SYS_KULOAD)");
else
{
opt_load.kmid = 0;
fprintf(stderr, "KernExt Successfully Unloaded \n");
}
}
break;

case 't': /* Terminate the kernel extension */
if (opt_load.kmid == 0)
fprintf(stderr, "Extension not loaded\n");
else
{
opt_kmod.kmid = opt_load.kmid;
opt_kmod.cmd = CFG_TERM; /* Terminate the kernel extension */
opt_kmod.mdiptr = NULL;
opt_kmod.mdilen = 0;

if (sysconfig(SYS_CFGKMOD|SYS_64BIT,&opt_kmod,sizeof(opt_kmod)))
perror("sysconfig(SYS_CFGKMOD)"); /* print error */
else
fprintf(stderr, "KernExtension Terminated \n");
}
break;

case 'q': /* query kernel extension existence */
query_load.path = opt_load.path;
query_load.libpath = opt_load.libpath ;
query_load.kmid = 0;

if (sysconfig(SYS_QUERYLOAD|SYS_64BIT,&query_load,sizeof(query_load)))
perror("sysconfig(SYS_QUERYLOAD)");User application program

else
{
if(query_load.kmid > 0)
fprintf(stderr, "Kernel Extension is loaded, with kmid %u \n", query_load.kmid);
else
fprintf(stderr, "Kernel Extension is not loaded \n");
}
break; /* done */
case 'e':
exit(0);
default:
fprintf(stderr, "Incorrect option \n");
break;
}
getchar();
}
return 0;
}

User application program

The following sample user application program can interact with the kernel extension and use the system call provided by the kernel extension.

Userapp.c

#include <stdio.h>
int main()
{
    int rc = 0;
    printf("invoking testke syscall...\n");

    if( (rc = testke_syscall(25) < 0))
    {
         perror("testke_syscall error");
         exit(1);
    }

    printf("exiting.....\n");
    return 0;
}

Makefile

Below is the content of the makefile to generate the binaries for sample programs kernext.c, userapp.c and kctrl.c

all: kernext Userapp
kernext: kernext.o
        cc -q64 -o kernext.o -c -DEBUG -D_KERNEL -DIBMR2 kernext.c -e     ke_entry_point
        ld -b64 -o kernext kernext.o -e ke_entry_point -bI:/usr/lib/kernex.exp -bE:testke.exp -bloadmap:kext.map -lsys -lcsys
Userapp: Userapp.o
      cc -q64 -o Userapp.o -c Userapp.c
cc -q64 -o Userapp.X -s Userapp.o -bI:testke.exp                      mv Userapp.X Userapp
cc -q64 -o kctrl -c kctrl.c
.PHONY: clean
clean:
      rm -rf *.o *.map Userapp kernext

After running the makefile, the executables kernext, kctrl, and Userapp are created.

Interaction between kernel extension, controller, and user application

When you run the Userapp application, it will try to call the system call testke_syscall and pass an argument to that. Before running Userapp, you need to load and initialize the kernel extension with the help of the controller application and then run Userapp. After Userapp execution is completed, you need to terminate and unload the kernel extension.

 (0) root @ saltc01: /home/goutham-LPM-KE/AdvKE
# ./kctrl kernext load

(0) root @ saltc01: /home/goutham-LPM-KE/AdvKE
# genkex | grep kernext
f1000000c069b000     2000 kernext
f1000000c04d4000     7000 /usr/lpp/htx/etc/kernext/nx
f1000000c04d1000     2000 /usr/lpp/htx/etc/kernext/sctKernExt64_64

(0) root @ saltc01: /home/goutham-LPM-KE/AdvKE
# ./kctrl kernext init

(0) root @ saltc01: /home/goutham-LPM-KE/AdvKE
# ./Userapp
invoking testke syscall...
exiting.....

(0) root @ saltc01: /home/goutham-LPM-KE/AdvKE
# ./kctrl kernext term
(0) root @ saltc01: /home/goutham-LPM-KE/AdvKE
# ./kctrl kernext unload

(0) root @ saltc01: /home/goutham-LPM-KE/AdvKE
# genkex | grep kernext
f1000000c04d4000     7000 /usr/lpp/htx/etc/kernext/nx
f1000000c04d1000     2000 /usr/lpp/htx/etc/kernext/sctKernExt64_64

Before running the main application, it is required to load and initialize the kernel extension and after completing the execution of the application, we need to terminate and unload the kernel extension. From the above scenario, we understood that we need an additional controller application for controlling the kernel extension to make use of the system calls defined in the kernel extension.

Interaction between kernel extension and user application without a controller application

One of the demerits of the above scenario is that an external controller application is required to manage kernel extension. This makes the controller application an important prerequisite for any user application interacting with kernel extension. This may not always be beneficial.

The best approach would be to remove the external controller application from the picture. Then user application must take the responsibility of managing the kernel extension. But there is a catch here. For user application to use the system calls provided by kernel extension, it must be loaded before the user application starts running. This means, for the user application to be active, the kernel extension should be up and running. But as said before, the user application by itself must load the kernel extension. That is, the user application must be active first and then load the kernel extension. This becomes a deadlock as you cannot start without the other.

To resolve this issue, we need to inform the user application that whatever kernel extension system calls it is using, those will be available later and this allows the user application to be active first before starting the kernel extension. Once the user application becomes active, it can load the kernel extension and use the system calls.

Perform the following steps to create static libraries for the kernel extension:

  • Include all the system calls provided by the kernel extension in a static library (for example, libkernext.a).
  • Write another static library to control kernel extension and dynamically load symbols of system calls (for example, libwrapper.a).
  • Link the static library libwrapper.a to the user application. This makes sure that user application knows about the kernel extension system calls and it can be active irrespective of whether actual kernel extension is loaded or not.

This way, user application can be loaded first, and it can manage and control all behaviors of kernel extension and use its system calls.

The following sample programs explains this scenario in detail.

The sample kernel extension program (testke.c) handles the code for initialization and termination of the kernel extension, definition of the system call, creation of kernel process and kernel thread, and detection of shutdown notification.

The sample wrapper library program (libwrapper.c) handles the code to load, unload, initialize, and terminate the kernel extension using sysconfig kernel service. It also handles the dynamic loading of the symbols of the system calls and provides these wrapper functions available for the applications running at user level.

The sample application program (Userapp.c) uses the APIs provided by the wrapper library to control the kernel extension.

Sample kernel extension program (testke.c)

#include <strings.h>
#include <sys/errno.h>
#include <sys/types.h>
#include <sys/syslog.h>
#include <sys/device.h>
#include <sys/pin.h>
#include <sys/errno.h>
#include <sys/thread.h>
#include <sys/signal.h>
#include <unistd.h>
#include <sys/proc.h>
#include <sys/time.h>
#include <sys/lock_def.h>
#include <sys/lock_alloc.h>
#include <sys/reboot.h>
#include <sys/ndd_var.h>
#include <sys/socket.h>
#include<sys/sleep.h>

int ke_entry_point(int const cmd, struct uio* const uio_buf );
void InitKE( struct uio* );
void TermKE(void);
void ke_main_func(int, void*, int );
void kern_thread_func(void*);
long notifyShutdown(shutdown_notify_t *sn);

#define TRUE  1
#define FALSE 0

enum {
    STATE_UNINITIALIZED = 0,
    STATE_INITIALIZED,
    STATE_INIT_FAILED,
    STATE_TERMINATION_REQUESTED,
    STATE_TERMINATED,
};
typedef struct ke_main_cb
{
    pid_t               kern_process_pid;
    boolean_t           isInitialized;
    Simple_lock         m_lock;
    tid_t               m_cond;
    int                 flags;
    boolean_t           isShutdownNotified;
}mcb;

mcb  ke_cb;
mcb  *mcbP = &ke_cb;

shutdown_notify_t m_notify;
shutdown_notify_t *sn = &m_notify;

int ke_entry_point(int const cmd, struct uio* const uio_buf)
{
    int rc=0;

    bsdlog(LOG_DEBUG|LOG_KERN,"Enter ke_entry_point:: command = 0x%d \n",cmd);
    if( cmd == CFG_INIT )
    {
        bsdlog(LOG_DEBUG|LOG_KERN,"Initializing ke_entry_point KernExt \n");
        InitKE( uio_buf );
    }
    else if( cmd == CFG_TERM )
    {
        bsdlog(LOG_DEBUG|LOG_KERN,"Terminating ke_entry_point KernExt \n");
        TermKE();
    }

    return rc;
}

void InitKE( struct uio *io )
{
    int rc=0,isFailure=FALSE;
    pid_t   pid;

    if ( mcbP->isInitialized )
    {
        bsdlog(LOG_DEBUG|LOG_KERN,"Extension is already intialized.\n" );
        return;
    }

    /*Pin this kernel extension.*/
    rc = pincode( (int (*)())ke_entry_point );
    if( rc )
    {
        bsdlog(LOG_DEBUG|LOG_KERN,"pincode failed with rc=%d\n",rc);
        return;
    }

    do
    {

    lock_alloc( &mcbP->m_lock,LOCK_ALLOC_PIN,0,-1);
    simple_lock_init( &mcbP->m_lock );
    mcbP->m_cond = EVENT_NULL;
    simple_lock( &mcbP->m_lock );
    mcbP->flags = STATE_UNINITIALIZED;
    simple_unlock( &mcbP->m_lock );
    bsdlog(LOG_DEBUG|LOG_KERN,"Creating a kernel process..\n");

 /*create a kernel process also make the kernel process runnable, allowing           it to start asap.*/
    pid = creatp();
    if ( pid < 0 )
    {
        bsdlog(LOG_DEBUG|LOG_KERN,"Failed to create kernel process.\n" );
        isFailure = TRUE;
        break;
    }

    rc = initp( pid, (int (*)())ke_main_func, NULL, 0, "ke_main");
    if ( rc )
    {
        bsdlog(LOG_DEBUG|LOG_KERN,"initp() failed with rc=%d\n", rc );
        isFailure = TRUE;
        break;
    }

        mcbP->kern_process_pid = pid;
    }
    while(0);

/* if isFailure is set to TRUE, it indicates failure in kernel process        initialization we need to unpin the kernel extension*/
    if( isFailure )
    {
        lock_free(&mcbP->m_lock);
        mcbP->m_cond = EVENT_NULL;
        rc = unpincode((int (*)())ke_entry_point );
        if( rc )
        {
        bsdlog(LOG_DEBUG|LOG_KERN,"Failed to unpin code. rc=%d\n",rc);
        }
        return;
    }

bsdlog(LOG_DEBUG|LOG_KERN,"Waiting for the main process:%d to           initialize...\n",pid);
    simple_lock( &mcbP->m_lock );
    /* will wait for initialization to complete by kernel process*/
    while( mcbP->flags == STATE_UNINITIALIZED )
    e_sleep_thread( &mcbP->m_cond,&mcbP->m_lock,LOCK_SIMPLE);

    /*If initialization fails then clean up and return*/
    if( mcbP->flags == STATE_INIT_FAILED )
    {
        simple_unlock( &mcbP->m_lock);
        lock_free(&mcbP->m_lock);
        mcbP->m_cond = EVENT_NULL;
        rc = unpincode((int (*)())ke_entry_point );
        if( rc )
        {
            bsdlog(LOG_DEBUG|LOG_KERN,"Failed to unpin code. rc=%d\n",rc);
        }
        return;
    }

    mcbP->isInitialized = TRUE;
    simple_unlock( &mcbP->m_lock );

bsdlog(LOG_DEBUG|LOG_KERN,"Successfully created and initialized main process. pid=%d\n", pid );

    return;
}

void TermKE()
{
    int rc;

/* if initialization flag is not set, then there is nothing to proceed for the termination*/
    if ( !mcbP->isInitialized )
    {
bsdlog(LOG_DEBUG|LOG_KERN,"Invalid operation: Kernel Extension is not intialized.\n" );
      return;
    }
    simple_lock( &mcbP->m_lock );

    if( mcbP->isShutdownNotified == FALSE )
    {
        rc = shutdown_notify_unreg(sn);
        if(rc)
  {
bsdlog(LOG_DEBUG|LOG_KERN,"'shutdown_notify_unreg' failed with rc:%d sn:%p\n",rc,sn);
  }
        else
  {
bsdlog(LOG_DEBUG|LOG_KERN,"'shutdown_notify_unreg' unregistered successfully. sn:%p\n",sn);
  }
    }

    /* set the termination flag and wake up the main thread*/
    mcbP->flags = STATE_TERMINATION_REQUESTED;
    e_wakeup( &mcbP->m_cond );

bsdlog(LOG_DEBUG|LOG_KERN,"Signaling the main thread to perform the termination.\n" );
    while ( mcbP->flags != STATE_TERMINATED )
        e_sleep_thread( &mcbP->m_cond,&mcbP->m_lock,LOCK_SIMPLE );

bsdlog(LOG_DEBUG|LOG_KERN,"Terminating the kernel process.pid=%d\n",mcbP-> kern_process_pid);
    mcbP->isInitialized = FALSE;
    simple_unlock( &mcbP->m_lock );
    lock_free(&mcbP->m_lock);
    mcbP->m_cond = EVENT_NULL;
    pidsig( mcbP->kern_process_pid, SIGTERM );
   /*Unpin the kernel extension.*/
    rc = unpincode(ke_entry_point);
    if( rc )
    {
        bsdlog(LOG_DEBUG | LOG_KERN,"Failed to unpin the code. rc = %d\n",rc );
    }
    return;
}

void ke_main_func(int flag, void* init_parms, int parms_length)
{
    int rc=0;
    /* set the init process as parent for this kernel process*/
    (void)setpinit();
    /* creates a new session and make it as session leader */
    (void)setsid();
    bsdlog(LOG_DEBUG|LOG_KERN, "Entered into main process.\n");
    simple_lock( &mcbP->m_lock);
    rc =  createKThread();
   if( rc )
    {
bsdlog(LOG_DEBUG|LOG_KERN,"Main thread failed to create kernel thread.         rc: %d\n",rc);
      mcbP->flags = STATE_INIT_FAILED;
      e_wakeup( &mcbP->m_cond );
      simple_unlock( &mcbP->m_lock);
      return;
    }

    else
    {
bsdlog(LOG_DEBUG|LOG_KERN,"Successfully created the kernel thread. pid=%d ppid=%d\n",getpid(),getppid());
      mcbP->flags = STATE_INITIALIZED;
      e_wakeup( &mcbP->m_cond );
    }

    /*fill the call back handler to get notified for the shutdown event and
      pass this as an argument to the kernel service */
    sn->func = &notifyShutdown;

   rc = shutdown_notify_reg(sn);
    if(rc)
    {
bsdlog(LOG_DEBUG|LOG_KERN,"'shutdown_notify_reg' failed with rc:%d sn:%p\n",rc,sn);
    }
    else
    {
bsdlog(LOG_DEBUG|LOG_KERN,"'shutdown_notify_reg' registered successfully. sn:%p\n",sn);
    }

    while( 1 )
    {
bsdlog(LOG_DEBUG|LOG_KERN,"Main thread going to sleep till Termination requested...\n");

      while( mcbP->flags != STATE_TERMINATION_REQUESTED )
      {
      e_sleep_thread( &mcbP->m_cond,&mcbP->m_lock,LOCK_SIMPLE );
}
      if ( mcbP->flags == STATE_TERMINATION_REQUESTED )
            break;
    }

    mcbP->flags = STATE_TERMINATED;
    e_wakeup( &mcbP->m_cond );
    simple_unlock( &mcbP->m_lock );
    bsdlog(LOG_DEBUG|LOG_KERN,"Main thread successfully terminated.\n");
    return;
}
int createKThread()
{
    tid_t tid;
    int rc=0;

/* kernel service used for creating kernel thread, upon success returns thread id and sets the state to IDLE, upon failure returns -1*/
    tid = thread_create();

    if ( tid == -1 )
    {
bsdlog(LOG_DEBUG|LOG_KERN, "Failed to create kernel thread. errno=%d\n", getuerror() );
      return -1;
    }

    /* kernel service used to make the kernel thread runnable */
    rc = kthread_start( tid, kern_thread_func, NULL, NULL, NULL, NULL );
    if ( rc != 0 )
    {
bsdlog(LOG_DEBUG|LOG_KERN,"Failed to start the kernel thread 'tid=%d'. rc=%d\n", tid, rc );
      return -1;
    }

    bsdlog(LOG_DEBUG | LOG_KERN, "createKThread: tid=%d\n",tid )
    return 0;
}

void kern_thread_func(void* arg)
{
        bsdlog(LOG_DEBUG | LOG_KERN,"In kern_thread_func....\n");
        return;
}

long notifyShutdown(shutdown_notify_t *sn)
{
    int rc=0;

    if( sn->reason == SHUTDOWN_REASON_USER )
    {
        bsdlog(LOG_DEBUG|LOG_KERN,"User initiated shutdown.\n");
    }
    else if( sn->reason == SHUTDOWN_REASON_EPOW )
    {
        bsdlog(LOG_DEBUG|LOG_KERN,"EPOW event.\n");
    }

    if( sn->status == SHUTDOWN_STATUS_PREPARE )
    {
        bsdlog(LOG_DEBUG|LOG_KERN,"Preparing for shutdown.\n");
        return SHUTDOWN_STATUS_FINISH;
    }
    else if( sn->status == SHUTDOWN_STATUS_COMMENCE )
    {
        bsdlog(LOG_DEBUG|LOG_KERN,"Commencing shutdown.\n");
    }
    else if( sn->status == SHUTDOWN_STATUS_FINISH )
    {
        bsdlog(LOG_DEBUG|LOG_KERN,"Shutdown has completed.\n");
    }

    if( sn->oper == SHUTDOWN_NOTIFY_PREPARE )
    {
        bsdlog(LOG_DEBUG|LOG_KERN,"Shutdown has started.\n" );
    }
    else if( sn->oper == SHUTDOWN_NOTIFY_REBOOT ||
             sn->oper == SHUTDOWN_NOTIFY_HALT )
    {
        sn->status = SHUTDOWN_STATUS_FINISH;
    }

    mcbP->isShutdownNotified = TRUE;
    return SHUTDOWN_STATUS_FINISH;
}

int testke_syscall( int arg )
{
bsdlog(LOG_DEBUG|LOG_KERN,"In testke_syscall: recevied arg %d.....\n",arg);
    return 0;
}

Dynamic loading of system calls

Dynamic loading allows an application to load a new module into its address space for either adding a new functionality or implementing an existing interface, without having to link the required libraries.

The following system calls help to dynamically load, unload, start, and stop kernel extensions from a user application.

dlopen()

This function loads the specified module into the address space of the calling process and returns a handle to the loaded module. Dependents of the module are automatically loaded as well. If any of its dependents cannot load due to say, an unresolved symbol, the whole module fails to load. If the module is already loaded, it does not load again, but a new unique handle is returned. The handle is used in subsequent calls in dlsym() and dlclose() functions.

Note that dlopen() on AIX also allows loading of archive members of shared libraries using the RTLD_MEMBER flag.

dlsym()

This function looks up the specified symbol in the loaded module (and its dependents) and returns its address.

dlclose()

This function is used to close access to a module loaded with the dlopen() subroutine. In addition, access to dependent modules of the module being unloaded is removed as well. Modules being unloaded will however, not be removed from the address space of the process if they are still required by other modules. Nevertheless, subsequent uses of that handle are invalid and further uses of the symbols that were exported by the module being unloaded result in an undefined behavior.

dlerror()

This function is used to obtain information about the last error that occurred in an immediately preceding call to dlopen(), dlclose(), or dlsym(). After a call is made to this function, subsequent calls, without any intervening dynamic loading errors, return NULL. Hence it may be a better idea for multi-threaded applications to perform error determination using the status held in errno.

Note: errno is a global variable provided by operating system to store the last known error value of any system call failure.

Sample wrapper program (libwrapper.c)

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <memory.h>
#include <errno.h>
#include <dlfcn.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <sys/sysconfig.h>
#include <sys/device.h>
#include <syslog.h>
#define KERNEXT_PATH      "/usr/lib/drivers/testke"
#define LIBNAME                 "/usr/lib64/libkernext.a(shr.o)"
#define TESTKE_SYCALL_SYMB_NAME     "testke_syscall"

int loadKernelExtension (void);
int unloadKernelExtension ( void );
mid_t queryKernelExtension( char *kernExtPath );
int load_testke_syscall(int);

void *ke_sym = NULL;
void *_libhandle = NULL;
int (*ke_syscall)(int)=NULL;

static int validateKernExt( char *kernExtPath )
{
   struct stat statbuf;
   /* Check existence of file */

   if ( stat(kernExtPath , &statbuf) != 0 )
   {
     printf("Kernel extension not found at '%s'\n",  kernExtPath );
     return (-1);
   }
   return 0;
}

mid_t queryKernelExtension( char *kernExtPath )
{
   struct cfg_load queryCmd;
   bzero( &queryCmd, sizeof( queryCmd ) );
   queryCmd.path       = kernExtPath;
   queryCmd.libpath    = NULL;

   if ( sysconfig( SYS_QUERYLOAD, &queryCmd, sizeof( queryCmd ) ) < 0 )
   {
     printf("ERROR: sysconfig(SYS_QUERYLOAD)\n");
     return (mid_t) -1;
   }
   return queryCmd.kmid;
}

int loadKernelExtension( void )
{
   int                 rc;
   struct cfg_load     loadCmd;
   struct cfg_kmod     configCmd;
   mid_t               kmid;
   char *const         kernExtPath = KERNEXT_PATH;
   const char *error = NULL;

   do
   {
     if ( validateKernExt( kernExtPath ) == -1 )
     return -1;

     kmid = queryKernelExtension( kernExtPath );
     if ( kmid == (mid_t) -1 )
     return -1;

     if ( kmid )
     {
       printf("Kernel module '%s' already loaded with id %u'\n",kernExtPath, kmid );
       break;
     }
     if(!kmid)
     {
     loadCmd.path    = kernExtPath;
     loadCmd.libpath = NULL;
     /* Load driver into kernel. */
     rc= sysconfig( SYS_KLOAD|SYS_64BIT, &loadCmd, sizeof(loadCmd) );
     if ( rc )
     {
       printf("ERROR: sysconfig(SYS_KLOAD)\n");
       return -1;
     }
    }

     /* Now initialize the driver */
     kmid = queryKernelExtension( kernExtPath );

     if ( kmid == (mid_t) -1 )
     return -1;
     if ( !kmid )
     {
       printf("Kernel module '%s' not loaded with id '%u'\n", kernExtPath, kmid );
       return -1;
     }

     configCmd.kmid              = kmid;
     configCmd.cmd               = CFG_INIT;
     configCmd.mdiptr            = NULL;
     configCmd.mdilen            = 0;

     rc=sysconfig(SYS_CFGKMOD|SYS_64BIT,&configCmd,sizeof(configCmd));

     if ( rc )
     {
       printf("Failed to Initialize Kernel Extension '%s' after a successfull load. kmid=%u\n", kernExtPath, configCmd.kmid );
       return -1;
     }

     printf("Kernel module '%s' Successfully loaded with id '%u'\n",kernExtPath,kmid );
  }while(0);

  if ( !_libhandle )
  {if(!(_libhandle=dlopen(LIBNAME,RTLD_NOW|RTLD_MEMBER|RTLD_GLOBAL)))
  {
     printf("dlopen error %s\n",dlerror());
     return -1;
  }
  }

  if (!(ke_sym = dlsym(_libhandle, TESTKE_SYCALL_SYMB_NAME)))
  {
     error = (const char*)dlerror();
     printf("Unable to get testke_syscall symbol. dlerror : %s \n", error);
     return -1;
   }
   else
   {
      ke_syscall = (int(*)(int))ke_sym;
   }
return 0;
}

int unloadKernelExtension ( void )
{
   int                 rc;
   struct cfg_load     loadCmd;
   struct cfg_kmod     configCmd;
   mid_t               kmid;
   char *const         kernExtPath = KERNEXT_PATH;

   kmid = queryKernelExtension( kernExtPath );
   if ( kmid == (mid_t) -1 )
   return -1;
   if ( !kmid )
   {
       printf("Failed to find Kernel module '%s' loaded in kernel.\n", kernExtPath );
       return -1;
    }

    /* Send Termination cmd to the kernel extension */
    configCmd.kmid              = kmid;
    configCmd.cmd               = CFG_TERM;
    configCmd.mdiptr            = (caddr_t) NULL;
    configCmd.mdilen            = 0;

    rc = sysconfig( SYS_CFGKMOD|SYS_64BIT, &configCmd, sizeof(configCmd) );
    if ( rc )
    {
       printf("Termination request to Kernel Extension failed '%s' kmid=%u\n",kernExtPath, kmid );
       return -1;
    }

    if ( _libhandle )
    {
       if(dlclose(_libhandle) == EINVAL)
       {
          printf("dlclose error %s\n", dlerror());
          return -1;
       }
    _libhandle = NULL;
    }

     bzero( &loadCmd, sizeof( loadCmd ) );
     loadCmd.kmid = kmid;
     /* Unload the kernel extension. */
    rc = sysconfig( SYS_KULOAD|SYS_64BIT, &loadCmd, sizeof( loadCmd ) );
    if ( rc )
    {
        printf("ERROR: sysconfig(SYS_KULOAD)");
        return -1;
    }

    printf("Kernel module '%s' with id '%u' Successfully unloaded.\n",kernExtPath, kmid );

    ke_syscall = NULL;

return 0;
}

int load_testke_syscall(int arg)
{
   if( !ke_syscall )
   {
      if (!_libhandle)
      {
        if(loadKernelExtension() != 0)
        {
          return NULL;
        }
        else
        {
          return (((int (*)(int))ke_syscall)(arg));
        }
      }
      return NULL;
    }

  return (((int (*)(int))ke_syscall)(arg));
}

wrapper export file

The following wrapper APIs are used by the user application to manage kernel extension:

  • #!/usr/lib/libwrapper.a(shr.o)
  • loadKernelExtension
  • unloadKernelExtension
  • queryKernelExtension
  • load_testke_syscall

Sample application program (Userapp.c)

#include <stdio.h>
int main ()
{
    int rc = 0;
    fprintf(stderr,"load and init testke\n");
    if( loadKernelExtension() != 0 )
    {
     fprintf(stderr, "loading kernel extension failed\n");
return 0;
    }
    fprintf(stderr, "Invoking testke_syscall...\n");
    if( (iVal = load_testke_syscall(25) < 0))
    {
        perror("testke_syscall error");
        exit(1);
    }
    sleep(120);
    fprintf(stderr,"term and unloading testke\n");
    if( unloadKernelExtension() != 0 )
    {
fprintf(stderr, "unloading kernel extension failed\n");
return 0;
    }
    return 0;
}

Makefile to build all the programs

all: testke libkernext.a  libwrapper.a Userapp

testke: testke.o
        cc -q64 -o testke.o -c -DEBUG -D_KERNEL -DIBMR2 testke.c -e ke_entry_point
        ld -b64 -o testke testke.o -e ke_entry_point -bI:/usr/lib/kernex.exp -bE:testke.exp -bloadmap:kext.map -lsys -lcsys
        ld -b64 -o shr.o -bI:testke.exp -bE:testke.exp -bM:Sre -bnoentry -bexpall -lc
        ar -X64 crlo libkernext.a shr.o

libwrapper.a: libwrapper.o
        cc -q64 -o libwrapper.o -c libwrapper.c -bnoentry
        ld -b64 -o shr.o libwrapper.o -bE:libwrapper.exp -bM:Sre -bnoentry -bexpall -lc
        ar -X64 crlo libwrapper.a shr.o

Userapp: Userapp.o
        cc -q64 -o Userapp.o -c Userapp.c
        cc -q64 -o Userapp.X -s Userapp.o -bI:libwrapper.exp -L. -lwrapper
        mv Userapp.X Userapp

.PHONY: clean
clean:
        rm -rf *.o *.a *.map Userapp testke

After copying the source code, run the make command to generate the binaries. If the above makefile is run, the following binary files are created.

  • testke – This is the kernel extension binary file, place this in /usr/lib/drivers.
  • libkernext.a – This is a static library which consists of the system call symbol (place this in /usr/lib64/).
  • libwrapper.a – This is a static library that provides the API to operate kernel extension and this will be linked with the application program.
  • Userapp – This is the application program to control the kernel extension.

Output of user-application execution

# ./Userapp
load and init testke
Kernel module '/usr/lib/drivers/testke' Successfully loaded with id '2698199040'
Invoking testke_syscall...
term and unloading testke
Kernel module '/usr/lib/drivers/testke' with id '2698199040' Successfully unloaded.

The following output is generated when you run the shutdown command.

alt

The following output is generated when shutdown -Fr is run on the system:

alt

kernel services for detecting shutdown and kernel sockets

For detecting shutdown, kernel services, shutdown_notify_reg() and shutdown_notify_unreg() are used. The sample kernel extension program mentioned earlier has this code implemented.

You can use the following kernel service for kernel socket communication.

socreate()

This function is used to create a kernel socket of a specified family.

struct socket *ksocket_ptr = NULL;
struct mbuf   *mbuf_l = NULL;

sockfd = socreate(AF_NDD, &ksocket_ptr,SOCK_DGRAM, NDD_PROT_ETHER);
if( sockfd > 0 )
{
bsdlog(LOG_DEBUG|LOG_KERN,"socreate: socket creation failed.\n");
return -1;
}
else
{
bsdlog(LOG_DEBUG|LOG_KERN,"socreate: successfully created socket rc:%d\n",sockfd);
}

/* using NDD sockets for ethernet layer */
struct sockaddr_ndd_8022 socket_ndd_8022;

socket_ndd_8022.sndd_8022_family = AF_NDD;
socket_ndd_8022.sndd_8022_len = sizeof(socket_ndd_8022);
socket_ndd_8022.sndd_8022_filtertype = NS_ETHERTYPE;
socket_ndd_8022.sndd_8022_ethertype = htons( 0x600 );
socket_ndd_8022.sndd_8022_filterlen = sizeof(struct ns_8022);
strncpy( (char*)socket_ndd_8022.sndd_8022_nddname,intfname,sizeof(intfname));

/* allocate the memory buffer */
if (( mbuf_l = m_get ( M_DONTWAIT, MT_SONAME )) == 0 )
{
bsdlog(LOG_DEBUG|LOG_KERN,"m_get() failed\n");
soshutdown(ksocket_ptr,2);
soclose(ksocket_ptr);
return -1;
}

mbuf_l->m_len = socket_ndd_8022.sndd_8022_len;

memcpy((mbuf_l)-> m_hdr.mh_data,&socket_ndd_8022,socket_ndd_8022.sndd_8022_len);

soconnect()

This function is used to establish a connection.

rc = soconnect(ksocket_ptr,mbuf_l);
if( rc )
{
bsdlog(LOG_DEBUG|LOG_KERN,"soconnect failed rc:%d\n",rc);
m_free( mbuf_l );
soshutdown(ksocket_ptr,2);
soclose(ksocket_ptr);
return -1;
}

sosend()

This function is used to send the data.

/* this method needs socket pointer, message buffer and void* which consists of data need to send as parameters */
int sendPacket( struct socket *kptr,struct mbuf *mbuf_data, void *data)
{
    int rc=0;
    struct iovec  iovect;
    struct uio    uio_buf;

    iovect.iov_base     = data;
    iovect.iov_len      = 1024;

    uio_buf.uio_iov     = &iovect;
    uio_buf.uio_iovcnt  = 1;
    uio_buf.uio_iovdcnt = 0;
    uio_buf.uio_segflg  = UIO_SYSSPACE;
    uio_buf.uio_offset  = 0;
    uio_buf.uio_resid   = 1024;
    /* kernel service used for sending the message */
    if( kptr != NULL )
    {
    rc = sosend( kptr,mbuf_data,&uio_buf,NULL,NULL,0);
if( rc )
{
bsdlog(LOG_DEBUG|LOG_KERN,"sosend failed rc:%d\n",rc);
return rc;
}
else
{
bsdlog(LOG_DEBUG|LOG_KERN,"sosend: successfully sent %d byte`s \n",sizeof(ethFrame));
}
    }
    else
    {
bsdlog(LOG_DEBUG|LOG_KERN,"sosend failed: Socket pointer is NULL!!");
    }
    return rc;
}

soreceive()

This function is used to receive data.

struct iovec  iovect;
struct uio    uio_buf;
char data[1024];

iovect.iov_base     = data;
iovect.iov_len      = 1024;

uio_buf.uio_iov     = &iovect;
uio_buf.uio_iovcnt  = 1;
uio_buf.uio_iovdcnt = 0;
uio_buf.uio_segflg  = UIO_SYSSPACE;
uio_buf.uio_offset  = 0;
uio_buf.uio_resid   = 1024;
rc = soreceive( kptr,NULL,&uio_buf,NULL,NULL,NULL);
if( rc )
{
bsdlog(LOG_DEBUG|LOG_KERN,"soreceiv failed rc:%d\n",rc);
return rc;
}

soshutdown()

This function closes read and write operations on the connection.

soshutdown( kptr,2 );

soclose()

This function closes the socket.

soclose(kptr );

Apart from the above kernel services, there are other services (such as kern_socreate(), kern_sobind(), kern_soconnect(), kern_sosend() and so on) that can be used for kernel socket communication. etc.

Useful commands for verification

Command to check the object file content of kernel extension:

# dump -X64 -HTv /usr/lib/drivers/testke

/usr/lib/drivers/testke:

                        ***Loader Section***
                      Loader Header Information
VERSION#         #SYMtableENT     #RELOCent        LENidSTR
0x00000001       0x00000018       0x00000029       0x00000046

#IMPfilID        OFFidSTR         LENstrTBL        OFFstrTBL
0x00000002       0x00000508       0x00000142       0x0000054e


                        ***Import File Strings***
INDEX  PATH                          BASE                MEMBER
0      /usr/local/staf/lib:/opt/freeware/lib64:/usr/local/staf/lib
1      /                             unix

                        ***Loader Symbol Table Information***
[Index]      Value      Scn     IMEX Sclass   Type           IMPid Name

[0]     0x00000000    undef      IMP     DS EXTref           /unix bsdlog
[1]     0x00000000    undef      IMP     DS EXTref           /unix creatp
[2]     0x00000000    undef      IMP     DS EXTref           /unix e_sleep_thread
[3]     0x00000000    undef      IMP     DS EXTref           /unix e_wakeup
[4]     0x00000000    undef      IMP     DS EXTref           /unix getpid
[5]     0x00000000    undef      IMP     DS EXTref           /unix getppid
[6]     0x00000000    undef      IMP     DS EXTref           /unix getuerror
[7]     0x00000000    undef      IMP     DS EXTref           /unix initp
[8]     0x00000000    undef      IMP     DS EXTref           /unix kthread_start
[9]     0x00000000    undef      IMP     DS EXTref           /unix pidsig
[10]    0x00000000    undef      IMP     DS EXTref           /unix setpinit
[11]    0x00000000    undef      IMP     DS EXTref           /unix setsid
[12]    0x00000000    undef      IMP     DS EXTref           /unix pincode
[13]    0x00000000    undef      IMP     DS EXTref           /unix unpincode
[14]    0x00000000    undef      IMP     DS EXTref           /unix simple_lock_init
[15]    0x00000000    undef      IMP     DS EXTref           /unix simple_lock
[16]    0x00000000    undef      IMP     DS EXTref           /unix simple_unlock
[17]    0x00000000    undef      IMP     DS EXTref           /unix lock_alloc
[18]    0x00000000    undef      IMP     DS EXTref           /unix lock_free
[19]    0x00000000    undef      IMP     DS EXTref           /unix shutdown_notify_reg
[20]    0x00000000    undef      IMP     DS EXTref           /unix shutdown_notify_unreg
[21]    0x00000000    undef      IMP     DS EXTref           /unix thread_create
[22]    0x000005b8    .data    ENTpt     DS SECdef        [noIMid] ke_entry_point
[23]    0x00000618    .data      EXP SV3264 SECdef        [noIMid] testke_syscall

Command to check the content of static library:

# nm -X64 /usr/lib64/libkernext.a
/usr/lib64/libkernext.a[shr.o]:
testke_syscall       U         -

Command to check linking of libraries:

# ldd Userapp
Userapp needs:
         /usr/lib/libc.a(shr_64.o)
         ./libwrapper.a(shr.o)
         /unix
         /usr/lib/libcrypt.a(shr_64.o)

Summary

This tutorial is helpful to write, compile, and run kernel extensions. It can be more helpful in scenarios where there is a tight relationship between a user application and a kernel extension (for example, user application itself loading kernel extension and using the system calls provided by the kernel extensions). These are tricky scenarios where there are high chances of build failures and kernel extension not working as expected. Proper synchronization needs to be maintained between kernel extension and user application to make it work. Also, this tutorial provided some less-known kernel interfaces such as L2 raw sockets to communicate with user space applications.

Goutham Jamalpur Sai
Alok Chandra Mallick