Tag: opensource

Demystifying request_module()

Introduction

request_module() is a function used in kernel modules to load another kernel module on fly.

It is defined in linux/kmod.h.

#define request_module(mod...) __request_module(true, mod)

Apart from module name, we can also specify module command-line parameters
accepted by the module.

request_module() internally calls userspace program modprobe to
load a specified module.
It returns 0, if modprobe is successfully called. Return value 0 doesn’t guarantee
that module gets loaded successfully by modprobe. request_module() will only
invoke modprobe (and if it is able to invoke it successfully, it returns 0)
but doesn’t check return status of modprobe program.
Hence if due to some reason modprobe failed, still request_module() would have
returned 0 in the kernel module.

How request_module() works and its workflow

request_module() calls __request_module().

__request_module() will wait until modprobe returns. Function checks whether
global variable “modprobe_path” is set by kmod subsystem. This is to check for cases
where request_module() is called in early booting stage, when modprobe is
yet started.
“modprobe_path” is set via /proc/sys/

[root@kashish kmod_subsystem_usage]# cat /proc/sys/kernel/modprobe
/sbin/modprobe

Then __request_module() will copy  module name and its arguments in “module_name” variable.

va_start(args, fmt);
ret = vsnprintf(module_name, MODULE_NAME_LEN, fmt, args);
va_end(args);

and finally calls

ret = call_modprobe(module_name, wait ? UMH_WAIT_PROC : UMH_WAIT_EXEC);

call_modprobe() prepares the command and call call_usermodehelper_exec()

 
static int call_modprobe(char *module_name, int wait)
{
        struct subprocess_info *info;
        static char *envp[] = {
                "HOME=/",
                "TERM=linux",
                "PATH=/sbin:/usr/sbin:/bin:/usr/bin",
                NULL
        };

        char **argv = kmalloc(sizeof(char *[5]), GFP_KERNEL);
        if (!argv)
                goto out;

        module_name = kstrdup(module_name, GFP_KERNEL);
        if (!module_name)
                goto free_argv;

        argv[0] = modprobe_path;
        argv[1] = "-q";
        argv[2] = "--";
        argv[3] = module_name;  /* check free_modprobe_argv() */
        argv[4] = NULL;

        info = call_usermodehelper_setup(modprobe_path, argv, envp, GFP_KERNEL,
                                         NULL, free_modprobe_argv, NULL);
        if (!info)
                goto free_module_name;

        return call_usermodehelper_exec(info, wait | UMH_KILLABLE);

call_usermodehelper_setup() is always called prior function call_usermodehelper_exec(), to initialize
a helper process structure which contains details of command to be issued in userspace.
call_usermodehelper_setup() is a exported function.

The user space program is invoked using workqueues. The subprocess_info{} contains a
work structure where work function __call_usermodehelper() is added.
And the work is enqueued in workqueue by call_usermodehelper_exec().
The workqueue is created when kmod subsystem(kmod.ko) is loaded.

void __init usermodehelper_init(void)
{
        khelper_wq = create_singlethread_workqueue("khelper");
        BUG_ON(!khelper_wq);
}
struct subprocess_info *call_usermodehelper_setup(char *path, char **argv,
                char **envp, gfp_t gfp_mask,
                int (*init)(struct subprocess_info *info, struct cred *new),
                void (*cleanup)(struct subprocess_info *info),
                void *data)
{
        struct subprocess_info *sub_info;
        sub_info = kzalloc(sizeof(struct subprocess_info), gfp_mask);
        if (!sub_info)
                goto out;

        INIT_WORK(&sub_info->work, __call_usermodehelper); <path = path;
        sub_info->argv = argv;
        sub_info->envp = envp;

        sub_info->cleanup = cleanup;
        sub_info->init = init;
        sub_info->data = data;
  out:
        return sub_info;
}
EXPORT_SYMBOL(call_usermodehelper_setup);

All the magic of calling a user program is done by call_usermodehelper_exec()

 
int call_usermodehelper_exec(struct subprocess_info *sub_info, int wait)
{
...
        /*
         * Set the completion pointer only if there is a waiter.
         * This makes it possible to use umh_complete to free
         * the data structure in case of UMH_NO_WAIT.
         */
        sub_info->complete = (wait == UMH_NO_WAIT) ? NULL : &done;
        sub_info->wait = wait;

        queue_work(khelper_wq, &sub_info->work); <retval;
out:
        call_usermodehelper_freeinfo(sub_info);
unlock:
        helper_unlock();
        return retval;
}

__call_usermodehelper() is called by scheduler and it will spwan a kthread based on whether
__request_module() was called with wait or without wait.


/* This is run by khelper thread  */
static void __call_usermodehelper(struct work_struct *work)
{
        struct subprocess_info *sub_info =
                container_of(work, struct subprocess_info, work);
        pid_t pid;

        if (sub_info->wait & UMH_WAIT_PROC)
                pid = kernel_thread(wait_for_helper, sub_info,
                                    CLONE_FS | CLONE_FILES | SIGCHLD);
        else
                pid = kernel_thread(____call_usermodehelper, sub_info,
                                    SIGCHLD);

        if (pid < 0) { sub_info->retval = pid;
                umh_complete(sub_info);
        }
}

kernel_thread() is an internal function of kernel thread subsystem which ultimately
creates kernel thread. When we call kthread_create() from kernel module, this is the
function which is called internally.

In case of __request_module() is called with wait=true, kthread is created to execute
wait_for_helper().

wait_for_helper(), a parent kthread, will also create a new child
kthread to execute ____call_usermodehelper() and waits until child exits.

In case of __request_module is called with wait=false, kthread is created directly to
invoke ____call_usermodehelper().

And finally function ____call_usermodehelper() calls do_execve() to execute usermode program.

Pre-requisites of using request_module()

In order to make request_module() work, Makefile must do some magic to copy modules object files in “/lib/modules/$(KREL)/extras/” or “/lib/modules/$(KREL)/kernel/”

“/lib/modules/$(KREL)/kernel/” generally contains kernel modules which are being installed during kernel compilation.
The kernel modules are copied to “/lib/modules/$(KREL)/kernel/” when we execute
“make modules_install” during kernel compilation.

Similarly external modules are copied to “/lib/modules/$(KREL)/extras/” by default when we execute “modules_install” target of module’s Makefile.
To copy .ko’s to some custom path, we can use “INSTALL_MOD_DIR” in Makefiles

The .ko’s need to present in above directories because modprobe try to search the modules inside “/lib/modules/$(KREL)/extras/” or “/lib/modules/$(KREL)/kernel/”

KDIR – Path to kernel source tree
KREL/KERNELRELEASE – Version of kernel, i.e. ‘uname -r’ output
INSTALL_MOD_DIR – an alternate path to “/lib/modules/$(KERNEL_RELEASE)/extras/”
INSTALL_MOD_PATH – an alternate path to where the sources are residing. If INSTALL_MOD_PATH=/my_kernel
then kernel modules will be copied to “/my_kernel/lib/modules/$(KERNEL_RELEASE)/kernel/”

If we don’t write Makefile to copy .ko’s at correct location, call to request_module()
might fail.

EXPORT_SYMBOL() and its history

EXPORT_SYMBOL() is a function macro to export symbol across loadable modules.

In releases prior to 2.4, inter_module_* were used to share variables across modules.
Now all inter_module_* apis are deprecated by linux kernel

void inter_module_register(const char *string, struct module *module,
const void *data);
void inter_module_unregister(const char *string);
void inter_module_unregister(const char *string);
const void *inter_module_get_request(const char *string, const char *module);
void inter_module_put(const char *string);

Example

signal_handling.ko will spawn a thread and load another module.
The other module(send_signal.ko) will send signal to the module (signal_handling.ko)
which loaded it.



signal_handling.c


#include <linux/init.h>
#include <linux/module.h>       /* Specifically, a module */
#include <linux/kernel.h>       /* We're doing kernel work */
#include <linux/kthread.h>
#include <linux/delay.h>        /* for msleep() */
#include <asm/siginfo.h>        /* for signals */
#include <linux/signal.h>
#include <linux/kmod.h>         /* for request_module() */

MODULE_DESCRIPTION("dynamically loading module through kmod subsystem and "
                   "receiving signal from another module");

struct task_struct *k1;
struct task_struct *saved_task;

//Global vars
int val = 0;
int flag = 0;                   // a flag is set when val reached 10

//Thread func for k1
int inc_func(void *arg)
{
        int ret;
        saved_task = current;
        printk("%s : pid of current thread %u\n", current->comm, current->pid);
        kthread_pid = current->pid;

        msleep(1000);
        //load module "send_signal.ko"
        ret = request_module("send_signal");
        if (ret != 0)
                printk("%s : unable to load module, Status=%d\n", current->comm, ret);

        allow_signal(SIGUSR1);

        set_current_state(TASK_INTERRUPTIBLE);

        while (!kthread_should_stop()) {
                int ret;
                ret = schedule_timeout(msecs_to_jiffies(1000));

                if (signal_pending(current)) {
                        printk("%s : signal is pending****\n", current->comm);
                        flush_signals(current);
                }

                if (val <=10 && !flag) {
                        val++;
                        printk("%s : current value is %d\n", current->comm, val);
                        if (val == 10) {
                                printk("%s : setting flag\n", current->comm);
                                flag = 1;
                        }
                }
                set_current_state(TASK_INTERRUPTIBLE);
        }
        set_current_state(TASK_RUNNING);
        printk("%s : out of while loop, exiting \"%s\"\n", current->comm, __FUNCTION__);
        return 0;
}

int __init startup(void)
{
        k1 = kthread_run(inc_func, NULL, "K1");
        return 0;
}

void __exit cleanup(void)
{
        printk("cleaning module \n");
        kthread_stop(k1);
        printk("stopped kernel thread 1 \n");
}

module_init(startup);
module_exit(cleanup);
MODULE_LICENSE("GPL");



send_signal.c


#include <linux/init.h>
#include <linux/module.h>       /* Specifically, a module */
#include <linux/kernel.h>       /* We're doing kernel work */
#include <linux/kthread.h>
#include <linux/delay.h>        /* for msleep() */
#include <asm/siginfo.h>        /* for signals */
#include <linux/signal.h>
#include <linux/kmod.h>

MODULE_DESCRIPTION("sending signal from 1 kthread to another"
                   " and using default signal handler");

//Follow me on twitter
MODULE_AUTHOR("@Kashish_Bhatia");

struct task_struct *k2;
struct task_struct *saved_task;

struct siginfo data_for_handler;

//signal descriptor
struct sigaction sig;

//Global vars
int val = 0;

int signal_receive_cnt = 0;     //max # of times signal should be serviced

extern pid_t kthread_pid;
struct task_struct *target_kthread;
struct pid *pid_struct;

//Thread func for k2
int send_signal_to_task(void *arg)
{
        int ret;
        while (!kthread_should_stop()) {
                msleep_interruptible(2000);
                if (kthread_pid && signal_receive_cnt != 2) {
                        printk("%s : pid of kthread to which signal is sent = %u\n", current->comm, kthread_pid);
                        //TODO: siginfo is passed to default sighandler.
                        //Can't check the info, as i dont have access to default
                        //signal handler.
                        data_for_handler.si_code = SI_KERNEL;
                        data_for_handler.si_int = val;
                        printk("%s : sending signal to process with pid %u...\n", current->comm, kthread_pid);

                        //TODO: convert pid to task_struct
                        pid_struct = find_get_pid(kthread_pid);
                        target_kthread = pid_task(pid_struct, PIDTYPE_PID);

                        //send signal
                        ret = send_sig_info(SIGUSR1, &data_for_handler, target_kthread);
                        if (ret) {
                                printk("%s : Error sending signal to thread:%s with pid:%d, status %d\n",
                                        current->comm, saved_task->comm, saved_task->pid, ret);
                                //return ret;
                        } else {
                                signal_receive_cnt++;
                        }
                }
        }
        printk("%s : exiting \"%s\"\n", current->comm, __FUNCTION__);
        return 0;
}

int __init startup(void)
{
        printk(KERN_ALERT "send_signal module is starting\n");
        k2= kthread_run(send_signal_to_task, NULL, "K2");
        return 0;
}

void __exit cleanup(void)
{
        printk("cleaning module \n");
        kthread_stop(k2);
        printk("stopped kernel thread 2 \n");
}

module_init(startup);
module_exit(cleanup);
MODULE_LICENSE("GPL");



Makefile


obj-m += send_signal.o
obj-m += signal_handling.o

INSTALLED_PATH=/lib/modules/$(shell uname -r)/extra/
clean-files := $(INSTALLED_PATH)/*.ko

all:
        make -C /lib/modules/$(shell uname -r)/build M=$(PWD) modules
modules_install:
        make -C /lib/modules/$(shell uname -r)/build INSTALL_MOD_DIR=extra M=$(PWD) modules_install
clean:
        make -C /lib/modules/$(shell uname -r)/build M=$(PWD) clean
Advertisements

Fixing yum install issues

Sometimes after installation of a new linux machine, we come across following failures while trying to install packages.

yum utility doesn’t  work due to repositories issues.

[root@kashish ~]# yum install lsscsi
Loaded plugins: presto, refresh-packagekit
Error: Cannot retrieve repository metadata (repomd.xml) for repository: fedora.

 

To resolve above errors nicely and cleanly, do as follows

  • Go to /etc/yum.repos.d

[root@kashish ~]# cd /etc/yum.repos.d

  • In all the *.repo files, change https to http in the mirror_list URLs

Now run following commands to rebuild rpm database

[root@kashish ~]# yum clean all
[root@kashish ~]# rm -f /var/lib/rpm/__db.*
[root@kashish ~]# rpm –rebuilddb

There is very useful YUM cheat sheet compiled by redhat. To learn more about yum
command, checkout this.

FUDCon 2015, Pune

I went Pune to participate in FUDCon 2015 and it was a wonderful experience for me.
I came to know about the event through one of my friend and after going through the details, I decided to participate in the event. I submitted the talk proposal on FUDCon website and fortunately my talk got accepted in the main event.

I left for Pune on 24th June via KSRTC bus at night and reached Pune nearly at noon, the next day. Bus dropped me at Swargate bus stop.
All the speakers stay was arranged at Cocoon hotel, Magarpatta city by FUDCon organizers, so I headed towards Cocoon hotel from Swargate . Thanks to the organizers for arranging accommodation and for being so welcoming.

Next Day on 26th June, I had to present my talk on “Linux IO : Native SCSI target implementation in linux kernel”. You can find the slides here. Frankly I was little nervous but was also excited as this was the first time I was presenting a technical paper in any event.

And the day finally arrived : Day 1 of FUDCon 2015
In the morning, before leaving for the event, the organizers gave some cool fedora bottles and bags to each speakers 🙂

When I reached MIT college (the venue of the event), I was amazed, as the event started phenomenally well . Many students and open source enthusiasts arrived (even from other cities) to attend the FUDCon event.

There were three tracks focusing on storage, cloud and container technologies, respectively scheduled for three days of the event.
I attended sessions on Glusterfs and ABI compatibility before lunch. My talk was scheduled after lunch and I got a really good response. I was not hoping that I would be able to explain the significance and intricacies of SCSI targets but the interest shown by the audience proved me wrong. I found students very curious to understand the topic. Some of the students even approached me after the session to know more about the topic.

Day 2
I observed only focused audience (interested students and people from Software companies). There were some cool sessions and workshops scheduled on openstack and linux systems on second day. I attended “Zero to Hero kernel module development” by Suchakra and “contributing to openstack” by Rohan. Both the talks were very informational.
Day 2 finished up and we all headed to BlueO for FUDPub. We did bowling, played foosball and danced till we got tired 😛

Day 3
Third day was focused on container related workshops (Docker, project atomic, kubernetes, etc). I attended mostly all container workshops and they were really enlightening for newbies who want to sink their heads in container technologies 🙂

As an addendum, I can just say that it was a great event and I really enjoyed, made new friends, interacted with open source enthusiasts and contributors and got inspired. Thanking all the FUDCon organizers and volunteers for this laudable endeavor and giving me opportunity to participate in FUDCon 2015.