Category: opensource

Demystifying Linux Kernel Timers

Some basics

jiffies – repesents time taken between two successive clock interrupt.
Incremented at every timer interrupt.
It is a global variable in kernel, declared as volatile so to avoid reading stale
value from memory.
Defined in linux/jiffies.h

HZ – represents clock interrupt frequency.
Its a macro set during compile time of kernel, based on different architectures.
Defined in linux/param.h

 Interval after which jiffies increments = 1000/HZ

If HZ=100, i.e. the frequency of timer interrupt is 100. In other words,
there will be 100 interrupts in 1 sec. That means at every 10ms, “jiffies”
is incremented. So “jiffies” value will equal to 100 after 1 sec.

System uptime = jiffies/HZ seconds

To convert seconds to jiffies, jiffies = seconds * HZ.

Using timers

Linux provides the timer subsystem to delay the execution of work/task and
execute them asynchronously. There can be cases that the timer fires when
the process which scheduled timer has exited already.
Timers are some underlying subsystem of kernel, which used by apis like schedule_timeout().

Timer data structure

struct timer_list {

struct list_head entry;                          /* entry in linked list of timers */
unsigned long expires;                         /* expiration value, in jiffies */
void (*function)(unsigned long);       /* the timer handler function */
unsigned long data;                              /* lone argument to the handler */
struct tvec_t_base_s *base;               /* internal timer field, do not touch */
};

What not-to-do while using timers and some of its weird properties

1. don’t pass auto variables as timer arguments
2. Timer function is called in interrupt context. Hence we cannot use any api which sleeps,
like kmalloc(), wait_event_*, semaphores, etc.
3. Timers have nasty property of binding themselves to the CPU on which they are created
4. There is no direct api to make periodic timers. mod_timer() has to be called time to time
to make them periodic.
5. Can’t use schedule_timeout(), msleep()
6. There is no relevance of “current” macro in timer function, as timer function is
executing in timer(interrupt) context

Useful Timer APIs

1. init_timer(timer)
add_timer(timer) / add_timer_on(timer, CPU)
They are used together generally.

2. setup_timer(timer, callback_fn, cb_data)
It internally calls init_timer()

3. del_timer(timer)
Destroy timer and not wait for timer callback to complete, if it is already fired.
It is safe to call del_timer() inside timer callback function.
This api works for both active or inactive(which is initialized but not running) timer.
If the timer is active, it will return 1, else return 0.

4. del_timer_sync(timer)
Destroy timer and wait for (spins until) timer callback to complete, if it is already fired.
This api cannot be called within the timer callback function, as it will lead a deadlock.

del_singleshot_timer_sync() is same as del_timer_sync().

When del_timer() returns, it guarantees only that the timer is no longer active (that is,
that it will not be executed in the future). On a multiprocessing machine, however, the
timer handler might already be executing on another processor.To deactivate the timer
and wait until a potentially executing handler for the timer exits, use del_timer_sync().
Unlike del_timer(), del_timer_sync() cannot be used from interrupt context.

5. try_to_del_timer_sync(timer)
It can be used before del_timer() to check if timer can be destroyed.
It will not destroy the timer if timer has hold any resource through spinlock, etc.

6. mod_timer(timer, expires)
It modifies the “expires” field of the timer to schedule it to fire again.
It is safe to modify the “expires” field of the deleted timer. In that
case, it will reactivate the timer

7. mod_timer_pending(timer, expires)
It allow modifying the timer’s “expires” field for only active timers. Check pending arg of the function.

8. mod_timer_pinned(timer, expires)
It should be used along with add_timer_on(), to pin timer on a particular CPU

9. set_timer_slack(timer, slack_hz)
It loosens the “expires” field of the timer. By using this API, the timer subsystem will
schedule the actual timer somewhere between the time mod_timer asks for, and that time plus the slack.

Other Useful links

1. Do not use in_atomic()
http://www.gossamer-threads.com/lists/linux/kernel/1142308

2. Something on in_interrupt()
http://yarchive.net/comp/linux/in_interrupt.html

3. About try_to_del_timer_sync()
https://lwn.net/Articles/137087/

4. About mod_timer_pinned()
https://lwn.net/Articles/327516/

Example


/*
 * Key Takeaways:
 * 1. How to use init_timer() and add_timer()
 * 2. How/where to use add_timer_on() and mod_timer_pinned()
 * 3. How to get CPU id (smp_processor_id())
 */

#include <linux/init.h>
#include <linux/kernel.h>
#include <linux/module.h>
#include <linux/timer.h>        /* for timers apis */
#include <linux/delay.h>        /* for msleep() */
#include <linux/hardirq.h>      /* for in_interrupt(), in_atomic() , moved to preempt_mask.h, 3.0 onwards */

MODULE_DESCRIPTION("Usage of init_timer(), add_timer(), add_timer_on() and mod_timer_pinned()");

//Twitter Handle
MODULE_AUTHOR("@Kashish_Bhatia");

struct timer_list timer;
struct timer_list timer2;
unsigned long *data;
spinlock_t data_lock = SPIN_LOCK_UNLOCKED;

void timer_func(unsigned long data)
{
        unsigned long *d = (unsigned long *)data;
        printk("%s() is running on CPU-%u\n", __FUNCTION__, smp_processor_id());
        spin_lock(&data_lock);
        (*d)--;
        printk("%s : data = %lu...\n",__FUNCTION__, *d);
        if (in_interrupt())
                printk("%s : In interrupt context...\n", __FUNCTION__);

        if (in_atomic())
                printk("%s : In atomic context...\n", __FUNCTION__);

        if ((*d) != 0) {
                //decrementing "data" faster
                mod_timer(&timer, jiffies + msecs_to_jiffies(500));
        }
        spin_unlock(&data_lock);
}

void timer2_func(unsigned long data)
{
        unsigned long *d = (unsigned long *)data;
        printk("%s() is running on CPU-%u\n", __FUNCTION__, smp_processor_id());
        spin_lock(&data_lock);
        (*d)++;
        printk("%s : data = %lu***\n",__FUNCTION__, *d);

        if (in_interrupt())
                printk("%s : In interrupt context***\n", __FUNCTION__);

        if (in_atomic())
                printk("%s : In atomic context***\n", __FUNCTION__);

        if ((*d) != 0) {
                //scheduling it again and pinning to same CPU
                mod_timer_pinned(&timer2, jiffies + msecs_to_jiffies(1000));
        }
        spin_unlock(&data_lock);
}

int __init start_module(void)
{
        data = kmalloc(sizeof(unsigned long), GFP_KERNEL);
        *data = 100;

        //create timer 1
        setup_timer(&timer, timer_func, (unsigned long)data);
        //set expiry time
        mod_timer(&timer, jiffies + msecs_to_jiffies(1000));

        //create timer 2
        init_timer(&timer2);
        timer2.function = timer2_func;
        timer2.expires = jiffies + msecs_to_jiffies(1000);
        timer2.data = (unsigned long)data;              //type-casting pointer to unsigned long
        //now schedule timer on specified CPU
        add_timer_on(&timer2, 1); //on CPU 1

        return 0;
}

void __exit stop_module(void)
{
        int i = 0;
        printk("%s() is running on CPU-%u\n", __FUNCTION__, smp_processor_id());
        while (timer_pending(&timer)) {
                printk("timer 1 is pending\n");
                msleep(500);
                i++;
                if (i == 5)
                        del_timer(&timer);
        }
        if (i == 0) {
                printk("deleting timer 1\n");
                del_timer(&timer);
        }

        i = 0;
        while (timer_pending(&timer2)) {
                printk("timer 2 is pending\n");
                msleep(500);
                i++;
                if (i == 5)
                        del_timer(&timer2);
        }
        if (i == 0) {
                printk("deleting timer 2\n");
                del_timer(&timer);
        }

        kfree(data);
        printk("exiting from module\n");

}

module_exit(stop_module);
module_init(start_module);
MODULE_LICENSE("GPL");

FUDCon 2015, Pune

I went Pune to participate in FUDCon 2015 and it was a wonderful experience for me.
I came to know about the event through one of my friend and after going through the details, I decided to participate in the event. I submitted the talk proposal on FUDCon website and fortunately my talk got accepted in the main event.

I left for Pune on 24th June via KSRTC bus at night and reached Pune nearly at noon, the next day. Bus dropped me at Swargate bus stop.
All the speakers stay was arranged at Cocoon hotel, Magarpatta city by FUDCon organizers, so I headed towards Cocoon hotel from Swargate . Thanks to the organizers for arranging accommodation and for being so welcoming.

Next Day on 26th June, I had to present my talk on “Linux IO : Native SCSI target implementation in linux kernel”. You can find the slides here. Frankly I was little nervous but was also excited as this was the first time I was presenting a technical paper in any event.

And the day finally arrived : Day 1 of FUDCon 2015
In the morning, before leaving for the event, the organizers gave some cool fedora bottles and bags to each speakers 🙂

When I reached MIT college (the venue of the event), I was amazed, as the event started phenomenally well . Many students and open source enthusiasts arrived (even from other cities) to attend the FUDCon event.

There were three tracks focusing on storage, cloud and container technologies, respectively scheduled for three days of the event.
I attended sessions on Glusterfs and ABI compatibility before lunch. My talk was scheduled after lunch and I got a really good response. I was not hoping that I would be able to explain the significance and intricacies of SCSI targets but the interest shown by the audience proved me wrong. I found students very curious to understand the topic. Some of the students even approached me after the session to know more about the topic.

Day 2
I observed only focused audience (interested students and people from Software companies). There were some cool sessions and workshops scheduled on openstack and linux systems on second day. I attended “Zero to Hero kernel module development” by Suchakra and “contributing to openstack” by Rohan. Both the talks were very informational.
Day 2 finished up and we all headed to BlueO for FUDPub. We did bowling, played foosball and danced till we got tired 😛

Day 3
Third day was focused on container related workshops (Docker, project atomic, kubernetes, etc). I attended mostly all container workshops and they were really enlightening for newbies who want to sink their heads in container technologies 🙂

As an addendum, I can just say that it was a great event and I really enjoyed, made new friends, interacted with open source enthusiasts and contributors and got inspired. Thanking all the FUDCon organizers and volunteers for this laudable endeavor and giving me opportunity to participate in FUDCon 2015.