本博客（http://blog.csdn.net/livelylittlefish ）贴出作者（三二一@小鱼）相关研究、学习内容所做的笔记，欢迎广大朋友指正！
analysis of ACE_Task::putq with timeout=0 when queue is full on Linux platform
Linux平台调用ACE_Task::putq分析（timeout=0，队列满）
content
0. Introduction
1. ACE_Task::putq introduction
2. call stack
3. call stack analysis
3.1 block analysis
3.2 system call analysis
4. analysis of ACE_Task::putq
4.1 the call process of ACE_Task::putq
4.2 each function analysis on the call path
(1) ACE_Task<ACE_SYNCH_USE>::putq
(2) ACE_Message_Queue<ACE_SYNCH_USE>::enqueue_tail
(3) ACE_Message_Queue<ACE_MT_SYNCH>::wait_not_full_cond
(4) ACE_Condition_Thread_Mutex::wait(const ACE_Time_Value *abstime)
(5) ACE_Condition_Thread_Mutex::wait(ACE_Thread_Mutex &mutex, const ACE_Time_Value *abstime)
(6) ACE_OS::cond_timedwait
(7) __pthread_cond_wait
(8) lll_futex_wait, lll_futex_timed_wait
(9) sys_futex
(10) do_futex
(11) futex_wait
(12) schedule_timeout
(13) schedule
5. conclusion
0. Introduction
In this article, we will discuss most actions of calling ACE_Task::putq with timeout=0. It includes ACE, glibc and Linux kernel source analysis, such as ACE_Task::putq, ACE_Message_Queue::enqueue_tail, pthread_cond_wait, sys_futex and schedule.
1. ACE_Task::putq introduction
ACE_Task::putq will put a message into the message queue. It has two parameters, and the second is timeout with default value NULL, as follows. It is declared in file ace/Task_T.h as below.
// For the following five method if @a timeout == 0, the caller will
// block until action is possible, else will wait until the
// <{absolute}> time specified in *@a timeout elapses). These calls
// will return, however, when queue is closed, deactivated, when a
// signal occurs, or if the time specified in timeout elapses, (in
// which case errno = EWOULDBLOCK).
/// Insert message into the message queue. Note that @a timeout uses
/// <{absolute}> time rather than <{relative}> time.
int putq (ACE_Message_Block *, ACE_Time_Value *timeout = 0);
from the comment, we can see that if timeout=0 and the queue is full, the caller will be blocked until action is possible, such as signal. Then it will NEVER return back until you kill the process.
2. call stack
When timeout=NULL, in the test, we will put many messages to fill the queue to the full, then it will be blocked and NEVER return. The call stack of it is as follows. We can see that it is blocked in __kernel_vsyscall at 0x00110402.
l pstack result
#0 0x00110402 in __kernel_vsyscall ()
#1 0x007065d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#2 0x008aa7ac in ACE_Condition_Thread_Mutex::wait ()
#3 0x008aa7eb in ACE_Condition_Thread_Mutex::wait ()
#4 0x0804b132 in ACE_Message_Queue<ACE_MT_SYNCH>::wait_not_full_cond ()
#5 0x0804d132 in ACE_Message_Queue<ACE_MT_SYNCH>::enqueue_tail ()
#6 0x0804e15a in ACE_Task<ACE_MT_SYNCH>::putq ()
...
l gdb bt result
(gdb) bt
#0 0x00110402 in __kernel_vsyscall ()
#1 0x007065d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#2 0x008aa7ac in ACE_Condition_Thread_Mutex::wait () from /usr/lib/libACE.so.5.6.2
#3 0x008aa7eb in ACE_Condition_Thread_Mutex::wait () from /usr/lib/libACE.so.5.6.2
#4 0x0804b132 in ACE_Message_Queue<ACE_MT_SYNCH>::wait_not_full_cond (this=0x9d425f8, timeout=0x0)
at /usr/include/ace/Message_Queue_T.cpp:1588
#5 0x0804d132 in ACE_Message_Queue<ACE_MT_SYNCH>::enqueue_tail (this=0x9d425f8, new_item=0xb7404588, timeout=0x0)
at /usr/include/ace/Message_Queue_T.cpp:1753
#6 0x0804e15a in ACE_Task<ACE_MT_SYNCH>::putq (this=0x9d42594, mb=0xb7404588, tv=0x0) at /usr/include/ace/Task_T.inl:36
...
3. call stack analysis
3.1 block analysis
Now, we disassemble the codes at 0x00110402 in gdb, like the following.
(gdb) disassemble 0x00110402
Dump of assembler code for function __kernel_vsyscall:
0x00110400 <__kernel_vsyscall+0>: int $0x80
0x00110402 <__kernel_vsyscall+2>: ret //it is blocked at ‘ret’, which is a return instruction
End of assembler dump.
or,
(gdb) x/i 0x00110402
0x110402 <__kernel_vsyscall+2>: ret //it is blocked at ‘ret’, which is a return instruction
This return instruction is at <__kernel_vsyscall+2>, then, <__kernel_vsyscall> is at 0x00110400.
(gdb) p __kernel_vsyscall //we can print this symbol
$1 = {<text variable, no debug info>} 0x110400 <__kernel_vsyscall>
(gdb) x/5i __kernel_vsyscall //display 5 instructions from this symbol address
0x110400 <__kernel_vsyscall>: int $0x80
0x110402 <__kernel_vsyscall+2>: ret
0x110403: nop
0x110404: nop
0x110405: nop
(gdb) x/5i 0x00110400 // display 5 instructions from this symbol address
0x110400 <__kernel_vsyscall>: int $0x80
0x110402 <__kernel_vsyscall+2>: ret
0x110403: nop
0x110404: nop
0x110405: nop
(gdb) disassemble 0x110400 //disassemble codes from this address
Dump of assembler code for function __kernel_vsyscall:
0x00110400 <__kernel_vsyscall+0>: int $0x80
0x00110402 <__kernel_vsyscall+2>: ret
End of assembler dump.
Besides this way, we can use ‘disassemble’ directly by default, then, it disassemble the codes surrounding the pc of the selected frame, as below. And this is the most direct way.
(gdb) help disassemble
Disassemble a specified section of memory.
Default is the function surrounding the pc of the selected frame.
With a single argument, the function surrounding that address is dumped.
Two arguments are taken as a range of memory to dump.
(gdb) disassemble
Dump of assembler code for function __kernel_vsyscall:
0x00110400 <__kernel_vsyscall+0>: int $0x80 //system call by ‘operating system trap’
0x00110402 <__kernel_vsyscall+2>: ret
End of assembler dump.
At last, from the disassemble code, we can see that this system call is done by interrupt with ‘operating system trap’ ‘int $0x80’.(通过操作系统陷入进行系统调用，对本例来讲，通过0x80软中断实现系统调用). The system call service handler sys_futex is declared as follows.
asmlinkage long sys_futex(u32 __user *uaddr, int op, u32 val, struct timespec __user *utime, u32 __user *uaddr2, u32 val3)
then, from registers of frame 0, we can determine some parameters of this system call like the following.
(gdb) frame 0 //enter frame 0
#0 0x00110402 in __kernel_vsyscall ()
(gdb) info registers
eax 0xfffffe00 -512
ecx 0x80 128 //the second parameter op=128
edx 0x3 3
ebx 0x9d42678 164898424
esp 0xbfca1b1c 0xbfca1b1c
ebp 0xbfca1b68 0xbfca1b68
esi 0x0 0 //the third parameter utime=NULL
edi 0x3 3
eip 0x110402 0x110402 <__kernel_vsyscall+2> //this instruction of the address will be executed
eflags 0x200206 [ PF IF ID ]
cs 0x73 115
ss 0x7b 123
ds 0x7b 123
es 0x7b 123
fs 0x0 0
gs 0x33 51
How to know these parameters will be discussed below.
Note, each time you run this program, only %ebx, %esp, %ebp are different, other registers’ value are all the same.
3.2 system call analysis
Now, we disassemble code at 0x007065d5, that is, pthread_cond_wait@@GLIBC_2.3.2, as follows.
(gdb) disassemble 0x007065d5
Dump of assembler code for function pthread_cond_wait@@GLIBC_2.3.2:
...
0x007065ab <pthread_cond_wait@@GLIBC_2.3.2+107>: call 0x709060 <__pthread_enable_asynccancel>
0x007065b0 <pthread_cond_wait@@GLIBC_2.3.2+112>: mov %eax,(%esp) //%eax=0, %ebx=0x862c63c, %ecx=0
0x007065b3 <pthread_cond_wait@@GLIBC_2.3.2+115>: cmpl $0xffffffff,0x20(%ebx)
0x007065b7 <pthread_cond_wait@@GLIBC_2.3.2+119>: sete %cl
0x007065ba <pthread_cond_wait@@GLIBC_2.3.2+122>: sub $0x1,%ecx //%ecx=-1, that is, 0xffffffff
0x007065bd <pthread_cond_wait@@GLIBC_2.3.2+125>: and %gs:0x20,%ecx //%ecx=0x80, the soft interrupt
0x007065c4 <pthread_cond_wait@@GLIBC_2.3.2+132>: mov %edi,%edx //%edi=1, then %edx=1
0x007065c6 <pthread_cond_wait@@GLIBC_2.3.2+134>: add $0x4,%ebx //%ebx=0x862c640
0x007065c9 <pthread_cond_wait@@GLIBC_2.3.2+137>: mov $0xf0,%eax //%eax=0xf0=240=SYS_futex, the system call number
0x007065ce <pthread_cond_wait@@GLIBC_2.3.2+142>: call *%gs:0x10 //jump to 0x00110400 ‘int $0x80’, use stepi to see it
0x007065d5 <pthread_cond_wait@@GLIBC_2.3.2+149>: sub $0x4,%ebx //the return address of call the system-call-handler
0x007065d8 <pthread_cond_wait@@GLIBC_2.3.2+152>: mov (%esp),%eax
0x007065db <pthread_cond_wait@@GLIBC_2.3.2+155>: call 0x709020 <__pthread_disable_asynccancel>
...
Contrast with the source code of glibc, we can find the system call is done at 0x007065ce. Wehre, %eax=0sf0=240 is the system call number.
/* Enable asynchronous cancellation. Required by the standard. */
cbuffer.oldtype = __pthread_enable_asynccancel ();
/* Wait until woken by signal or broadcast. */
lll_futex_wait (&cond->__data.__futex, futex_val, pshared);
/* Disable asynchronous cancellation. */
__pthread_disable_asynccancel (cbuffer.oldtype);
(gdb) frame 1
#1 0x007065d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
(gdb) info registers
eax 0xfffffe00 -512
ecx 0x80 128
edx 0x1 1
ebx 0x93b7678 154891896
esp 0xbfd55c90 0xbfd55c90
ebp 0xbfd55cd8 0xbfd55cd8
esi 0x0 0
edi 0x1 1
eip 0x7065d5 0x7065d5 <pthread_cond_wait@@GLIBC_2.3.2+149> //this instruction will be executed
eflags 0x200206 [ PF IF ID ]
cs 0x73 115
ss 0x7b 123
ds 0x7b 123
es 0x7b 123
fs 0x0 0
gs 0x33 51
if want to get all parameters passed to system call service handler (系统调用服务程序), it is necessary to arrive at the codes before call ‘int $0x80’. Then, set a breakpoint at 0x007065ce, and run to this breadpoint to watch all registers.
(gdb) b *0x007065ce
Breakpoint r at 0x7065ce
(gdb) r
...
(gdb) info registers
eax 0xf0 240 //the interrupt number SYS_futex=240, is passed by %eax
ecx 0x80 128 //the second parameter op=128
edx 0x1 1 //_val=1
ebx 0x83c9640 138188352 //the first parameter uaddr=0x83c9640
esp 0xb7f74200 0xb7f74200
ebp 0xb7f74248 0xb7f74248
esi 0x0 0 //the third parameter utime=NULL
edi 0x1 1
eip 0x7065ce 0x7065ce <pthread_cond_wait@@GLIBC_2.3.2+142>
eflags 0x200212 [ AF IF ID ]
cs 0x73 115
ss 0x7b 123
ds 0x7b 123
es 0x7b 123
fs 0x0 0
gs 0x33 51
(gdb) stepi
0x00110400 in __kernel_vsyscall () //int $0x80
(gdb) info registers
eax 0xf0 240 //the interrupt number SYS_futex=240, is passed by %eax
ecx 0x80 128 //the second parameter op=128
edx 0x1 1 //_val=1
ebx 0x83c9640 138188352 //the first parameter uaddr=0x83c9640
esp 0xb7f741fc 0xb7f741fc
ebp 0xb7f74248 0xb7f74248
esi 0x0 0 //the third parameter utime=NULL
edi 0x1 1
eip 0x110400 0x110400 <__kernel_vsyscall> //will execute ‘int $0x80’
eflags 0x200212 [ AF IF ID ]
cs 0x73 115
ss 0x7b 123
ds 0x7b 123
es 0x7b 123
fs 0x0 0
gs 0x33 51
(gdb)
About this system call, it will be detailed discussed later again.