How to use mod_dtrace to use DTrace with Apache on FreeBSD


June 2013.
Prefetch Technologies' Apache DTrace module (http://prefetch.net/projects/apache_modtrace/index.html) can be compiled and used under FreeBSD to efficiently monitor an running Apache instance.

System used:

FreeBSD hostmaster1.example.com 9.1-STABLE FreeBSD 9.1-STABLE #0 r248486: Thu Jun 13 00:23:59 CEST 2013 root@hostmaster1.example.com:/usr/obj/usr/src/sys/HOSTMASTER1 amd64

Fetch the module



fetch http://prefetch.net/projects/apache_modtrace/mod_dtrace.c
fetch http://prefetch.net/projects/apache_modtrace/apache.d

Compile and install the module


Make sure you have apache's compiler installed (should be installed with the standard apache22 port).

apxs -c -o mod_dtrace.so mod_dtrace.c && rm .libs/mod_dtrace.so
/usr/sbin/dtrace -G -o apache.o -s apache.d .libs/mod_dtrace.o
gcc -fPIC -shared -lelf -o .libs/mod_dtrace.so .libs/mod_dtrace.o apache.o

Copy the module:

cp .libs/mod_dtrace.so /usr/local/libexec/apache22/

Enable in your usual Apache configuration file (usually /usr/local/etc/apache22/httpd.conf).

LoadModule dtrace_module libexec/apache22/mod_dtrace.so

Restart Apache and check that it is loaded.

httpd -M | grep dtrace
Syntax OK
dtrace_module (shared)

List probes.

dtrace -l | grep apache
51871 apache61601 mod_dtrace.so apache_accept_connection accept-connection
51872 apache61601 mod_dtrace.so apache_check_access check-access
51873 apache61601 mod_dtrace.so apache_check_authorization check-authorization
51874 apache61601 mod_dtrace.so apache_check_user check-user-credentials
51875 apache61601 mod_dtrace.so apache_create_child create-child
51876 apache61601 mod_dtrace.so apache_log_request log-request
51877 apache61601 mod_dtrace.so apache_receive_request receive-request
51878 apache61600 mod_dtrace.so apache_accept_connection accept-connection
...

Use the module


Unfortunately, the scripts provided by Prefetch Technologies can't be used directly since they were created for Solaris 32bit. If you use a 64bit FreeBSD, you'll have to read the Apache and APR header files to find the location of the relevant fields in the different structures (struct request_rec in httpd.h for example).

Sample script:

#!/usr/sbin/dtrace -s

dtrace:::BEGIN
{
printf("%42s %-5s", "Request", "Status");
}

::apache_log_request:log-request
{
this->the_request = copyinstr(*(uintptr_t *)copyin(arg0 + 48,sizeof(uintptr_t)));
/* this->protocol = copyinstr(*(uintptr_t *)copyin(arg0 + 72,sizeof(uintptr_t))); */
/* this->status_line = copyinstr(*(uintptr_t *)copyin(arg0 + 104,sizeof(uintptr_t))); */
this->responsecode = (int)*(uintptr_t *)copyin(arg0 + 112,sizeof(int));
printf("%42s %5d\n", this->the_request, this->responsecode);
}

Output:

CPU ID FUNCTION:NAME
0 1 :BEGIN Request Status
0 51876 apache_log_request:log-request GET / HTTP/1.1 200
0 51876 apache_log_request:log-request GET / HTTP/1.1 200
0 51904 apache_log_request:log-request GET /does_not_exist HTTP/1.1 404

Problems I got



dtrace: failed to compile script apache.d: "/usr/lib/dtrace/psinfo.d", line 13: operator -> cannot be applied to a forward declaration: no struct devstat definition is available

I was using the dtrace command within a jail. Using it outside solved the problem.


dtrace: failed to compile script apache.d: "/usr/lib/dtrace/regs_x86.d", line 2: type redeclared: struct devinfo

I don't know what the problem was. I just moved /usr/lib/dtrace/io.d away temporarily and it worked.