c0t0d0s0.org

Looks like Perl will have integrated dtrace support soon. The Install file of the Release Candidate 1 for Perl 5.10.1 states:

On platforms where DTrace is available, it may be enabled by using the -Dusedtrace option to Configure. DTrace probes are available for subroutine entry (sub-entry) and subroutine exit (sub-exit). Here's a simple D script that uses them:

Peter Galvin publishes an column in the ;login: magazine - a publication of the USENIX and often an really insightful read. This time he wrote about the Storage Analytics part of our new storage appliances - Galvin.info: My latest column at ;login:

Overall, the use of the analytics component of the Sun Storage 7000 NAS appliance is amazing, allowing exploration of every aspect of the performance, load, and status of the system. [...] On the whole I’m glad to leave behind the Middle Ages of performance analysis and look forward to the new tools of the modern age

The destructive mode of DTrace is a really interesting way to do cool things - for example triggering a ZFS snapshot whenever you an application uses the rm. Brian Nitz describes such a dtrace script in his article Anti-destruction destructive dtrace script and y2k9. And Bernd Finger describes, how you can keep short-living processes alive for a deeper analysis by intercepting the exit calls.

When you really thing about harddisks, you could get the feeling, that this subsystem is a really delicate one. You read data from disk spinning at up to 15.000 rpm, trying to catch tracks as narrow as 50 nm, and you don´t read the data from the hard disk, you read a waveform and compute the most probable data pattern out of it. All together it´s a little bit of an suffciently advanced technology indistinguishable from magic.But at the end the system "harddisk" is really susceptible for perturbations from outside.

Brendan Gregg was able to measure the latency impact of sonic waves hitting the casing of hard disks - with a low-tech approach as he wrote in Unusual disk latency.



I don´t want to know how he found out about this effect ... perhaps a debugging session or a harddisk falling on his feet. Nevertheless it´s an impressive usecase for the Analytics feature of our new 7000 series storage. Try to find such effects with the tools from our beloved competitors. Their products are susceptible for "Brendans war scream" as well (as they use hard disks, too) ... but you can´t measure it.

At the moment tcpdump, snoop and others are the predominant tool to analyse network traffic and network problems on a host. But with the IP provider Dtrace makes the first steps to offer a tool for this class of problems as well. It found it´s way into Solaris with nv93. You should give it try and and it´s only one of the first steps ... more to come.

Ben Rockwood shows in the blog of his company, how dtrace can help you to find bottlenecks in a fast an efficient manner: DTrace, MySQL, Ganglia, and Digging for Solutions.

I found two really interesting interesting presentation. At first Brent Paulson wrote an really interesting presentation about practical Opensolaris Security. It´s a good overview about the possibilities of Solaris to increase the level of security at your installation. Ben Rockwood of cuddletech.com wrote an good preso about the usage of dtrace in conjunction with mysql.

On the foundations of Dtrace the security specialists Tiller Beauchamp and David Weston created a tool for finding vulnerabilties in your code (or writting exploits, depends on your objectives). It uses the mechanisms of DTrace and extends them. As Internetnews writes in Blackhat: Dtrace a Rootkit:.

Sun's Dtrace application was developed primarily as a tool to help monitor functions on Solaris. According to a pair of security researchers at the Black Hat conference, you can also use Dtrace as the basis for a rootkit-like tool for offensive and defensive security operations.

Finding vulnerabilties and writting exploits are different sides of a special case of debugging code. And debugging is exactly the job, we´ve developed dtrace for. It was really obvious, that someone will use dtrace for such an usecase.

Adam Leventhal made an interesting observation while using Dtrace on Mac OS X and wrote about his findings in"Mac OS X and the missing probes". According to this articles, Apple do not allow the dtrace-ing of all processes. Whenever a process has the flag "P_LNOATTACH" you can´t look into it with dtrace.

Perhaps, they are in fear of disabling or examination of certain DRM functions for example. It leaves a very bad aftertaste. But as Darran Moffat wrote in the comments: "Now we just need a port of mdb so that the proc flags can be changed ."

After Solaris, NetBSD and MacOS there is a fourth operating capable of Dtrace: QNX Neutrino, a real time operating system with vast similarities to unix. In I trace, you trace … what about dtrace? a first prototype implementation of dtrace in QNX was announced by the both developers of the port.

My coworker Dirk Wriedt had some problems with an application at a customer site. Dtrace told him soon, that the application used some libary call in an incorrect way. Normally you would wait some days for a patch for the application. But DTrace can do more for you: dtrace in destructive mode: working around a non-working syscall. Well, it´s certainly a nasty hack from the "don´t do this at home"-department, but ... well ... it seemed to work and it saved the progress of the project until the patch was available.

John Rice describes one of the more advanced techniques with dtrace: DTrace: digging into exec'd comand line args. Really usefull, when you try to find out, what the heck this %§$"$"§ ... fscking program executes and why it doesn´t run.

You should read the blog entry of to read the perception of one of the DTrace developers regarding the relationship of DTrace and Systemtap. But i really think the whole issue opens up the view to a different problem inside of Linux: The "not invented here (NIH)" problem. Linux isn´t immune to that problem.

Andrew Leventhal and Brendan Gregg gave an interesting presentation how DTrace can help you to find nasty performance problems in your software (and how dtrace helped Twitter to find performance problems).

There is an additional provider for monitoring your system: The DTrace provider for iSCSI.You will find a nice bunch of D scripts at the Solaris Internals Wiki, ranging from iSCSI snoop to measuring latencies in iSCSI operations