The individual owning this blog works for Oracle in Germany. The opinions expressed here are his own, are not necessarily reviewed in advance by anyone but the individual author, and neither Oracle nor any other party necessarily agrees with them.
Saturday, February 18. 2012
It's a few days since the event in Munich, the last one in the series in Germany. The last two weeks were really cool. There were many many people at the Solaris 11 Techdays 2012. That are the moments where i really love my job. Speaking in front of an interested audience about technology. I hope that Zurich will be as great as the events in Germany, but after the last two weeks i have no doubts about it.
That said we had really luck with the dates: A day after the event in Munich snow got a major problem there and as you may know flying to or from FRA is a major pain in the asymmetric photons at the moment due to the apron.
Sunday, January 8. 2012
Buffer Extermination? WTF? Normally i'm seeing wait events like "buffer busy", "log sync" or "db file sequential read" when doing my research in Oracle installation in Top5 events when i'm called because of a situation where the performance is not quite at the level the customer wants. I was sitting in front of the console of an system still using 10g as it's database.
I want to add, that the performance problem had its root somewhere else and was quickly found, however this log wait sparked my interest. A much simpler reason. It was the curiosity afterwards, why there were peaks in the wait event statistics in regular intervals with this wait event i never saw before.
"Buffer exterminate"? WTF ... again. Sounds dangerous. Never saw that before in that list, and than my brain rotated … what the heck is "Buffer Exterminate", i have an idea, something is ringing in my head, but somehow my long-term memory management unit of my brain was unable to stage this information in to current working set. Okay … ask Dr. Google.
Metalink [ID 259137.1] is of great help here. The "buffer exterminate" wait occurs (and can only occur), when the buffer cache is shrunk dynamically and a session wants to access data that is in the granule of the buffer that is chosen by Oracle for removal from the buffer cache. The session wanting the block has to wait until the buffer to be removed has been freed to read it from disk then. You can't simply read the block from disk without waiting, as the block in the granule may represent a new state of the block than the one on the disk an simply reading the one from the disk would yield just outdated data. So you can just wait until the granule has been released.
Before you ask, what a granule is: Oracle doesn't allocate memory in the SGA bytewise, but in so called granules. A granule is 4 MB of memory, when your SGA is up to 1 GB when the instance starts. It's 16 MB when your SGA is larger than 1GB at startup.
In Oracle DB 10g, there is a feature called "Automatic Shared Memory Management". The idea is, that Oracle itself monitors the load and configures the layout of the SGA. I think of automatic means as a very good feature. It's like with manual and automatic gearboxes. Surely, a good driver can accelerate faster with a manual gearbox than with an automatic gearbox, however an automatic gearbox is faster and better than 99% of all drivers. That said, given the existence of behaviour patterns explained by the Dunning-Kruger-effect (h/t to Chris Colomb for hinting me to this interesting phenomenon), 99% of drivers think are part of the 1%. This is especially epidemic in Germany. But back to the issue. It's the same with tuning of systems
You activate the ASMM by setting the parameter
Of course: When you have fixed
This works really good and this relieves the admin from investing time to find good values for some of the most important SGA parameters.
However if your database tries to move memory back and forth from one kind of shared memory to another tens of times per hour this is surely not without impact on your database performance. I had such a situation in this case. The system started to move around memory in minute intervals just to move it back a minute or two later. As most automatic systems they will work perfectly within their specification, but you may hit a situation where tries to get most out of a situation with restricted resources, where the SGA is confronted with the situation that all components want more memory and as soon you remove memory from one parts, the other part cries and wants its memory back. That's similar to the argument with your significant other about what's the half of the blanket. Better have two blankets Or to get back to the topic: Have enough SGA ...
How do you find out, how many resizing operations took place? You can look that up by a select statement as described in this blog:
With this statement you will see the recent history of resizings.
In this case a slight increase (4 gigs) of the target size of the SGA moved the system away from growing and shrinking the buffers back and forth. And not a single "buffer extermination" was seen afterwards and no peaks in the wait time statistics and the number of resizing ops was down to one per hour. And that was more than okay.
Other solutions would be the deactivation of ASMM (by setting
Friday, December 30. 2011
The year almost over. As usual at this time i'm finalizing my tax declaration at the moment. But nevertheless i wanted to write a short article about the last year.
The year started quite bad. In March i had a really bad argument with a good friend and we didn't talked a word since that. Albeit my communication habits could have been better in that days, i assume the writing were on the wall before that day in march and it was just a matter of when, not if, when the argument broke loose.
But starting in June the world just gone downhill for me. It started with the events that finally led to the state the blog currently is. That is really a large thorn sitting in the flesh hurting me all the time. However the consequences were inevitable.
The job had two really great satisfying moments for me with larger successful projects, and a number of others were i was able to help others to find problems. That was really and deeply satisfying moments.
Other things weren't that well as well: Just two main ones … in my vacation i was ill at the weekend and needed almost the complete vacation to get really rid of that nasty bug, and then just 2 days in the job again i broke my ankle and had a torn ankle ligament. While it was painful at least my body seems still to have its good regeneration capabilties in regard of injuries. I don't need painkillers any longer and i'm just wearing the aircast orthotic device at the moment. 24 days is not that bad.
On the good side: My house is almost completed … perhaps 3 month of work in the evening left. So i will have more time to other things soon: Like really pursuing new opportunities - career as well as personal - , finding a new hobby in the year one after almost two years of construction work that rules out any other free time things. Like reading through the heap of books that piled up this year, to get rid of several kilos. Finding a new car in February. Thinking of a nice S60 at the moment. Trying not to break my bones again. And perhaps a longer vacation abroad this year.
I wish you all a good transition into the next year and successful 2012.
Saturday, December 24. 2011
Friday, December 16. 2011
Ankle is really hurting. A big thank you to the pharmaceutical industry for providing pain killers. However i would like to concentrate a little bit better again. However things are getting better day by day ...
Friday, December 16. 2011
Just thought, that our perception of the world is heavily influenced by the representations of it ... like a map. Just to demonstrate, a question: Assume you are a pilot. You are flying from Frankfurt to San Francisco. You are a lazy pilot and you are allowed by air traffic control to set the shortest direct course on the autopilot and don't touch it again. What is the initial heading you are setting? Just answer with the first thought! No degrees needed ... just something like west,east,south our southeast.
Saturday, December 10. 2011
Damned, now i have a lot of time ... because of a mismatch between the real number of residual steps and the perceived number of residual steps on a stairway. Now i'm a proud owner of a fractured ankle join (the doc is not totally sure if it's a ankle ligament injury as well, next x-ray middle of next week will show it). Can't need that right now ...
Saturday, December 3. 2011
The newest revision (the rev valid as of December, 2nd) of the document "Partitioning - Topic: Server/Hardware Partitioning" naming the technologies allowing you just to license a subset of the available processors in a system for a Oracle software has been modified: Oracle VM for SPARC is now explicitly named as a hard partitioning technology.
Sunday, October 30. 2011
Damned axe … is it really that hard to believe that an automatic system in Debian makes a dumb decision based on dumb assumptions (classic BIBO - bullshit in , bullshit out) ? It has nothing to do with a butchered up distribution by a hosting provider or an error between the keyboard and the chair. I would consider it a bug if it would deliver a differen result given the input to the system and the ruleset codified in the system.
Per default an init.d script is inserted by the
This information is used to put the init.d script at the right place of the sequence of startup.
There is a number of scripts just dependent on the existence of the local filesystem. This ones are run with the sequence number 01 thus first.
One of the services started first is syslog, and as you may have recognized in the head of the ssh init.d script, ssh depends on it. So it gets a higher sequence number and thus it's started later:
But wait, what has happened with 02 and 03? They are used by a single service each:
Why do they have this special treatment. It's pretty simple. Both services are flagged as being interactive, as they could ask for some user interaction.
In this cases both might probably ask for a password for the key. They have to start early to be sure that nothing can gets and block the tty in order to enable both scripts to show the password dialog. Keep in mind that we are in a state where no getty is running. However they can't earlier, as both rely on syslog. Essentially they are started as soon as possible and before other scripts with the same dependencies. And thats basically a part af the problem.
And this essentially leads to the situation, that
Another reader hinted me to the fact that lightttpd starts after ssh. At first this is just of cursory interest at a situation where the webserver in question is apache. At second the dependencies are different. At first the init.d script for lighty has no
I won't comment what i'm thinking about such a … a… a… solution ...
Friday, October 7. 2011
Thursday, October 6. 2011
Wednesday, October 5. 2011
Friday, September 23. 2011
Recently I was doing some work in regard of tuning systems. There is something i really hate about this topic of computing: Tuning scripts. You find them on google easily and i find them on systems at customers quite often.
Simply said: I hate them. The reasons for it are simple. For example recently I found a system with a networking tuning script dating back into 2003 or so. The problem: It was meant to increase some of the settings. However many of them were already higher in the default config of current Solaris 10 versions, thus the tuning script essentially reduced the parameters and thus reduced the performance.
Futhermore: Tuning is a lot about understanding things. Understanding how things work together. On a systemic, on an architectural level. How an application loads all the rest of components. Just dropping a script downloaded from a website found by Google - into /etc/init.d is not about understanding things.
You have to carefully consider each change from the default about the impacts. You have to check each setting, if the setting hasn’t already overtaken by the years. You have to recheck it it with every major update of your environment. You have to recheck it with each new technology you are using in your system. Network tuning scripts dating back to a time when 100 MB/s were normal and 1 GB/s are fast aren’t necessarily up to the task in a time when 10 GBit/s are fast and Infiniband IPoIB networks deliver even more. You had to turn different knobs in a time, when cpu time was precious. You’ve tuned for minium cpu utilization. CPU isn’t a large factor today, you tune for minimize latency or maximize throughput. You have to know what you want to aim for, because minimum latency and maximum throughput are often mutually exclusive. Do you want an extreme or a target in between.
Just using a script to tune something doesn’t lead you through all the thought to make really good tuning decisions.
There are some basic rules from my point of view:
Tuesday, September 20. 2011
A commentator at hackernews asked how i think about -9. In my opinion: It's widespread use is a similar plaque like the –f switch. And this is pretty easy to explain (I'm simplifying things a bit).
-9 is a shorthand for SIGKILL. When you send a SIGKILL to a process, the process is terminated immediately. You can’t catch this signal, you can’t ignore it. A kill with -9 sends this SIGKILL to a process.
A kill without -9 sends a SIGTERM to process. It terminates the process like SIGKILL. However a process is allowed to catch it in order to execute a signal handler … or just ignores to ignore it. A signal handler is nothing more than a code path that is executed when the process receives a signal.
So when you kill a process with a normal kill you give the process the chance to clean up behind itself, to make files consistent, to roll back changes in the case the process isn’t using some transactional mechanisms when changing data, to delete temporary files … and so on ...
It's a good style to write such signal handlers and in many programming languages it's pretty easy. For example in perl:
When you send a -9 to a process you take away this chance from the process. It’s killed instantly … even if it just started to modify your files, fscking up your data in order to put it in a new form, even when you have created dozens of temporary files filling up /tmp. Things like that …
Killing a process with -9 is the last possibility. However I see people using it too often too early. A second after the normal kill is send a pgrep on the process follows. Still there and the sword of -9 is falling down.
When a process doesn’t disappear immediately after sending the SIGTERM, it may be just busy to follow your order of terminating itself and is cleaning up things. When your application is dependent to precious resources at cleaning up (for example IOPS on your rotating rust) the process of cleaning up may take a while.
The implicit question in any process, that doesn't react to a normal kill via SIGTERM is the question why it doesn't react to the signal. Just sending a -9 when a normal kill didn't worked is like "Do not care". Monitoring the process with truss or strace what the heck the process is doing after getting the SIGTERM is a good first step. Perhaps you see some cleanup work and know that you just have to wait a little bit longer. Writing a core dump of the process with
And then … and only then … a kill -9 may be feasible.
Sunday, August 21. 2011
Yesterday i held my deduplication talk at the Froscon 2011. I think it was acceptable and the lecture room 3 was really filled. To be honest: I don't expected such an audience at that time. So a big thank you for all who attended the talk.
My talk started at 10:00 o'clock and thanks to apron parking, a rental car pickup in the feeled middle of nowhere in Düsseldorf and a larger traffic jam near Cologne led to an arrival at 09:58 … with a presentation at 10:00 and the urgent need to visit the restroom before a disaster happens.
I had just a short time at Froscon … i had a date with my house, that still needs work and don't accept when i'm saying "No … not this weekend". And thus i was back in the rental car at 11:45 and was back in Lueneburg shortly after 15:00, ten hours after leaving home. 3.30h for 430km … not that bad for an Opel Corsa.
However there is something i've recognized: I'm really missing standing in front of customers and technically interested people and trying to transfer my enthusiasm for technology. I'm really missing being on the road.
The LKSF book
The book with the consolidated Less known Solaris Tutorials is available for download here
Martin about End of c0t0d0s0.org
Mon, 01.05.2017 11:21
Thank you for many interesting blog posts. Good luck with al l new endeavours!
Hosam about End of c0t0d0s0.org
Mon, 01.05.2017 08:58
Joerg Moellenkamp about tar -x and NFS - or: The devil in the details
Fri, 28.04.2017 13:47
At least with ZFS this isn't c orrect. A rmdir for example do esn't trigger a zil_commit, as long as you don't speci [...]
Thu, 27.04.2017 22:31
You say: "The following dat a modifying procedures are syn chronous: WRITE (with stable f lag set to FILE_SYNC), C [...]