ZFS Deduplication features in PSARC

Quicksearch

Disclaimer

The individual owning this blog works for Oracle in Germany. The opinions expressed here are his own, are not necessarily reviewed in advance by anyone but the individual author, and neither Oracle nor any other party necessarily agrees with them.

Navigation

< About that data loss at Microsoft/Danger | 6. HHOSUG Meeting "Hochverfügbarkeits-Cluster" >

ZFS Deduplication features in PSARC

Tuesday, October 20. 2009

There are two really interesting features for Solaris in the PSARC process. The first feature is the PSARC case "2009/571 - ZFS Deduplication Properties". The description of the properties gives some interesting insight into the feature:

the mechanism is based on checksums to find duplicates
selectable checksum mechanism
you can switch off the verify of deduplication - to check if a duplicate is really a duplicate and not just a hash collision - when using a strong checksum mechanism like sha256
There is a configurable number of blocks that gets deduplicated into a single block before keeping a second copy. This should protect you from the situation that you may lose 1000 files if you lose the block that is used as the "master block" for the deduplication. The default setting is to have a second block as the data source when more than 100 blocks were deduplicated to a single block.

Another interesting PSARC is the "2009/557 - ZFS send dedup" case. This case describes the changes to the stream generated by zfs send to support deduplication on this stream of data. Due to this PSARC blocks just have to be transmitted once in a stream. Interesting point: You can use stream deduplication to reduce transfer time without using deduplication on disks and the feature uses the already computed checksums of the data on disk for deduplication when the choosen mechanism is cryptographically strong enough.

Posted by Joerg Moellenkamp in English, Solaris, Sun/Oracle at 18:06 | Comments (10) | Trackback (1)
View as PDF: This entry | This month | Full blog

Trackbacks

Trackback specific URI for this entry

Deduplication integrated to
I've reported about the PSARC cases a few days ago. Now the code for deduplication in ZFS has been integrated into Opensolaris as stated by this mail on onnv-notify: "PSARC 2009/571 ZFS Deduplication Properties, 6677093 zfs should have dedup capability".

Weblog: c0t0d0s0.org
Tracked: Nov 02, 12:31

Comments

Display comments as (Linear | Threaded)

Is this on a block or file level? Block level (or chunks) work better for VM Images but are of course more overhead. Is this an async deduplication or for each write? is it only performed on new files or also on updates?

#1 Bernd Eckenfels (Homepage) on 2009-10-20 19:50 (Reply)

Block level. And there is not much overhead, as the checksums are already there. I think we will see the exact features soon as such PSARC cases are a good sign, that some changes to the code are imminent.

#1.1 Jörg+M. (Homepage) on 2009-10-20 22:41 (Reply)

What's PSARC?

#2 Maciek (Homepage) on 2009-10-20 20:28 (Reply)

PSARC - platform software architecture review comitee ... to say it a little bit simplified: the architectural steering comitee for the OS ...

#2.1 Joerg M. (Homepage) on 2009-10-20 22:07 (Reply)

So it (PSARC working on this issue) means, that this feature is already designed/implemented and it will be included in forhcoming updates for Storage 7000 firmware / Opensolaris Nevada distribution? Cool!

#2.1.1 Maciek on 2009-10-20 23:19 (Reply)

The PSARC doesn't work on an issue. Developers work on it. Before you can integrate it into the code base, you have to go through the PSARC, discussing the feature, the architecture and the possible impact to other parts of Solaris.

And yes, when a PSARC is approved, you should see the feature sooner or later in Opensolaris (of course, when there aren't unforeseen implementation difficulties that prevents an implementation without harming the rest of the system). From my observation, features introduced by the S7000 people are sooner than later in Solaris.

#2.1.1.1 Joerg Moellenkamp (Homepage) on 2009-10-23 21:12 (Reply)

http://www.opensolaris.org/os/community/arc/arc-faq/sun-arc-charter/

#3 Frank on 2009-10-20 21:08 (Reply)

I always thought the extra block should come sooner rather than later, as in don't activate dedup untill you have the 3rd user of a block then you know you will always have 2 good copies. If the data is common enough to appear more than once.

#4 James Dickens (Homepage) on 2009-10-21 01:17 (Reply)

1. It should be possible to yield such a behavior by configuring a really low dittolimit.

2. On the other side, you have already RAID1 for example with two versions of the blocks and with checksums to find the correct one. I think the situation is a little bit diferent than at other filesystems.

#4.1 Jörg+M. (Homepage) on 2009-10-21 06:48 (Reply)

Very cool.

According to this video: http://www.youtube.com/watch?v=Fski5cOKGEs,

Jeff Bonwick said that the deduplication table will be stored in flash(L2ARC). 1PB storage would take 1TB of flash for the dedupliation table. Now my question is this: if L2ARC isn't persistent, then upon reboot, it would take a long time to read the deduplication table into L2ARC again for any new writes that are deduped.

So technically, on a deduped large pool, deduplication depends on L2ARC persistence, which depends on ZFS crypto. How is ZFS crypto and L2ARC persistence going? Any new info?

#5 TS on 2009-10-21 05:07 (Reply)

Add Comment

Name
Email
Homepage
In reply to
Comment	E-Mail addresses will not be displayed and will only be used for E-Mail notifications. To prevent automated Bots from commentspamming, please enter the string you see in the image below in the appropriate input box. Your comment will only be submitted if the strings match. Please ensure that your browser supports and accepts cookies, or your comment cannot be verified correctly. Enter the string from the spam-prevention image above: Standard emoticons like :-) and ;-) are converted to images. Enclosing asterisks marks text as bold (word), underscore are made via _word_.
	Remember Information? Subscribe to this entry