{"id":1529,"date":"2011-09-12T22:06:19","date_gmt":"2011-09-12T14:06:19","guid":{"rendered":"http:\/\/blog.pighead.cc\/whsyu\/?p=1529"},"modified":"2011-09-13T13:42:22","modified_gmt":"2011-09-13T05:42:22","slug":"zfsv28-dedup","status":"publish","type":"post","link":"https:\/\/blog.pighead.cc\/whsyu\/2011\/09\/12\/zfsv28-dedup\/","title":{"rendered":"zfsv28 dedup"},"content":{"rendered":"

[\u83ef\u8a9e, cmn-Hant-TW] <\/strong><\/p>

\u5728\u516d\u6708\u4e2d\u5de6\u53f3 FreeBSD 8.2-STABLE \u7684 zfs \u5347\u7d1a\u5230\u4e86 v28<\/h5>\n

\u9019\u7248\u6700\u4e3b\u8981\u7684\u7279\u8272\u662f dedup\uff0c\u4e5f\u5c31\u662f block level \u7684\u00a0deduplication\uff0c\u50b3\u8aaa\u4e2d\u7684 virtual machine disk image \u5099\u4efd\u6551\u661f (\u8aa4)\u3002 \u56e0\u70ba\u662f block level\uff0c\u6240\u4ee5\u6a94\u6848\u4f4d\u7f6e\u600e\u9ebc\u653e\uff0c\u6a94\u6848\u662f\u4e0d\u662f\u53ea\u4fee\u6539\u4e00\u9ede\u9ede\u9019\u4e9b\u901a\u901a\u4e0d\u7528\u7ba1\u4e86\uff0c\u53cd\u6b63 zfs \u89ba\u5f97\u9019\u5169\u500b block \u662f\u9577\u5f97\u4e00\u6a23\u7684\u8a71\uff0c\u5c31\u81ea\u52d5\u6703\u8655\u7406\uff0c\u5982\u679c\u786c\u789f\u88e1\u9762\u6709\u4e00\u5806\u5f7c\u6b64\u9593\u5dee\u7570\u4e0d\u5927\u7684\u5927\u6a94\u6848\uff0c\u90a3\u6548\u679c\u5c31\u6703\u975e\u5e38\u597d\uff0c\u9060\u6bd4 lzma \u6216\u662f gzip \u58d3\u7e2e\u66f4\u6709\u5feb\u611f (\u4e8b\u5be6\u4e0a dedup \u8ddf\u58d3\u7e2e\u662f\u53ef\u4ee5\u4f75\u5b58\u7684\uff0c\u6b61\u8fce\u6311\u6230\u786c\u9ad4\u6975\u9650) \u3002<\/p>\n

\u8981\u958b dedup \u5f88\u7c21\u55ae\uff0c\u5728\u6709 zfsv28 \u7684\u7cfb\u7d71\u4e0b zfs set dedup=on filesystem \u6307\u4ee4\u5c31\u53ef\u4ee5\u6253\u958b\uff0c\u8981\u6ce8\u610f\u7684\u662f dedup \u662f\u5f9e\u6574\u500b pool \u4f86\u770b\u7684\uff0c\u540c\u4e00\u500b pool \u4e0b\u53ef\u4ee5\u6709\u4e9b filesystem \u6709\u958b\uff0c\u6709\u4e9b\u6c92\u6709\uff0c\u7b97\u662f\u76f8\u7576\u6709\u5f48\u6027\u3002dedup \u7684\u6548\u679c\u53ef\u4ee5\u7c21\u55ae\u7684\u7528 zpool list \u770b\u51fa\uff0cv28 \u591a\u4e86\u4e00\u683c DEDUP\uff0c\u90a3\u500b\u7684\u500d\u7387\u5c31\u662f dedup \u7684\u6548\u679c\u3002<\/p>\n

NAME         SIZE  ALLOC   FREE    CAP  DEDUP  HEALTH  ALTROOT\r\nbackup       696G   445G   251G    63%  1.80x  ONLINE  -<\/pre>\n
\u5728\u6709\u958b dedup \u7684\u5730\u65b9\uff0c\u6bcf\u4e00\u500b block \u7684 checksum \u6703\u5b58\u5728\u4e00\u500b\u53eb DDT \u7684\u8868\u88e1\u9762\uff0c\u6839\u64da dedup FAQ<\/a>\u00a0\u7684\u8b1b\u6cd5\uff0c\u4e00\u822c record size \u00a0128K \u7684\u60c5\u6cc1\u4e0b\uff0c20TB \u7684\u8cc7\u6599\u6703\u9700\u8981 32 GB \u7684\u5be6\u9ad4\u8a18\u61b6\u9ad4\u4f86\u653e DDT\uff0c\u5982\u679c\u6c92\u90a3\u9ebc\u5927\u7684\u8a18\u8b69\u9ad4\u90a3\u5c31\u6703\u5403\u5230\u786c\u789f\uff0c\u90a3\u5c31\u6703\u975e\u5e38\u975e\u5e38\u6162\u3002\u6298\u8877\u65b9\u6848\u5c31\u662f\u627e\u4e00\u9846 SSD \u7576 L2ARC\uff0cDDT \u4e3b\u8981\u662f\u8b80\u53d6 (\u56e0\u70ba\u8981\u6bd4\u5c0d)\uff0cSSD \u7684\u8b80\u53d6\u6548\u80fd\u9060\u6bd4\u786c\u789f\u4f86\u5f97\u597d\uff0c\u9019\u500b\u5de5\u4f5c\u9084\u7b97\u625b\u5f97\u8d77\u4f86\u3002<\/p>\n
\u8a18\u61b6\u9ad4\u4e0d\u8db3\u786c\u4e0a dedup \u6703\u5c0d\u6548\u80fd\u6709\u4ec0\u9ebc\u885d\u64ca\u5462\uff0c\u7c21\u55ae\u7684\u6e2c\u8a66<\/h5>\n
( 2G \u8a18\u61b6\u9ad4\uff0c640G \u7684\u786c\u789f\u7a7a\u9593\uff0czfs compression = gzip; atime = off)
\n<\/span>\u6e2c\u8a66\u6a94\u6848\u70ba\u00a0boost_1_45_0.tar.bz2 \u5927\u5c0f\u7d04 38MB\uff0c\u89e3\u58d3\u7e2e\u5f8c\u7d04275MB\uff0c\u7528 FreeBSD \u5167\u5efa\u7684 tar \u4f86\u89e3\u58d3\uff0c\u7136\u5f8c\u7528 rm \u522a\u6389\uff0c\u540c\u6a23\u7528\u7cfb\u7d71\u7684 time \u4f86\u6e2c\u6642\u9593<\/p>\n
DEDUP on:<\/h6>\n
tar 19.063u 6.914s 1:03:02.62<\/strong>  0.6%   88+1735k 0+0io 0pf+0w\r\nrm   0.171u 8.741s 1:12:41.98<\/strong>  0.2%   26+2530k 0+0io 0pf+0w<\/pre>\n
DEDUP off:<\/h6>\n
tar 19.372u 6.963s    1:05.41<\/strong> 40.2%   74+1464k 0+0io 0pf+0w\r\nrm   0.029u 3.826s    0:08.04<\/strong> 47.7%   16+1528k 0+0io 0pf+0w<\/pre>\n
\u9019\u662f\u6700\u6158\u70c8\u7684\u60c5\u6cc1 (\u516b\u6210\u662f\u6574\u500bDDT\u90fd\u5728\u786c\u789f\u4e0a\u4e86)\uff0cDEDUP on \u4e00\u822c\u60c5\u6cc1\u5176\u5be6\u6c92\u90a3\u9ebc\u6162\uff0c\u5927\u7d04\u6578\u5341\u5206\u9418\u5427\uff0c\u5982\u679c\u786c\u789f\u88e1\u9762\u5df2\u7d93\u6709\u76f8\u540c\u7684\u8cc7\u6599\u7684\u8a71\u90a3\u9084\u6703\u518d\u66f4\u5feb\u4e9b\uff0c\u4e0d\u904e\u518d\u5feb\u4e5f\u662f\u88ab DEDUP off \u00a0\u6642\u7684\u901f\u5ea6\u6158\u96fb\u5c31\u662f\u3002\u5f9e\u5be6\u969b\u61c9\u7528\u4f86\u770b\uff0c\u89e3\u58d3\u7e2e\u4e00\u500b\u6578\u5341MB \u7684\u6a94\u6848\u8981\u82b1\u6578\u5341\u5206\u9418\u5230\u4e00\u5c0f\u6642\uff0c\u780d\u6389\u89e3\u58d3\u7e2e\u51fa\u4f86\u7684\u6771\u897f\u9084\u8981\u518d\u82b1\u6578\u5341\u5206\u9418\u5230\u4e00\u500b\u5c0f\u6642\u662f\u4ec0\u9ebc\u60c5\u6cc1\uff1f \u53ea\u80fd\u8aaa\u786c\u9ad4\u4e0d\u5920\u529b\u7684\u8a71\uff0c\u9084\u662f\u4e0d\u8981\u4e82\u958b DEDUP \u5427\u3002 (\u4e0d\u904e\u4e00\u60f3\u5230 DEDUP \u7684\u5feb\u611f\u2026 \u518d\u52a0\u4e0a RAM \u73fe\u5728\u4fbf\u5b9c\u7684\u8ddf\u25cf\u25cf\u4e00\u6a23\uff0c\u597d\u50cf\u5347\u7d1a\u4e00\u4e0b\u518d\u4f86\u958b\u4e5f\u662f\u4e0d\u932f\u7684\u9078\u64c7)<\/p>\n","protected":false},"excerpt":{"rendered":"
[\u83ef\u8a9e, cmn-Hant-TW] \u5728\u516d\u6708\u4e2d\u5de6\u53f3 FreeBSD 8.2-STABLE \u7684 zfs \u5347\u7d1a\u5230\u4e86 v28 \u9019\u7248\u6700\u4e3b\u8981\u7684\u7279\u8272\u662f dedup\uff0c\u4e5f\u5c31\u662f block level \u7684\u00a0deduplication\uff0c\u50b3\u8aaa\u4e2d\u7684 virtual machine disk image \u5099\u4efd\u6551\u661f (\u8aa4)\u3002 \u56e0\u70ba\u662f block level\uff0c\u6240\u4ee5\u6a94\u6848\u4f4d\u7f6e\u600e\u9ebc\u653e\uff0c\u6a94\u6848\u662f\u4e0d\u662f\u53ea\u4fee\u6539\u4e00\u9ede\u9ede\u9019\u4e9b\u901a\u901a\u4e0d\u7528\u7ba1\u4e86\uff0c\u53cd\u6b63 zfs \u89ba\u5f97\u9019\u5169\u500b block \u662f\u9577\u5f97\u4e00\u6a23\u7684\u8a71\uff0c\u5c31\u81ea\u52d5\u6703\u8655\u7406\uff0c\u5982\u679c\u786c\u789f\u88e1\u9762\u6709\u4e00\u5806\u5f7c\u6b64\u9593\u5dee\u7570\u4e0d\u5927\u7684\u5927\u6a94\u6848\uff0c\u90a3\u6548\u679c\u5c31\u6703\u975e\u5e38\u597d\uff0c\u9060\u6bd4 lzma \u6216\u662f gzip \u58d3\u7e2e\u66f4\u6709\u5feb\u611f (\u4e8b\u5be6\u4e0a dedup \u8ddf\u58d3\u7e2e\u662f\u53ef\u4ee5\u4f75\u5b58\u7684\uff0c\u6b61\u8fce\u6311\u6230\u786c\u9ad4\u6975\u9650) \u3002 \u8981\u958b dedup \u5f88\u7c21\u55ae\uff0c\u5728\u6709 zfsv28 \u7684\u7cfb\u7d71\u4e0b… <\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[8],"tags":[4503,4591,107],"class_list":["post-1529","post","type-post","status-publish","format-standard","hentry","category-freebsd","tag-dedup","tag-freebsd","tag-zfs"],"_links":{"self":[{"href":"https:\/\/blog.pighead.cc\/whsyu\/wp-json\/wp\/v2\/posts\/1529","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.pighead.cc\/whsyu\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.pighead.cc\/whsyu\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.pighead.cc\/whsyu\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.pighead.cc\/whsyu\/wp-json\/wp\/v2\/comments?post=1529"}],"version-history":[{"count":14,"href":"https:\/\/blog.pighead.cc\/whsyu\/wp-json\/wp\/v2\/posts\/1529\/revisions"}],"predecessor-version":[{"id":1541,"href":"https:\/\/blog.pighead.cc\/whsyu\/wp-json\/wp\/v2\/posts\/1529\/revisions\/1541"}],"wp:attachment":[{"href":"https:\/\/blog.pighead.cc\/whsyu\/wp-json\/wp\/v2\/media?parent=1529"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.pighead.cc\/whsyu\/wp-json\/wp\/v2\/categories?post=1529"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.pighead.cc\/whsyu\/wp-json\/wp\/v2\/tags?post=1529"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}