I Can't Believe This is Butter! A tour of btrfs. - Avi Miller

##########################################
Good morning i am from the dark side and as if you've seen my tagged my tag this year is come to the dark side we have cookies I had cookies but they're all gone.
So the new tag is come quickly because the cookies run out fast so I'm the fool that decided to do a filesystem talking linux conve you which was a bad idea.
This is to give you this is not an internal discussion of butter face ok for that I can pointed some lovely very very in-depth presentations that Christmas and management have done on how the stuff actually works on disk or doesn't this is a beginner's tortue why butter faces shine in why should start looking at using it uncut so if hopefully that's kind of what you were very understanding if not you can go come back to the excess talk which is where i'll be next as well right so what about her face it was originally developed by Christmas an article and then now jointly developed by a whole bunch of people are equal redhead fujitsu Intel tuesday and more those of you who follow the butter fists lift and advice gonna SiC that's good i have yes no no they stay show up when it's the next slide that unsharp when it's the current slide or some let's do that and see if they do.
That's better i go i'm using PowerPoint and unfortunate corporate template styles.
Ok so there are those of you from the butterface list Christians pushed a poor request to let us this morning from a whole bunch of stuff out of fujitsu with the new raid rebalancing code which I'll get to in a couple of slides time the key of butter face is that all data and metadata is copy-on-write ok so we never go and overwrite data on disk gotta file you want to change it we read the file and then we write it out will be not the file we write our blocks that are changed.
Ok so we don't ever actually go and change box on discs we just copy them and write them up and ride out the copies we maintain CRC's for all metadata and data.
Okay so everything is checksum so we can tell when the data has gone patient and until recently we could tell you in the data is gone pear-shaped we couldn't do anything about it we could just say look your dad's gonna pay.
Now we could do something about it okay and one of the games i'm not gonna spend a lot of time talking I want to spend more time actually showing you butterfish so I'm going to kind of run through the overviews fairly quickly efficient writable snapshot snapshots of snapshots snapshots of snapshots of snapshots and subvolume so butterface includes not lvm style value management doesn't create block devices they just create subvolumes inside the file system and you can mount subvolumes and we do some fun trickery.
I in the last demo which I added this morning so you'll forgive me if that one doesn't actually work that we do some really interesting things with mounting subvolumes multi-device support built-in because what the linux really needed was another implementation of red we have built in reg supports at the moment we have raid 0 red one raid 10 raise five and six and triple mirroring is in Chris's to merge q after he releases butterface check online resizing and defragmentation and in fact you can online size up and size down so you can remove devices in and out of a butterface file system while its online you can also now replace devices but butter faces online so you can add a new disc and you can say rebalance and benefits will move extends across the new discipline you can drop the oldest out of a butterface aret we have transparent compression and we do that profile basis as well and we back out if the file won't compress so we just write it out efficient storage was more files so we packed the files into the metadata as well and lots of a 60 optimizations which by the way is the optimizations aren't there just turning off all the optimizations we do for rotational media okay so we just that I don't know look and trim support now as well so what are we working on at the moment extend for performance and stability fixes I spoke to chris this morning and he said yeah that stability thing was very important because he is pull request for 3.3 wouldn't allow you to boot system at all that was a really good one ok so that's now been fixed if those of you who are looking for a very special announcement today are not going to get it because he was fixing the 3.3 pool request sufficient I've ever a lot of code cleanup so I'm moving around a lot for the video i just realized finally some free space casing across reboots so we actually care case the free space map i delayed metadata insertion and deletion for those of you still do have spinning disk background scrubbing and that becomes very important once you get into red one and raid 10 and we look at what scrubbing is and why that's very cool.
El dedo compression is in mainline snappy compression is coming in developments in the pool windows so the difference pretty much as I understand between snappy and l0 the resulting compression levels are about the same snappy is just faster determining whether the rocks can be compressed or not.
So what basically what benefits does is it says can I compress this yes okay i will compress and right if not i will just write it out i pressed ok so it doesn't waste time trying to compress it that's God's that we have lives discarding so mark- Oh discard for ssds so it'll do trim discarding on the flyer or you can turn that often batch discard most ssds that do this don't need it but it's very nifty for the embedded devices that have less than stellar flash in them put on I flags to control copy-on-write and compression so you can actually flag files to not enable copy-on-write will not be compressed and automatic file defragment understood that actually someone I got one of the upcoming changes that we use key ordering to group blocks for metadata the problem is the moment we're sitting in a 4k block for better fess that's in the.
3.2 linux kernel we're pushing about seven eight megabytes per second once we push up to very very large block 16 case of four times the size hundred eighty-six megabytes per second for metadata rights so the bigger the block the list fragmented rb3 metadata blocks are less overhead because everything gets copied on right so we just write all the stuff that much quicker I get so that cue that code basically what's going to happen is we're going to allow you to create bigger blocks in 3.3 the default will still be four though and then based on tasting results from the 3.3 merge chris is looking to impact change the default from 416 later on okay but the current one when you get 3.3 you can create bigger blocks later we may automatically create the box but me one.
Yes just as we saw the fun bits crc the last to verify data stored on disk so we take some metadata and data which means we can correct by reading a good copy of the block from another dr ok so the new scrubbing code now as scans on Reed automatically so if you're in red one or a 10.
We're automatically doing check some validation and if the checksum fails on the first read will pull it off another copy or we recommend for those of you have Enterprise sand devices those enterprise sand devices scrub on a regular basis because what you don't want you know everybody's always we've got a scrubbing on read that's great i'll never worry about this the problem is is a butterface is deterministic s2 which copy it reads normally so if you've got a red one array you've got two copies of every block metadata and data it's always going to read block a off one copy and if that copy always remains good it's always going to read that copy it's never going to scrub.
If the disk on which their copy fails and now you're left with disc 2 and that one's gone bad and you haven't been scrubbing you're not in a position where the checksum will fail and you can't read the data so we recommend that you scrub Christmas sort of saying very very high usage filesystems once-a-week not-so-great filesystems once a month i would scrub one see how long it takes and then adjust accordingly it does thankfully scrub in the background so you can start a scrub and I oh nice the scrubs down so they don't impact your performance and then just work out how long a scrub is going to take and then the shade you described to happen in the points of low utilization of your filesystem ok so it will be extended to track it offline bad devices something to note though.
Butter faces raid is not what we think of as sort of md rate it's not block raid.
Ok it's not mirroring block devices it's just ensuring that for every block we have x semana copies so the moment for red one.
It means that we have two copies but we can have devices of different sizes inside a raid 1 array so you can create a raid 1 array of five devices or butter face will do is ensure that for every block we have at least two copies on two different devices raid 10 is obviously we need repairs the stripe of red once so let's give this a whirl do to do to do we all see guys so this is my lovely oracle linux 632 running our brand-new unbreakable enterprise Colonel to release 2 which is currently in beta which you can download so public hi phenomenon oracle.com / betta you can grab the colonel and butter face Prague's if you want to play with butter office this is.
Pretty much the version that will go into production with Oracle Linux 6 later this year so we are taking butterface into production when we release this Colonel guru we will also be releasing an update to the Oracle Linux 6 installer it actually allows you to install with a butterface root filesystem ok so you will be able to run better fit on route for reasons which in my last demo should become obvious if my pops demo works ok so that's the one step that I always forget because this is a brand-new I keep resetting my snapshot and i forget that i haven't actually turn on butter if it does turn on butter fists i'm going to make a fist yeah yeah it'sit's production-ready pretty much yes I'm not production-ready with the file systems fun ok Stevie and see the one cool thing about butterface.
It doesn't matter how big your discs off better make butter face is that fast all the time because we don't actually write anything.
Ok because it's small tiny little book and that's it i just created a red one so i have metadata is red one data is read one on devices hdb and see you can see currently we're at 4k blocks 16 gigs total because it's too and yeah here's where size becomes really interesting with butterface because it's to eight gig drives and it's telling me it's 16 gigs total but it's not because it's red wine i can only fit eight gigs worth of data on here and in fact it's lace because I need space for metadata as well so there's no data space and this is why DX minus H lies to you okay never trust a TF minus H with butter fest we actually have if i can remember the command for that need to do to do file system vs so let me mount this butterface mount and the cool thing I just have to mount.
The first I just have to reference the first device in the array and others mount the whole thing for me so adding and removing devices from the array is really simple i don't have to worry about you know md devices what I'm referring to just and you can affect refer to any of the devices in the array doesn't matter which one sure.
HD mining butterface second device correct.
No it doesn't worry.
Yes you can you can do by label you can do about uuid and you should then everything is that if you have all their first tabs that refer to give sdb that will continue to work with regardless of how many more devices you add and remove assuming FSDB remains in the earth DC it's not something you also listed is fun.
No they don't it should set in you sexually i haven't I haven't tested that much i got no sure I've been a little time come on.
Ok so I'm out there I'm just going to copy in some ok so there's my original idea of Saudi of minus H shows that i'm using its at 6k awesome yes and it is 16 gigs available whereas i don't actually have 16 because available also because i haven't signal so now it does actually show that very point one makes but still 16 gigs which is kind of a lie.
So if i do better this BTW orifice filesystem DF butter first so you can feed data in raid 1 i'm using total one gig allocated because we allocate in one big chunk so far it's only allocated a one big chunk of red one so if i remember it's not actually read one in the entire device yet i can have different grade levels for files on a butterface filesystem i can say some files in fact not read one I can turn off their raid currently using 3.98 megs data in total eight megs on the disk system red one total of eight makes allocated used or fork metadata is a dedicated a gig as well I've used 24k of it so it's kind of fun i got so now what I'm going to do because i have copies and I'll just sing a few times again let's take a look at five.
Mmm yeah we got so this is sitting physical block to think 9312 it's a 4k block so 4096 x 26 9312 that's what i want to corrupt we have a very very very lovely dangerous tool so one of the things that you should be very careful of with butter face corrupt block it's defaulted to corrupt all the copies of the block so we do want to do the first copy and do that and rips I need to know I need to unite divis to be they got you can t want my corruption I got so what I'm basically done i have corrupted the first copy of that block and now i'm going to remount thristy be at and i'm going to some so one of two things is going to happen here either it's gonna order scrub or it's not gonna order scribes I don't know which copy it's going to try and use when I some vm limits so indie msg.
There we go it auto scrub so check something failed in that block it automatically corrected it and then copied on righted the working block back from the copy so it's not a very useful so now what I need to do is in fact you mount and go back and actually kill the second copy because that's the copy that butter faces reading from now have corrupted the second copy i can mount i can sum and this is where things get really interesting this is why scrubbing is really important.
The sum is the same as a butterface hasn't actually detected the on-disk corruption yet so what I really need to do and this is why files system scrub that fire butter face scrub stocks scrub start this it starts the scribe i can do a status of it and it has scrubbed out with zero errors.
Huh why don't you fix my work for me because it probably went and error hey and fixed my Reed area again so we're very good at fixing the best things but this is obviously only works with red wine or higher you need to have another copy of the block somewhere you know if you don't have a single disc butter face scrub it will alert you that this on the wrong cause it's still check sums even on a single disc or in raid 0 but it has nothing to copy from so it has no known good.
Okay so there we go.
So that's scrubbing awesome so we now have discarded trim so anybody had not know what discarded trim is it's for ssds it's basically to tell the assist you were done with the block ok essentially an optimization ssds these internal copy-on-write load-leveling kind of mechanism can say i canjust array i can just right over that entire block without worrying about what might be there we do it I both real-time and batch to the default if you mount discard is real-time will actually send discards don't this is these all you can turn it off and you can do it by and ioctl which is coming that you can do a batch shrimp to all the free space Drive shopping so we can now do for as a google summer of code project the Ray rebuild works like the rebalance code so one of the nifty things is use God a you can retry between raid levels soon but you can change you can add devices to butterface at any time and then you can start a rebalancing butterfish so you can tell it he had two discs their red 1i can then add two more discs change to rate antenna to rebalance and it starts making sure that we've got copies across and start striking across that rate an array so the restriping between raid levels actually does it via the rebalance code.
Ok so basically is rebalances because again this is not block-level raid this is this is not brought device level read this.
Data block level rail drive swapping in place dr swapping ok so I just shove a new disc into my device and I tell it ok move from one device to another we are currently working on and this is my lovely friends over in the solaris department at big evil tell me that benefits send and receive is the best thing since sliced cheese and makes replication of Defense volumes really very easy butterface and receivers and development it actually works now so there is a working version of butter face and receive on at Smith skip repo which I couldn't find when i was writing the slide last night and it transmits a neutral third actually transmits it doesn't transmit the original block it's transmitting in neutral so we hopefully do not have transmitted the corruption but races fairly fairly small machines and likes interface particularly interfaces deduplication mechanisms which requires lots and lots of ram it's not as friendly too small discs though as you still with that file system DF butterface went out and allocated a gig of data and a gig of metadata the first time i wrote at eight megabytes the file system so in that role for my incubi virtual disks in this virtual box i'm probably going to be able to fit seven gigs worth of data in a gig of metadata because I probably won't need more than a gig of metadata for seven gigs worth of real data one hopes depends on how often I actually again this is all copy on her act so it depends on how much change goes on to these things chris has a and I point you to his his fun tools location in a sec but he has one of the things that emulates a colonel compile the file system activity of a colonel compile without actually doing a couple compile it takes and we don't we're not doing cpu testing we just want to see what the file system does and there's some really interesting things if you keep writing data over and over again you might need more metadata clan up to a point where you've got two gigs of metadata.
A dedicated and six gigs free kind of it ok but it works very well on low in flash because we can do SSD optimizations inside butter face instead of sort of relying on Flash's own load-leveling capabilities the copy on write nature but of butter FS is load leveling ok because as you'll see in a really fun animation when i create a billion 0 x files you can actually see that low-level in an operation i get so the rain five six the raid five and six code was implemented by Intel sometime ago it is merged painting the fastest and check work it adds triple mirroring so the equivalent of equivalent of raids if you like.
Ok so r 86 triple mirroring what's really cool is mixed raid mode for metadata and data so you can have data rated a trade 10 but metadata at red one or data raid 6 metadata read ten you know is you can start mixing your red modes on a single butterface follow which is really nifty this is the fun one right.
I would really love all of us just hide also i got so when bad things happen to good data I caso butter fists about and still has some bad flag for when things go bad they go terribly terribly bad there is currently a read-only recovery tool from Joseph this basically attempts to get all the data off of butter of his file system and copy it somewhere else it's read-only it doesn't touch the existing butter office file system so you need another file system to which you can copyright data and that actually works really well.
Ok we have now the tree root history to recover from a lot of hardware errors ok because of course butterface is copy-on-write so we'll just walk back through the root history and we'll find a known good copy of that and not that ok a new father some chick releases on the way I spoke to chris this morning the only thing I can say about this is it is talking about better face in LA on saturday and ok very subtle yet various little sister her husband the currents recovery be tools are at that git repo for chris's said that's the read-only recovery and I will now show you a full file system recovery which is kind of the I assume when butterface check gets released it will be in viruses products as well the other indicator about the the release of butter face check is that Oracle has publicly stated at open-world last year that we are taking butterface into production with the release of our new unbreakable enterprise Colonel that release has already gone into feature freeze in the colonel ok so the colonel has feature frozen butter face products creek Chris uses this is an excuse for not delivering because butter faces frogs is not technically in the colonel so the feature freeze didn't didn't apply to him.
Ok I'm bit is the indicators of this as to sort of time frames we released a tpc benchmark with that Colonel in late November early December last year and tpc benchmarks are only valid six months from the release of the benchmark so the you know I you might my usual joke of winners oracle going to release a product it's not between now and the heat death of the universe we have a six-month window from the release of that tpc benchmark to get the colonel out because we did the bench walking towards the tpc benchmark on that Colonel for those of you who are following because he doesn't follow me on twitter tpc transaction processing council to database benchmark okay so last year oracle with our partners cisco we published a tpc benchmark on a single server of 1.2 million transactions per minute and this morning because it just got announced by yahoo and news and everybody else but epc's website hasn't actually been updated yet we announced our new non-clustered tpc result on a single Sun 400 box of 4.8 million transactions per minute ok so on a single Intel box the only two devices they can beat us is the IBM power 7 and the bullets collar power 7 which is the same thing with a mess ok so two big power 7 mainframes and a5 are you son 4800 it's pretty awesome and then there was anyone in fact that that benchmark uses four terabytes of ramp and we had to wait for graduate for the Sun guys to actually put the 410 United the ability to buy four terabytes because you have to be able to buy the system when you publish the tpc benchmark be able to buy the system so we have to wait for them to put that on the priceless so you can occupy a system with full terabyte around I guess let's have a look at file system recovery what happens when everything goes pear-shaped ok so what I'm going to do e is you mount butter face again i'm going to use something called debug tree- our snow so what this is showing me the extent and device file system and check some trees this is where their current points are on the file system and now you can start seeing the rolling back up so there's my route back up that's gin 13 there's jan-14 ok Jen 15 and jen 16 they're actually in order but that's a rolling back up.
Okay so every 30 seconds butterface create the backup so i have two minutes worth of known good rollback point at this junk truck that's just gonna keep rolling these things correct yes it's going to continuously do this in 30 seconds and just roll because even if it doesn't change the blocks it's just going to roll the regeneration and this makes sink faster as well one of the things that we did is we change the way the metadata logs happen frigid in fact changed their metadata and it spelled up if sync on individual files like seventy-five percent or something which is really cool.
Ok so what I'm going to do yes yes every 30 seconds at any rate yes you can turn this off that you can disable as a run-up are it's possible for a bit to get corrupted but the likelihood of all of it getting corrupted game because this is copy on rides so I never overwrite any of this i'm just writing it somewhere else on the no no note but what I am going to do is I'm going to take the check some key which is sitting over there and I'm going to use the local soul corrupt lock is already much unwanted now I've corrupted that on both copies if i try to mount now well yeah but I'm rig this is you've got the corruption scrubbing the the negative of corruption happening yet hopefully describing the live scrubbing should take care of that because we'll just copy of a known good so run-rate one race are because of virtualbox done that's this virtual boxes is trying to create a the the virtual cd-rom device there we go whoo.
Crapper look at dms d there's my bad tree it's as though I couldn't open the checksum tree I'm just fairly what do i do how do i recover its really easy done it read the bad trees it's still mounted because now what it's done there's my lovely file for those of you remember anybody remember numbers because that is where the numbers it basically has done if i do i need to you mount butterface because butter fists debug tree once order stp it's rolled that check some tree back to you can see this is not Jane 17 jan 18 we were 15 16 we're not 17-18 was going out but if I go lower down you can see there's 15 and 16 15 16 year-old to 17 and 18 we've actually rolled I think that's a walk back to 17 and so your worst-case scenarios you've lost 30 seconds of data changes again this is copy-on-write so what we've done is we put the system back to 30 seconds ago ok if that was also corrupt we would try the next backup and the next one the next one the likelihood of none of them working is why we need breakfast check okay i'm getting the 10-minute mark button is the last time that I really want to do so that's the recovery as it stands millions of files this is just showing off some right back and metadata bottlenecks explore next face tended to get that behind a lock successive dramatically improved here in the next talk will show that improvement we know that is a current right back recreation we are working on it one of the 3.3 merge changes was to solve that but that's kind of what the file creation benchmark looks like on a 54 butterface on spinach for better fess what I will show you because the i/o animations for this is pretty cool actually so this is it's just to lose if you let the new come back.
Come back full screen ok so this is what x4 looks like when it creates a billion files you see the really busy part.
Ok it's doing 50 megabytes a second okay and it's pretty much bottleneck on its journals.
Ok that's essentially just wakes force its excess is nifty this is X face doing the same thing but it has the most amazing pattern the green is read that there.
The green is right there is read but he's still at this point when we did this animation stuck behind the log that's gotten much better but you can see what happens with the six per second ok and if you want to see load-leveling so then we discover back there whoops stop stop let's go back to Pirate Fest this is what load-leveling looks like so that's the copy on write of the metadata happening it's a better face is just load leveling a billion files with the metadata across the disk ok so it's sustaining you know upwards of a hundred megabytes per second and this is on a 4k block we reckon on a 16k block will get 200 and something megabytes per second which is kind of nasty if you want to know how those animations were created it's part of chris's fun tool bucket so-called seek watcher it basically takes the block trace and create animations are blocked race ok but it gets me to my last demo which is converting rooted butterface and the glory that is young plug-in FS snapshot so you unplug and if it snaps already exists.
Basically what it does is snapshots of file system prior to doing a young upgrade again the only file system the currently support snapshots is butter fest so what we're going to do unfortunately our installer doesn't support butter faces part of the installers for the simplest way to convert a file system obviously we convert in place we can convert x3 and x4 file systems to better fess what we do is we actually could we store all that x.x situ data as a snapshot so you can roll back to where you were before you did the convert but we will do is come back.
They're like that role yes or x3 at the point where you converted it so if you're going to x3 or x4 system you can convert about her face you can run with butter face and if you don't like it you can just tell better face to go back to where you were before you started because we store the x2 x3 as a snapshot of butter if s so that the easiest way at the moment to convert a better face is to put the fedora 16 live CD which has better face inside the installer and has better fess convert so here's how you convert this stuff as a chugs away to delete this is my quick plug for Oracle VM VirtualBox for all your free desktop virtualization needs my dismay had known 3i this is what I now no longer load the virtualbox guest additions on fedora because then I'm always assured that gnome 3 will never it's great i got so let's become roots I've got my logical volume as you can see there's my RV route so what I'm going to do is butterfish but BTW or if this convert give VTOL 6 root and the guys and creates a butterface metadata for me and at the same time it's taking and creating a snapshot of the x2 system works because it doesn't actually snapshot the journal that it creates a copy of the x two days that it's just converted my system her and the thing that I always forget which hopefully i will now remember I mean I need to make two routes mount give because what have I not done yet.
On routes yeah I reach see if this tab where did they go.
That's all I needed to do actually need to pull the CD out of this machine remove disc somehow able to i got it so there we go reboot into go and remounted route.
Yes I didn't write it I got so let me go back to my terminal so now if I look at Mount route is mounted butter fest if i look at the subvolumes butter fists sub full list you can see there's the ex2 the save dx2 subvolumes I can always go back to the point which I converted ok so now what I'm going to do because yeah and just make sure that i actually in this copy have if a snapshot installed which had turned cd-rw engaged to marry young and all six latest thank you say young group install I'm going to do something very silly it does.
Is no no I actually have yama on the same p.m. so there you go it's nap shuttered and if you so there's the snapshot happening it's snapshot of both of my butterface file systems because in fact the yum repositories also a butterface snapshot and route and butter face so now it's chugging away installing a whole bunch of stuff i was going to do this game up by installing a whole bunch of centos rpms but there's a conflict between because i'm using the new kernel for this and sentence doesn't have that Colonel there was an interesting conflict so while that chugs along doing some stuff anymore questions yes.
Yeah I believe the new metadata logging that frigate suits admit I think about 75% faster.
Certainly I mean this young which does similar things is much much faster running on butterface and it's actually running butterface with both the location of the Apaches yum repositories and riding into root so it's much better at it I got today we've got so now we've installed what I'm going to do bonus spt orifice of all this that there's the second to the ID 257.
Decide to easier than me having to type the 4x will turn around a minute yet give me two seconds because this is cool three flags equals sub ID equals 257 ohh 257 wasn't yes.
Ok so now i reboot reboot they're good little box chugs away I didn't change in a tab doing it five by default but ok.
There goes and it starts from firing off first boot because i haven't didn't actually run that before because first boot is not installed in minimal and I could start X and X comes up and we're running as root you know that's not really what I wanted to do to my server so if i restart.
The 4x100 the top left-hand Conard it's mounted sub all I d257 x is even there god I know I now all we have to get my first spoon for firstborn child to the god of demos because that actually worked but you said that that rollback is very very nifty so you can snap shot at any time we we do have enough time for one of the war with the love there I can't read that from Orange choking her and usually raised it I'll take two I saw this hand first I'm going to randomly pick her hand that one there in the green shirt we had different questions during the talk so we're only gonna have a couple of questions after the talk so far away.
Thank you i got all right so if you have to rate of course you should be using anymore i'm in your case since you have checksums if one drive is returning a bad block i take it you can detect that we write the block correct ok yes and outside of having checked with an English wanted to drive is wrong which software doesn't do you have other benefits from being read directly in the file system because we can read balance that at any time now so you're not stuck at any rate level we can start with raid 0 make it red one then rebalance and butter face will go and replicated the goal eventually do not duplicate the red code as in fact you know chris said it's silly that linux now has three red implementations in various places but the way it's that now it makes more sense to be in better face and hopefully over time they'll be one liberated some way.
Potentially but that's not on our roadmap and if using software raid you should just basically start over from scratch.
Right there you can you can do better face on software right ya ready kid you don't want to because your code is better than software raid for that you don't get it at it depends on what you're bored you want to achieve essentially in fact I know a lot of people who have software raid ones and then do a butterface read 10 on software raid one's kind of thing and use both balancing codes and you know it all work until finally i think yes it was Green there did you could technically possible ok I mean it's one of those things that we know that the the code can do it it's on a road map but it's not something that we find particularly interesting right now discs are so large that it's cheaper in terms of processing power just to buy more discs which are very cheap then to work with cpu processing your workout that I'm deduplicated because it's all copy-on-write so one of the challenges and I have to compare against what am i comparing against the stuff that I've just written everything else on the disk.
I and and that becomes very interesting from cpp when the defense doesn't for example they said internal honest you were like four to six gigs of ram to hold nice big deep cases and stuff so yeah so go.
I'm but you can obviously come and find me if you have more questions about butterface because if you want to test this obviously Oracle Linux 6 32 is free to download and all of the Colonel's and butter face problems free download so or attempted or 60 now as well but flora 16 doesn't have the latest so we've actually emerged in all the three dot to changes not the 323 change but all three are two main line changes in fact in our kernel as well so all of that metadata improvements and stuff is there as well cool thank you very much for the talk RV guys.
Hi very popular subject to city and one which of course as you said you will so against a few questions if there are people want to call him afterwards we do have a small token of appreciation a glass penguin related gold Louis LX thank you very much.


##########################################