DDRAID
“A Cluster for the Rest of Us”

Daniel Phillips

Red Hat Software Inc.

http://sourceware.org/cluster/ddraid

phillips@redhat.com

RAID 3.5

Or: How to Run Your Disks Five Times Faster with Software than a Hardware Raid Controller

Daniel Phillips

(at that time)

Sistina Software Inc.

Why?

Raid 3 is an old idea

Used on early Super Computers

Disk array rotation is synced

Data split across disks with byte resolution

Extremely fast serial transfers

Why Not?

Sucks for Random Access

Too much rotational latency

Disks can't seek in parallel

Hard to get disks that can sync

'Multimedia' disks are out there but cost big $$$

No longer used

but...

The Story

VA Research sent me a 5 disk SCSI Array

Thanks Walt

Unfortunately, RAID controller performance is pathetic

Reads 21 MB/s

Writes 18 MB/s

Slower than my IDE drives

The Story

So unplug the controller

Plug disk array into onboard SCSI controller

Set up a Device-Mapper striped target

Wow! 133 MB/sec, read and write

But with no redundancy

RAID 3.5

So write a new Device-Mapper target

Write a paper about it even

That's how Alasdair found me :-)

Keep it simple: RAID 3

RAID 3.5

Disk array is a power of two, plus 1

Each filesystem block split across N-1 disks

Parity on remaining block

All parity blocks on the same disk

Wrinkle: rotate parity bandwise

Compared to RAID 5

Effectively a 1K stripe size for 4K blocksize

Avoids need to buffer blocks for partial writes

It's a lot simpler

RAID 5

RAID 3

Implementation

It's a Device-Mapper target

Started life as a copy of dm-stripe.c

Split up each BIO transfer

bios, biovecs and parity blocks

Bios are nice to work with

Write some really fast xor code

Uses bout 20% of a 1GHz PIII at 100 MB/s

Make it faster! (SSE, Opteron)

Performance

Optimal for serial reads and writes

Random Access is not as slow as you'd think

Rotational latency actually happens only on the first seek

Doesn't have to buffer

Fewer bus transfers than RAID 5

RAID 5 has to do a lot of read-before-write

Boilerplate

Degraded mode

Rebuild mode

Need synchronization between virtual device access and rebuild daemon

Basic metadata

Degraded state

Rebuild progress

For now, just tell the driver what to do

Volume manager metadata

Wrinkles

Subblock transfers can be handled as special case

But Ext2 for one doesn't do any

2**K + 1 restriction can be removed

Procedural scatter-gather interface is possible

Reduces size of biovec vectors

Increase read performance (!)

Exercise for the interested reader

DDRAID
“A Cluster for the Rest of Us”
Part 2

Daniel Phillips

Red Hat Software Inc.

http://sourceware.org/cluster/ddraid

phillips@redhat.com

DDRAID

For the first time, here at LCA...

Introducing:

Distributed Data Clusters

Clusters for the rest of us.

No SAN!

Redundant data

Hotplug for free! (just pull the network cable)

Fast!

Linear throughput scale-up

DDRAID

Mar 14 18:09:35 <dopp>	So, how many evil geniuses does RedHat employ?

Mar 14 18:09:59 <dopp>	this ddraid stuff by phillips is crazy insane (in a super, wicked-cool way, of course)

Mar 14 18:10:09 <Hymie>	none, they are all good people, none of them are evil, you can't prove that they are evil, stop with those accusations

Mar 14 18:10:16 <dopp>	hah

Mar 14 18:11:07 <Hymie>	ddraid?

Mar 14 18:11:21 <dopp>	see the latest post to linux-cluster

Mar 14 18:11:22 *	dopp digs up url

Mar 14 18:11:33 <dopp>	https://www.redhat.com/archives/linux-cluster/2005-March/msg00034.html

Mar 14 18:11:39 <Hymie>	k, checking

Mar 14 18:13:07 <Hymie>	woo, raid 3.5... man, got to get me some of that, heh

Mar 14 18:17:46 <Hymie>	this sounds like the bee's knees

Mar 14 18:18:47 <--	windex has quit ("going-home.")

Mar 14 18:23:18 <Hymie>	dopp: I owe you a beer, some day

Mar 14 18:26:38 <dopp>	oh? I can't pass up beer ;)

Mar 14 20:30:18 <phillips>	dopp, heh, thanks

DDRaid Demo

The Machine:

12 Nodes. Aggregate power:

28 GigaFlops

960 Gigbytes disk

480 MB/Sec disk read/write throughput

12 GB memory

24 Gbits/sec internode network

10 GigE out the back

Possibly the fastest desktop in existence

Thanks for reading this far!
And now, a word about:
“Evil Patents”

ffii.org

Foundation for a Free Information Infrastructure

Go there, read it.

Add your voice! We don't need this. We don't want this. Help keep software patents out of Europe.

DDRAID
“Clusters for the Masses”

Daniel Phillips

Red Hat Software Inc.

http://sourceware.org/cluster/ddraid

phillips@redhat.com

2003-07-11T18:48:29