Berkeley CSUA MOTD:Entry 28351
Berkeley CSUA MOTD
 
WIKI | FAQ | Tech FAQ
http://csua.com/feed/
2025/04/07 [General] UID:1000 Activity:popular
4/7     

2003/5/6-7 [Computer/SW/Unix, Computer/HW/Drives] UID:28351 Activity:moderate
5/6     I was looking at a csua user homepage a few months ago and someone
        had worked on a project involving something about making disk
        backups to physically seperate locations, arguing that a flood or
        fire would destroy any local backups just as easily as it would your
        normal disk. anyone know the name of this project or user?
        \_ The project you describe is the Distributed Internet Backup
           System (DIBS) developed by Emin Martinian and available at
           http://www.csua.berkeley.edu/~emin/source_code/dibs/index.html -emin
           \_ thanks
              \- how are you determining what to incrementally backup?
                 are you looking at file timestamps? is the unit of change
                 a file or a block of data? i.e. if 1 byte is appended
                 to a 100meg file, does a 100mb get xferred? does this use
                 rsync under the hood? ok tnx. --psb
                 \_ if you're interested in that sort of stuff, try
                    http://oceanstore.cs.berkeley.edu
           \_ Emin, were you a part of Kubi's group?
                    \_ OceanStore is a great project but it has different
                       goals.  DIBS is designed to provide a way for people
                       to exchange files for backups in a secure, robust
                       way without requiring a central authority. -emin
                 \_ DIBS stores an MD5 hash for each file and does a full
                    backup of any file which changes.  Rsync is not used
                    because DIBS uses encryption and erasure correction
                    coding and that is difficult to combine with rsync.
                    Also, DIBS is peer-to-peer while rsync requires you
                    to have accounts on both ends.  -emin
                    \- yeah i have some limited familiarity with oceanstor
                       but again for something lightweight i am curious
                       about this dibs thing. we hacked up something we
                       call the "storagelocker" with ssh keys and some
                       other access control technology and a big perl
                       script ... it works ok as a palce to write and
                       recover a bitstream but it would be nice to have
                       some smarts to reduce the traffic volume.
                       BTW, does anyone know if the hummingbird fs
                       was ever released and what performance stats
                       look like? --psb
                       \_ Although I'm tooting my own horn, I think DIBS
                          is what you are looking for.  The way I use DIBS
                          is to put links to everything I want backed up in
                          a special directory and DIBS automatically and
                          incrementally makes backups to other machines. -emin
                                \- ok i will analyze it and let you know
                                   if i incorporate it into a front end
                                   to the StorageLocker(tm) ok tnx --psb
           \_ Emin, were you a part of Kubi's group?
              \_ No.  I was in Zakhor's group at Berkeley and I'm in the
                 DSP group at MIT.  I work on DIBS because I think it's
                 cool; the stuff I do for "research" is much different. -emin
        \_ Haven't companies been storing backup tapes or disks at separate
           physical locations for years?  He's not the first to realize that
           a flood or fire would destroy any local backups, is he?
2025/04/07 [General] UID:1000 Activity:popular
4/7     

You may also be interested in these entries...
2012/8/30-11/7 [Computer/SW/Apps, Computer/SW/Unix] UID:54470 Activity:nil
8/30    Is wall just dead? The wallall command dies for me, muttering
        something about /var/wall/ttys not existing.
        \_ its seen a great drop in usage, though it seems mostly functional.
            -ERic
        \_ Couldn't open wall log!: Bad file descriptor
           Could not open wall subscription directory /var/wall/ttys: No such file or directory
	...
2011/11/20-2012/2/6 [Computer/Companies/Apple, Computer/SW/Unix] UID:54237 Activity:nil
11/20   Are there tools that can justify a chunk of plain ASCII text by
        replacing words with words of similar meaning and inserting/removing
        commas into the text?  I received a 40-line plain text mail where
        all the lines are justified on left and right.  Every word and comma
        is followed by only one space, and every period is followed by two
        spaces.  The guy is my kid's karate instructor which I don't think is
	...
2011/10/26-12/6 [Computer/SW/Unix] UID:54202 Activity:nil
10/24  What's an easy way to see if say column 3 of a file matches a list of
       expressions in a file? Basically I want to combine "grep -f <file>"
       to store the patterns and awk's $3 ~ /(AAA|BBB|CCC)/ ... I realize
       I can do this with "egrep -f " and use regexp instead of strings, but
       was wondering if there was some magic way to do this.
       \_ UNIX has no magic. Make a shell script to produce the ask or egrep
	...
2010/11/21-2011/1/13 [Computer/SW/Languages/Web] UID:53988 Activity:moderate
11/21   Lifehacker's recommending Dreamhost as a personal web hosting service.
        Apart from csua, who do you guys use? --erikred
        \_ What do you want to use it for? Do you need CGI or PHP?  My
           brother worked for Dreamhost and said they are unethical. In
           fact, he sued them. This refers to their treatment of customers
           and employees both. I don't know who  or what "Lifehacker" is,
	...
2010/8/29-9/30 [Recreation/Humor, Politics/Foreign/Europe, Computer/Theory] UID:53940 Activity:nil
8/29    http://www.google.nl/trends?q=ramadan,+porno&ctab=0&geo=ma&date=all&sort=0
        Funny graph.  -- linkpusher
	...
2012/1/4-2/6 [Computer/HW/Drives] UID:54281 Activity:nil
1/4     I want to test how my servers behave during a disk failure and
        a RAID reconstruction so I want to simulate a hardware failure.
        How can I do this in Linux without having to physically pull
        a drive? These disks are behind a RAID card and run Linux. -ausman
        \_ According to the Linux RAID wiki, you might be able to use mdadm
           to do this with something like the following:
	...
2011/9/14-10/25 [Computer/HW/Drives] UID:54173 Activity:nil
9/13    Thanks to Jordan, our disk server is no longer virtualized. Our long
        nightmare of poor IO performance should hopefully be over. Prepare for
        another long nightmare of poor hardware reliability!
        ...
        Just kidding! (I hope)
        In any case, this means that cooler was taken out back and shot, and
	...
2011/2/14-4/20 [Computer/SW/Unix] UID:54039 Activity:nil
2/14    You sure soda isn't running windows in disguise?  It would explain the
        uptimes.
        \_ hardly, My winbox stays up longer.
        \_ Nobody cares about uptime anymore brother, that's what web2.0 has
           taught us.  Everything is "stateless".
           \_ You;d think gamers would care more about uptime.
	...
2011/2/18-4/20 [Computer/SW/Unix] UID:54044 Activity:nil
2/18    Why does the system seem so sluggish lately?
        \_ Slow NFS is basically always the answer. --toulouse
        \_ Any truth to the rumor that soda will be decommissioned this summer?
           \_ Absolutely none. Soda might go down temporarily while disks are
              reorganized and stuff so soda doesn't suffer from such shitty
              performance nearly as much, but no, we've gotta maintain NFS and
	...
2010/7/22-8/9 [Computer/SW/OS/FreeBSD, Computer/HW/Drives] UID:53893 Activity:nil
7/22    Playing with dd if=/dev/random of=/dev/<disk> on linux and bsd:
        2 questions, on linux when <disk>==hda it always gives me this off
        by one report i.e. Records out == records in-1 and says there is an
        error. Has anyone else seen this?  Second, when trying to repeat this
        on bsd, <disk>==rwd0 now, to my surprise, using the install disk and
        selecting (S)hell, when I try to dd a 40 gig disk it says "409 records
	...
2009/10/27-11/3 [Computer/HW/Drives] UID:53474 Activity:nil
10/27   I just read an article that Facebook had moved their database
        to all SSD to speed throughput, but now I can't find it. Has
        anyone else seen this? Any experience with doing this? -ausman
        \_ I hope you're not running mission critical data:
           http://ask.slashdot.org/story/09/10/27/1559248/Reliability-of-PC-Flash-SSDs?from=rss
        \_ Do you have any idea how much storage space is used by Facebook,
	...
2009/8/4-13 [Computer/SW/OS/Windows] UID:53239 Activity:kinda low
8/3     VMWare + Windows XP + Validation question. I need to test stuff with
        Service Pack 3 installed. I have a valid key that I own (yeah yeah I
        actually *bought* a copy, please don't flame me for supporting evil
        M$). Is it possible to register the key once, and then duplicate it
        for testing purposes?  Will Windows or Microsoft detect copies and
        disable the rest the copies?
	...
2009/7/28-8/6 [Computer/HW/Drives] UID:53216 Activity:nil
7/28    Does it make sense to defragment disks on VMWare? My 80GB disk
        on VMWare isn't really using 80GB, it just uses what it needs.
        Will defragment do anything to it?
        \_ If you want to speed up disk operation in your VM, it's best to
           defragment the disks in your VM, then defragment the disk on your
           host machine where the VM files are.
	...
Cache (202 bytes)
oceanstore.cs.berkeley.edu
Research Opportunities Possible Oceanstore projects for undergrads and grads. Downloads Software releases and other items for download. NEW: Pond (the OceanStore prototype) source code is now available!
Cache (707 bytes)
www.csua.berkeley.edu/~emin/source_code/dibs/index.html
Of course it does not help to mirror your data by adding more disks to your own computer because a virus, fire, flood, power surge, robbery, etc. Instead, you should give your files to peers (and in return store their files) so that if a catastrophe strikes your area, you can recover data from surviving peers. The Distributed Internet Backup System (DIBS) is designed to implement this vision. Note that DIBS is a backup system not a file sharing system like Napster, Gnutella, Kazaa, etc. In fact, DIBS encrypts all data transmissions so that the peers you trade files with can not access your data. A detailed discussion of how to use DIBS as well as the Python source code for dibs are available below.