4/18    I'm interested in doing some traffic analysis to see if
        the sshd trojan can be detected by looking at traffic patterns.
        I seem to remember people's inbound sshd connections
        being dropped now fairly frequently [but soda stayed up].
        Can anybody authoritatively speak to whether just some
        sshds were dropped or when one was dropped all were dropped.
        Also I assume outbound sshes were not dropped. I'm curious
        whether the sshd bug was in maybe the checkpointing routine
        when it was writing out to the sniffer log, or it was
        something more random/complex. Unless I get a good lead
        I probably wont pursue this because I'm sort of busy
        now and it's a lot of data to trawl though potentially or
        lot of work to reconstruct. Basically looking for a large
        clustering of sshd drops in time and space without evidence
        of a reboot [other protocols dropped] and not a normal shutdown
        might be smoke -> fire signal.
        \_ Even if this particular ssh trojan was causing the daemon to drop
           connections, why would you assume that this would be true of other
           ssh trojans? -dans
           \- why do you assume i assume it is true of other trojans.
              obviously my concern is we dont know where the soda hacker
              came from and what he did with the sniffed info. assuming
              this same person installed the same buggy trojan elsewhere
              is hardly a stretch. a better question might be: is the
              trojan buggy on just freebsd. and the issueis sshd not
              ssh. ssh trojan and sshd trojan have different implications.

4/17    Rootstaff:  You guys fucking ROCK!  Thank you so much for putting in
        all the extra time and effort necessary to get soda back up running
        especially after all the disasters!                -mice
        \_ seconded!
        \_ Good job, thanks.  -John
        \_ Excellent work. --erikred
        \_ thanks.   kngharv

4/18    Some thoughts about securing a machine.  Feel free to add your
        expert opinions. --ricky
        * Securing a machine that allows interactive logins by users
          is _very_ hard.
        * Reduce suid binary to absolute bare minimum.
        * Perform automatic _remote_ checksums from a machine that is
          separate and is not accessible by regular users.  Usually,
          NFS is recommended for this.  Basically, have a remote
          machine regularly check critical files on the machine and
          alert root if anything changed.
          \_ This existed a while ago, called Tripwire. Started as a
             a research project and grew to a startup. Many people tried
             it but gave it. The concept is easy, but in practice, it takes
             damn too much time. All of the above suggestions are good,
             but in the end, if the cost of manageability is high, no
             one will care. Lastly root and politburo aren't paid to do any
             of the above stuff and most people have better use of their
             time so... why cares. Would YOU like to volunteer
                \_ Why are suggestions being taken as a demand that they
                   do it.  If alumni (or "members") do all this stuff, aren't
                   they just "fucking the undergrads" ?
                   \_ No.  If they storm into the machine room or the office
                      and insist that it be done there way and be done right
                      this minute, then they are fucking the undergrads.
                      Historically, asking nicely and accepting a polite `No.'
                      is not one of the strong suits of the alumni.  Though
                      anecdotal, it's also worth noting that the amount a
                      given alumnus bitches appears to be inversely
                      proportional to the amount of meaningful contributions
                      (time, money, hardware, etc.) he makes to the
                      organization. -dans
                      \_ so you contribute absolutely nothing, eh?  -tom
                         \_ Ah let me clarify that.  The amount a given
                            alumnus bitches at the current undergrads appears
                            to be inversely proportional to the amount of
                            meaningful contributions he makes to the
                            organization.  If the alumni bitch at each other,
                            it has no bearing on the CSUA or its future.
                            -dans
             doing these things ricky? You should attend politburo.
             \_ Agreed, I tried to set up a modern version of tripwire on
                hosts I administered in my last job, and it's nigh unusable.
                It smacks of overengineering, and has too many features
                apparently added by marketing folks trying to sell to the
                enterprise software market.  Furthermore, if you want to be
                really secure, running _remote_ checksums isn't good enough
                since the credentials for soda are likely the same as the
                credentials for other CSUA hosts.  Thus, checksumming soda's
                binaries from screwdriver takes a non-trivial amount of work
                for a trivial gain.  Also, what happens when people trojan
                libraries not binaries?  Should we checksum those to?  Which
                libs? -dans
                \_ ideally you checksum everything, and flag what is 'volatile'
                   and likely to change from day to day.
                   \_ ideally, yes, but that's a really time consuming,
                      tedious, manual process.  Unless you have some '1337 tool
                      to do that for us.  If so, please post a url. -dans
             \_ I have used aide, a tripwire-like tool that checksums files
                in two ways. It works pretty well, and isn't that difficult to
                use.  I found it annoying if I didn't check/update signatures
                before doing package upgrades, which meant I couldn't tell
                whether the changes were intentional from the update or if
                someone had done something to the binaries the same day.
                While there are certain more-secure "ideal" ways to set things
                up (binary on immutable media, running on a separate system,
                database on immutable media, etc.) A simple "on this system"
                "aide running out of /usr/sbin" "database stored locally" while
                not great from a security standpoint, as long as one doesn't
                rely on the lack of warnings and messages to mean you are
                secure, is still a useful tool.
        * Educate users about ssh.  For example, unless the user is
          extremely certain that their private keys are safe (resides
          in encrypted partition, etc.) having empty passphrase is a
          bad idea.  Assuming above is met, using passphrase protected
          key pair and setting up authorized_keys is safer than using
          passwords.
          \_ Education works the best, when people are willing to
             be educated. Do you think people like to be educated?
        \_ It's also vital to keep up with patches to OS and utilities.
           \- ssh wont solve the problem. the problem is a combination of
              clueless users and users who dont care about security [and
              are willing to login from machines with kbd sniffers]
              combined with the close to inevitability of local account ->
              local exploit -> root. i think sloda should adopt the
              position: 1. soda will be broken into and should not be
              trusted ... meaning it should not be used as an outbound
              stepping stone ... no rsh, rlogin, ssh, telnet. i suppose
              you can leave ftp on and i guess scp. 2. do what you can
              about prevention [applying patches etc but also invest some
              in rapid detection. tripwire is a piece of crap but there are
              other tools to do this with ... i maintain checksums on about
              50 things [in some cases OSes, in other cases various data
              trees] and while i dont look at all the data everyday, with
              disk being cheap i can store enough snapshots i can at least
              go back and tell a story if there is a problem found at some
              point. even a half asses checksumming system will get you
              pretty far ... and would certainly pickup a trojaned daemon
              or client. we have some not-very-portable hacks to address the
              case of trojaned libs [these check low level information in
              inodes and compare them to higher level queries and look
              for inconsistencies ... like say in the link count] but these
              are probably not worth the effort ... they were crafted for
              very specific rootkits.

4/18    Soda just rebooted. What happened?
        \_ For some reason the CPU load was growing out of control.  I don't
           know the culprit. -!root
           \_ Do you know the process id? Was it wait time or cpu or user?
           \_ It was NFS hosage to keg.  Unknown why (keg seemed to be fine)
              \_ That seems likely, but was not definitively confirmed.  BSD's
                 NFS implementation and Linux's NFS implementation have a
                 history of not playing nice.  Though I've always heard of BSD
                 NFS clients not talking to Linux NFS servers, and, in this
                 case it would be a BSD NFS server (keg) hosing a Linux NFS
                 client (soda). -dans
                 \_ pave keg
                    \_ Let's not be so hasty.  Perhaps we should take steps to
                       make sure that the problem is, indeed, BSD NFS <->
                       Linux NFS interaction before doing a lot of work to
                       pave keg. -dans

4/15    11:09:12 up 3 days,  9:25, 18 users,  load average: 1.39, 1.67, 1.56
        ... and counting.  Props to the current undergrads.
        \_ The kids are alright.
           \_ Seconded. Everything looks good so far. Keep up the good work!

4/18    Soda and Keg had a spot of trouble talking to each other and as a
        result, NFS mounts had a slight problem, which was fixed for the
        moment with a restart. Investigation into the problem is proceeding
        apace. We apologize for the inconvenience.

4/18    Thanks mrauser for the call just now.
        root:  I think one of the next priorities can be enabling POP3/SSL
        and IMAP/SSL.  I'm going to download e-mail with the unencrypted
        connection, but I'll probably change my password once every couple
        weeks until the above gets online.
        Most if not all of the official UC e-mail systems now require SSL
        for downloading and sending e-mail, right?
        \_ Actually, all password transactions must be encrypted according
           to the Minimum Standards for Networked Devices policy.  -tom
        \_ IMAP/SSL is now up, POP3 is down entirely. That should suffice
           for the moment. -michener

4/18    Hi guys, just downloaded ~ 200 messages via IMAP using Outlook 2003.
        I manually deleted all my spam before moving the good e-mail to a
        local folder.  A lot of my messages have the wrong content for a
        given subject line, and those e-mails are also messed up in general.
        Not sure if this happened to anyone else -- already deleted my soda
        copies, so too late to check if it would work using POP (perhaps just
        Outlook weirdness).

4/18    Pres. Bush: "I'm the decider, and I decide what's best. And what's
        best is for Don Rumsfeld to remain as the secretary of defense."
        \_ DUBYA! DUBYA! DUBYA is the STANDARD!!! political troll
        \_ http://pollingreport.com/BushJob.htm
        \_ http://decider.cf.huffingtonpost.com

4/18    http://tinyurl.com/h8oe3
        Buy your 830 sq ft dream house in Redwood City for only $729K!
        Silicon Valley rules.

4/18    [Deleted because it seems I'm the only one who had that prob]

4/15    Bummer.  Reboot due to what looked like NFS woes.
        Regardless, props to the current undergrads, they're doing some great
        work.
        \_ The kids are alright.
           \_ Seconded. Everything looks good so far. Keep up the good work!