Director's Blog
tech

Web site security got some press last month when it was revealed that hackers had broken into a system at the Tang Center and stolen thousands of Social Security numbers.  The mechanism for that break-in was a “SQL injection attack“; without getting too geeky, SQL injection is an all-too-common vulnerability, caused when your application doesn’t check the input it’s receiving from the user.  If the application doesn’t check the input, it’s often possible to trick the system into doing something it’s not supposed to do.

The important thing to note is that the machine which had the SQL injection vulnerability was not the same as the machine which held the social security numbers.  The hackers broke into a low-security system, and used that as a platform to find and attack a high-security system.  The trust relationship between the systems allowed the hackers to escalate their privileges on the insecure system and get at some very high-value data.

It turns out that there is a ton of code with similar vulnerabilities out there on web pages, including on the departmental sites we host on the college web server.   It is almost guaranteed that PHP code written by someone who doesn’t understand the security implications will be vulnerable to this kind of attack.  Our web server has been under almost constant attack for the past two months, and at least three different departmental sites have been compromised.  Once the site is compromised, the hackers attempt to escalate their privileges (looking for confidential data), and try to use our server to support spamming, phishing, and other nefarious activities. All of these things are more likely to work when the activity is coming from a berkeley.edu (trusted) machine than when it’s coming from an ultranet.ru (suspicious) machine.

We’re doing what we can to reduce the ability for these hackers to escalate their privileges through our servers, but we have little control over the quality of the code on departmental web sites.  Our server hosts web sites for dozens of departments (75 unique domains at last count), and once a department is set up on our server, it can write and upload whatever code it wants.  The people running web sites for departments range from serious geeks to relative novices, so some of them are just fine in terms of security, while others are wide open.

If we see that you have vulnerable code that is being attacked or exploited, we’ll contact you and ask you to fix it.  Many departments who have vulnerable code don’t have anyone on staff who knows how to fix it–the fix is usually not very difficult, but it does require understanding of the issues.  If you don’t know how to deal with the issue, our web team can also fix security problems, charging on an hourly basis.

We received $100K this year in Campus Technology Council funding for a pilot program to investigate Virtual Desktop Infrastructure (VDI).  VDI is both a new technology, and something of a throwback; the idea is that you can install what is essentially a dumb terminal at the desktop, do all your processing on servers in the data center, and save yourself money and time managing hardware and software on a whole bunch of distributed desktops.  The thing that makes VDI different than a dumb terminal solution, or existing technologies like Microsoft Terminal Server, is that VDI allows you to customize and virtualize individual desktop environments.

What that means is that you can give each person a computing environment that’s specific to his or her needs; you can get the advantages of thin clients while still being able to install customized software on an individual basis.  Or, if your business needs dictate uniformity (such as in a computing lab), you can create a single image to be used by multiple devices, which always reverts to the default state when the user logs out.

If the technology works, it would mean that we could shift from replacing relatively expensive desktop machines every 3-4 years, to replacing cheap thin clients every 5-6 years.  We would save money on hardware, but more importantly, we would save a lot of time and effort in setting up computers, and users would have less downtime.  If something were to go wrong with a client, a replacement with identical functionality, and access to all the same files and applications could be installed within minutes.

VDI can also be accessed from a normal PC (or Mac or Linux box) using Microsoft’s RDC protocol.  That means you can have access to the same session from multiple computers; for example, a lecturer could work on his presentation on a thin client in his office, continue working on it on his laptop in a cafe, and then display it on a classroom computer, all without having to log out or re-open the document.  VDI can also be used as a lightweight replacement for Parallels or VMWare Fusion on Macs for folks who need access to a Windows environment.

In L&S, we are looking at testing several use cases:

• Lecturers–folks who often have the worst computers, shared office space, and a high degree of mobility.
• Labs (and drop-in machines)–places where we want to maintain a clean, standard desktop environment.
• Administrative task workers–front desk staff, or others whose work is fairly routine.
• Mac users needing access to Windows.

We just rolled out our first thin client (in the French library), and will be rolling out more over the next few weeks.  We have also begun setting up Mac users to access our infrastructure.  There have been a few glitches–the technology is definitely not yet fully mature.  But it basically seem to work for most use, and it has a lot of promise.  We’re using VMWare’s View product, which is undergoing rapid development, and the vendor has been pretty responsive to our needs.

We have funding to run the program through the summer, and we’re looking for funding to extend it through the fall.  As long as we have funding, there will be no cost for participating in the pilot.  We’ll ask users to fill out an evaluation survey at the end of the pilot.

If you are interested in seeing whether this would work for you, contact Seth Novogrodsky, who’s our lead on the project.

Side note about Macs: We would love to be able to provide Mac OS X over VDI, as well, but Apple’s licensing doesn’t allow it.  It would not be technically difficult; VMWare already provides Mac server virtualization.

I wrote the following in response to an email I received from a department chair; I thought it would be of general interest.  The subject is Google Books, and UCOP’s endorsement of a settlement agreement for the class-action lawsuit against them.  (Google is being sued for violating copyrights by scanning and publishing books which are in copyright, but out of print).

Relevant articles:

• UCOP’s endorsement of the settlement agreement
• Robert Darnton (Harvard’s librarian) article on the risks of the settlement.

I’ll note that this isn’t really in my area of expertise.  I think there are reasonable arguments to be made on both sides of the issue.  (See, for example, Courant’s response to Darnton’s article, and Darnton’s response to that: http://www.nybooks.com/articles/22496).

Academics have competing needs related to copyright.  On the one hand, universities and libraries can almost universally applaud greater access to public-domain works.  Easy access to digital versions of the chart of Beethoven’s 9th or the works of Isaac Newton provide great benefit to instruction and research, without impinging on copyright.  [There is a distinction between works whose essential character can be easily duplicated, such as books or music, and other media such as sculpture or painting.  The Louvre may not exactly claim copyright on the Mona Lisa, but they won't let you go in and take a high-quality digital image of it without paying a fee.]

Even digitizing the works of Beethoven or Newton has an effect; publishers who might otherwise produce new printed versions might be less likely to do so, because the size of their market has been reduced by the easy availability of electronic versions.  Still, I think most academics would agree that the overall benefit to the public of having the electronic versions available is the primary consideration.

The Google Books project goes a step further, by digitizing copyrighted works which are currently out of print.  This is very much aligned with Google’s corporate philosophy of collecting and providing as much information as they possibly can.  In some ways it’s a clear public benefit–people all over the world can get access to books that they aren’t able to buy–but the copyright holders are understandably concerned.  Just because something’s out of print now doesn’t mean it will be out of print forever–except that once it’s in Google Books, it’s probably less likely to get reprinted.  This is part of the academic objection, since many of our faculty write books which could end up in Google Books.  I think it’s a real effect, but I also think that many faculty would choose to have more readers of their work, even if it meant fewer actual book sales.

The concern that Google will disadvantage universities the way that the journal companies have, I think is largely unfounded.  It’s true that we don’t know what Google will do in the future, and corporate interests are often not aligned with academic interests.  But Google’s corporate philosophy is bound (no pun intended) to the concept of free content; I can’t imagine them charging thousands of dollars for content access to any of their properties, including Google Books.

Then there’s the underlying concern of the library, that Google Books and services like it could threaten the library by making it appear obsolete.  I think this is a real possibility, but it’s a real possibility no matter what happens with the Google Books situation. Right now, the technology to read books electronically is very immature and only marginally usable, but if someone comes up with a good e-book solution, the library as we know it will have to change radically to remain relevant.

We’ve received numerous reports from our customers about an apparent increase in spam in recent weeks.  Indeed, looking at spam statistics shows that spam activity has risen significantly in the past couple of months; see the charts at MessageLabs, which are complete through March.  I’m guessing that April will be even higher.  They list the spam rate as 75.7%; over three-quarters of all the mail sent on the Internet is spam.

Now, our spam filters are generally pretty good about catching spam; probably 80-90% of the total spam volume is caught before users see it.  But spam protection is always an arms race, and right now the spammers have come up with some new techniques which are succeeding at getting around many of the spam filters.  Getting around spam filters is a bounded problem; blocking spam is an unbounded problem.

We have a page with some suggestions on how to deal with spam.  One of those is that you can forward a spam message to spam@berkeley.edu to report it as spam; Calmail will use your message to help improve their spam filters.  (The message has to be forwarded as an attachment).

We’re also in the process of moving all of our mail service over to Calmail.  Calmail allows us to host a mail domain such as LS.berkeley.edu on their own servers.  They’re a much bigger operation, and they have many more resources to put into spam protection.  We’ve moved the math.berkeley.edu domain over, and are initiating projects to migrate the rest of the domains we run on departmental servers.  We expect that migrating to Calmail will reduce everyone’s spam counts, and provide more reliable service as well.  (See my post on high availability).

Universities everywhere are seeing pressure to adopt “cloud computing” services.  Cloud computing is a general class of application, also called “Software As A Service (SAAS)”, where a third-party vendor offers a web-based application service instead of a traditional desktop-based application.  An example that everyone is familiar with is gmail–to use gmail, you don’t need to install anything on your computer except a web browser.  The service is fully portable (you can get it from anywhere), it usually lacks platform dependencies, and in most cases it’s free or very inexpensive.  Google is offering universities the option to use gmail for their student email at no cost to the institution; on its surface, that option looks very attractive.  Google has a number of other cloud-based services, notably Google Docs, which offer great functionality at no or low cost.  Microsoft, Yahoo, and Amazon also offer cloud-based services, and a number of smaller vendors, such as Salesforce.com (more on Salesforce below) offer more targeted applications via cloud infrastructure.

So what’s the downside?  The reason it’s called cloud computing is that the application and the data have no specific location; the servers can be located anywhere in the world, and data backup is handled by storing data in multiple locations.  The problem this causes is that there’s very little control over what happens to data stored in the cloud; when we have legal or policy requirements to protect data security or privacy, it is often difficult or impossible to get assurances from vendors that the data will be handled according to our requirements.  This can put us at risk for audits or lawsuits.

The campus is now providing guidance on outsourcing.  The key part of the new policy is:

Before “sourcing” your technology offsite — campus individuals, departments, managers, and support staff must consider risks to the following:

• privacy and confidentiality of personal, sensitive, or restricted information
• availability of business data and electronic communications (e.g. backup retrieval, evidence for legal disputes)
• cyber security and support for forensics
• access to records in the event a company is acquired or goes out of business

When you process, store, or otherwise use University information (including information about colleagues, research subjects, correspondents, customers, etc.) in an off-campus site, legal and business consequences need to be expertly reviewed, documented in writing, and must be accepted or modified by an authorized individual for your department or the Campus.

The standard agreements a user might click through to sign up for a free service normally do not provide protection to the university in these areas–in fact, they usually explicitly waive our rights to protection and indemnify the vendor from harm.  It is important to consider the implications of conducting university business through cloud services.

That being said, the services offered are in some cases compelling, and the campus is interested in enabling access to them.  One example is the new agreement we’ve signed with Salesforce.com.

Salesforce is a company that started out providing Customer Relations Management (CRM) software as a cloud service, but now has expanded to offer a development platform where organizations and third parties can build applications related to tracking information about customers.  IST is deploying Salesforce to start keeping track of all of its customers–with any luck, their implementation will lead to a better shopping cart and better billing system.  Departments might be interested in using Saleseforce to track alumni, or current students.  LSCR will consider whether it makes sense for us as well.

The agreement the campus has signed verifies that Salesforce meets our criteria for data protection and liability.  Departments who want to try it out can sign on to the umbrella agreement and know they’re within campus policy and recommended practice.  I’m hoping to see similar agreements in the future for Google and other cloud vendors.  For now, if you have interest in using cloud services for university business, feel free to contact me for guidance.

Per Kathleen Valerio’s message, CalPACT and LearnIT are offering free 90-minute demos of Microsoft Office 2007, on March 24, 25 and 26.  (Sign up through the UCB Learning Center on blu).

Office 2007 for Windows, and Office 2008 for the Mac, have interface changes which most users will find disconcerting.  I recently moved to Office 2008 myself, and it took me a bit of poking around to figure out how to return the environment to something I could reasonably work in.  (For a start, close the toolbox, enable the Formatting toolbar, and view in Normal or Draft mode).

The new Office also has a completely revised file format–you’ve probably already received “docx” or “xlsx” files which need to be converted to be read on older versions of Office.  The new file formats are actually a lot better–they are XML-based, which means they’re simpler, more extensible, and should be less prone to corruption.  But, the change will definitely cause problems for collaborators.

Those who use Word or Excel regularly will probably find it useful to attend one of these sessions.

Departmental security contacts today received a message from SNS entitled “Report on Host Vulnerabilities.”  This report was generated by SNS’s Foundstone scanner, which looks at every machine on campus and tries to determine if any network security vulnerabilities exist.  To do this, the scanner connects to the machine to see what services are running, and, to the extent that it is possible, tries to determine what version of the software is running for each service.  It then compares those version numbers against a database of known vulnerabilities, and generates a report as follows:

128.32.x.x (hostname1.Berkeley.EDU) MAC: Not available

  Severity:    High           CVE-2000-0923   Aplio IP Phone authenticate.cgi Command Execution

128.32.x.x (hostname2.Berkeley.EDU) MAC: Not available

  Severity:    Medium         CVE-1999-0517   SNMP Default Community Name

  Severity:    Medium         No CVE-ID       HP Printer FTP Access

128.32.x.x (hostname3.Berkeley.EDU) MAC: Not available

  Severity:    High           CVE-2006-3747   Apache HTTP Server mod_rewrite Vulnerability

  Severity:    Medium         CVE-2007-5000   Apache mod_imap Module Vulnerability

There are three components in the report; a description of the supposed vulnerability, a reference number which you can use to look up the vulnerability and (in some cases) learn how to fix it, and a subjective rating on the severity of the vulnerability.

The methods Foundstone uses to determine the services and versions are not entirely reliable, so these reports often have a number of false positives.  The version numbers are usually self-reported by the applications, and they can vary from platform to platform.  In addition, the severity cannot take into consideration your local configuration, which can mean that you may be running an application in which a vulnerability exists, but which cannot be exploited on your configuration.  Foundstone is working with very limited information.

So, what do you do when you get one of these reports?  First, you have to find the machines and figure out who they belong to; these messages are going to the overall security contact for the department, so some departments will get large reports referencing many different machines.  Departments need to be able to track down the machine owners, even for machines which are being managed by PIs or grad students.  Many departments do not currently have good mechanisms for locating machines by IP address; that’s a problem.

Once you’ve found the machine, the owner needs to sanity-check the report.  Often, Foundstone scans will discover services running on machines that the owner was unaware of; check if there is really a service running, and if so, whether it’s really necessary.  Disable it if you don’t need it.  Check if the machine is set up to automatically receive software updates; if not, configure it to do so.

Today’s reports include a number of non-traditional computing devices like copiers and printers.  These devices, as they become more and more sophisticated, also become increasingly vulnerable to security problems.  The “hostname2.Berkeley.EDU” machine referenced above is probably an HP printer with a default network configuration.  In most cases, the services listed on the report can be shut down, but it can depend on how you’re using the machine.  You’ll have to contact your local IT folks, or your copier/printer manufacturer to look into it.

SNS is planning to run these scans on a regular basis; we hope that the reports continue to improve in accuracy (they’re way better now than when we first saw them), and that the security and configuration of our machines also continues to improve.

In the past couple of weeks, two servers that LSCR manages have had serious hardware problems. One was a new server located in the campus data center; the other was a very old server located in the basement of Campbell Hall.  In both cases, users experienced functional problems and downtime over an extended period.  Coincidentally, both problems were related to the machine’s disk controller.

Server-class computers are generally quite reliable.  Disk drives are the most common hardware failure, and even those have MTBF (Mean Time Between Failures) of 300,000 hours or more.  That’s a pretty big number (about 35 years), although when you consider that a major server often has 12 or more disks, you can see that a typical server has a decent chance of seeing a disk failure during its normal lifetime.  We build in redundancy for disk failures; using RAID (Redundant Array of Inexpensive Disks), we can install an array of disks and set them up so that no single disk failure will take the server down or cause data loss.  Similarly, we use redundant power supplies on different circuits to avoid local power problems; the campus data center has both a UPS (Uninterruptible Power Supply) to handle short power grid outages, and a diesel generator to handle longer outages.  It can keep running as long as there are still trucks to deliver gas.

That level of redundancy brings us up to something like 99.9% uptime.  99.9% (referred to in the industry as “three nines”) sounds like a lot, but it’s equivalent to having your server down for a little more than an hour a month, or one full day a year.  When that downtime is planned, it’s not too bad, but when it’s unplanned, it can be a huge disruption to the departments using our servers.

“High Availability” is an industry term generally used to refer to systems designed for availability of “three nines” or above.  To get to “four nines” (99.99% uptime, 1 hour of downtime per year) or “five nines (99.999% uptime, 5 minutes of downtime per year) requires a much larger investment in hardware.  A typical configuration will include wholly redundant hardware, including spare servers that don’t do anything except wait for another server to fail.  In front of that might sit a hardware load balancer, which makes the multiple machines look like one server to the outside world.  Then you might have redundant network paths, with two or more different Ethernet connections going to two or more different routers, which have different fiber-optic connections to different service providers.

With all this stuff, you have to evaluate how much it would cost relative to how much additional uptime you would gain.  For our operation, we don’t really have the funding to go above three nines, and in most cases it’s not really necessary; there are campus services which provide higher availability for someone who needs four or five nines.  This is why we can offer a free web hosting service to departments, when IST’s service costs $30/month; IST’s service has a more robust (and therefore more expensive) infrastructure that we can’t hope to replicate on the cheap.

We will continue to look for ways to make our servers more reliable, and to improve our disaster recovery procedures.  In these cases, if we had migrated the files on the servers to our NetApp storage device, we could have relatively easily brought the services back up on a different piece of hardware.  However, our NetApp itself isn’t designed for high availability–it has redundant disk, but the controller is a single point of failure.  This is an example of the kind of thing you have to deal with to build a highly available system; not only do you have to build redundancy into all of your hardware, but you have to build redundancy into everything it connects to, also.  Otherwise you’re just moving the point of failure.

Both of our servers are back running normally right now.  We’re accelerating our migration off the old one, and trying to improve our recovery procedures on the new one.  Unfortunately, the fact that we’ve already had 8 hours of downtime this year doesn’t mean it can’t happen again; all we can do is learn from the history and try to plan for the next problem.

Everyone at Berkeley and at other universities is seeing more messages of this sort:

We are currently carrying-out a mentainace process to your berkeley.edu account, to complete this process you must reply to this email immediately, and enter your User Name here (———-) And Password here(———-)  if you are the rightful owner of this account.
This process we help us to fight against spam mails.Failure to summit your password, will render your email address in-active from our database.

You can also confirm your email address by logging into your account at:https://calmail.berkeley.edu/

NOTE: You will be send a password reset messenge in the 48Hrs working days after undergoing this process for security reasons.
Your response should be sent to admin manager
Email: customer.careservice@live.com

Thank you for using berkeley.edu!
THE BERKELEY.EDU TEAM

As you’re probably aware, this message was not sent by anyone at Berkeley; it is an attempt at “spear phishing”–targeted messages sent to specific populations to attempt to trick them into giving up information they shouldn’t, such as credit card numbers, social security numbers, or account passwords.  In this case, the phisher (who appeared to be located in Singapore) figured out the URL of our webmail interface, and a couple of other details about the campus environment, to try to make the message more convincing.  Usually a few people get caught every time the phishers change tactics.  Typically what they’ll do is use the compromised password to send more phishing messages and a ton of spam.

The setup at Calmail does a pretty good job of blocking these messages, but because the messages keep changing, and can come from anywhere on the planet, there is always a window of vulnerability when the phishers come up with something new.

Remember:

• Never send your bank account or Social Security number through email
• System administrators will never ask you for your password through email.  If you get a request for your password, you should know it’s bogus.
• If you get a questionable message, check out the From: and the Reply-To: addresses.  Phishers will try to disguise the message to make it look like it’s coming from a local person, but if you look closely you’ll see that you’re responding to someone in another country or at a yahoo.com address.

In the past few weeks on campus, many users and mailing lists have been affected by a spam-related phenomenon known as “backscatter.”  Backscatter occurs when a spammer sends out a bunch of mail with a forged, but legitimate “From” address.  When they do this, servers which reject the mail often bounce the message back to the sender listed in the “From” field.  The result is that a person or mailing list which really had nothing to do with sending out the spam can get dozens or hundreds of bounce messages related to it.

The tactic is used primarily because mail with a legitimate From address is more likely to get through spam filters.  In general, the spammers are not targeting any individual or institution; they’re just doing whatever they can to improve their chances of having their messages delivered.

Users who are victimized by large amounts of backscatter often worry that their computer was broken into, or that they  have a virus.  Generally, backscatter does not indicate any problems with your own computer or mail server.  There have been some cases where a virus sent out messages designed to look like backscatter, with the virus payload as an attachment to the message, but even these cases were not a problem for users unless they clicked on the attachment.

As with most spam issues, backscatter is a pernicious problem.  When we send out a legitimate email that doesn’t get through, we want to get a bounce message that informs us of the problem, so we can resend or readdress the message.  It’s quite difficult for mail servers to tell the difference between a legitimate and an illegitimate message, so as long as mail servers are configured to deliver bounce messages, and as long as spammers are still spamming, backscatter will continue to occur.  We are looking at moving more of our mail services to the CalMail domain hosting environment; CalMail has better spam protection than we can easily implement at the departmental level, including better protection against backscatter.  Unfortunately, there is no magic bullet; CalMail users also experience spam and backscatter problems, though generally with less frequency than our other mail server users.

For now, our best weapon in the spam wars  remains the same; take a deep breath, let it go, and hit the delete key.