Director's Blog

As noted in an earlier blog entry, the campus is moving towards an FTE-based model for network funding.  Thursday I participated in the latest meeting of the advisory committee, and we got a lot more detail about the plan.

The biggest news is that implementation has been delayed until January 2010.  The committee and the Cabinet are in agreement that the plan is not ready for implementation this coming July 1; there are too many open questions and logistical problems.  So for at least the first six months of the fiscal year, network charges will be the same as they currently are.

Also big news for certain L&S departments is that completion of campus infrastructure projects, including ubiquitous AirBears, is included in the funding model.  That means that some of the buildings with the worst networking, such as Tolman, Wheeler, and Kroeber, will receive network upgrades as part of the adoption of the new model.  Timelines are unclear, and there will probably still be a required departmental contribution for new horizontal cables, but this is definitely good news for our most underserved departments.

Undergraduate students are now included in the funding model at a 0.15 FSE rate.  The proposal is to pay for students via a student technology fee; the campus is working with UCOP (and probably going to the Regents) to implement such a fee on the January 2010 timeframe.  [Not knowing anything about the politics involved, I'd have to say that timeframe looks unrealistic to me, but we'll see.]

The inclusion of students in the model has brought the cost per FSE down below $40/month/FSE, from original projections of $45/month.  The figure we are currently seeing is $38.08/month; that is subject to change due to negotiations over FSE counts, but it should remain below $40.

The committee’s strong preference is to use the existing $1.3M in funding allocation to pay the monthly fees for academic titles and GSIs.  It appears that $1.3M is roughly equivalent to the amount needed to cover those groups, and it looks to me like that’s the direction the campus is going to take.

The overall impact to L&S will be significant in any case.  According to draft figures shared with us, L&S departments in total are currently spending approximately $14K/month for network charges; under the new model (with no funding allocations included), L&S costs rise to $104K/month–an increase of $90K/month, $1.08M/year.  If L&S garners half of the $1.3M funding allocation (a reasonable guess based on faculty FTE counts), that still leaves us with additional costs of nearly $500K/year.

They gave us projected cost breakdowns by department; here are some examples of monthly costs from a large and a small department in each  division (again, this is before any funding is allocated):

• MCB: Current $969, projected $18,084, increase $17,115/month
• IB: Current $1,230, projected $7,501, increase $6,271/month
• Physics: Current $1,486, projected $7,947, increase $6,461/month
• Statistics: Current $192, projected $1,656, increase $1,447/month
• History: Current $410, projected $4,128, increase $3,718/month
• Geography: Current $96, projected $912, increase $816/month
• English: Current $297, projected $3,577, increase $3,280/month
• Scandinavian: Current $56, projected $443, increase $387/month
• UGIS: Current $306, projected $2,495, increase $2,189/month
• L&S Advising: Current $198, projected $1,644, increase $1,447/month

It’s obvious that impacts at these levels would be devestating to departments.  Funding allocation should bring the direct costs down quite a bit; my guess is that the proposal would cut 50-75% off the above numbers for most academic departments, depending on the makeup of departmental FTE.  Departments whose FTE are mostly in academic titles and GSIs would benefit the most from the proposed funding allocations; departments with many administrative staff and GSR positions would benefit less.  Fully administrative departments (such as mine) will likely be paying the full costs.  Even with funding allocations, many departments will struggle to find funding to pay these charges.

Thursday’s meeting is the first time the committee discussed the logistics of implementing this model.  I brought up the example of MCB, our largest department.  MCB has 475 FSE under the model, accounting for possibly 1000 people; the department would probably want to allocate network charges to 100 or more chart strings.  Imagine a scenario where the department manager gets a list of 800 people and has to manually specify the chart string (or multiple chart strings) for each one of them; it sounds ugly.  One possibility for simplifying the logistics would be to implement a GAEL-style payroll charge; the network would be charged to the same chart string as the individual’s payroll.  For some departments this would work well; for others it would still be a major headache.  We’re very early in the process of deciding how the model will be implemented; please let me know any comments you have, and I will continue to keep you informed.

Universities everywhere are seeing pressure to adopt “cloud computing” services.  Cloud computing is a general class of application, also called “Software As A Service (SAAS)”, where a third-party vendor offers a web-based application service instead of a traditional desktop-based application.  An example that everyone is familiar with is gmail–to use gmail, you don’t need to install anything on your computer except a web browser.  The service is fully portable (you can get it from anywhere), it usually lacks platform dependencies, and in most cases it’s free or very inexpensive.  Google is offering universities the option to use gmail for their student email at no cost to the institution; on its surface, that option looks very attractive.  Google has a number of other cloud-based services, notably Google Docs, which offer great functionality at no or low cost.  Microsoft, Yahoo, and Amazon also offer cloud-based services, and a number of smaller vendors, such as Salesforce.com (more on Salesforce below) offer more targeted applications via cloud infrastructure.

So what’s the downside?  The reason it’s called cloud computing is that the application and the data have no specific location; the servers can be located anywhere in the world, and data backup is handled by storing data in multiple locations.  The problem this causes is that there’s very little control over what happens to data stored in the cloud; when we have legal or policy requirements to protect data security or privacy, it is often difficult or impossible to get assurances from vendors that the data will be handled according to our requirements.  This can put us at risk for audits or lawsuits.

The campus is now providing guidance on outsourcing.  The key part of the new policy is:

Before “sourcing” your technology offsite — campus individuals, departments, managers, and support staff must consider risks to the following:

• privacy and confidentiality of personal, sensitive, or restricted information
• availability of business data and electronic communications (e.g. backup retrieval, evidence for legal disputes)
• cyber security and support for forensics
• access to records in the event a company is acquired or goes out of business

When you process, store, or otherwise use University information (including information about colleagues, research subjects, correspondents, customers, etc.) in an off-campus site, legal and business consequences need to be expertly reviewed, documented in writing, and must be accepted or modified by an authorized individual for your department or the Campus.

The standard agreements a user might click through to sign up for a free service normally do not provide protection to the university in these areas–in fact, they usually explicitly waive our rights to protection and indemnify the vendor from harm.  It is important to consider the implications of conducting university business through cloud services.

That being said, the services offered are in some cases compelling, and the campus is interested in enabling access to them.  One example is the new agreement we’ve signed with Salesforce.com.

Salesforce is a company that started out providing Customer Relations Management (CRM) software as a cloud service, but now has expanded to offer a development platform where organizations and third parties can build applications related to tracking information about customers.  IST is deploying Salesforce to start keeping track of all of its customers–with any luck, their implementation will lead to a better shopping cart and better billing system.  Departments might be interested in using Saleseforce to track alumni, or current students.  LSCR will consider whether it makes sense for us as well.

The agreement the campus has signed verifies that Salesforce meets our criteria for data protection and liability.  Departments who want to try it out can sign on to the umbrella agreement and know they’re within campus policy and recommended practice.  I’m hoping to see similar agreements in the future for Google and other cloud vendors.  For now, if you have interest in using cloud services for university business, feel free to contact me for guidance.

Per Kathleen Valerio’s message, CalPACT and LearnIT are offering free 90-minute demos of Microsoft Office 2007, on March 24, 25 and 26.  (Sign up through the UCB Learning Center on blu).

Office 2007 for Windows, and Office 2008 for the Mac, have interface changes which most users will find disconcerting.  I recently moved to Office 2008 myself, and it took me a bit of poking around to figure out how to return the environment to something I could reasonably work in.  (For a start, close the toolbox, enable the Formatting toolbar, and view in Normal or Draft mode).

The new Office also has a completely revised file format–you’ve probably already received “docx” or “xlsx” files which need to be converted to be read on older versions of Office.  The new file formats are actually a lot better–they are XML-based, which means they’re simpler, more extensible, and should be less prone to corruption.  But, the change will definitely cause problems for collaborators.

Those who use Word or Excel regularly will probably find it useful to attend one of these sessions.

For the past year or so I’ve been sitting on the advisory committee which is providing input for a project to totally change in how the campus network is funded.  Our current node-based network funding model has a number of major problems: the two most significant of those are that the charges don’t map well onto actual network costs (among other issues, wireless service isn’t included in the model at all), and that the model was never truly funded.  Originally, the model had the campus contributing permanent money to fund a “node bank,” which essentially grandfathered in all existing network connections as of July 1, 2000.  The campus never contributed that money; the chancellor has been covering some of the expense with temporary funding (which he wants to stop doing), and even so, the network has been underfunded.

Before you start throwing things–and when you read the rest of this post, you will likely want to throw things–please know that the advisory committee does not have any authority to change the basic parameters of the project, which are that the model should allow for direct charges to grants, be more or less equally charged to all departments, and be implemented by July 1, 2009.

We looked at what other similar institutions are doing, and the strong consensus is that higher ed institutions are moving to headcount, FTE, or FSE (full service equivalent, with a multiplier for non-communication workers and/or students) models.  A major advantage of an FTE-based model is that it is technology-neutral.  Apple introduced their first AirPort wireless access point, which was the first major success of Wi-Fi networking, just three weeks after the campus decided on a node-based model for network funding.  We cannot predict what new network technologies will arrive in the next five years–but we can surely predict that there will be some, and that implementing them will have cost implications.  Our funding model needs to be flexible as technology changes.

The FTE model also is neutral with respect to user behavior.  Our current node-based model encourages undesirable user behavior, such as using AirBears instead of wired connections to avoid network charges, or connecting network switches or hubs to a single wall connection, and running network cables down hallways or poking holes in walls.  (If you can imagine it, we’ve seen it done.)  When charges are FTE-based, networks can be built based on what works best, technologically, as opposed to what incurs the least cost based on a faulty model.

So, in theory I’m in favor of moving to an FTE-based funding model, and it seems clear that the campus is committed to do so at this point.  However, the devil is in the details, and today we were told there are two major details which still need to be worked out: defining “FTE”, and deciding on subsidy (or “funding allocation” as the budget office would have us term it.)

The advisory committee had come to a decision on the FTE definition which would include all faculty and staff (including “non-knowledge-worker” staff such as gardeners), all GSI and GSR positions, and a partial cost for students living in the dorms.  Undergraduates, including undergraduate student workers, would not be included.  When the plan was brought to the cabinet, there was push-back on the non-knowledge-worker issue (Facilities wants a multiplier for them), and on the student issue (the Recharge Committee wants them all included, ostensibly to comply with federal regulations).  There are ongoing discussions on exactly how the FTE/FSE model would shake out, and what the multipliers would be for different types of users.  Those discussions should be concluded reasonably quickly.

The bigger issue is subsidy.  There is a sizable amount of subsidy in the network right now, but most of it has been allocated on a yearly basis from temporary emergency funds.  The advisory committee is requesting $4 million in permanent funds be allocated to offset the impact of the network charges; in the current budget climate, it is not clear what will come of that request.  Without any funding allocation, the network charges will come out to something in the neighborhood of $45 per FTE per month ($540 per year).

Most L&S departments are currently paying very low, or zero per-month network fees.  Because all existing connections from July 1, 2000 were grandfathered in, and most network growth since that time has been wireless, departments have been able to manage their node banks to avoid charges.  Under the FTE-based model, these departments would begin to have to pay.  A  department with 10 staff, 50 faculty, and 40 GSI/GSR/student FTE could go from paying zero now, to paying $54,000/year for network charges.  Even a small department with 5 staff, 10 faculty, and 5 GSI/GSI/students could be hit with over $10,000 in ongoing yearly charges.  And as noted above, the campus plans to roll this out as of July 1 this year.  Given the above costs, it seems impossible that the roll out could successfully occur; subsidy will be required to plausibly implement this plan for the coming fiscal year.  (Or really, for any other fiscal year).

As of today, we are not close to agreement on the size of the funding allocation, or how it should be allocated.  One plan has the $4 million in requested permanent funding going to subsidize all academic titles (faculty/GSI/GSR); in the above scenarios, that would reduce the budget impact for those two departments to something like $6,000 and $3,000/year, respectively (staff FTE plus student worker FTE).  So even if $4 million in funding is allocated, there will still be a significant impact to many departmental budgets–and there are several on the committee who doubt that $4 million in funding allocation is possible.  There is some talk of a student fee, but serious doubts about whether the students would vote for it in referendum.

The financial people are going back to run another set of numbers.  Soon, Shel Waggener will talk to the Council of Deans and the Academic Senate about implementing this plan.  I expect there will be significant resistance from both bodies, and I don’t have a good sense of what the final result will be.  I can’t imagine how departments could take this kind of budget hit, in the midst of all the other budget cutting they are having to do.

I will do what I can to keep people informed.

Departmental security contacts today received a message from SNS entitled “Report on Host Vulnerabilities.”  This report was generated by SNS’s Foundstone scanner, which looks at every machine on campus and tries to determine if any network security vulnerabilities exist.  To do this, the scanner connects to the machine to see what services are running, and, to the extent that it is possible, tries to determine what version of the software is running for each service.  It then compares those version numbers against a database of known vulnerabilities, and generates a report as follows:

128.32.x.x (hostname1.Berkeley.EDU) MAC: Not available

  Severity:    High           CVE-2000-0923   Aplio IP Phone authenticate.cgi Command Execution

128.32.x.x (hostname2.Berkeley.EDU) MAC: Not available

  Severity:    Medium         CVE-1999-0517   SNMP Default Community Name

  Severity:    Medium         No CVE-ID       HP Printer FTP Access

128.32.x.x (hostname3.Berkeley.EDU) MAC: Not available

  Severity:    High           CVE-2006-3747   Apache HTTP Server mod_rewrite Vulnerability

  Severity:    Medium         CVE-2007-5000   Apache mod_imap Module Vulnerability

There are three components in the report; a description of the supposed vulnerability, a reference number which you can use to look up the vulnerability and (in some cases) learn how to fix it, and a subjective rating on the severity of the vulnerability.

The methods Foundstone uses to determine the services and versions are not entirely reliable, so these reports often have a number of false positives.  The version numbers are usually self-reported by the applications, and they can vary from platform to platform.  In addition, the severity cannot take into consideration your local configuration, which can mean that you may be running an application in which a vulnerability exists, but which cannot be exploited on your configuration.  Foundstone is working with very limited information.

So, what do you do when you get one of these reports?  First, you have to find the machines and figure out who they belong to; these messages are going to the overall security contact for the department, so some departments will get large reports referencing many different machines.  Departments need to be able to track down the machine owners, even for machines which are being managed by PIs or grad students.  Many departments do not currently have good mechanisms for locating machines by IP address; that’s a problem.

Once you’ve found the machine, the owner needs to sanity-check the report.  Often, Foundstone scans will discover services running on machines that the owner was unaware of; check if there is really a service running, and if so, whether it’s really necessary.  Disable it if you don’t need it.  Check if the machine is set up to automatically receive software updates; if not, configure it to do so.

Today’s reports include a number of non-traditional computing devices like copiers and printers.  These devices, as they become more and more sophisticated, also become increasingly vulnerable to security problems.  The “hostname2.Berkeley.EDU” machine referenced above is probably an HP printer with a default network configuration.  In most cases, the services listed on the report can be shut down, but it can depend on how you’re using the machine.  You’ll have to contact your local IT folks, or your copier/printer manufacturer to look into it.

SNS is planning to run these scans on a regular basis; we hope that the reports continue to improve in accuracy (they’re way better now than when we first saw them), and that the security and configuration of our machines also continues to improve.

In the past couple of weeks, two servers that LSCR manages have had serious hardware problems. One was a new server located in the campus data center; the other was a very old server located in the basement of Campbell Hall.  In both cases, users experienced functional problems and downtime over an extended period.  Coincidentally, both problems were related to the machine’s disk controller.

Server-class computers are generally quite reliable.  Disk drives are the most common hardware failure, and even those have MTBF (Mean Time Between Failures) of 300,000 hours or more.  That’s a pretty big number (about 35 years), although when you consider that a major server often has 12 or more disks, you can see that a typical server has a decent chance of seeing a disk failure during its normal lifetime.  We build in redundancy for disk failures; using RAID (Redundant Array of Inexpensive Disks), we can install an array of disks and set them up so that no single disk failure will take the server down or cause data loss.  Similarly, we use redundant power supplies on different circuits to avoid local power problems; the campus data center has both a UPS (Uninterruptible Power Supply) to handle short power grid outages, and a diesel generator to handle longer outages.  It can keep running as long as there are still trucks to deliver gas.

That level of redundancy brings us up to something like 99.9% uptime.  99.9% (referred to in the industry as “three nines”) sounds like a lot, but it’s equivalent to having your server down for a little more than an hour a month, or one full day a year.  When that downtime is planned, it’s not too bad, but when it’s unplanned, it can be a huge disruption to the departments using our servers.

“High Availability” is an industry term generally used to refer to systems designed for availability of “three nines” or above.  To get to “four nines” (99.99% uptime, 1 hour of downtime per year) or “five nines (99.999% uptime, 5 minutes of downtime per year) requires a much larger investment in hardware.  A typical configuration will include wholly redundant hardware, including spare servers that don’t do anything except wait for another server to fail.  In front of that might sit a hardware load balancer, which makes the multiple machines look like one server to the outside world.  Then you might have redundant network paths, with two or more different Ethernet connections going to two or more different routers, which have different fiber-optic connections to different service providers.

With all this stuff, you have to evaluate how much it would cost relative to how much additional uptime you would gain.  For our operation, we don’t really have the funding to go above three nines, and in most cases it’s not really necessary; there are campus services which provide higher availability for someone who needs four or five nines.  This is why we can offer a free web hosting service to departments, when IST’s service costs $30/month; IST’s service has a more robust (and therefore more expensive) infrastructure that we can’t hope to replicate on the cheap.

We will continue to look for ways to make our servers more reliable, and to improve our disaster recovery procedures.  In these cases, if we had migrated the files on the servers to our NetApp storage device, we could have relatively easily brought the services back up on a different piece of hardware.  However, our NetApp itself isn’t designed for high availability–it has redundant disk, but the controller is a single point of failure.  This is an example of the kind of thing you have to deal with to build a highly available system; not only do you have to build redundancy into all of your hardware, but you have to build redundancy into everything it connects to, also.  Otherwise you’re just moving the point of failure.

Both of our servers are back running normally right now.  We’re accelerating our migration off the old one, and trying to improve our recovery procedures on the new one.  Unfortunately, the fact that we’ve already had 8 hours of downtime this year doesn’t mean it can’t happen again; all we can do is learn from the history and try to plan for the next problem.

Last month, Susan Tobes of IST’s Technical Account Management (TAM) group announced that IST-TAM would be sponsoring one or two internship positions within TAM.  I’m pleased to announce that Jeanette Robinson has been selected for one of the positions. She’ll be helping IST develop consistent Service Level Agreements and doing comparative analsyes for the services they offer. The internship is a 40% position, running through the end of April.

Jeanette is the head of our “Tilden” desktop support team, and also manages our Windows servers, among other responsibilities.  While Jeanette’s LSCR time is reduced, Farnaz Stosik will be helping out with day-to-day management issues on the Tilden team, and we’ll be identifying other folks to assist with Windows server issues.

This is a great opportunity for Jeanette, and it will help us build a closer relationship with the TAM group and with IST in general.

Congratulations to Jeanette!

As I noted in a blog post back in March, IST is ending support for socrates.  Since that posting, I have been participating on the Socrates/Arachne Abatement Steering Committee, trying to find a way to remove those servers without adversely affecting teaching and research within L&S.  I am pleased to note that IST has been responsive to our concerns, and has come up with some plans which should meet the needs of nearly all of our socrates users.

IST’s information about the abatement project is available at http://ist.berkeley.edu/services/tam/serverabatement.html.  This project has been dragging on for a while, but now has a fairly aggressive timeline for finally removing those servers.  Most importantly, there is a user forum tomorrow (Tuesday, October 21) at 11:00 AM in Sibley Auditorium; if you have any questions or concerns about the abatement and migration projects, it would behoove you to attend and make sure your voice is heard.

The highlights of the new plan are:

• Web hosting will be outsourced to a third-party vendor (Dreamhost).  If you currently have a “tilde” account on socrates, and your site is simple HTML, you are eligible for free hosting on the outsourced service.  (Most sites on socrates, something like 80% of them, fall into this category).
• Users with more advanced web needs will be able to contract directly with Dreamhost for full-featured web hosting; IST will collaborate with Dreamhost to provide http://hostname.berkeley.edu URLs for your sites.  This hosting will be much cheaper than IST’s enterprise-level web hosting; something like $10/month.
• Merrill Shanks of the Social Sciences Computing Lab is developing a plan to provide access for socrates users or class accounts which need access to statistical and mathematical software.
• People who use pine to read email on socrates will be encouraged to migrate to a modern mail client (we recommend the free Thunderbird client).  There are also options to use pine on your own desktop computer if you really must.  (I recommend against that configuration, for support reasons).

As I write this message, socrates has been down for most of the day for unscheduled maintenance.  This highlights the fragile nature of the socrates system, and validates the aggressive timeline IST is pursuing for abating the system.  The current plan is to migrate all web users off socrates by December 19; if you are a socrates user, I highly recommend that you migrate as soon as the option is given to you.

Overall, I think this process has produced a result that is likely to be more positive both for IST and for the socrates users.  It is unfortunate that it took so long to get to this point, but on balance I am pleased with how IST has responded to our user demands.  Direct messages from individual faculty members played a key role in encouraging and helping develop a better solution.

LSCR has hired two new employees this month to fill vacant positions.

Craig Carlson joins us on the “Tilden” desktop support team.  Craig comes to us from UCSF, where he was working as the front-line desktop support person in the Pediatrics department.  (Coincidentally, our open position was created when Mical Wilson took a job at UCSF.  It’s all interconnected.)  Craig will initially be sitting with Mary Wielski in Wheeler Hall, and helping support the Tilden departments in Wheeler, Dwinelle, Barrows, and the PowerBar building, among others.

Ray Spence has joined our Unix team, replacing Julie Ashworth, who took a position at the Helen Wills Neuroscience Institute.  Ray comes to us from LBNL’s NERSC research computing facility, where he was in charge of some of NERSC’s infrastructural Unix systems.  Ray’s primary office will be in Evans Hall with Igor Savine, but he will also have space in Le Conte where he can hold office hours.  Ray’s primary responsibilities will be Unix systems outside the Math department, mostly in Physics and the biological sciences.

Peggy Huston posted a message today about changes to the campus’ network funding model.  The current model, which is based on a per-month, per-connection charge, has a number of signifcant problems.  The four main issues with the current model are:

• It fails to accurately track costs.  There are a number of reasons for this, but most importantly, wireless networking isn’t included in the cost model at all.
• It encourages undesirable behavior.  From a technical perspective, we would recommend that every desktop computer be connected via its own individual wired connection.  The per-port installation and monthly costs of the current model encourage departments to use other connection mechanisms, such as commodity hubs or wireless, which provide poorer service and false economy.
• It does not include the cost of upgrading legacy networking.  L&S has several buildings which are still using shared 10-megabit networking that was installed in the early 1990s.  The current funding model provides no way to replace those old, slow, unreliable networks.
• Subsidies are asymmetrical and insufficiently funded.  Each department which existed on June 30, 2000, has a certain number of nodes in its “node bank.”   Some departments are node-rich and others are node-poor, for reasons which are historical rather than .  Newly-created departments have no node bank and thus no subsidy.  But most importantly, the campus has never truly provided enough funding for the node bank, which has forced us to run the network in deficit.

Most universities are moving towards some kind of per-head model for network funding.  Charing by head (or by knowledge worker, or FTE or whatever) is attractive for a number of reasons.  Primarily, FTE models adapt much better to changes in technology; when our current node-based model was developed, it may have made sense given the technology we were using at the time, but now that wireless is a large and growing part of our network costs, our model no longer maps onto our costs.  FTE models can be adjusted and trued up as the technology changes.  Also, FTE models tend to be neutral in terms of their effects on user behavior; they don’t provide incentives to engage in bandit networking.

At this point, it looks like the campus is going to move to an FTE model that will include all staff and faculty FTE.  It appears that students will not be included in the FTE count.

The biggest issues remaining to be discussed are around subsidy; is how much will be involved, and how it will be implemented.  It is clear that L&S departments cannot absorb significant new charges without an offsetting addition of funding; I will continue to advocate on the network funding committee for full subsidy of baseline networking for all faculty and staff.  Cal Moore, as a faculty representative on the same committee, is also looking out for the interests of academic departments.

There should be some interesting discussions over the next few months; I will continue to provide updates as I have new information.