Director's Blog

IST has asked me to join a new group they’re forming called the Services Advisory Council, which will act as something of a public utilities commission overseeing IST’s service offerings.  This is a function that has been requested for a long time, dating back at least the Commission on Computing report led by Cal Moore.

I am hopeful that having some direct input on IST’s service offerings and delivery will give us some ability to influence the overall direction, and hopefully the quality of the services provided.

Many campus units are considering removing wired telephones as a cost-cutting measure.  Wired phones have certainly become less important as most of our communications move to email and cell phones, and the budget situation in most departments is severe enough that everyone is looking to save money where ever they can.  Some departments have already begun to disconnect their wired phones and replace them with internet-based solutions; many others are evaluating the idea.

Unfortunately, internet telephony is still immature relative to wired telephony, and there are concerns with any approach, many of which may not be apparent until after the decision has been made.

The biggest concern is in the area of life safety: 911 service from internet phones is different in important ways than 911 service from wired phones.  One major difference is that internet service generally doesn’t work during a power outage, while wired phones often continue functioning.  (This could be critical during an earthquake).  Also, internet or cell phone 911 service does not provide accurate location information to the emergency responders; with a wired phone, they will immediately know your exact building and room, while cell 911 gives only an approximate, 2-dimensional location, and internet phone service may provide no location information at all.

UCPD will soon come out with a policy which requires departments who’ve removed their wired phones to install some number of red emergency phones per floor.  Note that leaving a normal wired phone or two will not be enough to meet the requirements of this policy.  Imagine a scenario where there’s been an earthquake, or someone is having a heart attack; no one is going to think to run over to the fax machine to call 911.  Emergency phones, visible and colored, ideally with a signal light, will be required to address the safety issue.

There are other unexpected issues which arise from internet telephony.  For example, some departments are using a product called MagicJack which you may have seen advertised on late-night TV.  As a product on late-night TV, “the large print giveth, and the small print taketh away.”  The caveats with MagicJack are that you’re basically agreeing to allow them to install spyware on your computer, and to allow advertisers to know what phone numbers you’re dialing and target ads based on those phone numbers.  Creepy.  And at least one department is setting up a full-on Voice-Over-IP (VOIP) PBX, essentially becoming their own 24×7 telecom service provider.  It seems hard to imagine that that will really be a cost savings in the long run.

I won’t tell you not to pull out your wired phones; I know how bad the budget situation is for most departments.  But I will tell you to consider all the implications, particularly around life safety, and take appropriate steps to manage the risks involved.

Most of you have probably heard that Nancy Schimmelman, developer of the Our Unit system used by many L&S departments for faculty searches and graduate admissions (among other uses), was a victim of budget cuts in her department.  Because Nancy handled all of the programming and most of the support for Our Unit, her absence leaves a pretty big gap.

Fortunately, IST’s Applications Services division has agreed to take on support and development of Our Unit.  It will take them a while to figure out the code and deal with some of the issues related to supporting it centrally, so for now they’ve instituted a code freeze–no new features will be added until they understand things better.  You’ll still be able to use the existing system, and IST should be able to set up new instances for departments who aren’t currently using the system.  Our local team of Farnaz Stosik and Caroline Boyden will continue to provide user-level support for L&S folks.

I met with Bill Allison from IST-AS today, and he said that for at least the next 3-6 months, there will be no cost for using Our Unit.  Long-term, IST may require some recharge revenue for support and development of the system; there has not yet been a business model developed.  The goal will be to keep it “cheap.”

Nancy herself has landed on her feet; she’ll be doing IT management for Summer Sessions.

We’re now three weeks into LSCR’s reorganization, and things continue to move along.  It seems like most people have gotten used to using their new email addresses for support, and the feedback I’ve gotten from customers has been positive.  Internally, we’re still figuring out some of the coverage and escalation responsibilities. We’re doing fine with our current workload, but we anticipate a few glitches when things get busy at the start of the semester.  We hope you’ll be patient with us as we work on our new procedures.

Next Wednesday, July 29, we’ll be getting our whole group together for a team-building exercise, which I think is an important thing to do at this point in our process.  We’re forming a completely new teams, and need to get them to be as functional as our old teams were.

So, for the afternoon of 7/29, we’ll have only limited coverage for customer calls.  We will have a skeleton crew available to handle emergencies; non-emergencies will likely be deferred to Thursday morning.

Let me know if this poses a problem for you or your department.  Thanks for your continued support.

I’ve been working for several years now on a new strategic plan for computing in L&S, and I’m pleased to announce that the first major change related to that plan is due to take effect July 1.

We got a ton of input from our current customers and others within L&S, through surveys, committee discussions, and in-person interviews.  One thing we heard loud and clear was, “I want my own person.”  L&S folks want to get to know a specific IT person in their department, who understands their computing needs, and who they can grab in an emergency.  They want someone who gives them advice and help managing their IT, not just someone who shows up to fix problems.  They don’t want to submit requests to a faceless help desk system, and they don’t want to have nickle and dime charges every time they ask for something new or different.  Departments want a partner, not a consultant.

LSCR’s current structure and funding model will need to adapt to the way that our customers’ needs have changed.  We are perfectly structured to meet the technology needs of 1995: 2009’s demands will force us to look at ourselves completely differently.

We will take the first step on July 1.  We are not yet prepared to change our financial model–we’ll be looking at that during the coming fiscal year–but we are ready to implement significant changes in the way we interact with our customers.  Here’s how it will work:

For each department we’re supporting, we’re going to create a support team.  That team will work a little like an account team from a vendor; you’ll have one person who’ll be your lead partner, your first contact for any IT request.  In many cases, that person will be physically located in your department or your building (especially if you’re able to provide space for him or her), and he or she will handle requests from both staff and faculty.  Your partner will be backed up by technical experts in all the areas you need tech support; we’ll identify a Mac lead, PC lead, and web, database, or Unix leads depending on the department’s needs.  The goal will be to have one single person handling the bulk of requests in your department, while giving that person the ability to call upon the wide range of skills and experience that LSCR has, in order to solve customer problems.

We will set up a single departmental mailing list, such as “english@LSCR.berkeley.edu,” which will be your primary way of requesting support of any type.  Everyone on your account team will receive your department’s emails, and jobs will be tracked in a queue specific to your department.

I think this plan will provide better communication, clearer responsibilities, and stronger relationships between LSCR and the folks we support.

This reorganization is not related to the budget difficulties we are all facing; we are implementing this functional change without changing our financial model.  That means that for the 09-10 fiscal year, you can expect little to no change in your LSCR recharges compared with 08-09.  Like all departments, LSCR is likely to be affected by budget cuts and salary reductions or mandatory furloughs; the exact impact is not yet known.  We will keep our customers informed when we have any useful information.

We will soon be contacting each department individually to introduce the account team and provide more specifics about how the new scheme will work.

Web site security got some press last month when it was revealed that hackers had broken into a system at the Tang Center and stolen thousands of Social Security numbers.  The mechanism for that break-in was a “SQL injection attack“; without getting too geeky, SQL injection is an all-too-common vulnerability, caused when your application doesn’t check the input it’s receiving from the user.  If the application doesn’t check the input, it’s often possible to trick the system into doing something it’s not supposed to do.

The important thing to note is that the machine which had the SQL injection vulnerability was not the same as the machine which held the social security numbers.  The hackers broke into a low-security system, and used that as a platform to find and attack a high-security system.  The trust relationship between the systems allowed the hackers to escalate their privileges on the insecure system and get at some very high-value data.

It turns out that there is a ton of code with similar vulnerabilities out there on web pages, including on the departmental sites we host on the college web server.   It is almost guaranteed that PHP code written by someone who doesn’t understand the security implications will be vulnerable to this kind of attack.  Our web server has been under almost constant attack for the past two months, and at least three different departmental sites have been compromised.  Once the site is compromised, the hackers attempt to escalate their privileges (looking for confidential data), and try to use our server to support spamming, phishing, and other nefarious activities. All of these things are more likely to work when the activity is coming from a berkeley.edu (trusted) machine than when it’s coming from an ultranet.ru (suspicious) machine.

We’re doing what we can to reduce the ability for these hackers to escalate their privileges through our servers, but we have little control over the quality of the code on departmental web sites.  Our server hosts web sites for dozens of departments (75 unique domains at last count), and once a department is set up on our server, it can write and upload whatever code it wants.  The people running web sites for departments range from serious geeks to relative novices, so some of them are just fine in terms of security, while others are wide open.

If we see that you have vulnerable code that is being attacked or exploited, we’ll contact you and ask you to fix it.  Many departments who have vulnerable code don’t have anyone on staff who knows how to fix it–the fix is usually not very difficult, but it does require understanding of the issues.  If you don’t know how to deal with the issue, our web team can also fix security problems, charging on an hourly basis.

We received $100K this year in Campus Technology Council funding for a pilot program to investigate Virtual Desktop Infrastructure (VDI).  VDI is both a new technology, and something of a throwback; the idea is that you can install what is essentially a dumb terminal at the desktop, do all your processing on servers in the data center, and save yourself money and time managing hardware and software on a whole bunch of distributed desktops.  The thing that makes VDI different than a dumb terminal solution, or existing technologies like Microsoft Terminal Server, is that VDI allows you to customize and virtualize individual desktop environments.

What that means is that you can give each person a computing environment that’s specific to his or her needs; you can get the advantages of thin clients while still being able to install customized software on an individual basis.  Or, if your business needs dictate uniformity (such as in a computing lab), you can create a single image to be used by multiple devices, which always reverts to the default state when the user logs out.

If the technology works, it would mean that we could shift from replacing relatively expensive desktop machines every 3-4 years, to replacing cheap thin clients every 5-6 years.  We would save money on hardware, but more importantly, we would save a lot of time and effort in setting up computers, and users would have less downtime.  If something were to go wrong with a client, a replacement with identical functionality, and access to all the same files and applications could be installed within minutes.

VDI can also be accessed from a normal PC (or Mac or Linux box) using Microsoft’s RDC protocol.  That means you can have access to the same session from multiple computers; for example, a lecturer could work on his presentation on a thin client in his office, continue working on it on his laptop in a cafe, and then display it on a classroom computer, all without having to log out or re-open the document.  VDI can also be used as a lightweight replacement for Parallels or VMWare Fusion on Macs for folks who need access to a Windows environment.

In L&S, we are looking at testing several use cases:

• Lecturers–folks who often have the worst computers, shared office space, and a high degree of mobility.
• Labs (and drop-in machines)–places where we want to maintain a clean, standard desktop environment.
• Administrative task workers–front desk staff, or others whose work is fairly routine.
• Mac users needing access to Windows.

We just rolled out our first thin client (in the French library), and will be rolling out more over the next few weeks.  We have also begun setting up Mac users to access our infrastructure.  There have been a few glitches–the technology is definitely not yet fully mature.  But it basically seem to work for most use, and it has a lot of promise.  We’re using VMWare’s View product, which is undergoing rapid development, and the vendor has been pretty responsive to our needs.

We have funding to run the program through the summer, and we’re looking for funding to extend it through the fall.  As long as we have funding, there will be no cost for participating in the pilot.  We’ll ask users to fill out an evaluation survey at the end of the pilot.

If you are interested in seeing whether this would work for you, contact Seth Novogrodsky, who’s our lead on the project.

Side note about Macs: We would love to be able to provide Mac OS X over VDI, as well, but Apple’s licensing doesn’t allow it.  It would not be technically difficult; VMWare already provides Mac server virtualization.

I wrote the following in response to an email I received from a department chair; I thought it would be of general interest.  The subject is Google Books, and UCOP’s endorsement of a settlement agreement for the class-action lawsuit against them.  (Google is being sued for violating copyrights by scanning and publishing books which are in copyright, but out of print).

Relevant articles:

• UCOP’s endorsement of the settlement agreement
• Robert Darnton (Harvard’s librarian) article on the risks of the settlement.

I’ll note that this isn’t really in my area of expertise.  I think there are reasonable arguments to be made on both sides of the issue.  (See, for example, Courant’s response to Darnton’s article, and Darnton’s response to that: http://www.nybooks.com/articles/22496).

Academics have competing needs related to copyright.  On the one hand, universities and libraries can almost universally applaud greater access to public-domain works.  Easy access to digital versions of the chart of Beethoven’s 9th or the works of Isaac Newton provide great benefit to instruction and research, without impinging on copyright.  [There is a distinction between works whose essential character can be easily duplicated, such as books or music, and other media such as sculpture or painting.  The Louvre may not exactly claim copyright on the Mona Lisa, but they won't let you go in and take a high-quality digital image of it without paying a fee.]

Even digitizing the works of Beethoven or Newton has an effect; publishers who might otherwise produce new printed versions might be less likely to do so, because the size of their market has been reduced by the easy availability of electronic versions.  Still, I think most academics would agree that the overall benefit to the public of having the electronic versions available is the primary consideration.

The Google Books project goes a step further, by digitizing copyrighted works which are currently out of print.  This is very much aligned with Google’s corporate philosophy of collecting and providing as much information as they possibly can.  In some ways it’s a clear public benefit–people all over the world can get access to books that they aren’t able to buy–but the copyright holders are understandably concerned.  Just because something’s out of print now doesn’t mean it will be out of print forever–except that once it’s in Google Books, it’s probably less likely to get reprinted.  This is part of the academic objection, since many of our faculty write books which could end up in Google Books.  I think it’s a real effect, but I also think that many faculty would choose to have more readers of their work, even if it meant fewer actual book sales.

The concern that Google will disadvantage universities the way that the journal companies have, I think is largely unfounded.  It’s true that we don’t know what Google will do in the future, and corporate interests are often not aligned with academic interests.  But Google’s corporate philosophy is bound (no pun intended) to the concept of free content; I can’t imagine them charging thousands of dollars for content access to any of their properties, including Google Books.

Then there’s the underlying concern of the library, that Google Books and services like it could threaten the library by making it appear obsolete.  I think this is a real possibility, but it’s a real possibility no matter what happens with the Google Books situation. Right now, the technology to read books electronically is very immature and only marginally usable, but if someone comes up with a good e-book solution, the library as we know it will have to change radically to remain relevant.

We’ve received numerous reports from our customers about an apparent increase in spam in recent weeks.  Indeed, looking at spam statistics shows that spam activity has risen significantly in the past couple of months; see the charts at MessageLabs, which are complete through March.  I’m guessing that April will be even higher.  They list the spam rate as 75.7%; over three-quarters of all the mail sent on the Internet is spam.

Now, our spam filters are generally pretty good about catching spam; probably 80-90% of the total spam volume is caught before users see it.  But spam protection is always an arms race, and right now the spammers have come up with some new techniques which are succeeding at getting around many of the spam filters.  Getting around spam filters is a bounded problem; blocking spam is an unbounded problem.

We have a page with some suggestions on how to deal with spam.  One of those is that you can forward a spam message to spam@berkeley.edu to report it as spam; Calmail will use your message to help improve their spam filters.  (The message has to be forwarded as an attachment).

We’re also in the process of moving all of our mail service over to Calmail.  Calmail allows us to host a mail domain such as LS.berkeley.edu on their own servers.  They’re a much bigger operation, and they have many more resources to put into spam protection.  We’ve moved the math.berkeley.edu domain over, and are initiating projects to migrate the rest of the domains we run on departmental servers.  We expect that migrating to Calmail will reduce everyone’s spam counts, and provide more reliable service as well.  (See my post on high availability).

As noted in an earlier blog entry, the campus is moving towards an FTE-based model for network funding.  Thursday I participated in the latest meeting of the advisory committee, and we got a lot more detail about the plan.

The biggest news is that implementation has been delayed until January 2010.  The committee and the Cabinet are in agreement that the plan is not ready for implementation this coming July 1; there are too many open questions and logistical problems.  So for at least the first six months of the fiscal year, network charges will be the same as they currently are.

Also big news for certain L&S departments is that completion of campus infrastructure projects, including ubiquitous AirBears, is included in the funding model.  That means that some of the buildings with the worst networking, such as Tolman, Wheeler, and Kroeber, will receive network upgrades as part of the adoption of the new model.  Timelines are unclear, and there will probably still be a required departmental contribution for new horizontal cables, but this is definitely good news for our most underserved departments.

Undergraduate students are now included in the funding model at a 0.15 FSE rate.  The proposal is to pay for students via a student technology fee; the campus is working with UCOP (and probably going to the Regents) to implement such a fee on the January 2010 timeframe.  [Not knowing anything about the politics involved, I'd have to say that timeframe looks unrealistic to me, but we'll see.]

The inclusion of students in the model has brought the cost per FSE down below $40/month/FSE, from original projections of $45/month.  The figure we are currently seeing is $38.08/month; that is subject to change due to negotiations over FSE counts, but it should remain below $40.

The committee’s strong preference is to use the existing $1.3M in funding allocation to pay the monthly fees for academic titles and GSIs.  It appears that $1.3M is roughly equivalent to the amount needed to cover those groups, and it looks to me like that’s the direction the campus is going to take.

The overall impact to L&S will be significant in any case.  According to draft figures shared with us, L&S departments in total are currently spending approximately $14K/month for network charges; under the new model (with no funding allocations included), L&S costs rise to $104K/month–an increase of $90K/month, $1.08M/year.  If L&S garners half of the $1.3M funding allocation (a reasonable guess based on faculty FTE counts), that still leaves us with additional costs of nearly $500K/year.

They gave us projected cost breakdowns by department; here are some examples of monthly costs from a large and a small department in each  division (again, this is before any funding is allocated):

• MCB: Current $969, projected $18,084, increase $17,115/month
• IB: Current $1,230, projected $7,501, increase $6,271/month
• Physics: Current $1,486, projected $7,947, increase $6,461/month
• Statistics: Current $192, projected $1,656, increase $1,447/month
• History: Current $410, projected $4,128, increase $3,718/month
• Geography: Current $96, projected $912, increase $816/month
• English: Current $297, projected $3,577, increase $3,280/month
• Scandinavian: Current $56, projected $443, increase $387/month
• UGIS: Current $306, projected $2,495, increase $2,189/month
• L&S Advising: Current $198, projected $1,644, increase $1,447/month

It’s obvious that impacts at these levels would be devestating to departments.  Funding allocation should bring the direct costs down quite a bit; my guess is that the proposal would cut 50-75% off the above numbers for most academic departments, depending on the makeup of departmental FTE.  Departments whose FTE are mostly in academic titles and GSIs would benefit the most from the proposed funding allocations; departments with many administrative staff and GSR positions would benefit less.  Fully administrative departments (such as mine) will likely be paying the full costs.  Even with funding allocations, many departments will struggle to find funding to pay these charges.

Thursday’s meeting is the first time the committee discussed the logistics of implementing this model.  I brought up the example of MCB, our largest department.  MCB has 475 FSE under the model, accounting for possibly 1000 people; the department would probably want to allocate network charges to 100 or more chart strings.  Imagine a scenario where the department manager gets a list of 800 people and has to manually specify the chart string (or multiple chart strings) for each one of them; it sounds ugly.  One possibility for simplifying the logistics would be to implement a GAEL-style payroll charge; the network would be charged to the same chart string as the individual’s payroll.  For some departments this would work well; for others it would still be a major headache.  We’re very early in the process of deciding how the model will be implemented; please let me know any comments you have, and I will continue to keep you informed.