Director's Blog
administrative

As noted in an earlier blog entry, the campus is moving towards an FTE-based model for network funding.  Thursday I participated in the latest meeting of the advisory committee, and we got a lot more detail about the plan.

The biggest news is that implementation has been delayed until January 2010.  The committee and the Cabinet are in agreement that the plan is not ready for implementation this coming July 1; there are too many open questions and logistical problems.  So for at least the first six months of the fiscal year, network charges will be the same as they currently are.

Also big news for certain L&S departments is that completion of campus infrastructure projects, including ubiquitous AirBears, is included in the funding model.  That means that some of the buildings with the worst networking, such as Tolman, Wheeler, and Kroeber, will receive network upgrades as part of the adoption of the new model.  Timelines are unclear, and there will probably still be a required departmental contribution for new horizontal cables, but this is definitely good news for our most underserved departments.

Undergraduate students are now included in the funding model at a 0.15 FSE rate.  The proposal is to pay for students via a student technology fee; the campus is working with UCOP (and probably going to the Regents) to implement such a fee on the January 2010 timeframe.  [Not knowing anything about the politics involved, I'd have to say that timeframe looks unrealistic to me, but we'll see.]

The inclusion of students in the model has brought the cost per FSE down below $40/month/FSE, from original projections of $45/month.  The figure we are currently seeing is $38.08/month; that is subject to change due to negotiations over FSE counts, but it should remain below $40.

The committee’s strong preference is to use the existing $1.3M in funding allocation to pay the monthly fees for academic titles and GSIs.  It appears that $1.3M is roughly equivalent to the amount needed to cover those groups, and it looks to me like that’s the direction the campus is going to take.

The overall impact to L&S will be significant in any case.  According to draft figures shared with us, L&S departments in total are currently spending approximately $14K/month for network charges; under the new model (with no funding allocations included), L&S costs rise to $104K/month–an increase of $90K/month, $1.08M/year.  If L&S garners half of the $1.3M funding allocation (a reasonable guess based on faculty FTE counts), that still leaves us with additional costs of nearly $500K/year.

They gave us projected cost breakdowns by department; here are some examples of monthly costs from a large and a small department in each  division (again, this is before any funding is allocated):

• MCB: Current $969, projected $18,084, increase $17,115/month
• IB: Current $1,230, projected $7,501, increase $6,271/month
• Physics: Current $1,486, projected $7,947, increase $6,461/month
• Statistics: Current $192, projected $1,656, increase $1,447/month
• History: Current $410, projected $4,128, increase $3,718/month
• Geography: Current $96, projected $912, increase $816/month
• English: Current $297, projected $3,577, increase $3,280/month
• Scandinavian: Current $56, projected $443, increase $387/month
• UGIS: Current $306, projected $2,495, increase $2,189/month
• L&S Advising: Current $198, projected $1,644, increase $1,447/month

It’s obvious that impacts at these levels would be devestating to departments.  Funding allocation should bring the direct costs down quite a bit; my guess is that the proposal would cut 50-75% off the above numbers for most academic departments, depending on the makeup of departmental FTE.  Departments whose FTE are mostly in academic titles and GSIs would benefit the most from the proposed funding allocations; departments with many administrative staff and GSR positions would benefit less.  Fully administrative departments (such as mine) will likely be paying the full costs.  Even with funding allocations, many departments will struggle to find funding to pay these charges.

Thursday’s meeting is the first time the committee discussed the logistics of implementing this model.  I brought up the example of MCB, our largest department.  MCB has 475 FSE under the model, accounting for possibly 1000 people; the department would probably want to allocate network charges to 100 or more chart strings.  Imagine a scenario where the department manager gets a list of 800 people and has to manually specify the chart string (or multiple chart strings) for each one of them; it sounds ugly.  One possibility for simplifying the logistics would be to implement a GAEL-style payroll charge; the network would be charged to the same chart string as the individual’s payroll.  For some departments this would work well; for others it would still be a major headache.  We’re very early in the process of deciding how the model will be implemented; please let me know any comments you have, and I will continue to keep you informed.

Universities everywhere are seeing pressure to adopt “cloud computing” services.  Cloud computing is a general class of application, also called “Software As A Service (SAAS)”, where a third-party vendor offers a web-based application service instead of a traditional desktop-based application.  An example that everyone is familiar with is gmail–to use gmail, you don’t need to install anything on your computer except a web browser.  The service is fully portable (you can get it from anywhere), it usually lacks platform dependencies, and in most cases it’s free or very inexpensive.  Google is offering universities the option to use gmail for their student email at no cost to the institution; on its surface, that option looks very attractive.  Google has a number of other cloud-based services, notably Google Docs, which offer great functionality at no or low cost.  Microsoft, Yahoo, and Amazon also offer cloud-based services, and a number of smaller vendors, such as Salesforce.com (more on Salesforce below) offer more targeted applications via cloud infrastructure.

So what’s the downside?  The reason it’s called cloud computing is that the application and the data have no specific location; the servers can be located anywhere in the world, and data backup is handled by storing data in multiple locations.  The problem this causes is that there’s very little control over what happens to data stored in the cloud; when we have legal or policy requirements to protect data security or privacy, it is often difficult or impossible to get assurances from vendors that the data will be handled according to our requirements.  This can put us at risk for audits or lawsuits.

The campus is now providing guidance on outsourcing.  The key part of the new policy is:

Before “sourcing” your technology offsite — campus individuals, departments, managers, and support staff must consider risks to the following:

• privacy and confidentiality of personal, sensitive, or restricted information
• availability of business data and electronic communications (e.g. backup retrieval, evidence for legal disputes)
• cyber security and support for forensics
• access to records in the event a company is acquired or goes out of business

When you process, store, or otherwise use University information (including information about colleagues, research subjects, correspondents, customers, etc.) in an off-campus site, legal and business consequences need to be expertly reviewed, documented in writing, and must be accepted or modified by an authorized individual for your department or the Campus.

The standard agreements a user might click through to sign up for a free service normally do not provide protection to the university in these areas–in fact, they usually explicitly waive our rights to protection and indemnify the vendor from harm.  It is important to consider the implications of conducting university business through cloud services.

That being said, the services offered are in some cases compelling, and the campus is interested in enabling access to them.  One example is the new agreement we’ve signed with Salesforce.com.

Salesforce is a company that started out providing Customer Relations Management (CRM) software as a cloud service, but now has expanded to offer a development platform where organizations and third parties can build applications related to tracking information about customers.  IST is deploying Salesforce to start keeping track of all of its customers–with any luck, their implementation will lead to a better shopping cart and better billing system.  Departments might be interested in using Saleseforce to track alumni, or current students.  LSCR will consider whether it makes sense for us as well.

The agreement the campus has signed verifies that Salesforce meets our criteria for data protection and liability.  Departments who want to try it out can sign on to the umbrella agreement and know they’re within campus policy and recommended practice.  I’m hoping to see similar agreements in the future for Google and other cloud vendors.  For now, if you have interest in using cloud services for university business, feel free to contact me for guidance.

For the past year or so I’ve been sitting on the advisory committee which is providing input for a project to totally change in how the campus network is funded.  Our current node-based network funding model has a number of major problems: the two most significant of those are that the charges don’t map well onto actual network costs (among other issues, wireless service isn’t included in the model at all), and that the model was never truly funded.  Originally, the model had the campus contributing permanent money to fund a “node bank,” which essentially grandfathered in all existing network connections as of July 1, 2000.  The campus never contributed that money; the chancellor has been covering some of the expense with temporary funding (which he wants to stop doing), and even so, the network has been underfunded.

Before you start throwing things–and when you read the rest of this post, you will likely want to throw things–please know that the advisory committee does not have any authority to change the basic parameters of the project, which are that the model should allow for direct charges to grants, be more or less equally charged to all departments, and be implemented by July 1, 2009.

We looked at what other similar institutions are doing, and the strong consensus is that higher ed institutions are moving to headcount, FTE, or FSE (full service equivalent, with a multiplier for non-communication workers and/or students) models.  A major advantage of an FTE-based model is that it is technology-neutral.  Apple introduced their first AirPort wireless access point, which was the first major success of Wi-Fi networking, just three weeks after the campus decided on a node-based model for network funding.  We cannot predict what new network technologies will arrive in the next five years–but we can surely predict that there will be some, and that implementing them will have cost implications.  Our funding model needs to be flexible as technology changes.

The FTE model also is neutral with respect to user behavior.  Our current node-based model encourages undesirable user behavior, such as using AirBears instead of wired connections to avoid network charges, or connecting network switches or hubs to a single wall connection, and running network cables down hallways or poking holes in walls.  (If you can imagine it, we’ve seen it done.)  When charges are FTE-based, networks can be built based on what works best, technologically, as opposed to what incurs the least cost based on a faulty model.

So, in theory I’m in favor of moving to an FTE-based funding model, and it seems clear that the campus is committed to do so at this point.  However, the devil is in the details, and today we were told there are two major details which still need to be worked out: defining “FTE”, and deciding on subsidy (or “funding allocation” as the budget office would have us term it.)

The advisory committee had come to a decision on the FTE definition which would include all faculty and staff (including “non-knowledge-worker” staff such as gardeners), all GSI and GSR positions, and a partial cost for students living in the dorms.  Undergraduates, including undergraduate student workers, would not be included.  When the plan was brought to the cabinet, there was push-back on the non-knowledge-worker issue (Facilities wants a multiplier for them), and on the student issue (the Recharge Committee wants them all included, ostensibly to comply with federal regulations).  There are ongoing discussions on exactly how the FTE/FSE model would shake out, and what the multipliers would be for different types of users.  Those discussions should be concluded reasonably quickly.

The bigger issue is subsidy.  There is a sizable amount of subsidy in the network right now, but most of it has been allocated on a yearly basis from temporary emergency funds.  The advisory committee is requesting $4 million in permanent funds be allocated to offset the impact of the network charges; in the current budget climate, it is not clear what will come of that request.  Without any funding allocation, the network charges will come out to something in the neighborhood of $45 per FTE per month ($540 per year).

Most L&S departments are currently paying very low, or zero per-month network fees.  Because all existing connections from July 1, 2000 were grandfathered in, and most network growth since that time has been wireless, departments have been able to manage their node banks to avoid charges.  Under the FTE-based model, these departments would begin to have to pay.  A  department with 10 staff, 50 faculty, and 40 GSI/GSR/student FTE could go from paying zero now, to paying $54,000/year for network charges.  Even a small department with 5 staff, 10 faculty, and 5 GSI/GSI/students could be hit with over $10,000 in ongoing yearly charges.  And as noted above, the campus plans to roll this out as of July 1 this year.  Given the above costs, it seems impossible that the roll out could successfully occur; subsidy will be required to plausibly implement this plan for the coming fiscal year.  (Or really, for any other fiscal year).

As of today, we are not close to agreement on the size of the funding allocation, or how it should be allocated.  One plan has the $4 million in requested permanent funding going to subsidize all academic titles (faculty/GSI/GSR); in the above scenarios, that would reduce the budget impact for those two departments to something like $6,000 and $3,000/year, respectively (staff FTE plus student worker FTE).  So even if $4 million in funding is allocated, there will still be a significant impact to many departmental budgets–and there are several on the committee who doubt that $4 million in funding allocation is possible.  There is some talk of a student fee, but serious doubts about whether the students would vote for it in referendum.

The financial people are going back to run another set of numbers.  Soon, Shel Waggener will talk to the Council of Deans and the Academic Senate about implementing this plan.  I expect there will be significant resistance from both bodies, and I don’t have a good sense of what the final result will be.  I can’t imagine how departments could take this kind of budget hit, in the midst of all the other budget cutting they are having to do.

I will do what I can to keep people informed.

As I noted in a blog post back in March, IST is ending support for socrates.  Since that posting, I have been participating on the Socrates/Arachne Abatement Steering Committee, trying to find a way to remove those servers without adversely affecting teaching and research within L&S.  I am pleased to note that IST has been responsive to our concerns, and has come up with some plans which should meet the needs of nearly all of our socrates users.

IST’s information about the abatement project is available at http://ist.berkeley.edu/services/tam/serverabatement.html.  This project has been dragging on for a while, but now has a fairly aggressive timeline for finally removing those servers.  Most importantly, there is a user forum tomorrow (Tuesday, October 21) at 11:00 AM in Sibley Auditorium; if you have any questions or concerns about the abatement and migration projects, it would behoove you to attend and make sure your voice is heard.

The highlights of the new plan are:

• Web hosting will be outsourced to a third-party vendor (Dreamhost).  If you currently have a “tilde” account on socrates, and your site is simple HTML, you are eligible for free hosting on the outsourced service.  (Most sites on socrates, something like 80% of them, fall into this category).
• Users with more advanced web needs will be able to contract directly with Dreamhost for full-featured web hosting; IST will collaborate with Dreamhost to provide http://hostname.berkeley.edu URLs for your sites.  This hosting will be much cheaper than IST’s enterprise-level web hosting; something like $10/month.
• Merrill Shanks of the Social Sciences Computing Lab is developing a plan to provide access for socrates users or class accounts which need access to statistical and mathematical software.
• People who use pine to read email on socrates will be encouraged to migrate to a modern mail client (we recommend the free Thunderbird client).  There are also options to use pine on your own desktop computer if you really must.  (I recommend against that configuration, for support reasons).

As I write this message, socrates has been down for most of the day for unscheduled maintenance.  This highlights the fragile nature of the socrates system, and validates the aggressive timeline IST is pursuing for abating the system.  The current plan is to migrate all web users off socrates by December 19; if you are a socrates user, I highly recommend that you migrate as soon as the option is given to you.

Overall, I think this process has produced a result that is likely to be more positive both for IST and for the socrates users.  It is unfortunate that it took so long to get to this point, but on balance I am pleased with how IST has responded to our user demands.  Direct messages from individual faculty members played a key role in encouraging and helping develop a better solution.

LSCR has hired two new employees this month to fill vacant positions.

Craig Carlson joins us on the “Tilden” desktop support team.  Craig comes to us from UCSF, where he was working as the front-line desktop support person in the Pediatrics department.  (Coincidentally, our open position was created when Mical Wilson took a job at UCSF.  It’s all interconnected.)  Craig will initially be sitting with Mary Wielski in Wheeler Hall, and helping support the Tilden departments in Wheeler, Dwinelle, Barrows, and the PowerBar building, among others.

Ray Spence has joined our Unix team, replacing Julie Ashworth, who took a position at the Helen Wills Neuroscience Institute.  Ray comes to us from LBNL’s NERSC research computing facility, where he was in charge of some of NERSC’s infrastructural Unix systems.  Ray’s primary office will be in Evans Hall with Igor Savine, but he will also have space in Le Conte where he can hold office hours.  Ray’s primary responsibilities will be Unix systems outside the Math department, mostly in Physics and the biological sciences.

Peggy Huston posted a message today about changes to the campus’ network funding model.  The current model, which is based on a per-month, per-connection charge, has a number of signifcant problems.  The four main issues with the current model are:

• It fails to accurately track costs.  There are a number of reasons for this, but most importantly, wireless networking isn’t included in the cost model at all.
• It encourages undesirable behavior.  From a technical perspective, we would recommend that every desktop computer be connected via its own individual wired connection.  The per-port installation and monthly costs of the current model encourage departments to use other connection mechanisms, such as commodity hubs or wireless, which provide poorer service and false economy.
• It does not include the cost of upgrading legacy networking.  L&S has several buildings which are still using shared 10-megabit networking that was installed in the early 1990s.  The current funding model provides no way to replace those old, slow, unreliable networks.
• Subsidies are asymmetrical and insufficiently funded.  Each department which existed on June 30, 2000, has a certain number of nodes in its “node bank.”   Some departments are node-rich and others are node-poor, for reasons which are historical rather than .  Newly-created departments have no node bank and thus no subsidy.  But most importantly, the campus has never truly provided enough funding for the node bank, which has forced us to run the network in deficit.

Most universities are moving towards some kind of per-head model for network funding.  Charing by head (or by knowledge worker, or FTE or whatever) is attractive for a number of reasons.  Primarily, FTE models adapt much better to changes in technology; when our current node-based model was developed, it may have made sense given the technology we were using at the time, but now that wireless is a large and growing part of our network costs, our model no longer maps onto our costs.  FTE models can be adjusted and trued up as the technology changes.  Also, FTE models tend to be neutral in terms of their effects on user behavior; they don’t provide incentives to engage in bandit networking.

At this point, it looks like the campus is going to move to an FTE model that will include all staff and faculty FTE.  It appears that students will not be included in the FTE count.

The biggest issues remaining to be discussed are around subsidy; is how much will be involved, and how it will be implemented.  It is clear that L&S departments cannot absorb significant new charges without an offsetting addition of funding; I will continue to advocate on the network funding committee for full subsidy of baseline networking for all faculty and staff.  Cal Moore, as a faculty representative on the same committee, is also looking out for the interests of academic departments.

There should be some interesting discussions over the next few months; I will continue to provide updates as I have new information.

Shel Waggener put together an interesting presentation (PDF) on campus IT spending from 2001-2007.  Some of the highlights:

• Total campus IT spending is estimated at $136 million.  $56 million of that is within IST or the CIO’s office; the remainder is elsewhere (including L&S).
• Total IT spending under the EVCP control unit (basically, the academic units minus IST) is approximately $50 million, just a bit less than IST spending.
• Only 1% of EVCP spending is for IT; that percentage is less than half of any other control unit.  (NB: There is probably some IT spending in academic units that is not captured by these numbers, such as having GSIs/GSRs do computer support for faculty labs.  I don’t recommend the use of grad students for IT support, but it is common practice in the lab sciences).
• IT spending per campus FTE has been basically flat since 2001.
• IT spending as a percentage of total campus budget has been trending downwards since 2001.
• IT spending as a percentage of the institutional budget is in line with other universities; but spending per FTE is quite low.  (This basically indicates that IT isn’t the only thing that’s under-funded).
• Two-thirds of our IT spending is on people (salaries and benefits).  Salaries and benefit costs have gone up since 2001, while hardware and licensing costs have gone down.
• IT spending as a percentage of total budget has actually dropped significantly in physical and biological sciences.  This might be due to large initiatives like QB3 coming online without sufficient funding for IT, or could be an artifact of how they’re looking at the data.
• State funding has dropped from 61% to 56.9% of campus IT funding.  “Indirect cost recovery” (which I assume is synonymous with “recharge”) has risen from 4.9% to 7%.  “Other sources” has risen from 8.9% to 12.1%.  We’re having to get creative to fund IT, even more so than in other areas.

The overall message is unsatisfying but not unexpected; during a period when IT has become more important to our core mission every year, we have actually reduced the resources dedicated to IT.

We’ve submitted our recharge proposal for 2008-2009.  As of this point, there are no significant changes to our lines of business; our rates in all lines of business are going up slightly (4-9%), mostly due to increased salary expense.

The new rates are:

• Annual Desktop Support: $1410/year (was $1290/year in 07-08, $1320/year in 06-07)
• Unix and server administration: $89/hour (was $83/hour in 07-08, $75/hour in 06-07)
• Web and database development: $77/hour (was $70/hour in 07-08, $70/hour in 06-07)
• General hourly work: $75/hour (was $70/hour in 07-08, $69/hour in 06-07)

(See our “What we do” page for descriptions of these services.)

Our strategic planning process is nearing a close, and we are beginning work on a significant reorganization of LSCR’s business.  We expect to develop a significantly improved financial and support model that better meets the needs of our customers and the College.  I think we’ll have something specific to talk about in the next month or so.

Today, a number of departments received messages from CDW-G about needing to renew Software Assurance for one or more of their licenses. The messages look like this:

Dear HENRIETTA TWITTLEWHEEZE

The Software Assurance (SA) portion of the current three year UC Microsoft Select license agreement will expire on January 31, 2008.

Select SA coverage provides you with the right to upgrade to newer versions released during the term in which you own SA. SA is valid for the term of the agreement, purchased in one, two or three year increments depending on when during the three year agreement period you purchase the license with SA. Renewals are all for the full three year period. SA also provides you with certain learning tools and trainings that Microsoft makes available to customers who own SA.

If you purchased or renewed Select SA coverage at any time from January 2005 through December 2007 and wish to renew that coverage, *you will need to submit your renewal order to CDWG by February 26, 2008* to ensure process time before the deadline with Microsoft.

Your order(s) that need to be renewed are:
Q5877392

Items ordered were
ACAD MS SEL VIRTUAL PC MAC LIC/SA 1Y

I think I can speak for the majority of our departments when I say: Huh?

What happened here is that the department (often unwittingly) purchased a license for Microsoft software that includes Software Assurance (SA) for one year. (That’s what “SA 1Y” means at the end of the license). Usually this was unintentional and caused by the confusing CDW-G price lists. What it means is that you purchased the right to upgrade your software to the latest version, for a period of up to one year. (Actually, it’s not really a full year; it’s whatever portion of the year was left until January 31, and it’s not pro-rated). If you renew your SA (by paying an additional license fee), you’ll continue to have the right to upgrade to the latest version; if you don’t renew, you’ll have to buy a new license when you want to upgrade.

Our general recommendation on SA is, don’t buy it unless there’s a specific reason to. The way the pricing works is that you pay almost double the stand-alone license cost for the right to upgrade to a future version you don’t even know you’ll want. Better just to buy the stand-alone license, and buy another one when you know you want to upgrade. (Back in the old days, you could buy a cheaper upgrade, but Microsoft and most other software companies have done away with upgrade pricing.)

But, if you got one of these messages, you already bought SA, so what should you do? First of all, you should make sure you have the most recent version of the software; you’ve purchased the rights to it, so grab it before your rights expire. If you already have the latest version, you probably don’t need to renew SA; you can just buy a new license when the time comes.

Licensing is one of the issues we’re examining as we develop a strategic plan; this situation highlights the high cost of managing and keeping track of software licenses the way we currently do. I would like to move to a model where a standard suite of software licenses is paid for centrally or included in our yearly rates, so that departments don’t have to spend so much energy on these kinds of issues. It will take a fairly significant reworking of how computing in L&S is funded, but I think we can do it.

For several years now, I’ve coordinated a bulk purchase of computers for L&S units. The idea of the bulk purchase is to save us the time we spend configuring identical or nearly-identical computers one at a time, and the extra 10-20% more we spend for those computers than we would if we bought them all together. Purchasing in bulk just makes sense.

However, coordinating the bulk purchases has been challenging. The vendors give me pretty tight timelines, so there’s not much opportunity to publicize the deal before the deadline for the discount expires. I always get requests from managers the day after the deal expires. And the rest of the year, managers have to decide whether to buy a computer when they need one, or wait for the bulk purchase to save a little money.

So this year, I’m trying something new. In collaboration with The Scholar’s Workstation, I’ve managed to get a web page for L&S computer recommendations placed on the TSW web site. This page will include Dell desktops, and Apple desktops and notebooks, with discounted pricing and our configuration recommendations. I plan to add Windows laptop configurations as well. We will keep the page current, so you’ll be able to get our recommendation and pricing any time during the year, not just during a two-week period in September.

I think this will work a lot better for L&S than the bulk purchase program did; the challenge will be keeping up with the changes in products and technology.