Director's Blog
network

Many campus units are considering removing wired telephones as a cost-cutting measure.  Wired phones have certainly become less important as most of our communications move to email and cell phones, and the budget situation in most departments is severe enough that everyone is looking to save money where ever they can.  Some departments have already begun to disconnect their wired phones and replace them with internet-based solutions; many others are evaluating the idea.

Unfortunately, internet telephony is still immature relative to wired telephony, and there are concerns with any approach, many of which may not be apparent until after the decision has been made.

The biggest concern is in the area of life safety: 911 service from internet phones is different in important ways than 911 service from wired phones.  One major difference is that internet service generally doesn’t work during a power outage, while wired phones often continue functioning.  (This could be critical during an earthquake).  Also, internet or cell phone 911 service does not provide accurate location information to the emergency responders; with a wired phone, they will immediately know your exact building and room, while cell 911 gives only an approximate, 2-dimensional location, and internet phone service may provide no location information at all.

UCPD will soon come out with a policy which requires departments who’ve removed their wired phones to install some number of red emergency phones per floor.  Note that leaving a normal wired phone or two will not be enough to meet the requirements of this policy.  Imagine a scenario where there’s been an earthquake, or someone is having a heart attack; no one is going to think to run over to the fax machine to call 911.  Emergency phones, visible and colored, ideally with a signal light, will be required to address the safety issue.

There are other unexpected issues which arise from internet telephony.  For example, some departments are using a product called MagicJack which you may have seen advertised on late-night TV.  As a product on late-night TV, “the large print giveth, and the small print taketh away.”  The caveats with MagicJack are that you’re basically agreeing to allow them to install spyware on your computer, and to allow advertisers to know what phone numbers you’re dialing and target ads based on those phone numbers.  Creepy.  And at least one department is setting up a full-on Voice-Over-IP (VOIP) PBX, essentially becoming their own 24×7 telecom service provider.  It seems hard to imagine that that will really be a cost savings in the long run.

I won’t tell you not to pull out your wired phones; I know how bad the budget situation is for most departments.  But I will tell you to consider all the implications, particularly around life safety, and take appropriate steps to manage the risks involved.

As noted in an earlier blog entry, the campus is moving towards an FTE-based model for network funding.  Thursday I participated in the latest meeting of the advisory committee, and we got a lot more detail about the plan.

The biggest news is that implementation has been delayed until January 2010.  The committee and the Cabinet are in agreement that the plan is not ready for implementation this coming July 1; there are too many open questions and logistical problems.  So for at least the first six months of the fiscal year, network charges will be the same as they currently are.

Also big news for certain L&S departments is that completion of campus infrastructure projects, including ubiquitous AirBears, is included in the funding model.  That means that some of the buildings with the worst networking, such as Tolman, Wheeler, and Kroeber, will receive network upgrades as part of the adoption of the new model.  Timelines are unclear, and there will probably still be a required departmental contribution for new horizontal cables, but this is definitely good news for our most underserved departments.

Undergraduate students are now included in the funding model at a 0.15 FSE rate.  The proposal is to pay for students via a student technology fee; the campus is working with UCOP (and probably going to the Regents) to implement such a fee on the January 2010 timeframe.  [Not knowing anything about the politics involved, I'd have to say that timeframe looks unrealistic to me, but we'll see.]

The inclusion of students in the model has brought the cost per FSE down below $40/month/FSE, from original projections of $45/month.  The figure we are currently seeing is $38.08/month; that is subject to change due to negotiations over FSE counts, but it should remain below $40.

The committee’s strong preference is to use the existing $1.3M in funding allocation to pay the monthly fees for academic titles and GSIs.  It appears that $1.3M is roughly equivalent to the amount needed to cover those groups, and it looks to me like that’s the direction the campus is going to take.

The overall impact to L&S will be significant in any case.  According to draft figures shared with us, L&S departments in total are currently spending approximately $14K/month for network charges; under the new model (with no funding allocations included), L&S costs rise to $104K/month–an increase of $90K/month, $1.08M/year.  If L&S garners half of the $1.3M funding allocation (a reasonable guess based on faculty FTE counts), that still leaves us with additional costs of nearly $500K/year.

They gave us projected cost breakdowns by department; here are some examples of monthly costs from a large and a small department in each  division (again, this is before any funding is allocated):

• MCB: Current $969, projected $18,084, increase $17,115/month
• IB: Current $1,230, projected $7,501, increase $6,271/month
• Physics: Current $1,486, projected $7,947, increase $6,461/month
• Statistics: Current $192, projected $1,656, increase $1,447/month
• History: Current $410, projected $4,128, increase $3,718/month
• Geography: Current $96, projected $912, increase $816/month
• English: Current $297, projected $3,577, increase $3,280/month
• Scandinavian: Current $56, projected $443, increase $387/month
• UGIS: Current $306, projected $2,495, increase $2,189/month
• L&S Advising: Current $198, projected $1,644, increase $1,447/month

It’s obvious that impacts at these levels would be devestating to departments.  Funding allocation should bring the direct costs down quite a bit; my guess is that the proposal would cut 50-75% off the above numbers for most academic departments, depending on the makeup of departmental FTE.  Departments whose FTE are mostly in academic titles and GSIs would benefit the most from the proposed funding allocations; departments with many administrative staff and GSR positions would benefit less.  Fully administrative departments (such as mine) will likely be paying the full costs.  Even with funding allocations, many departments will struggle to find funding to pay these charges.

Thursday’s meeting is the first time the committee discussed the logistics of implementing this model.  I brought up the example of MCB, our largest department.  MCB has 475 FSE under the model, accounting for possibly 1000 people; the department would probably want to allocate network charges to 100 or more chart strings.  Imagine a scenario where the department manager gets a list of 800 people and has to manually specify the chart string (or multiple chart strings) for each one of them; it sounds ugly.  One possibility for simplifying the logistics would be to implement a GAEL-style payroll charge; the network would be charged to the same chart string as the individual’s payroll.  For some departments this would work well; for others it would still be a major headache.  We’re very early in the process of deciding how the model will be implemented; please let me know any comments you have, and I will continue to keep you informed.

For the past year or so I’ve been sitting on the advisory committee which is providing input for a project to totally change in how the campus network is funded.  Our current node-based network funding model has a number of major problems: the two most significant of those are that the charges don’t map well onto actual network costs (among other issues, wireless service isn’t included in the model at all), and that the model was never truly funded.  Originally, the model had the campus contributing permanent money to fund a “node bank,” which essentially grandfathered in all existing network connections as of July 1, 2000.  The campus never contributed that money; the chancellor has been covering some of the expense with temporary funding (which he wants to stop doing), and even so, the network has been underfunded.

Before you start throwing things–and when you read the rest of this post, you will likely want to throw things–please know that the advisory committee does not have any authority to change the basic parameters of the project, which are that the model should allow for direct charges to grants, be more or less equally charged to all departments, and be implemented by July 1, 2009.

We looked at what other similar institutions are doing, and the strong consensus is that higher ed institutions are moving to headcount, FTE, or FSE (full service equivalent, with a multiplier for non-communication workers and/or students) models.  A major advantage of an FTE-based model is that it is technology-neutral.  Apple introduced their first AirPort wireless access point, which was the first major success of Wi-Fi networking, just three weeks after the campus decided on a node-based model for network funding.  We cannot predict what new network technologies will arrive in the next five years–but we can surely predict that there will be some, and that implementing them will have cost implications.  Our funding model needs to be flexible as technology changes.

The FTE model also is neutral with respect to user behavior.  Our current node-based model encourages undesirable user behavior, such as using AirBears instead of wired connections to avoid network charges, or connecting network switches or hubs to a single wall connection, and running network cables down hallways or poking holes in walls.  (If you can imagine it, we’ve seen it done.)  When charges are FTE-based, networks can be built based on what works best, technologically, as opposed to what incurs the least cost based on a faulty model.

So, in theory I’m in favor of moving to an FTE-based funding model, and it seems clear that the campus is committed to do so at this point.  However, the devil is in the details, and today we were told there are two major details which still need to be worked out: defining “FTE”, and deciding on subsidy (or “funding allocation” as the budget office would have us term it.)

The advisory committee had come to a decision on the FTE definition which would include all faculty and staff (including “non-knowledge-worker” staff such as gardeners), all GSI and GSR positions, and a partial cost for students living in the dorms.  Undergraduates, including undergraduate student workers, would not be included.  When the plan was brought to the cabinet, there was push-back on the non-knowledge-worker issue (Facilities wants a multiplier for them), and on the student issue (the Recharge Committee wants them all included, ostensibly to comply with federal regulations).  There are ongoing discussions on exactly how the FTE/FSE model would shake out, and what the multipliers would be for different types of users.  Those discussions should be concluded reasonably quickly.

The bigger issue is subsidy.  There is a sizable amount of subsidy in the network right now, but most of it has been allocated on a yearly basis from temporary emergency funds.  The advisory committee is requesting $4 million in permanent funds be allocated to offset the impact of the network charges; in the current budget climate, it is not clear what will come of that request.  Without any funding allocation, the network charges will come out to something in the neighborhood of $45 per FTE per month ($540 per year).

Most L&S departments are currently paying very low, or zero per-month network fees.  Because all existing connections from July 1, 2000 were grandfathered in, and most network growth since that time has been wireless, departments have been able to manage their node banks to avoid charges.  Under the FTE-based model, these departments would begin to have to pay.  A  department with 10 staff, 50 faculty, and 40 GSI/GSR/student FTE could go from paying zero now, to paying $54,000/year for network charges.  Even a small department with 5 staff, 10 faculty, and 5 GSI/GSI/students could be hit with over $10,000 in ongoing yearly charges.  And as noted above, the campus plans to roll this out as of July 1 this year.  Given the above costs, it seems impossible that the roll out could successfully occur; subsidy will be required to plausibly implement this plan for the coming fiscal year.  (Or really, for any other fiscal year).

As of today, we are not close to agreement on the size of the funding allocation, or how it should be allocated.  One plan has the $4 million in requested permanent funding going to subsidize all academic titles (faculty/GSI/GSR); in the above scenarios, that would reduce the budget impact for those two departments to something like $6,000 and $3,000/year, respectively (staff FTE plus student worker FTE).  So even if $4 million in funding is allocated, there will still be a significant impact to many departmental budgets–and there are several on the committee who doubt that $4 million in funding allocation is possible.  There is some talk of a student fee, but serious doubts about whether the students would vote for it in referendum.

The financial people are going back to run another set of numbers.  Soon, Shel Waggener will talk to the Council of Deans and the Academic Senate about implementing this plan.  I expect there will be significant resistance from both bodies, and I don’t have a good sense of what the final result will be.  I can’t imagine how departments could take this kind of budget hit, in the midst of all the other budget cutting they are having to do.

I will do what I can to keep people informed.

Peggy Huston posted a message today about changes to the campus’ network funding model.  The current model, which is based on a per-month, per-connection charge, has a number of signifcant problems.  The four main issues with the current model are:

• It fails to accurately track costs.  There are a number of reasons for this, but most importantly, wireless networking isn’t included in the cost model at all.
• It encourages undesirable behavior.  From a technical perspective, we would recommend that every desktop computer be connected via its own individual wired connection.  The per-port installation and monthly costs of the current model encourage departments to use other connection mechanisms, such as commodity hubs or wireless, which provide poorer service and false economy.
• It does not include the cost of upgrading legacy networking.  L&S has several buildings which are still using shared 10-megabit networking that was installed in the early 1990s.  The current funding model provides no way to replace those old, slow, unreliable networks.
• Subsidies are asymmetrical and insufficiently funded.  Each department which existed on June 30, 2000, has a certain number of nodes in its “node bank.”   Some departments are node-rich and others are node-poor, for reasons which are historical rather than .  Newly-created departments have no node bank and thus no subsidy.  But most importantly, the campus has never truly provided enough funding for the node bank, which has forced us to run the network in deficit.

Most universities are moving towards some kind of per-head model for network funding.  Charing by head (or by knowledge worker, or FTE or whatever) is attractive for a number of reasons.  Primarily, FTE models adapt much better to changes in technology; when our current node-based model was developed, it may have made sense given the technology we were using at the time, but now that wireless is a large and growing part of our network costs, our model no longer maps onto our costs.  FTE models can be adjusted and trued up as the technology changes.  Also, FTE models tend to be neutral in terms of their effects on user behavior; they don’t provide incentives to engage in bandit networking.

At this point, it looks like the campus is going to move to an FTE model that will include all staff and faculty FTE.  It appears that students will not be included in the FTE count.

The biggest issues remaining to be discussed are around subsidy; is how much will be involved, and how it will be implemented.  It is clear that L&S departments cannot absorb significant new charges without an offsetting addition of funding; I will continue to advocate on the network funding committee for full subsidy of baseline networking for all faculty and staff.  Cal Moore, as a faculty representative on the same committee, is also looking out for the interests of academic departments.

There should be some interesting discussions over the next few months; I will continue to provide updates as I have new information.