Lessons from the CCSF debacle

In January 2012, some fairly sensational news stories were published about a major data breach at City College of San Francisco.  According to the early reports, tens of thousands of student records may have been compromised.  Even more interesting, the reports said that some systems may have been infected for over a decade and that there were connections to China and Russia.  While the reports were interesting, they were short on details and I hoped to eventually read more after the school had some time to sort things out.

In May, the CTO of CCSF was suspended at least in part for his reaction to the breach.  The Guardsman, CCSF's newspaper, published a series of articles that described controversy within CCSF over the handling of the breach, the CTO's management and accusations that the breach was a false alarm.

The CTO's tenure sounds like it was a disaster.  It's also full of lessons for IT and security managers.

I suggest reading these two articles before you continue, but if you'd rather just plow ahead you can; I'll provide enough context to understand what I'm talking about.

Disclaimer: I have no inside knowledge.  I'm basing my comments entirely on what was published.  I'm not looking to bash the CTO or the college.  I think we can learn from the situation and in this post I will offer some constructive suggestions for how IT and security managers could handle similar situations better.  I won't spend much time analyzing the academic/business side.  Other parties made mistakes too, but I'm interested in what we can apply to IT and/or security.

Know Your Environment

The CTO came from a military background and the articles suggest that one of the problems was a clash of cultures.  The CTO himself was dismissive of the idea of shared governance and according to a (disputed) accusation, he tried to bypass the college's technology steering committee on a proposed $750k purchase. Additionally, the IT staff complained that the CTO imposed a rigid hierarchy and wasn't open to opinions from employees below a certain "status".

It sounds to me like the CTO failed to understand his environment.  To start with, shared governance is an important part of the community college system in California and it is not optional.  Like it or not, if you want to work in a community college, you have to be willing to work within the confines of shared governance.  The administration and board have to seek faculty input on most major decisions and classified employees (and their union) are generally involved as well although I'm not sure if it has the same legal backing as faculty involvement.  The CTO's statement that the Academic Senate doesn't affect his career is naive.  The Academic Senate and other major committees are part of the governance and politics of a community college and when you sign up to be a manager, especially at senior level, you sign up to play politics.

In any organization, it's critical that you understand how IT governance works.  IT governance dictates how an organization makes decisions about technology.  Some operational decisions will rest solely with IT, but no successful organization leaves all of the IT decisions to the IT department.  It's up to the business side of the organization to make most of the strategic and policy decisions.  IT may be (and should be) involved, but a lot of these are "business decisions about IT", not "IT decisions".

Community colleges are not a "do what I say and don't ask questions" environment.  Classified staff generally expect to be consulted or at least to be able to approach their boss and their boss's boss.  They participate in committees alongside faculty and managers.  Union agreements may restrict the work that employees can do outside their job descriptions, especially if the person is being asked to do work at a higher level than the job they were hired for.  This doesn't meant that the boss has no authority; he does.  But, he may need a different approach than in some other environments.

Security Has to Come From the Top

One of the most disconcerting things that I saw in the articles was the push-back against the CTO for claiming that CCSF had security problems.  After the breach, the college began preparing a response titled "The ITS Department is on Top of Security."  The breach itself was chalked up as an overreaction.

The claims of security problems didn't just come from the CTO, they also came from the report from the forensics firm hired by the CTO, one or two previous external audits and an internal audit.  The college also suffered another breach in 2007.  My guess is that they have some security problems.  Unfortunately, not everyone seems interested in hearing that.

The problem here, for the CTO, is that change has to come from the top.  A senior IT or security manager can make some progress on his own, but big sweeping changes require higher and broader support.  If the chancellor and other senior managers don't recognize that there is a problem, the CTO is going to be limited in what he can do. 

The CTO (or CIO, CISO) can try to make the other senior managers (and the board) more aware.  In fact, it's an important part of his job to make sure they are aware of the risks.  In the end, however, its not his decision to decide how to deal with those risks. If the CTO doesn't like the response he gets, he can change his approach or move on.

While the CTO should make the administration aware, it's important not to over-sell security risks.  Security is important.  I'd even argue that organizations have a moral obligation to protect the personal information they collect.  But the sky isn't going to fall if they decide not to.  One of the sad facts of security is that it's often not as important (from a business perspective) as we'd like it to be.  Many of the risks are externalities.  The college may have to spend some money recovering from a breach, but they won't go out of business.  Students may have some ID theft problems, but classes won't stop.  Even a six-figure payroll theft isn't going to shut down the college or cost a chancellor his job.  As long as the cost to the college is low, the college doesn't have much incentive to increase spending or time spent on security.

In many ways, the college is better off not knowing if there is a data breach.  A breach requires notification in some states (including California) but ignorance is bliss.

Get Buy-in

One professor suggested that much of the conflict came from the CTO's focus on security.  Community colleges generally do not place much emphasis on security.  There's nothing wrong with the CTO wanting to do more in this regard, but if he tried to jump in and shake things up without getting support first, he made a mistake.  If he realized that he needed to make major changes, his next step was to sell those changes.

I mentioned in the previous section the importance of making senior managers aware of the risks, but once the administration is on board with the idea of doing something about security, you still need to convince them that you have the right plan.  You also need to get the rest of the college on board with the plan.  Speak to the key players, including the technology steering committee.  Develop security awareness training and use it to help spread the message.  People are generally resistant to change, but some salesmanship and patience will go a long way.  Having open, not just tacit, support from the other managers is also key.

Realize, however, that you may not get to implement the grand security plan that you want.  Accomplish what you can.  It's foolish to burn political capital with nothing to show for it, but that's exactly what you'll do if you try to drag the organization along, kicking and scream.  Instead, identify the most serious risks and focus on those first.  Start small and phase things in.  Get input from other managers; don't assume that you know what's important.  Ask.

I believe in over-communication.  Managers should be fairly transparent in their intentions.  If you've got a big plan, sell it and communicate it over and over.  Post details publicly, hold meetings, send emails.  Don't surprise people. Get input early and incorporate it into your plan.  You don't know everything so be willing to listen.

Talk to HR

One of the complaints was that the CTO didn't follow the standard process for disciplining and employee.  That's a dumb mistake.  Managers should talk to HR if they're considering any sort of formal discipline, especially if the manager is new to the organization.  HR handles these situations on a regular basis, most managers rarely deal with it.   There's a pretty good chance you'll run afoul of organizational policy, union agreements or the law if you decide to wing it.

Establish Procedures in Advance

One of the complaints was that the CTO hired an outside firm after the breach was detected.  According to those complaining, he should have relied on his existing technical staff and not doing so showed a lack of trust.  That's rubbish.  He had an IT staff of 70 people.  I don't know how many full-time security people he had, but the typical number would be one or two in an IT group that size.  And, the security people he had were probably not digital forensics or incident response experts.  DFIR is pretty specialized, not something you just have your network security people do as needed.

I think the CTO was right to hire an outside firm.  The mistake he made was in not developing and communicating the college's incident response procedures ahead of time.  The college should have had a written procedure that included a contact list and assigned responsibilities.  The procedure should also have identified the point at which CCSF would seek outside expertise and possibly listed some acceptable vendors.  The CTO could have addressed any misgivings during the planning process.

It's much easier to do things right and to get others to come along, when the policies, procedures, guidelines and standards are developed ahead of time.  People get to provide input, everyone has documentation and there are fewer surprises.

I'm a big fan of documentation and procedures.  Done properly, they help to guide people along and make processes repeatable and dependable.  This is especially important with incident response.  They security and incident response team need to know who to involve (i.e. HR and legal) and when to involve them.  They need to know what they can and should handle and when they should call in an outside expert.  If the organization hopes to seek a legal remedy or pursue criminal charges after a breach, it's important that they preserve evidence and establish a chain of custody.  If the team is trying to answer these questions as they go along, they're going to make a lot of mistakes.