GHOSTs in the machine; water in the machine room
2015 didn't get off to an auspicious start for UIS. While January saw us working hard to upgrade and update many of our online services, and implement our plans to resolve a number of infrastructure issues that had plagued our machine rooms during the Michaelmas term, we also had to contend with some fallout from the ongoing building works on the New Museums Site, where the Central Network Hub (CNH) is located.
On the morning of 30 January, a building contractor cut through a water main which was believed to have been disconnected from the water source. It turned out to have been fed from a different source, and as a result, a significant volume of water traversed the fabric of the building, eventually flowing down into the CNH.
UIS staff swiftly arrived to find water both coming from above and standing on the floor, which combined with the live electrics, created a health and safety risk that precluded instant assessment of the damage. Half an hour later, when it was safe to enter, we were able to establish that – by luck – the Janet network equipment had escaped damage. One rack, however, which hosted the StASSH backup service had not been so lucky and would need replacing.
Following-up on the unfortunate incident, Kier reviewed their working procedures, and UIS provisioned a temporary backup storage via our high performance computing infrastructure in the West Cambridge Data Centre (WCDC).
Had water penetrated other racks within the CNH, it is likely that elements of the CUDN and GBN would have failed, potentially including all off-site connectivity. As such, the incident was considered a 'near miss', and the likelihood of risks considered at the CNH has since been reviewed. An 'in principle' discussion with Janet staff agreed an emergency plan to connect this equipment directly to the CUDN in the event of future loss of services within the CNH. The incident accelerated our ongoing discussions to investigate the provision of a second Janet regional PoP, and Networks are working with Janet to provide a fully resilient second connection ASAP.
Following-up on another incident from last November in which the Soulsby Building machine room on the West Cambridge Site suffered a brief loss of power, two corrective maintenance activities were completed this month.
Aside from the unexpected infrastructure issues, the start to the year saw another cycle of software updates and infrastructure upgrades: the University Map, the UIS News Service, the MCS remote login servers, DS-Filestore SFTP service, CamTools, Jackdaw, the Janet Network Accounting Service, UniOfCam/eduroam , the , the RADIUS server, the Managed Web Service (MWS), the MCS desktops and the Cognos reporting system all had software updates. Meanwhile, the mailing lists service, Raven, the UIS Password Management App and the DS-Print/Managed print service all benefitted from upgraded hardware.
January also brought news of the 'GHOST' vulnerability, which left the GNU C Library (glibc) used by many Unix/Linux operating systems potentially vulnerable to remote attackers using the flaw to cause users to execute arbitrary instructions. Although the bug was fixed in 2013, it had not been recognised as a security vulnerability at the time, and the fix had not been backported to many Linux distributions. We recommend that Linux sys admins apply the update to affected systems.
Power issues continued to plague us on the West Cambridge Site during February. On the evening of 23rd, another power outage event occurred. The UPSs in our machine rooms in the Roger Needham Building, West Cambridge Data Centre (WCDC) and the Soulsby Building all successfully detected the event, and operated as expected. Unlike the event last November, the Soulsby switched over to the UPS without tripping any circuit breakers. Unfortunately, a second incident caused the Soulsby's air handling units to go offline, adversely affecting a number of key services provided by Astronomy, the Library and UIS. Our staff were again quickly on the scene, however, and worked with the Estates Management team to bring the services back online as rapidly as possible.
As if that wasn't enough 'excitement', the following day we received news that some of the GBN infrastructure providing network services to south Cambridge had been damaged, disrupting services running over some privately rented GBN fibres. University users remained unaware of the damage, however, because UIS designed the CUDN to have redundant connectivity for precisely this type of event. Connections running over the damaged GBN fibres were transferred to spare GBN fibres the following day, and the damage rectified.
February saw the High Performance Computing Service (HPCS) physically relocating the Darwin and Wilkes clusters to the WCDC. Teething toubles with the centre's cooling system meant that they initially operated a reduced production service for a few days while the cooling issues were rectified.
On a more positive note, UIS delivered more service enhancements. The upgraded Raven servers deployed earlier in the year support IPv6 (developed to deal with the long-anticipated problem of IPv4 address exhaustion), and as of 18 February, we started advertising an IPv6 address for raven.cam.ac.uk. This change will cause most IPv6-capable clients to start using the new protocol. For correctly configured clients, this change will have been transparent. There is a web page available to test IPv6 addresses.
The DNS server set-up was completely replaced, introducing a number of enhancements including automatic failover for recursive DNS servers and much improved server management and administration functionality. Webmasters now benefit from hourly DNS updates, meaning that users of other recursive DNS servers around the university and elsewhere will see DNS changes within 2 hours of changes to the IP Register database. Similarly, the zone refresh timer has been reduced from 4 hours to just 30 minutes, enabling 'stealth slave' nameservers to fetch DNS updates within 90 minutes of a change to the IP Register database, where previously, the delay could have been up to 8 hours.
In software licensing news, Adobe® announced that they would be withdrawing their Creative Suite® products under the CLP agreement as of 27th February 2015. Adobe is switching to the software-as-a-service (SaaS) business model, c , instead of being able to purchase a single product licence and own a copy of the software. This leaves our software sales team with the unenviable task of trying to work out how – and if – we can use the VIP subscription-based agreement to make Adobe Creative Cloud licences available.
At a strategic level, the University has been evaluating the benefits of taking out a Microsoft Enrolment for Education Solutions (EES) volume licensing agreement, which would enable UIS to provide a range of services including Office 365 ProPlus for all staff and students. Cambridge is currently the only UK university not to have such an agreement in place, but before any decisions were taken, we wanted to ensure that the viewpoints of the University's IT community and constituent Institutions were clearly understood and factored-in.
To this end, senior representatives from Microsoft visited UIS on 18 February for an open meeting with our wider tech community, to outline the details of the agreement as it may relate to Cambridge users and Institutions, and to answer questions directly. Many senior Departmental and College IT staff and administrators came along to voice their unique requirements, and get straight answers to some of their concerns. Overall, the proposal would seem to offer a number of benefits to both the University and its Colleges, Departments and Institutions. As a result of the consultation work, a paper went to the ISC at the end of April recommending that we proceed with the agreement. The paper was approved.
February also saw us reach 400 followers on the UIS News Twitter feed (@UcamISNews) (@UcamISNews) – an easy way to have important UIS service announcements delivered to your mobile device.
Back to (thinking about) the futureThe highlight of the month was the official opening of the West Cambridge Data Centre, by the University Chancellor, Lord Sainsbury. Over forty University dignitaries, including some Heads of Schools and representatives from UIS, Cambridge Assessment and the High Performance Computing Service (HPCS) attended the launch event to hear presentations around HPCS issues and have a tour of the state-of-the-art facility.
Otherwise, March brought less drama, affording us the opportunity to focus on planning the future and implementing new measures to mitigate the recent power issues. We started thinking about the software requirements for the Managed Cluster Service desktop installation in AY 2015/6, and work continued on designing a comprehensive IT service catalogue which will, in time, provide University members with a 'one-stop-shop' for accessing all the IT services available to them, whether provided by UIS, their Department, School or Institution, or a third party.
The joint offered UIS the chance to work closely with the School of Arts and Humanities in developing the University's IT Service Catalogue that will present UIS' services alongside those offered by the School. This collaborative approach is giving UIS a better insight into the issues facing the School's IT staff, and the requirements of its users, and the lessons learnt will be incorporated into further development phases as we start to include other Institutions' services in future.
The project website explains more about the aims of the pilot, what a Service Catalogue is, and why we believe it will be of great strategic benefit to the University. The project website links to a live beta release of the current catalogue and we are inviting everyone to take a look. At this stage, the beta is light on detailed content, as the aim is only to show how we are thinking about rationalising and presenting the 350+ services currently available. There is a feedback form (for logged-in Raven users) on the project website to collate everyone's comments.
We also published our report on the forward planning in response to the operational incidents over the last two months.
Hello HPCS, goodbye VPDN
April saw our colleagues from the former HPCS move from their Mill Lane offices to join us in the Roger Needham Building. A new HPC website was also launched, and Dr Peter Braam, a recognised global leader and multidisciplinary innovator in computing, data and science, was invited to the RNB on 30 April to deliver a talk on Trends in HPC Storage, which was open to the Techlink community.
As of 15 April, the latest release of the Google Chrome browser now flags https websites that use SHA-1 certificates (which are set to expire 31 December 2016) as being insecure. This shouldn’t affect University sites, as UIS issued replacement SHA-2 certificates in November last year, in readiness for the decline of SHA-1.
In another strategic move by Google, April also saw a big change in the way it ranks the search results delivered to mobile device users. The new Google algorithm favours websites it deems to be 'mobile-friendly' (you can check your site on their Mobile Friendly Test page), and penalises content not designed specifically for mobile devices. For the 300+ University webmasters using the Falcon web service, this is not an issue, as Falcon sites already meet Google's preference for responsive design. Site owners whose pages were designed only for large screens, however, "may see a significant decrease in rankings in mobile search results", according to Google.
The University’s old VPDN service was officially withdrawn on 27 April. It has been superseded by the new VPN service, which provides a much easier way for people to access resources remotely that normally require them to be at the University using a CUDN-connected device. The new service seems to be proving popular with users, who are reporting that they find it much easier to use. UIS also offers a managed VPN service that allows institutions to offer their own VPN using UIS' infrastructure.
Speedier software sales and web server certificates
May's service upgrades included the launch of the new Software Distribution system. This major upgrade makes ordering software much quicker, with most orders being ready for download via the website on the same day of ordering – no more waiting up to four days for CDs to arrive in the post. The new Order History page lets you easily keep track of all the licences for which you are responsible.
UIS obtains web server certificates from Jisc/Janet. Their previous arrangement with Comodo has expired and they now have an arrangement with QuoVadis. The range of certificates available, their costs, and the way that they are obtained hasn't changed significantly, but some details of the acquisition and deployment process are different. To coincide with this change, UIS has launched a new Web Server Certificate Portal for requesting, downloading and managing web server certificates, giving webmasters greater visibility of their current requests' statuses.UIS' city centre videoconferencing services were used for a very high profile event this month. An Evening with Stephen Hawking, at the Sydney Opera House, was the professor’s first live appearance before Australian audiences, but thanks to some clever 3D video technology and the ideal studio facilities, he didn’t have to leave Cambridge as the Cockroft Building became a virtual annexe to the Sydney Opera House. The famous cosmologist’s image was transformed into a live 'holographic' projection, and beamed 17,000 miles to his eager audience in Australia. UIS staff were on-hand leading up to, and during the event, to ensure Cisco's tech team had everything they needed for a successful live broadcast.
Closer to home, the UIS change programme ramped up another gear this month, as the much-heralded 'Day One' finally arrived. 18 May has, for many months, been strategically known within UIS as 'Day One' – the day on which UIS' new organisational structure officially goes live, with our new divisional leads in place.
Changing to our new divisional structure has meant that some of our Institutional liaison staff have changed roles. The UIS website features an article on the new divisional structure for Information Services describing how this will affect you and your Institutions in the short- and medium-term, and the new contact information for Institutions.
Our new senior leadership team, reporting to Martin Bellamy, comprises Paul Calleja (former director of the HPCS) who now heads the Research & Institutional Services division, Chris Edwards (a former deputy director of MISD) will head the Build & Development division, and Steve Kearsey (formerly deputy director of UCS) assumes responsibility for Departmental Operations.
They are joined by three people new to the University, who saw-off competition from a strong field of over 300 world-class applicants: Mark Ferrar has joined us from Microsoft to become our Chief Architect; Steve Riley joined us from the Department of Work and Pensions to lead the Service Operations division, and Richard Young, formerly of BUPA, will lead our Education, Administration & Student Services division.
The new team's work will start with the fine-grained internal restructuring of the Department, realigning staff to more effectively deliver our IT services, alongside developing new working practices and forging a common Departmental culture.
As ever, UIS welcomes your comments via email to firstname.lastname@example.org