As of 09:30 on Monday, 14 March 2016 all University websites have returned to their normal behaviour of loading the corporate font. This incident is now closed and we believe websites are running normally.
Between approximately 15:00 on Friday, 11 March and 09:30 on Monday, 14 March some University websites were temporarily modified to remove the use of the University’s corporate font. This would have caused some differences in presentation: most notably some menus at the top of web pages would have overrun onto two rows.
The effects of the modification were:
- Websites running on the central Drupal service (which include www.cam.ac.uk and the Science Festival site) were affected.
- Sites running on Falcon and the Managed Web Service, and those run by institutions themselves, were not affected.
This incident is now closed and we believe websites are running normally.
At approximately 14:00 on Friday, 11 March 2016 we noticed that University websites were loading very slowly. Our initial investigations showed that this was due to problems loading the corporate font in the Project Light website template from Adobe Typekit.
At 14:04 Adobe indicated that they were aware of widespread problems for their customers in the United Kingdom (tweeting at 14:10). It later became apparent that this was due to a technical fault with their content delivery network provider suffering a problem in London.
As the problem was outside our direct control our staff started to consider what immediate steps we could take to bypass the loading of the corporate font. In consultation and with the agreement of the Office of External Affairs and Communications (“the Office of Comms”) we were able to use some of our network technology to strip references to Adobe’s systems, which causes webpages to immediately use the backup font. This enabled us to ensure that the main University website loaded quickly albeit with a slightly different look to normal. Some other websites hosted on the central Drupal platform also benefitted from the use of this workaround.
Our crisis team and the Office of Comms reviewed the situation at 16:00 to determine what we should do over the weekend. As Adobe had not confirmed that the problem was resolved (although there were strong indications that the underlying problem was starting to clear) we agreed that those sites being modified by the workaround would continue to operate in this way until Monday, 14 March, when the situation would be reviewed at 09:00.
The crisis team met at 09:00 on Monday, 14 March and agreed that the changes that had been put in place should be removed.
Questions and answers
Q: Why did some websites look different?
A: The corporate font (Myriad Pro) is relatively narrow and the backup font (which is not loaded from the internet) is slightly wider. This caused headings and menus to become wider on the screen, which makes some wide menus flow onto two lines.
Q: Why were only some websites been modified?
A: The networking technology we used to manipulate webpages can only be used for requests that pass through it. At present only sites hosted on the Drupal platform pass through this system.
Q: Adobe reported that everything was OK at 16:15 on Friday, why did you leave the workaround in place over the weekend?
A: We were aware of Adobe's reports but felt it safer to leave the modifications in place over the weekend in the event of further problems with their content delivery network. With a number of important activities taking place (the Science Festival and the Library's 600th anniversary programme) we wanted to try to ensure that people could rely on sites working. This decision was made with the agreement of the Office of Comms.
Q: Why do you use a font that has to load over the Internet?
A: The corporate font is a core element of Project Light and has to be loaded from the Internet.
Q: Are there plans to remove this risk? If something goes wrong the websites take too long to load.
A: A review of the use of the corporate font on University websites had already started and UIS is working with the Office of Comms to review options.
Adobe's content delivery network provider: status.edgecast.com
Incident reference: 401633
Last updated 11:06 14-Mar-2016 by ihc22