Recovery plan completed successfully, though there may still be some disruptions on Tuesday 19 May

A fire at a data centre in Almere caused problems with Utrecht University’s network, applications and websites. This had a significant impact on the day-to-day work, research and teaching of and for staff and students.

Please keep an eye on this page for further information.

Update: Tuesday 19 May, 4.00 p.m.

Utrecht University’s systems are now working properly again. Margot van der Starre: “We, the Executive Board, are delighted that work and study can resume as normal. We are immensely proud that our organisation has managed to cope with such an exceptional crisis. We would like to thank all our colleagues who have worked so hard on this over the past week.”

This concludes this live blog.

Update: Tuesday 19 May, 8.30 a.m.

Throughout the night from Monday to Tuesday, we closely monitored whether all of Utrecht University’s systems had been restarted following the outage on the evening of Monday 18 May at the data centre in Almere. All our critical systems are now working again. Colleagues from ITS are still working on restarting all systems. As a result, certain systems may be experiencing glitches and/or users may experience some delays when using them.

Evaluation

The Executive Board recognises that the fire at the NorthC data centre has had a significant impact on students and staff, and that there are questions regarding the extent to which Utrecht University was prepared for this.

“We understand that these questions are being asked,” says Margot van der Starre, Vice-Chair of the Executive Board. “My colleagues and I have the same questions. Especially after another system failure occurred last night affecting some of our servers at NorthC.”

“After the Pentecost weekend, we will begin the review and consider what lessons can be learned from this situation. We believe it is sensible to wait until then before providing answers to the questions that are currently being asked. This also does justice to all our colleagues who have worked so hard on this over the past period, and who continue to do so.”

Some important points

• The issues with Osiris have now been resolved. All registrations, changes or submissions made via Osiris and Studielink have been successfully recorded. We are doing our best to process the applications as quickly as possible, but it may take a little longer before you receive a response. 
• Due to scheduled routine maintenance, ValidSign is not yet available. 

Contact 

If you are still experiencing IT issues, please contact the IT Service Desk.

Update: Monday 18 May, 9.40 p.m.

Due to a power cut at the NorthC data centre on Monday evening, a number of Utrecht University’s servers went down. The restart of all our systems appears to be going smoothly. Students and staff may still experience some disruption to their online work or studies on Tuesday 19 May.

Update: Monday 18 May, 9 p.m.

Due to power supply issues at the Almere data centre, a limited number of systems have been unavailable since early this evening. Power has now been restored. We are monitoring whether our systems will become available again following the restart. Further information will follow as soon as possible.

Update: Sunday 17 May, 12.15 p.m.

The recovery plan for restarting the servers has been successfully implemented over the past few days. All systems will be available again from Monday 18 May and Utrecht University’s buildings will reopen as normal from Monday 18 May. We continue to urge you not to log in to UU systems and to work offline until Monday morning, 18 May.

Do not log in until Monday morning

We continue to urge you not to log in to UU systems until Monday morning, 18 May, and to work offline. On Monday, we will be monitoring closely for any glitches so that we can resolve them as quickly as possible, for example if large groups log in at the same time. Please be aware that when you open your inbox, you may have received a large number of automated emails from various systems, as previously reported in this live blog on Wednesday 13 May.

Exams, exam reviews and preparation

Exams will take place as usual on Monday 18 May, but not digitally: an analogue alternative will be organised for this. From 19 May, exams and exam reviews can take place digitally again. Students who have exams from 18 May onwards can access study materials via the usual systems to prepare.

Additional details

There are still some specific issues regarding certain systems: 

• Work is still ongoing on a number of HPC (high-performance computing) systems at the Faculty of Science. 
• Osiris will be up and running again from Monday 18 May. If you have attempted to carry out any actions via Osiris, or via Studielink for example, in recent days, it is not certain whether these have been registered correctly. We are investigating this. You will find more information on this later in the frequently asked questions section.
• Due to scheduled routine maintenance, ValidSign is not yet available. 

Contact

If you do experience any IT issues from Monday 18 May onwards, please contact the IT Service Desk.

Update: Wednesday 13 May, 4.45 p.m.

Data centre has power again, controlled restart of UU servers

The NorthC data centre has had power again since the morning of Wednesday 13 May. This is an important step towards recovery. However, this does not mean that all Utrecht University systems are available again.

What to expect 

Utrecht University will begin the controlled restart of the servers at 17:00 on Wednesday 13 May. As servers are switched back on and systems start running again, it may appear that some applications are already working. This does not yet mean that these systems can be used safely and reliably. We therefore ask staff and students not to use these systems yet and to wait until we announce that they are ready for use again. Saving data and retrieving previously saved information via UU-systems is strongly discouraged until Monday morning, 18 May, as we cannot guarantee that this will work properly.

Students who wish to access course materials to study for their exams are still able to do so. If you are unable to do so straight away, please try again later. 

When systems are restarted, a large number of automated emails may be sent simultaneously. This can cause your inbox to fill up quickly. We ask that you do not immediately interpret these automated messages as a sign that systems are back to normal. Please wait until IT services are fully available. 

These messages may also include an automatic request to change your password. This is part of the standard UU process whereby staff members are required to renew their passwords. If you receive such a request, you may change your password. You can do this via the usual procedure. Once you have done this, we ask that you refrain from using the systems any further. 

Extension of the deadline for the matching form 

Due to a technical issue with Osiris, it is currently not possible to submit a matching form. Prospective students will receive an email as soon as Osiris is available again. The deadline for submitting the matching form has been extended to 22 May. 

Exams 

Exams will take place as usual on Monday 18 May. Exams will not be held online. An in-person alternative will be organised for this purpose. 

Students who have exams from Monday 18 May onwards will be able to access study materials via the usual systems over the Ascension Day weekend to prepare. If this does not work immediately, please try again at a later time. Please bear in mind the above-mentioned limitations regarding the stability of the system and the urgent advice not to save any new data. 

An important step, but recovery takes time 

“The fact that the data centre has power again is good news,” says Jan-Paul van Staalduinen, Director of ITS. “But this is not yet the moment when we can release the systems again. Now, more than ever, we must work with great care. We are starting up hundreds of servers in a controlled manner and in the correct order, so that we can quickly identify and resolve any issues.” 

Next update 

Keep an eye on this page for updates. The next update will be posted here on Sunday 17 May at around 12 noon. 

Update: Monday 11 May, 7.00 p.m.

NorthC’s data centre is expected to have power restored at 12 noon on Wednesday 13 May. NorthC announced this on Monday morning. As soon as power is actually restored to the data centre in Almere, we can begin the controlled restart of our servers. In this update, you can read what this means for the coming days.

Working offline after 5.00 pm on Wednesday

We expect that Utrecht University will be able to start booting up our servers after 5.00 pm on Wednesday 13 May. From that moment onwards, staff and students will not be able to work or study online, as we cannot guarantee the availability of systems. We do not yet know exactly how long this will take and will keep you informed via this live blog. We advise that, from 5.00 pm on Wednesday until further notice, you do not save any data and/or retrieve previously saved information. Lecturers giving lectures on Wednesday after 5.00 pm are asked to ensure that they have saved the necessary documents locally/offline (for example, on their desktop or a USB stick).

No additional systems will become available beyond those that are currently operational until the servers are brought back online. Once the servers are back online, the remaining systems can gradually be made available. This may take several days.

Patient care at the Faculty of Veterinary Medicine is a particular focus for us. ITS will provide tailor-made solutions where necessary. 

Exams

Exams will take place as usual on Wednesday 13 May and Monday 18 May. Exams will not be held online; an in-person alternative will be organised. Students who have exams from Monday 18 May onwards will be able to access study materials via the usual systems over the Ascension weekend to prepare. If this does not work immediately, please try again at a later time. Please bear in mind the aforementioned advice on saving data.

Major operation

I can imagine that many colleagues and students assume that all systems will be up and running again as soon as the data centre in Almere has power, says Jan-Paul van Staalduinen, Director of ITS. However, it is more complicated than that. In that respect, it is not comparable to being at home, where the lights come back on the moment you replace a blown fuse. We have to check hundreds of servers and switch them on in a specific order so that we can detect any problems immediately. This is a major operation that must be carried out carefully and therefore takes time. This means it will take a few more days before all UU systems are working properly again. Dozens of our colleagues are working on this day and night.

Next update

Keep an eye on this page for updates. The next update will be posted here no later than Sunday 17 May at around 12 noon.

Update: Sunday 10 May, 12 p.m.

Utrecht University’s buildings will reopen on Monday 11 May, but not all digital systems will be functioning as usual. Students and staff should therefore be aware of the restrictions.

Since Thursday 7 May, we have been working hard to restore the systems affected by the fire at the data centre in Almere. Our backup from another data centre is now sufficiently robust and stable for a number of (components of) essential systems. This means that buildings can be opened, and work and study activities can resume from Monday morning (with restrictions).

Utrecht University is now focusing on bringing the servers in Almere back online, and with that, the restart of all systems. At first glance, the UU’s servers do not appear to have been damaged by the fire, but we will have to wait and see what we actually find when we restart them. The timeline for full recovery is not yet known.

• From Monday 11 May, students and staff will once again be able to access the University of Utrecht’s buildings during normal opening hours. Laboratories will also be open as usual. 
• Not all digital systems will be fully available again from Monday morning; please see the list further down in this text.
• Teaching will resume from Monday according to the existing timetables.
• As announced on Friday, no exams will take place on Monday and Tuesday, with the exception of exams of the faculty of Medical Sciences held in the Hijmans van den Bergh Building, the faculty of Veterinary Medicine, and at University College Utrecht (see further details below). Where necessary, the exam will therefore take place in a modified format. Exams and reviews will take place as scheduled on Wednesday 13 May and Monday 18 May. Exams will not be held digital (on Chromebooks); a paper (written) alternative will be organised.

Systems to use 

As not all digital systems are working flawlessly yet, we ask you to use only those systems that are already sufficiently stable and are listed below. You may not be able to log in straight away. If that happens, please try again five minutes later.

Exams on 11 and 12 May, and from Wednesday 13 May 

As decided on Friday 8 May, no digital or in-person exams will be held on Monday 11 and Tuesday 12 May. With the exception of exams organised by the Faculty of Medical Sciences taking place in the Hijmans van den Bergh building, exams organised by the Faculty of Veterinary Medicine, and exams organised by University College Utrecht. Exams cannot yet be held digitally. Where necessary, the exam will therefore take place in a modified format. From Wednesday 13 May, exams across the University of Utrecht will resume as planned, possibly also in a modified format. 

Great dedication 

“The fact that we are partially operational again on Monday is thanks to the dedication of many colleagues, and in particular our IT organisation, which is working hard to restore robust and stable systems,” said Vice President Margot van der Starre. “We greatly appreciate this enormous dedication. We are not there yet, but the essential systems are running on the backup. In addition, the reports regarding our servers in Almere are encouraging. As soon as the data centre is back on the grid, we can begin rebooting all systems. I would like to thank all colleagues and students for their flexibility, understanding and patience regarding this situation.”

Please do not contact the IT service desk 

Until further notice, please do not contact the IT helpdesk regarding any system glitches. We are aware of these issues. Our colleagues are still working hard to restore the systems.

Update: Friday 8 May, 5 p.m.

Utrecht University has decided that no examinations will take place on Monday 11 May and Tuesday 12 May. This excludes assessments from the Faculty of Medical Sciences taking place in the Hijmans van den Bergh building, assessments from the Faculty of Veterinary Medicine, and assessments at University College Utrecht. Teaching at the Faculties of Medical Sciences and Veterinary Medicine will also continue next week.

The reason for this decision is that we are currently unable to guarantee that our buildings will be accessible again on Monday morning. In addition, the advice to refrain from using university systems remains in effect, meaning students are unable to prepare for their exams in the digital learning environment. This does not apply to Veterinary Medicine and Medical Sciences, as these faculties work with different systems.

Provide clarity

We understand that this is a significant decision for both students and staff. However, we wish to provide clarity today so that students and staff know where they stand going into the weekend. We consider this preferable to leaving uncertainty about whether exams will go ahead after the weekend.

Update on Sunday 10 May

An update on whether Utrecht University will reopen on Monday will be provided around noon (12:00) on Sunday 10 May. Should the university reopen, the decision remains unchanged: no digital or in-person examinations will be held on Monday or Tuesday, excluding examinations from the Faculties of Medical Sciences, Veterinary Medicine and University College Utrecht.

Update: Friday 8 May, 2.30 p.m.

Update: Thursday 7 May, 8:10 p.m., in addition to the 5:40 p.m. update

The vast majority of Utrecht University’s buildings will be closed on Friday, May 8. This includes the university libraries (UBU and UBB), the University Museum (UMU), and Parnassos. The university libraries will stay closed on Saturday 9 and Sunday May 10. For the UMU and Parnassos, a decision regarding possible weekend opening will be made and communicated on Friday.

These buildings and locations will open on Friday, May 8

• The buildings and clinics of the Faculty of Veterinary Medicine
• The Hijmans van den Bergh building of the Faculty of Medical Sciences
• The Botanical Gardens
• Olympos Sports Center
• The Dining Hall on International Campus Utrecht (ICU) will open - other University College (UCU) buildings, offices, and classrooms will be closed
• Utrecht University Hall will open for ceremonies such as doctoral defenses, inaugural lectures, and graduation ceremonies
   

Update: Thursday 7 May, 5.40 p.m.: Utrecht University closed on Friday, May 8

Due to the fire at the data center in Almere, digital systems and access passes for university buildings will not function as usual tomorrow (Friday, May 8). As a result, staff and students will not be able to access work and study spaces. The necessary safety measures cannot be guaranteed. The outage is also still affecting login processes. Therefore, Utrecht University will be closed on Friday, May 8.

Today (Thursday, May 7), it was not necessary to close the university buildings because many staff members had already “checked in” to their buildings before the outage began. Building access resets on a new day, and this cannot be restored before the weekend.

Do not use UU systems on Friday

Due to the outage, Utrecht University advises against online work and online teaching on Friday, as not all systems are functioning optimally. As a staff member or student, you can help the university by not using UU systems on Friday. This will help reduce the strain on the systems while recovery efforts are underway. 

Veterinary Medicine, Medical Sciences, and academic ceremonies

Different arrangements may apply for the Faculty of Veterinary Medicine, the University Veterinary Hospital, and the Faculty of Medical Sciences, as they are (still) using a different access system. Separate communication will follow for staff and students of these faculties.

Academic ceremonies such as PhD defenses, inaugural lectures and graduation ceremonies at the Academy Building will continue as planned.

Update: Thursday 7 May, 4:00 p.m.

UU students will receive updates about potential changes in their courses through their respective faculties, programmes, and/or teachers. 

We ask UU staff not to keep trying to log in if you are unable to access a system immediately. If possible, stay logged in to applications that are still functioning. This approach helps us minimise the strain on our systems as much as possible while we work on restoring them. 

More information about this disruption and any potential impact on tomorrow (Friday, May 8) will be posted on our website by 7:30 pm at the latest.

Please note: due to the disruption, the accessible restrooms are closed in some buildings.

Update Thursday 7 May, 1:00 p.m.

The disruption is affecting login processes, amongst other things. We advise staff and students to remain logged in to the UU systems as much as possible, as logging out and back in may prevent you from accessing the system again.

Are you a UU staff member and unable to carry out your work due to this disruption? Discuss with your manager whether you can work from home and/or carry out tasks that are not affected by this disruption.