Service Restored: Desire2Learn

Submitted by krlu2891 on

Start Date & Time

End Date & Time

Description

Service Restored at 1:19 p.m. Feb. 7:  The D2L vendor has indicated that the D2L service is stable.

Service Issue Updated at 1:30 p.m. Feb 5: The Content Browser widget has been re-enabled in D2L. Those instructors who were using a custom homepage that had the Content Browser widget in it, may need to add the widget back into their homepage.

Service Issue Updated at 1:00 p.m.:  The D2L system continues to be available. There is currently a minor issue with the Content widgets. The D2L vendor has temporarily removed the Content Browser widget from all CU Boulder D2L Course home pages, so that they can address some technical issues. This change does not affect your ability to access course content. You may access course content by choosing "Content" from the top navigation menu (located in the gold bar). More information will be posted here as it becomes available.

Service Issue Updated at 3:00 p.m.:  The issue causing degraded D2L logins has been resolved.

Service Issue Updated at 1:55 p.m. Feb 2: OIT has discovered that logins to the D2L system are showing signs of degradation. We have contacted Desire2Learn to investigate this issue. We will provide an update as soon as we know more. 

Service Issue Updated at 7:35 p.m.: The D2L system continues to be available, and OIT is actively monitoring the stability of the system. The restoration of inaccessable files was completed this evening. Given the severity of this outage, system stability could continue to be an issue, and again OIT is monitoring everything very closely.

Service Issue Reported at 10:12 a.m.:  The D2L system is again available, and we are actively monitoring the stability of the system. You may find you are unable to access some materials that were in the system before the outage. The D2L vendor continues the process to restore the inaccessible files. Given the severity of this outage, system stability could continue to be an issue, and again OIT is monitoring everything very closely.

Throughout the day, we expect many people to be using the system, and high volumes could result in system slowness. Wherever possible, we do recommend you space out work activities in D2L throughout the day, to help mitigate high volumes. Thank you for your patience and understanding. We apologize for the inconvenience of this outage and will be continuing extensive work to address the root cause.
 
Service Issue Updated at 9:50 a.m.:  The D2L system is unavailable. The D2L vendor is investigating this new outage. We will provide an update as soon as we know more.
 
Service Issue Updated at 6:07 a.m. Feb. 1: D2L remains online and responsive; however, the vendor believes some files may currently be inaccessible. You may find that you are unable to access some materials that were in the system before the outage. D2L is currently restoring the missing files. More information will be posted here when available.
 

Service Issue Updated at 10:49 p.m.:  After significant technical work the D2L vendor has made good progress, and as a result the D2L system is currently back online. Given the significance of the technical challenges the D2L system has experienced, OIT will continuously monitor the system very closely. Throughout the day, we expect many people to begin using the system, and high volumes could result in system slowness. Wherever possible, we do recommend you space out work activities in D2L throughout the day, to help mitigate high volumes. Thank you for your patience and understanding. We apologize for the inconvenience of this outage and will be continuing extensive work to address the root cause.

Service Issue Updated at 5:21 p.m.:  D2L continues to be down for many universities. One workaround (listed in the Resources section above) is Rosters in the MyCUInfo portal. OIT rebooted the processor for Rosters due to high volumes. This reboot is completed, and Rosters are again available. OIT continues to monitor Rosters.

Service Issue Updated at 4:57 p.m.:  D2L continues to be down for many universities. One workaround (listed in the Resources section above) is Rosters in the MyCUInfo portal. OIT is rebooting the processor for Rosters due to high volumes. This reboot is expected to take approximately 15 minutes, and mail will not be lost as a result of this action.

Service Issue Updated at 8:39 a.m., Jan. 31:  The Desire2Learn (D2L) outage continues for many universities, including CU Boulder. We apologize for the inconvenience of this unacceptable outage. Additional information will be posted here as it becomes available.

Service Issue Updated at 7:14 a.m. Jan. 31: While the Desire2Learn vendor has been working on several technical solutions day and night, as of this time D2L users at multiple universities including CU Boulder still cannot access D2L.

OIT has been in constant contact with D2L's leadership as the troubleshooting work continues.We apologize for the inconvenience of this unacceptable outage. Here are resources for the campus as the outage continues.

  • D2L Workarounds:  With D2L unavailable, we have created several D2L workarounds that can help faculty support students’ learning needs. We will continue to update this page, so please check back for more information.
  • OIT's Academic Technology Consultants (ATCs):  The ATCs assist the faculty with integrating technologies into their teaching, research and creative works. They are also able to assist faculty with learning the workarounds suggested below. Here is a listing of ATC and their contact information.
  • Drop-in Sessions to Get Help with D2L Workarounds: Academic Technology Consultants will be available from 3:30 pm to 5:30 pm in Eaton Humanities 1B60 today (Thursday, January 31st) to help you find and implement alternative solutions  to support students' learning in your classes. More sessions will be scheduled for Friday if D2L is not available by then.
  • Feedback Sessions:  Students, faculty and staff are invited to feedback sessions so that you can voice your frustrations about the outage. No registration is required.
  • Register for Service Alert emails and/or texts:  Visit the Service Alert Subscription page for instructions.
  • Campus E-Memo about the Outage:  CIO Larry Levine has issued an e-memo to the campus about the outage and plans of action for the future.

Service Issue Updated at 5:25 p.m.:  D2L continues to be down for many universities including CU Boulder. We apologize for the inconvenience of this unacceptable outage.

CIO Larry Levine has issued an e-memo to the campus about the outage and plans of action for the future. The e-memo is posted on the OIT website's News section at www.colorado.edu/oit/d2l-outage. The D2L vendor continues to troubleshoot this issue, and OIT continues to be in direct contact with the D2L leadership.

OIT is setting up face-to-face feedback sessions, starting tomorrow, so that you can voice your frustrations. Those sessions will be announced on the OIT homepage at www.colorado.edu/oit. You are also always welcome to send your feedback about D2L or any campus IT services to oitfeedback@colorado.edu.

Please check this Service Alert for status about the D2L service outage. To sign up for Service Alerts via email and/or text, visit this instruction page: www.colorado.edu/oit/service-alert-subscription.

Service Issue Updated at 1:20 p.m.:  The Desire2Learn system has been down for many universities including CU Boulder since yesterday, and this is unacceptable. OIT expected D2L to be restored by this morning, but the proposed technology solutions D2L had implemented did not restore the service. We are in discussions with D2L management now to understand exactly what the solution is and when the D2L service will be restored. As soon as we have an update, we will let you know the status. We apologize for this ongoing problem.

Service Issue Updated at 8:53 a.m.:  People are unable to log into D2L. Please see "Additional Information from Vendor" below, for an update from D2L. We'll provide more information as we know more. 

Service Issue Updated at 7:44 a.m.:  Individuals on campus are able to log into D2L, however content is not loading. We'll provide more information as we know more. 

Service Issue Updated at 6:28 a.m.:  D2L continues to implement a technical solution to this issue, and they report that the D2L system is seeing steady improvements overall. People can log in at this time, however, D2L indicates that they will continue to experience system slowness as D2L fully implements the solution. We will update the status as we know more, and we apologize for any inconvenience.

Service Issue Updated at 5:10 a.m. Jan. 30: Over the night, D2L has been working to implement a solution to this issue. They expect system performance to improve over the night but content to still be slow. More information will be posted here as it becomes available.

Service Issue Reported at 1:25 p.m. Jan. 29:  Desire2Learn (D2L), the campus' online learning environment, is experiencing technical issues. As a result, people are currently not able to log into D2L. OIT has engaged the D2L vendor to troubleshoot this issue. More information will be posted here as it becomes available.

Affected Buildings

Desire2Learn (D2L)

Additional Information from Vendor

Jan. 31, 3:00 p.m. ET, update from Desire2Learn:

Desire2Learn has been experiencing a severe degradation of service in our DC1 SaaS environment.  This affects approximately 25% of our customer base.  Symptoms can be anything from slow page loads to an outright unusable environment. 
 
Yesterday we articulated a solution that was a result of hours of collaboration between our top engineers and our hardware vendors.  This solution involved a configuration change in the File Virtualization layer (ARX) that we estimated to take around 4 hours as large amounts of data needed to be re-indexed.  This change ended up taking 3 times longer than originally thought, improved performance for some clients, however did not completely solve the root issue of contention between the file virtualization layer and the storage unit.  This root issue is still under intense investigation.  Our top engineers, our vendors’ top engineers, and other outside experts are continuing to work through this.  In parallel to the hardware root cause investigation, we have a number of other efforts underway to improve performance.  Our core SaaS development team is continuing to find ways to optimize traffic patterns and reduce loads on the affected file and storage devices.  For other clients, we are able to move their environment (with no data loss) from the affected storage unit to a different array we have recently procured and stood up. 
 
We understand the extreme frustration this is causing our clients and continue to put everything we have behind fixing this as soon as possible.