Date posted: 4/1/2022 4:54:04 PM
Posted by: Casey Coughlen
What: Maintenance was performed on gitlab.engr.illinois.edu to perform critical security updates.
Services affected: There was a short ~4 minute interruption to gitlab.engr.illinois.edu
Event Date and Type: 2022-04-01 2:09pm – 2:13pm
Estimated time of resolution: 2022-04-01 2:13pm
Root cause: A critical vulnerability was discovered with gitlab-ce.
Details: Select users have had their unused local account passwords reset and they have been notified via email that this change has happened. This will have no impact on the end users' ability to log into the service with their NetID.
Details for this critical vulnerability can be found here: https://about.gitlab.com/releases/2022/03/31/critical-security-release-gitlab-14-9-2-released/#static-passwords-inadvertently-set-during-omniauth-based-registration
Date posted: 3/17/2022 9:41:17 AM
Posted by: Bobbi Hardy
*Resolved* WHAT: EWS home directories and class directories were unavailable to clients use Distributed File System (DFS) paths (all of the lab machines and documentation instruct everyone to use DFS paths)
SERVICES AFFECTED: Access to EWS home directories and class directories from Windows and Macintosh clients
EVENT DATE AND TYPE: Morning of March 16, 2022 / unplanned outage
ROOT CAUSE: The DFS server for the EWS environment failed
DETAILS: The failure has been corrected, and services should now be restored.
Date posted: 1/26/2022 12:51:20 PM
Posted by: David Mussulman
Resolved as of 12:30pm, per their status page
Echo360 is reporting a nationwide degraded performance with capture processing. Classroom recordings are capturing lectures as expected, but there can be a delay in the availability of the recording due to degraded processing.
From the Echo360 Status page:
What:Echo360 Capture Processing Degraded Performance
Echo360 has identified an issue with capture processing in the US region causing a delay to capture availability.
When: 01/26/2022, approximately 08:22 am - 12:30 pm
Date posted: 1/26/2022 11:15:11 AM
Posted by: David Mussulman
Echo360 is reporting a nationwide degraded performance with capture processing. Classroom recordings are capturing lectures as expected, but there can be a delay in the availability of the recording due to degraded processing.
From the Echo360 Status page:
What:Echo360 Capture Processing Degraded Performance
Echo360 has identified an issue with capture processing in the US region causing a delay to capture availability.
When: 01/26/2022, approximately 08:22 am - Currently
Date posted: 12/14/2021 6:31:37 PM
Posted by: Bobbi Hardy
RESOLVED as of 12/8/2021 at 9:00am : AWS Outage - Echo360, AWS Console Page, and other AWS dependent Services
Audience: Entire College: Instructors, Researchers
Echo360 logins were impacted by this national AWS outage.
AWS was reporting an outage for US-East-1, that was impacting our college Echo360 service and access to the AWS Console Page. The AWS issue was impacting their services across the US. The AWS issue may have impacted other services.
Workaround for Console: Tech Services posted a way to get through to the Console page: https://status.illinois.edu/SystemStatus/jsp/view_events.jsp?eventId=5556&callFrom=currentEventPage&pageId=CC
Date posted: 12/7/2021 12:24:49 PM
Posted by: Bobbi Hardy
AWS Outage - Echo360, AWS Console Page, and other AWS dependent Services
Audience: Entire College: Instructors, Researchers
Echo360 logins are impacted by this national AWS outage.
AWS is reporting an outage for US-East-1, this is impacting our college Echo360 service and will impact access to the AWS Console Page. The AWS issue is impacting their services across the US. This AWS issue may impact other services.
Workaround for Console: Tech Services posted a way to get through to the Console page: https://status.illinois.edu/SystemStatus/jsp/view_events.jsp?eventId=5556&callFrom=currentEventPage&pageId=CC
Date posted: 10/14/2021 11:00:57 AM
Posted by: Bobbi Hardy
Closed: Some CS Virtual Machines not powering on
Services affected: student virtual machines on CS Virtual Farm are unable to be powered on
Event Date and Type: 2021-10-13 ~7:00 p.m. – ongoing / unplanned maintenance
Estimated time of resolution: Thursday, 2021-10-14 11:00 a.m.
Problem has been mitigated, future maintenance work may be done to address the root problem.
Root cause: One of the VMware hosts is exhibiting a disk issue without any logged messages or indications of what has failed
Impacts:The exact number of VMs affected cannot be determined. It appears to mainly affect a subset of CS241 student VMs and some Citrix hosted applications.
Date posted: 10/7/2021 7:08:57 PM
Posted by: Bobbi Hardy
Printers in departmental computer labs may be unavailable
Who: Printers in departmental computer labs and instructional spaces
Timeframe: on-going from the start of the semester
Status: In progress
Over the course of the pandemic, printers in computer labs across the college were infrequently used and many are in need of repairs, supplies and replacement parts, or other attention. Engineering IT Instructional Services is experiencing a high volume of requests this semester and it will take several weeks to return the many printers to working condition.
Additionally, the security vulnerability known as PrintNightmare was recently patched by Microsoft, which frequently results in limited or broken printing ability. Even printers in working condition and computers needing to print may require direct attention before full printing capability can be restored. The campus security team has released a
statement about PrintNightmare.
Thank you for your patience during this time.
Alternative Solutions: Print to another known working printer, like EWS printers or department administrative printers until the in-lab printer is working For specific issues or concerns, please report them to engrit-help@illinois.edu
Date posted: 8/19/2021 9:23:05 AM
Posted by: Bobbi Hardy
*RESOLVED* What: CS Virtual Farm hosts and VM’s hosted on their VSAN storage is unreachable.
Services affected: CS VMFarm Instructional VM’s, Github-dev.cs.illinois.edu, Assorted Course Infrastructure machines.
Event Date and Type: 2021-08-17 ~3pm – 2021-08-18 11:30 am / unplanned outage
Estimated time of resolution: TBD, updates will be posted here.
Root cause: A networking change caused an unforeseen outage.
Any questions or comments about this event can be directed to infrastructure staff and open a ticket to engrit-help-infra@illinois.edu
Date posted: 8/17/2021 3:52:38 PM
Posted by: Bobbi Hardy
What: CS Virtual Farm hosts and VM’s hosted on their VSAN storage is unreachable.
Services affected: CS VMFarm Instructional VM’s, Github-dev.cs.illinois.edu, Assorted Course Infrastructure machines.
Event Date and Type: 2021-08-17 ~3pm – ongoing / unplanned outage
Estimated time of resolution: TBD, updates will be posted here.
Root cause: A networking change caused an unforeseen outage.
Any questions or comments about this event can be directed to infrastructure staff and open a ticket to engrit-help-infra@illinois.edu