Announcements

What: Maintenance was performed on gitlab.engr.illinois.edu to perform critical security updates.
Services affected: There was a short ~4 minute interruption to gitlab.engr.illinois.edu
Event Date and Type: 2022-04-01 2:09pm – 2:13pm
Estimated time of resolution: 2022-04-01 2:13pm
Root cause: A critical vulnerability was discovered with gitlab-ce.
Details: Select users have had their unused local account passwords reset and they have been notified via email that this change has happened. This will have no impact on the end users' ability to log into the service with their NetID.
Details for this critical vulnerability can be found here: https://about.gitlab.com/releases/2022/03/31/critical-security-release-gitlab-14-9-2-released/#static-passwords-inadvertently-set-during-omniauth-based-registration

*Resolved* WHAT: EWS home directories and class directories were unavailable to clients use Distributed File System (DFS) paths (all of the lab machines and documentation instruct everyone to use DFS paths)

SERVICES AFFECTED: Access to EWS home directories and class directories from Windows and Macintosh clients

EVENT DATE AND TYPE: Morning of March 16, 2022 / unplanned outage

ROOT CAUSE: The DFS server for the EWS environment failed

DETAILS: The failure has been corrected, and services should now be restored.

RESOLVED as of 12/8/2021 at 9:00am : AWS Outage - Echo360, AWS Console Page, and other AWS dependent Services

Audience: Entire College: Instructors, Researchers

Echo360 logins were impacted by this national AWS outage.

AWS was reporting an outage for US-East-1, that was impacting our college Echo360 service and access to the AWS Console Page. The AWS issue was impacting their services across the US. The AWS issue may have impacted other services.

Workaround for Console: Tech Services posted a way to get through to the Console page: https://status.illinois.edu/SystemStatus/jsp/view_events.jsp?eventId=5556&callFrom=currentEventPage&pageId=CC

AWS Outage - Echo360, AWS Console Page, and other AWS dependent Services

Audience: Entire College: Instructors, Researchers

Echo360 logins are impacted by this national AWS outage.

AWS is reporting an outage for US-East-1, this is impacting our college Echo360 service and will impact access to the AWS Console Page. The AWS issue is impacting their services across the US. This AWS issue may impact other services.

Workaround for Console: Tech Services posted a way to get through to the Console page: https://status.illinois.edu/SystemStatus/jsp/view_events.jsp?eventId=5556&callFrom=currentEventPage&pageId=CC

Closed: Some CS Virtual Machines not powering on

Services affected: student virtual machines on CS Virtual Farm are unable to be powered on

Event Date and Type: 2021-10-13 ~7:00 p.m. – ongoing / unplanned maintenance

Estimated time of resolution: Thursday, 2021-10-14 11:00 a.m.
Problem has been mitigated, future maintenance work may be done to address the root problem.

Root cause: One of the VMware hosts is exhibiting a disk issue without any logged messages or indications of what has failed
Impacts:The exact number of VMs affected cannot be determined. It appears to mainly affect a subset of CS241 student VMs and some Citrix hosted applications.

Printers in departmental computer labs may be unavailable
Who: Printers in departmental computer labs and instructional spaces

Timeframe: on-going from the start of the semester

Status: In progress

Over the course of the pandemic, printers in computer labs across the college were infrequently used and many are in need of repairs, supplies and replacement parts, or other attention. Engineering IT Instructional Services is experiencing a high volume of requests this semester and it will take several weeks to return the many printers to working condition.

Additionally, the security vulnerability known as PrintNightmare was recently patched by Microsoft, which frequently results in limited or broken printing ability. Even printers in working condition and computers needing to print may require direct attention before full printing capability can be restored. The campus security team has released a statement about PrintNightmare.

Thank you for your patience during this time.

Alternative Solutions: Print to another known working printer, like EWS printers or department administrative printers until the in-lab printer is working For specific issues or concerns, please report them to engrit-help@illinois.edu

*RESOLVED* What: CS Virtual Farm hosts and VM’s hosted on their VSAN storage is unreachable.
Services affected: CS VMFarm Instructional VM’s, Github-dev.cs.illinois.edu, Assorted Course Infrastructure machines.
Event Date and Type: 2021-08-17 ~3pm – 2021-08-18 11:30 am / unplanned outage
Estimated time of resolution: TBD, updates will be posted here.
Root cause: A networking change caused an unforeseen outage.
Any questions or comments about this event can be directed to infrastructure staff and open a ticket to engrit-help-infra@illinois.edu

What: CS Virtual Farm hosts and VM’s hosted on their VSAN storage is unreachable. Services affected: CS VMFarm Instructional VM’s, Github-dev.cs.illinois.edu, Assorted Course Infrastructure machines. Event Date and Type: 2021-08-17 ~3pm – ongoing / unplanned outage Estimated time of resolution: TBD, updates will be posted here. Root cause: A networking change caused an unforeseen outage. Any questions or comments about this event can be directed to infrastructure staff and open a ticket to engrit-help-infra@illinois.edu