This issue has been resolved.
The remaining five Ubuntu servers have been restored:
- itf-rs-store35.hpc.uiowa.edu
- itf-rs-store38.hpc.uiowa.edu
- lc-rs-store31.hpc.uiowa.edu
- lc-rs-store32.hpc.uiowa.edu
- lc-rs-store33.hpc.uiowa.edu
ITS Research Services has started restoration of services for the remaining nine LSS (Ubuntu) servers with the new authentication architecture. Until you see that your Ubuntu share has been restored, you will continue to get authorization denied messages.
List of restored Ubuntu LSS servers:
- itf-rs-store34.hpc.uiowa.edu
- itf-rs-store36.hpc.uiowa.edu
- lc-rs-store30.hpc.uiowa.edu
- lc-rs-store34.hpc.uiowa.edu
As a reminder:
LSS shares will no longer be available with Healthcare IDs. There will be no changes or impact to access with HawkID. Users who have the drive mapped with their Healthcare ID will need to disconnect the share and remap it.
Instructions for how to map LSS shares can be found here:
https://its.uiowa.edu/services/large-scale-storage-lss/accessing-your-storage
For individuals using a Mac, please be sure to delete the keychain before attempting to map the share. Instructions on how to clear the keychain can be found at the bottom of the LSS Mac mapping instructions.
Please reach out to the HCIS Help Desk, ITS Help Desk, your ETS consultant, or other local IT support for assistance.
ITS Research Services continues seeing stabilized performance for CentOS LSS nodes listed below. Progress continues for returning service to nine Ubuntu LSS nodes with a current focus on inconsistencies with Apple Mac SMB mapping to the node.
Service has been restored to itf-rs-store18. Please be aware the system will likely be slow to respond initially as service is restored.
Work continues with our campus partners on the configuration changes needed for our remaining Ubuntu systems.
ITS Research Services was able to restore additional LSS systems; including the hardware issues for itf-rs-store14. Here is the full list of systems:
itf-ihdr-store0.hpc.uiowa.ed
itf-rs-store14.hpc.uiowa.eud
itf-rs-store19.hpc.uiowa.edu
itf-rs-store22.hpc.uiowa.edu
itf-rs-store23.hpc.uiowa.edu
itf-rs-store24.hpc.uiowa.edu
itf-rs-store25.hpc.uiowa.edu
itf-rs-store26.hpc.uiowa.edu
itf-rs-store27.hpc.uiowa.edu
itf-rs-store28.hpc.uiowa.edu
itf-rs-store29.hpc.uiowa.edu
itf-rs-store30.hpc.uiowa.edu
itf-rs-store31.hpc.uiowa.edu
itf-rs-store32.hpc.uiowa.edu
itf-rs-store33.hpc.uiowa.edu
lc-rs-storage12.hpc.uiowa.edu
lc-rs-storage13.hpc.uiowa.edu
lc-rs-storage14.hpc.uiowa.edu
lc-rs-storage19.hpc.uiowa.edu
lc-rs-storage20.hpc.uiowa.edu
lc-rs-store21.hpc.uiowa.edu
lc-rs-store22.hpc.uiowa.edu
lc-rs-store23.hpc.uiowa.edu
lc-rs-store24.hpc.uiowa.edu
lc-rs-store25.hpc.uiowa.edu
lc-rs-store26.hpc.uiowa.edu
lc-rs-store27.hpc.uiowa.edu
lc-rs-store28.hpc.uiowa.edu
lc-rs-store29.hpc.uiowa.edu
We are continuing to troubleshoot and make progress on the hardware issues on itf-rs-store18.hpc.uiowa.edu. Work continues with our campus partners on the configuration changes needed for our remaining Ubuntu systems.
ITS Research Services continues to work with our campus partners on the configuration changes needed for our remaining Ubuntu systems. Unfortunately this taking longer than expected.
We continue to troubleshoot two systems (itf-rs-store18 and itf-rs-store14) that we have identified as having hardware issues.
ITS Research Services has resolved most of the issues with mapping/mounting shares on the restored LSS servers. We are having issues now with itf-rs-store18.hpc.uiowa.edu and lc-rs-storage12 that we are continuing to troubleshoot.
As a reminder:
LSS shares will no longer be available with Healthcare IDs. There will be no changes or impact to access with HawkID. Users who have the drive mapped with their Healthcare ID will need to disconnect the share and remap it.
Instructions for how to map LSS shares can be found here:
- https://its.uiowa.edu/services/large-scale-storage-lss/accessing-your-storage
- For individuals using a Mac, please be sure to delete the keychain before attempting to map the share. Instructions on how to clear the keychain can be found at the bottom of the LSS Mac mapping instructions.
Please reach out to the HCIS Help Desk, ITS Help Desk, your ETS consultant, or other local IT support for assistance.
ITS Research Services is investigating reports of failed mapping/mounting of LSS systems that have been restored. Most of the reports are from failed authentication attempts when mounting the share in Argon/IDAS.
Access has been restored to all LSS CentOS servers (itf-rs-store14.hpc.uiowa.edu continues to be unavailable for reasons unrelated to authentication service). ITS Research Services has paused configuration troubleshooting on the 9 LSS Ubuntu servers until Monday.
List of restored LSS servers:
lc-rs-store22.hpc.uiowa.edu
itf-ihdr-store0.hpc.uiowa.edu
itf-rs-store19.hpc.uiowa.edu
itf-rs-store22.hpc.uiowa.edu
itf-rs-store23.hpc.uiowa.edu
itf-rs-store24.hpc.uiowa.edu
itf-rs-store25.hpc.uiowa.edu
itf-rs-store26.hpc.uiowa.edu
itf-rs-store27.hpc.uiowa.edu
itf-rs-store28.hpc.uiowa.edu
itf-rs-store29.hpc.uiowa.edu
itf-rs-store30.hpc.uiowa.edu
itf-rs-store31.hpc.uiowa.edu
itf-rs-store32.hpc.uiowa.edu
itf-rs-store33.hpc.uiowa.edu
lc-rs-storage19.hpc.uiowa.edu
lc-rs-storage20.hpc.uiowa.edu
lc-rs-store21.hpc.uiowa.edu
lc-rs-store23.hpc.uiowa.edu
lc-rs-store24.hpc.uiowa.edu
lc-rs-store25.hpc.uiowa.edu
lc-rs-store26.hpc.uiowa.edu
lc-rs-store27.hpc.uiowa.edu
lc-rs-store28.hpc.uiowa.edu
lc-rs-store29.hpc.uiowa.edu
Until you see that your share has been restored, you will continue to get authorization denied messages. We will not be able to provide a list of which shares are on individual systems as staff are focusing on the restoration of service.
Please note that a result of this configuration change LSS shares will no longer be available with Healthcare IDs. There will be no changes or impact to access with HawkID. Users who have the drive mapped with their Healthcare ID will need to disconnect the share and remap it.
Instructions for how to map LSS shares can be found here:
- https://its.uiowa.edu/services/large-scale-storage-lss/accessing-your-storage
- For individuals using a Mac, please be sure to delete the keychain before attempting to map the share. Instructions on how to clear the keychain can be found at the bottom of the LSS Mac mapping instructions.
Please reach out to the HCIS Help Desk, ITS Help Desk, your ETS consultant, or other local IT support for assistance.
Until you see that your share has been restored, you will continue to get authorization denied messages. We will not be able to provide a list of which shares are on individual systems as staff are focusing on the restoration of service.
Please note that a result of this configuration change LSS shares will no longer be available with Healthcare IDs. There will be no changes or impact to access with HawkID. Users who have the drive mapped with their Healthcare ID will need to disconnect the share and remap it.
Instructions for how to map LSS shares can be found here:
- https://its.uiowa.edu/services/large-scale-storage-lss/accessing-your-storage
- For individuals using a Mac, please be sure to delete the keychain before attempting to map the share. Instructions on how to clear the keychain can be found at the bottom of the LSS Mac mapping instructions.
Please reach out to the HCIS Help Desk, ITS Help Desk, your ETS consultant, or other local IT support for assistance.
ITS Research Services has paused work on the Ubuntu configuration troubleshooting until Monday. This will impact approximately 9 of the 40 LSS nodes. We will continue to work on restoring services to the other CentOS LSS servers.
LSS access has been restored to the following servers. Please be aware the systems will likely be slow to respond initially as service is restored:
itf-ihdr-store0.hpc.uiowa.ed
itf-rs-store19.hpc.uiowa.edu
itf-rs-store22.hpc.uiowa.edu
itf-rs-store23.hpc.uiowa.edu
itf-rs-store24.hpc.uiowa.edu
itf-rs-store25.hpc.uiowa.edu
itf-rs-store26.hpc.uiowa.edu
itf-rs-store27.hpc.uiowa.edu
itf-rs-store28.hpc.uiowa.edu
itf-rs-store29.hpc.uiowa.edu
itf-rs-store30.hpc.uiowa.edu
its-rs-store31.hpc.uiowa.edu
its-rs-store32.hpc.uiowa.edu
itf-rs-store33.hpc.uiowa.edu
lc-rs-storage19.hpc.uiowa.edu
lc-rs-store20.hpc.uiowa.edu
lc-rs-store21.hpc.uiowa.edu
lc-rs-store23.hpc.uiowa.edu
lc-rs-store24.hpc.uiowa.edu
lc-rs-store25.hpc.uiowa.edu
lc-rs-store29.hpc.uiowa.edu
Until you see that your share has been restored, you will continue to get authorization denied messages. We will not be able to provide a list of which shares are on individual systems as staff are focusing on the restoration of service.
Please note that a result of this configuration change LSS shares will no longer be available with Healthcare IDs. There will be no changes or impact to access with HawkID. Users who have the drive mapped with their Healthcare ID will need to disconnect the share and remap it.
Instructions for how to map LSS shares can be found here:
- https://its.uiowa.edu/services/large-scale-storage-lss/accessing-your-storage
- For individuals using a Mac, please be sure to delete the keychain before attempting to map the share. Instructions on how to clear the keychain can be found at the bottom of the LSS Mac mapping instructions.
Please reach out to the HCIS Help Desk, ITS Help Desk, your ETS consultant, or other local IT support for assistance.
LSS access has been restored to the following servers. Please be aware the systems will likely be slow to respond initially as service is restored:
- itf-rs-store22.hpc.uiowa.edu
- itf-rs-store24.hpc.uiowa.edu
- its-rs-store31.hpc.uiowa.edu
- itf-rs-store33.hpc.uiowa.edu
- lc-rs-store20.hpc.uiowa.edu
- lc-rs-store24.hpc.uiowa.edu
Until you see that your share has been restored, you will continue to get authorization denied messages. We will not be able to provide a list of which shares are on individual systems as staff are focusing on the restoration of service.
Please note that a result of this configuration change LSS shares will no longer be available with Healthcare IDs. There will be no changes or impact to access with HawkID. Users who have the drive mapped with their Healthcare ID will need to disconnect the share and remap it.
Instructions for how to map LSS shares can be found here:
- https://its.uiowa.edu/services/large-scale-storage-lss/accessing-your-storage
- For individuals using a Mac, please be sure to delete the keychain before attempting to map the share. Instructions on how to clear the keychain can be found at the bottom of the LSS Mac mapping instructions.
Please reach out to the HCIS Help Desk, ITS Help Desk, your ETS consultant, or other local IT support for assistance.
ITS Research Services and our campus partners continue to make progress on the Ubuntu configurations.
ITS Research Services is pausing effort on the Ubuntu configurations to work on the automation for deploying the changes to CentOS LSS nodes. Our campus partners continue to look at the configuration issues on the Ubuntu systems. This should allow us to start restoration of service to LSS nodes. Access to the shares will depend on which LSS node the share is located on.
We will provide updates of which LSS nodes have been restored as we can. Until you see that your share has been restored, you will continue to get authorization denied messages.
Pleas note that a result of this configuration change LSS shares will no longer be available with Healthcare IDs. There will be no changes or impact to access with HawkID. Users who have the drive mapped with their Healthcare ID will need to disconnect the share and remap it.
Instructions for how to map LSS shares can be found here:
- https://its.uiowa.edu/services/large-scale-storage-lss/accessing-your-storage
- For individuals using a Mac, please be sure to delete the keychain before attempting to map the share. Instructions on how to clear the keychain can be found at the bottom of the LSS Mac mapping instructions.
Please reach out to the HCIS Help Desk, ITS Help Desk, your ETS consultant, or other local IT support for assistance.
Research Services continue to work toward changing LSS authentication services in response to recent Microsoft Security patches. After initial success, the team has run into difficulty with the Ubuntu operating system and is working through these issues with assistance from campus partners.
As part of the automation process, ITS Research Services has been validating and documenting the configurations changes on a secondary LSS node. This is taking longer than expected as the two test systems are running different operating systems (CentOS and Ubuntu). We anticipate being able to finalize the configurations today/early tomorrow then start applying the fix to all LSS systems.
ITS Research Services has successfully tested a solution for resolving the authentication issues on a test LSS system. We are currently working on automating the configuration changes to be able to apply to the rest of the LSS systems. Once access is restored some users may need to remap or mount their LSS shares on their workstations or through Argon/IDAS.
ITS Research Services believes they have identified the root cause. Last week a Microsoft update was applied to address a high security vulnerability, and this broke the method that was used for authentication to LSS. Research Services is currently testing alternatives authentication methods.
ITS Research Services continues to troubleshoot the ongoing issues with LSS. After further investigation, this issue does not appear to be related to the power outage and is impacting LSS systems in both data centers. When attempting to mount their LSS share users may receive a message that they are unable to authenticate.
ITS Research Services is also troubleshooting issues with itf-rs-store31.hpc.uiowa.edu and itf-rs-store14.hpc.uiowa.edu.
Due to a power outage this morning at Lindquist Center, some users may be having issues accessing or mounting their LSS Shares.
Shares that are impacted will start with "lc-rs-store*" or "lc-rs-storage*"
ITS Research Services is working to resolve this issue.