You can also find these real time updates on the storage issue on the IT Status page at it.status.mtu.edu.
==== SUMMARY ====
The new storage system to which we recently migrated home directories is exhibiting serious performance issues. IT staff have been working around the clock with vendor support to correct the issue, and also to prepare a fail-back procedure in the case it is necessary.
Updates will be posted here, and you can find more information about the issue at http://blogs.mtu.edu/it/ and http://www.facebook.com/MichiganTechIT .
==== DETAILED UPDATES ====
** Update 09:00 **
At this time we are holding all storage expansion requests for shares on this storage system. Please contact us for other solutions if your needs are time-sensitive.
** Update 08:00, 12/10/13 **
We expect the additional storage servers to be onsite tomorrow along with vendor staff. Until then we’re continuing with efforts to make performance acceptable as much as possible.
** Update 07:00, 12/9/13 **
Progress has been made reducing space usage in the higher performance storage tier. This puts us at slightly less risk for a situation that would significantly impact performance and possibly data protection.
We continue with efforts to increase storage system performance and maintain its health until additional storage servers arrive.
** Update 18:00 **
Work continues. There may be some performance impact starting at 6:00 PM, and possibly additional impact after 11:00 PM. We are continually monitoring and will pause optimization processes at 7:00 AM if necessary.
** Update 12:00 **
We are working with vendor engineers today to troubleshoot unexpected behavior in some of the automatic optimization and clean-up processes used by the storage system. As a result, we are using targeted manual processes to balance the system. This has been somewhat successful, and we are constantly monitoring and adjusting to make improvements without having an impact on performance during the day.
An update on the shipping status of the additional storage servers is expected today. We will also update later on the progress of Linux permissions problems and other outstanding issues.
Thanks again for your patience.
** Update 07:00, 12/8/13**
The space reorganization has been successful in reducing usage on the higher performance tier. We’re continuing with both space and performance optimization processes today. Impact to normal use should be low during the day, but may be noticable in the evening.
** Update 22:45 **
We are running close to space capacity on the higher performance tier of storage in this system. Additional processes will be started to move seldom-accessed files to a lower performance tier. This may have a performance impact.
** Update 15:00 **
We’re continuing with both storage cluster monitoring and optimization. This includes identifying areas where we can reduce overhead on the storage cluster and adjusting storage system policies based on the kinds of usage that have the largest storage impact.
** Update 07:00, 12/7/13 **
Optimization processes have made good progress. We will be pausing them at 8:00 AM, as well as taking steps to limit the impact of backups if necessary.
The targeted optimiztion of ADMIN home directories completed at 10:37 PM last night.
** Update 17:00 **
We have started an optimization process targeting ADMIN domain home directories. At 7:00 PM, a more general optimization process will start, and is scheduled to allow it to run concurrently with backups without having a severe effect on performance. As before, we are continually monitoring the storage system and will take action if performance degrades further.
We also expect to have an update soon on the status of the additional storage servers being shipped to us. Preparations are complete for the physical installation in our datacenter, and we are developing a plan to add them to the storage cluster.
** Update 14:30 **
We are again monitoring performance and observing results of the optimizations that ran overnight. Vendor support is working continually with us to identify other areas to reduce load, and we are acting on these suggestions. In addition, we are updating our plan for tonight’s optimizations.
A permissions change is now complete for all Desktop directories that corrects the issue with Chrome and other programs not being able to save downloads to the Desktop. We’re continuing with other permissions corrections, as well as working through reported Linux permissions issues.
** Update 09:00, 12/6/13 **
Optimization processes were paused at 07:30. We’re currently taking steps to reduce the performance impact of backups.
In addition, we are working on correcting some of the artifacts of the data migration, such as trouble saving downloaded files for some users and incorrect apparent file permissions in Linux.
** Update 22:45 **
Performance is degraded as both optimization processes and regular backups run. We will pause the optimization process at 07:00 and will reduce the impact of backups at 08:00 if they have not yet completed.
** Update 17:00 **
Additional optimization work will begin at 7:00 PM tonight. The types of optimization processes that will be run automatically adjust to limit performance impact, but we will be monitoring them closely.
The vendor is also shipping us additional storage servers that will nearly double the number of “fast” tier storage servers. On Tuesday, vendor engineers found that, fundamentally, the system that was specified is not capable of handling our storage load. The additional servers are being shipped as quickly as possible, but weather is a factor. The work being done until these arrive is the vendor’s and our best effort at mitigating the fundamental limitations of the current system.
** Update 15:00 **
We’ve worked with vendor support to verify that last night’s optimizations are contributing to improved performance. Additional work is planned for tonight, and could reduce performance while it’s underway. Again, optimization processes will be stopped at the beginning of the day tomorrow.
** Update 12:00 **
Email notifications for customers over their storage limits were turned on this morning. This resulted in many email messages being sent at once, with some customers receiving up to 50 messages in a short time. We have disabled further email notifications until we correct how this works.
** Update 09:30 **
We see evidence that the optimization has made a positive change in how data is is being handled by the storage system, but aren’t yet confident that it is directly resulting in improved performance.
** Update 07:00 **
Optimization processes have been stopped. We will be working with vendor support to closely monitor storage performance as normal daily load is applied.
** Update 05:30, 12/5/13 **
The optimization processes have progressed as expected. We will stop any processes not completed at 07:00 to prevent impact to normal load as the day starts.
** Update 21:00 **
Optimizations are running, and are progressing at a good rate. We’re closely monitoring the system to make sure the optimization processes do not have an unacceptable impact to performance.
** Update 17:00 **
We have finalized tonight’s optimization procedure. There will be some performance impact while this optimization runs, but we will limit the impact before 11:00 PM. If the optimization procedure is not complete before 7:00 AM tomorrow, we will pause it to limit impact during normal usage hours.
** Update 14:25 **
IT and EMC staff have continued to adjust parameters on the array to optimize performance and things have remained at a tolerable level. We will be continuing this effort around the clock to improve performance and are working closely with EMC (the vendor) to resolve the issue.
** Update 13:15 **
Performance is somewhat improved as file reclassification continues.
** Update 12:45 **
Performance is again degraded.
** Update 12:40 **
We have made additional configuration changes that appear to be improving performance.
** Update 11:40 **
The offline node is now back online and will begin handling load.
** Update 11:20 **
One storage node has unexpectedly gone offline. Connections are rebalancing to other nodes. Performance is still poor, though the impact of this latest event hasn’t had a further effect.
** Update 09:55 **
Performance is again degraded as normal work load increases.
** Update 09:30 **
Performance is somewhat improved, but the file policies aren’t yet completely effective. We continue to work directly with vendor engineers to apply the policies.
We’re contacting the vendor account team to see what proposal resulted from their meeting.
** Update 07:00, 12/4/13 **
Some but not all of the optimizations have completed at this time. We are taking other measures to reduce storage system load as much as possible. Some of these optimizations will continue to take effect during the day.
The vendor account management team and engineers are meeting this morning to take action on a plan developed last night. More information will be posted here as we have it.
** Update 21:30 **
We are aggressively reorganizing data to optimize performance. Currently performance remains poor as a result of this operation, but we are continuing to work through the night to prepare for normal load tomorrow morning.
** Update 18:15 **
We are continuing to apply these file type policies and are monitoring the effect as we test.
** Update 16:15 **
Vendor engineers have identified a mechanism for the storage system to handle frequently accessed files more quickly. We are working with the engineers to identify these types of files and make these changes as soon as possible, hopefully within the next hour. This is expected to improve performance.
** Update 15:30 **
We are closing on hardware latency contributing to the performance issues. Vendor engineers are determining the precise cause and corrective action.
** Update 13:00 **
At this time performance remains poor. The vendor has been unable to identify the cause of the issue, and we are preparing for the possibility of a failback.
** Update 11:40 **
We have made a configuration change that unexpectedly dropped connections and required computers to reconnect. We are now testing this further configuration change.
** Update 11:00 **
We’re seeing mixed reports of storage performance, but still consider the overall performance to be unacceptable. We are preparing to execute an alternate plan if necessary.
** Update 09:30 **
Performance is again degraded. We have taken steps to reduce other load on this storage cluster and are actively troubleshooting with vendor support.
In addition, we are investigating other potential authentication/authorization issues that might be affecting storage performance.
** Update 08:00, 12/3/13 **
We are continuing to test and monitor performance, and will be reengaging vendor support.
** Update 18:15 **
Responsiveness is much improved after disabling a specific protocol version causing to system to fall back to an earlier version. Logs have been gathered and sent to the vendor and will be passed on to developers to investigate the root cause. Additional work will be completed tomorrow.
** Update 15:20 **
We are applying a vendor supplied patch. After the patch is applied we will re-evaluate performance and take appropriate actions. Applying the patch will most likely cause intermittent connectivity but the duration for this will be short.
** Update 13:50 **
A configuration change has been made on the recommendation of the storage vendor. This should improve performance on CIFS clients (Windows, some Mac OS) as it reduces the amount of network traffic overhead associated with those connections.
** Update 12:20 **
A faulty fiber cable was found and replaced. This appears to have contributed to the performance issue but does not completely account for it.
We are still actively working with vendor support.
** Update 11:40 **
We have a case open with the storage vendor at the highest priority.
** Update 11:20 **
The problematic storage node has been rebooted, but we are still not seeing expected performance. The issue is being escalated with the vendor.
** Update 10:45, 12/2/13 **
We have corrected some configuration issues and are now testing. One problematic node in the new storage cluster is also being rebooted to restore functionality. This reboot should have minimal impact to storage clients.
** Original Notice **
We are seeing slow performance with home directories due to a load issue on the new storage system as well as configuration issues with supporting systems. These issues are causing slow login times, access problems with Windows XP clients, and other intermittent problems.
We are working with vendor support to correct the performance issues, and are correcting the configuration issues affecting access.
We apologize for the high number of storage and service issues today and ask for your patience as we work to restore normal service.
Older Entries