A brief reminder that we have a scheduled outage this evening of websrv02 so that maintenance can be performed on the server. We will be upgrading to Plesk 10.3, PHP 5.3, and also doing some operating system upgrades. Please expect intermittent outages over the next few hours as these upgrades are completed. I will be sure to post a tweet when they are finished.
A quick follow up regarding the emergency maintenance that took place last night. Work started at roughly 9:50PM EST, and service was fully restored by approximately 10:15PM EST. During the rebuilding of the RAID array all services were available, but servers’ performance was degraded. As of 2:30AM EST the RAID array was fully rebuilt, and all services are functioning normally at this time.
If you require any assistance or have any questions please feel free to let me know.
Please be aware that one of the hard drives in websrv01 has predictive failure, meaning that it is possible the drive could fail in the not-so-distant future. I have scheduled a Dell technician to be onsite this evening at 10:00PM EST to replace the drive in the machine, which requires me to schedule 30 to 60 minutes of downtime.
Downtime: Thursday, October 13th, 2011 @ 10:00PM EST for approximately 30 to 60 minutes.
Once the drive is replaced the server will be brought back online and will continue to operate while the RAID array is being rebuilt. You may notice some slowness of service over the next 24 hours while the array is being restored. We apologize for any inconvenience.
This is an important notice to inform websrv02 users that we will be replacing the websrv02 server hardware on Friday, February 19th, 2010. The current hardware is about four years old, and while it has not given us any problems we like to make sure our hardware is always in tip-top shape.
The websrv02 server will be unavailable for a short period of time while we shut the old system down and bring the new one online (approximately 20 minutes). This downtime is currently scheduled for:
Friday, February 19th, 2010 around 1:00PM EST
Important Website Warning: While the server downtime itself will be short, we will be starting the data transfer between the old server to the new server at approximately 10:45AM. It is important that you *do not make any changes to your website* between 10:30AM and 2:00PM EST on Friday or your changes may be lost once the new server is brought online.
Important E-Mail Warning: We will be shutting down incoming mail service on websrv02 at approximately 11:00AM EST to prevent any e-mail loss. If someone attempts to send you an e-mail on Friday between 11:00AM and 1:00PM EST the message will stay in their servers mail queue and will automatically resend once the mail service on the new server is brought online again.
We do apologize for any inconvenience this may cause; however we feel that this is an important upgrade to ensure the long term quality of service you have come to expect.
Just a quick note to inform websrv01 users that I have upgraded both Apache and PHP on websrv01 this afternoon. This server is now running PHP 5.2.12, and previous pdo_mysql issues should now be resolved. Our next PHP upgrade (likely the end of February) will be into the PHP 5.3 branch, so please ensure any PHP applications you may be running are ready for this.
Update: 9:29AM EST
Dell has successfully completed the maintenance work and replaced the failed hard drive on websrv01. All services are currently online; however, the server will be a little slow until it fully syncs the newly installed hard drive with its mirrored drive. We expect that this will take another few hours to complete. If you have any trouble, please let me know.
Original Post: Oct 27 @ 8:02PM EST
I confirmed earlier today with Dell that a service technician will be on-site in our Montreal data centre on Thursday, October 29th at 8:00AM EST to replace the failed hard drive on websrv01. While this maintenance is taking place, all services on the server will be unavailable. We expect that the downtime will no longer than 1 hour; however, your patience as we replace the failed hardware is appreciated.
Just to confirm, all services on websrv01 will be unavailable:
Thursday, October 29th, 2009 @ 8:00AM EST
Update: Oct 21 2:24AM
websrv01 is currently back online. Diagnostic tests are currently running on the server as we speak (thanks to Greg), but initial reports indicate that 1 of the 2 hard drives on websrv01 has died. Luckily we run a RAID 1 (mirror) configuration, so the other drive is picking up the slack (whew). Dell is aware of the issue and will get back to me later in the day to schedule a time for them to visit the data centre and investigate further. I will post more information as it becomes available.
Initial Report: Oct 20, 11:03PM
websrv01 is currently off-line. It is highly upsetting to say that; however, we are currently experiencing some major hardware issues / failures. I am currently working with Dell and our co-location provider to resolve the issue; however, we expect that the server will be down most of the day on Wednesday while we recover service.
This morning we experienced a short unscheduled service outage on websrv01 due to spam attack that took place early in the morning. This incident could have easily been avoided if a select few users had e-mail address passwords that were not incredibly simple. If you have a simple e-mail address password, please change it immediately. Passwords should be alphanumeric and contain a minimum of 6 characters, and no dictionary words.
We are currently experiencing an unscheduled service outage on websrv01 due to what we believe may be a hardware issue on the server. In fact, I think this could be the same issue we encountered on April 25th, and I hate to say it but our co-location provider *still* has not resolved the misconfigured the power port that our server is plugged into, so I am still unable to reboot the machine.
A technician has been informed of the problem, and someone is going down to the server to reboot it right now. Luckily, I am told there are people in the building today, so it should be back shortly. I will post an update as soon as I know anything.
Data centre technicians are making their way to the server right now to fix the APC switch and restart the machine.
I’m still waiting, and getting more angry by the minute. I apologize for the inconvenience.
websrv01 is back online after the technician finally rebooted the server, I apologize once again for the inconvenience. I am fairly certain that they assigned John to my support ticket:
We are currently experiencing an unscheduled service outage on websrv01 due to what we believe may be a hardware issue on the server. Unfortunately our co-location provider misconfigured the power port that our server is plugged into, so we were unable to reboot the machine ourselves. Currently we have a technician assigned in Montreal who is on his way to the data centre to reboot the server and investigate further. We will update this post as more information becomes available.
We are still working with our co-location provider to determine the exact cause of the problem. One theory currently being investigated is that we may be experiencing a distributed denial of service attack on the server. As soon as we have any further information, we will post it.
The problem has now been resolved, and all service has been fully restored. It does in fact appear to have been a distributed denial of service attack, which fortunately ceased on it’s own. We sustained 1Mbit of http traffic to websrv01 for only a short period of time before the server was unable to handle the requests. The 1Mbit wall continued until just after 6PM when it stopped just as mysteriously as it began. Further investigation is on-going and any new information will be made available.
We apologize for the inconvenience.