DreamHost: Email connectivity issues (homiemail-sub4) (RESOLVED)

We are currently investigating connectivity issues on our homiemail-sub4 mail cluster. This is affecting POP3, IMAP, and SMTP connections. Our team is hard at work looking into this, and we will post more information as we have it. Thank you very much for your patience and understanding!

Early this morning, sometime around 8:30am, we started receiving reports of performance degradation on our homiemail-sub4 email cluster. This degradation was primarily caused by an extremely distributed DoS attack.

Our mitigation service kicked in and started blocking various traffic; the traffic block was mitigated to make sure that valid traffic went through around 9am. The attack continued, and is still ongoing at this moment, and at 12pm our provider decided to turned off our port for about 30-90 seconds.

Our team is currently blocking part of the attack right now, in attempt to help resolve the long standing issue. Unfortunately, due to the scale of this attack, this is a long and ongoing process.

Update 2:00pm PDT: We are happy to announce that this matter is fully resolved. Thank you, once more, for your patience while we worked to bring this issue to a full resolution. If you are still experiencing problems connecting to email (or any other DreamHost service), it is not likely related to this, so please do not hesitate to contact support via your DreamHost panel under Support / Contact Support, here.

Update 1:50pm PDT: Our engineers believe we have found a workaround that should be continuing to stabilize our homiemail-sub4 services. However, we are continuing to closely monitor the situation, and will continue to post updates until this is fully resolved. We sincerely appreciate your patience and understanding.

Update 1:20pm PDT: Early this morning, sometime around 8:30am, we started receiving reports of performance degradation on our homiemail-sub4 email cluster. This degradation was primarily caused by an extremely distributed DoS attack.

Our mitigation service kicked in and started blocking various traffic; the traffic block was mitigated to make sure that valid traffic went through around 9am. The attack continued, and is still ongoing at this moment, and at 12pm our provider decided to turned off our port for about 30-90 seconds.

Our team is currently blocking part of the attack right now, in attempt to help resolve the long standing issue. Unfortunately, due to the scale of this attack, this is a long and ongoing process.

Unfortunately, we do not currently have an estimate as to when this will be resolved, however we are continuing to work toward a resolution as quickly as possible. We are still seeing issues with POP3 and SMTP connectivity, and IMAP connections are working intermittently.

Update 12:30pm PDT: IMAP connectivity remains intermittent, and we are still continuing to get POP3 and SMTP connections working properly again. Our team is working diligently to get everything back up again. Thanks for your continued patience and understanding.

Update 12:05pm PDT: Unfortunately, we are still seeing issues with connectivity to our sub4 mail server. We sincerely apologize for the amount of time this is taking. Our teams are continuing to work hard at getting everything working properly again. Thanks again for your patience!

Update 11:18am PDT: We are still continuing to work on this. We’re definitely seeing improvements to IMAP connectivity, however SMTP and POP3 are still experiencing issues. Thanks again for your patience and understanding while we work to resolve this!

Update 10:50am PDT: Connectivity to IMAP  should be much improved. We are still seeing issues with SMTP and POP3 connections and are continuing to investigate. Your continued understanding and patience while we work on this is greatly appreciated.

Update 10:28am PDT: Our team is hard at work on this to make sure this gets resolved as quickly as possible, and remains that way. We will continue posting more information as we have it. Thanks again for your patience!

Update 9:59am PDT: Unfortunately, we are seeing an upswing in connectivity issues again and our teams are investigating this once more. Stay tuned for more information and we thank you very much for your patience and understanding.

Update 9:33am PDT: We are happy to announce everything should be working properly again now, and we are continuing to monitor the situation to prevent further issues. If you are still noticing problems connecting to email, please do not hesitate to contact support via your DreamHost panel under Support / Contact Support, here.

Update 9:20am PDT: We’re seeing significant improvement in connectivity now and are testing a bit more. More information to come!

Update 8:55am PDT: We’ve identified the issue, and are currently working toward a resolution. More updates to come as we have them!

Go to Source


WebHostsStatus.com info on DreamHost

DreamHost: Email connectivity issues (homiemail-sub4)

We are currently investigating connectivity issues on our homiemail-sub4 mail cluster. This is affecting POP3, IMAP, and SMTP connections. Our team is hard at work looking into this, and we will post more information as we have it. Thank you very much for your patience and understanding!

Update 9:59am: Unfortunately, we are seeing an upswing in connectivity issues again and our teams are investigating this once more. Stay tuned for more information and we thank you very much for your patience and understanding.

Update 9:33am PDT: We are happy to announce everything should be working properly again now, and we are continuing to monitor the situation to prevent further issues. If you are still noticing problems connecting to email, please do not hesitate to contact support via your DreamHost panel under Support / Contact Support, here.

Update 9:20am PDT: We’re seeing significant improvement in connectivity now and are testing a bit more. More information to come!

Update 8:55am PDT: We’ve identified the issue, and are currently working toward a resolution. More updates to come as we have them!

Go to Source


WebHostsStatus.com info on DreamHost

HostExcellence: DDoS (Distributed Denial of Service) attack – Resolved

We are currently experiencing a DDoS (Distributed Denial of Service) attack against shared IP 98.130.148.2 on web415. In order to stop the attack on it will be blocked on 2 hours.

We will update this status blog post as we get additional information from our systems administrators.

Read the original post here


WebHostsStatus.com info on HostExcellence

ixwebhosting: Server Maintenance – Wednesday, July 29, 2015 & Thursday, July 30, 2015 – Resolved

Starting July 29, 2015, IX Web Hosting will be adding new hardware to the existing servers that currently house your data.  These upgrades will provide additional capacity, as well as improved disk performance to our existing cloud infrastructure.
The upgrades to the server pool will take place July 29, 2015 and July 30, 2015 at our Data Center located in Columbus Ohio.

These upgrades will be applied so as not to impact any customer VMs.  This upgrade will be seamless, and the only noticeable change will be an increase in overall disk performance.

 

We do not anticipate any customer to experience downtime during this maintenance.  Additionally, as a precaution, you can also perform a local backup of your content. Refer to our Knowledge Base for assistance or via web chat, or please call us at anytime.

To follow the progress of the update, please watch this status blog for updates.

Thank you,
Your IX Web Hosting Team

Original post


WebHostsStatus.com info on ixwebhosting

ixwebhosting: Server Maintenance – Wednesday, July 29, 2015 & Thursday, July 30, 2015

Starting July 29, 2015, IX Web Hosting will be adding new hardware to the existing servers that currently house your data.  These upgrades will provide additional capacity, as well as improved disk performance to our existing cloud infrastructure.
The upgrades to the server pool will take place July 29, 2015 and July 30, 2015 at our Data Center located in Columbus Ohio.

These upgrades will be applied so as not to impact any customer VMs.  This upgrade will be seamless, and the only noticeable change will be an increase in overall disk performance.

 

We do not anticipate any customer to experience downtime during this maintenance.  Additionally, as a precaution, you can also perform a local backup of your content. Refer to our Knowledge Base for assistance or via web chat, or please call us at anytime.

To follow the progress of the update, please watch this status blog for updates.

Thank you,
Your IX Web Hosting Team

Original post


WebHostsStatus.com info on ixwebhosting

ixwebhosting: DDoS (Distributed Denial of Service) attack

We are currently experiencing a DDoS (Distributed Denial of Service) attack against shared IP 98.130.148.2 on web415. In order to stop the attack on it will be blocked on 2 hours.

We will update this status blog post as we get additional information from our systems administrators.

Original post


WebHostsStatus.com info on ixwebhosting

HostExcellence: Mail310 RAID Failure – Resolved

Mail310 encountered a catastrophic RAID error and had to be recreated. Unfortunately, any mail stored on the server was lost.

Updated description of the issue:

Mail 310 is a mailserver made up of several hard drives working together in a RAID array.  The RAID array spreads data across several drives allowing for faster access speed and helps protect the server from hardware failure.  If a drive in the array fails, the server uses the data spread across the rest of the drives to recreate the data on a spare drive that remains unused in the server until needed.  Yesterday, a drive failed on this mailserver but the RAID array failed to recreate the data on the spare.  That happens sometimes if the spare encounters an error and is usually easily corrected by taking the server down for maintenance to replace the failed or spare drive.

At 11:58AM EDT yesterday, the mail server was taken off-line to replace the failed drive and upon booting the server, the RAID failed to rebuild again.  Next, we replaced the RAID card that controls the array for the server but it still failed to rebuild.

Upon further investigation, we found that the remaining RAID data had become corrupted which was the root cause of the initial array failure.  This meant we had to completely recreate the mail server and all of the data on the failed drives was unrecoverable.

Any mail sent to a domain using this mail server after the mail server was taken offline on July 28th 11:58AM EDT should be delivered to your mailbox, though it may be delayed, as some servers will delay resend attempts up to 16 hours or more.  If you have your mail client set-up to store a copy of your mail to a local computer, then anything received after the last time you connected to your mail and before the server was taken offline is lost.  If you do not have your mail client set-up to store a copy of your mail to a local computer, then any mail stored on the server has been lost.

We keep regular backups of most server types, but are unable to do so on mailservers due to the size of mailservers and the changing nature of the data on them. Shared hosting mail servers are large, serving a great number of customers, and backups of this amount of data, without affecting server performance, requires several hours.  This wouldn’t be a problem, except that the data on mail servers is different from the data on a web or database server because it’s constantly changing.  Over a several hour period, the majority of data on a mail server changes completely as mail arrives, is sent, is deleted, etc, so backup systems are not able to keep up with the constantly changing data.

If email is essential to you, we highly recommend that you set-up your email client to store a local copy of all email received, or investigate a permanent mailserver solution that services just your domains and can be backed up, such as a Cloud hosting server.

We are in the process of moving all of our shared services to a virtualized environment which will provide higher availability and further protection against problems like this in the future.

Read the original post here


WebHostsStatus.com info on HostExcellence

HostExcellence: Mail310 Maintenance – Resolved

During maintenance of our mail server mail310 our System Administrators discovered a failed drive. Unfortunately this will require a complete recreation of this server.

Due to miss-configuration during server creation our server administrators had to restart the physical creation from scratch. Our system admins tried all possible solutions. Unfortunately, As we don’t have backups for mail content the server had to be re-created without content. We apologize for any inconvenience this may have caused.

Read the original post here


WebHostsStatus.com info on HostExcellence

HostExcellence: Mail310 Maintenance

During maintenance of our mail server mail310 our System Administrators discovered a failed drive. Unfortunately this will require a complete recreation of this server.

Read the original post here


WebHostsStatus.com info on HostExcellence

ixwebhosting: Mail310 RAID Failure – Resolved

Mail310 encountered a catastrophic RAID error and had to be recreated. Unfortunately, any mail stored on the server was lost.

Updated description of the issue:

Mail 310 is a mailserver made up of several hard drives working together in a RAID array.  The RAID array spreads data across several drives allowing for faster access speed and helps protect the server from hardware failure.  If a drive in the array fails, the server uses the data spread across the rest of the drives to recreate the data on a spare drive that remains unused in the server until needed.  Yesterday, a drive failed on this mailserver but the RAID array failed to recreate the data on the spare.  That happens sometimes if the spare encounters an error and is usually easily corrected by taking the server down for maintenance to replace the failed or spare drive.

At 11:58AM EDT yesterday, the mail server was taken off-line to replace the failed drive and upon booting the server, the RAID failed to rebuild again.  Next, we replaced the RAID card that controls the array for the server but it still failed to rebuild.

Upon further investigation, we found that the remaining RAID data had become corrupted which was the root cause of the initial array failure.  This meant we had to completely recreate the mail server and all of the data on the failed drives was unrecoverable.

Any mail sent to a domain using this mail server after the mail server was taken offline on July 28th 11:58AM EDT should be delivered to your mailbox, though it may be delayed, as some servers will delay resend attempts up to 16 hours or more.  If you have your mail client set-up to store a copy of your mail to a local computer, then anything received after the last time you connected to your mail and before the server was taken offline is lost.  If you do not have your mail client set-up to store a copy of your mail to a local computer, then any mail stored on the server has been lost.

We keep regular backups of most server types, but are unable to do so on mailservers due to the size of mailservers and the changing nature of the data on them. Shared hosting mail servers are large, serving a great number of customers, and backups of this amount of data, without affecting server performance, requires several hours.  This wouldn’t be a problem, except that the data on mail servers is different from the data on a web or database server because it’s constantly changing.  Over a several hour period, the majority of data on a mail server changes completely as mail arrives, is sent, is deleted, etc, so backup systems are not able to keep up with the constantly changing data.

If email is essential to you, we highly recommend that you set-up your email client to store a local copy of all email received, or investigate a permanent mailserver solution that services just your domains and can be backed up, such as a Cloud hosting server.

We are in the process of moving all of our shared services to a virtualized environment which will provide higher availability and further protection against problems like this in the future.

Original post


WebHostsStatus.com info on ixwebhosting