melon.org system status

Between around 11:00am and noon today (Oct. 22, 2010) the hosting provider for ambrosia.melon.org was the victim of a DDOS attack, and access to ambrosia was prevented. The attack appears to have been blocked/stopped.

The hosting provider for ambrosia.melon.org has indicated that the system will have to be moved in a window starting at 11pm EST tonight (March 23rd) and ending by 6am tomorrow morning. The move is to a temporary location, so there will be another such outage in a couple of days. Actual downtime is expected to last a few hours in both cases, as the temporary location is in another (nearby) city, and there are several machines being moved.

Logs indicate that ambrosia.melon.org was down between 02:06 and 02:55 EDT, last night (i.e. March 21). This is currently assumed to be due to a power interruption, and will be checked with the hosting provider.

ambrosia.melon.org will also need to be physically moved some time during the next month and will be down during the move. An update will be posted when the move is scheduled.

At roughly midnight on Apr. 4, melon.org stopped responding to pings and other network traffic. Physical access to the server was delayed until late on the night of Apr. 6. After a few false starts bringing it back up, it was determined that the filesystem journal in the root filesystem was corrupt, and that this failure was crashing the system (ironically, filesystem journals are intended to hasten recovery of a downed system after a crash). The corrupt journal has been disabled and will remain so until it can be determined whether there are underlying hardware problems (and they are fixed, if they exist).

All services were resumed at around 3:40am on Apr. 6.

If the cause is a failing disk in the root filesystem’s array, there will be some scheduled downtime in the near future to replace that disk.

Postfix and mailman configurations have now been successfully adapted, and mailing list mail is getting delivered.

The recent switch in mailservers from Exim to Postfix is preventing mailman from working properly. Mailing list mail sent before Friday would bounce as undeliverable (postfix not recognizing the list addresses as existing). Now mailman receives the mail, but is unable to send it out, owing to postfix’s refusal to relay the message to the expanded list. Another update will be posted when this is fixed.

The mailservice on ambrosia has been changed from exim to postfix, and the default mailbox format has been changed from mbox to Maildir.

The RAID array containing the /home filesystem continues to be short a disk, and is running non-degraded as a smaller (2+1 instead of 3+1) array. While there will be downtime to replace the disk and grow the array (and filesystem) back to (3+1)x500GB, the scheduling of this downtime is not yet forseeable.

All other systems are working properly.