melon.org system status

I forgot to mentioned some scheduled downtime tonight, during the window of 9pm to midnight for a physical move of the server. Total downtime shouldn’t exceed 15 minutes.

Due to a bug in the bootloader reconfiguration scripts, information about previous drive changes in the disk array on ambrosia were not properly propagated to the bootloader. Absent this information, the server was unable to boot after the physical move last night. Diagnosis and a work-around for this bug took roughly 3.5h, extending the scheduled outage until around midnight last night. A bug report has been submitted to the script maintainers, and the workaround has been noted for future drive changes.

The server hosting all melon.org services is scheduled for a physical move during the window of 2000-2200 on Tuesday July 17, 2012. During this time, the server will need to be powered down, rendering all services unavailable. Total downtime is not expected to exceed 15 minutes.

Due to a sudden need to change IP addresses, there was some unscheduled intermittent connectivity with all melon.org services (roughly 4 outages of no more than 2.5minutes each) yesterday afternoon and evening, and additional intermittent connectivity with melon.org virtual servers (i.e. resonant.melon.org, healthy.melon.org, poisonous.melon.org, hidden.melon.org, and seedless.melon.org) earlier today (roughly 3 outages per virtual server, each lasting no more than a minute).

The migration has been completed, and no additional loss of connectivity is anticipated at this point.

ambrosia.melon.org and all its associated services will be down for a few minutes later this week, in order to physically move the machine from its current rack to an adjacent rack in the datacentre. Additional details will follow when the move window is scheduled with the hosting provider.

All melon.org services have been fully restored. The server is still running on a degraded RAID array, but a replacement drive to restore redundance should be installed by Thursday morning (Dec. 22).

Mailservice on ambrosia.melon.org has been restored, and the MX records for melon.org have been repointed at the new hardware. Additionally, anarreshealth.ca and shiny.melon.org have been successfully migrated to the new hardware. These fixes may not yet be visible to some users at the time of this posting, due to DNS propagation delays.

Still not working on the new hardware are spotted.melon.org (which did not have a current backup at the time of the crash and will have to be rebuilt entirely) and mailman/fleshy.melon.org. Since spotted.melon.org is embedded into deadly.melon.org, that bit of functionality is also unavailable.

Mail for melon.org has been moved off-site pending recovery/migration of mail service on the new hardware.

Mailing lists hosted at fleshy.melon.org are offline until melon.org mail service is restored locally.

Websites based on static content or on WordPress have been successfully migrated to and recovered on the new hardware. These sites are:

• conspiracy.melon.org
• deadly.melon.org
• dispossessed.melon.org
• juicy.melon.org
• kiwano.melon.org
• rolling.melon.org
• status.melon.org
• unripe.melon.org

Websites based on drupal and on gallery have not yet been migrated/recovered, but some are still half-working on the old hardware. These sites are:

• www.anarreshealth.ca (and any additional domains pointing to it)
• shiny.melon.org
• spotted.melon.org

Additionally, with spotted.melon.org out of service, the photo galleries in deadly.melon.org are also out of service.

The new hardware continues to run with a degraded RAID array (leaving it vulnerable to a single disk failure) but a warranty replacement for its failed disk is reportedly on its way.

The hardware upgrade and physical move has taken long enough to coordinate that a second disk in the RAID array hosting /home has failed. Consequently many melon.org services are non-functional or severely hobbled. I am hassling my hosting provider to hasten the migration to the new hardware, so that this problem can be remedied as quickly as possible. Some data on the new hardware may be out of date.

Owing to age, physical size, power consumption, and a recent disk failure, the existing hardware for ambrosia.melon.org is going to be retired and everything migrated over to new hardware (which will be faster, have more space, and on which I will also set up vservers). Additionally, the new hardware will be set up in a facility to which I’ll have full access (so any future outages should produce less downtime–unless I’m on vacation or something).

During the transition between machines, some services may become read-only, or be completely inaccessible. More details will become available as I sort things out with my hosting provider.