Recovery complete.

2011.12.19

All melon.org services have been fully restored. The server is still running on a degraded RAID array, but a replacement drive to restore redundance should be installed by Thursday morning (Dec. 22).

Further recovery update.

2011.12.19

Mailservice on ambrosia.melon.org has been restored, and the MX records for melon.org have been repointed at the new hardware. Additionally, anarreshealth.ca and shiny.melon.org have been successfully migrated to the new hardware. These fixes may not yet be visible to some users at the time of this posting, due to DNS propagation delays.

Still not working on the new hardware are spotted.melon.org (which did not have a current backup at the time of the crash and will have to be rebuilt entirely) and mailman/fleshy.melon.org. Since spotted.melon.org is embedded into deadly.melon.org, that bit of functionality is also unavailable.

Recovery update.

2011.12.14

Mail for melon.org has been moved off-site pending recovery/migration of mail service on the new hardware.

Mailing lists hosted at fleshy.melon.org are offline until melon.org mail service is restored locally.

Websites based on static content or on WordPress have been successfully migrated to and recovered on the new hardware. These sites are:

  • conspiracy.melon.org
  • deadly.melon.org
  • dispossessed.melon.org
  • juicy.melon.org
  • kiwano.melon.org
  • rolling.melon.org
  • status.melon.org
  • unripe.melon.org

Websites based on drupal and on gallery have not yet been migrated/recovered, but some are still half-working on the old hardware. These sites are:

  • www.anarreshealth.ca (and any additional domains pointing to it)
  • shiny.melon.org
  • spotted.melon.org

Additionally, with spotted.melon.org out of service, the photo galleries in deadly.melon.org are also out of service.

The new hardware continues to run with a degraded RAID array (leaving it vulnerable to a single disk failure) but a warranty replacement for its failed disk is reportedly on its way.

Disk failure.

2011.12.08

The hardware upgrade and physical move has taken long enough to coordinate that a second disk in the RAID array hosting /home has failed. Consequently many melon.org services are non-functional or severely hobbled. I am hassling my hosting provider to hasten the migration to the new hardware, so that this problem can be remedied as quickly as possible. Some data on the new hardware may be out of date.