melon.org system status

Owing to the world going cloud, the colo provider for ambrosia.melon.org is relinquishing his physical space to host servers. People who had virtual servers living on ambrosia were all contacted, and either expressed a willingness to give up their server, or were assisted in migrating to a new virtual hosting provider. Most of the functionality of various melon.org vservers was rolled into horned.melon.org, and horned.melon.org was migrated to ambrosia’s colo provider’s virtual hosting. There was negligable disruption to any hosted services during the migration.

At around 18:00 EDT on 2022-04-28, all of the virtual servers had been stopped (whether their eventual fate was to migrate, or to be decommissioned themselves), and ambrosia.melon.org was halted.

The move scheduled for the coming Sunday (Feb 5) has been postponed indefinitely. It may prove unnecessary to physically move ambrosia.melon.org at all, though I am still waiting for a solid decision one way or another from the hosting provider. The previous notice can be safely ignored.

ambrosia.melon.org is being physically moved within the datacentre where it’s hosted (probably between floors). While it is being moved, it will be powered down and disconnected from the internet. The maintenance window for this move has been scheduled for 1900-2100 EDT on Sun. Feb. 5. Actual outage time is expected to be around 20-30min. An update will follow when the move has been completed.

UPDATE: This move has been postponed indefinitely. Details here.

The reboot for which tonight’s maintenance window was opened, has successfully completed. A sampling of services has been selected, and all of them appear to have restarted normally.

There will be a maintenance window on ambrosia.melon.org from midnight to 6am on Sunday October 30th, EDT. During this time, the server will be rebooted in order to boot into a new kernel that is not vulnerable to the dirtycow exploit. Although the maintenance window is scheduled to last 6 hours, actual downtime is not expected to exceed 15 minutes near the beginning of the window.

In addition to status.melon.org, rolling.melon.org and kiwano.melon.org have been fully restored. deadly.melon.org has also been partially restored; because the Gallery 2 site spotted.melon.org which was integrated with deadly.melon.org has not yet been restored, the integration functionality has not been restored either.

Further updates will follow as the remaining functionality is restored.

In mid-late May, an OS upgrade on the various melon.org containers tried to switch to systemd, while the kernel on the host server did not support this version of systemd. This permanently wedged a few containers, which had to be rebuilt, and have their files copied over. This also led to about a day of intermittent outage on the host server, as verious attempts to unwedge the server were made. Since then, the special config line to tell the OS to continue to use sysvinit instead of systemd has been deployed on all the servers, host and container alike, and the upgrade to jessie has been made. Most services were restored within a couple of days, but some of the CMS-based webservices (like wordpress) refused to restore cleanly from their backups.

Owing to other things to do, restoring wordpress sites got a low priority, and the first restoration has only just happened (this site: status.melon.org). Updates will be provided as the other CMS-based webservices get restored.

Additionally, there are problems with SSH. healthy.melon.org is inaccessible by SFTP, and logging from horned.melon.org to itself via SSH generates spurious error messages in the terminal. These issues will be addressed after the CMS-based webservices are restored.

A while ago, virtual servers based on LXC were set up on amprosia.melon.org. More recently, services have been migrated from the host server to container servers. During this process, some services have been deemed to be abandoned, and were simply removed instead of being migrated. The services that have been removed are:

conspiracy.melon.org (website for organizing activities that never really attracted users)
dispossessed.melon.org (disused blog)
dense.melon.org (phpmyadmin for ambrosia’s DB)
shiny.melon.org (gallery site that hasn’t seen any use in years)
winter.melon.org (abandoned blog)

The mailman mailing lists were split up, with Anarres Natural Health lists migrating to lists.anarreshealth.ca, and the remaining lists continuing as fleshy.melon.org, but on its own virtual server. It should be noted that list@melon.org is no longer a deliverable address for any of these lists, since melon.org and fleshy.melon.org no longer have their mail handled by the same server.

Most other services have been migrated either to horned.melon.org, or to vsrv1.annareshealth.ca (AKA healthy.melon.org)

On Aug. 8 at roughly 22:45, there was a UPS malfunction at the facility hosting ambrosia.melon.org. This interrupted the power only temporarily, but due to a bug in a configuration script for the bootloader that was recently run during an upgrade, the system could not be brought back up until approximately 01:45. The virtual servers were not restarted until this morning, as they don’t see heavy enough use to have justified staying up even later.

ambrosia.melon.org and all hosted sites and virtual servers were down yesterday evening for a scheduled move. This occured between roughly 10:30pm and 11:00pm. All services appear to be have been restored to normal function by the end of this time.