Fixing ZFS after power failure
I finally decided to buy a Uninterruptible Power Supply (UPS) for my lab environment after a short power failure crashed my environment last week. After the restart, the Network Attached Storage (NAS) refused to come online again, and while visible as datastore on ESXi/vSphere, it was unreachable.
I use a ZFS file system on NexentaStor Community Edition. Clearing errors and alerts in the admin UI didn’t bring the storage back online, apparently something more low level was broken. I checked the physical disks but they were spinning and had a good SMART status. After a fair amount of time - which is why I’m writing this blog - I finally figured out what to do.
NAS
I had to go into the admin CLI of the appliance and check for errors:
fdadm faulty
For each error that was related to the power outage I cleared the error:
fdadm repaired zdev://xxxxx
This wasn’t registered until I rebooted the appliance.
ESXi
Then on the ESXi host I had to remove the storage and attach it again through the CLI (GUI doesn’t work).
First list the attached storage devices:
esxcli storage nfs list
Remove the unreachable device
esxcli storage nfs remove -v <datastore_name>
And reattach it:
esxcli storage nfs add -H <nas_hostname> -s <path_to_share_on_nas> -v <datastore_name>