Brain Byte - vCenter Emergency Mode
Welcome back all to another Brain Byte post, this one come about following a power cut while working in my lab. It wasn’t until I attempted to bring my vCenter up after the power had returned. The vCenter would no longer boot and displayed:
Failed to start file System Check....
Dependency failed for....
I had seen this before and thought this time round I would use the opportunity to capture the steps required towards fixing the issue and getting your vCenter back up and running.
This issue occurred for me because of a power failure, but this issue is also very common during storage failures, all of which can cause inconsistencies to occur within the file system.
One of the file systems was not mounting in my case, but you can use the same steps when troubleshooting any of these types of issues related to "Failed to start file System".
What to do?
1: First, power off the vCenter and take a snapshot
2: Once we have captured the snapshot, power on vCenter and open a console window on the ESXi host
3: On boot, if not already done, make a note of the file system affected, mine was /dev/db_vg/db.
If the screen has scrolled past too fast you can use “Shift + Page Up” key combination to view all information out of view.
4: When prompted log on using your root password type
To launch BASH.
5: First, do a quick check on the file systems to ensure none are full or near to full. Use the following command to show the file systems and check for space available.
Followed by the command below to show the file system configuration file
You can use both outputs to compare and confirm the affected filesystem.
You can see free space looks good, but as you will see in my case, the filesystem in question is not listed or mounted.
The Fix with e2fsck
So, knowing we have a snapshot available should we need to roll back and with the problem validated lets attempt repair.
1: Lets use the e2fsck tool to check the file systems. I used the “y” option in this case, which assumes “yes” to any fix questions and errors.
e2fsck -y /dev/db_vg/db
2: Reboot your vCenter with the following command
3: Give some time for the vCenter to boot and then carry out a health check of your vCenter, remembering, you still have a snapshot in place. When your happy all is well don't forget to remove it.
4: Finally, ensure you have recent backups of both your vCenter Configuration and Distributed Switch exports.
If you're not sure on how to do this, follow my posts around these backup essentials linked below in recent posts.
Happy holidays and all the best for 2021!!