Fixing ESXi when RAMdisks run out of disk space

I ran into an interesting problem a few weeks back with one of my ESXi hosts.  While trying to do some normal things – like vMotion – I noticed an error recorded for the tasks – nothing that seemed to point to a lot of detail – just “A general system error occurred.”  On further investigation, I found that the underlying message was an out of disk space message while trying to proceed with a Storage vMotion.

Observed errors

Attempting vMotion – “A general system error occurred:”

Attempting Storage vMotion – “/var/log/vmware/journal/xxxx error writing file. There is no space left on the device.”

Troubleshooting steps

  • Go to Configuration tab on host in vCenter client, go to Security Profile, click Properties link on the Services section.
  • Scroll down to SSH and highlight – click options – click start to start SSH service.
  • Use putty or reflections to ssh to the host.
  • If you get a connection rejected – root filesystem ramdisk is probably full.
  • Go to console (either through KVM or OA for blades)
  • F2 to login, login, arrow down to Troubleshooting Options, select Enable ESXi Shell.
  • Press ALT-F1 to change to management shell and login (same root credentials).
  • Run ‘vdf -h’ and look for root filesystem – should look like:
    Ramdisk                   Size      Used Available Use% Mounted on
    root                       32M        3M       28M  10% --
  • If it is 0M available and 100% used, that’s the problem.  Try to clear up space:
    cd /var/log/
     ls -la
  • Check size of the hpHelper.log file – likely pretty large.  Reset the file, if large.
    > hpHelper.log

Host is back online and working and it looks like the most likely culprit was the HP agents inside of the custom ESXi image provided by HP.  It seems in some circumstances that the hpHelper.log file can become very large, filling the RAMdisk and causing the issues.  Its a first for me and I have not observed the issues on any of my other ESXi hosts running on Proliant rack-mount or blade servers.  

Helpful Knowlegebase Articles

Tags: , , , , , , ,

 

About the Post

Author Information

Philip is a IT solutions engineer working for AmWINS Group, Inc., an insurance brokerage firm in Charlotte, NC. With a focus on data center technologies, he has built a career helping his customers and his employers deploy better IT solutions to solve their problems. Philip holds certifications in VMware and Microsoft technologies and he is a technical jack of all trades that is passionate about IT infrastructure and all things Apple. He's a part-time blogger and author here at Techazine.com.

3 Responses to “Fixing ESXi when RAMdisks run out of disk space”

  1. Thanks for the write up. This was my exact problem with some HP DL 360 servers running ESXi 6.

    .

    October 1, 2015 at 8:19 am Reply
    • Philip Sellers #

      Happy to hear it helped!

      October 20, 2015 at 7:10 pm Reply

Trackbacks/Pingbacks

  1. Patching HP customized ESXi installs with latest agents and drivers | Techazine - April 22, 2014

    […] problem a couple months back where excessive logging from one HP tool filled up my ESXi RAMdisk.  Filling the RAMdisk made the system unmanageable even though the VM’s continued to operate on the host.  So like any other software, you need […]

Leave a Reply

%d bloggers like this: