Well it seems that both sets of support people are confused by what seems to be a simple issue. The version of SNMP on ESXi 5.1 [including Update 2] appears to report large file systems incorrectly. The SolarWinds' gentleman doesn't seem to appreciate that with anything above ESXi 4 VMWare uses an imbedded SNMP daemon in hostd. If you want to use something else you won't get to the VM specific OIDs. The VMWare gentleman seems to think that it is the SolarWinds SNMP polling agent that needs to be upgraded. I read through the source code for net-snmp 5.7.1 for storage and clearly there was a recognition back in 2012 that large filesystems were not reported correctly and the code has a fix for that. As I said in my note to VMWare it just appears that we will be stuck with incorrect file system sizes for anything larger than 8 TB.
Zach,
It is the version on the HOST not the agent to which SolarWinds is referring. The information polled requires a calculation of the two OIDs in order to achieve the actual drive size. In this case something is not correct in what the OID is reporting via the SNMP agent on the host. I used another version of SNMPWALK from my PC and the HOST returns the same numbers.
I do not think this is a SolarWinds issue, but an SNMP reporting issue. The problem is that VMWare ESXi 5.1 contains an embedded version of SNMP agent in HOSTD which is not upgradeable as a separate package. At least to my understanding. The version of SNMP that is supposed to contain the patch SolarWinds' development team is speaking about is over two years old. The version of snmp on the patched ESXi system show this:
--------------------------------
VMware ESX SNMP agent version 5.1.0 build-1483097
SNMP Research EMANATE/Lite Agent Version 17.2.2.0
Copyright 1989-2011 SNMP Research, Inc.
----------------------------------
Which appears to be older than the one Solarwinds development team was pointing me to at source forge for net-snmp. You folks are using an SNMP agent maintained by SNMP RESEARCH as an embedded SNMP agent in your host daemon.
I cannot tell from their web site what version of SNMP the EMANATE product is based upon and the last NEWS updates they have posted are from 2011. Yet in 2012 the NET-SNMP 5.7.1 fixed the problem for large filesystems:
- reworked hrStorageTable to support large filesystems. The hrStorageAllocationUnits does not report real allocation unit size, but some calculated value so hrStorageAllocationUnits * hrStorageSize gives real size of the filesystem. This calculation happens only when hrStorageSize is too small (32bits) for filesystem size, e.g. filesystems larger than 8TB with 4096 bytes block size. This calculation can be turned off by 'realStorageUnits' config option.
Looking at the data from the SNMPWALK from my PC I get the following results:
[1.3.6.1.2.1.25.2.3] hrStorageTable(3) - [1.3.6.1.2.1.25.2.3.1] hrStorageEntry(1)
hrStorageIndex(1) hrStorageType(2) hrStorageDescr(3) hrStorageAllocationUnits(4) hrStorageSize(5) hrStorageUsed(6) hrStorageAllocationFailures(7)
1 1.3.6.1.2.1.25.2.1.8 /vmfs/volumes/505362c7-7861d1fe-f9dd-842b2b067cf4 65,536 65,515 503 -
2 1.3.6.1.2.1.25.2.1.8 /vmfs/volumes/5abbfc74-475286a1-1ede-f2814cc9279c 4,096 63,929 39,937 -
3 1.3.6.1.2.1.25.2.1.8 /vmfs/volumes/68c7ed7a-329bfe3c-ed87-4e2fec1ef742 4,096 63,929 39,725 -
4 1.3.6.1.2.1.25.2.1.8 /vmfs/volumes/505362bf-c1ecfda4-e210-842b2b067cf4 8,192 36,586 26,662 -
5 1.3.6.1.2.1.25.2.1.10 /vmfs/volumes/MUS-ESX-VMPAGEFILES 4,096 154,886,396 38,481,882 -
6 1.3.6.1.2.1.25.2.1.10 /vmfs/volumes/MUS-VMDK-1 4,096 2,147,483,647 2,147,483,647 -
7 1.3.6.1.2.1.25.2.1.10 /vmfs/volumes/MUS-VMDK-2 4,096 2,147,483,647 2,147,483,647 -
8 1.3.6.1.2.1.25.2.1.4 /vmfs/volumes/505362c6-37c34786-a33a-842b2b067cf4 1,048,576 134,144 971 -
9 1.3.6.1.2.1.25.2.1.10 /vmfs/volumes/MUS-VMDK-Logs 4,096 158,628,972 118,021,214 -
10 1.3.6.1.2.1.25.2.1.10 /vmfs/volumes/MUS-VMDK-Performance 4,096 1,334,536,356 561,207,686 -
11 1.3.6.1.2.1.25.2.1.20 Real Memory 1,024 201,313,300 19,609,268 -
Using the formula in the NET-SNMP documentation the file system sizes in gigabytes would be: 3.999; 0.244; 0.244; 0.279; 590.845; 8192; 8192; 131; 605.122; 5090.852; 191.987. All of which are correct except for the 8,192 GB or 8.00 TB.
-------------------
TO BE ABSOLUTLEY CLEAR HERE IS THE QUOTE FROM VMWARE DOCUMENTATION RELATED TO SNMP AND ESXI VERSIONS LATER THAN 4.0
Host-based embedded SNMP agent. ESXi 4.0 and later includes an SNMP agent embedded in the host daemon (hostd) that can send traps and receive polling requests such as GET requests.
You can manage SNMP on ESXi hosts with the vicfg-snmp vCLI command, but not with the vSphere Client or with the ESXCLI command.
Net-SNMP-based agent. Versions of ESX released before ESX/ESXi 4.0 include a Net-SNMP-based agent. You can continue to use this Net-SNMP-based agent in ESX 4.x with MIBs supplied by your hardware vendor and other third-party management applications. However, to use the VMware MIB files, you must use the host-based embedded SNMP agent.
----------------------
So at this point I am just stuck with the incorrect reporting of the file system sizes.
Respectfully,
Thomas A. Iannelli, Information Services Manager
Stanley Consultants, Inc.
-----Original Message-----
From: "VMware Technical Support" <webform@vmware.com> [mailto:webform@vmware.com]
Sent: Tuesday, February 11, 2014 12:35 PM
To: Iannelli, Thomas
Subject: VMware Support Request 14435542801 [ ref:_00D409hQR._50080Txbph:ref ]
** Please do not change the subject line of this email if you wish to respond. **
Hello Thomas,
This is a follow up to the voicemail I left.
After doing more research, I do not believe that they want you to update your hosts snmp version but your SolarWinds application. Based off of the site I found that build number applies to the SolarWinds application. I would suggest contacting them to update that and see if it resolves the issue. Our esxi snmp agent uses only versions like this, V1, V2, and V3.
Thank you for using VMware products.
Zackery Gregory
Technical Support Engineer
VMware Global Support Services
1-877-4VMWARE
1-877-486-7273 Office Hours: Monday - Friday 10:00 AM - 7:00 PM Mountain Time Please visit us at: http://www.vmware.com/support/