VMware

ICMP ping issue by Windows base End Point Operations Agent

kisstib0r VMware , , , ,

As you know vRealize Operation Manager can monitor remote devices by ICMP check. It is a simple checking method, we use it to monitor at basic level, our remote routers or switches.

Implementation process:

  1. Go Environment / All Objects / EP OPS Adapter / Remote Cheks Word
  2. From Actions menu select “Monitor this object remotely”
    • Add Display nama E.g.: WAN-1
    • Monitored from select your Agent
    • Under Check Method select ICMP check
    • Hostname: IP address of device
  3. Under Advanced settings you can change “sotimeout” and “Collection Interval” (we use one minute)
  4. Click OK

Under Environment / All Objects / EP OPS Adapter / ICMP Check tree you will find all your ICMP checked objects, and you can select one of them for details or All Metrics data.

Now, you are able to create an Alert, base on resource availability state. Yes, but we received lots of alert from vROps, becouse the resources wasn’t available many times. We tried rise up sotimeout. Not worked.

We changed Agent log level to Debug and we saw many of this messages.

26-09-2017 20:23:36,043 CEST DEBUG [pool-1-thread-11] [Collector] name=ICMP Check, thread=pool-1-thread-11, result=Tue Sep 26 20:23:36 CEST 2017 (/192.168.xxx.xxx) null values={Availability=0.0, ResponseTime=19969.0}

(In my next Article I will show you how you can change EP OPS Agent’s log level and file size.)

ICMP

We opened a new case at GSS (VMware Global Service Support) about this problem, but fortunetly in the maintime (under weeks – waited for GSS) we find a Release Note for vRealize Operation Manager 6.2. The subscription in this note looks like very similar as our problem 🙂

No data returned when running ICMP check
When attempting to run an ICMP check for remote monitoring from an Agent running on a Windows platform, no data is returned.
Workaround: Do not use ICMP checking from an agent that is deployed on a Windows platform.

Link: http://pubs.vmware.com/Release_Notes/en/vrops/62/vrops-62-release-notes.html

We changed the Agent from Windows base OS to Linux OS, and in the last weeks we didn’t revecive fals alerts. I’m really sad becouse we use vRealize Operation 6.6.1 and the problem, wich was discovered in 6.2,  is still exists in the latest release.

Add Certificate to vRealize Operation Manager tcServer Truststore

kisstib0r VMware , , ,

Why interesting this? We use VMware vRealize Operation Manager and vRealize Log Insight. This two products is integrated eachother. E.g.: when I select a VM in vROps than I select Log tab, vRLI will open in the same window, and shows me the logs wich is belong to the selected VM. In that case we won’t get cert. error message in vROps. So we can import or add vRLI cert into vROps certifiacet store.

vRealize Operation Manager handle only PEM format certificate. If you have cer file in DEM format you can convert it by OpenSSL.

Convert DER to PEM

openssl x509 -inform der -in certificate.cer -out certificate.pem

View PEM cert:

openssl x509 -in aaa_cert.pem -noout -text

Upload the file to vROps server e.g.: /tmp folder by WinSCP. Open Putty or your favorite SSH application and log in to vROps Nodes by root. Type the following command on console:

$VCOPS_BASE/jre/bin/keytool -import -alias <alias_name> -file /tmp/<cert.pem> -keystore “$VCOPS_DATA_VCOPS/user/conf/ssl/tcserver.truststore” -storepass <thisisstorepasskey> -trustcacerts

The <truststore_password> is generated by vROps and is located in /storage/vcops/user/
conf/ssl/storePass.properties. Copy the password from the ssltruststorePassword= field and paste it in the <truststore_password> placeholder.

Press enter, and say yes for question about trustid certifiaction.

Repeat the above steps on every vROps node. Don’t forget reboot the host after you ran the command successfully.

Import_cert_vRealize_Operation

SWAP drive Alerts in vROps

kisstib0r VMware , ,

In our MS SQL environment we use dedicated swap drive wich almost is full becouse of we set it up Custom size. Despite of it is a normal behavior, vROps generate “Guest file system space usage” Alert. In this article I would like to show you how to manage this situation in vRealize Operation Manager and avoid unnecessary alerts.

In this example we have two similar VM with three drive: C: for OS, H: for Kernel and S: for SWAP. From Kernel drive you could figure out, it is a SAP environemt 🙂

SWAP_Alert_vRealize_Operation_001

We want to keep alert to C: and H: drive, but we don’t get alert from S: (swap) drive, unless it is full.

First of all we can create new symptoms to this VMs. You can see the table below. Create Warning (85%) , Immediate (90%) and Critical (95%) symptom to C: and H: drive too. As you can see in table.

For S: drive enough Info level and condition will be “is greather than or equal to” with value 100 (precent). So the swap drive is full, vROps will create an Alert. 100% is just an advice, you can change the value as you wish, customize your environment.

SWAP_Alert_vRealize_Operation_003

We can use “Guest File System stats / Guest File System Usage (%)” metric under effected drive. As you can see, in this list we don’t see S:\ drive.

In the interest of we see our VM’s S:\ drive click the little grey icon next to Metrics select list form.

In the new window you can find a list of all VMs, select that VM wich want to configure, and click OK, than you can see the all drive under Guest File System stats.

Choose drive eg. C:\ and under the drive letter you will find the “Guest File System Usage (%)” metric. Drag and drop to right side of window.

Create all symptoms from list of below.

 

 

 

 

SWAP_Alert_vRealize_Operation_004

Create this Symptomes

Base Object Type Metrics Threshold Name Level Condition Value
vCenter Adapter / Virtual Machine Guest File System stats / C:\ Guest File System Usage (%) Static C Drive space usage at Warning level Warning is greather than or equal to 85
vCenter Adapter / Virtual Machine Guest File System stats / C:\ Guest File System Usage (%) Static C Drive space usage at Immediate level Immediate is greather than or equal to 90
vCenter Adapter / Virtual Machine Guest File System stats / C:\ Guest File System Usage (%) Static C Drive space usage at Critical level Critical is greather than or equal to 95
vCenter Adapter / Virtual Machine Guest File System stats / H:\ Guest File System Usage (%) Static H Drive space usage at Warning level Warning is greather than or equal to 85
vCenter Adapter / Virtual Machine Guest File System stats / H:\ Guest File System Usage (%) Static H Drive space usage at Immediate level Immediate is greather than or equal to 90
vCenter Adapter / Virtual Machine Guest File System stats / H:\ Guest File System Usage (%) Static H Drive space usage at Critical level Critical is greather than or equal to 95
vCenter Adapter / Virtual Machine Guest File System stats / S:\ Guest File System Usage (%) Static S Drive space usage at Info level (SWAP) Info is greather than or equal to 100

From this symptoms we can create new Alert.

Create new Alert

Name Base Object Type Impact Criticality Alert Type and Subtype Wait Cycle Cancel Cycle
SAP LFP Apps virtual machine guest file systems are running out of disk space vCenter Adapter/Virtual Machine Health Symptom Based Virtualization/Hypervisor: Capacity 1 1

If you did everything right, you will see similar things as picture below.

SWAP_Alert_vRealize_Operation_002

Create new Custom Group

Ok, now you can create a new Custom group under Environement / Environment overview menu. Hit the green cross on top of the menu, and configure membership criteria. Part of Policy leave empty for now. In this eximpe I add exact VMs to this group. The name of the new group: “Disable SWAP drive alert”

SWAP_Alert_vRealize_Operation_005

Create new Policy

Go Administration / Policies / Policy Library and add new one. In the “6. Alert / Symptom Definitions” menu disable “One or more virtual machine guest file systems are running out of disk space” default alert, and Enable the newly created alert, in this case “SAP LFP Apps virtual machine guest file systems are running out of disk space”

SWAP_Alert_vRealize_Operation_006

In “8. Apply Policy to Groups” part select the custom group wich was created before, and click OK.

SWAP_Alert_vRealize_Operation_007

If you have any question pls. leave comment.

vRealize Operation 6.6 “hidden” enhancement

kisstib0r VMware ,

You can find lots of infomration about VMware vRealize Operation 6.6 or 6.6.1 release. You should have to read about new H5 User Interface, new DRS capability and new Dashbords.

But do not forget the small things. I really like (and nowhere find information about this enhancement) the full screen option especially on Policy window.

vRealize_hidden_enhancement_001

I use 24″ display even so when I had to modify something in the Policy, I can’t resize the window and it was annoying. In policy window we can manage batch of options like major badge treasholds, alerts, collected metric and so on. So you can jump here really offten, if you want to chenge anything in vROps you can do it here. I really missed something like feature, but looks like not only me 🙂

I discovered this option on Symtomps Definition window and Dashboard widget windows too.

vRealize_hidden_enhancement_002

vRealize_hidden_enhancement_003

Change vROps “Datastore is running out of disk space” precent centric alert

kisstib0r VMware ,

Datastores is an importent objects in every VMware environtment. Especially free space of Datastore. So I think we need collect information about free space and we want to get alert message when the free space is low or it is getting low on DS.

Fortunetly VMware vROps has a built-in alert message about this topic. Name of this alert is “Datastore is running out of disk space” wich alert send to Us e-mail when “Datastore space usage reaching “ALERT LEVEL” limit” (85/90/95%) and “Datastore space time remaining is low” symptom definitions meet the values.

VMware_vROps_Symptoms_Alert_002
Default Alert settings of Datastore is running out of disk space

As you remember in vROps Alerts is working different way as other monitoring tools (as I know, like Nagios).

vROps collect metrics from environment, in this case from Datastore through vCenter Server. We can create Symptome Definitions (Alert / Alert Settings / Symptom Definitions/Add (green cross)) from Metrics or Properties.

VMware vROps Symptoms

Alerts consists of symptoms. If symptoms reaching the level wich was seted up and match all conditions (Any / All logical separation) we have got Alert from vROps.

More informations about vROps Alerts:
https://docs.vmware.com/en/vRealize-Operations-Manager/6.6/com.vmware.vcom.core.doc/GUID-06380281-4B99-4E4B-9D4E-574E5D0A9194.html

 

Jump back to subject of this blog article. My problem was with default DS alert that we should have leave huge free spaces on big datastores becouse of this symptoms is watching free space in precent. Eg. I got alert from 8TB datastore despite of I had 700GB free space on this DS.

I created new symptom definitions:

Base Object Type: vCenter Adapter / Datastore
Metrics: Capacity / Available Space (GB)
Threshold: Static

Name Level Condition Value
Datastore Available Space reaching Warning limit Warning is less than or equel to 150
Datastore Available Space reaching Immediate limit Immediate is less than or equel to 100
Datastore Available Space reaching Critical limit Critical is less than or equel to 50

VMware vROps Symptoms

After I added this tree symptoms to “Datastore is running out of disk space” alert as new Self-Datastore objects by “Any” condition.

VMware_vROps_Symptoms_Alert_003

This alert mean: The DataStore free space is lower than 85% or 90% or 95% and 150GB or 100GB ot 50GB and Disk space time remaining <= 60 days we are going to get an Alert message from vROps.

If you have any qestion pls. leave a comment!

VMworld Europe 2017 – First day experiences #vmworld

kisstib0r VMware , ,

I would like to share my opinion and experiences about my first VMworld EU (Barcelona) happening 🙂

11/09/17 – We pick up our badge and look oround a little. We met Valentin Czuczumanov (Senior Engineer at VMware) and Nyitrai Csaba (VMUG Hungary leader)

VMworld 2017
VMworld 2017
VMworld 2017
Backpack

12/09/17 – Becouse of morning rush hour in Barcelona we arrived at Keynote a little bit late. The CEO of VMware Pat Gelsinger @PGelsinger has already started the speach. Fortunetly I got a quite good place in the “arena”

In the vmvillage had big screen where can you see the live streem from next room 🙂

Pat Gelsinger show us how we are able to migrate our VMs from on-prem to cloud in Virtual Reality. It was a funny demo with Alan Renouf @alanrenouf.

VMworld 2017 Pat Gelsinger
Pat Gelsinger at VMworld General Session

End of the general session I had to run Hall 8.0 Room 21, becouse I scheduled lots of breakout session. Fira Gran Via is a big place but VMworld also is a big event.

I catched up my session “Implementing Advanced vSphere Features with Veeam Availability Suite” (#PBO3309BES) in the time. Here is my tweet and a picture from session.

The presenters Michael Cade from UK and Clinton Wyckoff show us what is new in Veeam v10, and end of the session they demonstrate benefit of itegration Veeam and vRealize Log Insight. They also talked about how we can backup encrytped VMs and why vVOL is better then VMFS perspective of snapshot base backups.

Implementing Advanced vSphere Features with Veeam Availability Suite

Implementing Advanced vSphere Features with Veeam Availability Suite

Implementing Advanced vSphere Features with Veeam Availability Suite
Veeam 10 enhacements
Implementing Advanced vSphere Features with Veeam Availability Suite
vRealize LogInsigt and Veeam integration

After this session I met my colluages at Lunch. Until lunch time I visited some booth. I hope tumorrow I can visit more as today. Fortunetly I can spoke Saurabh Chandrate, Solution Architect at Blue Medora, about our new Blue Medora pack introduction. If somebady don’t know Blue Medora pls. visit they website. They have solution packs to VMware vRealize Operation Manager, and we have extended our vROps environment with this capabilities.

My next session was VMware Cloud on AWS: Storage deep dive  with Ben Meadowcroft

We close this day at Barcelona vs. Juventus match (3:0)

FC Barcelon vs. Juventus

Some other great guys was there also 😀

 

VMware Mirage – Only Download Layer

kisstib0r VMware , ,

I have worked a VMware Mirage introduction as i mentioned earlier. Our project team specified very short time window for restart. It is only 30 minutes. The PCs are hosts to a cash register sofware witch is start a special way. In registry of Windows the shell was changed from explorer.exe to software’s exe.

Mirage will manage this settings that way it will restart PC without any question. NAG scrren will not ask the user about restart. Becouse of PCs connect by 3G network to Mirage, the assign process time of new bas layer is unpredictable. Other side we got a short restart time.

I figured out that we upload the changes between two base layer, and we will apply the changes a later time.

VMware_Mirage_PIC01
Assign Base Layer to VCD / Select “Only Download Layer”

Follow the wizard’s steps. The changes will upload to the selected CVDs, and thats all. After this process, PC will not restart automatically. We can save about 30-45 minutes.

VMware_Mirage_PIC02

In the Tasks menu we can see the state of process “Base Layer Download” is done. Mirage server uploaded the changes to CVD.

We can apply this changes (I call that Install process) if we select task and from right click menu choose “Apply Layers” option.

VMware_Mirage_PIC03

VMware Mirage will “install” the changes about 10-15 minutes, than the PC will restart automatically. It is not depens on 3G bandwith, becouse layer update process is going on PC without network usage.

My previously blog articles about Mirage