vRealize

Change vRealize Operation Agent’s log level and file size

kisstib0r VMware , ,

As I promised in my previous blog article, I will show you how do you can change vRealize Operation Manager Agent’s log level and file size limit. If you want to debug a problem these two changes are helpful because you can store log information for a longer time than default 5Mb. Ok, let’s go.

The default location of bin files of the agent is “C:\ep-agent\conf\” on Windows environment. Under this folder, you will find “agent.properties” file. Open it by Notepad or your favorite editor.

agent.logLevel=INFO

Change_vROps_Agent_log_level_001

Search for “agent.logLevel” config, and change from INFO to DEBUG. Save the file and restart agent’s service. Ok, you will see more details in the log file, but the file size will grow fast, and if it will be higher then 5Mb the agent overwrite the old data. You can prevent it, so change the file size limit. The file size setting is very close to log level settings.

Change the value (in KB) end of the line, and save the file, and restart agent’s service.

log4j.appender.R.MaxFileSize=5000KB

Change_vROps_Agent_log_level_002

If you want to know more about agent property, click the link below:

https://docs.vmware.com/en/vRealize-Operations-Manager/6.6/com.vmware.vcom.core.doc/GUID-75B39838-E8A4-4FE2-B19C-DB55482D9961.html

vRealize Log Insight alert integrate with Operation Manager

kisstib0r VMware , , , ,

First of all, you can install and configure Veeam Backup and Replication content pack in vRealize Log Insight. Follow instructions of this document.

https://helpcenter.veeam.com/docs/loginsight/userguide/about_vcp.html?ver=10

  • In Log Insight jump to Content Packs / Veeam Backup & Replication / Alerts menu
  • Click “vbr – Backup job failed”

vRealize_LogInsight_alert_integrate_OperationManager_004

The content pack will create a query of failed backups.

  • Change the name of alert as you wish or leave default
  • You can edit Description and Recommendation
  • Fill “Sent to vRealize Operation Manager” checkbox
  • Hit the “Select…” button and select your Veeam Backup&Replication server (VM or Windows layer)
  • Set criticality to Critical
  • Check Auto Cancel option

vRealize_LogInsight_alert_integrate_OperationManager_000

  • Click “Send test alert” button
  • Save to my alerts

If you do everything right you will see the test alert in vRealize Operation Manager (wait 5 minutes before you start troubleshooting)

vRealize_LogInsight_alert_integrate_OperationManager_001

You can see the alert on VM or Windows layer dash or if you select Alert button, you will see the alert under it.

vRealize_LogInsight_alert_integrate_OperationManager_002

vRealize_LogInsight_alert_integrate_OperationManager_003.JPG

The alert comes as Notification event so you can create Notification settings under Alerts / Notification settings if you want to receive an e-mail alert.

vRealize_LogInsight_alert_integrate_OperationManager_005

vmware failed to read header on stream tcp n7vmacore15systemexceptione connection reset by peer

kisstib0r VMware , , ,

As I mentioned we have introduced VMware Log Insight application. I really like this VMware product, but the others too 🙂 Simple and it is working well.

LogInsight_001

Today I find this error message on our one host:

Hostd: error hostd[66282B70] [Originator@6876 sub=SoapAdapter.HTTPService.HttpConnection] Failed to read header on stream <io_obj p:0x64b7765c, h:42, <TCP ‘0.0.0.0:0’>, <TCP ‘0.0.0.0:0’>>: N7Vmacore15SystemExceptionE(Connection reset by peer)

I started to Google it, and I find this VMware Kb Article:

https://kb.vmware.com/kb/2079299

So this is known issue and no solution yet.

ICMP ping issue by Windows base End Point Operations Agent

kisstib0r VMware , , , ,

As you know vRealize Operation Manager can monitor remote devices by ICMP check. It is a simple checking method, we use it to monitor at basic level, our remote routers or switches.

Implementation process:

  1. Go Environment / All Objects / EP OPS Adapter / Remote Cheks Word
  2. From Actions menu select “Monitor this object remotely”
    • Add Display nama E.g.: WAN-1
    • Monitored from select your Agent
    • Under Check Method select ICMP check
    • Hostname: IP address of device
  3. Under Advanced settings you can change “sotimeout” and “Collection Interval” (we use one minute)
  4. Click OK

Under Environment / All Objects / EP OPS Adapter / ICMP Check tree you will find all your ICMP checked objects, and you can select one of them for details or All Metrics data.

Now, you are able to create an Alert, base on resource availability state. Yes, but we received lots of alert from vROps, becouse the resources wasn’t available many times. We tried rise up sotimeout. Not worked.

We changed Agent log level to Debug and we saw many of this messages.

26-09-2017 20:23:36,043 CEST DEBUG [pool-1-thread-11] [Collector] name=ICMP Check, thread=pool-1-thread-11, result=Tue Sep 26 20:23:36 CEST 2017 (/192.168.xxx.xxx) null values={Availability=0.0, ResponseTime=19969.0}

(In my next Article I will show you how you can change EP OPS Agent’s log level and file size.)

ICMP

We opened a new case at GSS (VMware Global Service Support) about this problem, but fortunetly in the maintime (under weeks – waited for GSS) we find a Release Note for vRealize Operation Manager 6.2. The subscription in this note looks like very similar as our problem 🙂

No data returned when running ICMP check
When attempting to run an ICMP check for remote monitoring from an Agent running on a Windows platform, no data is returned.
Workaround: Do not use ICMP checking from an agent that is deployed on a Windows platform.

Link: http://pubs.vmware.com/Release_Notes/en/vrops/62/vrops-62-release-notes.html

We changed the Agent from Windows base OS to Linux OS, and in the last weeks we didn’t revecive fals alerts. I’m really sad becouse we use vRealize Operation 6.6.1 and the problem, wich was discovered in 6.2,  is still exists in the latest release.

Add Certificate to vRealize Operation Manager tcServer Truststore

kisstib0r VMware , , ,

Why interesting this? We use VMware vRealize Operation Manager and vRealize Log Insight. This two products is integrated eachother. E.g.: when I select a VM in vROps than I select Log tab, vRLI will open in the same window, and shows me the logs wich is belong to the selected VM. In that case we won’t get cert. error message in vROps. So we can import or add vRLI cert into vROps certifiacet store.

vRealize Operation Manager handle only PEM format certificate. If you have cer file in DEM format you can convert it by OpenSSL.

Convert DER to PEM

openssl x509 -inform der -in certificate.cer -out certificate.pem

View PEM cert:

openssl x509 -in aaa_cert.pem -noout -text

Upload the file to vROps server e.g.: /tmp folder by WinSCP. Open Putty or your favorite SSH application and log in to vROps Nodes by root. Type the following command on console:

$VCOPS_BASE/jre/bin/keytool -import -alias <alias_name> -file /tmp/<cert.pem> -keystore “$VCOPS_DATA_VCOPS/user/conf/ssl/tcserver.truststore” -storepass <thisisstorepasskey> -trustcacerts

The <truststore_password> is generated by vROps and is located in /storage/vcops/user/
conf/ssl/storePass.properties. Copy the password from the ssltruststorePassword= field and paste it in the <truststore_password> placeholder.

Press enter, and say yes for question about trustid certifiaction.

Repeat the above steps on every vROps node. Don’t forget reboot the host after you ran the command successfully.

Import_cert_vRealize_Operation

Resetting VMware vRealize Infrastructure Navigator database

kisstib0r VMware , ,

Today we have deployed VMware vRealize Infrastructure Navigator appliance. This app is part of our vRealize Operation Manager Advanced and Enterprise environment. This intriduction is a new project at my Company, and I’m glad to lead that.

The reason why I’m writing this post, because when we deployed Infrastructure Navigator, we got this error message: “An unknown VM access error has occurred”

We find this article: http://www.thegurleyman.com/how-to-fix-infrastructure-navigator-error-an-unknown-vm-access-error-has-occurred/

Unfortunately it isn’t worked 😦 but I’m sharing this link, maybe will help for somebody who are looking for solution similar problem.

We find an other blog: http://www.virtualiseme.net.au/index.php/resetting-vin-database/ whitch explan how could reset database of Infrastructure Navigator.

  1. Log onto the VIN server as root
  2. Run the command : monit stop vadm-engine This will stop the VIN service
  3. Execute the file: /opt/vadm-db-config/bin/drop_db.sh This will drop the VIN database
  4. Execute the file /opt/vadm-db-config/bin/update_db.sh This will re configure the database
  5. run the command: monit start vadm-engineThis will start the VIN service The appliance should then start preparing the environment.
  6. Another good command to run is monit summary this will show the service status.

A good advice: don’t overwrite Infr. Navigator host name, because the first step won’t be work!!