ExaGrid EX13000E installation experience

Short story

We have decided to change our backup appliance to ExaGrid solution. Why, I will explain it.

End of the 2017 we we changed from Networker backup sofware to Veeam Backup & Repliaction. Our environment is 100% virtualized by VMware vSphere. Restore test were part of the Powerpuff (… save the day… :)) project. We realized restore process, especially Instant VM recovery feature was very slow. One normal sized VM can’t start under 15 minutes. Reason of the slow restore was rehydration deduped data.

We didn’t want to lose this great feature becouse of our backup appliance, especially our purpose with the Veeam was made faster our restore process. We want to restore whole VM or service (multiple VMs) minutes, not hours like previously solution.

We have started calculate maintenance fee and compire it with new Dell EMC DataDomain and ExaGrid. Fee of 3Y support of Dell EMC 2500 we can buy a new Dell EMC DD 3300. It’s looks like a good deal, but when we dive into technical details realize that DD3300 has capacity limit. The next size from DataDomain is 6300 schould be oversized, and over our budget. We calculated EX13000E is pass our DR environment. It was necessary becouse of size of landing zone. We calculated 5Y TCO and ExaGrid price was lower than competitors.

We have wrote to ExaGrid and they helped us sizeing, and chooese the right equipment. Fortunately they said EX13000E. The ExaGrid approach: store your last week backups on Landing Zone, witch is a non deduplicated area. If you have to fast restore, you want it from latest data, and you can it thanks Landing zone.

Fastest-Restore-Recovery-VM-Boots-Tape-Copies

I would like highlited some interesting capability:

  • Landing zone of course
  • Scale out architecture: if you add node to your cluster, your throughput will extend
  • Every node is a controller too
  • Grow at easy, just add any type of node to your grid
  • Global deduplication amongs nodes
  • Integrated Veeam Data mover (More: http://www.exagrid.com/wp-content/uploads/ExaGrid-Veeam-End-to-End-Backup-Storage-Data-Sheet.pdf)
  • DR: I recommend to you use ExaGrid on your Primary and DR site to repliaction too. Otherwise your replicated data will rewrite your data in landing zone. Eg.: if you use Veeam replication, your data will landing on landing zone. If you replicated data among nodes, data are going to store retention repository

Veeam Backup & Replication Enterprise Plus is recommanded to ExaGrid. Scale-out Backup Repository with unlimited repository supported only Ent. Plus licenc.

Details: https://www.veeam.com/veeam_availability_suite_9_5_editions_comparison_ds.pdf

You can reach ExaGrid’s data sheet on this link: http://www.exagrid.com/wp-content/uploads/ExaGrid-Product-Overview-Data-Sheet.pdf 

 

Hardware:

It is brilliant. Normal Supermicro server with Seagate Enterprise Capacity 3.5 HDDs. CPU and memroy depends on you choose. 1G network is default but you can add optional 10G network too. The magic happen in the Software! 🙂 It is a customized Debian base linux.

Full specification: http://www.exagrid.com/wp-content/uploads/ExaGrid-Technical-Specifications-Data-Sheet.pdf

Pictures from installation day

After 2 weeks our order the magix box was arrived at our DR site.

EXAGRID BOX
EXAGRID EX13000E box

The guys from CZ (Distributor of ExaGrid) put it into rack.

ExaGrid
Install ExaGrid EX 13000E

Cabling, Power On and set up IPs, add mail server…

cof
Power On ExaGrid 13000E

The installation process was 2 hours only. Engineer from ExaGrid checked the setups remotely, fix NTP and some config. Next day we add it to our Veeam environment.

Complete ExaGrid EX13000E
Complete ExaGrid EX13000E

I really proud of Hungary first ExaGrid installation! 🙂

 

vRealize Orchestrator Management Pack for vRealize Operations

I have red an interesting blog article from John Dias about VMware vRealize Operation Manager and vRealize Orchestrator integration capability. Link below.

https://blogs.vmware.com/management/2018/04/introducing-vrealize-orchestrator-management-pack-vrealize-operations.html

John shows us an example of an alert recommendation that launches a vRO workflow to decommission a host. It is amazing, we are able to create a workflow to alerts. The operator just pushes the “RUN ACTION” button and the pre-implemented flow will guide her. I like it! 🙂

2018-04-30-10_38_56-10-172-209-86-vrealize-operations-manager

VMware vRealize Operation 6.7 upgrade experience

I have upgraded our vRealize Operation 6.6.1 test environment to version 6.7. Please let me share my experience about the test.

Our test environment consists of one Remote Connector and one Data node.

First of all, I checked this blog articles:

https://blogs.vmware.com/management/2018/04/assess-upgrade-readiness-vrealize-operations-6-7.html

https://docs.vmware.com/en/vRealize-Operations-Manager/6.7/rn/vRealize-Operations-Manager-67.html#installUpdate

vRealize Operations Upgrade Assessment Tool

From my.vmware.com site (under 6.7 download page) you can download vRealize Operations Upgrade Assessment Tool. “APUAT-6.7.0.8224833.pak”

Install was very easy, just install pak file on the admin page. Install created a log file automatically. Download it, and browse to \apuat-data\report\ folder, and open the index.html file. As you can red in the VMware’s blog article, you are able to check your dashes, widgets and so on. Fortunately, I didn’t have to worry about metrics, tools find only one matric witch should be a problem.

vRealize_6.7_002_mod

vRealize Operations Upgrade Assessment Tool
vRealize Operations Upgrade Assessment Tool

I downloaded OS (vRealize_Operations_Manager-VA-OS-6.7.0.8183616.pak) and VA (vRealize_Operations_Manager-VA-6.7.0.8183616.pak) upgrade pak files, from my.vmware.com site.

vRealize Operation OS update process

I opened vRealize Operation’s admin page (“https://FQDNorIP/admin”) on System Status page hit “TAKE OFFLINE” button. Wait a few minutes.

Take Offline vRealize
Take Offline vRealize

After the nodes went to offline mode, switch to vSphere client and take snapshot from VM-s (deselect: Snapshot the virtual machine’s memory).

vRealize_6.7_000_SnapShots

Then go back vRealize Operation admin page and click “System Updates” link. Click “INSTALL A SOFTWARE UPDATE” blue icon and browse OS upgrade pak file (vRealize_Operations_Manager-VA-OS-6.7.0.8183616.pak). Select Install the PAK file is already to upload and install option. Then click “UPLOAD” button.

vRealize_6.7_006_Install_OS_Upgrade

Wait approximately 5-10 minutes, and you will see similar the picture below. The PAK file signature is valid. Next.

vRealize_6.7_007_Install_OS_Upgrade

Accept EULA, and go next.

vRealize_6.7_008_Install_OS_Upgrade

Read update information and click Next.

vRealize_6.7_009_Install_OS_Upgrade

Click “INSTALL” button.

You see the OS update process on the screen.

vRealize_6.7_010_Install_OS_Upgrade_MOD

Click the icon next to Status column for details.

vRealize_6.7_011_Install_OS_Upgrade

Wait 30-45 min for the end of the process. If you did everything right you will see a similar screen on your monitor. Status Completed. Don’t worry, under the update process admin page will disappear. Be patient, and hit the refresh button, the page will be returned. You have to relogin.

vRealize_6.7_012_Install_OS_Upgrade_MOD

vRealize Operation VA update process

Ok, now we can upgrade vRealize Operation, the OS upgrade process is done. Similar steps as OS upgrade, except you can browse the vRealize_Operations_Manager-VA-6.7.0.8183616.pak file and you can select “Reset Default Content…” option. After you fill the box, a warning message will appear. Read it and select I understood button 🙂

vRealize_6.7_013_Install_VA_Upgrade

vRealize_6.7_014_Install_VA_Upgrade

The admin page under this upgrade process will disappear too, don’t panic.

vRealize_6.7_018_Install_VA_Upgrade_MOD

It’s time to login to the new vRealize Operation Manager.

vRealize_6.7_019_NewVersion

vRealize_6.7_020_NewVersion

The OS and vRealize Operation upgrade process is done. Now I go and discover new features in vRealize Operation 6.7 🙂

If U have any question, don’t hesitate to contact me.

Log Analytics by VMware Log Insight – VMUG presentation

I would like to share my VMUG presentation about VMware Log Insight.

Advice for installation:

  • Install one medium size appliance, use different IP wich U use to cluster
  • Use Eager Zeroed DS because of avoiding performance issues
  • End of configuration don’t configure vSphere hosts
  • Create VIP despite of you don’t want a cluster!!!
  • Use DNS name resolution
  • Now U can integrate Log Insight with your vSphere environment

If you miss creating VIP address for Log Insight, and you integrate it with your vSphere environment, the IP address of Log Insight node will write to Syslog.global.logHost value. If you want to change your IP of Log Insight, or you want to create cluster environment (eg.: scaling) you should change log value on every vSphere host.

In case of you rerun vSphere integration process in Log Insight, it does not solve your problem, because of Log Insight will add the new address to value (don’t overwrite it).

Yes, I know, you can change it by script, but I think it is a better solution, isn’t it? 🙂

Sizing: http://www.vmware.com/go/loginsight/calculator

I’m planning to create a new article series, where I will share more details about the install process and content packs integration. I have created more than 100+ page documentation in this topic under introduction. It is not finished yet! 🙂


 

vThing has created a great new blog post, you can reach details of the event on this link:
https://vthing.wordpress.com/2018/03/05/vmug-hungary-2018-q1-recap/


If you have any question, don’t hesitate to contact me!

Change vRealize Operation Agent’s log level and file size

As I promised in my previous blog article, I will show you how do you can change vRealize Operation Manager Agent’s log level and file size limit. If you want to debug a problem these two changes are helpful because you can store log information for a longer time than default 5Mb. Ok, let’s go.

The default location of bin files of the agent is “C:\ep-agent\conf\” on Windows environment. Under this folder, you will find “agent.properties” file. Open it by Notepad or your favorite editor.

agent.logLevel=INFO

Change_vROps_Agent_log_level_001

Search for “agent.logLevel” config, and change from INFO to DEBUG. Save the file and restart agent’s service. Ok, you will see more details in the log file, but the file size will grow fast, and if it will be higher then 5Mb the agent overwrite the old data. You can prevent it, so change the file size limit. The file size setting is very close to log level settings.

Change the value (in KB) end of the line, and save the file, and restart agent’s service.

log4j.appender.R.MaxFileSize=5000KB

Change_vROps_Agent_log_level_002

If you want to know more about agent property, click the link below:

https://docs.vmware.com/en/vRealize-Operations-Manager/6.6/com.vmware.vcom.core.doc/GUID-75B39838-E8A4-4FE2-B19C-DB55482D9961.html

vRealize Log Insight alert integrate with Operation Manager

First of all, you can install and configure Veeam Backup and Replication content pack in vRealize Log Insight. Follow instructions of this document.

https://helpcenter.veeam.com/docs/loginsight/userguide/about_vcp.html?ver=10

  • In Log Insight jump to Content Packs / Veeam Backup & Replication / Alerts menu
  • Click “vbr – Backup job failed”

vRealize_LogInsight_alert_integrate_OperationManager_004

The content pack will create a query of failed backups.

  • Change the name of alert as you wish or leave default
  • You can edit Description and Recommendation
  • Fill “Sent to vRealize Operation Manager” checkbox
  • Hit the “Select…” button and select your Veeam Backup&Replication server (VM or Windows layer)
  • Set criticality to Critical
  • Check Auto Cancel option

vRealize_LogInsight_alert_integrate_OperationManager_000

  • Click “Send test alert” button
  • Save to my alerts

If you do everything right you will see the test alert in vRealize Operation Manager (wait 5 minutes before you start troubleshooting)

vRealize_LogInsight_alert_integrate_OperationManager_001

You can see the alert on VM or Windows layer dash or if you select Alert button, you will see the alert under it.

vRealize_LogInsight_alert_integrate_OperationManager_002

vRealize_LogInsight_alert_integrate_OperationManager_003.JPG

The alert comes as Notification event so you can create Notification settings under Alerts / Notification settings if you want to receive an e-mail alert.

vRealize_LogInsight_alert_integrate_OperationManager_005

vmware failed to read header on stream tcp n7vmacore15systemexceptione connection reset by peer

As I mentioned we have introduced VMware Log Insight application. I really like this VMware product, but the others too 🙂 Simple and it is working well.

LogInsight_001

Today I find this error message on our one host:

Hostd: error hostd[66282B70] [Originator@6876 sub=SoapAdapter.HTTPService.HttpConnection] Failed to read header on stream <io_obj p:0x64b7765c, h:42, <TCP ‘0.0.0.0:0’>, <TCP ‘0.0.0.0:0’>>: N7Vmacore15SystemExceptionE(Connection reset by peer)

I started to Google it, and I find this VMware Kb Article:

https://kb.vmware.com/kb/2079299

So this is known issue and no solution yet.