							 Perfstat v8.4 Release Guide:
							--------------------------------

1.	Introduction:
	-------------

Perfstat v8.4 is a diagnostics data collection tool for clustered Data ONTAP and Data ONTAP running in 7-Mode
Perfstat 8.4 GUI tool:
	*	Captures performance data from multiple clusters by executing a single command. 
	*	Automatically removes clusters that are not reachable and continues with the remainder of the clusters. 
	*	Captures all of the required performance information for multiple clusters with one command. 
	*	Automatically gathers information for MetroCluster and collects performance data from a single command. CLI option (-mcc) 
	*	Captures information from node(s). 
	*	Captures information from local host and remote host(s). 
	*	Captures information from network switch(es). 
	*	Captures all of the information simultaneously for cross correlation. 
	*	Clears data separation for multiclusters run in the output folder. 
	*	All captured data is returned as a zip/tar file.
GUI Capabilities [https://kb.netapp.com/support/index?page=content&id=S:1015436&actp=LIST]
	*	Auto-configuration of the controllers.
	*	Auto detect controller configuration.
	*	SSH-key encryption on local host.
	*	Incremental runs of the Perfstat.
	*	Functionality to schedule Perfstat.
	*	Keeps track of Perfstat runs.


Perfstat v8.4 comes in exactly four variants:
	*	Linux [32-bit] 
	*	Linux [64-bit] 
	*	Windows version (perfstat8.exe)
	*	Apple Mac OS X (perfstat8)

For more information, see Building_Linux_Dependencies.txt file.


2.	Supported platforms
	--------------------
	 Perfstat v8.4 supports various Data ONTAP versions, local hosts, remote hosts, and switches.

	2.1	Supported Data ONTAP versions
	Perfstat 8.4 tool supports the following Data ONTAP versions:
		*	Clustered Data ONTAP 8.0*
		*	Clustered Data ONTAP 8.1*
		*	Clustered Data ONTAP 8.2*
		*	Clustered Data ONTAP 8.3*
		*	Clustered Data ONTAP 9.0

	2.2	Supported Client OS versions
Perfstat 8.4 tool supports the following versions of clients OS:
		*	Windows 2003
		*	Windows 2008
		*	Windows XP 
		*	Windows Vista
		*	Windows 7
		*	Windows 8
		*	Linux
		*	Apple Mac OS X Snow Leopard [ Intel x86-64 ]
		*	Apple Mac OS X Lion [ Intel x86-64 ]

	2.3	Supported switches

	Perfstat v8.4 is capable of collecting data from network switches using SSH. 
	The prerequisite to collect data from the switch is that the switch must support SSH and SSH authentication and network communication must be enabled. 
	The currently supported switch vendors/models are:
		*	Cisco: Cisco nexus 5000 series
		*	Brocade: Brocade 4100 switch series
		*	Qlogic: SANbox 5602 FC switch

	2.4	Supported Local host OS versions

	Perfstat v8.4 collects data from local host by default. The following operating systems are supported for local host.

		1.	Linux

		2.	Windows
			*	Windows 8
			*	Windows 7
			*	Windows XP
			*	Windows Vista
			*	Windows 2003
			*	Windows 2008
		3.	Apple Mac OS X
			*	Mac OS X Snow Leopard [ Intel x86-64 ]
			*	Mac OS X Lion [ Intel x86-64 ]
			
	2.5	Supported Remote host OS versions

	Perfstat v8.4 collects data from the remote host(s) if specified with command line. The prerequisite to collect data from remote hosts is that it should have support for SSH. The only exception to this rule is for hosts running AIX where RSH is mandatorily configured. 
	The following operating systems are supported as remote host:

		*	Linux
		*	Solaris
		*	HP-UX
		*	FreeBSD
		*	ESX server 3.5
		*	AIX
		
	2.6	Supported GUI Local host OS versions
		1.	Linux
			*	Ubuntu 14.04 LTS
		1.	Windows
			*	Windows 8
			*	Windows 7
			* 	Windows XP
			*	Windows 2008r2
		2.	Apple Mac OS X
			*	Mac OS X Snow Leopard [ Intel x86-64 ]
			*	Mac OS X Lion [ Intel x86-64 ]


3.	Perfstat v8.4 Cluster-Mode
	--------------------------

	 Perfstat v8.4 running on Cluster-Mode executes only in the current mode by establishing the interactive SSH session with the node. The Legacy mode is not supported in clustered Data ONTAP.


4.	Prerequisite to run  Perfstat v8.4 in cluster-Mode:
	--------------------------------------------------

	The set of rules which need to be followed while using  Perfstat v8.4 to collect data from the controllers running with clustered Data ONTAP are

	*	The 'diag' user account must be unlocked on the controller to collect SystemShell data.
	*	Data ONTAP 8.3 and higher do not need diag account to be unlocked for collecting SYSTEMSHELL data.
	*	PROFILES collection failed if the diag password has not been provided in the CLI.


5.	Caveats & Known Issues
	----------------------
	a.	Perfstat v8.4 may fail or behave unexpectedly if:
			*	FS.0 and higher do not require diag password to execute SYSTEMSHELL commands. 
			*	PROFILES collection will fail if diag password has not been given while running Perfstat. 
			*	SYSTEMSHELL commands do not execute if diag-password authentication failed in cluster-Mode. 
			*	Directory clean up has issues in conjunction with CLI options >--PROFILES and >--Clean-Up. 
			*	A controller name cannot be resolved from the host running the Perfstat utility. 
			*	User does not have write permissions in the directory in which Perfstat is running. 
			*	A system shell account for the "diag" user is necessary for Perfstat to collect the system shell commands. 
			*	If individual commands fail to execute they will be logged in the console log.  

	b.	When using Windows clients to run Perfstat, when specifying the password as part of the Perfstat authentication, if the password meets the following conditions:
		It has an "@" character, place quotes at the beginning and end of the password. 

	c.	The "--nodes" option is not supported for collected performance information from multiple nodes running in different Data ONTAP modes. 
		(7-Mode/Cluster-Mode).  If performance data is required from Data ONTAP operating in 7-Mode and clustered Data ONTAP nodes at the same 
		Time. Two separate instances of Perfstat  should be used, one for each mode.

	d.	Perfstat running on Linux clients supports only empty passphrases.

	e.	Known BURTs
 
		BURT	Description	
		405679	Multiple instances of Perfstat running against the same cluster will cause one instance of Perfstat to fail.	   
		316710	Requests can time out if a controller is under severe load, resulting in all of the data not being collected.	   
		574443	In 7-Mode, if the system shell password is provided either incorrectly at the first time or Perfstat hangs during the second attempt.
		690021	Sometimes Perfstat fails to make an ssh connection with Data ONTAP 8.2 running on 7-Mode controller. 
		

6.	Structure of the output directory (Cluster-Mode)
	------------------------------------------------
	Output file-
		*	console.log
		*	Perfstat.preset
				*	cluster mgmt IP1	 	
						1.	node-ip1(Under Cluster mgmt IP1)	
									*	output.data
									*	output.index
									*	Logs (if rastrace (or) profile (or) sktrace (or) log is used).
	
						2.	node-ip2(Under Cluster mgmt IP1)	
									*	 output.data
									*	output.index
									*	Logs (if rastrace (or) profile (or) sktrace (or) log is used).
							...
				*	cluster mgmt IP2
						1.	node-ip1(Under Cluster mgmt IP2)
									*	output.data
									*	output.index
									*	Logs (if rastrace (or) profile (or) sktrace (or) log is used).
						2.	node-ip2(Under Cluster mgmt IP2)	
									*	 output.data
									*	output.index
									*	Logs (if rastrace (or) profile (or) sktrace (or) log is used).
							...
					...
				*	host		
						1.	Local-host-IP	
									*	output.data
									*	output.index
						2.	Remote-host-ip1	
									*	output.data
									*	output.index
						3.	Remote-host-ip2	
									*	output.data
									*	output.index
						        .
				*	switch
						1.	switch-ip1	
									*	output.data
									*	output.index
						2.	switch-ip2 
									*	output.data
									*	output.index
							..


7.	Structure of the output directory(7-Mode)
	-----------------------------------------
	Output file-
		*	console.log
		*	Perfstat.preset
				*	0.0.0.0	 	
						1.	node-ip1(Under Cluster management IP1)	
									*	output.data
									*	output.index
									*	Logs (if rastrace (or) profile (or) sktrace (or) log is used).
	
						2.	node-ip2(Under Cluster management IP2)	
									*	output.data
									*	output.index
									*	Logs (if rastrace (or) profile (or) sktrace (or) log is used).
							...
				*	host		
						1.	Local-host-IP	
									*	output.data
									*	output.index
						2.	Remote-host-ip1	
									*	output.data
									*	output.index
						3.	Remote-host-ip2	
									*	output.data
									*	output.index
						        .
				*	switch
						1.	switch-ip1	
									*	output.data
									*	output.index
						2.	switch-ip2 
									*	output.data
									*	output.index
							..


8.	Help Manual (run: perfstat8 --help)
	-----------------------------------

NAME 
perfstat v8.4 (20160624_133824) Supported Converged ONTAP

SYNOPSIS
Perfstat is a performance diagnostic tool supplied and supported by NetApp.
Usage:  perfstat [OPTIONS] <a node-management IP of a node in the cluster>

OPTIONS:
-h      --help                To see a complete detailed usage
-v      --version             Prints perfstat version, and
                              supported ONTAP versions.
-q      --quiet               Suppress all non-error messages
        --verbose             Increase verbosity 
                              (beware: prints all command executions)
        --debug               Beware verbose output. Both 'verbose'
                              and 'debug' will add client-side timestamps.
        --preset-file         Use a different preset file.
        --print-preset        Prints the internal preset file.
-o      --output-file         Use a different output file name.
                              (default: "perfstat_data_yyyymmdd_hhmmss.zip").
        --sshprivatekey-file  SSH private key filename for perfstat user.
        --sample-interval     Specifies statit/sysstat frequency.
        --config-all          captures config information in every iteration.
                              (default: CONFIG info is captured only during 
the first iteration).
-t      --time                sample time in minutes, per iteration.
                              (default: 2 minutes).
-i      --iteration=N[,M]     repeat N times with M seconds between samples.
                              (default: N=1, M=0)..
-n      --nodes=N1[,N2,N3,..] node names to collect from.
                              (defaults to all nodes).
        --max-run-time        Maximum Perfstat runtime in minutes.
        --full-stutter-statit Split entire time (-t)
                              into all stutter statit collections.
        --no-stutter-statit   Disable 'stuttered' statit commands.
        --enable-flag=PATTERN Enables data collection for specified flags.
                              (default: All flags are disabled.
                              Command with !<flag-name> flag value will be executed).
        --unzip=FILE          Unzips the specified file.
        --include=PATTERN     Includes *only* commands matching 
                              PATTERN, and excludes everything else.
        --exclude=PATTERN     Excludes commands matching PATTERN,
                              and includes everything else.
-H      --hosts=H1:username:passWord[,H2:userName:passWord,H3:passWord,..]Remote host names to collect from.
-m      --mode=[7-mode/cluster-mode/cluster/c]	Specify the mode for the target filers (default: 7-mode).
-z      --FILER_ONLY           Specify the flag  to capture information from controller(s) only (default: Host and controller(s)
-s      --SSH                 How the 7-mode parallel commands should be executed, using SSH or RSH. (default: RSH)
-r      --MODULE_INST =ID1[ID2,ID3..]Capture RAS trace for the instances of pre-configured modules(default: instance-id 1)
-p      --PROFILES=[kahuna,flat,storage,(more)]Capture MP Domain profiles from filers
-l      --login               controller login user in 7-mode [ default: root ]
-a      --app_name            Name of the application for which data to be collected on host side.
-w      --app_args            List of parameters for the application data collection on host side. 
-S      --switch              Collect the data from target switches.The format should <SwitchIP>:<SwitchType>:<loginUser>:<loginPassword>
-k      --SKTRACE_POINT       Collect the sktrace for the modules. The format should be <Module>:<debug level> (deafult: 'SK:7,WAFL:4'). The debug level should be in old data ONTAP7.x fashion
-L      --log                 Collect back up log information
        --LEGACY_MODE         Collects output in Legacy Mode. This option work only for 7-mode. In Legacy mode, Systemshell and SKTrace data will not be collected.
        --diag-passwd         diag (systemshell) password. Format should be <systemShellPassword> for C-Mode. <IP1>:<systemShellPassword1>,<IP2>:<systemShellPassword2> for 7-Mode
        --wafl-bufstats       Execute the " wafl bufstats " command.
        --stats-format        Parse stats stop command output
        --mcc                 Metro Cluster support
        --SSHKey-passPhrase   SSHKey file password. Passpharse used to encrypt the private key file.
        --SSHKnownHosts-FilePathOption is vaild in-case OpenSSH flag set. Otherwise sliently Ignored. Deafult path to look into know_hosts file is sshprivatekey file path.
-R      --maxRetries          Default Retries count are 100. Each Retry take around 3 seconds. Chang in maximum retries count increase the timeout with the controller. E.g. 100 retries take around 5 minutes(3*100 seconds) timeout with the controller.
-C      --Clean-Up            Clean up output uncompressed directory from local host and perfstat user from controller.
        --vol_db              Collect the waffi_id for each volume under each node.
        --iter-wafl-cpstats   Enable collection of multiple samples of " wafl cpstats " command output over the sampling period. 
        --ip-version=[4/6]    Force use of specific IP version for node mgmt LIF discovery (useful if a node has both an ipv4 and ipv6 node mgmt LIF).

OPTION DESCRIPTION
1. --include=PATTERN/--exclude=PATTERN
The include/exclude patterns consist of semicolon separated parameter tags, which match those from the preset file.
The parameter tags themselves have comma separated values. Any definition lines in the preset file which match ANY
parameter tag and value will be included or excluded from execution. include/exclude option may not be used together.
 Here are a few examples:

 a. --exclude="SHELL=SYSTEMSHELL"
 With this option perfstat will not execute any systemshell commands.
 b. --include="TYPE=PERF"
 With this option perfstat will only execute commands with TYPE as "PERF".

2. --enable-flag=PATTERN
The enable-flag pattern consist of comma separated flag names, which match those from the preset file.
 Any definition lines in the preset file which match ANY flag value will be enabled and added for execution.
Here are a few examples:

 a. --enable-flag="snapmirror,wafl" With this option perfstat will enable "snapmirror" and "wafl" flags. Command with "snappmirror" and
 "wafl" flag will be executed. Command with "!snappmirror" and "!wafl" flag will be excluded.

3. --diag-password
With this option, user doesn't need to input the systemShell password during perfstat run.
As all the nodes in the cluster will be having same systemShell password, provide only the systemShell password of a single node in case of C-Mode
In case of 7-Mode, the node ip and its respective systemShell password should be provided. 
Note: Patterns matching will not be case sensitive. 

EXAMPLES FOR C-MODE

%perfstat8 10.10.2.10 --mode="cluster-mode" 
%perfstat8 10.10.2.10 --mode="cluster" 
%perfstat8 10.10.2.10 --mode="c" 
This command runs perfstat with 1 iterations. Here IP address "10.10.2.10" is a cluster-management IP address of the cluster. 

%perfstat8 -i 5,10 --verbose 10.10.2.10 --mode="cluster-mode" 
This command runs perfstat with 5 iterations, waiting 10 seconds between each iteration. 
Here IP address "10.10.2.10" is a cluster-management IP address of the cluster.

%perfstat8 10.10.2.10 -a "oracle" -w "runas=someDbaUser,sysdba=true" --mode="cluster-mode" 
This command runs perfstat with default preset file and also collects the host side data along with the oracle AWR report. 
Here 'someDbaUser' is a pre-existing DBA user. The another option for oracle params is '-w "oracle_login=testUser/testUser123" ' 

%perfstat8 -i 5,10 --verbose 10.10.2.10 --enable-flag=snapmirror --mode="cluster-mode" 
This command enables "snapmirror" flag in preset file and runs perfstat with 5 iterations, waiting 10 seconds between each iteration. 
Here IP address "10.10.2.10" is a cluster-management IP address of the cluster.

Example to provide systemShell password at command Line.
%perfstat8 -i 5,10 --verbose 10.10.2.10 --diag-passwd=abcxyz --mode="cluster-mode" 
This command runs perfstat with 5 iterations. Here IP address "10.10.2.10" is a cluster-management IP address of the cluster. 
And "abcxyz" is the systemShell password.

%perfstat8 --verbose --nodes=10.10.2.10,10.10.2.11,10.10.2.12 --diag-passwd=abcxyz --mode="cluster-mode" 
This command runs perfstat with 1 iteration. Here IP address "10.10.2.10", "10.10.2.11", "10.10.2.12" are the address of a nodes in the cluster. 
"abcxyz" is the systemShell password.

%perfstat8 --verbose --nodes=10.10.2.10,10.10.2.11,10.10.2.12 --client-creds=root:xyzabc --PROFILES=kahuna,flat --MODULE_INST=1 --mode="cluster-mode" 
This command runs perfstat with 1 iteration. Here IP address "10.10.2.10", "10.10.2.11", "10.10.2.12" are the address of a node-management in the cluster. 
"root" and "xyzabc" are the username and password of the local host.

EXAMPLES FOR 7-MODE

%perfstat8 --nodes=10.72.57.222 --mode="7-mode" 
%perfstat8 --nodes=10.72.57.222 
This command runs perfstat with 1 iterations. Here "10.72.57.222" is the node-management IP for a 7-mode controller. 

%perfstat8 -i 5,10 --verbose --nodes=10.72.57.222 --mode="7-mode" 
This command runs perfstat with 5 iterations, waiting 10 seconds between each iteration.  

%perfstat8 -i 5,10 --verbose --nodes=10.72.57.222 --mode="7-mode" --enable-flag=snapmirror
This command enables "snapmirror" flag in preset file and runs perfstat with 5 iterations, waiting 10 seconds between each iteration. 

%perfstat8 --verbose --nodes=10.72.57.222 --mode="7-mode" -a "oracle" -w "runas=someDbaUser,sysdba=true" 
This command runs perfstat with default preset file and also collects the host side data along with the oracle AWR report. 
Here 'someDbaUser' is a pre-existing DBA user. The another option for oracle params is '-w "oracle_login=testUser/testUser123" ' 

Example to provide systemShell password at command Line.
%perfstat8 --verbose --nodes=10.10.2.10,10.10.2.11,10.10.2.12 --mode="7-mode" --diag-passwd=10.10.2.10:abcxyz1,10.10.2.11:abcxyz2,10.10.2.12:abcxyz3 
This command runs perfstat with 1 iteration. Here IP address "10.10.2.10", "10.10.2.11", "10.10.2.12" are the address of the respective nodes. 
"abcxyz1", "abcxyz2", "abcxyz3" are the systemShell passwords of the respective nodes.

%perfstat8 --verbose --nodes=10.10.2.10 --mode="7-mode" --PROFILES=kahuna
This command runs perfstat with 1 iteration. Here IP address "10.10.2.10" is the address of the node. 
Example to show the usage of LEGACY MODE.
%perfstat8 --verbose --nodes=10.10.2.10 --mode="7-mode" --LEGACY_MODE
This command runs perfstat in Legacy Mode with 1 iteration. Here IP address "10.10.2.10", is the address of the node. 
Note: System shell account for "diag" user needs to be setup before running perfstat with system shell commands.

Note: Converged perfstat does not accept empty password
COPYRIGHT
Copyright 2009, NetApp  All Rights Reserved.


9.	Examples:
	---------
9.1.	Option : Cluster IP run

	Windows syntax : prompt% perfstat8.exe 10.10.2.10 													

	Linux syntax: prompt% perfstat8 10.10.2.10	   

	Description: 	This command runs Perfstat with 1 iteration. Here, IP address "10.10.2.10" is a node-management IP address of a node in the cluster.

9.2.	Option:		7-Mode default run
	Windows syntax :	prompt% perfstat8.exe --nodes=10.10.2.10 --mode="7-mode"	
	
	Linux syntax   :	prompt% perfstat8 --nodes=10.10.2.10 --mode="7-mode"	

	Description	   : 	This command runs Perfstat with 1 iteration. Here, IP address "10.10.2.10" is a management IP address for a 7-mode controller.	   
		

9.3.	Option:		--enable-flag
	Windows syntax :	prompt% perfstat8.exe -i 5,10 10.10.2.10 --enable-flag=snapmirror	
	
	Linux syntax   :	prompt% perfstat8 -i 5,10 10.10.2.10 --enable-flag=snapmirror

	Description	   :	This command enables "snapmirror" flag in preset file and runs Perfstat with 5 iterations, waiting 10 seconds between each iteration. Here, IP address "10.10.2.10" is a node-management IP address of a node in the cluster.


9.4.	Option:	--preset-file	
	Windows syntax :	prompt% perfstat8.exe 10.10.2.10 --preset-file user_defined_preset
	
	Linux syntax   :	prompt% perfstat8 10.10.2.10 --preset-file user_defined_preset

	Description	   : 	This command collects data based on the preset file provided. Here, IP address "10.10.2.10" is a node-management IP address of a node in the cluster and "user_defined_preset" is the user defined preset file.	


9.5.	Option:		-a and -w
	Windows syntax :	prompt% perfstat8.exe 10.10.2.10 -a "oracle" -w "runas=someDbaUser,sysdba=true"
	
	Linux syntax   :	prompt% perfstat8 10.10.2.10 -a "oracle" -w "runas=someDbaUser,sysdba=true"

	Description	   : 	This command collects Oracle AWR report. Here, IP address "10.10.2.10" is a node-management IP address of a node in the cluster and 'someDbaUser' is a pre-existing DBA user. Another option for oracle params is '-w "oracle_login=testUser/testUser123"	   


9.6.	Option:		-i (--iteration)
	Windows syntax :	prompt% perfstat8.exe 10.10.2.10 .i 5,10
	
	Linux syntax   :	prompt% perfstat8 10.10.2.10 .i 5,10

	Description	   : 	This command runs Perfstat with 5 iterations, waiting 10 seconds between each iteration. Here, IP address "10.10.2.10" is a node-management		IP address of a node in the cluster.		   

9.7.	Option:		--diag-passwd
	Windows syntax :	prompt% perfstat8.exe 10.10.2.10 .i 5,10 --diag-passwd=abcxyz
						prompt% perfstat8.exe .-nodes=10.10.2.11, 10.10.2.12 .i 5,10 --diag-passwd=abcxyz
						prompt% perfstat8.exe .-nodes=10.10.2.11, 10.10.2.12 .i 5,10 --diag-passwd=10.10.2.11:abcxyz 1, 10.10.2.12:abcxyz2
	
	Linux syntax   :	prompt% perfstat8 10.10.2.10 .i 5,10 --diag-passwd=abcxyz
						prompt% perfstat8 .-nodes=10.10.2.11, 10.10.2.12 .i 5,10 --diag-passwd=abcxyz
						prompt% perfstat8 .-nodes=10.10.2.11, 10.10.2.12 .i 5,10 --diag-passwd=10.10.2.11:abcxyz 1, 10.10.2.12:abcxyz2

	Description	   : 	This command runs Perfstat with 5 iterations, waiting 10 seconds between each iteration and with the SystemShell password mentioned in the command line. 
	
						for c-mode:
						Here, IP address "10.10.2.10" is a node-management IP address of a node in the cluster. 
						10.10.2.11 and 10.10.2.12 are the IP addresses of the nodes in the cluster. "abcxyz" is the SystemShell password of the cluster.

						for 7-Mode:
						10.10.2.11 and10.10.2.12 are the IP addresses of the nodes in the cluster. "abcxyz1", "abcxyz2" are the systemshell 
						passwords of the nodes	10.10.2.11 and 10.10.2.12.	

9.8.	Option:		--PROFILES (-g)
	Windows syntax :	prompt% perfstat8.exe 10.10.2.10 --PROFILES=kahuna
	
	Linux syntax   :	prompt% perfstat8 10.10.2.10 --PROFILES=kahuna .

	Description	   : 	This command runs Perfstat with 1 iteration and collects the profile data. Here, IP address "10.10.2.10" is a node-management IP address of a node in the cluster and "kahuna" is the profile domain.


9.9.	Option:		--MODULE_INST (-r)
	Windows syntax :	prompt% perfstat8.exe 10.10.2.10 --MODULE_INST=1
	
	Linux syntax   :	prompt% perfstat8 10.10.2.10 --MODULE_INST=1 

	Description	   : 	This command runs Perfstat with 1 iteration and collects the rastrace data. Here, IP address "10.10.2.10" is a node-management IP address of a node in the cluster and "1" is the module id.


9.10.	Option:		--SKTRACE_POINTS (-k)
	Windows syntax :	prompt% perfstat8.exe 10.10.2.10 --SKTRACE_POINTS="SK:7,WAFL:4"
	
	Linux syntax   :	prompt% perfstat8 10.10.2.10 --SKTRACE_POINTS="SK:7,WAFL:4" 

	Description	   : 	This command runs Perfstat with 1 iteration and collects the SKTRACE data. Here, IP address "10.10.2.10" is a node-management IP address of a node in the cluster. 


9.11.	Option:		--log
	Windows syntax :	prompt% perfstat8.exe 10.10.2.10 --log
	
	Linux syntax   :	prompt% perfstat8 10.10.2.10 --log

	Description	   : 	This command runs Perfstat with 1 iteration and collects the log data. Here, IP address "10.10.2.10" is a node-management IP address of a node in the cluster. 

9.12.	Option:		--hosts (-H)
	Windows syntax :	prompt% perfstat8.exe 10.10.2.10 --hosts=10.10.20.1:root:abcxyz, 10.10.20.2:root:abcpqr
	
	Linux syntax   :	prompt% perfstat8 10.10.2.10 --hosts=10.10.20.1:root:abcxyz, 10.10.20.2:root:abcpqr

	Description	   : 	This command runs Perfstat with 1 iteration and collects remote host data also. Here IP address "10.10.2.10" is a node-management IP address of a node in the cluster and 10.10.20.1 and 10.10.20.2 are the IP addresses of the remote hosts. 
	"root","abcxyz" and "abcpqr" are the username and passwords of their respective hosts.

9.13.	Option:		--switch (-S)
	Windows syntax :	prompt% perfstat8.exe 10.10.2.10 --switch=10.10.20.1:BROCADE:root:abcxyz, 10.10.20.2:CISCO:root:abcpqr
	
	Linux syntax   :	prompt% perfstat8 10.10.2.10 --switch=10.10.20.1:BROCADE:root:abcxyz, 10.10.20.2:CISCO:root:abcpqr

	Description	   : 	This command runs Perfstat with 1 iteration and collects switch data. Here, IP address "10.10.2.10" is a node-management IP address of a node in the cluster and 10.10.20.1 and 10.10.20.2 are the IP addresses of the switches. 

9.14.	Option:		--exclude
	Windows syntax :	prompt% perfstat8.exe 10.10.2.10 --exclude="TYPE=CONFIG"
	
	Linux syntax   :	prompt% perfstat8 10.10.2.10 --exclude="TYPE=CONFIG"

	Description	   :	This command runs Perfstat with 1 iteration and exclude the commands whose TYPE is "CONFIG". Here, IP address "10.10.2.10" is a node-management IP address of a node in the cluster. 

9.15.	Option:		--include
	Windows syntax :	prompt% perfstat8.exe 10.10.2.10 --include="TYPE=CONFIG"
	
	Linux syntax   :	prompt% perfstat8 10.10.2.10 --include="TYPE=CONFIG"

	Description	   : 	This command runs Perfstat with 1 iteration and includes only the commands whose TYPE is "CONFIG". Here, IP address "10.10.2.10" is a node-management IP address of a node in the cluster.
	
9.16.	Option:		--wafl-bufstats
	Windows syntax :	prompt% perfstat8.exe 10.10.2.10 --wafl-bufstats  
	
	Linux syntax   :	prompt% perfstat8 10.10.2.10 --wafl-bufstats  

	Description	   : 	This option executes wafl bufstats command. default it does not execute
	
9.17.	Option:		--stats-format
	Windows syntax :	prompt% perfstat8.exe 10.10.2.10 --stats-format
	
	Linux syntax   :	prompt% perfstat8 10.10.2.10 --stats-format

	Description	   : 	This command runs Perfstat with parsing stats stop command output.

9.18	Option:		--mcc
	Windows syntax :	prompt% perfstat8.exe 10.10.2.10 -m c --mcc
	
	Linux syntax   :	prompt% perfstat8.exe 10.10.2.10 -m c --mcc
	
	Description	   :	This command runs Perfstat v8.4. Runs Perfstat with 1 iteration and also collects data for MetroCluster. If failed to reach MetroCluster, Perfstat continues with Site-A management IP.

9.19	Option:		Multiple cluster management IPs
	Windows syntax :	prompt% perfstat8.exe 10.10.2.10,10.10.2.11 -m c
	
	Linux syntax   :	prompt% perfstat8.exe 10.10.2.10,10.10.2.11 -m c	
	
	Description    :	This command collects the Perfstat data for both the Ips[10.10.2.10,10.10.2.11] given in command line interface. Option valid in Perfstat v8.4

9.20	Option:		--vol_db
	Windows syntax :	prompt% perfstat8.exe 10.10.2.10,10.10.2.11 -m c --vol_db
	
	Linux syntax   :	prompt% perfstat8.exe 10.10.2.10,10.10.2.11 -m c --vol_db
	
	Description    :	Perfstat collects the waffi_id for each volume under each node.


10.	FAQs:
	-----

10.1	Where to get OpenSSH to generate SSH public/private key pair?

	Ans: 	Download OpenSSH windows setup from the following URL: 
         	http://www.mls-software.com/opensshd.html

10.2	 How to setup the ssh key on the 7-Mode controller?

	Ans: 	Here are the steps to generate the sshkey and exchange for Windows. 
		For Linux hosts, "ssh-keygen" command can be used to generate the public/private key pair.

		Note: Ensure that you type a real value for the password and the password field is not left blank. Using SSH instead of RSH ensures security, but requires interactive authentication unless you first perform a host key exchange. Using the free, open source implementation of OpenSSH, you can control the 
		controller from a Windows host and send non-interactive commands similar to the one you would use RSH, but without the security risks.
		Enable SSH Host Key Exchange. This document covers implementing OpenSSH as an alternative to RSH for security conscious environments.
	
	1.	Set up SecureAdmin on the controller (if you have not already done so).
	2.	Download and Install OpenSSH setup on Windows and place in Desktop or some directory in your system path.
	3.	Run ssh-keygen.exe (Click Start, Run, type cmd press return/enter key, cd to path_to\OpenSSH\bin directory, type ssh-keygen.exe and press return/enter key).
	4.	At this point, we need to save our private key file somewhere on the local host. Provide a Windows location to save the ssh keys.
		Example: C:\Users\Administrator\Desktop\OpenSSH\.ssh\id_rsa
	Note: Perfstat CLI does not support empty ssh key passphrase with OpenSSH
	5.	This will create Public key[id_rsa.pub] and Private key[id_rsa] on the given location.
	6.	Next, open your public key file with any editor. Copy and paste the text into WordPad and save the file as authorized_keys.

		This file needs to be saved to: \\<CONTROLLERNAME>\c$\etc\sshd\<USERNAME>\.ssh\authorized_keys. 
		However, the problem is, there is no such folder. If you are using CIFS to copy the file, 
		this can be problematic because CIFS does not let you create folders with a "." at the beginning of their names.

		To get around this, create a folder named <USERNAME> in the following location using Windows Explorer: \\<CONTROLLERNAME>\c$\etc\sshd\
		Example: - \\controllername\c$\etc\sshd\joeadmin
		Navigate to the newly created <USERNAME> folder using Windows Explorer and create a subfolder called ssh.

		From the controller console, run the following command:
		mv /etc/sshd/<USERNAME>/ssh /etc/sshd/[USERNAME]/.ssh
		Now you have the proper directory structure, so save the 'authorized_keys' file into this folder:
		\\<CONTROLLERNAME>\c$\etc\sshd\<USERNAME>\.ssh\authorized_keys
		We want to cache the controller's key in our registry, so run the following command from the Windows system:
		plink -ssh -l root <CONTROLLERNAME> -i c:\windows\privkey.ppk
		You will be prompted to save the key to the local registry. Answer "yes" (this is required for VFM to work with plink.exe).
		SSH access is now enabled and commands can be sent to the controller by simply preceding them with ssh.


10.3   How to setup the ssh key on the clustered Data ONTAP controller?

	Ans: 	
	
	1.	Set up SecureAdmin on the controller (if you have not already done so).
	2.	Download and install OpenSSH setup on Windows and place it on the desktop or some directory in your system path.
	3.	Run ssh-keygen.exe (Click Start, Run, type cmd press return/enter key, cd to path_to\OpenSSH\bin directory, type ssh-keygen.exe and press return/enter key).
	4.	At this point, we need to save our private key file somewhere on the local host. Provide a Windows location to save the ssh keys.
		Example: C:\Users\Administrator\Desktop\OpenSSH\.ssh\id_rsa
	5.	This will create Public key[id_rsa.pub] and Private key[id_rsa] on the given location.
	Note: Perfstat CLI does not support empty ssh key passphrase with OpenSSH
	6.	Create the user with a public key authentication method. Login in to the filer. 
		netapp::> security login create -username monitor -application ssh -authmethod publickey -profile admin
	7.	Copy the public key contents of the id_rsa.pub and place it between quotes in the security login publickey create command. (take caution not to add carriage
		returns or other data that modifies the keystring, leave it in one line)
		netapp::> security login publickey create -username monitor -index 1 -publickey "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA5s4vVbwEO1sOsq7r64V5KYBRXBDb2I5mtGmt0+3p1jjPJrXx4/	IPHFLalXAQkG7LhV5Dyc5jyQiGKVawBYwxxSZ3GqXJNv1aORZHJEuCd0zvSTBGGZ09vra5uCfxkpz8nwaTeiAT232LS2lZ6RJ4dsCz+	GAj2eidpPYMldi2z6RVoxpZ5Zq68MvNzz8b15BS9T7bvdHkC2OpXFXu2jndhgGxPHvfO2zGwgYv4wwv2nQw4tuqMp8e+	z0YP73Jg0T3jV8NYraXO951Rr5/9ZT8KPUqLEgPZxiSNkLnPC5dnmfTyswlofPGud+qmciYYr+cUZIvcFaYRG+Z6DM/HInX7w==  monitor@bjacobs-lnx"

		Alternatively, you can use the load-from-uri function to bring the public key from another source.
		netapp::> security login publickey load-from-uri -username monitor -uri http://bjacobs-lnx/id_rsa.pub
		
	8.	Verify creation.
		netapp::> security login publickey show -username monitor
		UserName: monitor Index: 1
		Public Key:
		ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA5s4vVbwEO1sOsq7r64V5KYBRXBDb2I5mtGmt0+3p1jjPJrXx4/
		IPHFLalXAQkG7LhV5Dyc5jyQiGKVawBYwxxSZ3GqXJNv1aORZHJEuCd0zvSTBGGZ09vra5uCfxkpz8nwaTeiAT232LS2lZ6RJ4dsCz+
		GAj2eidpPYMldi2z6RVoxpZ5Zq68MvNzz8b15BS9T7bvdHkC2OpXFXu2jndhgGxPHvfO2zGwgYv4wwv2nQw4tuqMp8e+
		z0YP73Jg0T3jV8NYraXO951Rr5/9ZT8KPUqLEgPZxiSNkLnPC5dnmfTyswlofPGud+qmciYYr+cUZIvcFaYRG+Z6DM/HInX7w== monitor@bjacobs-lnx
		Fingerprint:
		fd:cf:9e:06:50:4d:8c:19:5a:c6:36:0f:0f:9b:ef:bb
		Bubblebabble fingerprint:
		xunep-misif-magug-maryp-hikig-hycun-hisob-mymim-riryv-ryvam-toxox
		Comment:
		
		
10.4	Generated a public SSH key with ssh-keygen.exe but do not know what to do with the saved key?

	Ans: When the public/private key pair is generated by ssh-keygen.exe, you should put the public key on the controller. 
	To achieve the same, create the root directory under "/etc/sshd" directory and further create ".ssh" directory 
	under the newly created "root" directory. When the "/etc/sshd/root/.ssh" directory is ready, create the "authorized_keys" 
	file under this directory path by copying the content of the "public key" you just generated using ssh-keygen.exe. 
	So after it, effectively you would have a file called "/etc/sshd/root/.ssh/authorized_keys" having the public key content.
	Now, the private key from this public/private key pair should be used with the Perfstat command line.

10.5	The controller must have been added to the 'trusted known host list' on the client before running the Perfstat. 
	Where exactly do we need to add to this trusted host list?

	Ans: The meaning for the phrase, trusted known host list is the server's host "Keys" that need to be cached in the system registry. 
	This requirement is imposed because Perfstat executes openSSH in batch mode. However, you cannot respond to the request to update Window's registry. 

	So precisely, to achieve the preceding task, run ssh.exe directly from a command prompt, and answer "y" when ssh.exe asks you to update
	the registry. When the key is successfully cached, Perfstat should be able to execute ssh.exe in batch mode 
	The following example should be executed after public keys are shared on the controller.
	For example:
	%ssh.exe -l <anyUser>  <your controller name/IP> -i <ssh private key file path> "version"
	The server's host key is not cached in the registry. 
	Example: C:\Users\Administrator\Desktop\OpenSSH\bin>ssh.exe -l root 10.61.176.10 -i C:\Users\Administrator\Desktop\OpenSSH\.ssh\id_rsa "version"
	The authenticity of host '10.61.176.10 (10.61.176.10)' can't be established. RSA key fingerprint is SHA256:L9eq/IvFgWigV8j484EdemrmUuYaPOgaawfq/o+8+8Y.
	Are you sure you want to continue connecting (yes/no)? yes
	Warning: Permanently added '10.61.176.10' (RSA) to the list of known hosts.
	NetApp Release 8.2.2RC1 7-Mode: Tue Apr 29 11:13:29 PDT 2014


10.6	How can we test the whether the public and private key pair transfer has been successful or not?

	Ans: Assuming you wanted to run Perfstat without "password", so you need to modify the command line for the preceding run as follows:

	%perfstat8.exe -i 5,60 --nodes=<node-ip> --mode="7-mode" --sshprivatekey-file="priv_key.ppk"
	
	Note that the preceding Perfstat would run with "root" user. 
	If you want to run with some other user, you need to setup the key for that user and "-l" option need to be used at command line.


10.7	I am trying to run Perfstat against a 7-Mode system. When it asks for the nodeshell user, I give the SystemShell diaguser name and password because there is no node shell in 7-Mode that I know of?

	Ans: Nodeshell and Noderun shell are interchangeably used in 7-Mode as a default shell.  
	Being the SystemShell user, "diag" is not exposed to Nodeshell user directly. So, run Perfstat with any of 
	the users having administrative privileges ( could be found by the command "useradmin user list" on the controller).



10.8	What is this --SSH option for?

	Ans: In 7-Mode, the parallel commands will run either through "rsh" or "ssh". By default it is through "rsh". The prerequisite to run parallel commands using "rsh" is, the local host details should be specified at "/etc/hosts.equiv" in the CONTROLLER.	
	You can do this by using:
	wrfile -a /etc/hosts.equiv host_ip login_name
	Prerequisite to use SSH is:
	For Windows, keep the Perfstat binary under OpenSSH/bin/ folder.

10.9	What exactly is this --time or -t option?

	Ans: --time option does not mean that the time for which the Perfstat will run or the time for which an iteration will run.
	It specifies the sample time in minutes, per iteration. In other words, the time for which the parallel commands will run.

10.10	Is there a way that I can exclude or include specific commands for Perfstat run?

	Ans: Yes. Using the --exclude or --include options one can exclude or include specific commands for Perfstat run.
		For further information, see Perfstat help manual.

10.11	Why is this OpenSSH required?

	Ans:  Perfstat tool opens a SSH connection and executes the PRESTATS and POSTSTATS commands over this channel. Due to the limitation of 'only single interactive SSH session allowed' for 7-Mode, the parallel commands gets executed using "rsh.exe" on windows and "rsh binary on linux. There are few windows OS versions which does not have native RSH support for example windows2008/windows7, to make parallel commands work on those windows clients an open source ssh for windows and "ssh" binary for Linux are being used.


10.12	Why is the option "flat" used for profiling?

	Ans: Profiling is triggered by means of --PROFILES or -p at command line. 
	These options take the profile domains as the input. "flat" is the mixture of the other domains like kahuna and storage. 

10.13	How can I collect sktrace data?
	Ans:  SKTRACE can be triggered by  Perfstat by using --SKTRACE_POINTS (or) -k. 
	The format should be <Module>:<debug level> (default: 'SK:7,WAFL:4'). The debug level should be in old data ONTAP7.x manner

10.14	How can I make a customized preset file?

	Ans: The preset file is of the pattern:
	SECTION       TYPE    HARDWARE        OS      LICENSE SHELL   FLAG

	SHORTNAME   COMMAND

	For example:
	PRESTATS	PERF	NODE	*	*	NODERUN		

	options	options stats.wafltop.config volume,process,message.

	Each line in the preset file will have these parameter tags. These tags are "tab" separated. "*" means all. " " means nothing.
	
	Note: profile and rastrace commands should be present in the customized preset file, if --PROFILES or --MODULE_INST options are used.

10.15  	What and why is this -g option used for?

	Ans: There are a few Windows OS versions which do not have native RSH support for example Windows2008/Windows 7. On these machines/others,
	openssh is used to execute the commands using -g option. This -g option is available only with --LEGACY_MODE and --sshprivatekey-file. Please refer to "openssh_procedure.txt" file to know the steps to follow, so as to make "-g" option work.

10.16  How to enable RSH on Mac OS X systems ?

	Ans: The following procedure is valid for Mac OS X 10.6.x and higher.

	1)  There are files, /System/Library/LaunchDaemons/login.plist and /System/Library/LaunchDaemons/shell.plist that control the rlogind and rshd daemons in 
	 the same way as the files in /etc/xinetd.d. To enable those services, simply delete the lines,

	<key>Disabled</key>
	<true/>

	Note: In Mac OS x Lion, the *.plist files are provided as binary files, So, they can not be edited with current form in any text editor. So, they need to first be
	 converted into "XML" format {use plutil binary] and then delete those preceding lines.

       		1.a) $ sudo plutil -convert xml1 shell.plist
  	        1.b) $ sudo plutil -convert xml1 login.plist


	2) run $ sudo launchctl load /System/Library/LaunchDaemons/shell.plist
	3) run $ sudo launchctl load /System/Library/LaunchDaemons/login.plist
	4) run $ sudo launchctl list | egrep "rsh|login"
	The output you will see when loaded is similar to the following:

	$ sudo launchctl list | egrep "rsh|login"
	-     0     com.apple.rlogind
	-     0     com.apple.rshd
	$


	6) run $ sudo launchctl start com.apple.rshd
	7) run $ sudo launchctl start com.apple.rlogind
	8) run $ sudo launchctl list | egrep "rsh|rlogin"
	--if the processes are properly started. 0 means they are loaded, >0 means they are started):

	$ sudo launchctl list | egrep "rsh|login"
	608     -     com.apple.rlogind
	604     -     com.apple.rshd
	$


	9)  run sudo su .
	10) run 'echo "localhost root" >> /etc/hosts.equiv'
	11) run rsh localhost :

		$ rsh localhost
		Last login: Tue Jan 8 23:21:04 on ttys000
		bash$

	12) Now, revert the xml format files [Only if done in step-1.a/1.b] into the original binary format.

		$ sudo plutil -convert binary1 shell.plist
		$ sudo plutil -convert binary1 login.plist

10.17	How to increase maximum open files in MacOS(Error too many files open)?
	
	Ans: If maximum open file in MacOS is 256 then it fail to execute  Perftat parallel command for a big cluster, to overcome this problem we can set appropriate allowed number of open files.
	
	Check the current state:
		$ ulimit -a
		core file size          (blocks, -c) 0
		data seg size           (kbytes, -d) unlimited
		file size               (blocks, -f) unlimited
		max locked memory       (kbytes, -l) unlimited
		max memory size         (kbytes, -m) unlimited
		open files                      (-n) 256
		pipe size            (512 bytes, -p) 1
		stack size              (kbytes, -s) 8192
		cpu time               (seconds, -t) unlimited
		max user processes              (-u) 266
		virtual memory          (kbytes, -v) unlimited

	Change number of open files to 1024
		$ ulimit -n 1024

10.18	How to run Perfstat when ssh connection fails for SN 7-Mode filer?

	Ans: Run Perfstat with option --LEGACY_MODE
		For legacy mode rsh should be enabled on the filer or use OpenSSH
		
		Checking rsh option on filer:
		$options rsh
		rsh.access                   legacy
		rsh.enable                   off
		
		Enabling rsh option:
		$options rsh.enable on
		$options rsh
		rsh.access                   legacy
		rsh.enable                   on

