INE Workbook Vol 1 System Management

The command 'logging origin-id string ' will send syslog messages with the configured name as the origin. The 'service sequence-numbers' includes sequence numbers in syslog messages. This helps prevent against tampering with stored syslog information. 'logging count' allows to count all 'notification' and above system messages to provide statistics for quick analysis of the system history. Timestamps can be configured separately for logging and debugging messages (service timestamp [debug|log] [uptime|datetime] (msec year ...).

With archive logging, you can show the output that's suitable for direct application..

sh archive log config all provisioning

You can also show the differences (+/-) between two configs, and you can show the incremental, which will again output in a direct application format.


SW1#sh archive config differences flash:initial.cfg system:running-config
Contextual Config Diffs:
+archive
 +log config
  +logging enable
 +path tftp:155.1.58.100/sw1-config
 +rollback filter adaptive
 +write-memory
 +time-period 1440
interface Vlan67
 +ip address 155.1.76.7 255.255.255.0
 +shutdown
interface Vlan67
 -ip address 155.1.67.7 255.255.255.0


SW1#sh archive config incremental-diffs flash:initial.cfg
!List of Commands:
interface Vlan67
 ip address 155.1.67.7 255.255.255.0
end


Very cool. With the right configuration, you could get by without Ciscoworks/rancid in a small environment.You can even do a 'configure replace' to rollback changes.


Conditional debugging. Another feature I had no idea existed. You can set a debug condition for an interface, ip, username, mac, vlan, etc. Very cool. Be careful, undebug all does not remove the conditions. You can set most TCP client parameters with the 'ip tcp' configuration. The exceptions are 'service telnet-zeroidle' - idle outgoing telnet sessions should signal the remote host to pause output. The busy message 'busy-message # message # does just that - sets the busy message when the configured host is unavailable.


On the flip side, configure 'refuse-message # Sorry, the line is already in use #' to set the busy message for the VTY line directly. You can set the session limit for the console line; 'session-limit 1'. The vacant message will display the configured output when the line is idle. 


line con 0
vacant-message # Welcome to IOS#


To allow SNMP management stations to reload the device, you must enable the RW community string in addition to 'snmp-server system-shutdown' command. You can restrict access to certain MIB values using views. Simply create the view, and attach to the community string. Here we have created a view for the cisco subtree for the community PUBLIC.



snmp-server view ROVIEW cisco included
snmp-server community CISCO RW SNMP-HOSTS
snmp-server community PUBLIC view ROVIEW RO

To enable informs for a particular hosts, issue the following:

snmp-server host 155.1.146.100 inform version 2c CISCO




Traps can be configured globally or per-host, with some exceptions like snmp link up and down can only be enabled globally. The host specific configuration overrides the global snmp config. It is required to configure the 'snmp-server enable traps' and the 'snmp-server host' commands in order to send notifications.


You can set CPU and memory thresholds using the 'process cpu' and 'memory' IOS commands. When doing so, you may be required to also enable 'snmp-server enable traps cpu threshold'



memory reserve critical 512
memory free low-watermark processor 1000
process cpu threshold type total rising 50 interval 5

On to SNMPv3. For some reason, I thought this was difficult but in reality, it is not. You create groups ( attach read/write views) and add users to that group. You can do the same thing for trap hosts, NMS stations, by specifying the v3 username instead of the v1/v2c community. Great Article from Cisco on SNMPv3.

To send all debugging and high priority level messages to SNMP host, issue the following:


logging history size 100
logging history debugging
snmp-server enable traps syslog
snmp-server host 155.1.146.100 CISCO


Syslog first sends the logs to a history buffer, and then the SNMP agent replicates the messages as SNMP traps.

Now on to RMON. I have no major issues here except for absolute and delta sampling. Absolute is used for variables that increase or decrease over time and have an upper or lower bound when a log should be generated. Delta is used for variables that either constantly increase or constantly decrease. Absolute samplings would be interface queue depth, cpu, memory. Delta would be interface errors, input packets, output bytes. Remember that anytime you call a particular interface, you need to enable 'snmp-server ifindex persist'.


RSH/RCP on the IOS? Pretty cool! RSH does not use passwords, but instead an entry in the local .rhosts table. The table maps IP and username to local username and privilege level. So when asked for local username, typically you would just use the hostname.


R6:

ip rcmd rcp-enable
ip rcmd rsh-enable
ip rcmd remote-host R6 150.1.1.1 R1 enable
ip rcmd remote-host RCP 150.1.1.1 R1 enable


R1:

R1#rsh 150.1.6.6 /user R6 show run int s0/0


Building configuration...


Current configuration : 349 bytes
!
interface Serial0/0
 ip address 54.1.1.6 255.255.255.0
 ip hello-interval eigrp 10 5
 ip hold-time eigrp 10 15
 ip authentication mode eigrp 10 md5
 ip authentication key-chain eigrp 10 MD5_KEYS
 encapsulation frame-relay
 clock rate 2000000
 frame-relay map ip 54.1.1.254 101 broadcast
 no frame-relay inverse-arp
 frame-relay lmi-type ansi
end


R1#sh run | i ip rcmd
ip rcmd remote-username RCP
ip rcmd source-interface Loopback0


Now NTP. The part that always trips me up is NTP authentication. I finally get it. 'ntp authentication-key' simply creates the key on both masters and clients. Issue 'ntp authenticate' if you will be authenticating your time sources (ntp server, ntp broadcast/multicast, ntp peer commands). Now attach the key to the pertinent server/broadcast/multicast client statements (ntp server 1.1.1.1 key 1) and say these are trusted keys (ntp trusted-key 1). Boom! Everything should work. Verify with 'show ntp association detail'. Remember, key numbers must be the same on both master and clients.


ntp authentication-key 4 md5 062526126F615D 7
ntp authentication-key 6 md5 062526126F615F 7
ntp authentication-key 58 md5 0327723825207414 7
ntp authenticate
ntp trusted-key 4
ntp trusted-key 6
ntp clock-period 17179895
ntp source Loopback0
ntp server 150.1.4.4 key 4 prefer
ntp server 150.1.6.6 key 6


R5(config-if)#do sh ntp ass det
150.1.4.4 configured, authenticated, our_master, sane, valid, stratum 5
ref ID 127.127.7.1, time C02948F5.1BA2EA04 (00:25:25.107 UTC Fri Mar 1 2002)
...trim


150.1.6.6 configured, authenticated, insane, invalid, stratum 5
ref ID 127.127.7.1, time C0294872.9FBBF625 (00:23:14.623 UTC Fri Mar 1 2002)
...trim



Autoinstall over ethernet. The only difficult was finding out the client-id. In short, you need to enable a pool, let the client get an address, issue a 'show ip dhcp lease' to find the client ID, and then enter the client id (prepending 00 to the hex ID) in the dhcp pool. From there, you will need to run one of your routers as a DNS server, so the autoconfig host can find it's hostname as well as the hostname of the tftp server. 


Autoinstall over frame-relay was not drastically different. Instead of DHCP, the client will probe the hub router for an IP. The hub will look up the map statement for that particular DLCI and hand-out the IP address configured for that map statement. From there the client will broadcast tftp requests out all available DLCIs. The staging router should either be the tftp server, or use 'ip helper-address' pointing to the actual tftp server. 


You can also use RARP. Simply add a static arp entry to the RARP server, and set the rarp server address under the interface connected to the autoinstall client. Same tftp rules apply - the rarp server should be the tftp server itself, or use 'ip helper-address' pointing to actual tftp server.


Next came menus. The only two things I really need to remember here are 1) menu-exit is the command to exit from the menu and return to the shell; 2) username OPERTOR autocommand menu OPERATOR will automatically load the menu OPERATOR upon login for the user OPERATOR.


Banners can be confusing. Which banner performs what function? The MOTD banner will be shown to all connecting users. The login banner will be displayed before the username prompt. The exec banner is shown before the shell prompt and the incoming banner is used for reverse telnet connections. Special variables such as $(line) and $(hostname) can be used to create dynamic notifications. Also, you can disable banners per line.


line con 0
no motd-banner
no exec-banner


KRON was pretty simple except I didn't know you could do this...


sh running-config | redirect tftp:155.1.146.0/r4-conf

Ah! That explains a lot. So issue the following to configure KRON to save the running config to TFTP @ 155.1.146.100 daily at 8:00.


kron occurrence SAVE_DAILY at 8:00 recurring
 policy-list SAVE_CONFIG
kron policy-list SAVE_CONFIG
 cli show running-config | redirect tftp://155.1.146.100/r3-config

And now...EEM scripting, which I hate - simply indicated by the keyword 'scripting'. I'm no slouch when it comes to basic perl programming, grep statements, regex, etc but I am NOT a programmer. If I wanted to be a programmer, I wouldn't be worried about the damn CCIE....anyway, moving on....

The core of EEM is the EEM server that sits between the event detectors and event subscribers. There is a fixed amount of event detectors that post an event when a condition is met. They include:

1. CLI event detector - detects commands typed in CLI based on regex matching.
2. Syslog event detector - responds to syslog strings, allowing for matching on regex.
3. Interface Counter - responds to various interface counters that cross a threshold setting.
4. Counter - responds to the change of value of a generic counter.
5. SNMP - monitors SNMP objects and post an event upon the condition being met.
6. None - This is a special case. Called when a user issues 'event manager run' to execute a named EEM script/applet.
7. Watchdog - generates periodic timer events and allows the EEM script to be run at repeating intervals.

Event subscribers on the other hand, are defined and registered with the EEM server as applets or scripts. Applets are a simple program written using a very basic set of CLI commands that start with the 'action' keyword. Scripts are special TCL scripts written to hand the EEM events. Applets themselves are described as easy to write, yet powerful enough to perform many functions including CLI commands, e-mail generation, snmp/syslog message generation and implementing basic program logic. TCL is a full scripting language and requires more skills to develop. It would appear that the lab will focus on applets only - which is a good thing!

Every applet has a name and a detector condition defined to trigger said applet. The applet may access global variables, defined using the 'event manager environment' command or parameters passed to them by the detector. Every event detector has pre-defined variables. Documentation on writing EEM applets, which I will be reading later. You can list variables for a detector by using 'show event manager detector detailed' command. 

The first scenario request that you fire an action if the interface Serial0/0 rxload goes above 60%. Interface load is gauged by 1 - 255, so 60% is 153. Here is the event:

event interface name Serial0/0 parameter rxload entry-op gt entry-val 153 entry-val-is-increment false poll-interval 60

Nothing really that difficult. We are checking a interface, named Serial0/0, checking for rxload that is greater than 153 and we are not incrementing. The actions are really simple and should be self explanatory.

event manager applet INTERFACE_LOAD 

 event interface name "Serial0/0" parameter rxload entry-val 153 entry-op gt entry-val-is-increment false poll-interval 60

 action 0.0 cli command "enable"

 action 1.0 cli command "configure terminal"

 action 2.0 cli command "interface Serial0/0"

 action 3.0 cli command "ip access-group CRITICAL_TRAFFIC in"

 action 4.0 mail server "155.1.146.100" to "noc@INE.com" from "r5@INE.com" subject "Interface Alert" body "Interface Serial0/0 over 60% RX load"

Next up is syslog events. The tasks asks that if someone shutdown interface Serial0/0, to un-shut the interface and send an e-mail. At first I thought this would be a 'cli event' but I couldn't construct what I needed. Looking at the solution, you create a syslog event like so. Also notice the $_cli_result" variable used in the e-mail generation. NOW I know how to do that......

event manager applet INTERFACE_SHUTDOWN 

 event syslog pattern "Interface Serial0/0.*changed.*down"

 action 1.0 cli command "enable"

 action 2.0 cli command "conf t"

 action 3.0 cli command "interface Serial 0/0"

 action 4.0 cli command "no shutdown"

 action 5.0 cli command "end"

 action 6.0 cli command "show users"

 action 7.0 mail server "155.1.146.100" to "admin@ine.com" from "r5@ine.com" subject "Interface Shutdown Alert" body "Interface Serial 0/0 unshut, current users $_cli_result"

Looking at that now, not really that difficult. Just a little regex matching...moving on to CLI events....and the familiar but misunderstood sync option...

When the applet event is sync, the EEM server will hold the matched CLI command execution until the script terminates. The script should return an exit value in the variable $_exit_status and this will determine whether the triggered command will run (status 1) or not (status 0). Async will let the CLI command execute and the event will be posted after that. The script cannot affect the command execution. Async CLI events require a set of additional parameters such as # of occurances and the time window for occurances. You can also use the puts action which allows displaying arbitrary text on the console, provided the script is async. 

So if you need to modify the output, such as a show run, you need to set sync to yes.

event manager applet SHOW_RUN

   event cli pattern "show run" sync yes

   action 1.0 cli command "enable"

   action 2.0 cli command "show run | exclude username"

   action 3.0 puts $_cli_result

   action 4.0 set $_exit_status 0

I wish I could test this, but my IOS won't let me use the action 'puts'! Additionally, some of the sytax such as event tags is not there. 12.4(15)10T on C3725. Go figure. Anyway, to explain the above - we are triggering the event based on an entered cli command; show run. Since we set sync to yes, we can modify the output - excluding usernames. The 'puts' action displays the modified output to the user. In the end, we need to set $_exit_status to 0 so that the original command 'show run' does not run. Hope that helps someone out there.

Now periodic scheduling. Pretty easy - instead of matching an event on CLI, syslog, etc - you match on a watchdog timer value and set appropriate actions. Notice the syslog action below.

event manager applet SHOW_RUN_EVERY_5min 

 event timer watchdog time 300

 action 1.0 cli command "enable"

 action 2.0 cli command "write term"

 action 3.0 syslog msg "Configuration Saved"

 action 4.0 mail server "155.1.146.100" to "noc@ine.com" from "r5@ine.com" subject "Configuration" body " $_cli_result"

Now on to what INE calls "Advanced Features". We need to write an applet that clears interface counters every 3 minutes, excluding Serial interfaces and account for any future interfaces added to the router. I am guessing a watchdog timer and doing some SNMP ifindex stuff...

Yay. Loops and conditional constructs. A few interesting actions...

1. 'info type interface-name' pulls out the list of interface names in the system matching a regex
2. 'regexp PATTERN STRING' performs regex match against a string. If there is a match, the variable $_regexp_result is set to "1". This commands takes extra arguments for extracting the matched substrings. 
3. 'continue' - starts another interation
4. 'cli command COMMAND patter PATTERN' - allows the CLI commands that await a users response. The patter is the regex matching the command prompt.

Again, I couldn't test! I don't have a 'interface-names' info type option. Argh! I've also tried my 3640/12.4(16a) and my 2621xm/12.4(15)T13. Finally found a 2851 w/ 12.4(24)T3 that works. Are these ISR only commands?

Well, here is the config.

event manager applet CLEAR_INTERFACE

 event none

 action 1.0 info type interface-names regexp "Fa|Se"

 action 2.0 foreach _iface "$_info_interface_names"

 action 3.0  regexp ".*(Serial).*" "$_iface"

 action 4.0  if $_regexp_result eq 1

 action 5.0   continue

 action 6.0  else

 action 7.0   cli command "enable"

 action 8.0   cli command "clear counters $_iface" pattern "confirm"

 action 9.0   cli command "y"

 action 9.1  end

 action 9.2 end

To be honest, I'm not sure I could re-construct this in the lab. I'm just not a very good programmer and when you don't do things like foreach/loop logic every day, it is easy to forget. My programming days are WAY behind me. 

The info type action is pretty simple. Get the interface names, and extract those with Fa|Se. Attach the result to the _iface variable. For each result, if it's a serial interface, continue to the next one. If it is not a serial, clear the counters. I can look at that and make sense of it, but to construct this in a lab would be difficult. I think this is possibly the hardest EEM applet you could expect to see on the lab, and I would put those chances very low.

Well, that is it for system management. This section was not too bad, and I now feel I have a better understanding of things such as EEM, banners and NTP. Next is IP services, MPLS and then finally Switching. I am hoping by June 4th to be done with all of Volume 1, and then start on Volume 2 full labs while mixing in Troubleshooting labs and re-visiting certain Volume 1 topics. That will give my five solid weeks of doing full labs and troubleshooting and maybe a mock lab. Feeling pretty good on my progress.