Cloudifying IT through ITIL

•September 27, 2013 • Leave a Comment

For those who are stuck keeping their networks running, analyzing event logs, or implementing IT Gangnam style, ITIL may be a curse word, if it even exists in the vocabulary. When I first began directing the information services of an organization, frameworks seemed like bureaucracy that would delay implementation times, strangle innovation, and hinder improvement. In my defense, I came from a VAR world where I was tasked to do the heavy lifting or fighting fires, there was but a small portion of a lifecycle that ACTUALLY participated in. I look back today and see some pitfalls I should have avoided.

I think that personality drives how we handle projects and services. For the cautious and meticulous, ITIL is a welcome friend, for people who are “decisive”, and fast paced, well ITIL and I … let’s just say we didn’t get along.

Those feelings can begin to change in an organization that has experienced rapid growth and dramatic changes and are now reaching maturation.  Those types of departments can find that resources are stretched thin and as the shift from project oriented tasks moves toward maintenance tasks, IT managers begin to pull their hair out. This is where stressed IT departments and stretched work hours for the staff translate into Cloud based opportunities for offloading workloads. So where does ITIL fit in?

I have been noticing a great deal of Cloud management and sales positions asking for ITIL v3 Foundation certifications. I caught myself asking why, and I think I stumbled on the answer. In the late stages of a maturing IT organization, IT managers begin searching for ways to reign it all in. After they have brought the infrastructure up to speed, they then begin to align their IT goals with lines of business. They try to make a shift to help enhance business rather than just supporting it. After searching for ways to do that they will likely come back to a framework (at least that is how it happened for me) and ITIL is one of the more standardized and accepted frameworks for IT services delivery and lifecycle available.

At its heart, ITIL asks departments to define their services; what are you offering your users? We build little objects that define who our stakeholders are, what the service delivers, what the service consumes, how well the service works, and who is responsible for the service. By building blocks of services, we can begin to see how our services interact and (drumroll please) abstract services from the organization itself. That is why cloud companies want folks on the front lines knowledgeable in ITIL. By containerizing services we can compare, in an apples-to-apples way our services and the cloud version of those services.

KPI’s, or Key Performance Indicators are a common language or set of metrics whereby these comparisons can be made. How much does a service cost? How reliable is it? How much time does it take to maintain? All these metrics should be available to both the IT department and the cloud vendor for their own respective services. The problem is, without a framework of some sort, many organizations have absolutely no idea what the line item details are for any of their most mission critical systems, much less their ancillary support systems.

There is far more to ITIL than simply defining services, it consists of Service Strategy, Service Design, Service Transition, Service Operation, and Continual Service Improvement. It is essential for organizations to understand that business operations and IT systems meet together somewhere, and in that somewhere they must play nice. In most of today’s modern companies, every operational process is supported by an information system. In the best of today’s companies, that relationship is enhanced by both sides working in tight coordination to ensure that greater value can be delivered to the organizations customers. Cloud service providers are looking to plug themselves in, and if positioned properly, deliver value much easier to organizations which have adopted ITIL due to the common ground achieved through the vocabulary of ITIL.

Maintaining IT Relevance

•September 11, 2013 • Leave a Comment

I think it is important for IT leaders to ask themselves “Is my department relevant? Am I?” If the answer to either of those questions isn’t yes then it is time to retool and reengineer the circumstances to get the right answer. Achieving or maintaining relevancy is not simply a process of continuing education or completing projects within specifications. I find relevancy is achieved through both a tight integration of IT into the strategic vision of the organization and the personal ability of the IT leader to influence those around them to engage in making that vision real.

True, charisma alone cannot drive IT to be relevant, just as simply tagging IT onto corporate goals does not guarantee that IT is anything more than a valet for the corporate clown car. What these two elements do in concert is to put an exclamation point to the notion that IT can be operational partner and not simply a support structure in a modern enterprise.

As I have evolved my understanding of the way IT services impact and facilitate business processes, it became apparent every operational system in a modern enterprise has an Information System which keeps it moving. For that reason I have found it essential to define and engage stakeholders with IT, to write a story involving both parties one where ownership and accountability are clearly defined. As part of this understanding, I have abandoned the concept of servers and embrace the notion of services. Once we firmly define our services as an object, we are able to assess its priority, evaluate it independently for possible outsourcing, insourcing, or co-sourcing, and see how many resources are tied to its operation.

As an enterprise IT system and IT leader mature together, a critical milestone is reached once it becomes clear that every service has a lifecycle. If the full lifecycle it is not taken into consideration, the operational cost of maintenance, growth, and upgrades can stalemate technology growth and expansion. I realized that just because you can implement system after system does not mean you should. The necessity of IT Governance is a lesson too often learned this way. It is critical to relevance to strike a balance between agility and governance. We need to find ways to enhance the productivity of business units while maintaining our ability to support the portfolio of services as we grow. Engagement and evangelization of this new strategic IT relationship should occur from the top down and not only from the middle up. IT is most relevant when it is weaved into the very core of the organizational strategy, and ultimately the relevance of IT Leader who provides the thread we be tied in as well.

How to configure VMware SNMP settings

•May 29, 2013 • Leave a Comment

Alright, I had been fighting my various flavors of VMware Hosts in an effort to get them configured to poll properly with our Solarwinds Orion NPM and SAM.  We generally only had issues with one or two, but it seemed to take forever for those couple.  Despite all reason, it just didn’t want to take properly.  Here is our journey and how we did it (all explitives removed).

For ESXi 5.1

To enable SSH aka Tech Support Mode (TSM), in case you do not have it ready

From the vSphere Client
Select the host and click the Configuration tab.
Click Security Profile > (Firewall) Properties.

Click SSH Server Checkbox > OK Button to return to the Host Configuration screen.

Click Security Profile > (Services) Properties .
Click SSH > Options Button.

Click Start and stop with host Startup Policy >
Start button > OK.

To enable SNMP

From an SSH Session

From the esxcli system snmp set context, the following options are available:
Set default authentication protocol. Values: none, MD5, SHA1

Set up to ten communities each no more than 64 characters. Format is: community1[,community2,…] (this overwrites previous settings)

Start or stop SNMP service. Values: [yes|no, true|false, 0|1]

Set SNMPv3 engine id. Must be at least 5 to 32 hexadecimal characters. 0x is stripped if found as well as colons (:)

Where to source hardware events from IPMI sensors or CIM Indications. One of: indications|sensors

System Agent syslog logging level: debug|info|warning|error

Comma separated list of trap oids for traps not to be sent by agent. Use value ‘reset’ to clear setting

Set UDP port to poll snmp agent on. The default is udp/161

Set default privacy protocol. Values: none, AES128

Set up to five inform user ids. Format is: user/auth-proto/-|auth-hash/priv-proto/-|priv-hash/engine-id[,…] Where user is 32 chars max. auth-proto is none|MD5|SHA1, priv-proto is none|AES. ‘-‘ indicates no hash. engine-id is hex string ‘0x0-9a-f’ up to 32 chars max.

Return agent configuration to factory defaults

System contact string as presented in sysContact.0. Up to 255 characters

System location string as presented in sysLocation.0. Up to 255 characters.

Set up to three targets to send SNMPv1 traps to. Format is: ip-or-hostname[@port]/community[,…] The default port is udp/162. (this overwrites previous settings)

Set up to five local users. Format is: user/-|auth-hash/-|priv-hash/model[,…] Where user is 32 chars max. ‘-‘ indicates no hash. Model is one of (none|auth|priv)

Set up to three SNMPv3 notification targets. Format is: ip-or-hostname[@port]/remote-user/security-level/trap|inform[,…].

esxcli system snmp set -r
esxcli system snmp set -c anycommunity,anycommunity2
esxcli system snmp set -t
esxcli system snmp set -p 161
esxcli system snmp set -l warning
esxcli system snmp set -e yes
esxcli system snmp get

should return something like

~ # esxcli system snmp get
Communities: anycommunity, anycommunity2
Enable: true
Engineid: 00000063000000a1ac1d0c21
Hwsrc: indications
Loglevel: info
Port: 161
Targets: anycommunity

The file that is created is /etc/vmware/snmp.xml, to edit it (Not generally necessary) directly type
~ # vi /etc/vmware/snmp.xml
<targets> anycommunity</targets>

Hit the escape key then type :wq! to save the file (you should see it type on the bottom left hand corner of your SSH window.

To start the service:
You can try /etc/init.d/snmpd start but doesn’t always work, another way is…

From the vSphere Client
Select the host and click the Configuration tab.
Click Security Profile > (Firewall) Properties.

Click SNMP Server > Options Button.

Click Start and stop with host Startup Policy >
Start button > OK.

Troubleshooting SNMP
If SNMP is not working, try this (Commands in blue):
From an SSH Session

~ # esxcli network ip connection list | grep 161

udp 0 0 11701841 snmpd

~ # esxcli network firewall ruleset list | grep snmp

snmp true

~ # esxcli network firewall ruleset rule list | grep -i snmp

snmp Inbound UDP Dst 161 161

If you do not have a value returned from the esxcli network ip connection list | grep 161 command, try the following:
Make sure you are not out of space, delete the bad traps indicated by a bunch of .trp files. You may have a bad snmp target setting and the condition may lead to full disk.
Navigate to the directory by typing cd /var/spool/snmp
Delete the files by typing rm *

esxcli system snmp set -r
esxcli system snmp set -c anycommunity
esxcli system snmp set -e yes
esxcli system snmp get

For ESXi 5.0, 4.1
To enable SSH aka Tech Support Mode (TSM), in case you do not have it ready
From the vSphere Client

Select the host and click the Configuration tab.
Click Security Profile > (Services) Properties.

Click SSH > Options Button.

Click Start and stop with host Startup Policy >
Start button > OK.

Verify that the SSH service is running.

To enable SNMP
From an SSH Session

~ # vi /etc/vmware/snmp.xml

Edit the file as follows:

<targets> anycommunity</targets>

Type the escape key then :wq! to save the file.

~ # /etc/init.d/hostd restart

For ESX 4.0

To enable SSH (in case you do not have it ready)
Log in locally to the console using the “root” account
[root@vh-vmw-05 ~]# su
[root@vh-vmw-05 ~]# vi /etc/ssh/sshd_config

Change the variable for PermitRootLogin from no to yes

PermitRootLogin yes

Type the escape key then :wq! to save the file.
[root@vh-vmw-05 ~]# service sshd restart

To enable SNMP

From an SSH Session

[root@vh-vmw-05 ~]# esxcfg-firewall -e snmpd
[root@vh-vmw-05 ~]# vi /etc/vmware/snmp.xml

Edit the file as follows:
<targets> anycommunity</targets>

Type the escape key then :wq! to save the file.

[root@vh-vmw-05 ~]# service snmpd restart
Stopping snmpd: [FAILED]
Starting snmpd: [ OK ]

If you want to see if it is running type the following:
[root@vh-vmw-05 ~]# service snmpd status
snmpd (pid 13399) is running…

Sonny, I Remember Working on a Computer that Weighed like 20lbs!

•April 11, 2013 • Leave a Comment

A recent article from IDC PC Shipments Post the Steepest Decline Ever… shows that despite a hefty 7.7% expected decline, PC’s still exceeded expectations. With an actual drop of 13.9% to 14.2 million units in 1Q13 compared to the same period last year this year marks the greatest decline since IDC began tracking the metric in 1994. To put it in perspective, during 1Q03 34.5 million units were sold, a 58% drop in a decade. This calls into question the relevancy of not only the PC, but of local computing altogether.

For the past 4 years, when I began speaking about the cloud and what it can eventually bring to the party, I was amazed by how many Baby Boomers began to tell stories about programming mainframes with punch-cards. Centralized computing is old school they said, it all goes in circles they also said. I remember working on WYSE terminals connected to a VAX-VMS mainframe during college and I thought Bulletin Boards and email were the coolest things ever, but seriously? To compare Mainframes built in the 80’s with the cloud of today is like saying that the Apollo program is just a new version of the hot air balloon. Computing of today is not only mission critical, but I think it is easier to imagine a zombie apocalypse than a world without tech. There is no stuffing the genie back into the bottle.

With SaaS providers like Sales-Force, Azure, and Office365, the cloud is providing the horsepower. When all you need is a browser, all you need is a phone, tablet, or phablet. With the migration of ones pics, music, and video to the cloud, local storage plays a backseat to reliable and fast internet connections. For the masses who do actually need some local computing power, Samsung announced an 8 core processor for 2013, yea… a PHONE! What about gaming you say, well what about Xbox and Playstation? Have you been to an arcade lately, what happened to all the video games? What happened was that the power of Game consoles and the quality of the games got so good that the appeal to spend .50 (I know I am dating myself) for 5 minutes of gameplay just isn’t there. Now don’t get me wrong, there are still some uses for PC’s (for now), but the point is that as the sales of PC’s drop, the competition will get tighter until the profit just isn’t there. PC’s will eventually become a niche market.

What does that all mean? PC repair technicians should definitely have some other skills in their repertoire. KSA’s in mobility and security will be in every successful admin’s bag of tricks. CIO’s will need to look at not only SaaS options but IaaS and even PaaS (Cisco bought Meraki for 1.2 billion to play in that market). It’s not all bad, IT departments can really get away from the autonomic functions that keep the network breathing and move on to the cognitive functions that help a business achieve its objectives more efficiently and profitably, you know actually do something that others in the company will notice and maybe even thank you for.

Cisco WLC 5508 Visio Stencil

•May 29, 2010 • 3 Comments

I had a request for the Visio stencil I used to make the WLC part 1 intro diagram.  It was one I made myself from the web interface.  I refined it a bit adding some connection points and such.  Enjoy.

WLC-5508 Visio Stencil
WLC-5508 Visio Drawing

5508, GLC-T, GLC-SX-MM

WLC School for Network Admin’s Who Can Read Real Good: Part 2 (Ok, so it has been awhile)

•May 29, 2010 • Leave a Comment

As I promised, Part 2 is here, PEAP! It is the Protected Extensible Authentication Protocol and is used in our situation where machines are members of the domain and we are looking for a permanent method of wireless connectivity for users, tied to Active Directory. While there are multiple Radius type approaches, including using ACS, I like Microsoft’s IAS. If you are running an Active Directory, you have IAS’s Radius capabilities free. If you are starting from scratch, check out Part 1

First, since we have Active Directory, why don’t we use it? Create a group for your PEAP Authenticated Users, that way we can bind it to the Remote Access Policy we create later.

Step 1. Active Directory Users and Computers

Next, when you create your users, add them to the group you just created.

Step 2.

Now Click on the “Dial-in” tab and ensure “Control access through Remote Access Policy is selected.

Step 3.

Now that the user and group are set up to read the permissions set in the IAS Remote Access Policy, we need to get a spot on the network up and running for this configuration. On your Cisco switch you will need to create your Vlan and the layer 3 Interface for that Vlan.

Step 3.

interface Vlan20

description PEAP Users

ip address

no shutdown

vlan 20

name PEAP

state active

Since the interface from the switch to the WLC was already trunked from our last exercise, unless you have implemented vlan pruning or vlan allowed configurations, you should not need to change that LACP Vlan Trunk. Now it gets strange. I will be bouncing from IAS, the WLC, and the Wireless Client throughout the rest of this scenario. I do this because it will at least make sense to me, and hopefully that would imply it will make sense to others as well.

From the WLC web interface you will need to create the following: Network Interface, DHCP Scope, and Wireless SSID. We will do the Network Interface and DHCP Scope first.

Step 4. Interface Configuration

Once you are logged into the WLC, go to the “Controller” link at the top of the page, then to the “Interfaces” sidebar option. Click the “New” button in the upper right hand corner. Here you will enter the name of the interface (I would recommend against any spaces or funky symbols just in case). Take note of the “Interface Name,” you may use this later when dealing with “VSA” options in Remote Access Policy configurations. Then enter the Vlan Id you created on the switch for this network. Click “Apply” in the upper right hand corner.

Step 5.

This page actually defines the specific information regarding the Interface. Choose an IP Address on the network you have chosen. Since the Interface IP on the Switch was .1, I chose .2 (sometimes I choose .254, all depends on the situation). The Netmask of course is self-explanatory, and the Gateway is the IP Address you set as the Interface IP for the Vlan. The Primary DHCP Server IP is the IP of the Management Interface defined in the last exercise. Click “Apply” in the upper right hand corner of the page.

Step 6. DHCP Configuration

Then expand the “Internal DHCP Server” Sidebar option and click on “DHCP Scope”. Click the “New” button and enter the name for the DHCP scope that will answer requests on this PEAP network of ours.

Step 7.

You will then populate the respective information for this DHCP Scope, this screen is pretty much self-explanatory as well. Choose “Enabled” on the “Status” dropdown. Click “Apply”

Now we will update the WLC with the AAA settings from our IAS.

Step 8.

You will be clicking on the “Security” link at the top and then expand AAA -> RADIUS -> Authentication. Click on the “New” button and match the settings to those used in the “Add RADIUS Client” step in Part 1.

Now that we have all these pieces completed, we can join them together in the creation of the “WLAN”. So fittingly we will go to the “WLAN” link at the top and expand “WLANs” in the Sidebar. Click the “WLANs” option and then choose “Create New” in the drop down and click the “Go” button.

Step 9. WLAN Configuration

Here name the profile and the SSID, which is what people will see when they browse for wireless networks (unless broadcasting is disabled). Note the ID of the WLAN; this is also a VSA option. Click “Apply”.

Step 10.

On the “General” Tab, click the “Enabled” checkbox, choose the appropriate interface from the “Interface” dropdown, and then you will see the Broadcast SSID we spoke about earlier. Now check the box if you want to actually see it when you are searching for networks.

Step 11.

On the “Security” Tab -> “Layer 2” Tab, Choose the “WPA+WPA2” dropdown under Layer 2 Security, choose WPA2 Policy and WPA2 Encryption checkboxes with the 802.1x dropdown option in Auth Key Mgmt.

Step 12.

Next move over to the “AAA Servers” tab and chose the server you identified earlier as your IAS server.

Step 13.

Here we only have a couple settings to configure to round out our WLAN configuration. Here choose the “Allow AAA Override” and “DHCP Addr. Assignment” checkboxes are selected. Click “Apply” to wrap this bit up.

Now we must set up the actual policy which dictates how our user should be treated when connecting to this WLAN.

Open IAS and right-click on “Remote Access Policies” then choose “New Remote Access Policy”. This will open the “New Remote Access Policy Wizard”. On the first page choose a relevant Name for the Remote Access Policy. Remote Access Policies flow from top to bottom, like most ACL’s. So one option is to start with the most broad set of criteria and build rules down from there, similar to creating group policies in a hierarchical design. However for simplicity sake, and considering how few policies we will actually be creating, we will create all the rules and match criteria in a single policy.

Step 14. Remote Access Policy Wizard

For our purposes we choose “PEAP for Wireless Users” and the “Set up a custom policy” radio button. Click “Next”.

Step 15.

On the “Policy Conditions” step in the wizard, click the “Add” Button. Find the “Client-IP-Address” Attribute type then click the “Add…” button. You will enter the IP Address of the WLC and then move on to the next step.

Step 16.

Click the “Windows-Group” Attribute type. And choose the “Peap Wireless” (or whatever you created in the “Create Windows Group” Step.

Step 17.

Once you have the two Attribute types listed above click “Next”.

Step 18.

Choose the option to “Grant remote access permission” if the settings here match the connection settings RADIUS will receive. Click “Next”.

Step 19.

On the “Profile” page, click the “Edit Profile…” button then on the “Edit Dial-in Profile” window, click the “Advanced” tab and highlight any options you see here. Click the “Remove” button for each item until this window is empty.

Step 20.

Then click the “Add…” button. On the new “Add Attribute” window highlight the “Service-Type” Attribute and click “Add”. On the “Enumerable Attribute Information” window choose the “Login” drop down and click the “OK” button.

Step 21.

Next choose the “Vendor-Specific” Attribute and click the “Add…” button. On the “Multivalued Attribute Information” window click the “Add” button. The “Vendor-Specific Attribute Information” window will open and select the “Enter Vendor Code:” radio button and enter 14179 as the code. Choose the “Yes. It conforms.” Option and click the “Configure Attribute…” button

Step 22.

Now we are at the VSA settings I was forshadowing earlier on. VSA’s are Vendor Specific Attributes, attributes AAA servers can pass onto the client. The following table shows the VSA’s available when connecting to a WLC. In this option, I want a dynamic Vlan scenario. When a user connects to the PEAP network, I want to place the user on the “peapauth” interface, vlan 20. This may seem pretty straightforward given what we have configured so far, however if your enterprise is utilizing end-to-end Vlan’s, you may want particular users on particular vlan’s at all times. This VSA is how that can be done.

Attribute Name VSA Number Attribute Format Value
Airespace-WLAN-ID 1 Decimal WLAN ID
Airespace-QOS-Level 2 Decimal Blank-Silver, 1-Gold,
Airespace-DSCP 3 Decimal DSCP Value
Airespace-802.1p-Tag 4 Decimal 802.1p Tag
Airespace-Interface-Name 5 String Interface Name on WLC
Airespace-ACL-Name 6 String ACL Name


We will also choose Airspace-QOS-Level, I choose “Blank” which equates to 0 or “Silver”. This is useful if there is a class of individuals whose performance on the network is paramount. Albeit this option is generally set for political purposes, who ever said that IT decisions are purely “Technically” driven?

Step 23.

After adding all your VSA’s, they should appear on the window above. Click the “OK” button.

Step 24.

Back on the “Edit Dial-in Profile” window, choose the “Authentication” Tab and uncheck all boxes on this tab. Click the “EAP Methods” button and on the “Select EAP Providers” windows click the “Add…” button. On the “Add EAP” window highlight “Protected EAP (PEAP)” and click the “OK” button.

Step 25.

When you return to the “Select EAP Properties” windows, highlight “Protected EAP (PEAP)” and click the “Edit…” button. When the “Protected EAP Properties” window opens, check the “Enable Fast Reconnect” box and then click the “OK” button.

And that is it for the servers, but of course the backend would not exist were it not for the users connecting on the front end, which leads us to the configuration of the client.

PEAP configuration on the client will be focused around the basic “Wireless Zero Configuration” setup. If you are running a third party supplicant, try to match up these options as best as you can.

Step 26. Wireless Client Configuration

Get to your wireless network adapter and right-click it to get to its properties. Click on the “Wireless Networks” tab and select the appropriate wireless network. Change its settings to reflect the following configuration. On the “Association” tab choose WPA and AES for “Network Authentication:” and “Data encryption:” respectively.

Step 27.

On the “Authentication” tab make sure both checkboxes are clear then change the “EAP type:” choose “Protected EAP (PEAP)” and click the “Properties” button.

Step 28.

On the “Protected EAP Properties” window, uncheck the “Validate server certificate” checkbox and check the “Enable Fast Reconnect” checkbox. Click the “Configure…” button to show the “EAP MSCHAPv2 Properties” window and ensure the “Automatically use my Windows logon name and password (and domain if any).” is checked.

To troubleshoot any issues, begin with the IAS event viewer, I would normally run into issues revolving around mistyped VSA’s, bad windows groups, or faulty client configurations. The event viewer in IAS will normally spell out the issues if they are authentication related, and at the very least get you on the right track. This sum’s up Part 2.

Intel MFSYS25 Modular Server Visio

•May 19, 2010 • 4 Comments

So, I was inspired to better document my rack contents in the ‘ol Datacenter. I had some Dell Boxes, Cisco and HP Switches, no problems, Clariion CX4, check, Intel Blade Server… hrmm. I Googled, MSN’d, and Yahoo’d my brains out and came up negative. So I thought “I could have made my own visio stencil in the time I have spent searching (It really was a looooong time). That said I hope that while you do the same searching you will come across this (and if you are reading this, you obviously did).

These drawings are for the MFSYS25, I tried to put the connection points in, but the .vss only showed them when I had it open with the actual visio file itself, so I am included that one too. Here you go, hope you like it.

Components of Intel MFSYS25 Modular Server

Intel MFSYS25 Modular Server.vss

MFSYS Visio Elements.vsd