Issues with Cisco Networking / Ansible 2.3

So Ansible 2.3 was recently released. I'm particularly excited about this release as it introduces a single SSH connection for the playbook to the device instead of individual connections per play.

So I decided to give it a try, and straight out of the box, I have some issues. It appears that I have some connection challenges. I'm going to try to sort them out and document what was tried and the results here. Since I had an issue with moving to Ansible 2.3 initially (rolled back the update), I now have a "dev" box created to do my testing.

"Prod system"
- Centos 7
- GNS3 1.52 running on a different VM
- SSH keys setup from "ansible" to the routers

"Dev system"
- Ubuntu 16.04 LTS
- Same GNS3 target systems
- No SSH keys (so far, will likely get setup later)

Here are the results at this point in time (4/19/17)

$ ansible-playbook cisco_show_clock.yml -i inventories/lab-bulls/hosts -k
SSH password:

PLAY [Show Clock] **************************************************************

TASK [setup] *******************************************************************
ok: [mpls-pe1]
ok: [mpls-pe2]
ok: [mpls-p2]
ok: [mpls-p1]
ok: [mpls-edge]
ok: [mpls-pe3]
ok: [mpls-pe5]
ok: [mpls-pe7]
ok: [mpls-pe4]
ok: [mpls-pe6]

TASK [ios_command] *************************************************************
ok: [mpls-edge]
ok: [mpls-p1]
ok: [mpls-pe2]
ok: [mpls-p2]
ok: [mpls-pe1]
ok: [mpls-pe3]
ok: [mpls-pe4]
ok: [mpls-pe6]
ok: [mpls-pe5]
ok: [mpls-pe7]

TASK [debug] *******************************************************************
ok: [mpls-p1] => {
    "show_clock.stdout_lines[0]": [
        "*04:57:38.826 UTC Sun Apr 16 2017"
    ]
}
ok: [mpls-p2] => {
    "show_clock.stdout_lines[0]": [
        "*02:01:22.742 UTC Tue Apr 18 2017"
    ]
}
ok: [mpls-pe2] => {
    "show_clock.stdout_lines[0]": [
        "*01:09:02.111 UTC Sun Apr 16 2017"
    ]
}
ok: [mpls-pe1] => {
    "show_clock.stdout_lines[0]": [
        "*01:09:53.443 UTC Sun Apr 16 2017"
    ]
}
ok: [mpls-edge] => {
    "show_clock.stdout_lines[0]": [
        "*09:55:57.327 UTC Wed Apr 19 2017"
    ]
}
ok: [mpls-pe3] => {
    "show_clock.stdout_lines[0]": [
        "*01:11:59.303 UTC Sun Apr 16 2017"
    ]
}
ok: [mpls-pe4] => {
    "show_clock.stdout_lines[0]": [
        "*01:47:56.342 UTC Sun Apr 16 2017"
    ]
}
ok: [mpls-pe5] => {
    "show_clock.stdout_lines[0]": [
        "*01:08:07.459 UTC Sun Apr 16 2017"
    ]
}
ok: [mpls-pe6] => {
    "show_clock.stdout_lines[0]": [
        "*01:15:00.435 UTC Sun Apr 16 2017"
    ]
}
ok: [mpls-pe7] => {
    "show_clock.stdout_lines[0]": [
        "*01:10:03.495 UTC Sun Apr 16 2017"
    ]
}

PLAY RECAP *********************************************************************
mpls-edge                  : ok=3    changed=0    unreachable=0    failed=0
mpls-p1                    : ok=3    changed=0    unreachable=0    failed=0
mpls-p2                    : ok=3    changed=0    unreachable=0    failed=0
mpls-pe1                   : ok=3    changed=0    unreachable=0    failed=0
mpls-pe2                   : ok=3    changed=0    unreachable=0    failed=0
mpls-pe3                   : ok=3    changed=0    unreachable=0    failed=0
mpls-pe4                   : ok=3    changed=0    unreachable=0    failed=0
mpls-pe5                   : ok=3    changed=0    unreachable=0    failed=0
mpls-pe6                   : ok=3    changed=0    unreachable=0    failed=0
mpls-pe7                   : ok=3    changed=0    unreachable=0    failed=0

Not working from Ansible 2.3:

$ ansible-playbook cisco_show_clock.yml -i inventories/lab-bulls/hosts -k
SSH password:

PLAY [Show Clock] *****************************************************************************************************************

TASK [Gathering Facts] ************************************************************************************************************
ok: [mpls-p2]
ok: [mpls-pe2]
ok: [mpls-edge]
ok: [mpls-p1]
ok: [mpls-pe1]
ok: [mpls-pe4]
ok: [mpls-pe3]
ok: [mpls-pe5]
ok: [mpls-pe6]
ok: [mpls-pe7]

TASK [ios_command] ****************************************************************************************************************
fatal: [mpls-pe2]: FAILED! => {"changed": false, "failed": true, "msg": "unable to open shell. Please see: https://docs.ansible.com/ansible/network_debug_troubleshooting.html#unable-to-open-shell", "rc": 255}
fatal: [mpls-pe1]: FAILED! => {"changed": false, "failed": true, "msg": "unable to open shell. Please see: https://docs.ansible.com/ansible/network_debug_troubleshooting.html#unable-to-open-shell", "rc": 255}
fatal: [mpls-p2]: FAILED! => {"changed": false, "failed": true, "msg": "unable to open shell. Please see: https://docs.ansible.com/ansible/network_debug_troubleshooting.html#unable-to-open-shell", "rc": 255}
fatal: [mpls-p1]: FAILED! => {"changed": false, "failed": true, "msg": "unable to open shell. Please see: https://docs.ansible.com/ansible/network_debug_troubleshooting.html#unable-to-open-shell", "rc": 255}
fatal: [mpls-edge]: FAILED! => {"changed": false, "failed": true, "msg": "unable to open shell. Please see: https://docs.ansible.com/ansible/network_debug_troubleshooting.html#unable-to-open-shell", "rc": 255}
fatal: [mpls-pe4]: FAILED! => {"changed": false, "failed": true, "msg": "unable to open shell. Please see: https://docs.ansible.com/ansible/network_debug_troubleshooting.html#unable-to-open-shell", "rc": 255}
fatal: [mpls-pe3]: FAILED! => {"changed": false, "failed": true, "msg": "unable to open shell. Please see: https://docs.ansible.com/ansible/network_debug_troubleshooting.html#unable-to-open-shell", "rc": 255}
fatal: [mpls-pe7]: FAILED! => {"changed": false, "failed": true, "msg": "unable to open shell. Please see: https://docs.ansible.com/ansible/network_debug_troubleshooting.html#unable-to-open-shell", "rc": 255}
fatal: [mpls-pe5]: FAILED! => {"changed": false, "failed": true, "msg": "unable to open shell. Please see: https://docs.ansible.com/ansible/network_debug_troubleshooting.html#unable-to-open-shell", "rc": 255}
fatal: [mpls-pe6]: FAILED! => {"changed": false, "failed": true, "msg": "unable to open shell. Please see: https://docs.ansible.com/ansible/network_debug_troubleshooting.html#unable-to-open-shell", "rc": 255}
        to retry, use: --limit @/home/dir/ansible/cisco_show_clock.retry

PLAY RECAP ************************************************************************************************************************
mpls-edge                  : ok=1    changed=0    unreachable=0    failed=1
mpls-p1                    : ok=1    changed=0    unreachable=0    failed=1
mpls-p2                    : ok=1    changed=0    unreachable=0    failed=1
mpls-pe1                   : ok=1    changed=0    unreachable=0    failed=1
mpls-pe2                   : ok=1    changed=0    unreachable=0    failed=1
mpls-pe3                   : ok=1    changed=0    unreachable=0    failed=1
mpls-pe4                   : ok=1    changed=0    unreachable=0    failed=1
mpls-pe5                   : ok=1    changed=0    unreachable=0    failed=1
mpls-pe6                   : ok=1    changed=0    unreachable=0    failed=1
mpls-pe7                   : ok=1    changed=0    unreachable=0    failed=1

New Tool Found for NetFlow - NTOP

Today looking at my Cacti graphs I'm seeing a 5 Mbps sustained bandwidth. I'm guessing that is a stream that is going on from the PS Vue service, but I wanted to know for sure. Low and behold, my edge firewall (pfSesne) supports Netflow export. And in searching of Linux Netflow collectors I found my way to NTOP. So far I'm very impressed. More updates to come once I get flows going to it, but here is the install documentation that I used to install it on my CentOS 7 system.

https://devops.profitbricks.com/tutorials/install-ntopng-network-traffic-monitoring-tool-on-centos-7/

Home Lab Update - Cacti

Today I was able to successfully install Cacti on a Centos system. This was pretty straight forward with the help of the link:
http://www.tecmint.com/install-cacti-network-monitoring-on-rhel-centos-6-3-5-8-and-fedora-17-12/

Now that I have that marked off my list I will be able to monitor my lab environment (which I will be posting soon).

New Focus - Coding, then on to Ansible

TLDR Version
CS Minor -- Network Engineer -- DevOps -- Python -- Ansible

First I thought I'd write out how I got to the point that I decided to focus on Ansible as far as networking goes. I had been doing "standard" networking for some time (10+ years). Using commercial tools that were running a large company. Then come in this new culture shift into the environment called DevOps.

Having a CS minor from college and generally understanding programming/coding I started to think a little bit more about how work was getting done, especially during crunch time. I figured out that using code that a computer would run would get consistent results. I looked into the two predominant coding languages of the day (Ruby and Python) and came to the conclusion that Python would be the right language for the networking side of things, even though there was a lot of Ruby development going on. Networking gear has Python interpreters on them, and if it is good enough to run the core of YouTube, it is still worth while.

I set out to learn about how to use Python and here are the recommendations. First for learning the syntax in general and how python works, https://www.codecademy.com/learn/python. I recommend jumping through the lessons there to learn about how to use Python. I used the community several times, so if you have to, don't be discouraged.

After getting the basics down of Python I needed to figure out how to use it to interact with network devices over SSH. To that end I did a lot of searching and found that the key Python package for SSH was Paramiko. Very shortly there after I found out about a site with some tutorials on using Python for Network devices by Kirk Byers - https://pynet.twb-tech.com/blog/. There he has some excellent examples on using Python to do various things for network devices. To this end, Kirk Byers has also written a Python module on top of Paramiko called NetMiko. These are very handy for getting some Python code running and managing networks.

At some point along the journey I had also heard of Ansible to orchestrate an IT environment. This was a natural progression from Python as it was developed in Python. Ansible is my orchestrater of choice as it supports every major system today (Linux (SSH), Windows (WinRM), Network Devices (via SSH)).

So why not just do everything in Python since it is available and not use Ansible? Ansible has many extensions that do very similar capabilities that I was looking to write in Python. I think I would have gotten results quicker using Python, but there is concern around that what happens when the one person that knows the code leaves? So that seems to be a hurdle at this point, so to use a system that keeps things a little cleaner in YAML for code, that seems like a great way to get automation moving forward.

Much of the following posts will be about my experience getting started with Ansible. There were many hurdles that I had to get over that took a lot of my time that I had available. I have gotten to a point where I can interact with network devices with Ansible, both virtual and physical devices. I will start to outline of the hurdles that I had to overcome so you won't have to. I'll also look to give props out to the good articles that I have found out there. This is all about collecting information and helping to have others start using the systems quicker and learning the systems. It will not be about giving all of the answers. There is still a learning function to be done.

It's Been a While - Glad to be back

It has been a long time since I've posted on this site. Apparently I had an opinion piece that was written in 2012 last, that I don't really believe in any more (I have new information).

I've been doing some work in the space for connecting devices (networking). I'll have some new posts coming and another introduction as well. Looking forward to some good information that other Network Engineers may be able to pick up on as well.

-Josh

Google Wallet Card - My View Point

Google Wallet is an initiative by Google to provide mobile payment services predominantly via Google's Android operating system. As I understand the system it will utilize primarily NFC to provide information from the customer device to the retailer.

On November 6th, many of the blogging websites (techcrunch.com, theverge.com) all reported a momentary information page relating to a physical card. It appears that this is to be an accessory for Android only devices. I'm not here to attempt to recreate the news. That is pointless as I have a full time career that will not allow me to compete with those. So I am going to move this to talk about my view point for this.

First, for full disclosure, I will be adding a post with my history of devices. This will give others an idea that I am a user. So right now I am on predominantly Apple user of devices. I am typing this up on a MacBook Air, have an iPhone 5, Windows Desktop (home built i5), work I utilize a Windows based Windows 7 laptop.

My ViewPoint:
When I first heard about this store on Tech News Today from the TWIT.TV podcast network I became excited. The opportunity to have a single credit card in my billfold that I could choose the actual back end credit card is a tremendous opportunity. Most of the time I would want to get most of my purchases to go to a single card. But there are times that I'd like to be able to have the card "morph" into my ATM card that I could pull cash out of the ATM with.

This would be an ideal situation to me. Some day this may become a reality with various cards having different rewards. There are plenty of retailers and other gas cards that this would be ideal for.  So when you stop at a gas station, you utilize the card as the gas stations card. After that, you are on your way to a retail shop. You simply go into the new Google Wallet app and switch to the retailer card to get the benefits out of that card.

There are obviously a lot of components that Google would need to work on to get this completed. Google may want to keep this in Android only, however, Google could learn a lot about their users shopping habits. Creating an app for iPhone would be a good step in the right direction for this information as well. This may be a few years away, but I am looking forward to having only a couple of cards in my wallet.

Discontiguous Masks

Discontiguous Masking Explained
Version 1.0 6/26/2011
 
Before digging deep into general connectivity, the first thing you should be concerned about is security. Now this isn't going to go deep into security, but I wanted to write something about one of the items that I feel is very under utilized these days. Discontiguous Masks.

Simply put, discontiguous masks are those that are represented by a potentially alternating sets of 1s and 0s within a mask (subnet/standard mask or wild card). This does not really apply to a subnet mask as you have a network portion of the address, and then the host portion, so everything is contiguous. The primary place that you will see this is in Access Control Lists or ACLs (for more info on ACLs, here is a quick link to a Wikipedia article. This may be a topic later on within this blog http://en.wikipedia.org/wiki/Standard_Access_Control_List).

The first two use cases that these come in handy for are for Quality of Service ACLs and Security/Firewall ACLs. There are many more uses out there, but these are the two primary ones that I have come across. Feel free to leave comments, and I will dive into it, modifying this post for this.

My first tip for understanding this concept is that you need to think the way that a router (or other network device) thinks. Everything that passes through a router or switch is passing through the device in a logical method. The router and switch are looking at binary 1s and 0s as they cross the wire.

Everything for this discussion revolves around binary addressing. Remember that currently in the IPv4 space that there are 32 bits to a network address. That is the IP address 192.168.0.1 can be translated into binary bits of 1100 0000 . 1010 1000 . 0000 0000 . 0000 0001.

Discontiguous masks are as they state, a mask that can be discontiguous. Many devices in the industry have support for what is a contiguous mask. This does not have do with wild card or subnet masking yet. It just matters how the devices check to see if something matches. In a standard mask or subnet mask, the mask that you define may look something like 255.255.255.0 (11111111.11111111.11111111.00000000). In this case you will be doing a logical "AND" operation where the bits that you care about are represented by a "1".  The bits that are represented by a "0" are those that you do not care about. In the Cisco world of wild card masking, it is the inverse. Most of my examples will be in wild card notation, since this is where it is most prevalent in my world.

A discontiguous mask does not follow this. First, let's take the previous example of the mask 255.255.255.0 and turn it into a wild card mask, so that we can follow along in the same fashion moving forward. This then turns into a 0.0.0.255 mask. Simply changing the important bits, or the bits that you care about from a "1" to a "0" and vice versa. Binary representation is 00000000.00000000.00000000.11111111. So now, the discontiguous part. Let's say you run a network where you have 10 sites. They are all the same IP address wise, except that you change the third octet to represent your site. Let's use addressing 192.168.0.0 - 192.168.255.255 to represent this. Site A has the IP addresses 192.168.1.0/24 (contiguous masking) and site B has the IP addresses 192.168.2.0/24. You assign a server the address of X.X.X.2 at each site. Now you want to write an ACL that will match that at each site.

The process that I would follow, is we need to figure out what is important. As we look at the 10 site  addresses (we will just use two, but we will see the commonality) that we care about.


IP address              Binary
192.168.1.2            1100 0000 . 1010 1000 . 0000 0001 . 0000 0010
192.168.2.2            1100 0000 . 1010 1000 . 0000 0010 . 0000 0010
192.168.3.2            1100 0000 . 1010 1000 . 0000 0011 . 0000 0010
192.168.4.2            1100 0000 . 1010 1000 . 0000 0100 . 0000 0010
192.168.5.2            1100 0000 . 1010 1000 . 0000 0101 . 0000 0010
192.168.6.2            1100 0000 . 1010 1000 . 0000 0110 . 0000 0010
192.168.7.2            1100 0000 . 1010 1000 . 0000 0111 . 0000 0010
192.168.8.2            1100 0000 . 1010 1000 . 0000 1000 . 0000 0010
192.168.9.2            1100 0000 . 1010 1000 . 0000 1001 . 0000 0010
192.168.10.2          1100 0000 . 1010 1000 . 0000 1010 . 0000 0010

You notice a little bit of a pattern develop. So if I were to look at a wild card mask here, let's look at each individual octet one at a time. I encourage you to do so when trying to create your own. We know that in this world that all of the addresses will start with 192.168. on all of the sites. This makes the first two octets very easy to write the wild card mask. You care about all of the bits in the first two octets. So they are simply "0". So far we have 0.0.?.? for a mask.

The third octet is where it becomes more difficult. As we take a look at the binary information above, you will notice that there isn't a complete pattern in that octet (with only 10 sites, there is a little bit of a pattern, but if this were to get expanded out to say 100 or 200 sites with each the same addressing, then it becomes more difficult. We will say for our discussion, that we don't care what the third octet is. It can be 2, it could be 254 or 140. For simplicity sake of discontiguous explanation, this is the case, we don't care about any of the bits. So now we would have the mask 0.0.255.?

The last octet is once again easy in this case. We care about every bit. So it is "0" again. Our complete wild card mask would be 0.0.255.0 in this case. This is where discontiguous masks are very powerful. Now, let's say that we add a second server at each site. The easiest thing to do from a network perspective is set the IP address to .3 at the site. These two servers do the same functionality in a high availability design, so you want all traffic to be treated as equal. You simply change the very last bit in the octet to be a 1 (in that you don't care if it is a 0 or a 1), and you have your new wild card mask of 0.0.255.1 that will match traffic on two IP addresses at a site instead of just one.

In a contiguous state, there would need to be a clear "border" between the 0s and 1s, that you can't go back and forth on. So 0000000000000001111111111111111 is a contiguous mask. The 000000000000000011111111000000000 mask would not play well with hardware that does not support it.

Remember, think about how a router thinks in order to understand clearly what is happening.

Happy masking!