Issues with Cisco Networking / Ansible 2.3

So Ansible 2.3 was recently released. I'm particularly excited about this release as it introduces a single SSH connection for the playbook to the device instead of individual connections per play.

So I decided to give it a try, and straight out of the box, I have some issues. It appears that I have some connection challenges. I'm going to try to sort them out and document what was tried and the results here. Since I had an issue with moving to Ansible 2.3 initially (rolled back the update), I now have a "dev" box created to do my testing.

"Prod system"
- Centos 7
- GNS3 1.52 running on a different VM
- SSH keys setup from "ansible" to the routers

"Dev system"
- Ubuntu 16.04 LTS
- Same GNS3 target systems
- No SSH keys (so far, will likely get setup later)

Here are the results at this point in time (4/19/17)

$ ansible-playbook cisco_show_clock.yml -i inventories/lab-bulls/hosts -k
SSH password:

PLAY [Show Clock] **************************************************************

TASK [setup] *******************************************************************
ok: [mpls-pe1]
ok: [mpls-pe2]
ok: [mpls-p2]
ok: [mpls-p1]
ok: [mpls-edge]
ok: [mpls-pe3]
ok: [mpls-pe5]
ok: [mpls-pe7]
ok: [mpls-pe4]
ok: [mpls-pe6]

TASK [ios_command] *************************************************************
ok: [mpls-edge]
ok: [mpls-p1]
ok: [mpls-pe2]
ok: [mpls-p2]
ok: [mpls-pe1]
ok: [mpls-pe3]
ok: [mpls-pe4]
ok: [mpls-pe6]
ok: [mpls-pe5]
ok: [mpls-pe7]

TASK [debug] *******************************************************************
ok: [mpls-p1] => {
    "show_clock.stdout_lines[0]": [
        "*04:57:38.826 UTC Sun Apr 16 2017"
    ]
}
ok: [mpls-p2] => {
    "show_clock.stdout_lines[0]": [
        "*02:01:22.742 UTC Tue Apr 18 2017"
    ]
}
ok: [mpls-pe2] => {
    "show_clock.stdout_lines[0]": [
        "*01:09:02.111 UTC Sun Apr 16 2017"
    ]
}
ok: [mpls-pe1] => {
    "show_clock.stdout_lines[0]": [
        "*01:09:53.443 UTC Sun Apr 16 2017"
    ]
}
ok: [mpls-edge] => {
    "show_clock.stdout_lines[0]": [
        "*09:55:57.327 UTC Wed Apr 19 2017"
    ]
}
ok: [mpls-pe3] => {
    "show_clock.stdout_lines[0]": [
        "*01:11:59.303 UTC Sun Apr 16 2017"
    ]
}
ok: [mpls-pe4] => {
    "show_clock.stdout_lines[0]": [
        "*01:47:56.342 UTC Sun Apr 16 2017"
    ]
}
ok: [mpls-pe5] => {
    "show_clock.stdout_lines[0]": [
        "*01:08:07.459 UTC Sun Apr 16 2017"
    ]
}
ok: [mpls-pe6] => {
    "show_clock.stdout_lines[0]": [
        "*01:15:00.435 UTC Sun Apr 16 2017"
    ]
}
ok: [mpls-pe7] => {
    "show_clock.stdout_lines[0]": [
        "*01:10:03.495 UTC Sun Apr 16 2017"
    ]
}

PLAY RECAP *********************************************************************
mpls-edge                  : ok=3    changed=0    unreachable=0    failed=0
mpls-p1                    : ok=3    changed=0    unreachable=0    failed=0
mpls-p2                    : ok=3    changed=0    unreachable=0    failed=0
mpls-pe1                   : ok=3    changed=0    unreachable=0    failed=0
mpls-pe2                   : ok=3    changed=0    unreachable=0    failed=0
mpls-pe3                   : ok=3    changed=0    unreachable=0    failed=0
mpls-pe4                   : ok=3    changed=0    unreachable=0    failed=0
mpls-pe5                   : ok=3    changed=0    unreachable=0    failed=0
mpls-pe6                   : ok=3    changed=0    unreachable=0    failed=0
mpls-pe7                   : ok=3    changed=0    unreachable=0    failed=0

Not working from Ansible 2.3:

$ ansible-playbook cisco_show_clock.yml -i inventories/lab-bulls/hosts -k
SSH password:

PLAY [Show Clock] *****************************************************************************************************************

TASK [Gathering Facts] ************************************************************************************************************
ok: [mpls-p2]
ok: [mpls-pe2]
ok: [mpls-edge]
ok: [mpls-p1]
ok: [mpls-pe1]
ok: [mpls-pe4]
ok: [mpls-pe3]
ok: [mpls-pe5]
ok: [mpls-pe6]
ok: [mpls-pe7]

TASK [ios_command] ****************************************************************************************************************
fatal: [mpls-pe2]: FAILED! => {"changed": false, "failed": true, "msg": "unable to open shell. Please see: https://docs.ansible.com/ansible/network_debug_troubleshooting.html#unable-to-open-shell", "rc": 255}
fatal: [mpls-pe1]: FAILED! => {"changed": false, "failed": true, "msg": "unable to open shell. Please see: https://docs.ansible.com/ansible/network_debug_troubleshooting.html#unable-to-open-shell", "rc": 255}
fatal: [mpls-p2]: FAILED! => {"changed": false, "failed": true, "msg": "unable to open shell. Please see: https://docs.ansible.com/ansible/network_debug_troubleshooting.html#unable-to-open-shell", "rc": 255}
fatal: [mpls-p1]: FAILED! => {"changed": false, "failed": true, "msg": "unable to open shell. Please see: https://docs.ansible.com/ansible/network_debug_troubleshooting.html#unable-to-open-shell", "rc": 255}
fatal: [mpls-edge]: FAILED! => {"changed": false, "failed": true, "msg": "unable to open shell. Please see: https://docs.ansible.com/ansible/network_debug_troubleshooting.html#unable-to-open-shell", "rc": 255}
fatal: [mpls-pe4]: FAILED! => {"changed": false, "failed": true, "msg": "unable to open shell. Please see: https://docs.ansible.com/ansible/network_debug_troubleshooting.html#unable-to-open-shell", "rc": 255}
fatal: [mpls-pe3]: FAILED! => {"changed": false, "failed": true, "msg": "unable to open shell. Please see: https://docs.ansible.com/ansible/network_debug_troubleshooting.html#unable-to-open-shell", "rc": 255}
fatal: [mpls-pe7]: FAILED! => {"changed": false, "failed": true, "msg": "unable to open shell. Please see: https://docs.ansible.com/ansible/network_debug_troubleshooting.html#unable-to-open-shell", "rc": 255}
fatal: [mpls-pe5]: FAILED! => {"changed": false, "failed": true, "msg": "unable to open shell. Please see: https://docs.ansible.com/ansible/network_debug_troubleshooting.html#unable-to-open-shell", "rc": 255}
fatal: [mpls-pe6]: FAILED! => {"changed": false, "failed": true, "msg": "unable to open shell. Please see: https://docs.ansible.com/ansible/network_debug_troubleshooting.html#unable-to-open-shell", "rc": 255}
        to retry, use: --limit @/home/dir/ansible/cisco_show_clock.retry

PLAY RECAP ************************************************************************************************************************
mpls-edge                  : ok=1    changed=0    unreachable=0    failed=1
mpls-p1                    : ok=1    changed=0    unreachable=0    failed=1
mpls-p2                    : ok=1    changed=0    unreachable=0    failed=1
mpls-pe1                   : ok=1    changed=0    unreachable=0    failed=1
mpls-pe2                   : ok=1    changed=0    unreachable=0    failed=1
mpls-pe3                   : ok=1    changed=0    unreachable=0    failed=1
mpls-pe4                   : ok=1    changed=0    unreachable=0    failed=1
mpls-pe5                   : ok=1    changed=0    unreachable=0    failed=1
mpls-pe6                   : ok=1    changed=0    unreachable=0    failed=1
mpls-pe7                   : ok=1    changed=0    unreachable=0    failed=1

New Tool Found for NetFlow - NTOP

Today looking at my Cacti graphs I'm seeing a 5 Mbps sustained bandwidth. I'm guessing that is a stream that is going on from the PS Vue service, but I wanted to know for sure. Low and behold, my edge firewall (pfSesne) supports Netflow export. And in searching of Linux Netflow collectors I found my way to NTOP. So far I'm very impressed. More updates to come once I get flows going to it, but here is the install documentation that I used to install it on my CentOS 7 system.

https://devops.profitbricks.com/tutorials/install-ntopng-network-traffic-monitoring-tool-on-centos-7/