Ansible with 500 routers – NetDevOps tools performance testing 9 | stream 2020/01/05

Описание: The recording of the network programmability stream which occurred on 2020/01/05 - part of 2-week stream marathon which started on 2019-12-23
During this marathon I was streaming almost daily.
The project I was working on was performance testing of different network automation tools: Ansible / Python + Nornir / Python + asyncio / Golang. For that I built a huge lab with 500 routers: https://s3.us-east-2.amazonaws.com/me.... This will be later summarized in a blog article.

On this stream I continued testing Ansible on my GNS3 lab with 500 routers. We tested:
different strategies (linear and free)
various number of forks
different tasks (gathering commands + writing to a file / doing config changes)
different inventories (50 routers, 500 routers vs 500 routers + a lot of host_vars)

Some observations:
Ansible consumes A LOT of memory and CPU resources at scale
strategy free is faster than linear, but consumes more memory
bigger inventory (not only number of nodes, but also amount of variables) leads to significantly bigger memory consumption
registering variables leads to significantly more memory consumption
increasing number of forks leads to significantly more memory and CPU consumption, also many child zombie processes if you don't have enough resources

Also, at the very end, while we were doing one of the tests, I quickly wrote an asyncio script which was mimicking the first playbook and it ran ~6-7 times faster than the best ansible results with much smaller memory footprint

Timecode:
0:00 – Introduction
1:22 – The state of the project: what have been done already and what still needs to be done
8:56 – My thoughts/experience with Ansible on the previous stream
18:00 – Testing commands gathering playbook with various Ansible options
1:18:25 – Writing a new playbook to do simple configuration changes
2:04:35 – Writing a script to generate random host_vars for existing hosts to make the inventory bigger
2:35:00 – Testing an Ansible playbook which does configuration with different settings
3:03:10 – Testing both playbooks on the big inventory (500 routers + random host_vars)
3:12:10 – While waiting for results, writing an Python asyncio script which is mimicking the first playbook
3:26:10 – Testing, troubleshooting and optimizing my Python asyncio script
3:45:00 – Back to testing Ansible on big inventory
3:59:25 – Conclusions, summary and wrap-up of the stream and the whole streaming marathon

Please note that this content is stream-first and it is slow-paced by design. I recommend increasing playback speed in the player settings.
Don't miss my upcoming streams at   / dmfigol  

The code is on my GitHub: https://github.com/dmfigol/network-pr...
Twitter:   / dmfigol  
Blog: https://dmfigol.me

Background music (royalty-free):
https://www.pretzel.rocks/