first off, we add an oddly slow basic test of 10k item inventory
Before:
```
Ran 229 tests in 13.214s
OK
real 0m13.403s
user 0m12.106s
sys 0m1.155s
```
After:
```
Ran 230 tests in 21.328s
OK
real 0m21.516s
user 0m20.099s
sys 0m1.275s
```
since that seems like a bit long for the test to add to runtime, lets profile
`python -m cProfile -s time ./bin/ansible all -i test/units/inventory_test_data/huge_range --list-hosts`
Before:
```
1272607 function calls (1259689 primitive calls) in 8.497 seconds
Ordered by: internal time
ncalls tottime percall cumtime percall filename:lineno(function)
10000 4.393 0.000 4.396 0.000 __init__.py:395(_get_host)
20000 2.695 0.000 2.697 0.000 __init__.py:341(__append_host_to_results)
40369 0.113 0.000 0.113 0.000 {posix.lstat}
50006 0.102 0.000 0.153 0.000 __init__.py:1490(combine_vars)
40008 0.089 0.000 0.202 0.000 __init__.py:1546(_load_vars_from_path)
20195 0.088 0.000 0.088 0.000 {posix.stat}
10011 0.087 0.000 0.087 0.000 {posix.getcwd}
```
The top two lines are promising optimization targets
- populate Inventory's host cache more in _get_host, as we are looping
over all the groups anyways.
- eliminate duplicate check of whether we've already included a host
in the construction around __append_host_to_results we can infer
presence of a host in the results list implies the presence of its
name in the hostnames set, allowing us to only to the less expensive
of the two checks
After:
```
1252610 function calls (1239692 primitive calls) in 1.320 seconds
Ordered by: internal time
ncalls tottime percall cumtime percall filename:lineno(function)
40369 0.105 0.000 0.105 0.000 {posix.lstat}
50006 0.094 0.000 0.141 0.000 __init__.py:1490(combine_vars)
40008 0.081 0.000 0.184 0.000 __init__.py:1546(_load_vars_from_path)
10011 0.080 0.000 0.080 0.000 {posix.getcwd}
20195 0.074 0.000 0.074 0.000 {posix.stat}
10002 0.069 0.000 0.261 0.000 __init__.py:1517(load_vars)
```
This function takes a string like 'foo:bar[1:2]:baz[x:y]-quux' and
returns a list of patterns ['foo', 'bar[1:2]', 'baz[x:y]-quux'], i.e.
splits the string on colons that are not part of a range specification.
get_hosts → used externally, not changed
_get_hosts → _evaluate_patterns (takes a list, evaluates ! and &)
__get_hosts → _match_one_pattern (takes one pattern only, ignores !&)
This function takes a string like 'foo:bar[1:2]:baz[x:y]-quux' and
returns a list of patterns ['foo', 'bar[1:2]', 'baz[x:y]-quux'], i.e.
splits the string on colons that are not part of a range specification.
This was used earlier to implement serial, but that's now done using
restrict_to_hosts() (whose docstring is also suitably adjusted here)
and there are no more callers.
This is a slightly different fix than we originally committed, but fixes
the problem in a less invasive way (and I believe it's generally better
that we don't deal with relative paths internally past this point)
Fixes#11789
This fixes the case in which the delegated to host may not be in the
specified hosts list, in which cases facts/vars for the host were
not available in the injected hostvars.
This also fixes the inventory variable fetching function, so that an
unknown host raises a proper error as opposed to a NoneType exception.
Fixes#8224
Avoid resolving a pattern that is a plain host. When matching a hostname in the
hosts_cache, just use the host object from there.
When running a task on say 750 hosts, this yields a huge improvement.
Split out parsing of vars files to per host and per group
parsing, instead of reparsing all groups for each host. This enhances
performance.
Extend vars_plugins' API with two new methods:
* get host variables: only parses host_vars
* get group variables: only parses group_vars for specific group
The initial run method is still used for backward compatibility.
Parse all vars_plugins at inventory initialisation, instead of
per host when touched first by runner. Here we can also loop through
all groups once easily, then parse them.
This also centralizes all parsing in the inventory constructor.
modified: bin/ansible
modified: bin/ansible-playbook
modified: lib/ansible/inventory/__init__.py
modified: lib/ansible/inventory/vars_plugins/group_vars.py
If something is executable but doesn't look like it should be, or if
something is NOT executable and DOES looks like it should, show a
more apropos error with a hint on correcting the problem
Fixes#5113
As part of 94f3b9bfab the code was changed to support dynamically adding localhost to the inventory. This change introduced an crash when run via ansible-pull
```
Starting ansible-pull at 2014-01-20 23:09:57
Traceback (most recent call last):
File "/tmp/ansible/bin/ansible", line 157, in <module>
(runner, results) = cli.run(options, args)
File "/tmp/ansible/bin/ansible", line 82, in run
hosts = inventory_manager.list_hosts(pattern)
File "/tmp/ansible/lib/ansible/inventory/__init__.py", line 372, in list_hosts
result = [ h.name for h in self.get_hosts(pattern) ]
File "/tmp/ansible/lib/ansible/inventory/__init__.py", line 136, in get_hosts
subset = self._get_hosts(self._subset)
File "/tmp/ansible/lib/ansible/inventory/__init__.py", line 177, in _get_hosts
that = self.__get_hosts(p)
File "/tmp/ansible/lib/ansible/inventory/__init__.py", line 198, in __get_hosts
hpat = self._hosts_in_unenumerated_pattern(name)
File "/tmp/ansible/lib/ansible/inventory/__init__.py", line 275, in _hosts_in_unenumerated_pattern
ungrouped.add_host(new_host)
AttributeError: 'NoneType' object has no attribute 'add_host'
```
The root cause is there is no group for the host to be added to. I fixed this case by creating the ungrouped group when it doesn't exist and then adding the host to the newly added group. This fixes the regression for me.
For link-local addresses, it is sometimes necessary to append the
interface to use for the ipv6 address. This patch extends the ipv6
regex to allow for '%ifnameX' at the end.
See https://bugzilla.redhat.com/show_bug.cgi?id=136852 for more info
testing with a ipv6 :
ansible -u misc -i '[2002::c23e]:22,' '*' -m ping
fail due to parsing of ':' as a separator of port/ip with ipv4.
This commit add support for properly parsing 2002::c23 and the
bracket notation [2002::ce]:2222
to ensure consistent behavior, hosts should look like this:
hosts: webservers:&boston:!rack42
So when applying the host selectors, run those without the "&" first,
then the &s, then the !s.
Closes#3500
A host pattern of the form '!foo' by itself does not work, but
'all:!foo' does. If the first pattern is a negation, this commit
automatically prepends 'all'.
Signed-off-by: martin f. krafft <madduck@madduck.net>
This allows patterns such as webservers:!debian:&datacenter1 to target
hosts in the webservers group, that are not in the debian group, but are
in the datacenter1 group. It also parses patterns left to right.
commit 48069adf0f47b09f675a9ba59360ca216b695ceb
Author: Gregory Duchatelet <skygreg@gmail.com>
Date: Tue Nov 27 10:13:08 2012 +0100
Removing this plugin from this branch.
commit 15400fffe643ad3e66d6b5a296fe62d36d9a617a
Author: Gregory Duchatelet <skygreg@gmail.com>
Date: Tue Nov 27 09:53:16 2012 +0100
Enhance _match function in inventory with regex.
--limit ~regex could be used to filter hosts or group with a regex.
Tested on cli and ansible-playbook.
commit 63c1b2e17ec6daa282e0a3af2249bda8f734dcd3
Author: Gregory Duchatelet <skygreg@gmail.com>
Date: Tue Nov 27 09:03:41 2012 +0100
Revert pull request #1684
commit 7c2c6fee3a856c52c1960b96ec2e7ca1112c35a1
Merge: f023a2f dd5a847
Author: Gregory Duchatelet <skygreg@gmail.com>
Date: Tue Nov 27 08:52:53 2012 +0100
Merge remote branch 'upstream/devel' into devel
commit f023a2f3df4a4c2ab2dfcc5aa42c57c02358a47a
Author: Gregory Duchatelet <skygreg@gmail.com>
Date: Mon Nov 26 20:52:27 2012 +0100
Add an inventory plugin to fetch groups and host from our CMDB.
commit c64193b4c67053e6e197b89c7143b9770cf71f23
Author: Gregory Duchatelet <skygreg@gmail.com>
Date: Mon Nov 26 20:43:30 2012 +0100
Added possibility to filter hosts from a group, with a regex, separating
groupname and regex with a ~
Usage in group pattern: group~filterpattern
Samples:
ansible group~server-0[1236] -m ping
ansible web~proxy -m ping
ansible web~(proxy|frontend) -m ping
The default is 0, which means all hosts. If set to 1, each host would run a playbook all the way through before moving
on the next host. Fact gathering is still parallel, regardless of the serial setting.
to an additional pattern (a subset) specified on the command line. For instance, a playbook could be reusable
and target "webservers" and "dbservers", but you want to test only in the stage environment, or a few boxes at a time.
./group_vars/groupname.yml (for all groups the host is in)
./host_vars/hostname.yml (for the hostname given in the inventory)
This requires an actual inventory file, not script and the paths are relative
to the directory of the inventory file.
get_hosts() was treating [] (meaning complete restriction, no hosts allowed)
the same as None (meaning no restriction, all hosts allowed). Fixed logic.
Will West
Abhijit Menon-Sen
James Cammarata
Brian Coca
Toshio Kuratomi
Marc Abramowitz
alberto
Serge van Ginderachter
Michael DeHaan
Dan Slimmon
Christian Berendt
Cristian Ciupitu
Jesse Keating
James Cammarata
James Tanner
Rob Smith
Michael DeHaan
Sébastien Bocahu
Jasper Capel
Michael Scherer
ftao
René Moser
martin f. krafft
Rike-Benjamin Schuppner
Nicolas Grilly
Timothy Appnel
Rob Parrott
Daniel Hokka Zakrisson
Brian Coca
Silviu Dicu
Gregory Duchatelet
Dag Wieers
Seth Vidal
Peter Sankauskas
Michael Lambert
John Kleint
Peter Sankauskas
Derek Carter
Brad Olson