This adds a parse_address(pattern) utility function that returns
(host,port), and uses it wherever where we accept IPv4 and IPv6
addresses and hostnames (or host patterns): the inventory parser
the the add_host action plugin.
It also introduces a more extensive set of unit tests that supersedes
the old add_host unit tests (which didn't actually test add_host, but
only the parsing function).
Required some rewiring in inventory code to make sure we're using
the DataLoader class for some data file operations, which makes mocking
them much easier.
Also identified two corner cases not currently handled by the code, related
to inventory variable sources and which one "wins". Also noticed we weren't
properly merging variables from multiple group/host_var file locations
(inventory directory vs. playbook directory locations) so fixed as well.
Replace .iteritems() with six.iteritems() everywhere except in
module_utils (because there's no 'six' on the remote host). And except
in lib/ansible/galaxy/data/metadata_template.j2, because I'm not sure
six is available there.
This commit deprecates the earlier groupname[x-y] syntax in favour of
the inclusive groupname[x:y] syntax. It also makes the subscripting
code simpler and adds explanatory comments.
One problem addressed by the cleanup is that _enumeration_info used to
be called twice, and its results discarded the first time because of the
convoluted control flow.
The possibilities are complicated enough that I didn't want to make
changes without having a complete description of what it actually
accepts/matches. Note that this text documents current behaviour, not
necessarily the behaviour we want. Some of this is undocumented and may
not be intended.
Now we accept IPv6 addresses _with port numbers_ only in the standard
[xxx]:NN notation (though bare IPv6 addresses may be given, as before,
and non-IPv6 addresses may also be placed in square brackets), and any
other host identifiers (IPv4/hostname/host pattern) as before, with an
optional :NN suffix.
The new code parses INI-format inventory files in a single pass using a
well-documented state machine that reports precise errors and eliminates
the duplications and inconsistencies and outright errors in the earlier
three-phase parsing code (e.g. three ways to skip comments). It is also
much easier now to follow what decisions are being taken on the basis of
the parsed data. The comments point out various potential improvements,
particularly in the area of consistent IPv6 handling.
On the ornate marble tombstone of the old code, the following
inscription is one last baffling memento from a bygone age:
- def _before_comment(self, msg):
- ''' what's the part of a string before a comment? '''
- msg = msg.replace("\#","**NOT_A_COMMENT**")
- msg = msg.split("#")[0]
- msg = msg.replace("**NOT_A_COMMENT**","#")
- return msg
first off, we add an oddly slow basic test of 10k item inventory
Before:
```
Ran 229 tests in 13.214s
OK
real 0m13.403s
user 0m12.106s
sys 0m1.155s
```
After:
```
Ran 230 tests in 21.328s
OK
real 0m21.516s
user 0m20.099s
sys 0m1.275s
```
since that seems like a bit long for the test to add to runtime, lets profile
`python -m cProfile -s time ./bin/ansible all -i test/units/inventory_test_data/huge_range --list-hosts`
Before:
```
1272607 function calls (1259689 primitive calls) in 8.497 seconds
Ordered by: internal time
ncalls tottime percall cumtime percall filename:lineno(function)
10000 4.393 0.000 4.396 0.000 __init__.py:395(_get_host)
20000 2.695 0.000 2.697 0.000 __init__.py:341(__append_host_to_results)
40369 0.113 0.000 0.113 0.000 {posix.lstat}
50006 0.102 0.000 0.153 0.000 __init__.py:1490(combine_vars)
40008 0.089 0.000 0.202 0.000 __init__.py:1546(_load_vars_from_path)
20195 0.088 0.000 0.088 0.000 {posix.stat}
10011 0.087 0.000 0.087 0.000 {posix.getcwd}
```
The top two lines are promising optimization targets
- populate Inventory's host cache more in _get_host, as we are looping
over all the groups anyways.
- eliminate duplicate check of whether we've already included a host
in the construction around __append_host_to_results we can infer
presence of a host in the results list implies the presence of its
name in the hostnames set, allowing us to only to the less expensive
of the two checks
After:
```
1252610 function calls (1239692 primitive calls) in 1.320 seconds
Ordered by: internal time
ncalls tottime percall cumtime percall filename:lineno(function)
40369 0.105 0.000 0.105 0.000 {posix.lstat}
50006 0.094 0.000 0.141 0.000 __init__.py:1490(combine_vars)
40008 0.081 0.000 0.184 0.000 __init__.py:1546(_load_vars_from_path)
10011 0.080 0.000 0.080 0.000 {posix.getcwd}
20195 0.074 0.000 0.074 0.000 {posix.stat}
10002 0.069 0.000 0.261 0.000 __init__.py:1517(load_vars)
```
This function takes a string like 'foo:bar[1:2]:baz[x:y]-quux' and
returns a list of patterns ['foo', 'bar[1:2]', 'baz[x:y]-quux'], i.e.
splits the string on colons that are not part of a range specification.
get_hosts → used externally, not changed
_get_hosts → _evaluate_patterns (takes a list, evaluates ! and &)
__get_hosts → _match_one_pattern (takes one pattern only, ignores !&)
This function takes a string like 'foo:bar[1:2]:baz[x:y]-quux' and
returns a list of patterns ['foo', 'bar[1:2]', 'baz[x:y]-quux'], i.e.
splits the string on colons that are not part of a range specification.
This was used earlier to implement serial, but that's now done using
restrict_to_hosts() (whose docstring is also suitably adjusted here)
and there are no more callers.
This is a slightly different fix than we originally committed, but fixes
the problem in a less invasive way (and I believe it's generally better
that we don't deal with relative paths internally past this point)
Fixes#11789
* Moving connection creation until after the task is post_validated,
to make sure all fields are properly templated (#11230)
* Fixing problems related to the connection method and remote address
lookup on the delegated-to host
Fixes#11230
This fixes the case in which the delegated to host may not be in the
specified hosts list, in which cases facts/vars for the host were
not available in the injected hostvars.
This also fixes the inventory variable fetching function, so that an
unknown host raises a proper error as opposed to a NoneType exception.
Fixes#8224
and without hosts and vars
Without this patch, the simplified syntax is triggered when a group
is defined like this:
"platforms": {
"children": [
"cloudstack"
]
}
Which results in a group 'platforms' with 1 host 'platforms'.
modified: lib/ansible/inventory/script.py
Avoid resolving a pattern that is a plain host. When matching a hostname in the
hosts_cache, just use the host object from there.
When running a task on say 750 hosts, this yields a huge improvement.
In some cases, where a host is mentioned in multiple groups, and those
groups are referenced in multiple ini files, a group could still contain
multiple instances of a group in its host,groups list, where only one of them
is the right group, that exists in the inventory.
Split out parsing of vars files to per host and per group
parsing, instead of reparsing all groups for each host. This enhances
performance.
Extend vars_plugins' API with two new methods:
* get host variables: only parses host_vars
* get group variables: only parses group_vars for specific group
The initial run method is still used for backward compatibility.
Parse all vars_plugins at inventory initialisation, instead of
per host when touched first by runner. Here we can also loop through
all groups once easily, then parse them.
This also centralizes all parsing in the inventory constructor.
modified: bin/ansible
modified: bin/ansible-playbook
modified: lib/ansible/inventory/__init__.py
modified: lib/ansible/inventory/vars_plugins/group_vars.py
If something is executable but doesn't look like it should be, or if
something is NOT executable and DOES looks like it should, show a
more apropos error with a hint on correcting the problem
Fixes#5113
Make sure all hosts and groups are unique objects
and that those are referenced uniquely everywhere.
Also fixes test_dir_inventory unit tests which were broken after previous
patches.
modified: lib/ansible/inventory/dir.py
Fixes a bug where vault_password parameter was not passed through in
_load_vars_from_folder()
modified: lib/ansible/inventory/vars_plugins/group_vars.py
0. Uncomment the test.
1. Test fails.
2. Make vars unique per file in test inventory files.
3. Modify token addition to not ast.literal_eval(v) a variable containing a hash.
4. Modify vars to have an escape in test inventory file.
5. Catch exceptions explicitly. Any unknown exceptions should be a bug.
6. Test passes.
It came up that fixing this unit test may relate to another ticket that is open. This work allows us to uncomment this unit test by fixing how we pars variables allowing a quoted variable to contain a '#'.
Work also went into cleaning up some of the test data to clarify what was working.
Lastly work went into cleaning up formatting so that the code is easily read.
As part of 94f3b9bfab the code was changed to support dynamically adding localhost to the inventory. This change introduced an crash when run via ansible-pull
```
Starting ansible-pull at 2014-01-20 23:09:57
Traceback (most recent call last):
File "/tmp/ansible/bin/ansible", line 157, in <module>
(runner, results) = cli.run(options, args)
File "/tmp/ansible/bin/ansible", line 82, in run
hosts = inventory_manager.list_hosts(pattern)
File "/tmp/ansible/lib/ansible/inventory/__init__.py", line 372, in list_hosts
result = [ h.name for h in self.get_hosts(pattern) ]
File "/tmp/ansible/lib/ansible/inventory/__init__.py", line 136, in get_hosts
subset = self._get_hosts(self._subset)
File "/tmp/ansible/lib/ansible/inventory/__init__.py", line 177, in _get_hosts
that = self.__get_hosts(p)
File "/tmp/ansible/lib/ansible/inventory/__init__.py", line 198, in __get_hosts
hpat = self._hosts_in_unenumerated_pattern(name)
File "/tmp/ansible/lib/ansible/inventory/__init__.py", line 275, in _hosts_in_unenumerated_pattern
ungrouped.add_host(new_host)
AttributeError: 'NoneType' object has no attribute 'add_host'
```
The root cause is there is no group for the host to be added to. I fixed this case by creating the ungrouped group when it doesn't exist and then adding the host to the newly added group. This fixes the regression for me.
So instead of having:
group_vars/production.yml
A user could chose to reorganize to:
group_vars/production/staff.yml
group_vars/production/networks.yml
group_vars/production/dns.yml
(Backwards compatible.)
- Move all the supported YAML file extensions into a constant
- Use helper functions to avoid duplicate code for group/host vars
- Catch and disallow some confusing situations, such as the presence of
multiple group/host vars files for the same group/host, but with
different extensions. For example having both group_vars/all.yml and
group_vars/all.yaml.
- Catch and report file system permission issues, symlink errors,
unexpected file system objects
- Trivial performance improvement from making fewer stat system calls
- Restructuring that makes it easy for a following patch to support
directory recursion
For link-local addresses, it is sometimes necessary to append the
interface to use for the ipv6 address. This patch extends the ipv6
regex to allow for '%ifnameX' at the end.
See https://bugzilla.redhat.com/show_bug.cgi?id=136852 for more info
testing with a ipv6 :
ansible -u misc -i '[2002::c23e]:22,' '*' -m ping
fail due to parsing of ':' as a separator of port/ip with ipv4.
This commit add support for properly parsing 2002::c23 and the
bracket notation [2002::ce]:2222
to ensure consistent behavior, hosts should look like this:
hosts: webservers:&boston:!rack42
So when applying the host selectors, run those without the "&" first,
then the &s, then the !s.
Closes#3500
and the _meta hash contains a "hostvars", don't call --host hostname for any elements
and just serve them directly for performance enhancements with the external inventory
script and a large number of hosts.
Since ansible 1.2, it became possible to place a host_vars
directory in the same directory as a playbook, making it possible
to keep host_vars local to that playbook there. However, due to
python's os.path.dirname, a action such as:
$ ansible-playbook pb.yml
..would not pick up the host_vars as os.path.dirname("pb.yml")
returns "", unlike the unix command dirname that would return
".". Substituting "pb.yml" on the command line with "./pb.yml"
would do the trick, but is not always intuitive. This patch
solves the problem until python solves issue18547 [1].
[1] http://bugs.python.org/issue18547
A host pattern of the form '!foo' by itself does not work, but
'all:!foo' does. If the first pattern is a negation, this commit
automatically prepends 'all'.
Signed-off-by: martin f. krafft <madduck@madduck.net>
e.g. db[01:10:3]node-[01:10]
- to do this we split off at the first [...] set, getting the list
of hosts and then repeat until none left.
- also add an optional third parameter which contains the step. (Default: 1)
so range can be [01:10:2] -> 01 03 05 07 09
If someone has a " #" in a quoted var string, it
will interpret that as a comment and refuse to
load the inventory file due to an unbalanced
quote. Noisy failure > unexpected behavior.
Look for a file with the base name of the group/host, first without
a file extension, then with a '.yml' extension, and, finally, with
a '.yaml' extension, loading vars from only the first one found.
Victor Salgado
James Cammarata
Thomas Quinot
Abhijit Menon-Sen
Toshio Kuratomi
Marius Gedminas
Brian Coca
Ilya Epifanov
Harald Laabs
Will West
Brian Coca
Dann Bohn
Gerard Lynch
Benno Joy
Marc Abramowitz
alberto
Alvaro Lopez Ortega
Michael DeHaan
Serge van Ginderachter
Michael DeHaan
Dan Slimmon
Kevin Bell
Christian Berendt
Cristian Ciupitu
Oleg Bulatov
Richard C Isaacson
Michael DeHaan
Jesse Keating
James Cammarata
James Tanner
Ilya Kuznetsov
Rob Smith
jay
John Barker
Sébastien Bocahu
Stoned Elipot
Alan Fairless
Martynas Mickevicius
Brian Coca
Jasper Capel
Michael Scherer
Matt Saunders
ftao
René Moser
rishid
Jonas Eriksson
martin f. krafft
Rike-Benjamin Schuppner
Simon J Mudd
Chris Jacobs
Darryl Stoflet
Pol Llovet
Nicolas Grilly
Timothy Appnel
C. Morgan Hamill
George Miroshnykov
Blair Zajac
Seth Vidal
Rob Parrott
Daniel Hokka Zakrisson
Nigel Metheringham