Facts Refresh (2.4 roadmap)
This commit implements most of the 2.4 roadmap 'Facts Refresh'
- move facts.py to facts/__init__.py
- move facts Distribution() to its own class
- add a facts/utils.py
- move get_file_content and get_uname_version to facts/utils.py
- move Facts() class from facts/__init__ to facts/facts.py
- mv get_file_lines to facts/utils.py
- mv Ohai()/Facter() class to facts/ohai.py and facter.py
- Start moving fact Hardware() classes to facts/hardware/*.py
- mv HPUX() hardware class to facts/hardware/hpux.py
- move SunOSHardware() fact class to facts/hardware/sunos.py
- move OpenBSDHardware() class to facts/hardware/openbsd.py
- mv FreeBsdHardware() and DragonFlyHardware() to facts/hardware/
- mv NetBSDHardware() to facts/hardware/netbsd.py
- mv Darwin() hardware class to facts/hardware/darwin.py
- pep8/etc cleanups on facts/hardware/*.py
- Mv network facts classes to facts/network/*.py
- mv Virtual fact classes to facts/virtual
- mv Hardware.get_sysctl to facts/sysctl.py:get_sysctl
- Also mv get_uname_version from facts/utils.py -> distribution.py
since distribution.py is the only thing using it.
- add collector.py with new BaseFactCollector
- add a subclass for AnsibleFactCollector
- hook up dict key munging FactNamespaces
- add some test cases for testing the names of facts
- mv timeout stuff to facts.timeout
- rm ansible_facts()/get_all_facts() etc
- Instead of calling facts.ansible_facts(), fact collection
api used by setup.py is now to create an AnsibleFactCollector()
and call it's collect method.
- replace Facts.get_user_facts with UserFactCollector
- add a 'systems' facts package, mv UserFactCollector there
- mv get_dns_facts to DnsFactCollector
- mv get_env_facts to EnvFactCollector
- include the timeout length in exception message
- modules and module_utils that use AnsibleFactCollector
can now theoretically set the 'valid_subsets'
May be useful for network facts module that currently have
to reimplement a good chunk of facts.py to get gather_subsets
to work.
- get_local_facts -> system/LocalFactCollector
- get_date_time -> system/date_time.py
- get_fips_facts -> system/fips.py
- get_caps_facts() -> system/caps.py
- get_apparmor_facts -> system/apparmor.py
- get_selinux_facts -> system/selinux.py
- get_lsb_facts -> system/lsb.py
- get_service_mgr_facts -> system/service_mgr.py
- Facts.is_systemd_managed -> system/service_mgr.py
- get_pkg_mgr_facts -> system/pkg_mgr.py
- Facts()._get_mount_size_facts() -> facts.utils.get_mount_size()
- add unit test for EnvFactCollector
- add a test case for minimal gather_subsets
- add test case for collect_ids
- Make gather_subset match existing behavior or '!all'
If 'gather_subset' is provided as '!all', the existing behavior
(in 2.2/2.3) is that means 'dont collect any facts except those
from the Facts() class'. So 'skip everything except
'apparmor', 'caps', 'date_time', 'env', 'fips', 'local', 'lsb',
'pkg_mgr', 'python', 'selinux', 'service_mgr', 'user', 'platform', etc.
The new facts setup was making '!all' mean no facts at all, since
it can add/exclude at a finer granularity. Since that makes more
sense for the ansible collector, and the set of minimal facts to
collect is really more up to setup.py to decide we do just that.
So if setup.py needs to always collect some gather_subset, even
on !all, setup.py needs to have the that subset added to the
list it passes as minimal_gather_subset.
This should fix some intg tests that assume '!all' means that
some facts are still collected (user info and env for example).
If we want to make setup.py collect a more minimal set, we can do that.
- force facts_dicts.keys() to a list so py3 works
- split fact collector tests to test_collectors.py
- convert Facter(Facts) -> other/facter.py:FacterFactCollector
- add FactCollector.collect_with_namespace()
regular .collect() will return a dict with the key names
using the base names ('ip_address', 'service_mgr' etc)
.collect_with_namespace() will return a dict where the key names
have been transformed with the collectors namespace, if there is
one. For most, this means a namespace that adds 'ansible_' to the
start of the key name.
For 'FacterFactCollector', the namespace transforms the key to
'facter_*'.
- add test cases for collect_with_namespace
- move all the concrete 'which facts does setup.py' stuff to setup.py
The caller of AnsibleFactCollector.from_gather_subset() needs to
pass in the list of collector classes now.
- update system/setup.py to import all of the fact classes and pass
in that list.
- split the Distribution fact class up a bit
extracted the 'distro release' file handling (ie, linux
boxes with /etc/release, /etc/os-release etc) into its
own class.
- extract get_cmdline_facts -> cmdline.py
- extract get_public_ssh_host_keys -> system/ssh_pub_keys.py
- extract get_platform_facts -> system/platform.py
platform.py may be a good candidate for further splitting.
- rm test for plain Facts() base class
- let the base class for Collector unit tests provide collected_facts
some Collectors and/or their migrated Facts() subsclasses need
to look at facts collected by other modules ('ansible_architecture'
the main one...).
Collector.collect() has the collected_facts arg for this, so add
a class variable to BaseFactsTest so we can specify it.
- mv Ohai to other/ohai.py and convert to Collector
- update hardware/*.py to return facts (no side effects)
- mv AnsibleFactCollector to setup.py
- extra collector class gathering to module method in
facts/__init__.py (collector_classes_from_gather_subset)
- add a CollectorMetaDataCollector collector used to provide
the 'gather_setup' fact
- add unit test module for 'setup' module
(test/units/modules/system/setup.py)
- Collector init now doesnt need a module, but collect does
An instance of a FactCollector() isnt tied to a AnsibleModule
instance, but the collect() method can be, so optionally pass
in module to FactCollector.collect() (everywhere)
- add a default_collectors for list of default collectors
import and use it from setup.py module
eventually, would like to replace this with a plugin loader
style class finder/loader
- unit tests for module_utils/facts/__init__.py
- add unit tests for ohai facts collector
- remove self.facts side effect on populate() in hardware/sunos.py
- convert OpenBSDHardware() to rm side effects on self.facts
- try to rm some self.facts side effects in Network()
plumb in collected_facts from populate() where it is needed.
stop passing collected_facts into Network() [via cached_facts=,
where it eventually becomes self.facts]
- nothing provides Fact() cached_facts arg now, rm it
Facts() should be internal only implementation so nothing
should be using it.
Of course, now someone will.
- add a Collector.name attr to build a map of name->_fact_ids
To properly exclude a gather_subset spec like '!hardware', we
need to know that 'hardware' also means 'devices', 'dmi', etc.
Before, '!hardware' would remove the 'hardware' collector name
but not 'devices'. Since both would end up in id_collector_map,
we would still end up with the HardwareCollector in the collector
list. End result being that '!hardware' wouldn't stop hardware
from being collected.
So we need to be able to build that map, so add the Collector.name
attribute that is the primary name (like 'hardware') and let
Collector._fact_ids be the other fact ids that a collector is
responsible for.
Construct the aliases_map of Collector.name -> set of _fact_ids
in fact/__init__.py get_collector_names, and use it when we are
populating the exclude set.
- refactor of distribution.py
make the big OS_FAMILY literal a little easier to read
Also keys can now be any string instead of python literals
99% sure the test for 'KDE Neon' was wrong
I don't see how/where it should or could get 'Neon' instead
of 'KDE Neon' as provided in os-release NAME=
Use 'distribution' string for key to OS_MAP
ie, we dont need to make it a valid python label anymore so dont.
move _has_dist_file to module as _file_exists
easier to mock without mucking with os.path
mv platform.system() calls to within get_distribution_facts() instead
of Distribution() init.
- remove _json compat module
The code in here was to support:
-a 'json' python module that was not the standard one included
with python since 2.6.
- potentially fallback to simplejson if 'json' was not available.
'json' is available for all supported python versions now so
no longer needed.
- mv get_collector_names -> facts.collector
- mv collector_classes_from_gather_subset -> facts.collector
- mv collector tests from test_facts -> test_collector
- Use six's reduce() in sunos/netbsd hardware facts
- rm extraneous get_uname_version in utils
only system/distribution.py uses it
- Remove Facts() subclass metaclass usage
- using fact_id and a platform id for matching collectors
gut most of Facts() subclasses
rm Facts() subclasses with weird metaclass
only add collectors that match the fact_ids and the platform_info
to the list of collectors used.
atm, a collectors platform_id will default to 'Generic', and
any platform matches 'Generic'
goal is to select collector classes including matching the
systems platform in collector.py, instead of relying on
metaclasses in hardware/*. To finish this, the various
Facts() subclasses will need to be replaced entirely with
Collector() subclasses.
use collector classmethod platform_match() to match the platform
This lets the particular class decide if it is compatible with
a given platform_info. platform_info is a dict like obj, so it could be
expanded in the future.
Add a default platform_match to BaseFactCollector that matches
platform_info['system'] == cls._platform
They were needed previously to trigger a module
load on all the collector classes when we import
facts/hardare so that the Hardware() and related
classes that used __new__ and find_all_subclasses()
would work.
Now that is done in collectors based on platform matching
at runtime we dont need to do it py module import/parse
time. So the non empty __init__.pys are no longer needed
and their is a more flexible mechanism for selection
platform specific stuff.
facts/facts.py is no longer used, rm'ed
- if we dont find an implement class for gather spec.. just ignore it.
Would be useful to add a warn to warn about this case.
- Fix SD-UX typo (should be HP-UX)
- Port fix for #21893 (0 sockets) to this branch
This readds the change from 8ad182059d
that got lost in merge/rebase
Fixes#21893
- port sunos fact locale fix for #24542 to this branch
based on e558ec19cdFixes#24542
Solaris fact fix (#24793)
ensure locale for solaris fact gathering
fixes issue with locale interfering with proper reading of decimals
- raise exceptions in the air like we just dont care.
Pretty much ignore any not exit exception in facts
collection. And add some test cases.
- added new selinux fact to clarify python lib
the selinux fact is boolean false when the library is not installed,
a dictionary/hash otherwise, but this is ambigous
added new fact so we can eventually remove the type dichtomy and normalize it as a dict
Re-add of devel commit 85c7a7b844 to
the new code layout, since it got removed in merge/rebase
Fix adds check for values provided by user for
name and value in sysctl module.
While providing name and value as in-line params,
check for blank values
Fixes#20176
Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
With newer versions of ansible, module arguments are assumed to
be strings unless otherwise specified. Our 'facts' argument is
expected to be a dictionary, so tell ansible that.
Without this, the argument will arrive as a string and be written
to the facter file inside string quotes. Facter will produce the
following error:
undefined method `each' for #<String:0x000000016ee640>
This was originally fixed and found in the Ansible Puppet role which
is maintained by the OpenStack infrastructure team.
8d0f0bfd0a
Puppet modules are not always installed in the default location
(i.e, /etc/puppet/modules) so it is useful to be able to specify
an alternate location.
* Fix systemd in chroot
The 'request ignored' message is in stderr, not stdout.
* Check both stdout and stderr for systemd message
Some versions of systemd report to stderr, others to stdout. Also check
whether output could be a valid normal response to avoid false positives.
The cron module forces changed=True when there was no real change,
but the original crontab did not contain a final newline, which is
mandatory.
When the user has no crontab or the user does not exist at all,
crontab -l exits with 1 and the cron module correctly interprets
this as "no crontab" and stores the old crontab as "".
However this triggers changed=True, even if we're not going to
change anything, e.g. when removing a crontab entry from a user
who has no crontabs at all.
Let's special-case the fact that the old crontab is empty and not
force changed=True in that case.
Changes to the metadata format were approved here:
https://github.com/ansible/proposals/issues/54
* Update documentation to the new metadata format
* Changes to metadata-tool to account for new metadata
* Add GPL license header
* Add upgrade subcommand to upgrade metadata version
* Change default metadata to the new format
* Fix exclusion of non-modules from the metadata report
* Fix ansible-doc for new module metadata
* Exclude metadata version from ansible-doc output
* Fix website docs generation for the new metadata
* Update metadata schema in valiate-modules test
* Update the metadata in all modules to the new version
* Add simple module to import/delete ssl certificate from/to java keystore.
* add diff/check mode support, fix reported issues and simplify decision logic
* Fix build by adding new line at the end of file.
* Some updates to module requested PR review
* Add simple executable parameter
* Fix build error
* fix module doc fields
* More module docs corrections
* More module docs corrections
* More module docs corrections
* More module docs corrections
* correct aliases
* Review comments
* Must quote ':'
* More authors
* Use suboptions:
* restore type: bool
* type should be in the same place
* More tidyups
* authors
* Use suboptions
* revert
* remove duplicate author
* More issues post rebase
* Added parted module
* Amended documentation
* Improved documentation
* Managed the case of parted not returning device information.
* Fixed Shippable test not passing
* Fixed compatibility with Python 2.4
* Cleanup of the state option, improved flags management, documentation refinements.
* Compacted format function, code style cleanups, amended comments.
* Fixed bug related to parted data parsing.
* Support for check-mode, amended size data type.
* Fixed Shippable test not passing.
* Added full suport for CHS and CYL units, applied suggested changes.
get_masquerade_* functions only take one arg. The action_handler
wrapper function expected a tuple, but was being passed (zone)
instead of (zone,) making for an ambiquous tuple. The
(zone) arg was being treated as a tuple/list of six chars
(the zone name) instead of a tuple of one string.
This would cause errors like:
get_masquerade_enabled_permanent() takes exactly 1 argument (6 given)
Fixes#21632
* Added the aix_inittab module to be able to modify the inittab at AIX systems.
* fixed identation errors
* fixed identations and trailing whitespace
changed if conditions
* repaired tab-identation
* adjusted to long line and identation
* Adjusted the yaml in the example documentation
* Adjusted the documentation yaml
* Repaired return yaml
* repaired typo
* Removed unnecessary parameters from comment
changed action to state, and changed the way to call this module
changed proccessaction to action
* adjusted result['warnings'] to module.warn()
* adjusted the documentation
* changed warning to failed
ran module against autopep8
* added check_mode
* fixed typo
* changed description and short description, to be able to push again after a failure at shippable
'#' and ';' are both valid comment chars for sysctl.conf files
according to the 'man sysctl.conf':
"Lines which begin with a # or ; are considered comments and ignored."
Fixes#20569
* Avoid having module documentation links to itself
A lot of modules use M(own_module) in their documentation causing a link
in the documentation to itself.
* Make note more clear now
* known_hosts: support --diff
* known_hosts: support --diff also without --check
* Add unit tests and fix incorrect diff in one corner case
Tests are good!
* Refactor for readability
* Python 3 compat
* More Python 3 compat
* Add an integration test for known_hosts
* Handle ssh-keygen -HF returning non-zero exit code
AFAICT this is a bug in ssh-keygen in some newer OpenSSH versions
(>= 6.4 probably; see commit dd9d5cc670):
when you invoke ssh-keygen with -H and -F <host> options, it always
returns exit code 1. This is because in ssh-keygen.c there's a function
do_known_hosts() which calls
exit (find_host && !ctx.found_key);
at the end, and find_host is 1 (because we passed -F on the command line),
but ctx.found_key is always 0. Why is found_key always 0? Because the
callback passed to hostkeys_foreach(), which is known_hosts_hash(),
never bothers to set found_key to 1.
* This test does not need root
* Avoid ssh-ed25519 keys in sample known_hosts file
Older versions of OpenSSH do not like them and ssh-keygen -HF
aborts with an error when it sees such keys:
line 5 invalid key: example.net...
/root/ansible_testing/known_hosts is not a valid known_hosts file.
* Fix Python 3 errors
Specifically, the default mode of tempfile.NamedTemporaryFile is 'w+b',
which means Python 3 wants us to write bytes objects to it -- but the
keys we have are all unicode strings.
* Add Check Mode capability to kernel_blacklist module
* Add suggested changes from @tmshn
* Pass a bool into `Blacklist` that'll just be `module.check_mode`
* Move detection and creation of a file to a separate function within `Blacklist`
* If there's no file on the system and we are running under `Check Mode`, set `self.filename` to `os.devnull` AND mark a change, as the module would have created an empty file.
* Whenever a `self.filename` is being opened in a mode where changes can be made, replace with an if statement that checks `self.checkmode` and if true then open up `os.devnull` instead
* Update validate-modules
* Validates ANSIBLE_METADATA
* Ensures imports happen after documentation vars
* Some pep8 cleanup
* Clean up some left over unneeded code
* Update modules for new module guidelines and validate-modules checks
* Update imports for ec2_vpc_route_table and ec2_vpc_nat_gateway
* Fix service's exec_command() for python3
exec_command() was mixing text and bytes in several places on python3.
Made changes so that we explicitly convert between the two.
Fixes#20818
* Also handle basestring and shlex.split in a python2/python3 compatible fashion
* Tweaks for SmartOS:
- prevent attempting from changing timezone in the global zone (read-only)
- provide meaningful error message in the unlikely case smtools isn't present
* Add support for FreeBSD and NetBSD to timezone module
In our environment we have custom services that need to be stopped and
restarted very gracefully to not interrupt active sessions.
A stop job, depending on the state, can take up to 20 minutes until the
process exits. It simply reacts to SIGTERM with a graceful shutdown.
By default, systemctl blocks until the job has completed, which leads to
Ansible hanging on this task for up to 20 minutes.
Thankfully systemctl supports the `--no-block` flag which lets the job
continue in the background.
This PR adds support for that flag as the `no_block` boolean option.
From the man page:
--no-block
Do not synchronously wait for the requested operation to
finish. If this is not specified, the job will be
verified, enqueued and systemctl will wait until the
unit's start-up is completed. By passing this argument,
it is only verified and enqueued. This option may not be
combined with --wait.
Not all file-related modules consistently use "path" as the attribute to specify a single filename, some use "dest", others use "name". Most do have aliases for either "name" or "destfile".
This change makes "path" the default attribute for (single) file-related modules, but also adds "dest" and "name" as aliases, so that people can use a consistent way of attributing paths, but also to ensure backward compatibility with existing playbooks.
NOTE: The reason for changing this, is that it makes Ansible needlessly harder to use if you have to remember that e.g. the xattr module requires the name attribute, the lineinfile module requires a dest attribute, and the stat module requires a path attribute.
* Small tweaks for timezone module:
- small textual fixes
- ensure the generated docs list either hwclock or name as required
by using a non-generated value for required_one_of
* Update docs with the DOCUMENTATION block about either name/hwclock being required
Adrian Likins
Łukasz Jendrysik
Aurimas Lapiene
Evan Kaufman
tdtrask
Abhijeet Kasurde
Fabrizio Colonna
Brian Coca
cheko
Anton Kasimov
John R Barker
Dylan Silva
Dag Wieers
Mohammed Naser
Toshio Kuratomi
taksekine
Joris (J.J.M.) Weijters
jctanner
David Moreau Simard
Colin Chan
Andrea Tartaglia
Kai
Bart van Bragt
adejoux
Colin Nolan
Guillaume Coré
Strahinja Kustudić
Jeremy L. Gaddis
Evgeni Golov
Matt Martz
Alexander Gubin
Adam Hamsik
Hao
Kamil Cholewiński
Pierre-Louis Bonicoli
molekuul
Matt Clay
Brian Coca
Jiri Tyr
Marius Gedminas
Imran Hussain
James Sumners
Jasper Lievisse Adriaanse
Fabio Alessandro Locati
Tobias Wolf
Michael Kwardakov