Commit Graph

711 Commits (a6a73594ecc5ea40cf5aa3ca486e85f952554e0f)

Author SHA1 Message Date
Matt Davis f86345f777
Collection content loading (#52194)
* basic plugin loading working (with many hacks)

* task collections working

* play/block-level collection module/action working

* implement PEP302 loader

* implicit package support (no need for __init.py__ in collections)
* provides future options for secure loading of content that shouldn't execute inside controller (eg, actively ignore __init__.py on content/module paths)
* provide hook for synthetic collection setup (eg ansible.core pseudo-collection for specifying built-in plugins without legacy path, etc)

* synthetic package support

* ansible.core.plugins mapping works, others don't

* synthetic collections working for modules/actions

* fix direct-load legacy

* change base package name to ansible_collections

* note

* collection role loading

* expand paths from installed content root vars

* feature complete?

* rename ansible.core to ansible.builtin

* and various sanity fixes

* sanity tweaks

* unittest fixes

* less grabby error handler on has_plugin

* probably need to replace with a or harden callers

* fix win_ping test

* disable module test with explicit file extension; might be able to support in some scenarios, but can't see any other tests that verify that behavior...

* fix unicode conversion issues on py2

* attempt to keep things working-ish on py2.6

* python2.6 test fun round 2

* rename dirs/configs to "collections"

* add wrapper dir for content-adjacent

* fix pythoncheck to use localhost

* unicode tweaks, native/bytes string prefixing

* rename COLLECTION_PATHS to COLLECTIONS_PATHS

* switch to pathspec

* path handling cleanup

* change expensive `all` back to or chain

* unused import cleanup

* quotes tweak

* use wrapped iter/len in Jinja proxy

* var name expansion

* comment seemingly overcomplicated playbook_paths resolution

* drop unnecessary conditional nesting

* eliminate extraneous local

* zap superfluous validation function

* use slice for rolespec NS assembly

* misc naming/unicode fixes

* collection callback loader asks if valid FQ name instead of just '.'
* switch collection role resolution behavior to be internally `text` as much as possible

* misc fixmes

* to_native in exception constructor
* (slightly) detangle tuple accumulation mess in module_utils __init__ walker

* more misc fixmes

* tighten up action dispatch, add unqualified action test

* rename Collection mixin to CollectionSearch

* (attempt to) avoid potential confusion/conflict with builtin collections, etc

* stale fixmes

* tighten up pluginloader collections determination

* sanity test fixes

* ditch regex escape

* clarify comment

* update default collections paths config entry

* use PATH format instead of list

* skip integration tests on Python 2.6

ci_complete
2019-03-28 10:41:39 -07:00
Matt Martz 6996926d89
Expose loop_control.loop_var as ansible_loop_var (#54240)
* Expose the loop_var as ansible_loop_var

* Add docs

* fix assert test

* Indicate version added
2019-03-26 11:07:54 -05:00
dw 81deb8f132 Avoid WorkerProcess._new_stdin FD sharing (#51623) (#51624)
This avoids holding open _new_stdin within the parent process, where
subsequent WorkerProcess forks will duplicate it, producing significant
noise in the FD table of every worker.

Fix by overriding start() and moving the work to there, with a finally:
to ensure parent FD is closed after start().
2019-03-26 09:41:45 -05:00
Martin Krizek 5ff1ce28d4
Remove "legacy" code from PlayIterator (#53754) 2019-03-25 16:32:27 +01:00
Brian Coca b793f08a92
fixes for stripping (#52930)
function changed to do in place replacement, should be less expensive even with copy as it avoids 'sub copies', can compose with module_args_copy to create replacement for old behavior

  attempt to fix #52910

* handle lists and subdicts correctly
* added  missing exception case, which was not noticed since 'cleaning' was not working
* added comments to clarify exceptions
2019-03-12 18:18:38 -04:00
Jordan Borean c8c326d88e
Use unicode string for auto interpreter warnings (#53671)
* Use unicode string for auto interpreter warnings

* Added some unit tests for interpreter selection

* Fix python 3 syntax issues
2019-03-12 14:16:49 +10:00
Matt Clay 887ab35656
Add support for ansible-test --coverage-check. (#53573) 2019-03-11 15:13:09 -07:00
Brian Coca 8940732b58
Configurable and parallel gather facts (#49399)
* Configurable list of facts modules (#31783)

 - allow for args dict for specific modules
 - add way to pass parameters
 - avoid facts poluting test
 - move to 'facts gathered' flag
 - add 'gathering' setting tests
 - allow parallel option in case serialization is too slow
 - added support to automatically map network facts
   uses "smart" connection mapping
2019-03-08 13:08:37 -05:00
Matt Davis 9c644d9bcc
interpreter discovery message tweaks (#53155)
* fixes #53120 (doclink generator now understands devel)
* added hosts to some warnings
2019-02-28 17:53:55 -08:00
Matt Davis 4d3a6123d5
Python interpreter discovery (#50163)
* Python interpreter discovery

* No longer blindly default to only `/usr/bin/python`
* `ansible_python_interpreter` defaults to `auto_legacy`, which will discover the platform Python interpreter on some platforms (but still favor `/usr/bin/python` if present for backward compatibility). Use `auto` to always use the discovered interpreter, append `_silent` to either value to suppress warnings.
* includes new doc utility method `get_versioned_doclink` to generate a major.minor versioned doclink against docs.ansible.com (or some other config-overridden URL)

* docs revisions for python interpreter discovery

(cherry picked from commit 5b53c0012ab7212304c28fdd24cb33fd8ff755c2)

* verify output on some distros, cleanup
2019-02-27 23:52:02 -08:00
Brian Coca cfba6dfe91
Ensure play order is obeyed (#49897)
* Ensure play order is obeyed

 it was being ignored depending on other options
 also added tests for each order (except shuffle) both serial and not
 fixes #49846
2019-02-27 18:20:16 -05:00
Jordan Borean d55ddec923
Revert Clear failed state in always only if we did rescue (#52968)
* Revert "Clear failed state in always only if we did rescue (#52829)"

This reverts commit f135960fc2.

* Add tests for failed scenario

* Set failed task with EXPECTED FAILURE
2019-02-26 16:22:32 +10:00
Brian Coca 40a053ce78
better error for bad module options (#52726)
also document clearly inline vaults don't work on 'options'
  fixes #52707
2019-02-25 15:58:51 -05:00
Martin Krizek f135960fc2
Clear failed state in always only if we did rescue (#52829)
Fixes #52561
2019-02-25 09:30:17 +01:00
Martin Krizek be9f07279e Add stats on rescued/ignored tasks (#48418)
* Adding rescued/ignored tasks to stats gathering

Fixes #31245

* Amend integration tests to pass

* callback/dense.py: fix too-many-format-args

* Add changelog

* Amend counter_enabled and unixy callbacks

* Fix syntax error

* Fix typo in the changelog

* Remove not needed comment

* Re-add skipped

* Add test for rescued

* Fix colors...

* Fix unstable tests?

* Add a note to the porting guide

* Re-word the note in the porting guide

Fixes #20346
Fixes #24525
Fixes #14393

Co-authored-by: James Cammarata <jimi@sngx.net>
Co-authored-by: Martin Krizek <martin.krizek@gmail.com>
2019-02-20 13:00:47 +10:00
Matt Clay f5c92f6bc1
Allow setting resource.RLIMIT_NOFILE in modules (#51989) 2019-02-15 17:52:35 -08:00
Matt Clay 3e778d3f8f Fix pycodestyle E117 issues. 2019-02-13 23:35:39 -08:00
Matt Martz 445ff39f94
Become plugins (#50991)
* [WIP] become plugins

Move from hardcoded method to plugins for ease of use, expansion and overrides
  - load into connection as it is going to be the main consumer
  - play_context will also use to keep backwards compat API
  - ensure shell is used to construct commands when needed
  - migrate settings remove from base config in favor of plugin specific configs
  - cleanup ansible-doc
  - add become plugin docs
  - remove deprecated sudo/su code and keywords
  - adjust become options for cli
  - set plugin options from context
  - ensure config defs are avaialbe before instance
  - refactored getting the shell plugin, fixed tests
     - changed into regex as they were string matching, which does not work with random string generation
     - explicitly set flags for play context tests
 - moved plugin loading up front
 - now loads for basedir also
 - allow pyc/o for non m modules
 - fixes to tests and some plugins
 - migrate to play objects fro play_context
 - simiplify gathering
 -  added utf8 headers
 - moved option setting
 - add fail msg to dzdo
 - use tuple for multiple options on fail/missing
 - fix relative plugin paths
 - shift from play context to play
 - all tasks already inherit this from play directly
 - remove obsolete 'set play'
 - correct environment handling
 - add wrap_exe option to pfexec
 - fix runas to noop
 - fixed setting play context
 - added password configs
 - removed required false
 - remove from doc building till they are ready

future development:
  - deal with 'enable' and 'runas' which are not 'command wrappers' but 'state flags' and currently hardcoded in diff subsystems

* cleanup

  remove callers to removed func
  removed --sudo cli doc refs
  remove runas become_exe
  ensure keyerorr on plugin
  also fix backwards compat, missing method is attributeerror, not ansible error
  get remote_user consistently
  ignore missing system_tmpdirs on plugin load
  correct config precedence
  add deprecation
  fix networking imports
  backwards compat for plugins using BECOME_METHODS

* Port become_plugins to context.CLIARGS

This is a work in progress:
* Stop passing options around everywhere as we can use context.CLIARGS
  instead

* Refactor make_become_commands as asked for by alikins

* Typo in comment fix

* Stop loading values from the cli in more than one place

Both play and play_context were saving default values from the cli
arguments directly.  This changes things so that the default values are
loaded into the play and then play_context takes them from there.

* Rename BECOME_PLUGIN_PATH to DEFAULT_BECOME_PLUGIN_PATH

As alikins said, all other plugin paths are named
DEFAULT_plugintype_PLUGIN_PATH.  If we're going to rename these, that
should be done all at one time rather than piecemeal.

* One to throw away

This is a set of hacks to get setting FieldAttribute defaults to command
line args to work.  It's not fully done yet.

After talking it over with sivel and jimi-c this should be done by
fixing FieldAttributeBase and _get_parent_attribute() calls to do the
right thing when there is a non-None default.

What we want to be able to do ideally is something like this:

class Base(FieldAttributeBase):
    _check_mode = FieldAttribute([..] default=lambda: context.CLIARGS['check'])

class Play(Base):
    # lambda so that we have a chance to parse the command line args
    # before we get here.  In the future we might be able to restructure
    # this so that the cli parsing code runs before these classes are
    # defined.

class Task(Base):
    pass

And still have a playbook like this function:

---
- hosts:
  tasks:
  - command: whoami
    check_mode: True

(The check_mode test that is added as a separate commit in this PR will
let you test variations on this case).

There's a few separate reasons that the code doesn't let us do this or
a non-ugly workaround for this as written right now.  The fix that
jimi-c, sivel, and I talked about may let us do this or it may still
require a workaround (but less ugly) (having one class that has the
FieldAttributes with default values and one class that inherits from
that but just overrides the FieldAttributes which now have defaults)

* Revert "One to throw away"

This reverts commit 23aa883cbed11429ef1be2a2d0ed18f83a3b8064.

* Set FieldAttr defaults directly from CLIARGS

* Remove dead code

* Move timeout directly to PlayContext, it's never needed on Play

* just for backwards compat, add a static version of BECOME_METHODS to constants

* Make the become attr on the connection public, since it's used outside of the connection

* Logic fix

* Nuke connection testing if it supports specific become methods

* Remove unused vars

* Address rebase issues

* Fix path encoding issue

* Remove unused import

* Various cleanups

* Restore network_cli check in _low_level_execute_command

* type improvements for cliargs_deferred_get and swap shallowcopy to default to False

* minor cleanups

* Allow the su plugin to work, since it doesn't define a prompt the same way

* Fix up ksu become plugin

* Only set prompt if build_become_command was called

* Add helper to assist connection plugins in knowing they need to wait for a prompt

* Fix tests and code expectations

* Doc updates

* Various additional minor cleanups

* Make doas functional

* Don't change connection signature, load become plugin from TaskExecutor

* Remove unused imports

* Add comment about setting the become plugin on the playcontext

* Fix up tests for recent changes

* Support 'Password:' natively for the doas plugin

* Make default prompts raw

* wording cleanups. ci_complete

* Remove unrelated changes

* Address spelling mistake

* Restore removed test, and udpate to use new functionality

* Add changelog fragment

* Don't hard fail in set_attributes_from_cli on missing CLI keys

* Remove unrelated change to loader

* Remove internal deprecated FieldAttributes now

* Emit deprecation warnings now
2019-02-11 11:27:44 -06:00
Martin Krizek 9301c18428 Pass number of forks to TQM from CLI/config (#51669)
Fixes #51622
2019-02-05 15:23:12 -06:00
dw 30eea29373 Avoid accidental fork bombs due to try/except (#51633) (#51636)
WorkerProcess exceptions could previously cause StrategyBase loop to
resume in every child. See ticket description.
2019-02-04 13:52:26 -05:00
Matt Clay f6e129046a Fix performance issue with large task retries. (#51581) 2019-01-31 18:44:57 -05:00
Brian Coca 4ac0c23db6
added unsafe toggle to vars_prompt (#49219)
* added unsafe toggle to vars_prompt

	fixes #47534
2019-01-30 15:01:13 -05:00
Brian Coca 960388143f
faster config loading (#48333)
* faster config loading

  - used already loaded module var instead of doc functions
  - add preload to populate config defs cache
  - avoid debug work when not in debug
  - generators, force consumption
2019-01-23 12:06:54 -05:00
Martin Krizek 3dc5ba6186
Remove unnecessary calls to cache_block_tasks noop (#50715) 2019-01-17 08:29:26 +01:00
Brian Coca 119b65f919
Clarify clear facts (#50667)
* Revert "avoid x2 setting of set_fact when 'cacheable' (#50564)"

This reverts commit 207848f354.

* clarify clear_facts with set_fact cacheable

 revert previous 'fix' as it will break playbooks by changing precedence
 opted to leave current behaviour but document it on both plugins to mitigate confusion

 fixes #50556

 also fix grammer, add comment, remove unused e
2019-01-15 15:20:33 -05:00
Abhijeet Kasurde db8702cdb8 Close all open filehandle (#50544)
Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
2019-01-11 10:14:08 -05:00
Martin Krizek 4c72ef124b tqm._initialize_notified_handlers() is noop now (#50713) 2019-01-09 10:15:11 -05:00
Brian Coca 207848f354 avoid x2 setting of set_fact when 'cacheable' (#50564)
* avoid x2 setting of set_fact when 'cacheable'

  fixes #50556

* ammend docs to new behaviour

* added period
2019-01-08 10:12:49 -05:00
Abhijeet Kasurde bc6cd13874
Handle exception raised in recursive_finder API (#49590)
User module can contain Indentation errors or syntax errors.
Handle AST exceptions rather than showing traceback while importing such module.

Fixes: #21707

Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
2019-01-04 10:33:37 +05:30
Toshio Kuratomi afdbb0d9d5 Save the command line arguments into a global context
* Once cli args are parsed, they're constant.  So, save the parsed args
  into the global context for everyone else to use them from now on.
* Port cli scripts to use the CLIARGS in the context
* Refactor call to parse cli args into the run() method
* Fix unittests for changes to the internals of CLI arg parsing
* Port callback plugins to use context.CLIARGS
  * Got rid of the private self._options attribute
  * Use context.CLIARGS in the individual callback plugins instead.
  * Also output positional arguments in default and unixy plugins
  * Code has been simplified since we're now dealing with a dict rather
    than Optparse.Value
2019-01-03 18:12:23 -08:00
Nathaniel Case 1829a72885
Allow persistent connection plugins to queue messages back to ansible-connection (#49977)
* Connections can queue messages to be returned from ansible-connection

* Provide fallback for invalid display level

* Strip display from plugins

* Route messages through helper method to try to avoid improper appends
2018-12-19 10:54:42 -05:00
Toshio Kuratomi 3fba006207 Update bare exceptions to specify Exception.
This will keep us from accidentally catching program-exiting exceptions
like KeyboardInterupt and SystemExit.
2018-12-16 15:03:19 -08:00
Jordan Borean 190d1ed7f1 win become: refactor and add support for passwordless become (#48082)
* win become: refactor and add support for passwordless become

* make tests more stable

* fix up dep message for Load-CommandUtils

* Add further check for System impersonation token

* re-add support for become with accounts that have no password

* doc fixes and slight code improvements

* fix doc sanity issue
2018-12-12 17:15:25 -08:00
Nathaniel Case 02432565cd
Remove cliconf from httpapi connection (#46813)
* Bare minimum rip out cliconf

* nxapi changeover

* Update documentation, move options

* Memoize device_info

* Gratuitous rename to underscore use of local api implementation

Fixup eos module_utils like nxos

* Streamline version and image scans

* Expose get_capabilities through module_utils

* Add load_config to module_utils

* Support rpcs using both args and kwargs

* Add get_config for nxos

* Add get_diff

* module context, pulled from nxapi

We could probably do this correctly later

* Fix eos issues

* Limit connection._sub_plugin to only one plugin
2018-12-11 16:26:59 -05:00
Matt Martz 2a469fd959 Consolidate handler tracking (#49338)
* Consolidate handler tracking
  -  Remove unused code. ci_complete
  - unit test fixes. ci_complete
  - Restore previous behavior of matching a single handler
  - when notifying a host for a handler, return True if it was added, False otherwise, to reduce copied logic
  - rename funcitons for clarity. ci_complete
  - Remove handler logic for static includes which was disabled previously
2018-12-11 10:58:19 -05:00
Matt Martz 9007dbec2f
Add info about loop based on jinja2 loop var (#42134)
* Add info about loop based on jinja2 loop var

* ansible_loop

* Update test count

* Add extended loop_control that defines whether ansible_loop should be added

* Extended needs to be defaulted

* Revert "Update test count"

This reverts commit f1e93ee469825f4cdcd90fb28667d29aa088275c.

* Add docs about loop_control.extended

* Add revindex and revindex0

* Document ansible_loop in special vars

* Add changelog fragment

* Add tests, change items to allitems so that dot notation works, fix logic error with previtem
2018-12-07 13:49:50 -06:00
Matt Martz 69d230f4bf Make squash deprecation messages less misleading (#49551) 2018-12-06 10:49:01 -05:00
Monty Taylor 824c72da5f Set play context on stdout callback plugin (#35910)
The loop on self._callback_plugins does not include the stdout callback,
so the stdout_callback never has set_play_context called.
2018-11-29 23:15:56 -05:00
Matt Davis 8c1f701e6e
split PS wrapper and payload (CVE-2018-16859) (#49142)
* prevent scriptblock logging from logging payload contents
* added tests to verify no payload contents in PS Operational event log
* fix script action to send split-aware wrapper
* fix CLIXML error parser (return to -EncodedCommand exposed problems with it)
2018-11-26 15:28:21 -08:00
Matt Martz 9773a1f289
Add a Singleton metaclass, use it with Display (#48935)
* Add a Singleton class, use it with Display

* update six import

* Move remaining failes to display singleton

* Fix rebase issues

* Singleton improvements

* Add code-smell for 'from __main__ import display'. ci_complete

* s/self/cls/g

* Add docs for no-main-display

* Address linting issues

* Add changelog fragment. ci_complete

* Implement reentrant lock for class instantiation in Singleton

* Add Display singleton porting guide
2018-11-20 17:06:51 -06:00
Matt Martz 64bb54dab4 Catch per loop label templating exceptions. Fixes #48879 (#48896) 2018-11-20 15:29:44 -05:00
Abhijeet Kasurde a6fde8aef9 Add timeout information into async failure message
Add some extra information to an async failure message to reflect the
actually timeout value of the failure.

Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2018-11-20 08:46:27 -05:00
Martin Krizek 869d866aa1 display.error on no matching --start-at-task (#45221) 2018-11-15 19:35:51 -05:00
Jordan Borean 4fc013a7fa powershell - fix async for Ansible.Basic (#48404)
* powershell - fix async for Ansible.Basic

* moved old out outside of the try block
2018-11-13 12:08:27 -08:00
Brian Coca b047591dee fix tagged implicit gathering (#48294)
* fix tagged implicit gathering

 - None is not tHe default anymore, its an empty list

* Update changelogs/fragments/fix_taggged_gather.yml

Co-Authored-By: bcoca <bcoca@users.noreply.github.com>
2018-11-09 13:41:07 +00:00
Matt Martz 0d068f1e3a
Support list of dicts with omit. Fixes #45907 (#45923) 2018-11-07 13:41:22 -06:00
Jordan Borean 501acae5ab
Added basic equivalent to PowerShell modules (#44705)
* Added basic equivalent to PowerShell modules

* changes based on latest review

* Added tests

* ignore sanity test due to how tests are set up

* Changes to work with PSCore

* Added documentation and change updated more modules

* Add some speed optimisations to AddType

* fix some issues in the doc changes

* doc changes
2018-11-07 10:53:17 +10:00
Brian Coca 4ec2423a04
make fact gathering obey play tags (#44717)
fixes #44251
2018-11-06 13:19:44 -05:00
Matt Martz 77d32b8f57
Don't use the task for a cache, return a special cache var (#47243)
* Don't use task to cache loop results, use hostvars. Fixes #47207

* Avoid a race condition, supply _ansible_loop_cache through get_vars directly

* Add tests

* Add changelog fragment

* Remove unnecessary copy

* Remove unnecessary host from _get_delegated_vars signature
2018-10-18 15:25:43 -05:00
Matt Martz 02f4d0a57f
Ensure that an empty literal list with loop skips the task (#47129) 2018-10-16 13:35:10 -05:00