Commit Graph

274 Commits (dfaa72297ccec757aed0c6eeaadaa0188e8ec34f)

Author SHA1 Message Date
Brian Coca 1a893a48a3
Allow finer grained control for dupe YAML keys (#56933)
* Allow finer grained control for dupe YAML keys

  fixes #16903

* expand option to handle errors

* remove clog for previous version of toggle

* added missing parens

* fix typoe
2019-05-28 11:57:16 -04:00
Brian Coca 27dcf8aaab
Not native, but text (#55676)
* use to_text instead of to_native
* cleaned up some imports and other pyflakisms
* fix missing lib messages
2019-05-24 14:49:04 -04:00
Andrew Gaffney 7f0bf0a98a
Config toggle for duplicate dict key warnings (fixes #56799) (#56805) 2019-05-23 08:42:44 -05:00
Matt Martz db6cc60352
Migrate command line parsing to argparse (#50610)
* Start of migration to argparse

* various fixes and improvements

* Linting fixes

* Test fixes

* Fix vault_password_files

* Add PrependAction for argparse

* A bunch of additional tweak/fixes

* Fix ansible-config tests

* Fix man page generation

* linting fix

* More adhoc pattern fixes

* Add changelog fragment

* Add support for argcomplete

* Enable argcomplete global completion

* Rename PrependAction to PrependListAction to better describe what it does

* Add documentation for installing and configuring argcomplete

* Address rebase issues

* Fix display encoding for vault

* Fix line length

* Address rebase issues

* Handle rebase issues

* Use mutually exclusive group instead of handling manually

* Fix rebase issues

* Address rebase issue

* Update version added for argcomplete support

* -e must be given a value

* ci_complete
2019-04-23 13:54:39 -05:00
Matt Davis f86345f777
Collection content loading (#52194)
* basic plugin loading working (with many hacks)

* task collections working

* play/block-level collection module/action working

* implement PEP302 loader

* implicit package support (no need for __init.py__ in collections)
* provides future options for secure loading of content that shouldn't execute inside controller (eg, actively ignore __init__.py on content/module paths)
* provide hook for synthetic collection setup (eg ansible.core pseudo-collection for specifying built-in plugins without legacy path, etc)

* synthetic package support

* ansible.core.plugins mapping works, others don't

* synthetic collections working for modules/actions

* fix direct-load legacy

* change base package name to ansible_collections

* note

* collection role loading

* expand paths from installed content root vars

* feature complete?

* rename ansible.core to ansible.builtin

* and various sanity fixes

* sanity tweaks

* unittest fixes

* less grabby error handler on has_plugin

* probably need to replace with a or harden callers

* fix win_ping test

* disable module test with explicit file extension; might be able to support in some scenarios, but can't see any other tests that verify that behavior...

* fix unicode conversion issues on py2

* attempt to keep things working-ish on py2.6

* python2.6 test fun round 2

* rename dirs/configs to "collections"

* add wrapper dir for content-adjacent

* fix pythoncheck to use localhost

* unicode tweaks, native/bytes string prefixing

* rename COLLECTION_PATHS to COLLECTIONS_PATHS

* switch to pathspec

* path handling cleanup

* change expensive `all` back to or chain

* unused import cleanup

* quotes tweak

* use wrapped iter/len in Jinja proxy

* var name expansion

* comment seemingly overcomplicated playbook_paths resolution

* drop unnecessary conditional nesting

* eliminate extraneous local

* zap superfluous validation function

* use slice for rolespec NS assembly

* misc naming/unicode fixes

* collection callback loader asks if valid FQ name instead of just '.'
* switch collection role resolution behavior to be internally `text` as much as possible

* misc fixmes

* to_native in exception constructor
* (slightly) detangle tuple accumulation mess in module_utils __init__ walker

* more misc fixmes

* tighten up action dispatch, add unqualified action test

* rename Collection mixin to CollectionSearch

* (attempt to) avoid potential confusion/conflict with builtin collections, etc

* stale fixmes

* tighten up pluginloader collections determination

* sanity test fixes

* ditch regex escape

* clarify comment

* update default collections paths config entry

* use PATH format instead of list

* skip integration tests on Python 2.6

ci_complete
2019-03-28 10:41:39 -07:00
Abhijeet Kasurde 142732dba9 dataloader: check exact value of dir (#52021)
Include path in role with directory which has 'tasks' as end.
For example, roles/sometasks/templates is now considered while searching path.

Fixes: #42585

Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
2019-03-16 01:48:01 -04:00
Matt Clay 3e778d3f8f Fix pycodestyle E117 issues. 2019-02-13 23:35:39 -08:00
Abhijeet Kasurde 5c992fcc3f
ansible-vault: handle utf-8 filename in vault (#50341)
Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
2019-02-12 16:13:51 +05:30
Matt Martz 8c08d03989
Fieldattribute inheritance with defaults (#50891)
* Add tests for check_mode at play and task level

These test inheritance of check_mode from the various levels (command
line, as a play attribute and as a task attribute) so they will be
useful for checking that the change to fieldattribute inheritance with
defaults works

* Add a sentinel object

The Sentinel object can be used in place of None when we need to mark an
entry as being special (usually used to mark something as not having
been set)

* Start of using a Sentinel object instead of None.

* Handle edge cases around use of Sentinel

* _get_parent_attribute needs to deal in Sentinel not None

* No need to special case any_errors_fatal in task.py any longer

* Handle more edge cases around Sentinel

* Use Sentinel instead of None in TaskInclude

* Update code to clarify the vars we are copying are class attrs

* Add changelog fragment

* Use a default of Sentinel for delegate_to, this also allows 'delegate_to: ~' now to unset inherited delegate_to

* Explain Sentinel stripping in _extend_value

* Fix ModuleArgsParser tests to compare with Sentinel

* Fixes for tasks inside of roles inheriting from play

* Remove incorrect note. ci_complete

* Remove commented code
2019-01-23 11:40:07 -06:00
Matt Clay b971ebd343 More path encoding fixes. 2019-01-16 15:44:50 -08:00
Abhijeet Kasurde 4817dcd0fc Minor typos (#50371)
Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
2019-01-15 11:37:00 +00:00
Toshio Kuratomi 3fba006207 Update bare exceptions to specify Exception.
This will keep us from accidentally catching program-exiting exceptions
like KeyboardInterupt and SystemExit.
2018-12-16 15:03:19 -08:00
Dag Wieers baf0ad2309 Docs: Add a "seealso" section to the module docs (#45949)
* Docs: Add a separate  "seealso" section to the module docs
to list related modules and/or related references. This clears up the notes
section for things that are actual notes.

So you can add a section in your module documentation and four types of
references are possible.

    seealso:

    # Reference by module name
    - module: aci_tenant

    # Reference by module name, including description
    - module: aci_tenant
      description: ACI module to create tenants on a Cisco ACI fabric.

    # Reference by rST documentation anchor
    - ref: aci_guide
      description: Detailed information on how to manage your ACI infrastructure using Ansible.

    # Reference by Internet resource
    - name: APIC Management Information Model reference
      description: Complete reference of the APIC object model.
      link: https://developer.cisco.com/docs/apic-mim-ref/

This PR also includes:

- Implements ansible-doc support
- Implements schema support for the seealso options
- Updates to the development documentation
- Rename filter convert_symbols_to_format to rst_ify, cfr the existing html_ify and tty_ify filters
  - This makes the existing template a lot easier to read and fixes the confusion I had myself rereading the template (again).
- We fixed the possible suboption types (which was limited to 'bool' only)

* Use latest stable instead of devel docs
2018-12-12 14:19:58 -06:00
Matt Martz c9040d7579
Don't just raise AnsibleError with the exception message (#49654)
* Don't just raise AnsibleError with the exception message. Fixes #49252

* Add changelog fragment
2018-12-11 09:27:10 -06:00
Brian Coca 597c258e0e
explain 'bare variables' (#49577) 2018-12-07 14:47:56 -05:00
Brian Coca 8c43ce491e unsafe cleanup 2018-11-30 13:07:00 -08:00
Matt Martz 9773a1f289
Add a Singleton metaclass, use it with Display (#48935)
* Add a Singleton class, use it with Display

* update six import

* Move remaining failes to display singleton

* Fix rebase issues

* Singleton improvements

* Add code-smell for 'from __main__ import display'. ci_complete

* s/self/cls/g

* Add docs for no-main-display

* Address linting issues

* Add changelog fragment. ci_complete

* Implement reentrant lock for class instantiation in Singleton

* Add Display singleton porting guide
2018-11-20 17:06:51 -06:00
Martin Krizek a80c25cbd9
Do not use str() on exceptions (#46950) 2018-11-09 07:59:30 +01:00
Toshio Kuratomi dd906e20f4 Set the default metadata 2018-10-31 10:41:46 -07:00
Toshio Kuratomi 61ae6424a3 An earlier optimization of ansible-doc -l caused failures. (#47012)
* An earlier optimization of ansible-doc -l caused failures.

The optimization quickly searches the plugin code for short_description
fields and then uses that in the -l output.  The searching was a bit too
naive and ended up pulling out malformed yaml.  This caused those
plugins to be omitted from the list of plugins of that type with
a warning that their documentation strings were wrong.

This change makes the documentation parser aware that the documentation
string could have a relative indent for all of its fields which makes it
robust in the face of this particular problem.

* Don't search for space after short_description:

Any whitespace would be valid.  In particular newline
2018-10-17 11:15:59 -04:00
Brian Coca b2ac745461 add note to inform awx team on change 2018-10-11 12:45:53 -04:00
Eitan Adler 063a9dd911 Reduce noise in warnings from ansible (#40754)
When python is compiled in debug mode, or certain command line flags are
passed, python issues helpful warnings to let users know of files opened
by not closed.

This can be fairly spammy on an ansible run.  This change reduces the
number of such warnings by a factor of 10.
2018-10-09 18:53:03 -04:00
Matt Clay a11f631ee4 Python 3.8 collections compatibility fixes.
Includes a new pylint blacklist plugin to prevent regressions.
2018-10-08 11:26:37 -07:00
Dag Wieers b72e989e18 Ensure _raw_params retain exact spaces
This fixes a few issues related to multi-line YAML strings that eat
whitespace and add whitespaces after newlines (that weren't there).
2018-09-28 19:19:58 +02:00
Brendan Jurd ab96bbdef9 Clean up Vault docs in User Guide. (#46188)
* Fix spelling of 'separate' throughout.

* Various cleanups in the User Guide for Vault.

- Fix spelling of 'algorithm'
- Fix indentation of nested list in payload format
- Fix mysterious refernce to 'b_pkey1'.
- Fix reference to newline as '\n': the backslash is lost when rendered
to the docs website. Specify the hex value for newline instead of the
backslash escape.

* Fix formatting

* Update vault.rst
2018-09-28 10:16:58 -04:00
Dag Wieers 771a3983d2 Docs: Keep the module index clean (#46155)
A lot of modules have a short_description with a trailing dot even
though we don't want trailing dots in the index. This change removes
it when creating the document index.
2018-09-27 13:25:38 -05:00
Matt Martz c0915e2f5a
Support nested JSON decoding in AnsibleJSONDecoder (#45924)
* Support nested JSON decoding in AnsibleJSONDecoder

* Add tests for vault portion of AnsibleJSONDecoder
2018-09-24 14:33:19 -05:00
Toshio Kuratomi 0873d46617 Fix ansible-doc and docsite generation for removed modules
* Fix ansible-doc wrt removed modules
  * Fix listing of modules ia ansible-doc to not complain about removed modules
    Removed modules are marked as such in the metadata but nowhere else.
    Need to retrieve the metadata when a module doesn't have a doc so that
    we can tell if it falls under this case.
  * omit removed modules from json dump
  * Print an error that the module has been removed if attempting to run
    ansible-doc on that specific module

* Get plugin_formatter to stop outputting removed modules
2018-08-24 15:37:13 -07:00
Matt Martz 81ca04512d
Raise a nicer error when we cannot execute the editor (#44423)
* Raise a nicer error when we cannot execute the editor. Fixes #44419

* Don't use to_bytes when constructing an exception

* Add changelog fragment
2018-08-22 10:24:11 -05:00
Matt Martz 617372f8c0
Mass nuke deprecated items that are easily removed. ci_complete (#44320) 2018-08-20 16:26:10 -05:00
Matt Martz c1c229c6d4
Remove use of simplejson throughout code base (#43548)
* Remove use of simplejson throughout code base. Fixes #42761

* Address failing tests

* Remove simplejson from contrib and other outlying files

* Add changelog fragment for simplejson removal
2018-08-10 11:13:29 -05:00
Joren Vrancken b954917761 Surround top-level function and class definitions with two blank lines. 2018-07-31 12:06:56 -07:00
Eric Rinish 551501f326 use faster method to get short_description from DOCUMENTATION (#42705)
* use faster method to get short_description from DOCUMENTATION

* fixed pep8 whitespace

* fixed blank line
2018-07-27 17:27:44 -04:00
Abhijeet Kasurde 4f1746ee1d
vault: check dir value before passing (#43184)
This fix checks if dirname is not equal to '' before proceeding
to create actual directory with name.

Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
2018-07-24 16:27:17 +05:30
Troy Murray 15ce7c5bab change OS X to macOS (#41294)
* change OS X to macOS

<!--- Your description here -->

+label: docsite_pr

* Update all Mac OS X references to be macOS

* Drop extra Mac
2018-06-26 14:09:23 -04:00
Andrew Gaffney 0ff04ad41b Fix wrong python keyword (followup to #36357) (#41067)
Also add tests around that code path
2018-06-07 19:48:07 +05:30
Tom Wassenberg 6097776dc0 Fix flag in debug message (#40753)
The debug message stated that the vault password client script was invoked as `example.py --vault-id=example`, while it's actually `example.py --vault-id example`.
2018-06-01 13:48:22 -04:00
Adrian Likins 1613a739ad fix decrypted vault utf8 values (#37539)
* Fix errors decrypted non-ascii vault vars

AnsibleVaultEncryptedUnicode was just using b"".decode()
instead of to_text() on the bytestrings returned from
vault.decrypt() and could cause errors on python2
if non-ascii since decode() defaults to ascii.
Use to_text() to default to decoding utf-8.

add intg and unit tests for value of vaulted vars
being non-ascii utf8

based on https://github.com/ansible/ansible/issues/37258

Fixes #37258

* yamllint fixups
2018-05-24 23:22:46 -04:00
Brian Coca 5852df57f4 allow dataloader to use descriptors (#40627)
also fixed improper uses of str()

fixes #40383
2018-05-24 12:29:37 +10:00
Martin Krizek dccb0d035b
vault: check dir existence before creating a file (#40010) 2018-05-11 19:29:44 +02:00
Brian Coca cbb6a7f4e8
Enabled unsafe and vault in JSON (#38759)
* allow to load json marked as unsafe or vault

 * centralized json code/decode, add vault support
 * use generics to allow for more varied inputs
 * allow inventory to dump vault w/o decrypting
 * override simplejson also
 * add entry for unsafe also
 * load vaulted and unsafe json, support unvaulting if secrets provided
2018-05-09 14:01:51 -04:00
Toshio Kuratomi 3e64036684 Fix encoding issue when parsing plugin examples
This caused a non-fatal traceback when building, for example, the module
documentation for the debug module.
2018-05-08 12:21:52 -07:00
Andrew Gaffney 95ce00ff00 Allow loading dirs from role defaults/vars (#36357)
This commit moves code to look for vars files/dirs to a common place and
uses it for loading role defaults/vars. This allows things such as
'defaults/main' or 'vars/main' being a directory in a role, allowing
splitting of defaults/vars into multiple files. This commit also fixes
the role loading unit tests for py3 when bytestrings are used for paths
instead of utf8 strings.

This fixes #14248 and #11639.
2018-04-10 17:14:38 -04:00
Adrian Likins 6e737c8cb6
Fix 'New Vault password' on vault 'edit' (#35923)
* Fix 'New Vault password' on vault 'edit'

ffe0ddea96 introduce a
change on 'ansible-vault edit' that tried to check
for --encrypt-vault-id in that mode. But '--encrypt-vault-id'
is not intended for 'edit' since the 'edit' should always
reuse the vault secret that was used to decrypt the text.

Change cli to not check for --encrypt-vault-id on 'edit'.

VaultLib.decrypt_and_get_vault_id() was change to return
the vault secret used to decrypt (in addition to vault_id
and the plaintext).

VaultEditor.edit_file() will now use 'vault_secret_used'
as returned from decrypt_and_get_vault_id() so that
an edited file always gets reencrypted with the same
secret, regardless of any vault id configuration or
cli options.

Fixes #35834
2018-03-27 14:12:21 -04:00
Toshio Kuratomi 9890ce47e8 Optimizations in mod_args parsing
* Make it less likely that we have to identify all the modules during
  a playbook run.  PluginLoader is optimized to look for modules one
  directory at a time.  If we find a module before we've examined all
  the directories we never have to touch the other directories.
  Reordering this conditional makes it so tasks which don't have
  a module file will not force us to examine all the module directories
  before moving on to other sources of task actions.
* Change several variables we consult to see if a task is in a certain
  category from lists/tuples to frozensets.  These are static lists
  which we only do containment tests on so frozensets should be faster
2018-03-26 10:08:38 -07:00
Toshio Kuratomi b945846814 Remove dead code
I wrote this code because we had a traceback at some point with pyyaml
not able to handle our AnsibleUnicode type (the C library was doing an
exact match to python's unicode type rather than testing with the object
inheritance hierarchy.) Neither jimi think that AnsibleUnicode gets
passed into this function anymore, though, so we think it is safe to remove.
2018-03-12 12:13:19 -07:00
Andrew Gaffney dae737c8b7 Only template each hostvars var on-demand (fixes #33259) 2018-02-15 10:09:04 -05:00
Matt Martz f659cc5963
Guard against assignments that are not ast.Name in docs parsing (#35513) 2018-01-30 12:56:16 -06:00
Michael Scherer 060001b08d Remove deprecated code (#34828)
* Remove compat code for to_unicode, to_str and to_bytes

Code was marked as deprecated and to be removed after 2.4

* Remove is_encrypted and is_encrypted_file

Code was marked as deprecated after 2.4 release.
2018-01-23 12:02:09 -06:00
Toshio Kuratomi 340a7be7c3 Implement plugin filtering 2018-01-22 16:54:53 -08:00