Commit Graph

164 Commits (f9b43a0f6843bc8651c914b0b5619ac4f62fa241)

Author SHA1 Message Date
Martin Krizek 34e9d6781b Templar: encapsulate _available_variables (#55435)
Ensure variables are reset between iterations
2019-05-20 11:49:54 -04:00
Sloane Hertel 11279a909d fix combine filter using undefined vars (#55840)
* Check variables are defined before using combine filter

* Add tests for the combine filter

* Remove dependencies that should already be installed

* relocate the function to recursively check for undefined vars

add another test

* changelog
2019-04-30 11:10:19 -04:00
Matt Martz 9f83139dcb
Don't register tests as filters (#55332)
* Don't register tests as filters. Fixes #55319

* Remove tests for deprecated functionality

* Remove no-tests-as-filters sanity tests

* Remove docs too

* Revert "Remove docs too"

This reverts commit 7daf457a742e67ec6e91964852c7f804507b46b8.

* Make no-tests-as-filters doc an orphan
2019-04-16 14:10:14 -05:00
Matt Martz e89f8bae86
Extend jinja2 nested undefined support to keys/indices (#55094) 2019-04-10 10:35:31 -05:00
Matt Davis f86345f777
Collection content loading (#52194)
* basic plugin loading working (with many hacks)

* task collections working

* play/block-level collection module/action working

* implement PEP302 loader

* implicit package support (no need for __init.py__ in collections)
* provides future options for secure loading of content that shouldn't execute inside controller (eg, actively ignore __init__.py on content/module paths)
* provide hook for synthetic collection setup (eg ansible.core pseudo-collection for specifying built-in plugins without legacy path, etc)

* synthetic package support

* ansible.core.plugins mapping works, others don't

* synthetic collections working for modules/actions

* fix direct-load legacy

* change base package name to ansible_collections

* note

* collection role loading

* expand paths from installed content root vars

* feature complete?

* rename ansible.core to ansible.builtin

* and various sanity fixes

* sanity tweaks

* unittest fixes

* less grabby error handler on has_plugin

* probably need to replace with a or harden callers

* fix win_ping test

* disable module test with explicit file extension; might be able to support in some scenarios, but can't see any other tests that verify that behavior...

* fix unicode conversion issues on py2

* attempt to keep things working-ish on py2.6

* python2.6 test fun round 2

* rename dirs/configs to "collections"

* add wrapper dir for content-adjacent

* fix pythoncheck to use localhost

* unicode tweaks, native/bytes string prefixing

* rename COLLECTION_PATHS to COLLECTIONS_PATHS

* switch to pathspec

* path handling cleanup

* change expensive `all` back to or chain

* unused import cleanup

* quotes tweak

* use wrapped iter/len in Jinja proxy

* var name expansion

* comment seemingly overcomplicated playbook_paths resolution

* drop unnecessary conditional nesting

* eliminate extraneous local

* zap superfluous validation function

* use slice for rolespec NS assembly

* misc naming/unicode fixes

* collection callback loader asks if valid FQ name instead of just '.'
* switch collection role resolution behavior to be internally `text` as much as possible

* misc fixmes

* to_native in exception constructor
* (slightly) detangle tuple accumulation mess in module_utils __init__ walker

* more misc fixmes

* tighten up action dispatch, add unqualified action test

* rename Collection mixin to CollectionSearch

* (attempt to) avoid potential confusion/conflict with builtin collections, etc

* stale fixmes

* tighten up pluginloader collections determination

* sanity test fixes

* ditch regex escape

* clarify comment

* update default collections paths config entry

* use PATH format instead of list

* skip integration tests on Python 2.6

ci_complete
2019-03-28 10:41:39 -07:00
Brian Coca d1a688b1d7
dict is dict (#54057) 2019-03-20 09:37:34 -04:00
Abhijeet Kasurde 24d1886499
template: add additional variable for dest path (#52015)
Signed-off-by: Dustin Spicuzza <dustin@virtualroadside.com>
Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
2019-02-19 14:15:19 +05:30
Martin Krizek 3cc4981d63
Clarify StrictUndefined's side-effect in comment (#52508) 2019-02-19 08:47:43 +01:00
Martin Krizek 50d7483632
Native Jinja2: raise undefined error immediately (#52237)
Fixes #52158
2019-02-15 19:36:12 +01:00
Matt Clay 3f387f225c Fix safe_eval on Python 3.8. 2019-02-13 11:54:03 -08:00
Andrew Gaffney 9c35f18dd6 Custom jinja Undefined class for handling nested undefined attributes (#51768)
This commit creates a custom Jinja2 Undefined class that returns
Undefined for any further accesses, rather than raising an exception
2019-02-12 15:04:00 -05:00
Toshio Kuratomi 3fba006207 Update bare exceptions to specify Exception.
This will keep us from accidentally catching program-exiting exceptions
like KeyboardInterupt and SystemExit.
2018-12-16 15:03:19 -08:00
Matt Martz 3a4d476512
Add now() jinja2 global func for getting the date/time (#43792)
* Add now() jinja2 global func for getting the date/time

* Docs and changelog

* now isn't a lookup, it doesn't need disabled
2018-12-07 11:25:53 -06:00
Pilou 62c05033d6 templar: ensure that exceptions are handled, fix 'AttributeError' (#48792)
* templar: ensure that exceptions are handled

* Fix AttributeError: object has no attribute 'message'

'message' attribute is deprecated since Python 2.6 and not available
with Python 3.

Simple reproducer:

    - hosts: localhost
      vars:
        not_json: "{{ 'test str' | from_json }}"
      tasks:
        - command: "echo {{ not_json }}"
2018-11-29 09:56:23 -05:00
Martin Krizek d4568d97d4 Fix using vault encrypted data with jinja2_native (#49025)
Fixes #48950
2018-11-27 16:44:22 -05:00
Martin Krizek abdf46803b
Warn about falling back to jinja2_native=false (#49063) 2018-11-27 18:11:42 +01:00
Matt Martz 9773a1f289
Add a Singleton metaclass, use it with Display (#48935)
* Add a Singleton class, use it with Display

* update six import

* Move remaining failes to display singleton

* Fix rebase issues

* Singleton improvements

* Add code-smell for 'from __main__ import display'. ci_complete

* s/self/cls/g

* Add docs for no-main-display

* Address linting issues

* Add changelog fragment. ci_complete

* Implement reentrant lock for class instantiation in Singleton

* Add Display singleton porting guide
2018-11-20 17:06:51 -06:00
Martin Krizek 541255a2d8
Do not strip new lines in native jinja (#46751)
* Do not strip new lines in native jinja

* Add changelog/fragment
2018-10-19 00:14:43 +02:00
Matt Clay a11f631ee4 Python 3.8 collections compatibility fixes.
Includes a new pylint blacklist plugin to prevent regressions.
2018-10-08 11:26:37 -07:00
James Cassell 211aeebd0d remove unused templar._clean_data() (#42739) 2018-09-11 09:11:03 +02:00
Matt Martz 88509e75ad
Remove bare_deprecated functionality (#44517)
* Remove bare_deprecated functionality

* Change tests due to bare_deprecated removal
2018-08-23 11:31:16 -05:00
Joren Vrancken 7c13c4b207 Add specific exceptions to except 2018-08-01 16:42:05 -04:00
Matt Martz ad0827e5c0
When using ANSIBLE_JINJA2_NATIVE bypass our None filtering in _finalze (#41408)
* When using ANSIBLE_JINJA2_NATIVE bypass our None filtering in _finalize. Fixes #41392

* Add tests for _finalize bypass

* Address python3 failures in tests
2018-06-12 08:46:23 -05:00
jctanner a9e53cdb68 Allow config to enable native jinja types (#32738)
Co-authored-by: Martin Krizek <martin.krizek@gmail.com>
2018-05-31 10:38:29 +02:00
Toshio Kuratomi 81b2529159 Fix when template paths contain non-ascii chars and using the path in ansible_managed
Fixes #27262
2018-04-30 16:47:27 -07:00
Brian Coca 135ec86eb5 fix exception on lookup ignore error
no log, so only_log ...
2018-04-16 18:04:52 -04:00
Victor Bocharsky a30befa609 Remove duplicated "instead" word (#38800)
* Remove duplicated "instead" word

* Actually fix the message
2018-04-16 09:23:43 -04:00
Brian Coca d78da403b6
allow user control of lookup error behaviour (#35932)
* allow user control of lookup error behaviour

this does not affect undefined vars, only other exceptions raised by a lookup

i.e lookup('file' ..) not finding a file
2018-04-12 09:35:00 -04:00
Brian Coca 5a333859fe allow for parameterized string functions
now also allow for ending parens

from following, 1, 2, 4 and 5 match, rest are ignored.

	{doc | to_nice_json}}
	{{(doc | to_nice_json)}}
	{{doc | to_nice_json| from_json}}
	{{doc | to_nice_json()}}
	{{doc | to_nice_json(indent=2)}}
	{{doc | to_nice_json(indent=2) | from_json()}}
	{{doc | to_nice_json(indent=2) | from_json}}

fixes #37579
2018-04-10 17:24:27 -04:00
Matt Martz 1f824bd620 Don't overwrite builtin jinja2 filters with tests (#37881)
* Don't overwrite builtin jinja2 filters with tests. Fixes #37856

* Fix tests and other callers of _get_filters
2018-03-26 12:06:00 -05:00
Toshio Kuratomi 0633f73faf Fix loader for filters (#37748)
* Fix loading of filter and test plugins

Filter and test plugins are different than other plugins in that they
can have many plugins in a single file.  Therefore they need to operate
a little differently.  They need to have all of the potential files
returned.  Then the caller takes care of passing those onto jinja2 in
order for jinja2 to make use of them.

This problem was (most recently) introduced with f921369445

This commit also restructures how we deduplicate plugins to take paths
into account.  If we want to start scoping which set of modules are
loaded (due to roles, for instance) we'll need to hang on to the path
information.

* add integration test for override

* Fix style checks for bcoca code

* Implement jinja2 plugin loader as a subclass

Having a subclass allows us to customize the overriding of jinja
plugins.  We can then move common parts of common code into the Loader.
2018-03-22 17:23:10 -04:00
Toshio Kuratomi 03a6d72633 Revert "remove query/q (#36315)"
This reverts commit b47d2e07e1.

query is a feature for 2.5.  We're not reverting it now.
2018-02-21 10:37:51 -08:00
Brian Coca b47d2e07e1
remove query/q (#36315)
* remove query/q

remove and fixed tests for removed func

* paren fix
2018-02-21 10:42:54 -05:00
Andrew Gaffney dae737c8b7 Only template each hostvars var on-demand (fixes #33259) 2018-02-15 10:09:04 -05:00
Martin Krizek 63fdc3f08f
Fix jinja2>=2.9 nested include vars (#35099)
This fixes a bug when parent's local vars where not available in nested
includes. The bug can only be seen with jinja>=2.9 which changes
how the variable scopes work.

Fixes #34886
2018-02-07 10:58:29 +01:00
Pilou d608eb9530 Check that AnsibleUndefinedVariable doesn't occur when an unused variable references an undefined variable (#35571)
* Check that AnsibleUndefinedVariable doesn't occur

* AnsibleUndefinedVariable exc.: don't modify type
2018-02-02 09:25:13 -08:00
Toshio Kuratomi 3d614bfe84 Clarify lookup plugin TypeError recovery
Change the code and comments around recovering from a TypeError in
handling the return value from lookup plugins to make it clearer what
we're recovering from.
2018-01-30 07:21:36 -08:00
Thomas Quinot 101e983f07 Remove incorrect assumption on exception constructors (#35271)
Do not assume that all exception constructors accept a single string
argument. For example UnicodeError's __init__ takes 5 parameters:
encoding, object, start, end, reason.

Also, if e.message is present but empty, fall back on stringifying
the exception occurrence.

Fixes #35270
2018-01-26 11:07:58 -06:00
Sloane Hertel 7bc754674c
[template/vars] Use to_native(exception) as a fallback for Python 3 which doesn't have a .message attribute (#34044) 2017-12-19 17:34:46 -05:00
Brian Coca 67c83823f2 Add q/query aliases to lookup (#33466)
* Add L alias to lookup, defaults to real list
* renamed to q/query as per core meeting
* also force wantlist=true
2017-12-14 11:53:00 -06:00
Matt Martz 4fe08441be Deprecate tests used as filters (#32361)
* Warn on tests used as filters

* Update docs, add aliases for tests that fit more gramatically with test syntax

* Fix rst formatting

* Add successful filter, alias of success

* Remove renamed_deprecation, it was overkill

* Make directory alias for is_dir

* Update tests to use proper jinja test syntax

* Update additional documentation, living outside of YAML files, to reflect proper jinja test syntax

* Add conversion script, porting guide updates, and changelog updates

* Update newly added uses of tests as filters

* No underscore variable

* Convert recent tests as filter changes to win_stat

* Fix some changes related to rebasing a few integration tests

* Make tests_as_filters_warning explicitly accept the name of the test, instead of inferring the name

* Add test for tests_as_filters_warning

* Update tests as filters in newly added/modified tests

* Address recent changes to several integration tests

* Address recent changes in cs_vpc
2017-11-27 17:58:08 -05:00
Matt Martz 99d4f5bab4 Remove uses of assert in production code (#32079)
* Remove uses of assert in production code

* Fix assertion

* Add code smell test for assertions, currently limited to lib/ansible

* Fix assertion

* Add docs for no-assert

* Remove new assert from enos

* Fix assert in module_utils.connection
2017-11-13 11:51:18 -05:00
Brian Coca e010fcf7e1 added debug info when skipping undef 2017-10-27 12:16:16 -04:00
Matt Clay 68aeaa58a8 Fix dangerous default args. (#29839) 2017-09-12 00:11:13 -07:00
Brian Coca f921369445 Ansible Config part2 (#27448)
* Ansible Config part2

- made dump_me nicer, added note this is not prod
- moved internal key removal function to vars
- carry tracebacks in errors we can now show tracebacks for plugins on vvv
- show inventory plugin tracebacks on vvv
- minor fixes to cg groups plugin
- draft config from plugin docs
- made search path warning 'saner' (top level dirs only)
- correctly display config entries and others
- removed unneeded code
- commented out some conn plugin specific from base.yml
- also deprecated sudo/su
- updated ssh conn docs
- shared get option method for connection plugins
- note about needing eval for defaults
- tailored yaml ext
- updated strategy entry
- for connection pliugins, options load on plugin load
- allow for long types in definitions
- better display in ansible-doc
- cleaned up/updated source docs and base.yml
- added many descriptions
- deprecated include toggles as include is
- draft backwards compat get_config
- fixes to ansible-config, added --only-changed
- some code reoorg
- small license headers
- show default in doc type
- pushed module utils details to 5vs
- work w/o config file
- PEPE ATE!
- moved loader to it's own file
- fixed rhn_register test
- fixed boto requirement in make tests
- I ate Pepe
- fixed dynamic eval of defaults
- better doc code

skip ipaddr filter tests when missing netaddr
removed devnull string from config
better becoem resolution

* killed extra space with extreeme prejudice

cause its an affront against all that is holy that 2 spaces touch each other!

shippable timing out on some images, but merging as it passes most
2017-08-15 16:38:59 -04:00
Zac Medico 501fc7a248 template: fix KeyError: 'undefined variable: 0 (#27972)
* template: fix KeyError: 'undefined variable: 0

For compatibility with the Context.get_all() implementation
in jinja 2.9, make AnsibleJ2Vars implement collections.Mapping.
Also, make AnsibleJ2Template.newcontext() handle dict type
for the 'vars' parameter.

See: d67f0fd4cc
Fixes: https://github.com/ansible/ansible/issues/20494

* add units/template/test_vars

* intg tests for jinja-2.9 issues like 20494

test cases here are based on
https://github.com/ansible/ansible/issues/20494#issue-202108318
2017-08-09 18:50:53 -04:00
Adrian Likins 4befefd78c Try to show original exception info for yaml (and other) errors (#24468)
* show original exception for yaml (and other) errors

In places where we need to catch a yaml error and raise
an AnsibleError, add the orig yaml exc to the AnsibleError
via the orig_exc arg.

When the AnsibleError is displayed it will now include the
AnsibleError (AnsibleParserError for example) and the type
and message from the original yaml exception.

This provides more detail to the error messages related to
yaml errors.

This also improves errors from dataloader (for example,
previously if a wrong password was used for a vault encrypted
yaml file, the error was very vague and suggested yaml errors,
but now the message includes the original exception from vault
indicating the password was incorrect or missing).

Add a text note to playbook helper asserts. For playbook
syntax/layout errors that aren't yaml errors, but errors
indicating invalid data structures for a playbook/task/role/block,
we now include some info about where the assert was and
why it was raised.

In places we raise an AnsibleParserError in an except
clause, pass the original exception to AnsibleParserError via
orig_exc arg.

Make assorted error messages a little more specific (like
the playbook helper load methods)

* Revert "Include the original YAML error in syntax error messages"

This reverts commit 781bb44b02.
2017-06-09 13:13:15 -04:00
Dag Wieers 5553b20828 Collated PEP8 fixes (#25293)
- Make PEP8 compliant
2017-06-02 12:14:11 +01:00
Sergey Bondarev f39fa7fd22 fix from https://github.com/ansible/ansible/pull/23001 2017-05-19 09:41:09 -04:00
Sergey Bondarev 72fc9f5cbe fix deprecation import ansible.vars.unsafe_proxy to ansible.utils.unsafe_proxy 2017-05-19 09:41:09 -04:00