Commit Graph

129 Commits (2c63f453be0d5d140bf19cbb62abce09befe6c69)

Author SHA1 Message Date
James Cammarata a94a48f85f Partial revert of 76f7ce55 2017-01-11 15:53:04 -06:00
James Cammarata d316068831 Additional fixes for security related to CVE-2016-9587 2017-01-11 15:53:04 -06:00
Computest bcceada5d9 Fixing another corner case for security related to CVE-2016-9587 2017-01-11 15:52:58 -06:00
James Cammarata a6fff93967 Fixing security bugs for CVE-2016-9587 2017-01-09 10:43:03 -06:00
James Cammarata 85bbce9d6b Wrap unhandled errors from lookups in an AnsibleError
This provides better error handling, and prevents errors like KeyError
from bubbliing up to code in odd places.

Fixes #17482
2016-12-21 11:48:18 -06:00
Brian Coca bd70397e24 always template when called from template (#18490)
* Have template action plugin call do_template

Avoids all the magic done for 'inline templating' for ansible plays.
renamed _do_template to do_template in templar to make externally accessible.
fixes #18192

* added backwards compat as per feedback
2016-11-15 15:16:46 -05:00
Toshio Kuratomi 4ed88512e4 Move uses of to_bytes, to_text, to_native to use the module_utils version (#17423)
We couldn't copy to_unicode, to_bytes, to_str into module_utils because
of licensing.  So once created it we had two sets of functions that did
the same things but had different implementations.  To remedy that, this
change removes the ansible.utils.unicode versions of those functions.
2016-09-06 22:54:17 -07:00
Brian Coca ff34f5548d Dynamic role include (#17401)
* dynamic role_include

* more fixes for dynamic include roles

* set play yfrom iterator when dynamic

* changes from jimi-c

* avoid modules that break ad hoc

TODO: should really be a config
2016-09-05 20:07:58 -04:00
Adrian Likins e396d5d508 Implement vault encrypted yaml variables. (#16274)
Make !vault-encrypted create a AnsibleVaultUnicode
yaml object that can be used as a regular string object.

This allows a playbook to include a encrypted vault
blob for the value of a yaml variable. A 'secret_password'
variable can have it's value encrypted instead of having
to vault encrypt an entire vars file.

Add __ENCRYPTED__ to the vault yaml types so
template.Template can treat it similar
to __UNSAFE__ flags.

vault.VaultLib api changes:
    - Split VaultLib.encrypt to encrypt and encrypt_bytestring

    - VaultLib.encrypt() previously accepted the plaintext data
      as either a byte string or a unicode string.
      Doing the right thing based on the input type would fail
      on py3 if given a arg of type 'bytes'. To simplify the
      API, vaultlib.encrypt() now assumes input plaintext is a
      py2 unicode or py3 str. It will encode to utf-8 then call
      the new encrypt_bytestring(). The new methods are less
      ambiguous.

    - moved VaultLib.is_encrypted logic to vault module scope
      and split to is_encrypted() and is_encrypted_file().

Add a test/unit/mock/yaml_helper.py
It has some helpers for testing parsing/yaml

Integration tests added as roles test_vault and test_vault_embedded
2016-08-23 20:03:11 -04:00
Colin Hoglund 5a2b34e159 allow spaces before the filter string (#16312) 2016-06-15 14:17:17 -04:00
Brian Coca 005dc84aa7 moved last utils.debug to display.debug 2016-04-08 16:00:36 -04:00
Brian Coca 0b2f631197 Merge pull request #14504 from bcoca/template_comments
template also when only comments present
2016-03-06 10:45:18 -05:00
Brian Coca c022a43a47 fixed str to to_str 2016-03-03 14:40:35 -05:00
Brian Coca 4cfd042839 more complete and informative errors
mostly templating issues now all return data templated and actual error
fixes #14781
2016-03-03 13:52:09 -05:00
Toshio Kuratomi 1dbbd2a80c Remove debugging and fix the place where bytes are being handed to StringIO 2016-03-02 09:31:06 -08:00
Toshio Kuratomi 093b3422ec more information 2016-03-02 08:58:15 -08:00
Toshio Kuratomi 5aef65edcd Testing whether this fixes jenkins 2016-03-02 07:47:32 -08:00
James Cammarata 3247390123 Conditionally import StringIO in template so we only use io.StringIO on py3 2016-03-02 09:34:43 -05:00
Toshio Kuratomi b70bf3b056 Use io.StringIO and io.BytesIO instead of StringIO.StringIO for compat with py3 2016-02-26 16:43:05 -08:00
Matt Martz 439a385215 Catch TypeError on join, and if caught just return a single item, or a list of many 2016-02-20 16:17:58 -06:00
Brian Coca 6102a4b9b4 template also when only comments present 2016-02-16 09:42:33 -05:00
Marc Poirier 0792983ec8 Display the string which is causing an exception to be raised.
In the ansible template module, when there is an error while
expanding a templated string, displaying the string causing the
exception is very useful.
2016-02-11 15:42:50 -05:00
Brian Coca dc15eb806e deprecate all bare variable conversions (not debug)
now deprecation message appears with variable name in all spots where this occurs
debug's var= option is excluded as this is only place where bare variables shold actually
be accepted.
2016-02-01 18:59:14 -05:00
Brian Coca d82d65ee7b keep string type filters as strings
now we don't try to convert types if using a filter that outputs a specifically formated string
made list of filters configurable
2015-12-08 12:52:20 -08:00
James Cammarata f926e81782 Re-implement lookup wantlist
Fixes #13285
2015-11-29 23:45:54 -05:00
Yannig Perré 0c360d17cb New parameter for template method.
We do not compute hash when we know that the result does not need to be cached (like with_items loop).
It also result in a small speed improvement.
2015-11-10 07:19:19 +01:00
James Cammarata 7e04947599 Be more selective about what variables we cache during templating
Fixes #13087
2015-11-09 14:07:48 -05:00
James Cammarata 5040abaaf2 Merge branch 'devel_cache_for_do_template_call' of https://github.com/Yannig/ansible into Yannig-devel_cache_for_do_template_call 2015-11-06 22:04:52 -05:00
James Cammarata 63c47fb271 Fixing up performance 2015-11-05 16:22:37 -05:00
Yannig Perre ccbdd6229a Use static vars when computing host vars known to be static (inventory_hostname, inventory_dir etc.). 2015-11-04 22:16:14 +01:00
Yannig Perré 30f827d92d Fix python3 test. 2015-11-03 11:52:09 +01:00
Yannig Perré 130139dc80 Fix unicode issue introduced by previous commit. 2015-11-02 21:32:49 +01:00
Yannig Perré 4a8d1703d4 New patch against hostvars.py. With this patch, Ansible run lose 50% of time.
Little rewrite of previous patch to use sha1 signature.
Use fail_on_undefined to compute sha1 signature.
2015-11-02 21:04:20 +01:00
Brian Coca 79bf66336b switched clean_data from hardcoded delimiters to actual currently set jinja2 environment delimiters 2015-11-02 09:00:56 -05:00
Yannig Perré 805f768dab Clear cache when updating template variables. 2015-11-02 07:57:44 +01:00
Yannig Perré 87a9485b2f Cache for _do_template call. May result in nice speed improvement (4-5 times faster). 2015-11-01 22:30:55 +01:00
Toshio Kuratomi baa309309d Bundle a new version of python-six for compatibility along with some code to make it easy for distributions to override the bunndled copy if they have a new enough version. 2015-10-16 08:21:28 -07:00
James Cammarata 45b803efb4 Properly use test_loader from SharedPluginLoaderObj 2015-10-14 17:53:28 -04:00
Marius Gedminas 9b39af8e9a Python 3: the StringIO module is gone
Fixes 17 failing tests on Python 3.4, restoring the 8 failures we had
last Friday.
2015-09-28 09:28:53 +03:00
James Cammarata cdc6c5208e Clean string data run through the template engine
Also strip UnsafeProxy off of low level srings and objects to ensure
they don't cause issues later down the road

Fixes #12513
2015-09-25 16:49:31 -04:00
Marius Gedminas 56f2a25bff Python 3: there's no 'unicode' 2015-09-24 12:53:48 +03:00
Toshio Kuratomi 46bf3d1bc1 Remove plugin._basedirs as we now use different data structures to achieve that functionality. 2015-09-10 10:41:31 -07:00
Marius Gedminas ba0cf42ce9 Python 3: use six.string_types instead of basestring
Reduces the number of failing tests from 35 to 17.
2015-09-09 09:25:23 +03:00
James Cammarata 30399edada Use UnsafeProxy for lookup results too
Also fixes a couple of bugs that popped up when using the proxy class
2015-09-08 12:19:39 -04:00
James Cammarata 87f75a50ad Use proxy class to mark variables unsafe for templating
Fixes #12191
2015-09-04 16:38:06 -04:00
Toshio Kuratomi e218bf8da9 Do not escape backslashes when processing a template file.
Fixes #12198
Fixes #12241
2015-09-03 17:45:17 -07:00
Toshio Kuratomi 1998eddf8d Keep newlines by default now.
In v1, a trailing newline was kept if the parameter was passed as key=value.  If
the parameter was passed as yaml dict the trailing newline was
discarded.  Since key-value and yaml dict were unified in v2 we have to
make a choice as to which behaviour we want.  Decided that keeping trailing
newlines by default made the most sense.

Fixes #12200
Fixes #12199
2015-09-02 11:55:45 -07:00
Toshio Kuratomi 41da8de094 Speedup for counting newlines 2015-09-02 10:59:51 -07:00
Toshio Kuratomi 417bf1c805 Unittest the _count_trailing_newlines function 2015-09-02 08:59:31 -07:00
Toshio Kuratomi 7ed746ad45 Fix preserve_trailing_newlines (broken by 7f5080f64a )
Fix for one half of hte bug reported in #12198
2015-09-02 08:47:20 -07:00
James Cammarata 3e1a774ba5 Don't use iteritems() in templar to avoid (hostvars) dict size change errors 2015-09-01 14:11:23 -04:00
Toshio Kuratomi 7f5080f64a Fix backslash escaping inside of jinja2 expressions
Fixes #11891
2015-08-31 13:17:26 -07:00
Marius Gedminas df1b41d3d3 Avoid types.NoneType
types.NoneType was removed in Python 3.

None is a singleton in Python, so 'x is None' is equivalent to
'isinstance(x, NoneType)'.
2015-08-27 22:15:56 +03:00
Marius Gedminas 0c6ce31f76 Use 'except ... as' syntax
This syntax works on Python 2.6 through 3.x.  lib/ansible/module_utils
(and lib/ansible/modules) need to support Python 2.4, so I didn't touch
those.
2015-08-27 22:15:04 +03:00
Feanil Patel 892e230514 Don't convert nulls to strings.
This change is similar to https://github.com/ansible/ansible/pull/10465

It extends the logic there to also support none types.  Right now if you have
a '!!null' in yaml, and that var gets passed around, it will get converted to
a string.

eg. defaults/main.yml
```
ENABLE_AWESOME_FEATURE: !!null # Yaml Null
OTHER_CONFIG:
  secret1: "so_secret"
  secret2: "even_more_secret"

CONFIG:
  hostname: "some_hostname"
  features:
    awesame_feature: "{{ ENABLE_AWESOME_FEATURE}}"
  secrets: "{{ OTHER_CONFIG }}"
```

If you output `CONFIG` to json or yaml, the feature flag would get represented in the output
as a string instead of as a null, but secrets would get represented as a dictionary.  This is
a mis-match in behaviour where some "types" are retained and others are not.  This change
should fix the issue.

I also updated the template test to test for this and made the changes to v2.

Added a changelog entry specifically for the change from empty string to null as the default.

Made the null representation configurable.

It still defaults to the python NoneType but can be overriden to be an emptystring by updating
the DEFAULT_NULL_REPRESENTATION config.
2015-08-19 18:35:07 -04:00
Brian Coca ee2e31b37a now passes the test of skipping list when dict attribute is undefined, added deprecation warning as this seems like bad behaviour 2015-08-16 10:16:02 -04:00
Brian Coca 2c34a5bdbd pass list correctly to lookup 2015-08-12 17:42:06 -04:00
Brian Coca 79a1bca010 avoid circular import 2015-08-11 17:20:48 -04:00
Brian Coca a83b8220ef added import back 2015-08-11 16:42:51 -04:00
Brian Coca 6e825e8c22 fixed listify for lookups, made sure convert_bare is only on with_ 2015-08-11 16:42:09 -04:00
James Cammarata f904acbd14 Fixing bug related to listify changes 2015-08-11 01:33:43 -04:00
Brian Coca 5c1d850fbb now fixed to actually work 2015-08-10 13:51:56 -04:00
Brian Coca ee365c881f fixed typo 2015-08-10 13:44:52 -04:00
Toshio Kuratomi d35b956900 listify lookup plugin terms when they're specified as "{{ lookup(terms) }}"
Before this, they were not listified there but they were listified when
specified like this:

with_lookup: terms
2015-08-10 09:07:37 -07:00
James Cammarata dbd755e0f4 Fix jinja2 template search pathing
Fixes #9933
2015-08-04 22:50:16 -04:00
James Cammarata 5266679964 Use templar all the way down
Fixes bugs related to creating Templar() objects on the fly, where
the shared loader objects (serialized to TaskExecutor) aren't used
so information loaded into plugin loaders after forking is lost.

Fixes #11815
2015-08-04 12:25:53 -04:00
Brian Coca 07e5e98cbf removed jinaj2 override restrictions 2015-07-30 16:53:39 -04:00
James Cammarata 3a50c08c01 Merge branch 'pluggable_jinja_tests' of https://github.com/quixoten/ansible into quixoten-pluggable_jinja_tests 2015-07-29 15:16:27 -04:00
James Cammarata e526743b4f Allowing `args: "{{some_var}}"` for task params again
This is unsafe and we debated re-adding it to the v2/2.0 codebase,
however it is a common-enough feature that we will simply mark it
as deprecated for now and remove it at some point in the future.

Fixes #11718
2015-07-24 10:33:12 -04:00
James Cammarata 206ef27268 Add flag to template() so data is not converted to a datastructure
Fixes #11641
2015-07-22 15:05:51 -04:00
James Cammarata b09f1f8e0b Account for filters in bare variable conversion
Fixes #11530
2015-07-22 00:08:55 -04:00
Pierre-Louis Bonicoli 720e184f88 implement jinja2 header overrides 2015-07-03 22:27:49 +02:00
James Cammarata 927072546b Fixing up some issues with plugin loading 2015-06-29 22:49:25 -04:00
James Cammarata 18a9eff11f Properly use local variables from templates including other templates
Fixes #6653
2015-06-18 16:10:21 -04:00
James Cammarata 270eb4274c Make sure we safe_eval booleans too
Fixes #5779
2015-06-18 11:12:46 -04:00
Brian Coca 4098e8283e several fixes to template
- now obeys global undefined var setting and allows override (mostly for with_ )
- moved environment instanciation to init instead of each template call
- removed hardcoded template token matching and now use actually configured tokens, now it won't break if someone changes default configs in ansible.cfg
- made reenetrant template calls now pass the same data it got, dictionary and lists were loosing existing and new params
- moved fail_on_undeinfed parameter to template call, as it should only realky be set to false on specific templates and not globally
- added overrides, which will allow template to implement jinja2 header override features
- added filter list to overrides to disallow possibly insecure ones, TODO: check if this is still needed as facts should not be templated anymore
- TODO: actually implement jinja2 header overrides
2015-06-11 00:21:53 -04:00
James Cammarata 4bc7703db3 Fixing some small bugs related to integration tests (v2) 2015-06-01 16:42:10 -05:00
James Cammarata 9aa8676bdd More template unit tests for v2 2015-05-17 01:06:02 -05:00
James Cammarata ce3ef7f4c1 Making the switch to v2 2015-05-03 21:47:26 -05:00