Template lookup plugin now support variable_start_string and
variable_end_string, just like template module.
Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
Allow specifying the source and destination files' encodings in the template module
* Added output_encoding to the template module, default to utf-8
* Added documentation for the new variables
* Leveraged the encoding argument on to_text() and to_bytes() to keep the implementation as simple as possible
* Added integration tests with files in utf-8 and windows-1252 encodings, testing all combinations
* fix bad smell test by excluding windows-1252 files from the utf8 checks
* fix bad smell test by excluding valid files from the smart quote test
Under a non-utf-8 locale (for instance, LC_ALL=C), passing a non-ascii
filename to many APIs will traceback. Fix that by explicitly converting
to byte strings before passing to external APIs.
May fix#27262
* Fixes for mode=preserve
* Document mode=preserve for template and copy
* Make mode=preserve work with remote_src for copy
* Make mode=preserve work for template
* Integration tests for copy & template mode=preserve
Fixes#39279
* Changed mode option in win_copy to hidden option as it doesn't reflect copy mode
* template: Add integration tests for `lstrip_blocks'
Signed-off-by: Alex Tsitsimpis <alextsi@arrikto.com>
* template: Fix passing `trim_blocks' inline
Fix passing `trim_blocks' option to the template module as inline
argument. Previously passing the `trim_blocks' option inline instead of
using the YAML dictionary format resulted in it always being set to
`True', even if `trim_blocks=False' was used.
Signed-off-by: Alex Tsitsimpis <alextsi@arrikto.com>
* template: Add option to `lstrip_blocks'
Add option to set `lstrip_blocks' when using the template module to
render Jinja templates. The Jinja documentation suggests that
`trim_blocks' and `lstrip_blocks' is a great combination and the
template module already provides an option for `trim_blocks'.
Note that although `trim_blocks' in Ansible is enabled by default since
version 2.4, in order to avoid breaking things keep `lstrip_blocks'
disabled by default. Maybe in a future version it could be enabled by
default.
This seems to address issue #10725 in a more appropriate way than the
suggested.
Signed-off-by: Alex Tsitsimpis <alextsi@arrikto.com>
* template: Add integration tests for `trim_blocks'
Signed-off-by: Alex Tsitsimpis <alextsi@arrikto.com>
* template: Check Jinja2 support for `lstrip_blocks'
Since the `lstrip_blocks' option was added in Jinja2 version 2.7, raise
an exception when `lstrip_blocks' is set but Jinja2 does not support it.
Check support for `lstrip_blocks' option by checking `jinja2.defaults'
for `LSTRIP_BLOCKS' and do not use `jinja2.__version__' because the
latter is set to `unknown' in some cases, perhaps due to bug in
`pkg_resources' in Python 2.6.6.
Also update option description to state that Jinja2 version >=2.7 is
required.
Signed-off-by: Alex Tsitsimpis <alextsi@arrikto.com>
* First bit of fixing temporary to have one source of truth
* Fix pep8
* Remove explicit make_tmp_path() in copy
The copy action plugin sets TRANSFER_FILES=True so it does not need to
set the temporary directory explicitly; the base class's run() method
will do that for us.
* Fix for calling a module's run when a shell has already created a temp path.
* Remember to inform the rest of the world when tempdir is removed
* New strategy for how to warn on passing tmp
Now we just warn when calling the parent class run() early. If the
module does a late call to the parent run() and doesn't make use of the
temporary directory, then we don't check for the possibility that the
user mistakenly is sending tmp in. If we truly deprecate this (rather
than ignoring it forever) then we might want to switch back to checking
for someone passing a value in as tmp.
* Remove tmp parameter from _execute_module as well
* Port all action plugins to not send tmp explicitly
This is now handled inside of _execute_module via the
_connection._shell.tempdir attribute.
Also update warnings and docs to tell people to set the attribute
instead of using _execute_module's tmp parameter.
* Always set local tempdir variable
* allow shells to have per host options, remote_tmp
added language to shell
removed module lang setting from general as plugins have it now
use get to avoid bad powershell plugin
more resilient tmp discovery, fall back to `pwd`
add shell to docs
fixed options for when frags are only options
added shell set ops in t_e and fixed option frags
normalize tmp dir usag4e
- pass tmpdir/tmp/temp options as env var to commands, making it default for tempfile
- adjusted ansiballz tmpdir
- default local tempfile usage to the configured local tmp
- set env temp in action
add options to powershell
shift temporary to internal envvar/params
ensure tempdir is set if we pass var
ensure basic and url use expected tempdir
ensure localhost uses local tmp
give /var/tmp priority, less perms issues
more consistent tempfile mgmt for ansiballz
made async_dir configurable
better action handling, allow for finally rm tmp
fixed tmp issue and no more tempdir in ballz
hostvarize world readable and admin users
always set shell tempdir
added comment to discourage use of exception/flow control
* Mostly revert expand_user as it's not quite working.
This was an additional feature anyhow.
Kept the use of pwd as a fallback but moved it to a second ssh
connection. This is not optimal but getting that to work in a single
ssh connection was part of the problem holding this up.
(cherry picked from commit 395b714120522f15e4c90a346f5e8e8d79213aca)
* fixed script and other action plugins
ensure tmpdir deletion
allow for connections that don't support new options (legacy, 3rd party)
fixed tests
* Optimize template
* In fixing template to handle diff correctly #24477, I introduced more
round trips to the remote end which slowed things down The new code now
uses one fewer round trips than the old code.
* Reimplement a large part of template by calling the copy action plugin
instead of doing it in template's code. This reduces the code in
template and gives us one place to fix bugs and optimize.
* Add a follow parameter to template that mirrors the follow parameters
for file and copy.
* Fix copy's diff handling (probably broken in my rewrite for in 2.4
development)
* Adjusted when copy creates tmp dirs to rduce round trips in copy and
template.
Fixes#27956
Consolidate the module_utils, constants, and config functions that
convert values into booleans into a single function in module_utils.
Port code to use the module_utils.validate.convert_bool.boolean function
isntead of mk_boolean.
* Allow template files to be vaulted
* Make sure to import exceptions we need
* get_real_file can't take bytes, since it looks specifically for string_types
* Now that we aren't using open() we don't need b_source
* Expand playbooks_vault docs to include modules that support vaulted src files
* Add vaulted template test
Template can take a directory as the destination. When that's the case,
we need to diff between the source and the file inside of the directory.
That happened when the directory was specified with a trailing slash but
not when it was specified on its own. This change fixes that.
Fixes#24413
This change to the template action plugin make template use the
platform's native newline_sequence for Jinja.
We also added the option `newline_sequence` to change the newline
sequence using by Jinja if you need to use another newline sequence than
the platform default.
This was previously discussed in
https://github.com/ansible/ansible/issues/16255#issuecomment-278289414
And also relates to issue #21128
* Update module_utils.six to latest
We've been held back on the version of six we could use on the module
side to 1.4.x because of python-2.4 compatibility. Now that our minimum
is Python-2.6, we can update to the latest version of six in
module_utils and get rid of the second copy in lib/ansible/compat.
- centralized skipping
- also fixed module name broken by previous refactor
- let action modules handle async processing
- moved async into base action class's module exec
- action plugins can now run final action as async
- actually skip copy if base skips
- fixed normal for new paths
- ensure internal stat is never async
- default poll to 10 as per docs
- added hint for callback fix on poll
- restructured late tmp, now a pipeline query
- moving action handler to connection as networking does
- fixed network assumption invocation is always passed
- centralized key cleanup, normalized internal var
- _supress_tmpdir_delete now in _ansible_xxx and gets removed from results
- delay internal key removal till after we use em
- nicer tmp removing, using existing methods
- moved cleanup tmp flag to mking tmp func
This is a redesign in how plugins call _remote_checksum().
- _remote_stat() has been modified to report the real error as
AnsiblError
- Action plugin **unarchive** calls _remote_stat() directly instead of
_remote_checksum()
- Action plugin **unarchive** also handles the exceptions directly
- Ensure get_exception() returns native text
Two other action plugins, **template** and **fetch**, also do a remote checksum.
In **template** we already call _remote_stat(), just like we now do for
unarchive, in **fetch** we do call _remote_checksum() and we make the
exact same mistake as the unarchive plugin. So that one could use a
redesign as well.
This fixes#19494
Before:
```
[dag@moria ansible.testing]$ ansible-playbook -v test137.yml
Using /home/dag/home-made/ansible.testing/ansible.cfg as config file
PLAY [localhost]
******************************************************************************************************
TASK [unarchive]
******************************************************************************************************
fatal: [localhost]: FAILED! => {"changed": false, "failed": true, "msg":
"python isn't present on the system. Unable to compute checksum"}
PLAY RECAP
******************************************************************************************************
localhost : ok=0 changed=0 unreachable=0
failed=1
```
After:
```
[dag@moria ansible.testing]$ ansible-playbook -v test137.yml
Using /home/dag/home-made/ansible.testing/ansible.cfg as config file
PLAY [localhost]
*************************************************************************************************************
TASK [unarchive]
*************************************************************************************************************
fatal: [localhost]: FAILED! => {"changed": false, "failed": true, "msg":
"Failed to get information on remote file (/tmp/): sudo: unknown user:
foobar\nsudo: unable to initialize policy plugin\n"}
PLAY RECAP
*******************************************************************************************************************
localhost : ok=0 changed=0 unreachable=0
failed=1
```
* Have template action plugin call do_template
Avoids all the magic done for 'inline templating' for ansible plays.
renamed _do_template to do_template in templar to make externally accessible.
fixes#18192
* added backwards compat as per feedback
We couldn't copy to_unicode, to_bytes, to_str into module_utils because
of licensing. So once created it we had two sets of functions that did
the same things but had different implementations. To remedy that, this
change removes the ansible.utils.unicode versions of those functions.
This is enough to get minimal copy module working on python3
We have t omodify dataloader's path_dwim_relative_stack and everything
that calls it to use text paths instead of byte string paths
Run setfacl/chown/chmod on each temp dir and file.
This fixes temp file permissions handling on platforms such as FreeBSD
which always return success when using find -exec. This is done by
eliminating the use of find when setting up temp files and directories.
Additionally, tests that now pass on FreeBSD have been enabled for CI.
* smarter function to figure out relative paths
takes list of paths in order of relevance to current task
and does the dwim magic on them
* shared function for action plugins using new dwim
unify path construction and error info/messaging
made include and role non exclusive
corrected order and now smarter about tasks
includes inside roles are currently broken as they don't provide the correct role data
make dirname full match to avoid corner cases
* migrated action plugins to new dwim function
reported plugins to use exceptions instead of info
* clarified needle
- added new function for action plugins this avoids the very fragile checksum code that is shell dependant.
- ported copy module to it
- converted assemble to new stat function
- some corrections and ported temlpate
- updated old checksum function to use new stat one under the hood
- documented revamped remote checksum method
Abhijeet Kasurde
Julien Champseix
Brian Coca
Toshio Kuratomi
Alex Tsitsimpis
Jordan Borean
Matt Martz
Brian Coca
Dag Wieers
Matt Clay
James Cammarata
Brian Coca
Toshio Kuratomi
Rene Moser