* We need a directory walker that can handle symlinks, empty directories,
and some other odd needs. This commit contains a directory walker that
can do all that. The walker returns information about the files in the
directories that we can then use to implement different strategies for
copying the files to the remote machines.
* Add local_follow parameter to copy that follows local symlinks (follow
is for remote symlinks)
* Refactor the copying of files out of run into its own method
* Add new integration tests for copy
Fixes#24949Fixes#21513
Consolidate the module_utils, constants, and config functions that
convert values into booleans into a single function in module_utils.
Port code to use the module_utils.validate.convert_bool.boolean function
isntead of mk_boolean.
Copy module was walking over files in subdirectories repeatedly (a
directory tree a few levels deep could bring the time spent into the
tens of minutes)
This was traced to the fix for this bug report: https://github.com/ansible/ansible/issues/13013Fixed#13013 a different way and added an integration test to check for
regressions of #13013 as we optimize this code.
Fixes#21513
* win_copy: Add force parameter and check-mode support
The rationale behind this is that if you're working with +3GB files,
creating the checksum takes a lot of time, which we can avoid by simply
testing if the file exists.
I also took the liberty to put the various parameters together. It
probably takes a (neglible) performance hit but makes the code a bit
easier to inspect/work with, as its closer to all other windows modules.
On a normal run, the action plugin does a local checksum of the source
and a remote checksum of the destination. And afterwards, the module
will do another remote checksum of the copied source, a remote checksum
of the original destination, and another remote checksum of the copied
destination.
On a very huge file (think 4GB) that means 5x reading the complete file
(if you have a large cache you may get away with it, otherwise you're
doomed !).
This patch will ensure with `force: no` that not checksums are being
performed.
* Moving presence check before remote checksum
* Adapted to wishes
* Even more performance improvements
- centralized skipping
- also fixed module name broken by previous refactor
- let action modules handle async processing
- moved async into base action class's module exec
- action plugins can now run final action as async
- actually skip copy if base skips
- fixed normal for new paths
- ensure internal stat is never async
- default poll to 10 as per docs
- added hint for callback fix on poll
- restructured late tmp, now a pipeline query
- moving action handler to connection as networking does
- fixed network assumption invocation is always passed
- centralized key cleanup, normalized internal var
- _supress_tmpdir_delete now in _ansible_xxx and gets removed from results
- delay internal key removal till after we use em
- nicer tmp removing, using existing methods
- moved cleanup tmp flag to mking tmp func
* Use the local file's mode to for the argument if not explicitly given.
Fixes https://github.com/ansible/ansible-modules-core/issues/1124
* Fix octal mode for py3
* Implement preserve instead of null
* Remove duplicate line
* Update comment
* Use stat module per toshia's suggestion
We couldn't copy to_unicode, to_bytes, to_str into module_utils because
of licensing. So once created it we had two sets of functions that did
the same things but had different implementations. To remedy that, this
change removes the ansible.utils.unicode versions of those functions.
- Fix octal formatting of file mode in module response on py3.
- Convert file path to unicode in copy action.
- Enable file and copy module tests for py3 now that they pass.
This is enough to get minimal copy module working on python3
We have t omodify dataloader's path_dwim_relative_stack and everything
that calls it to use text paths instead of byte string paths
Run setfacl/chown/chmod on each temp dir and file.
This fixes temp file permissions handling on platforms such as FreeBSD
which always return success when using find -exec. This is done by
eliminating the use of find when setting up temp files and directories.
Additionally, tests that now pass on FreeBSD have been enabled for CI.
* smarter function to figure out relative paths
takes list of paths in order of relevance to current task
and does the dwim magic on them
* shared function for action plugins using new dwim
unify path construction and error info/messaging
made include and role non exclusive
corrected order and now smarter about tasks
includes inside roles are currently broken as they don't provide the correct role data
make dirname full match to avoid corner cases
* migrated action plugins to new dwim function
reported plugins to use exceptions instead of info
* clarified needle
Has already been transferred as a tempfile.
This fixes the error in https://github.com/ansible/ansible/issues/16125
but there may be higher level issues that should be fixed as well (other
modules might be able to cause status fields like failed and changed to
return a censored string instead of a bool). So leaving 16125 open for
now.
rm _del_ as it might leak memory
renamed to tmp file cleanup
added exception handling when traversing file list, even if one fails try rest
added cleanup to finally to ensure removal in most cases
- get_real_file will decrypt vault encrypted files and return a path to
a temporary file.
- cleanup_real_file will remove a temporary file created previously with
get_real_file
When checksums of local and remote files match, and when follow = True,
determine if remote destination is a symlink. If so, de-reference it and
pass the link target to the file module as 'dest'.
This change fixes an edge case in file copy behavior when:
- 'dest' is a symlink to some other file ('realdest')
- follow = True
- the checksums of the source file, 'src', and the symlink target, 'realdest',
match.
Because the checksums match, the copy module is skipped and the file module
is invoked directly with 'dest' = the symlink, and 'src' = the source of the
copy module, whether that source is present on the target machine or not.
When 'src' doesn't exist on the target machine, this leads to an error that
looks like this because it can't change the target of the symlink:
TASK [copy] ********************************************************************
fatal: [192.168.56.101]: FAILED! => {"changed": false, "checksum": "f572d396fae9206628714fb2ce00f72e94f2258f", "failed": true, "gid": 1000, "group": "ajdecon", "mode": "0777", "msg": "src file does not exist, use \"force=yes\" if you really want to create the link: /tmp/issue1568/dest_dir/source", "owner": "ajdecon", "path": "/tmp/issue1568/dest_dir/dest", "size": 8, "src": "source", "state": "link", "uid": 1000}
When the path 'src' *does* exist on the target machine, the file module makes
this the symlink "dest -> src" instead of "dest -> realdest"... even if the
checksum of 'src' on the target machine is different from the checksum of 'src'
on the machine where Ansible is running.
- added new function for action plugins this avoids the very fragile checksum code that is shell dependant.
- ported copy module to it
- converted assemble to new stat function
- some corrections and ported temlpate
- updated old checksum function to use new stat one under the hood
- documented revamped remote checksum method
* Fix the task_vars parameter to not default to a mutable type (dict)
* Implement invocation in the base class's run() method have each action
module call the run() method's implemention in the base class.
* Return values from the action plugins' run() method takes the return
value from the base class run() method into account so that invocation
makes its way to the output.
Fixes#12869
There doesn't appear to be anything that actually uses tmp_path in the
connection plugins so we don't need to pass that in to exec_command.
That change also means that we don't need to pass tmp_path around in
many places in the action plugins any more. there may be more cleanup
that can be done there as well (the action plugin's public run() method
takes tmp as a keyword arg but that may not be necessary).
As a sideeffect of this patch, some potential problems with chmod and
the patch, assemble, copy, and template modules has been fixed (those
modules called _remote_chmod() with the wrong order for their
parameters. Removing the tmp parameter fixed them.)
- Add support for inserting module args into PowerShell modules. Fixes#11661.
- Support Windows paths containing spaces. Applies changes from #10727 to v2. Fixes#9999. Should also fixansible/ansible-modules-core#944 and ansible/ansible-modules-core#1007.
- Change how execution policy is set for running remote scripts. Applies changes from #11092 to v2. Also fixesansible/ansible-modules-core#1776.
- Use codepage 65001 (UTF-8) for WinRM connection instead of default (CP437), convert command to UTF-8 and results from UTF-8. Replaces changes from #10024. Fixes#11198.
- Close WinRM connection when task completes.
- Use win_stat, win_file and win_copy modules instead of stat, file and copy when called from within other action plugins (only when using WinRM+PowerShell).
- Unquote Windows path arguments before passing to win_stat, win_file, win_copy and slurp modules (only when using WinRM/PowerShell).
- Check for win_ping module to determine if core modules are missing (only when using WinRM/PowerShell).
- Add stdout_lines to result from running low level commands (so stdout_lines is available when using raw/script).
- Update copy action plugin to use shell functions for joining paths and checking for trailing slash.
- Update fetch action plugin to unquote source path when using Windows paths.
- Add win_copy and win_template action plugins that inherit from copy and template.
- Support running .bat and .cmd scripts using default system encoding instead of UTF-8.
- Always send PowerShell commands as base64-encoded blobs to allow for running simple PowerShell commands via raw.
- Support running modules on Windows with interpreters other than PowerShell.
- Update integration tests to support above changes and test unicode fixes.
- Add test for win_user error from ansible/ansible-modules-core#1241 (fixed by ansible/ansible-modules-core#1774).
- Add test for additional win_stat output values (implemented by ansible/ansible-modules-core#1473).
- Add test for OS architecture and name from setup.ps1 (implemented by ansible/ansible-modules-core#1100).
All WinRM integration tests pass for me with these changes.
Toshio Kuratomi
Dag Wieers
James Cammarata
Brian Coca
Brian Coca
jctanner
Matt Clay
Brian Coca
Cambell
Toshio Kuratomi
Adam DeConinck
Chris Church
Mario de Frutos