Fixes several bugs exposed in #34893
* Fixes relative path handling in copy so that it splits directories and
reconstructs the correct file path
* Return failed in the proper circumstances
* First bit of fixing temporary to have one source of truth
* Fix pep8
* Remove explicit make_tmp_path() in copy
The copy action plugin sets TRANSFER_FILES=True so it does not need to
set the temporary directory explicitly; the base class's run() method
will do that for us.
* Fix for calling a module's run when a shell has already created a temp path.
* Remember to inform the rest of the world when tempdir is removed
* New strategy for how to warn on passing tmp
Now we just warn when calling the parent class run() early. If the
module does a late call to the parent run() and doesn't make use of the
temporary directory, then we don't check for the possibility that the
user mistakenly is sending tmp in. If we truly deprecate this (rather
than ignoring it forever) then we might want to switch back to checking
for someone passing a value in as tmp.
* Remove tmp parameter from _execute_module as well
* Port all action plugins to not send tmp explicitly
This is now handled inside of _execute_module via the
_connection._shell.tempdir attribute.
Also update warnings and docs to tell people to set the attribute
instead of using _execute_module's tmp parameter.
* Always set local tempdir variable
* Add transfer checksum verification in copy module, to ensure that the file was transferred to the remote successfully. Fixes#35029
* Guard on no checksum
* Add version_added
* allow shells to have per host options, remote_tmp
added language to shell
removed module lang setting from general as plugins have it now
use get to avoid bad powershell plugin
more resilient tmp discovery, fall back to `pwd`
add shell to docs
fixed options for when frags are only options
added shell set ops in t_e and fixed option frags
normalize tmp dir usag4e
- pass tmpdir/tmp/temp options as env var to commands, making it default for tempfile
- adjusted ansiballz tmpdir
- default local tempfile usage to the configured local tmp
- set env temp in action
add options to powershell
shift temporary to internal envvar/params
ensure tempdir is set if we pass var
ensure basic and url use expected tempdir
ensure localhost uses local tmp
give /var/tmp priority, less perms issues
more consistent tempfile mgmt for ansiballz
made async_dir configurable
better action handling, allow for finally rm tmp
fixed tmp issue and no more tempdir in ballz
hostvarize world readable and admin users
always set shell tempdir
added comment to discourage use of exception/flow control
* Mostly revert expand_user as it's not quite working.
This was an additional feature anyhow.
Kept the use of pwd as a fallback but moved it to a second ssh
connection. This is not optimal but getting that to work in a single
ssh connection was part of the problem holding this up.
(cherry picked from commit 395b714120522f15e4c90a346f5e8e8d79213aca)
* fixed script and other action plugins
ensure tmpdir deletion
allow for connections that don't support new options (legacy, 3rd party)
fixed tests
This was causing wrong behaviour when `prev_state` was `hard`-link,
since the `file` module tried to apply the same `state` on the new
file, causing unexpected errors.
Particularly, both `overlay` and `devicemapper` storage drivers in
docker use hardlinks to share files between layers. This causes
most ansible playbooks to fail when working with files from layers
below.
* Fail if an empty string is set as src for copy module
Fixes#27363
* Cleanup task formatting on copy tests
Use multi-line YAML
Add debug statements with verbosity: 1 rather than leave them in there commented out.
* Add test for empty string as source
* Do more checks in order to add more specific errors messages
Add more integration tests for the various failure scenarios.
Cleanup some syntax on existing integration test tasks.
* Optimize template
* In fixing template to handle diff correctly #24477, I introduced more
round trips to the remote end which slowed things down The new code now
uses one fewer round trips than the old code.
* Reimplement a large part of template by calling the copy action plugin
instead of doing it in template's code. This reduces the code in
template and gives us one place to fix bugs and optimize.
* Add a follow parameter to template that mirrors the follow parameters
for file and copy.
* Fix copy's diff handling (probably broken in my rewrite for in 2.4
development)
* Adjusted when copy creates tmp dirs to rduce round trips in copy and
template.
Fixes#27956
Absolute path trailing slash handling in absolute directories
find_needle() isn't passing a trailing slash through verbatim. Since
copy uses that to determine if it should copy a directory or just the
files inside of it, we have to detect that and restore it after calling
find_needle()
Fixes#27439
* We need a directory walker that can handle symlinks, empty directories,
and some other odd needs. This commit contains a directory walker that
can do all that. The walker returns information about the files in the
directories that we can then use to implement different strategies for
copying the files to the remote machines.
* Add local_follow parameter to copy that follows local symlinks (follow
is for remote symlinks)
* Refactor the copying of files out of run into its own method
* Add new integration tests for copy
Fixes#24949Fixes#21513
Consolidate the module_utils, constants, and config functions that
convert values into booleans into a single function in module_utils.
Port code to use the module_utils.validate.convert_bool.boolean function
isntead of mk_boolean.
Copy module was walking over files in subdirectories repeatedly (a
directory tree a few levels deep could bring the time spent into the
tens of minutes)
This was traced to the fix for this bug report: https://github.com/ansible/ansible/issues/13013Fixed#13013 a different way and added an integration test to check for
regressions of #13013 as we optimize this code.
Fixes#21513
* win_copy: Add force parameter and check-mode support
The rationale behind this is that if you're working with +3GB files,
creating the checksum takes a lot of time, which we can avoid by simply
testing if the file exists.
I also took the liberty to put the various parameters together. It
probably takes a (neglible) performance hit but makes the code a bit
easier to inspect/work with, as its closer to all other windows modules.
On a normal run, the action plugin does a local checksum of the source
and a remote checksum of the destination. And afterwards, the module
will do another remote checksum of the copied source, a remote checksum
of the original destination, and another remote checksum of the copied
destination.
On a very huge file (think 4GB) that means 5x reading the complete file
(if you have a large cache you may get away with it, otherwise you're
doomed !).
This patch will ensure with `force: no` that not checksums are being
performed.
* Moving presence check before remote checksum
* Adapted to wishes
* Even more performance improvements
- centralized skipping
- also fixed module name broken by previous refactor
- let action modules handle async processing
- moved async into base action class's module exec
- action plugins can now run final action as async
- actually skip copy if base skips
- fixed normal for new paths
- ensure internal stat is never async
- default poll to 10 as per docs
- added hint for callback fix on poll
- restructured late tmp, now a pipeline query
- moving action handler to connection as networking does
- fixed network assumption invocation is always passed
- centralized key cleanup, normalized internal var
- _supress_tmpdir_delete now in _ansible_xxx and gets removed from results
- delay internal key removal till after we use em
- nicer tmp removing, using existing methods
- moved cleanup tmp flag to mking tmp func
* Use the local file's mode to for the argument if not explicitly given.
Fixes https://github.com/ansible/ansible-modules-core/issues/1124
* Fix octal mode for py3
* Implement preserve instead of null
* Remove duplicate line
* Update comment
* Use stat module per toshia's suggestion
We couldn't copy to_unicode, to_bytes, to_str into module_utils because
of licensing. So once created it we had two sets of functions that did
the same things but had different implementations. To remedy that, this
change removes the ansible.utils.unicode versions of those functions.
- Fix octal formatting of file mode in module response on py3.
- Convert file path to unicode in copy action.
- Enable file and copy module tests for py3 now that they pass.
This is enough to get minimal copy module working on python3
We have t omodify dataloader's path_dwim_relative_stack and everything
that calls it to use text paths instead of byte string paths
Run setfacl/chown/chmod on each temp dir and file.
This fixes temp file permissions handling on platforms such as FreeBSD
which always return success when using find -exec. This is done by
eliminating the use of find when setting up temp files and directories.
Additionally, tests that now pass on FreeBSD have been enabled for CI.
* smarter function to figure out relative paths
takes list of paths in order of relevance to current task
and does the dwim magic on them
* shared function for action plugins using new dwim
unify path construction and error info/messaging
made include and role non exclusive
corrected order and now smarter about tasks
includes inside roles are currently broken as they don't provide the correct role data
make dirname full match to avoid corner cases
* migrated action plugins to new dwim function
reported plugins to use exceptions instead of info
* clarified needle
Has already been transferred as a tempfile.
This fixes the error in https://github.com/ansible/ansible/issues/16125
but there may be higher level issues that should be fixed as well (other
modules might be able to cause status fields like failed and changed to
return a censored string instead of a bool). So leaving 16125 open for
now.
rm _del_ as it might leak memory
renamed to tmp file cleanup
added exception handling when traversing file list, even if one fails try rest
added cleanup to finally to ensure removal in most cases
- get_real_file will decrypt vault encrypted files and return a path to
a temporary file.
- cleanup_real_file will remove a temporary file created previously with
get_real_file
When checksums of local and remote files match, and when follow = True,
determine if remote destination is a symlink. If so, de-reference it and
pass the link target to the file module as 'dest'.
This change fixes an edge case in file copy behavior when:
- 'dest' is a symlink to some other file ('realdest')
- follow = True
- the checksums of the source file, 'src', and the symlink target, 'realdest',
match.
Because the checksums match, the copy module is skipped and the file module
is invoked directly with 'dest' = the symlink, and 'src' = the source of the
copy module, whether that source is present on the target machine or not.
When 'src' doesn't exist on the target machine, this leads to an error that
looks like this because it can't change the target of the symlink:
TASK [copy] ********************************************************************
fatal: [192.168.56.101]: FAILED! => {"changed": false, "checksum": "f572d396fae9206628714fb2ce00f72e94f2258f", "failed": true, "gid": 1000, "group": "ajdecon", "mode": "0777", "msg": "src file does not exist, use \"force=yes\" if you really want to create the link: /tmp/issue1568/dest_dir/source", "owner": "ajdecon", "path": "/tmp/issue1568/dest_dir/dest", "size": 8, "src": "source", "state": "link", "uid": 1000}
When the path 'src' *does* exist on the target machine, the file module makes
this the symlink "dest -> src" instead of "dest -> realdest"... even if the
checksum of 'src' on the target machine is different from the checksum of 'src'
on the machine where Ansible is running.
- added new function for action plugins this avoids the very fragile checksum code that is shell dependant.
- ported copy module to it
- converted assemble to new stat function
- some corrections and ported temlpate
- updated old checksum function to use new stat one under the hood
- documented revamped remote checksum method
Toshio Kuratomi
Matt Martz
Tomasz Kontusz
Jordan Borean
Brian Coca
Mehran Kholdi
Sam Doran
Dag Wieers
James Cammarata
Brian Coca
jctanner
Matt Clay
Brian Coca
Cambell
Toshio Kuratomi
Adam DeConinck