Fixes#4063.
Tar does not use this parameter on extraction (-x) or diff (-d)(the
only two cases where it is passed in unarchive). It only uses it on
creation:
https://www.gnu.org/software/tar/manual/html_section/tar_33.html
Providing `unarchive` with a file mode of `0755` (octal) makes it pass
the argument `--mode 493` (493 = 0755 in decimal) to `tar`, which then
fails while verifying it (because it contains an invalid octal char
'9'). Not passing the parameter to tar solves the issue.
This means we will have to unarchive the complete archive if a single change is found.
Unfortunately we cannot fix this for `unzip`, the only hope is a pure-python reimplementation.
This fixes problems reported in the comments of #3810
* Ensure unicode characters in zip-compressed filenames work correctly
Another corner-case we are fixing hoping it doesn't break anything else.
This fixes:
- The correct encoding of unicode paths internally (so the filenames we scrape from the output and is returned by zipfile match)
- Disable LANG=C for the unzip command (because it breaks the unicode output, unlike on gtar)
* Fix for python3 and other suggestions from @abadger
* Python3 fixes to copy, file, and stat so that the connection integration tests can be run
* Forgot to audit the helper functions as well.
* Fix dest to refledt b_dest (found by @mattclay)
On python3, we want to use the surrogateescape error handler if
available for filesystem paths and the like. On python2, have to use
strict in these circumstances. Use the new error strategy for to_text,
to_bytes, and to_native that allows this.
Lineinfile deals heavily with Unic text files. Makes some sense to deal
with it all as byte strings. So there is a lot of work done here to
show that we're dealing with byte strings throughout.
* Improve the correct handling of gtar and unzip options
Add the option --show-transformed-names when extra_opts is being used
Ignore bogus warnings related to empty filenames
Properly quote _and_ escape filenames for unzip command
Rewrite gtar options and provide run_command with array, not string
This fixes#2480 and #4109.
* Make check-mode work for zip-files
Check-mode was disabled for zip-files since gtar did not support it.
This change enables check-mode support for zip-files, but does skip the task when used with gtar.
(Best of both worlds)
Also remove unused compress_mode variable.
This replaces PR #4401, the changes overlap somewhat so I merged them
* FreeBSD do not support --omit-header and --absolute-names
* The option for following symlink wth getfacl is different on FreeBSD
* ZFS on Freebsd use nfsv4 acls, who use a slightly different syntax
* FreeBSD do not have a --test flag, so always return 'True'
* FreeBSD do not have the --omit-headers options, so we have to filter by ourself
* Mark Freebsd as working for the acl module
When you try to remote unarchive files with the option copy=no the code always fail, as evidenced in issue #4202. That happens because the conditional to check "if remote_src=no or copy=yes" will always be true since the default value of them is remote_src=no and copy=yes.
My modification is only to change the condition from or to and, that way only if both the vars stay with the default value will be true, otherwise you can unarchive remote files.
A capital "S" appears when the the setuid or setgid bit are set but have no effect. Likewise, a capital "T" appears when the sticky bit is set but it has no effect.
* Revert PR #3575 since it causes problems related to exclude patterns
By using a different method for getting archive filelists, and extracting we introduced new problems related to excluding based on gtar patterns.
As a result files that would be excluded by gtar, would still be in the filelist. Implementing our own gtar compatible pattern exclusion mechanism is near to impossible (believe me, we looked at it...). The best way is to look at the original problem and deal with that, and ensure that extraction and filelists are done with the exact same tool and exact same options.
The solution is to decode the octal unicode representation in gtar's output back to unicode. Since gtar has no problem extracting these files in LANG=C, we simply has to compensate for it.
This reverts #3575 and fixes#11348.
* Implement codecs.escape_decode() instead of decode("string_escape") for python3
* A few more sanity checks for detecting unzip output that's not a file entry
Also note that there's a rounding error somewhere in the mtime
comparison code.
* Fix reference to sub-array
The example for delete=yes does not specify recursive although it is
required. In addition, the wording for the delete option is confusing
about from where files are really deleted. This should clarify that.
* Improve the unzip output scraping
Ensure we capture the complete file (also when it includes spaces).
Drop lines that do not conform (in length) to what we expect (e.g. header/footer).
This fixes#3813
* Fix how split() works
Fallback to unzip if zipfile fails and hope that unzip can deal with it
(sites have an easier time upgrading the unzip utility than all of
python).
https://bugs.python.org/issue3997Fixes#3560
* Detection of handler depends on the wrong handler failing to list the contents of the tarfile.
Use explicit compression types with the python tarfile library to
achieve that.
* bytearray isn't available in python2.4
* unarchive: use Python's tarfile module for tar listing
fixes https://github.com/ansible/ansible/issues/11348
Depending on the current active locale, `tar`'s file listing can end up
spitting backslash-escaped characters. Unfortunately, when that happens,
we end up with double-escaped backslashes, giving us a wrong path,
making our action fail.
We could try un-double-escaping our paths, but that would be complicated
and, I think, error-prone. The easiest way forward seemed to simply use
the `tarfile` module.
Why use it only for listing? Because the `unarchive` option also
supports the `extra_opts` option, and that supporting this would require
us to mimick `tar`'s interface.
For listing files, however, I don't think that the loss of `extra_opts`
support causes problems (well, I hope so).
* unarchive: re-add xz decompression support
Following previous change to use Python's `tarfile` module for tar file
listing, we lost `xz` decompression support. This commits re-add it by
adding a special case in `TarXzArchive` that pre-decompresses the source
file.
* WIP: Making unarchive idempotent
Currently unarchive is not idempotent and has many rough edges and bugs.
The current release is a workable improvement on many fronts:
- zip support is now idempotent (but gtar lacks check-mode)
- New option `exclude` to exclude specific paths/files
- New option `keep_newer` to exclude newer files on target
- New option `extra_opts` to influence unzip/gtar (like synchronize module)
The following items are still ongoing:
- Implement CRC32 support for .zip files
- Re-implement the zip support using native zipfile module
- Re-implement the gtar support using native tarfile/gzip/bz2 modules (lzma external)
- Implement check-mode (works in gzip, but fails using gtar)
- Implement diff-mode (discuss an appropriate output model, like synchronize module)
The re-implementation of unzip/gtar support using native python modules will not only simplify the codebase, additional functionality can be implemented correctly and identically, which is currently not possible. (Other archives could be implemented using native modules equally, incl. options)
* Assorted fixes to zip support (during quality checks)
- Support both rw---- and rwx--- permstr
- Better file type support (more qa needed)
- Symlink support
- Include fix from #3229
* Implement zip diff-mode (itemized change) and avoid changes permissions every time (!)
This commit implements:
- rsync-compatible itemized-change output in diff-mode (using zip)
- no longer changing permissions unconditionally (when idempotent)
* Small fixes to itemized change output
* Fixes to user/group ownership changes
- The implementation of user/group ownership is a bit more complex for idempotency
- We report when a ZIP file incorrectly tags a directory as a file/link
- We only offer diff output when there is a change
* Fix the handling of includes and excludes for unzip
* Remove test output from output (confuses easily)
* Logic and performance improvements to ownership handling, and umask fix
* Handle special files (type '?')
* Make exceptions compatible with python 2.4
* Implement CRC32 support
* Revert some unintended/unknown changes ?
* Taking over maintenance as offered by current maintainer
* Fix support for white-spaces in filenames
* Remove/rename incorrect regex
* Ensure that fat executables end up with execute permission
* Remove check_result from output when unchanged
* When unarchiving as a user, or when owner/group/mode is supplied --diff is insufficient
Only way to be sure is to check request with what is on disk (as we do for zip).
Leave this up to set_fs_attributes_if_different() instead of inducing a (false) change
* By default, don't send confusing check_results in verbose output
This fixes#74.
now uses atomic move to avoid data corruption
correclty cleans up temp files in every case
returns backup_file info if needed
validate validate before temp file gets created
backup AFTER validate
Without this change, a download failure may bail out with the message:
"Failure downloading http://foo/bar, 'NoneType' object has no attribute 'read'"
whereas with this fix, you'd get a proper error like:
"Failure downloading http://foo/bar, Request failed: <urlopen error [Errno 113] No route to host>"
or one of the many other possible download errors that can occur.
Using the difflist feature added in ansible/ansible@c337293 we can add
two diffs to the `diff` dict returned as JSON: A `before` and `after` pair of
changed file contents and the diff of the file attributes.
n.b.: the difflist handling from the above commit is logically broken.
PR will follow.
Example output:
TASK [change line and mode] ************************************************************
changed: [localhost]
--- before: /tmp/sshd_config (content)
+++ after: /tmp/sshd_config (content)
@@ -65,21 +65,21 @@
X11DisplayOffset 10
PrintMotd no
PrintLastLog yes
TCPKeepAlive yes
#UseLogin no
#MaxStartups 10:30:60
#Banner /etc/issue.net
# Allow client to pass locale environment variables
-AcceptEnv LANG LC_*
+AcceptEnv LANG LC_* GF_ENV_*
Subsystem sftp /usr/lib/openssh/sftp-server
# Set this to 'yes' to enable PAM authentication, account processing,
# and session processing. If this is enabled, PAM authentication will
# be allowed through the ChallengeResponseAuthentication and
# PasswordAuthentication. Depending on your PAM configuration,
# PAM authentication via ChallengeResponseAuthentication may bypass
# the setting of "PermitRootLogin without-password".
# If you just want the PAM account and session checks to run without
--- before: /tmp/sshd_config (file attributes)
+++ after: /tmp/sshd_config (file attributes)
@@ -1,3 +1,3 @@
{
- "mode": "0700"
+ "mode": "0644"
}
Toshio Kuratomi
Matt Clay
Andrew Gaffney
JesseEmond
nichivo
Brian Maddy
Dag Wieers
Brian Coca
Evan Kaufman
James Pharaoh
Gennady Trafimenkov
Michael Scherer
Chris Short
Lucas Costa Beyeler
Sam Doran
peter.jang
David Chwalisz
Brian Coca
Ryan S. Brown
tobixx
James Cammarata
Lukas Grossar
Keisuke Kamada
neo
Matt Fischer
Brendan Jurd
Matthias Frey
Jan Fader
Marcos Diez
Michael Scherer
Veaceslav Mindru
Virgil Dupras
yarick123
Mengdi Gao
Bence Nagy
Matt Martz
Bert Van Vreckem
Toshio Kuratomi
Jose Moreira
Guruprasad
Tobias Wolf
Adam Fields
quoing
Ales Nosek
krdlab
Dylan Martin