Recent Python3 versions require open() to specify binary mode if the data is anything other than text.
Python3: Use int() instead of long() in unarchive
Changes long() to int() for CRC values in the unarchive module. Affects unarchiving of zip files. Since CRC values in zipfile are 32 bits the behaviour should be unchanged even in Python 2.
* synchronize: Convert cmd to list and fix handling of the copy_links argument
Converting cmd from str to list stops the pain of argument quoting/escaping.
* synchronize: Update imports according to #pullrequestreview-28758614
This change to the template action plugin make template use the
platform's native newline_sequence for Jinja.
We also added the option `newline_sequence` to change the newline
sequence using by Jinja if you need to use another newline sequence than
the platform default.
This was previously discussed in
https://github.com/ansible/ansible/issues/16255#issuecomment-278289414
And also relates to issue #21128
Changes to the metadata format were approved here:
https://github.com/ansible/proposals/issues/54
* Update documentation to the new metadata format
* Changes to metadata-tool to account for new metadata
* Add GPL license header
* Add upgrade subcommand to upgrade metadata version
* Change default metadata to the new format
* Fix exclusion of non-modules from the metadata report
* Fix ansible-doc for new module metadata
* Exclude metadata version from ansible-doc output
* Fix website docs generation for the new metadata
* Update metadata schema in valiate-modules test
* Update the metadata in all modules to the new version
* fix module doc fields
* More module docs corrections
* More module docs corrections
* More module docs corrections
* More module docs corrections
* correct aliases
* Review comments
* Must quote ':'
* More authors
* Use suboptions:
* restore type: bool
* type should be in the same place
* More tidyups
* authors
* Use suboptions
* revert
* remove duplicate author
* More issues post rebase
This is the original iso_extract module from the ansible-provisioning project.
We are upstreaming all our modules, and this PR is part of that process.
* Avoid having module documentation links to itself
A lot of modules use M(own_module) in their documentation causing a link
in the documentation to itself.
* Make note more clear now
* Reminder that lookup(‘file’) can be used
Sometimes the block of text does not easily fit into a playbook, so this acts as a reminder (or a prompt for anyone who hasn't used this before) that the content could be stored in an separate file.
I've also included `backup: yes` as I think this is a good example of where a backup might be needed.
* Correct the not so "smart quotes"
* Update blockinfile.py
Replace – with an an ASCII -
* Update validate-modules
* Validates ANSIBLE_METADATA
* Ensures imports happen after documentation vars
* Some pep8 cleanup
* Clean up some left over unneeded code
* Update modules for new module guidelines and validate-modules checks
* Update imports for ec2_vpc_route_table and ec2_vpc_nat_gateway
The following playbook:
```yaml
- hosts: localhost
tasks:
- file:
path: /tmp/non-existing-foo-bar
state: absent
recurse: yes
```
causes this error:
```
fatal: [localhost]: FAILED! => {"changed": false, "failed": true, "msg": "recurse option requires state to be 'directory'", "path": "/tmp/non-existing-foo-bar", "state": "absent"}
```
The included fix ensures that when recurse is added, we no longer assume
it is a file, but accept that it is a directory.
when using "state: link", and particularly when using
"force: yes".
Symbolic link resolution can be expensive. In our case,
the symbolic links are legacy links to automounts, and
the "file" task was causing all of the legacy links to
be traversed and mounted on every host every time the
task executed, even when the links were correct and there
was nothing to do.
This change avoids the system calls that perform the
symbolic link resolution by taking advantage of the short
circuit behaviur of the boolean "and" operator. The code
behaviour is unchanged except that it no longer performs
unnecessary system calls.
As it turns out, this change is not sufficient to fully
solve the symbolic link resolution problem, as the "file"
module still performs a stat() at the end of execution to
provide the caller with information about the file.
However, this change is very simple, it will eliminate
unnecessary system calls in a number of use cases, and it
gets the "file" module closer to the desired end result.
Not all file-related modules consistently use "path" as the attribute to specify a single filename, some use "dest", others use "name". Most do have aliases for either "name" or "destfile".
This change makes "path" the default attribute for (single) file-related modules, but also adds "dest" and "name" as aliases, so that people can use a consistent way of attributing paths, but also to ensure backward compatibility with existing playbooks.
NOTE: The reason for changing this, is that it makes Ansible needlessly harder to use if you have to remember that e.g. the xattr module requires the name attribute, the lineinfile module requires a dest attribute, and the stat module requires a path attribute.
can be per run or per host, also aggregate or not
set_stats action plugin as reference implementation
added doc stub
display stats in calblack
made custom stats showing configurable
Sudoers is a great example to show how you can prevent shutting yourself
out. But SSHd is at least as important to avoid syntax errors causing a
lot of grieve. So I think it deserves a spot in this list :-)
- Adds the 'link' file_type for finding symbolic or hard links
- Use `os.lstat` instead of `os.stat` to prevent the following
of links when statting the file.
* Change example syntax on os_auth module
* Change example syntax on os_client_config module
* Change example syntax on os_image_facts module
* Change example syntax on os_networks_facts module
* Change example syntax on os_nova_flavor module
* Change example syntax on os_object module
* Change example syntax on os_server module
* Change example syntax on os_subnet_facts module
* Change example syntax on rax_files module
* Change example syntax on rax_files_objects module
* Change example syntax on mysql_db module
* Change example syntax on file module
* Change example syntax on uri module
* Change example syntax on cl_bond module
* Change example syntax on cl_bridge module
* Change example syntax on cl_img_install module
* Change example syntax on cl_interface module
* Change example syntax on cl_license module
* Change example syntax on cl_ports module
* Remove trailing colon
The new create option with the default value 'no' changes the
behavior from the previous Ansible releases. Change the default to
'yes' to create missing ini files by default.
Fixes: #5488
This fixes the behavior that the dest is directory,
when we set the "force: no" argument.
To be join the dest and the src's basename,
before checking the "force" argument.
In the description of the find module return value, the sample dict
has its key=value strings converted to key=value: None in the
web documentation. This commit updates the sample output to a 'real'
dict.
Minor additional edit in the description: "return list *of* files".
make format function 'format only'
added platform dependant info, when it is available
avoid rechecking same info
added comments to each info gathering section
(cherry picked from commit a79acf73d7eb79b76d808ff8a1d6c505dfd9ec82)
Since the module use re and os, we need to import them.
And rather than importing '*', we should limit to the
only object/function needed, so we can more easily refactor
later.
Since handler.files_in_archive is a list of files coming from
various executables output, that's a bytes list, and we use it
with dest who is a str. So we need to convert that to native
type.
Fixes#4063.
Tar does not use this parameter on extraction (-x) or diff (-d)(the
only two cases where it is passed in unarchive). It only uses it on
creation:
https://www.gnu.org/software/tar/manual/html_section/tar_33.html
Providing `unarchive` with a file mode of `0755` (octal) makes it pass
the argument `--mode 493` (493 = 0755 in decimal) to `tar`, which then
fails while verifying it (because it contains an invalid octal char
'9'). Not passing the parameter to tar solves the issue.
This means we will have to unarchive the complete archive if a single change is found.
Unfortunately we cannot fix this for `unzip`, the only hope is a pure-python reimplementation.
This fixes problems reported in the comments of #3810
* Ensure unicode characters in zip-compressed filenames work correctly
Another corner-case we are fixing hoping it doesn't break anything else.
This fixes:
- The correct encoding of unicode paths internally (so the filenames we scrape from the output and is returned by zipfile match)
- Disable LANG=C for the unzip command (because it breaks the unicode output, unlike on gtar)
* Fix for python3 and other suggestions from @abadger
* Python3 fixes to copy, file, and stat so that the connection integration tests can be run
* Forgot to audit the helper functions as well.
* Fix dest to refledt b_dest (found by @mattclay)
On python3, we want to use the surrogateescape error handler if
available for filesystem paths and the like. On python2, have to use
strict in these circumstances. Use the new error strategy for to_text,
to_bytes, and to_native that allows this.
Lineinfile deals heavily with Unic text files. Makes some sense to deal
with it all as byte strings. So there is a lot of work done here to
show that we're dealing with byte strings throughout.
* Improve the correct handling of gtar and unzip options
Add the option --show-transformed-names when extra_opts is being used
Ignore bogus warnings related to empty filenames
Properly quote _and_ escape filenames for unzip command
Rewrite gtar options and provide run_command with array, not string
This fixes#2480 and #4109.
* Make check-mode work for zip-files
Check-mode was disabled for zip-files since gtar did not support it.
This change enables check-mode support for zip-files, but does skip the task when used with gtar.
(Best of both worlds)
Also remove unused compress_mode variable.
This replaces PR #4401, the changes overlap somewhat so I merged them
* FreeBSD do not support --omit-header and --absolute-names
* The option for following symlink wth getfacl is different on FreeBSD
* ZFS on Freebsd use nfsv4 acls, who use a slightly different syntax
* FreeBSD do not have a --test flag, so always return 'True'
* FreeBSD do not have the --omit-headers options, so we have to filter by ourself
* Mark Freebsd as working for the acl module
When you try to remote unarchive files with the option copy=no the code always fail, as evidenced in issue #4202. That happens because the conditional to check "if remote_src=no or copy=yes" will always be true since the default value of them is remote_src=no and copy=yes.
My modification is only to change the condition from or to and, that way only if both the vars stay with the default value will be true, otherwise you can unarchive remote files.
A capital "S" appears when the the setuid or setgid bit are set but have no effect. Likewise, a capital "T" appears when the sticky bit is set but it has no effect.
* Revert PR #3575 since it causes problems related to exclude patterns
By using a different method for getting archive filelists, and extracting we introduced new problems related to excluding based on gtar patterns.
As a result files that would be excluded by gtar, would still be in the filelist. Implementing our own gtar compatible pattern exclusion mechanism is near to impossible (believe me, we looked at it...). The best way is to look at the original problem and deal with that, and ensure that extraction and filelists are done with the exact same tool and exact same options.
The solution is to decode the octal unicode representation in gtar's output back to unicode. Since gtar has no problem extracting these files in LANG=C, we simply has to compensate for it.
This reverts #3575 and fixes#11348.
* Implement codecs.escape_decode() instead of decode("string_escape") for python3
* A few more sanity checks for detecting unzip output that's not a file entry
Also note that there's a rounding error somewhere in the mtime
comparison code.
* Fix reference to sub-array
The example for delete=yes does not specify recursive although it is
required. In addition, the wording for the delete option is confusing
about from where files are really deleted. This should clarify that.
* Improve the unzip output scraping
Ensure we capture the complete file (also when it includes spaces).
Drop lines that do not conform (in length) to what we expect (e.g. header/footer).
This fixes#3813
* Fix how split() works
Fallback to unzip if zipfile fails and hope that unzip can deal with it
(sites have an easier time upgrading the unzip utility than all of
python).
https://bugs.python.org/issue3997Fixes#3560
* Detection of handler depends on the wrong handler failing to list the contents of the tarfile.
Use explicit compression types with the python tarfile library to
achieve that.
* bytearray isn't available in python2.4
* unarchive: use Python's tarfile module for tar listing
fixes https://github.com/ansible/ansible/issues/11348
Depending on the current active locale, `tar`'s file listing can end up
spitting backslash-escaped characters. Unfortunately, when that happens,
we end up with double-escaped backslashes, giving us a wrong path,
making our action fail.
We could try un-double-escaping our paths, but that would be complicated
and, I think, error-prone. The easiest way forward seemed to simply use
the `tarfile` module.
Why use it only for listing? Because the `unarchive` option also
supports the `extra_opts` option, and that supporting this would require
us to mimick `tar`'s interface.
For listing files, however, I don't think that the loss of `extra_opts`
support causes problems (well, I hope so).
* unarchive: re-add xz decompression support
Following previous change to use Python's `tarfile` module for tar file
listing, we lost `xz` decompression support. This commits re-add it by
adding a special case in `TarXzArchive` that pre-decompresses the source
file.
* WIP: Making unarchive idempotent
Currently unarchive is not idempotent and has many rough edges and bugs.
The current release is a workable improvement on many fronts:
- zip support is now idempotent (but gtar lacks check-mode)
- New option `exclude` to exclude specific paths/files
- New option `keep_newer` to exclude newer files on target
- New option `extra_opts` to influence unzip/gtar (like synchronize module)
The following items are still ongoing:
- Implement CRC32 support for .zip files
- Re-implement the zip support using native zipfile module
- Re-implement the gtar support using native tarfile/gzip/bz2 modules (lzma external)
- Implement check-mode (works in gzip, but fails using gtar)
- Implement diff-mode (discuss an appropriate output model, like synchronize module)
The re-implementation of unzip/gtar support using native python modules will not only simplify the codebase, additional functionality can be implemented correctly and identically, which is currently not possible. (Other archives could be implemented using native modules equally, incl. options)
* Assorted fixes to zip support (during quality checks)
- Support both rw---- and rwx--- permstr
- Better file type support (more qa needed)
- Symlink support
- Include fix from #3229
* Implement zip diff-mode (itemized change) and avoid changes permissions every time (!)
This commit implements:
- rsync-compatible itemized-change output in diff-mode (using zip)
- no longer changing permissions unconditionally (when idempotent)
* Small fixes to itemized change output
* Fixes to user/group ownership changes
- The implementation of user/group ownership is a bit more complex for idempotency
- We report when a ZIP file incorrectly tags a directory as a file/link
- We only offer diff output when there is a change
* Fix the handling of includes and excludes for unzip
* Remove test output from output (confuses easily)
* Logic and performance improvements to ownership handling, and umask fix
* Handle special files (type '?')
* Make exceptions compatible with python 2.4
* Implement CRC32 support
* Revert some unintended/unknown changes ?
* Taking over maintenance as offered by current maintainer
* Fix support for white-spaces in filenames
* Remove/rename incorrect regex
* Ensure that fat executables end up with execute permission
* Remove check_result from output when unchanged
* When unarchiving as a user, or when owner/group/mode is supplied --diff is insufficient
Only way to be sure is to check request with what is on disk (as we do for zip).
Leave this up to set_fs_attributes_if_different() instead of inducing a (false) change
* By default, don't send confusing check_results in verbose output
This fixes#74.
now uses atomic move to avoid data corruption
correclty cleans up temp files in every case
returns backup_file info if needed
validate validate before temp file gets created
backup AFTER validate
Without this change, a download failure may bail out with the message:
"Failure downloading http://foo/bar, 'NoneType' object has no attribute 'read'"
whereas with this fix, you'd get a proper error like:
"Failure downloading http://foo/bar, Request failed: <urlopen error [Errno 113] No route to host>"
or one of the many other possible download errors that can occur.
Using the difflist feature added in ansible/ansible@c337293 we can add
two diffs to the `diff` dict returned as JSON: A `before` and `after` pair of
changed file contents and the diff of the file attributes.
n.b.: the difflist handling from the above commit is logically broken.
PR will follow.
Example output:
TASK [change line and mode] ************************************************************
changed: [localhost]
--- before: /tmp/sshd_config (content)
+++ after: /tmp/sshd_config (content)
@@ -65,21 +65,21 @@
X11DisplayOffset 10
PrintMotd no
PrintLastLog yes
TCPKeepAlive yes
#UseLogin no
#MaxStartups 10:30:60
#Banner /etc/issue.net
# Allow client to pass locale environment variables
-AcceptEnv LANG LC_*
+AcceptEnv LANG LC_* GF_ENV_*
Subsystem sftp /usr/lib/openssh/sftp-server
# Set this to 'yes' to enable PAM authentication, account processing,
# and session processing. If this is enabled, PAM authentication will
# be allowed through the ChallengeResponseAuthentication and
# PasswordAuthentication. Depending on your PAM configuration,
# PAM authentication via ChallengeResponseAuthentication may bypass
# the setting of "PermitRootLogin without-password".
# If you just want the PAM account and session checks to run without
--- before: /tmp/sshd_config (file attributes)
+++ after: /tmp/sshd_config (file attributes)
@@ -1,3 +1,3 @@
{
- "mode": "0700"
+ "mode": "0644"
}
Dylan Silva
Matt Martz
Toshio Kuratomi
Erwin Lang
Andrea Tartaglia
Sloane Hertel
Matt Parkes
Ritesh Khadgaray
Hao Wang
Karl Bergström
Paul Montero
Brian Coca
Dag Wieers
Brian Coca
Jens Carl
John R Barker
Marius Gedminas
Benjamin Jolivot
Tim Rupp
Matt Clay
Evgeni Golov
Tim Bielawa
Tuomas-Matti Soikkeli
Craig Francis
Mark Mielke
HyungCheol Kim
Fabio Alessandro Locati
peay
sirkubax
Michael Herold
Carlos E. Garcia
Evan Kaufman
James Cammarata
Orion Poplawski
jctanner
David Wittman
Sam Doran
Gyorgy Szombathelyi
René Moser
zuiurs
bencomp
Timothy Appnel
Luca Berruti
Charles Zaffery
Michael Scherer
Andrew Gaffney
JesseEmond
nichivo
Brian Maddy
James Pharaoh
Gennady Trafimenkov
Michael Scherer
Chris Short
Lucas Costa Beyeler
peter.jang
David Chwalisz
Ryan S. Brown
tobixx
Lukas Grossar
Keisuke Kamada
neo
Matt Fischer
Brendan Jurd
Matthias Frey
Jan Fader
Marcos Diez
Veaceslav Mindru
Virgil Dupras
yarick123
Mengdi Gao
Bence Nagy
Bert Van Vreckem
Toshio Kuratomi
Jose Moreira
Guruprasad
Tobias Wolf