* Handle duplicate headers, and make it easier for users to use cookies, by providing a pre-built string
* Ensure proper cookie ordering, make key plural
* Add note about cookie sort order
* Add tests for duplicate headers and cookies_string
* Extend tests, normalize headers between py2 and py3
* Add some notes in test code
* Don't use AttributeError, use six.PY3. Use better names.
* Start of tests for ansible.module_utils.urls
* Start adding file for generic functions throughout urls
* Add tests for maybe_add_ssl_handler
* Remove commented out line
* Improve coverage of maybe_add_ssl_handler, test basic_auth_header
* Start tests for open_url
* pep8 and ignore urlopen in test_url_open.py tests
* Extend auth tests, add test for validate_certs=False
* Finish tests for open_url
* Add tests for fetch_url
* Add fetch_url tests to replace-urlopen ignore
* dummy instead of _
* Add BadStatusLine test
* Reorganize/rename tests
* Add tests for RedirectHandlerFactory
* Add POST test to confirm behavior is to convert to GET
* Update tests to handle recent changes to RedirectHandlerFactory
* Special test, just to confirm that aliasing http_error_308 to http_error_307 does not cause issues with urllib2 type redirects
* Required changes to support redirects on HTTP 307/308
This ensures HTTP 307 and 308 will redirect the request to the new
location without modification.
* Fix the unused newheaders reference
* Be more compliant
* Add integration tests for follow_redirects=all
* Improve other tests for new behaviour
* Make follow_redirects values more strict
This is required if we want to ensure that #36809 doesn't cause any
important behavioral changes.
This PR changes the uri module to support follow_redirects=urllib2
It also adds a better error message when the connection closes before
any data was returned.
* allow shells to have per host options, remote_tmp
added language to shell
removed module lang setting from general as plugins have it now
use get to avoid bad powershell plugin
more resilient tmp discovery, fall back to `pwd`
add shell to docs
fixed options for when frags are only options
added shell set ops in t_e and fixed option frags
normalize tmp dir usag4e
- pass tmpdir/tmp/temp options as env var to commands, making it default for tempfile
- adjusted ansiballz tmpdir
- default local tempfile usage to the configured local tmp
- set env temp in action
add options to powershell
shift temporary to internal envvar/params
ensure tempdir is set if we pass var
ensure basic and url use expected tempdir
ensure localhost uses local tmp
give /var/tmp priority, less perms issues
more consistent tempfile mgmt for ansiballz
made async_dir configurable
better action handling, allow for finally rm tmp
fixed tmp issue and no more tempdir in ballz
hostvarize world readable and admin users
always set shell tempdir
added comment to discourage use of exception/flow control
* Mostly revert expand_user as it's not quite working.
This was an additional feature anyhow.
Kept the use of pwd as a fallback but moved it to a second ssh
connection. This is not optimal but getting that to work in a single
ssh connection was part of the problem holding this up.
(cherry picked from commit 395b714120522f15e4c90a346f5e8e8d79213aca)
* fixed script and other action plugins
ensure tmpdir deletion
allow for connections that don't support new options (legacy, 3rd party)
fixed tests
We do not go through the effort of finding the right PROTOCOL setting if
we have SSLContext in the stdlib. So we do not want to hit the code
that uses PROTOCOL to set the urllib3-provided ssl context when
SSLContext is available. Also, the urllib3 implementation appears to
have a bug in some recent versions. Preferring the stdlib version will
work around that for those with Python-2.7.9+ as well.
Fixes#26235Fixes#25402Fixes#31998
* module_utils.urls - Encode the proxy connect as binary
Under Python3 the sendall method expects binary not a string.
Prior to this change the below exception was being thrown;
Traceback (most recent call last):
File "/tmp/ansible_umxox7_x/ansible_modlib.zip/ansible/module_utils/urls.py", line 1044, in fetch_url
client_key=client_key, cookies=cookies)
File "/tmp/ansible_umxox7_x/ansible_modlib.zip/ansible/module_utils/urls.py", line 951, in open_url
r = urllib_request.urlopen(*urlopen_args)
File "/opt/blue-python/3.6/lib/python3.6/urllib/request.py", line 223, in urlopen
return opener.open(url, data, timeout)
File "/opt/blue-python/3.6/lib/python3.6/urllib/request.py", line 524, in open
req = meth(req)
File "/tmp/ansible_umxox7_x/ansible_modlib.zip/ansible/module_utils/urls.py", line 729, in http_request
s.sendall((self.CONNECT_COMMAND % (self.hostname, self.port)).decode())
AttributeError: 'str' object has no attribute 'decode'
Encoding the value is inline with the lines below (Proxy-Authorization etc) which are being sent as binary.
This patch adds cookie parsing to the fetch_url/open_url module_utils
method. The overall result will still contain the key `set_cookie`, however
an additional key (`cookies`) will also be present. This new field is a
dictionary of values. Overall, this should make looking for individual
cookies in the response much easier, as currently the `set_cookie` field
is an amalgamation of the returned set-cookie headers and can be somewhat
difficult to parse.
The HTTP User-Agent "ansible-httpget" is already kind of the default,
it being the default value provided by the `url_argument_spec` helper
method. Yet, it may not be practical for all modules to get their
argument_spec that way.
Without a default User-Agent we fall back on the library
User-Agent. That being something like "Python-urllib/2.7".
While I'm no big fan of web servers making decisions based on the
provided User-Agent I still think that part of being a well-behaved
HTTP client is to provide an informative User-Agent. Not to mention
that it's a good thing for Ansible to behave consistently.
Indirectly fixes#26239
socket.create_connection is a higher-level function, which tries to
establish a socket connection using both AF_INET and AF_INET6. It got
introduced in Python 2.6, which ought to be fine with Ansible 2.4.
Fixes#26740
Based on issue 23642, add some info about the used python
executable and version to the error message when ssl connection
fail in a way that may be related to the version.
* Build HTTPSClientAuthHandler more similarly to how HTTPSHandler works
* Add docs for new client cert authentication
* Support older versions of python
* Simplify logic
* Initial support for client certs in urls.py
* Add an extra test
* Add a get_url test for client cert auth
* Add additional test for client cert auth, with validation and ssl mismatch
* Skip assert when http tester not available
* Update version_added for new options
When get_url or other functions receive an SSL failure it prints a
standard message regardless of the failure. Include the actual OpenSSL
message in the error message so alternative failures can be debugged.
Closes: #20024
The `except` block with exception matching throught
`if 'connection refused' in str(e).lower():` is funny,
but is not user-friendly.
Probably related issues:
- #15679
- #12161
- #9966
- #8221
- #7218
... and more
os.write() needs bytes objects on python3 while python2 can work with
either a byte or unicode string. Mark the DUMMY_CA_CERT string as
a byte string so it will work.
Fixes#19265Fixes#19266
If there is an intermittent network failure, we might be trying to reach
an URL multiple times. Without this patch, we would be re-adding the same
certificate to the OpenSSL default context multiple times.
Normally, this is no big issue, as OpenSSL will just silently ignore them,
after registering the error in its own error stack.
However, when python-cryptography initializes, it verifies that the current
error stack of the default OpenSSL context is empty, which it no longer is
due to us adding the certificates multiple times.
This results in cryptography throwing an Unknown OpenSSL Error with details:
OpenSSLErrorWithText(code=185057381L, lib=11, func=124, reason=101,
reason_text='error:0B07C065:x509 certificate routines:X509_STORE_add_cert:cert already in hash table'),
Signed-off-by: Patrick Uiterwijk <puiterwijk@redhat.com>
Slightly better handling of http headers from http (CONNECT) proxy. Buffers up to 128KiB of headers and raises exception if this size is exceeded.
This could be optimized further, but for the time being it does the trick.
- When there is no file at the destination yet, we have no modification time for the `If-Modified-Since`-Header. In this case trust the cache to make the right decision to either serve a cached version or to refresh from origin. This should help with mass-deployment scenarios where you want to use a local cache to relieve your uplink.
- If you don't trust the cache to make the right decision you can still force it to refresh by providing the `force: yes` option.
* Use six instead of urllib2, for python 3 compat
* Open the certificate file using binary mode
On python3, os.write requires 'bytes'. Also avoid
using a too broad exception, since the issue was hard
to spot due to it.
* Do not add the header User-agent if not set
Python3 module do raise a exception if a header is
not a string-like object, and the default value is None.
* Port urls.py to python3
Fixes (largely normalizing byte vs text strings) for python3
* Rework what we do with attributes that aren't set already.
* Comments
* HTTPError can also function as a non-exceptional file-like return value (the same thing that urlopen() returns)
* HTTPError - adding response to info dictionnary
* HTTPError - adding response to info dictionnary
* HTTPError - adding body response to info dictionnary
* Ziploader proof of concept (jimi-c)
* Cleanups to proof of concept ziploader branch:
* python3 compatible base64 encoding
* zipfile compression (still need to enable toggling this off for
systems without zlib support in python)
* Allow non-wildcard imports (still need to make this recusrsive so that
we can have module_utils code that imports other module_utils code.)
* Better tracebacks: module filename is kept and module_utils directory
is kept so that tracebacks show the real filenames that the errors
appear in.
* Make sure we import modules that are used into the module_utils files that they are used in.
* Set ansible version in a more pythonic way for ziploader than we were doing in module replacer
* Make it possible to set the module compression as an inventory var
This may be necessary on systems where python has been compiled without
zlib compression.
* Refactoring of module_common code:
* module replacer only replaces values that make sense for that type of
file (example: don't attempt to replace python imports if we're in
a powershell module).
* Implement configurable shebang support for ziploader wrapper
* Implement client-side constants (for SELINUX_SPECIAL_FS and SYSLOG)
via environment variable.
* Remove strip_comments param as we're never going to use it (ruins line
numbering)
* Don't repeat ourselves about detecting REPLACER
* Add an easy way to debug
* Port test-module to the ziploader-aware modify_module()
* strip comments and blank lines from the wrapper so we send less over the wire.
* Comments cleanup
* Remember to output write the module line itself in powershell modules
* for line in lines strips the newlines so we have to add them back in
Matt Martz
Dag Wieers
Toshio Kuratomi
acrewdson
Brian Coca
nathaniel gentile
David Hain
Damian Zaremba
Matt Clay
James Cammarata
Andreas Olsson
Adrian Likins
Abhijeet Kasurde
jjlorenzo
Jamie Lennox
Andrew Grigorev
bart2
Patrick Uiterwijk
Foxlik
Michael Riss
Michael Scherer
Victor Volle
Toshio Kuratomi
Olivier GROSJEANNE