User module can contain Indentation errors or syntax errors.
Handle AST exceptions rather than showing traceback while importing such module.
Fixes: #21707
Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
* Added basic equivalent to PowerShell modules
* changes based on latest review
* Added tests
* ignore sanity test due to how tests are set up
* Changes to work with PSCore
* Added documentation and change updated more modules
* Add some speed optimisations to AddType
* fix some issues in the doc changes
* doc changes
When python is compiled in debug mode, or certain command line flags are
passed, python issues helpful warnings to let users know of files opened
by not closed.
This can be fairly spammy on an ansible run. This change reduces the
number of such warnings by a factor of 10.
* win_exec: refactor PS exec runner
* more changes for PSCore compatibility
* made some changes based on the recent review
* split up module exec scripts for smaller payload
* removed C# module support to focus on just error msg improvement
* cleaned up c# test classifier code
* win_script: add support for become and centralise exec wrapper builder
* satisfying the pep8 gods
* do not scan for module dependencies when running as a script
The AnsiballZ optimization which only uses one pyton interpreter
currently monkeypatches the arguments into a global argument in module_utils
so we need to always include basic.py. In the future we should fix this
so that it monkeypatches its own file. That way we won't need to always
include basic.py
Now that we don't need to worry about python-2.4 and 2.5, we can make
some improvements to the way AnsiballZ handles modules.
* Change AnsiballZ wrapper to use import to invoke the module
We need the module to think of itself as a script because it could be
coded as:
main()
or as:
if __name__ == '__main__':
main()
Or even as:
if __name__ == '__main__':
random_function_name()
A script will invoke all of those. Prior to this change, we invoked
a second Python interpreter on the module so that it really was
a script. However, this means that we have to run python twice (once
for the AnsiballZ wrapper and once for the module). This change makes
the module think that it is a script (because __name__ in the module ==
'__main__') but it's actually being invoked by us importing the module
code.
There's three ways we've come up to do this.
* The most elegant is to use zipimporter and tell the import mechanism
that the module being loaded is __main__:
* 5959f11c9d/lib/ansible/executor/module_common.py (L175)
* zipimporter is nice because we do not have to extract the module from
the zip file and save it to the disk when we do that. The import
machinery does it all for us.
* The drawback is that modules do not have a __file__ which points
to a real file when they do this. Modules could be using __file__
to for a variety of reasons, most of those probably have
replacements (the most common one is to find a writable directory
for temporary files. AnsibleModule.tmpdir should be used instead)
We can monkeypatch __file__ in fom AnsibleModule initialization
but that's kind of gross. There's no way I can see to do this
from the wrapper.
* Next, there's imp.load_module():
* https://github.com/abadger/ansible/blob/340edf7489/lib/ansible/executor/module_common.py#L151
* imp has the nice property of allowing us to set __name__ to
__main__ without changing the name of the file itself
* We also don't have to do anything special to set __file__ for
backwards compatibility (although the reason for that is the
drawback):
* Its drawback is that it requires the file to exist on disk so we
have to explicitly extract it from the zipfile and save it to
a temporary file
* The last choice is to use exec to execute the module:
* https://github.com/abadger/ansible/blob/f47a4ccc76/lib/ansible/executor/module_common.py#L175
* The code we would have to maintain for this looks pretty clean.
In the wrapper we create a ModuleType, set __file__ on it, read
the module's contents in from the zip file and then exec it.
* Drawbacks: We still have to explicitly extract the file's contents
from the zip archive instead of letting python's import mechanism
handle it.
* Exec also has hidden performance issues and breaks certain
assumptions that modules could be making about their own code:
http://lucumr.pocoo.org/2011/2/1/exec-in-python/
Our plan is to use imp.load_module() for now, deprecate the use of
__file__ in modules, and switch to zipimport once the deprecation
period for __file__ is over (without monkeypatching a fake __file__ in
via AnsibleModule).
* Rename the name of the AnsiBallZ wrapped module
This makes it obvious that the wrapped module isn't the module file that
we distribute. It's part of trying to mitigate the fact that the module
is now named __main)).py in tracebacks.
* Shield all wrapper symbols inside of a function
With the new import code, all symbols in the wrapper become visible in
the module. To mitigate the chance of collisions, move most symbols
into a toplevel function. The only symbols left in the global namespace
are now _ANSIBALLZ_WRAPPER and _ansiballz_main.
revised porting guide entry
Integrate code coverage collection into AnsiballZ.
ci_coverage
ci_complete
* tolerate windows line endings when loading windows module utils. Helpful for old custom windows modules.
* add test modules to demonstrate win line ending module load behaviour.
* attempt to fix sanity check failures
* pep8 fix
* explict skip of test modules from shebang check (core modules must still have expected unix style line endings)
* switch to rstrip() following core team meeting feedback
* module_common: set required parameter templar
Fix the following error (related to b455901):
$ ./hacking/test-module -m ./lib/ansible/modules/system/ping.py -I ansible_python_interpreter=/usr/bin/python
Traceback (most recent call last):
File "./hacking/test-module", line 268, in <module>
main()
File "./hacking/test-module", line 249, in main
(modfile, modname, module_style) = boilerplate_module(options.module_path, options.module_args, interpreters, options.check, options.filename)
File "./hacking/test-module", line 152, in boilerplate_module
task_vars=task_vars
File "ansible/lib/ansible/executor/module_common.py", line 910, in modify_module
environment=environment)
File "ansible/lib/ansible/executor/module_common.py", line 736, in _find_module_utils
shebang, interpreter = _get_shebang(u'/usr/bin/python', task_vars, templar)
File "ansible/lib/ansible/executor/module_common.py", line 452, in _get_shebang
interpreter = templar.template(task_vars[interpreter_config].strip())
AttributeError: 'NoneType' object has no attribute 'template'
* module_common.modify_module: templar is required
* Fix types when evaluating interpreter. Fixes#36536
* Rename variables that contain bytes to b_*
* Get rid of to_text() and to_bytes() calls that do nothing (because the
data is already the proper type)
* windows: add #AnsibleRequires to set whether a module requires module or a specific version
* fix up pep8 issues
* changed psversion to use the actual ps Requires -Version syntax
* missed the check on #Requires -Version
* fix #Requires module extensions
* Ansible Config part2
- made dump_me nicer, added note this is not prod
- moved internal key removal function to vars
- carry tracebacks in errors we can now show tracebacks for plugins on vvv
- show inventory plugin tracebacks on vvv
- minor fixes to cg groups plugin
- draft config from plugin docs
- made search path warning 'saner' (top level dirs only)
- correctly display config entries and others
- removed unneeded code
- commented out some conn plugin specific from base.yml
- also deprecated sudo/su
- updated ssh conn docs
- shared get option method for connection plugins
- note about needing eval for defaults
- tailored yaml ext
- updated strategy entry
- for connection pliugins, options load on plugin load
- allow for long types in definitions
- better display in ansible-doc
- cleaned up/updated source docs and base.yml
- added many descriptions
- deprecated include toggles as include is
- draft backwards compat get_config
- fixes to ansible-config, added --only-changed
- some code reoorg
- small license headers
- show default in doc type
- pushed module utils details to 5vs
- work w/o config file
- PEPE ATE!
- moved loader to it's own file
- fixed rhn_register test
- fixed boto requirement in make tests
- I ate Pepe
- fixed dynamic eval of defaults
- better doc code
skip ipaddr filter tests when missing netaddr
removed devnull string from config
better becoem resolution
* killed extra space with extreeme prejudice
cause its an affront against all that is holy that 2 spaces touch each other!
shippable timing out on some images, but merging as it passes most
The AnsiBallZ wrapper is transferred to the remote machine with
a filename similar to the Ansible-module it runs. For modules like copy
and tempfile, this can end up conflicting with stdlib modules on the
remote machine depending on how python is setup there. We have a little
bit of code in the wrapper to deal with this by removing the path that
the ansible module resides in from sys.path.
On MacOSX, that code was having a problem. The path the module ends up
in included a symlinked directory so we were looking for a path in
sys.path but we had to look for the unsymlinked path instead.
Fix that by using os.path.realpath() instead of os.path.abspath()
If the temp directory creation failed in mkdtemp then temp_path is never
given a value. This would lead to a NameError exception which would
obfuscate the original error (out of disk space being a common one). By
catching NameError, python will raise the original exception as we want.
Fixes#17215
* fixes#22441
* fixes#22655
* moves all env handling into the exec wrapper; this should work for everything but raw, which is consistent with non-Windows.
* Update module_utils.six to latest
We've been held back on the version of six we could use on the module
side to 1.4.x because of python-2.4 compatibility. Now that our minimum
is Python-2.6, we can update to the latest version of six in
module_utils and get rid of the second copy in lib/ansible/compat.
* Make the module_utils path configurable
* Add a config value to define the path site module_utils files
* Handle module_utils that do not have source as an error
* Make an integration test for module_utils envvar working
* Add documentation for the ANSIBLE_MODULE_UTILS config option/envvar
* Add it to the sample ansible.cfg
* Add it to intro_configuration.
* Also modify intro_configuration to place envvars on equal footing with
the config options (will need to document the envvar names in the
future)
* Also add the ANSIBLE_LIBRARY use case from
https://github.com/ansible/ansible/issues/15432 so we can close out
that bug.
This version just gets the relevant paths from PluginLoader and then
uses the existing imp.find_plugin() calls in the AnsiballZ code to load
the proper module_utils.
Modify PluginLoader to optionally omit subdirectories (module_utils
needs to operate on top level dirs, not on subdirs because it has
a hierarchical namespace whereas all other plugins use a flat
namespace).
Rename snippet* variables to module_utils*
Add a small number of unittests for recursive_finder
Add a larger number of integration tests to demonstrate that
module_utils is working.
Whitelist module-style shebang in test target library dirs
Prefix module_data variable with b_ to be clear that it holds bytes data
On Ubuntu the scriptdir gets placed into sys.path. This makes some
modules (copy) fail because the ansible module gets loaded instead of
the stdlib copy module. So we remove scriptdir there. Unfortunately,
the scriptdir code uses abspath(). When pipelining, abspath() has to
find the cwd. On OSX, finding the cwd when that directory is not
executable by the user raises an OSError. Since OSX does not suffer
from the scriptdir problem we're able to just skip scriptdir handling if
we get that exception.
Fixes#19729
Some machines have system clocks which can fall behind (for instance,
a host without a CMOS battery like Raspberry Pi). When managing those
machines we have to workaround the fact that the zip format does not
handle file timestamps before 1980. The workaround is to substitute in
the timestamp from the controller instead of from the managed machine.
Fixes#18640
While doing evil things with action plugins, I hit a code path in which
the mkdir here was failing due to lack of parent dir. Changing this to
makedirs made everything happy. Now, I'd obviously like to understand
why the parent dir exists in some places and not others - but I could
not find anywhere that C.DEFAULT_LOCAL_TMP is ensured to be created.
We couldn't copy to_unicode, to_bytes, to_str into module_utils because
of licensing. So once created it we had two sets of functions that did
the same things but had different implementations. To remedy that, this
change removes the ansible.utils.unicode versions of those functions.
I'm not sure why that would be desirable -- we really want __version__
to come from the controller whereas importing will come from the client
node. If it turns out there was a reason to do that, please be sure to
use an exception handler that catches all exceptions instead of only
catching ImportError:
```
try:
from ansible.release import __version__, __author__
except:
__version__ = [...]
```
Fixes#16523
Since Ansiballz, we no longer need to import basic directly into
a new-style module. Some modules, like the Networking modules, may
import basic in their own module_utils files and the module will import
that specialized module_util file rather than basic.
Matt Clay
Abhijeet Kasurde
Matt Martz
Jordan Borean
Eitan Adler
Toshio Kuratomi
jhawkesworth
Pilou
Brian Coca
Matt Davis
Dag Wieers
Adrian Likins
Monty Taylor
James Cammarata
Lukas Pirl
nitzmahone