Commit Graph

72 Commits (be6628cd3d965b8aa742611c500b0784827bd616)

Author SHA1 Message Date
Will Thames ef8c9798d3 include_role handlers bug fix (#26335)
* Ensure that include_role properly fires handlers

include_role needs to ensure that any handlers included
with the role are added to the _notified_handler and
_listening_handler lists of the TaskQueueManager, otherwise
it fails when trying to run the handler.

Additionally, the handler needs to be added to the
PlayIterator's `_uuid_cache` or it fails after running
the handler

Add more uuid debug statements - this code was hard
to debug with existing debug statements, so add more
uuid information at little additional output cost.

Fixes #18411

* Add tests for include_role handlers

Tests for #18411
2017-07-19 15:02:32 -05:00
Toshio Kuratomi e238ae999b Cyptography pr 20566 rebase (#25560)
Make pyca/cryptography the preferred backend for cryptographic needs (mainly vault) falling back to pycrypto

pyca/cryptography is already implicitly a dependency in many cases
through paramiko (2.0+) as well as the new openssl_publickey module,
which requires pyOpenSSL 16.0+. Additionally, pyca/cryptography is
an optional dep for better performance with vault already.

This commit leverages cryptography's padding, constant time comparisons,
and CBC/CTR modes to reduce the amount of code ansible needs to
maintain.

* Handle wrong password given for VaultAES format

* Do not display deprecation warning for cryptography on python-2.6

* Namespace all of the pycrypto imports and always import them

  Makes unittests better and the code less likely to get stupid mistakes
  (like using HMAC from cryptogrpahy when the one from pycrypto is needed)

* Add back in atfork since we need pycrypto to reinitialize its RNG just in case we're being used with old paramiko

* contrib/inventory/gce: Remove spurious require on pycrypto

(cherry picked from commit 9e16b9db275263b3ea8d1b124966fdebfc9ab271)

* Add cryptography to ec2_win_password module requirements
  * Fix python3 bug which would pass text strings to a function which
    requires byte strings.

* Attempt to add pycrypto version to setup deps

* Change hacking README for dual pycrypto/cryptography

* update dependencies for various CI scripts

* additional CI dockerfile/script updates

* add paramiko to the windows and sanity requirement set

  This is needed because ansible lists it as a requirement. Previously
  the missing dep wasn't enforced, but cryptography imports pkg_resources
  so you can't ignore a requirement any more

* Add integration test cases for old vault and for wrong passwords

* helper script for manual testing of pycrypto/cryptography

* Skip the pycrypto tests so that users without it installed can still run the unittests

* Run unittests for vault with both cryptography and pycrypto backend
2017-06-27 06:00:15 -07:00
Dag Wieers 630185cb20 ansible/executor: PEP8 compliancy (#24695)
- Make PEP8 compliant
2017-05-30 18:13:53 +01:00
Brian Coca 586fcae398 tolerate 'batch' systems that mess with stdin (#23596)
* tolerate 'batch' systems taht mess with stdin

fixes #23541

* have pause on windows tolerate devnull

* tuplie
2017-04-18 11:39:49 -04:00
James Cammarata 3d65482927 Stash post-validated fields of the task in the TaskResult
This allows us to have a snapshot of the fields, which we can restore
on the pre-fork side so as to avoid having to re-template fields.
2017-02-21 14:35:13 -06:00
James Cammarata dfb1c0647e Revert "Move queuing tasks to a background thread"
This reverts commit b71957d6e6.
2016-09-15 17:00:06 -05:00
Toshio Kuratomi 4ed88512e4 Move uses of to_bytes, to_text, to_native to use the module_utils version (#17423)
We couldn't copy to_unicode, to_bytes, to_str into module_utils because
of licensing.  So once created it we had two sets of functions that did
the same things but had different implementations.  To remedy that, this
change removes the ansible.utils.unicode versions of those functions.
2016-09-06 22:54:17 -07:00
James Cammarata b71957d6e6 Move queuing tasks to a background thread 2016-08-31 13:33:01 -05:00
nitzmahone a9322e8b8b py3-friendly multi-exception fix for missing fileno under debugger 2016-08-23 19:55:39 -07:00
nitzmahone dbf7df4439 fix pycharm debugger issue on missing fileno
Under pycharm debugger, stdin has no fileno, but throws AttributeError instead of ValueError. Account for both.
2016-08-23 17:00:14 -07:00
James Cammarata d2b3b2c03e Performance improvements 2016-08-08 15:58:46 -05:00
Abhijit Menon-Sen 950cc26aab Use loop_control.loop_var directly
6eefc11c converted task.loop_control into an object, but while the other
callers were updated to use .loop_var instead of .get('loop_var'), this
site was overlooked.

This can be reproduced by including with loop_control a file that does
set_fact; a simple regression test along these lines is included.
2016-07-01 07:03:42 +05:30
James Cammarata 1f18f5c92d Tweak the poll time to 0.005 to improve perf with large playbooks 2016-06-06 11:24:32 -05:00
Andrey Shestakov 295c7ce1c8 Fix sleep intervals (14219) 2016-05-31 09:33:58 +03:00
James Cammarata 6eefc11c39 Make the loop variable (item by default) settable per task
Required for include+with* tasks which may include files that also
have tasks containing a with* loop.

Fixes #12736
2016-04-19 01:42:40 -04:00
Toshio Kuratomi b4b058794e Correct the worker.run documentation 2016-04-12 08:01:07 -07:00
Brian Coca 005dc84aa7 moved last utils.debug to display.debug 2016-04-08 16:00:36 -04:00
James Cammarata 0f2b1244d2 Make sure VariableManager has a view of HostVars
Fixes #15261
2016-04-05 11:32:28 -04:00
James Cammarata 13f3cbaf3b Renaming per-item and retry callbacks 2016-03-18 15:17:43 -04:00
Brian Coca 66ea464ebd Merge pull request #14793 from xiaket/devel
remove main_q for simplicity.
2016-03-07 12:51:38 -05:00
Brian Coca 25e9b5788b add per item diff handling
fixes #14843
2016-03-07 09:49:18 -05:00
夏恺(Xia Kai) b33074b703 remove main_q for simplicity.
main_q is not used anywhere in the codebase.

It is created in TaskQueueManager._initialize_processes, bundled with rslt_q
into TaskQueueManger._workers, later unwrapped in StrategyBase but not used.
This queue is closed in TaskQueueManger._cleanup_processes.

Historically, it is passed as a init parameter into WorkerProcess,
introduced in 62d7956, but this behavior is changed in 120b9a7.

Signed-off-by: 夏恺(Xia Kai) <xiaket@gmail.com>
2016-03-04 07:58:18 +00:00
James Cammarata e02b98274b issue callbacks per item and retry fails
- now workers passes queue to task_executor so it can send back events per item and on retry attempt
- updated result class to pass along events to strategy
- base strategy updated to forward new events to callback
- callbacks now remove 'items' on final result but process them directly when invoked per item
- new callback method to deal with retry attempt messages (also now obeys nolog)
- updated tests to match new signature of task_executor

fixes #14558
fixes #14072
2016-02-23 15:07:06 -05:00
Brian Coca 5a1887cc76 correctly handle term signals
- adhoc now terminates gracefully
- avoid race condition on terminations by ignoring errors if
  worker might have been reaped between checking if active and termination call
- ansible-playbook now properly exits on sigint/term
- adhoc and playbook now give exceptions that we should not normally capture
  and rely on top level finally to reap children
- handle systemexit breaks in workers
- added debug to see at which frame we exit
partial fix for #14346
2016-02-08 09:51:16 -05:00
Toshio Kuratomi f4d68b8860 Transform tracebacks into unicode before printing
Fixes #14042
2016-01-25 19:18:59 -08:00
James Cammarata 120b9a7ac6 Changing the way workers are forked 2015-12-11 23:35:07 -05:00
James Cammarata 80db6bacc4 Make sure run_once tasks properly set variables for all active hosts
Fixes #13267
2015-11-30 11:27:05 -05:00
James Cammarata 25807f5404 Don't modify things we've put on the queue
Fixes #12937
2015-11-17 14:22:52 -05:00
James Cammarata 5cbeab5a3c Performance improvements for HostVars and some bugfixes 2015-11-16 17:15:52 -05:00
James Cammarata 2b25b7974c Remove unnecessary sleep from result process 2015-11-13 16:45:45 -05:00
James Cammarata 13750f68ee Reduce spinning while waiting on queues 2015-11-12 14:41:09 -05:00
James Cammarata efbc6054a4 Add variable compression option 2015-11-05 16:22:37 -05:00
James Cammarata 63c47fb271 Fixing up performance 2015-11-05 16:22:37 -05:00
Brian Coca f5b095f6c8 properly sanitize registered vars 2015-10-30 16:42:06 -04:00
Brian Coca b6c92dacb5 removed key striping as they are read up the chain 2015-10-30 14:03:37 -04:00
Brian Coca 8ce864db6f sanatize results 2015-10-24 15:23:12 -04:00
James Cammarata 86de1429e5 Cleaning up FIXMEs 2015-10-22 16:03:50 -04:00
James Cammarata 1fa975d81a Simplify group_by by removing BYPASS_HOST_LOOP from the action
Fixes #12825
2015-10-19 14:57:07 -04:00
Toshio Kuratomi a526eae543 Clean up a few more pyflakes warnings 2015-10-19 11:20:21 -07:00
Toshio Kuratomi baa309309d Bundle a new version of python-six for compatibility along with some code to make it easy for distributions to override the bunndled copy if they have a new enough version. 2015-10-16 08:21:28 -07:00
James Cammarata a3e913da62 Don't assume handlers exist in the same role (if any)
Fixes #12536
2015-09-29 14:52:43 -04:00
James Cammarata 31d5f88a1d Use the task loop to calculate multiple delegated hosts
Due to the way we're now calculating delegate_to, if that value is based
on a loop variable ('item') we need to calculate all of the possible
delegated_to variables for that loop.

Fixes #12499
2015-09-25 01:41:09 -04:00
James Cammarata bde5ed9672 Catch worker errors that may be subclassed on others that lead to incorrect exits 2015-09-17 14:46:53 -04:00
James Cammarata c2115c3715 Re-adding part of 926f127 which was actually fixing #12277 2015-09-11 09:45:27 -04:00
James Cammarata 083aef65ad Revert "set_fact should put facts in the fact_cache, not vars_cache"
This reverts commit 926f127245.
2015-09-11 09:45:27 -04:00
James Cammarata 926f127245 set_fact should put facts in the fact_cache, not vars_cache
Also fixes a template path lookup issue when using an include

Fixes #12277
2015-09-09 11:10:18 -04:00
Marius Gedminas 37be9539ff Python 3: use six.text_type instead of unicode
Replace 'unicode' with six.text_type, everywhere but in module_utils.
2015-09-04 08:40:10 +03:00
Marius Gedminas 823677b490 Replace .iteritems() with six.iteritems()
Replace .iteritems() with six.iteritems() everywhere except in
module_utils (because there's no 'six' on the remote host).  And except
in lib/ansible/galaxy/data/metadata_template.j2, because I'm not sure
six is available there.
2015-09-03 09:23:27 +03:00
Marius Gedminas 0c6ce31f76 Use 'except ... as' syntax
This syntax works on Python 2.6 through 3.x.  lib/ansible/module_utils
(and lib/ansible/modules) need to support Python 2.4, so I didn't touch
those.
2015-08-27 22:15:04 +03:00
Toshio Kuratomi 6e107d2f22 Comments pointing the way towards substituting cryptography for pycrypto 2015-08-24 15:50:37 -07:00