Commit Graph

49 Commits (138fc11ee0ec3402885068eaa6362a42d1a46e7c)

Author SHA1 Message Date
Sloane Hertel 7da565b3ae
parse botocore.endpoint logs into a list of AWS actions (#49312)
* Add an option to parse botocore.endpoint logs for the AWS actions performed during a task

Add a callback to consolidate all AWS actions used by modules

Added some documentation to the AWS guidelines

* Enable aws_resource_actions callback only for AWS tests

* Add script to help generate policies

* Set debug_botocore_endpoint_logs via environment variable for all AWS integration tests

Ensure AWS tests inherit environment

(also remove AWS CLI in aws_rds inventory tests and use the module)
2019-03-18 08:29:03 -05:00
Matt Clay 0a461380a3 Fix incorrect use of `is` for comparisons.
See https://bugs.python.org/issue34850 for details.
2019-02-12 16:33:13 -08:00
Jordan Borean a39c4ad464
Final round of moving modules to new import error msg (#51852)
* Final round of moving modules to new import error msg

* readd URL to jenkins install guide

* fix unit tests
2019-02-08 10:07:01 +10:00
Tad Merchant 71ef69d51f check elb type in compare_subnets for elbv2 (#50203)
* Add check for elb type to fix difference between NLB and ALB AZ return structure

Fixes #49558

* Fix spacing for pep8 (hopefully) compliancE

* use empty array default instead of loadbalancer type detection

hat tip briantist

* spacing is probably important.
2019-01-29 13:28:39 +10:00
Brian Lamar 73133d7708 [aws] ALB/NLB support SubnetMappings in AWS create LB call (#42060) 2018-10-04 16:53:32 -04:00
Brian Scholer fbac32c5d0 Fix support for SubnetMappings and EIPs in NLB (#42979)
* Fix support for SubnetMappings and EIPs in NLB

* Fix style failures
2018-10-04 16:31:16 -04:00
Michael Scherer 518f80c3fb [module_utils/aws/urls] remove assigning variable itself (#45612)
Flagged by lgtm.com
2018-09-13 17:42:04 -04:00
Jonathan 93cd050250 Added default value for when CloudFrontOriginAccessIdentityList is (#44984)
missing.
2018-09-02 12:17:25 +10:00
Sloane Hertel 113336d6f1 rds_instance module and tests (#43789)
* Add functions to retrieve the allowed and required parameters for boto3 client methods

* Add custom waiter for stopping an RDS DB instance

* Add rds_instance module

* Add rds_instance integration tests

* address requested changes from ryansb

* address requested changes from willthames

* address requested changes from dmsimard

* Fix final snapshots

Fix idempotence with already-deleting DB instances

Remove unused import from module_utils/aws/core.py

Consolidate function to get all boto3 client method parameters and the subset of required parameters

* Add some additional rds_instance integration tests

* Add some common functions to module_utils/aws/rds

* Move common code out of rds_instance

* Remove hardcoded engine choices and require the minimum boto3

* Document wait behavior

* Provide a list of valid engines in the error message if it is invalid

Add supported methods to whitelist

Remove AWSRetry around waiter

Wait for a less crazy amount of time

Remove unused variables

* Add a test for an invalid engine option

* pep8

* Missed adding a method to the whitelist

* Use retries

* Fix some little things

* Fix more things

* Improve error message

* Support creating cross-region read replicas

* Remove unused imports

* Add retry when getting RDS instance

* Soft-check required options so module fails properly when options are missing

* Fix mariadb parameter version

* Fix cross-region read_replica creation and tests

* fix modify tests

* Fix a modification test

* Fix typo

* Remove test for option_group_name that exists for this account but may not for others and added as a TODO to do properly
2018-08-31 12:17:02 +10:00
Sloane Hertel 79ecb4c41f
Add diff mode for ec2_group (#44533)
* Add (preview) diff mode support ec2_group

* Add diff mode to some ec2_group integration tests

* Remove unnecessary arguments and add comment to the module notes

* Add changelog
2018-08-23 19:43:18 -04:00
Ryan Brown 6d52afeed6 New module cloudformation_stack_set (#41669)
* [AWS] new module cloudformation_stack_set with integration tests
2018-08-20 14:38:14 -04:00
Aaron Haaf 8a56aa322e Implement signed_request for sigV4 requests (#42758)
* Implement signed_request for sigV4 requests

* Correct linting errors

* More linting changes. Correct import.

* Final linting fix for inline comments

* Correct import of urllib.parse

* Update copyright and shebang line

* Remove shebang

* Put boto3 requirement. Abtract out get_aws_key_pair for module consumption.

* Dummy out unused region variable.

* Handle Boto3 ImportError

* - implement get_aws_credentials_object with willthames suggestion
- Handle session_token

* Make quote style consistant

* Chop arugment line up

* Correct indent
2018-08-03 08:31:10 +10:00
mjmayer 4549962f85 Fix comparison of priority (#43329)
The existing rule priority comes from aws as a string. It is then
compared to the new rule priority, which is defined as an int. This change
casts the new rule priority as a string making the comparison work. The
reason to cast it as a string rather than an int is used because a priority
can also be set to 'default'. When trying to case 'default' as an int, it creates
an error.
2018-08-02 17:05:20 +10:00
Martin Krizek 00c88f6497 [aws] Add missing methods to AnsibleAWSModule (#42679) 2018-07-12 09:47:28 -04:00
Deiwin Sarjas 6412cbf84b aws_eks_cluster: Add wait functionality (#42259)
* aws_eks_cluster: Improve output documentation

This data is already returned by the module, it just wasn't documented. These
fields are required for accessing the created Kubernetes API with e.g. the
k8s_raw module.

* aws_eks_cluster: Add wait functionality

This enables further cluster configuration once it's created and active.

20 minutes was chosen as an arbitrary default, so that if it takes longer than
the documented "usually less than 10 minutes" it's still likely to succeed.

* Correct security group name in aws_eks tests

* Improve teardown of aws_eks tests

Fix minor teardown issues. The `pause` step is a placeholder until
a waiter for `state: absent`
2018-07-04 22:30:57 +10:00
Rob b87e1a023d [aws] add support for http2 to AWS ALB (#40372) 2018-07-02 14:31:56 -04:00
Sloane Hertel 40d2df0ef3 Add AWS boto3 error code exception function is_boto3_error_code (#41202)
* Add aws/core.py function to check for specific AWS error codes

* Use sys.exc_info to get exception object if it isn't passed in

* Allow catching exceptions with is_boto3_error_code

* Replace from_code with is_boto3_error_code

* Return a type that will never be raised to support stricter type comparisons in Python 3+

* Use is_boto3_error_code in aws_eks_cluster

* Add duplicate-except to ignores when using is_boto3_error_code

* Add is_boto3_error_code to module development guideline docs
2018-06-12 12:15:16 -04:00
Will Thames 46886f8249 Improve aws_s3 permission handling for non S3 (#38574)
* Test case for missing permissions

* Update aws_s3 module to latest standards

* Use AnsibleAWSModule
* Handle BotoCoreErrors properly
* Test for BotoCoreErrors
* Check for XNotImplemented exceptions (#38569)

* Don't prematurely fail if user does not have s3:GetObject permission

* Allow S3 drop-ins to ignore put_object_acl and put_bucket_acl
2018-06-07 13:13:10 -04:00
Ryan Brown 7f8654d586 Add minimum botocore and boto3 checking to AnsibleAWSModule (#41005)
* Add minimum botocore and boto3 checking to AnsibleAWSModule
2018-06-01 18:45:22 -04:00
Rob 1b45a755a2 [AWS] Add helpful failure message when using wrong lb module (#40433)
Remove the default empty dict parameter for tags.
2018-05-25 13:10:39 -04:00
Rob 858f0fc000 New module: AWS Network load balancer (#33808)
* New module - elb_network_lb

* Fix creating a load balancer without tags

* Linter

Fix purging tags

Remove extra imports

* add support for cross zone lb, doc update and fix tagging

* pep8 fixes

* Add integration tests for elb_network_lb module

* more pep8

* Remove non-applicable option for NLBs

* fix target protocol

* pep8
2018-05-24 16:38:34 -04:00
Ryan Brown 858a1b09bb EC2_group module refactor (formerly pr/37255) (#38678)
* Refactor ec2_group

Replace nested for loops with list comprehensions

Purge rules before adding new ones in case sg has maximum permitted rules

* Add check mode tests for ec2_group

* add tests

* Remove dead code

* Fix integration test assertions for old boto versions

* Add waiter for security group that is autocreated

* Add support for in-account group rules

* Add common util to get AWS account ID

Fixes #31383

* Fix protocol number and add separate tests for egress rule handling

* Return egress rule treatment to be backwards compatible

* Remove functions that were obsoleted by `Rule` namedtuple

* IP tests

* Move description updates to a function

* Fix string formatting missing index

* Add tests for auto-creation of the same group in quick succession

* Resolve use of brand-new group in a rule without a description

* Clean up duplicated get-security-group function

* Add reverse cleanup in case of dependency issues

* Add crossaccount ELB group support

* Deal with non-STS calls to account API

* Add filtering of owner IDs that match the current account
2018-05-24 11:53:21 -04:00
Matt Clay 8deced3e04
Fix shebangs and file modes and update tests. (#40563)
* Add execute bit sanity test and apply fixes.
* Add shebang test for `lib` dirs and apply fixes.
* Shebang and execute bit cleanup.
2018-05-22 14:25:36 -07:00
Rob b5cffe8ced [aws] Create classes for Application Load Balancer (#33769)
* Create classes for Application Load Balancer
* Add unsupported CI alias
* Add AWSRetry
* Add integration tests using the ALB
2018-05-04 16:22:00 -04:00
Sloane Hertel 923f676836
[ec2_vpc_vgw] [ec2_vpc_vpn] stabilize modules for PR 35983 (#38666)
* Stabilize ec2_vpc_vgw and ec2_vpc_vpn so tests for ec2_vpc_vpn_facts in PR 35983 can be run in CI

* Add updated placebo recordings

* ensure find_vgw uses the virtual gateway id if available

Add AWSRetry.jittered_backoff to attach_vpn_gateway to deal with errors when attaching a new VPC directly after detaching

Add integrations tests for ec2_vpc_vgw

* Sort VPN Gateways by ID
2018-05-03 14:19:19 -04:00
Sloane Hertel ec9c59f52b [ec2_vpc_net] Add retries to describe_vpc_attribute call (#39256)
* [ec2_vpc_net] Add retries to describe_vpc_attribute call

* Use new AnsibleAWSModule client-based waiters
2018-04-30 15:27:22 -04:00
Ryan Brown 86db62c0e9 [AWS] report boto3/botocore versions during `fail_json_aws` (#39298)
* [AWS] report boto3/botocore versions during `fail_json_aws`

When modules call `fail_json_aws` and boto3 is installed, gather the
boto3 and botocore versions so that any new AWS module issues will
include the user's boto3 installation info. This will make debugging
issues where features aren't available yet easier.

* PEP8

* Switch to `dict` rather than tuple returns
2018-04-27 10:05:41 +10:00
Ryan Brown 40b7a63d1a Add docs to `AnsibleAWSModule.client` retry settings (#38689) 2018-04-16 08:48:10 -04:00
Ed Costello 0d31d1cd24 [cloud]Add aws_ses_identity_policy module for managing SES sending policies (#36623)
* Add aws_ses_identity_policy module for managing SES sending policies

* Add option to AnsibleAWSModule for applying a retry decorator to all calls.

* Add per-callsite opt in to retry behaviours in AnsibleAWSModule

* Update aws_ses_identity_policy module to opt in to retries at all callsites.

* Add test for aws_ses_identity_policy module with inline policy.

* Remove implicit retrys on boto resources since they're not working yet.
2018-04-05 15:11:12 -04:00
Ryan Brown 1c7b9e66b4 AWS WAF module custom waiter (#37026)
Consolidate waiters to a single file

* Add waiter message with token ID
* Add waiter

Add waiter for WAF change tokens

Working waiter for waf_condition module

Add support for waiters to waf_rule

* WAF data model refactor

* Fix ref to self.client

* Add custom waiters to aws_waf_web_acl

* Allow add/remove rule tasks to operate in parallel, then wait for their change tokens to complete

* Move waiter into run_func_with_change_token_backoff since it is generic to all WAF update operations

* Wait for deletes on waf_web_acl

* Remove always-wait

* Remove waiter retry catch
2018-04-05 11:30:57 +10:00
Sloane Hertel 873a9ddf8d [cloud] Add custom waiters to stabilize ec2_vpc_subnet module - Fixes #36083 (#37534)
* stabilize ec2_vpc_subnet module

* Add waiters for ec2_vpc_subnet

Clean up integration tests

* Reenable CI for stabilized ec2_vpc_subnet tests

* rename waiters

* Use module_json_aws where applicable

Handle WaiterError first if waiting failed

* Fix traceback when tagging with keys/values that look like booleans

* Fix check mode with tags

* Add integration tests for tags that look like booleans and check mode

* Add waiter for deleting subnet

* Sleep a few seconds after using aws command line
2018-04-03 15:25:00 -04:00
Ryan Brown 93946ba627 Use NormalizedOperationMethod to catch ClientErrors so the waiter can handle them properly (#37356) 2018-03-13 16:02:39 -04:00
Ryan Brown c8ef07e015 Route custom waiter (#36922)
This creates a way for us to use boto3's data-driven waiter support to use custom waiters where Boto3 hasn't implemented them yet.

The only waiter implemented so far is for VPC Route Tables to check that they exist, and this replaces some custom retry code.
2018-03-06 08:22:16 +10:00
Ed Costello 07bc98f019 Additions to AWS Module Guidlines from Pull Request Feedback (#36834)
* Update doc of AnsibleAWSModule to remove incorrect connect example.

Current example uses aws_connect method which no longer exists. Replace
this with the client and resource methods that do exist.

Also remove try/except block in connect example as guidelines imply that
there aren't connection errors thrown on connection, just when later
sing the connection.

* Update AWS module guidelines to include the AnsibleAWSModule connection methods.

* Add information on integration testing to the AWS module guidelines.

* Add information on common documentation fragments to AWS module
Guidelines.

* Changes as requested on PR.

* Restructure connection section to start with current best practice
* Explain the use of the CI build groups
* Use YAML Anchors for aws credentials example

* Replace guidance on use of test groups with link to aliases file.

This achieves the goal of explaining why this file is necessary without
introducing overhead of keeping documentation up to date when test
groups change.
2018-03-02 11:40:01 +10:00
Sloane Hertel f7d79d4789 [cloud] Retry WAF actions on WAFStaleDataException (#36405)
Add a util to run functions with AWSRetry to retry on WAFStaleDataExceptions and update ChangeToken for each attempt
2018-02-21 08:14:17 -05:00
Ryan Brown e71c6d8e17 New module: ec2_instance (#35749)
New module: ec2_instance

Integration tests for ec2_instance
2018-02-08 01:50:41 -05:00
Willem van Ketwich 4faad64579 [cloud][aws] New module: cloudfront_invalidation (#35601)
* Remove redundant default for target_paths

* Make invalidation requests idempotent

* Remove redundant exception handling around boto3_conn

* Add cloudfront_invalidation to the changelog
2018-02-02 15:09:55 -05:00
Jonathan Meran 8578b3c1c5 [cloud] new module aws_batch_job_queue: support for AWS Batch Job Queues (#26701)
1. Removed redundant default values in argument_spec
2. Enclosed non standard python libraries in Try Except ImportError block
3. Added region as a mandatory element in argument_spec
4. Updated the typo in api_params while accessing job queue. Changed it from job_queue to jobQueue
5. Updated the set_api_params() method to use snake_dict_to_camel_dict()
6. Updated the ANSIBLE_METADATA version from 1.0 to 1.1
7. Updated the version_added in Documentation from 2.4 to 2.5
8. Initialized response value to None
9. Updated the RETUN documentation
10. Updated the return from manage_state() method to not embed batch_job_queue_action in ansible_facts
11. Updated the module.exit_json()  in main() to convert any keys back to snake_dict format using camel_dict_to_snake_dict to maintain consistency throughout the module
12. Changed the filename from batch_job_queue.py to aws_batch_job_queue.py
2018-01-30 17:22:06 -05:00
Jonathan Meran 9e68ac9d86 Initial support for AWS Batch Compute Environment (#26699)
* Add AWS Batch Compute Environment module

* Move common code for AWS batch modules in module_utils/aws/batch
2018-01-30 13:24:38 -05:00
Sloane Hertel 6dda001a8c [cloud] new module - AWS Direct Connect virtual interface (#33211)
* Add an AWS Direct Connect virtual interface module

Fix DirectConnectError exception wrapper to be compatible with fail_json_aws()

* Fix documentation and handle BotoCoreErrors as well

* Add the aws and ec2 documentation fragments
2018-01-29 17:23:03 -05:00
Will Thames 8d733dbdf0 [cloud] New module cloudfront_distribution (#31284)
* added cloudfont.py, modified cloudfront_facts.py class name and fixed a minor bug

* Improvements to cloudfront_distribution

* Reduce the scope of the cloudfront_distribution module
    * Remove presigning
    * Remove streaming distribution functionality
* Add full test suite for cloudfront distribution
* Meet Ansible AWS guidelines

* Make requested changes

Fix tests

Use built-in waiter

Update copyright
2018-01-17 11:03:23 -05:00
Will Thames 04df2312e7 Provide AnsibleAWSModule with _name attribute (#34564)
_name attribute is used when providing generic error messages
(such as connection problems). As AnsibleAWSModule does not inherit
things from AnsibleModule by default, need to provide it.
2018-01-11 14:11:28 +10:00
Will Thames 3283f46ffa Create common waf module for use by future waf modules (#33003)
Move waf common code into waf module_utils.
This will be used by future waf modules
2017-12-08 15:50:26 -05:00
s-hertel bd0e1b8743 Fix header direct_connect module_utils 2017-10-18 12:08:19 -07:00
Matt Clay 68aeaa58a8 Fix dangerous default args. (#29839) 2017-09-12 00:11:13 -07:00
Sloane Hertel a48e0b5101 New Module to manage AWS direct connect link aggregation groups (#27250)
* Add module_utils/aws/direct_connect.py for frequently used functions

* new AWS Direct Connect link aggregation group module with tests and placebo recordings

* remove extra argument

* Remove use of undefined var

* Fix param name for extra exception codes for AWSRetry to use.

* Fix undefined var and line length and metadata version number

* Fix copyright headers
2017-08-22 18:24:54 -04:00
René Moser 0391ac6ece fix pep8, fixes build (#26660) 2017-07-12 00:26:59 +02:00
Toshio Kuratomi da540b5c48 Properly export the public portions of module_utils.aws.core 2017-07-11 14:07:06 -07:00
mikedlr 07a4079a81 aws.core in new aws dir in module utils - module with AnsibleAWSModule class and fail_json_aws (#25780)
* aws module utils including AnsibleAWSModule
* fail_json_aws method on AnsibleAWSModule to do fail_json nicely with AWS exceptions
* aws module util - feedback - rename to aws/core.py & improve doc strings
2017-07-11 14:01:35 -07:00