aws-encryption-sdk

Latest Version Supported Python Versions Code style: black Documentation Status

The AWS Encryption SDK for Python provides a fully compliant, native Python implementation of the AWS Encryption SDK.

The latest full documentation can be found at Read the Docs.

Find us on GitHub.

Security issue notifications

See Support Policy for details on the current support status of all major versions of this library.

Getting Started

Required Prerequisites

  • Python 3.8+

  • cryptography >= 3.4.6

  • boto3 >= 1.10.0

  • attrs

Installation

Note

If you have not already installed cryptography, you might need to install additional prerequisites as detailed in the cryptography installation guide for your operating system.

$ pip install "aws-encryption-sdk[MPL]"

The [MPL] suffix also installs the AWS Cryptographic Material Providers Library (MPL). This is a library that contains constructs for encrypting and decrypting your data. We highly recommend installing the MPL. However, if you do not wish to install the MPL, omit the [MPL] suffix.

Concepts

There are three main concepts that you need to understand to use this library:

Data Keys

Data keys are the encryption keys that are used to encrypt your data. If your algorithm suite uses a key derivation function, the data key is used to generate the key that directly encrypts the data.

Keyrings

Keyrings are resources that generate, encrypt, and decrypt data keys. You specify a keyring when encrypting and the same or a different keyring when decrypting.

Note: You must also install the AWS Cryptographic Material Providers Library (MPL) to create and use keyrings.

For more information, see the AWS Documentation for Keyrings.

Cryptographic Materials Managers

Cryptographic materials managers (CMMs) are resources that collect cryptographic materials and prepare them for use by the Encryption SDK core logic.

An example of a CMM is the default CMM, which is automatically generated anywhere a caller provides a keyring.

Note: You must also install the AWS Cryptographic Material Providers Library (MPL) to create and use CMMs that use keyrings. CMMs that use master key providers have been marked as legacy since v4 of this library.

Legacy Concepts

This section describes legacy concepts introduced in earlier versions of this library. These components have been superseded by new components in the AWS Cryptographic Material Providers Library (MPL). Please avoid using these components, and instead use components in the MPL.

Master Key Providers

Master key providers are resources that provide master keys.

To encrypt data in this client, a MasterKeyProvider object must contain at least one MasterKey object.

MasterKeyProvider objects can also contain other MasterKeyProvider objects.

NOTE: Master key providers are legacy components and have been superseded by keyrings provided by the AWS Cryptographic Material Providers Library (MPL). Please install this library and migrate master key providers to keyring interfaces.

Master Keys

Master keys generate, encrypt, and decrypt data keys. An example of a master key is an AWS KMS key.

NOTE: Master keys are legacy constructs and have been superseded by keyrings provided by the AWS Cryptographic Material Providers Library (MPL). Please install this library and migrate master key providers to keyring interfaces.

Usage

EncryptionSDKClient

To use this module, you (the caller) must first create an instance of the EncryptionSDKClient class. The constructor to this class accepts an optional keyword argument, commitment_policy, that controls which algorithm suites can be used for encryption and decryption. If no value is provided for this argument, a default value of REQUIRE_ENCRYPT_REQUIRE_DECRYPT is used. Unless you have specialized performance requirements or are in the process of migrating from an older version of the AWS Encryption SDK, we recommend using the default value.

import aws_encryption_sdk
from aws_encryption_sdk.identifiers import CommitmentPolicy


client = aws_encryption_sdk.EncryptionSDKClient(
    commitment_policy=CommitmentPolicy.REQUIRE_ENCRYPT_REQUIRE_DECRYPT
)

You must then create an instance of either a keyring (with the MPL installed) or a CMM. Note: You must also install the AWS Cryptographic Material Providers Library (MPL) to use keyrings. (You may also provide an instance of a legacy master key provider, but this is not recommended.)

AwsKmsMultiKeyring

An AwsKmsMultiKeyring is configured with a generator keyring and a list of child keyrings of type AwsKmsKeyring. The effect is like using several keyrings in a series. When you use a multi-keyring to encrypt data, any of the wrapping keys in any of its keyrings can decrypt that data.

On encryption, the generator keyring generates and encrypts the plaintext data key. Then, all of the wrapping keys in all of the child keyrings encrypt the same plaintext data key. The final encrypted message will include a copy of the data key encrypted by each configured key. On decryption, the AWS Encryption SDK uses the keyrings to try to decrypt one of the encrypted data keys. The keyrings are called in the order that they are specified in the multi-keyring. Processing stops as soon as any key in any keyring can decrypt an encrypted data key.

An individual AwsKmsKeyring in an AwsKmsMultiKeyring is configured with an AWS KMS key ARN. For keyrings that will only be used for encryption, you can use any valid KMS key identifier. For providers that will be used for decryption, you must use the key ARN. Key ids, alias names, and alias ARNs are not supported for decryption.

Because the AwsKmsMultiKeyring uses the boto3 SDK to interact with AWS KMS, it requires AWS Credentials. To provide these credentials, use the standard means by which boto3 locates credentials or provide a pre-existing instance of a botocore session to the AwsKmsMultiKeyring. This latter option can be useful if you have an alternate way to store your AWS credentials or you want to reuse an existing instance of a botocore session in order to decrease startup costs. You can also add KMS keys from multiple regions to the AwsKmsMultiKeyring.

See examples/src/aws_kms_multi_keyring_example.py for a code example configuring and using a AwsKmsMultiKeyring with the EncryptionSDKClient.

AwsKmsDiscoveryKeyring

We recommend using an AwsKmsMultiKeyring in order to ensure that you can only encrypt and decrypt data using the AWS KMS key ARN you expect. However, if you are unable to explicitly identify the AWS KMS key ARNs that should be used for decryption, you can instead use an AwsKmsDiscoveryKeyring for decryption operations. This provider attempts decryption of any ciphertexts as long as they match a DiscoveryFilter that you configure. A DiscoveryFilter consists of a list of AWS account ids and an AWS partition. If you do not want to filter the set of allowed accounts, you can also omit the discovery_filter argument.

Note that an AwsKmsDiscoveryKeyring cannot be used for encryption operations.

See examples/src/aws_kms_discovery_keyring_example.py for a code example configuring and using an AwsKmsDiscoveryKeyring with the EncryptionSDKClient.

Encryption and Decryption

After you create an instance of an EncryptionSDKClient and a Keyring, you can use the client’s encrypt and decrypt functions to encrypt and decrypt your data.

You can also provide an encryption context: a form of additional authenticating information.

See code in the examples/src/ directory for code examples configuring and using keyrings and encryption context with the EncryptionSDKClient.

Streaming

If you are handling large files or simply do not want to put the entire plaintext or ciphertext in memory at once, you can use this library’s streaming clients directly. The streaming clients are file-like objects, and behave exactly as you would expect a Python file object to behave, offering context manager and iteration support.

See examples/src/file_streaming_example.py for a code example streaming data to and from files.

Performance Considerations

Adjusting the frame size can significantly improve the performance of encrypt/decrypt operations with this library.

Processing each frame in a framed message involves a certain amount of overhead. If you are encrypting a large file, increasing the frame size can offer potentially significant performance gains. We recommend that you tune these values to your use-case in order to obtain peak performance.

Thread safety

The EncryptionSDKClient and all provided CryptoMaterialsManager in this library are thread safe. But instances of BaseKMSMasterKeyProvider MUST not be shared between threads, for the reasons outlined in the boto3 docs.

Because the BaseKMSMaterKeyProvider creates a new boto3 sessions per region, users do not need to create a client for every region in every thread; a new BaseKMSMasterKeyProvider per thread is sufficient.

(The BaseKMSMasterKeyProvider is the internal parent class of all the KMS Providers.)

Finally, while the CryptoMaterialsCache is thread safe, sharing entries in that cache across threads needs to be done carefully (see the !Note about partition name in the API Docs).

Important: Components from the AWS Cryptographic Material Providers Library (MPL) have separate thread safety considerations. For more information, see the note on thread safety in that project’s README (TODO-MPL: link)

Modules

aws_encryption_sdk

High level AWS Encryption SDK client functions.

aws_encryption_sdk.exceptions

Contains exception classes for AWS Encryption SDK.

aws_encryption_sdk.identifiers

AWS Encryption SDK native data structures for defining implementation-specific characteristics.

aws_encryption_sdk.caches

Common functions and structures for use in cryptographic materials caches.

aws_encryption_sdk.caches.base

Base class interface for caches for use with caching crypto material managers.

aws_encryption_sdk.caches.local

Local, in-memory, LRU, cryptographic materials cache for use with caching cryptographic materials providers.

aws_encryption_sdk.caches.null

Null cache: a cache which does not cache.

aws_encryption_sdk.key_providers.base

Base class interface for Master Key Providers.

aws_encryption_sdk.key_providers.kms

Master Key Providers for use with AWS KMS

aws_encryption_sdk.key_providers.raw

Resources required for Raw Master Keys.

aws_encryption_sdk.materials_managers

Primitive structures for use when interacting with crypto material managers.

aws_encryption_sdk.materials_managers.base

Base class interface for crypto material managers.

aws_encryption_sdk.materials_managers.caching

Caching crypto material manager.

aws_encryption_sdk.materials_managers.default

Default crypto material manager class.

aws_encryption_sdk.streaming_client

High level AWS Encryption SDK client for streaming objects.

aws_encryption_sdk.structures

Public data structures for aws_encryption_sdk.

aws_encryption_sdk.internal

Internal Implementation Details

aws_encryption_sdk.internal.crypto.authentication

Contains authentication primitives.

aws_encryption_sdk.internal.crypto.data_keys

Contains data key helper functions.

aws_encryption_sdk.internal.crypto.elliptic_curve

Contains elliptic curve functionality.

aws_encryption_sdk.internal.crypto.encryption

Contains encryption primitives and helper functions.

aws_encryption_sdk.internal.crypto.iv

Helper functions used for generating deterministic initialization vectors (IVs).

aws_encryption_sdk.internal.crypto.wrapping_keys

Contains wrapping key primitives.

aws_encryption_sdk.internal.defaults

Default values for AWS Encryption SDK.

aws_encryption_sdk.internal.formatting

Formatting functions for aws_encryption_sdk.

aws_encryption_sdk.internal.formatting.deserialize

Components for handling AWS Encryption SDK message deserialization.

aws_encryption_sdk.internal.formatting.encryption_context

Components for handling serialization and deserialization of encryption context data in AWS Encryption SDK messages.

aws_encryption_sdk.internal.formatting.serialize

Components for handling AWS Encryption SDK message serialization.

aws_encryption_sdk.internal.str_ops

Helper functions for consistently obtaining str and bytes objects in both Python2 and Python3.

aws_encryption_sdk.internal.structures

Public data structures for aws_encryption_sdk.

aws_encryption_sdk.internal.utils

Helper utility functions for AWS Encryption SDK.

Changelog

4.0.0 – 2024-10-29

Features

Breaking Changes

  • The MPL introduces the Required Encryption Context Cryptographic Materials Manager (“required EC CMM”) as a new construct for protecting your data. On encrypt, the required EC CMM will use specific configured encryption context key-value pairs to calculate the message signature, but will not store those pairs in the ESDK message. On decrypt, decryptors must supply these same pairs that were used when encrypting the message. All messages that have been encrypted with versions of the ESDK <4.0.0 are forward compatible with this change. However, messages that are constructed with the required EC CMM are not backward compatible with ESDK <4.0.0, as no version of ESDK <4.0.0 supports reading messages encrypted with the required EC CMM. A message that is encrypted with the required EC CMM from the MPL must be decrypted with a CMM from the MPL.

Fixes

  • fix: MKPs attempt to decrypt with remaining keys if a preceding raw RSA key failed to decrypt #707

3.3.0 – 2024-05-20

Deprecation

The AWS Encryption SDK for Python no longer supports Python 3.7 as of version 3.3; only Python 3.8+ is supported.

Fixes

Maintenance

3.2.0 – 2024-03-18

Features

Fixes

Maintenance

3.1.1 – 2022-06-20

Maintenance

  • Replace deprecated cryptography verify_interface with isinstance #467

3.1.0 – 2021-11-10

Deprecation

The AWS Encryption SDK for Python no longer supports Python 3.5 as of version 3.1; only Python 3.6+ is supported. Customers using Python 3.5 can still use the 2.x line of the AWS Encryption SDK for Python, which will continue to receive security updates, in accordance with our Support Policy.

Feature

  • Warn on Deprecated Python usage #368

  • Add Python 3.10 to CI

  • Remove Python 3.5 from testing

3.0.0 – 2021-07-01

Deprecation

The AWS Encryption SDK for Python no longer supports Python 2 or Python 3.4 as of major version 3.x; only Python 3.5+ is supported. Customers using Python 2 or Python 3.4 can still use the 2.x line of the AWS Encryption SDK for Python, which will continue to receive security updates for the next 12 months, in accordance with our Support Policy.

Maintenance

  • Move away from deprecated cryptography int_from_bytes #355

2.4.0 – 2021-07-01

Deprecation Announcement

The AWS Encryption SDK for Python is discontinuing support for Python 2. Future major versions of this library will drop support for Python 2 and begin to adopt changes that are known to break Python 2.

Support for Python 3.4 will be removed at the same time. Moving forward, we will support Python 3.5+.

Security updates will still be available for the Encryption SDK 2.x line for the next 12 months, in accordance with our Support Policy.

2.3.0 – 2021-06-16

Features

2.2.0 – 2021-05-27

Features

2.1.0 – 2020-04-20

Maintenance

  • New minimum cryptography dependency 2.5.0 since we’re using newer byte type checking #308

  • New minimum boto dependency 1.10.0 to ensure KMS Decrypt APIs know about the KeyId parameter #317

  • Add python 3.8 and 3.9 to CI and update setup.py to clarify we support them #329

  • Update decrypt oracle and test vector handlers with 2.0.0 changes #303

  • Added a number of CodeBuild specs to support integration tests and release processes

2.0.0 – 2020-09-24

Features

  • Updates to the AWS Encryption SDK. 73cce71

Breaking Changes

  • KMSMasterKeyProvider is removed. Customers must use StrictAwsKmsMasterKeyProvider with explicit key ids, or DiscoveryAwsKmsMasterKeyProvider to allow decryption of any ciphertext to which the application has access.

  • The encrypt, decrypt, and stream methods in the aws_encryption_sdk module are removed, replaced by identically named methods on the new EncryptionSDKClient class.

  • Key committing algorithm suites are now default.

See Migration guide for more details.

1.7.0 – 2020-09-24

Features

  • Updates to the AWS Encryption SDK. ef90351

Deprecations

  • KMSMasterKeyProvider is deprecated. Customers should move to StrictAwsKmsMasterKeyProvider with explicit key ids, or DiscoveryAwsKmsMasterKeyProvider to allow decryption of any ciphertext to which the application has access.

  • The encrypt, decrypt, and stream methods in the aws_encryption_sdk module are deprecated. Customers should move to the identically named methods on the new EncryptionSDKClient class.

See Migration guide for more details.

1.4.1 – 2019-09-20

Bugfixes

  • Fix region configuration override in botocore sessions. #190 #193

Minor

  • Caching CMM must require that max age configuration value is greater than 0. #147 #172

1.4.0 – 2019-05-23

Minor

  • Remove dependence on all source_stream APIs except for read(). #103

Potentially Backwards Incompatible

  • Encryption streams no longer close the source_stream when they themselves close. If you are using context managers for all of your stream handling, this change will not affect you. However, if you have been relying on the StreamDecryptor or StreamEncryptor to close your source_stream for you, you will now need to close those streams yourself.

  • StreamDecryptor.body_start and StreamDecryptor.body_end, deprecated in a prior release, have now been removed.

Maintenance

  • Move all remaining unittest tests to pytest. #99

Bugfixes

  • Fix MasterKeyprovider.decrypt_data_key_from_list error handling. #150

1.3.8 – 2018-11-15

Bugfixes

  • Remove debug logging that may contain input data when encrypting non-default unframed messages. #105

Minor

  • Add support to remove clients from KMSMasterKeyProvider client cache if they fail to connect to endpoint. #86

  • Add support for SHA384 and SHA512 for use with RSA OAEP wrapping algorithms. #56

  • Fix streaming_client classes to properly interpret short reads in source streams. #24

1.3.7 – 2018-09-20

Bugfixes

  • Fix KMSMasterKeyProvider to determine the default region before trying to create the requested master keys. #83

1.3.6 – 2018-09-04

Bugfixes

  • StreamEncryptor and StreamDecryptor should always report as readable if they are open. #73

  • Allow duck-typing of source streams. #75

1.3.5 – 2018-08-01

  • Move the aws-encryption-sdk-python repository from awslabs to aws.

1.3.4 – 2018-04-12

Bugfixes

  • AWS KMS master key/provider user agent extension fixed. #47

Maintenance

  • New minimum pytest version 3.3.1 to avoid bugs in 3.3.0 #32

  • New minimum attrs version 17.4.0 to allow use of converter rather than convert #39

  • Algorithm Suites are modeled as collections of sub-suites now #36

  • Selecting test suites is more sane now, with pytest markers. #41

1.3.3 – 2017-12-05

Bugfixes

  • Remove use of attrs functionality deprecated in 17.3.0 #29

Maintenance

1.3.2 – 2017-09-28

  • Addressed issue #13 to properly handle non-seekable source streams.

1.3.1 – 2017-09-12

Reorganization

  • Moved source into src.

  • Moved examples into examples.

  • Broke out internal.crypto into smaller, feature-oriented, modules.

Tooling

  • Added tox configuration to support automation and development tooling.

  • Added pylint, flake8, and doc8 configuration to enforce style rules.

Maintenance

  • Updated internal.crypto.authentication.Verifier to use Prehashed.

  • Addressed docstring issue #7.

  • Addressed docstring issue #8.

  • Addressed logging issue #10.

  • Addressed assorted linting issues to bring source, tests, examples, and docs up to configured linting standards.

1.3.0 – 2017-08-04

Major

  • Added cryptographic materials managers as a concept

  • Added data key caching

  • Moved to deterministic IV generation

Minor

  • Added changelog

  • Fixed attrs usage to provide consistent behavior with 16.3.0 and 17.x

  • Fixed performance bug which caused KDF calculations to be performed too frequently

  • Removed line_length as a configurable parameter of EncryptingStream and DecryptingStream objects to simplify class APIs after it was found in further testing to have no measurable impact on performance

  • Added deterministic length eliptic curve signature generation

  • Added support for calculating ciphertext message length from header

  • Migrated README from md to rst

1.2.2 – 2017-05-23

1.2.0 – 2017-03-21

  • Initial public release