Medium severity5.3GHSA Advisory· Published Dec 17, 2025· Updated Apr 15, 2026
CVE-2025-14762
CVE-2025-14762
Description
Missing cryptographic key commitment in the AWS SDK for Ruby may allow a user with write access to the S3 bucket to introduce a new EDK that decrypts to different plaintext when the encrypted data key is stored in an "instruction file" instead of S3's metadata record.
To mitigate this issue, upgrade AWS SDK for Ruby to version 1.208.0 or later.
Affected packages
Versions sourced from the GitHub Security Advisory.
| Package | Affected versions | Patched versions |
|---|---|---|
aws-sdk-s3RubyGems | < 1.208.0 | 1.208.0 |
Affected products
1- Range: < 1.208.0
Patches
1b633ba10cd2fUpdates to the S3 Encryption Client (#3328)
46 files changed · +7580 −188
gems/aws-sdk-s3/CHANGELOG.md+2 −0 modified@@ -1,6 +1,8 @@ Unreleased Changes ------------------ +* Feature - Updates to the S3 Encryption Client. The V3 S3 Encryption Client now requires key committing algorithm suites by default. See migration guide: [link to docs] + 1.207.0 (2025-12-15) ------------------
gems/aws-sdk-s3/lib/aws-sdk-s3/customizations.rb+1 −0 modified@@ -6,6 +6,7 @@ module S3 autoload :BucketRegionCache, 'aws-sdk-s3/bucket_region_cache' autoload :Encryption, 'aws-sdk-s3/encryption' autoload :EncryptionV2, 'aws-sdk-s3/encryption_v2' + autoload :EncryptionV3, 'aws-sdk-s3/encryption_v3' autoload :FilePart, 'aws-sdk-s3/file_part' autoload :DefaultExecutor, 'aws-sdk-s3/default_executor' autoload :FileUploader, 'aws-sdk-s3/file_uploader'
gems/aws-sdk-s3/lib/aws-sdk-s3/encryption/client.rb+2 −2 modified@@ -6,9 +6,9 @@ module Aws module S3 # [MAINTENANCE MODE] There is a new version of the Encryption Client. - # AWS strongly recommends upgrading to the {Aws::S3::EncryptionV2::Client}, + # AWS strongly recommends upgrading to the {Aws::S3::EncryptionV3::Client}, # which provides updated data security best practices. - # See documentation for {Aws::S3::EncryptionV2::Client}. + # See documentation for {Aws::S3::EncryptionV3::Client}. # Provides an encryption client that encrypts and decrypts data client-side, # storing the encrypted data in Amazon S3. #
gems/aws-sdk-s3/lib/aws-sdk-s3/encryption/default_cipher_provider.rb+2 −0 modified@@ -16,6 +16,8 @@ def initialize(options = {}) # envelope and encryption cipher. def encryption_cipher cipher = Utils.aes_encryption_cipher(:CBC) + ##= ../specification/s3-encryption/data-format/content-metadata.md#algorithm-suite-and-message-format-version-compatibility + ##% Objects encrypted with ALG_AES_256_CBC_IV16_NO_KDF MAY use either the V1 or V2 message format version. envelope = { 'x-amz-key' => encode64(encrypt(envelope_key(cipher))), 'x-amz-iv' => encode64(envelope_iv(cipher)),
gems/aws-sdk-s3/lib/aws-sdk-s3/encryption/encrypt_handler.rb+2 −0 modified@@ -38,6 +38,8 @@ def apply_encryption_cipher(context, cipher) io = StringIO.new(io) if String === io context.params[:body] = IOEncrypter.new(cipher, io) context.params[:metadata] ||= {} + ##= ../specification/s3-encryption/data-format/content-metadata.md#content-metadata-mapkeys + ##% - The mapkey "x-amz-unencrypted-content-length" SHOULD be present for V1 format objects. context.params[:metadata]['x-amz-unencrypted-content-length'] = io.size if context.params.delete(:content_md5) warn('Setting content_md5 on client side encrypted objects is deprecated')
gems/aws-sdk-s3/lib/aws-sdk-s3/encryption/kms_cipher_provider.rb+2 −0 modified@@ -26,6 +26,8 @@ def encryption_cipher end cipher = Utils.aes_encryption_cipher(:CBC) cipher.key = key_data.plaintext + ##= ../specification/s3-encryption/data-format/content-metadata.md#algorithm-suite-and-message-format-version-compatibility + ##% Objects encrypted with ALG_AES_256_CBC_IV16_NO_KDF MAY use either the V1 or V2 message format version. envelope = { 'x-amz-key-v2' => encode64(key_data.ciphertext_blob), 'x-amz-iv' => encode64(cipher.iv = cipher.random_iv),
gems/aws-sdk-s3/lib/aws-sdk-s3/encryptionV2/client.rb+98 −23 modified@@ -5,9 +5,17 @@ module Aws module S3 - REQUIRED_PARAMS = [:key_wrap_schema, :content_encryption_schema, :security_profile] - SUPPORTED_SECURITY_PROFILES = [:v2, :v2_and_legacy] + REQUIRED_PARAMS = [:key_wrap_schema, :content_encryption_schema, :security_profile].freeze + SUPPORTED_SECURITY_PROFILES = [:v2, :v2_and_legacy].freeze + SUPPORTED_COMMITMENT_POLICIES = [:forbid_encrypt_allow_decrypt].freeze + # [MAINTENANCE MODE] There is a new version of the Encryption Client. + # AWS strongly recommends upgrading to the {Aws::S3::EncryptionV3::Client}, + # which provides updated data security best practices. + # For migration guidance, see: https://docs.aws.amazon.com/sdk-for-ruby/v3/developer-guide/s3-encryption-migration-v2-v3.html + # Provides an encryption client that encrypts and decrypts data client-side, + # storing the encrypted data in Amazon S3. + # # Provides an encryption client that encrypts and decrypts data client-side, # storing the encrypted data in Amazon S3. The `EncryptionV2::Client` (V2 Client) # provides improved security over the `Encryption::Client` (V1 Client) @@ -307,15 +315,29 @@ class Client # @option options [KMS::Client] :kms_client A default {KMS::Client} # is constructed when using KMS to manage encryption keys. # + # @option options [Symbol] :commitment_policy (nil) + # Optional parameter for migration from V2 to V3. When set to + # :forbid_encrypt_allow_decrypt, this explicitly indicates you are + # maintaining V2 encryption behavior while preparing for migration. + # This allows the V2 client to decrypt V3-encrypted objects while + # continuing to encrypt new objects using V2 algorithms. + # Only :forbid_encrypt_allow_decrypt is supported. + # For migration guidance, see: https://docs.aws.amazon.com/sdk-for-ruby/v3/developer-guide/s3-encryption-migration-v2-v3.html + # def initialize(options = {}) validate_params(options) @client = extract_client(options) - @cipher_provider = cipher_provider(options) + @cipher_provider = build_cipher_provider(options) + @key_provider = @cipher_provider.key_provider if @cipher_provider.is_a?(DefaultCipherProvider) @envelope_location = extract_location(options) @instruction_file_suffix = extract_suffix(options) @kms_allow_decrypt_with_any_cmk = options[:kms_key_id] == :kms_allow_decrypt_with_any_cmk @security_profile = extract_security_profile(options) + @commitment_policy = extract_commitment_policy(options) + # The v3 cipher is only used for decrypt. + # Therefore any configured v2 `content_encryption_schema` is going to be incorrect. + @v3_cipher_provider = build_v3_cipher_provider_for_decrypt(options.reject { |k, _| k == :content_encryption_schema }) end # @return [S3::Client] @@ -341,6 +363,11 @@ def initialize(options = {}) # by this string. attr_reader :instruction_file_suffix + # @return [Symbol, nil] Optional commitment policy for V2 to V3 migration. + # When set to :forbid_encrypt_allow_decrypt, explicitly indicates + # maintaining V2 encryption behavior while preparing for migration. + attr_reader :commitment_policy + # Uploads an object to Amazon S3, encrypting data client-side. # See {S3::Client#put_object} for documentation on accepted # request parameters. @@ -410,6 +437,7 @@ def get_object(params = {}, &block) req.handlers.add(DecryptHandler) req.context[:encryption] = { cipher_provider: @cipher_provider, + v3_cipher_provider: @v3_cipher_provider, envelope_location: envelope_location, instruction_file_suffix: instruction_file_suffix, kms_encryption_context: kms_encryption_context, @@ -423,6 +451,50 @@ def get_object(params = {}, &block) private + def build_cipher_provider(options) + if options[:kms_key_id] + KmsCipherProvider.new( + kms_key_id: options[:kms_key_id], + kms_client: kms_client(options), + key_wrap_schema: options[:key_wrap_schema], + content_encryption_schema: options[:content_encryption_schema] + ) + else + key_provider = extract_key_provider(options) + DefaultCipherProvider.new( + key_provider: key_provider, + key_wrap_schema: options[:key_wrap_schema], + content_encryption_schema: options[:content_encryption_schema] + ) + end + end + + def build_v3_cipher_provider_for_decrypt(options) + if options[:kms_key_id] + Aws::S3::EncryptionV3::KmsCipherProvider.new( + kms_key_id: options[:kms_key_id], + kms_client: kms_client(options), + key_wrap_schema: options[:key_wrap_schema], + content_encryption_schema: options[:content_encryption_schema] + ) + else + # Create V3 key provider explicitly for proper namespace consistency + key_provider = if options[:key_provider] + options[:key_provider] + elsif options[:encryption_key] + Aws::S3::EncryptionV3::DefaultKeyProvider.new(options) + else + msg = 'you must pass a :kms_key_id, :key_provider, or :encryption_key' + raise ArgumentError, msg + end + Aws::S3::EncryptionV3::DefaultCipherProvider.new( + key_provider: key_provider, + key_wrap_schema: options[:key_wrap_schema], + content_encryption_schema: options[:content_encryption_schema] + ) + end + end + # Validate required parameters exist and don't conflict. # The cek_alg and wrap_alg are passed on to the CipherProviders # and further validated there @@ -452,36 +524,19 @@ def extract_client(options) options.delete(:encryption_key) options.delete(:envelope_location) options.delete(:instruction_file_suffix) + options.delete(:commitment_policy) REQUIRED_PARAMS.each { |p| options.delete(p) } S3::Client.new(options) end end def kms_client(options) - options[:kms_client] || begin + options[:kms_client] || (@kms_client ||= KMS::Client.new( region: @client.config.region, credentials: @client.config.credentials, ) - end - end - - def cipher_provider(options) - if options[:kms_key_id] - KmsCipherProvider.new( - kms_key_id: options[:kms_key_id], - kms_client: kms_client(options), - key_wrap_schema: options[:key_wrap_schema], - content_encryption_schema: options[:content_encryption_schema] - ) - else - @key_provider = extract_key_provider(options) - DefaultCipherProvider.new( - key_provider: @key_provider, - key_wrap_schema: options[:key_wrap_schema], - content_encryption_schema: options[:content_encryption_schema] - ) - end + ) end def extract_key_provider(options) @@ -564,7 +619,27 @@ def validate_security_profile(security_profile) end security_profile end + + def extract_commitment_policy(options) + validate_commitment_policy(options[:commitment_policy]) + end + + def validate_commitment_policy(commitment_policy) + return nil if commitment_policy.nil? + + unless SUPPORTED_COMMITMENT_POLICIES.include? commitment_policy + raise ArgumentError, "Unsupported commitment policy: :#{commitment_policy}. " \ + "The V2 client only supports :forbid_encrypt_allow_decrypt for migration purposes. " \ + "For migration guidance, see: https://docs.aws.amazon.com/sdk-for-ruby/v3/developer-guide/s3-encryption-migration-v2-v3.html" + end + commitment_policy + end end end end end + +##= ../specification/s3-encryption/data-format/content-metadata.md#v1-v2-shared +##= type=exception +##= reason=This has never been supported in Ruby +##% This string MAY be encoded by the esoteric double-encoding scheme used by the S3 web server.
gems/aws-sdk-s3/lib/aws-sdk-s3/encryptionV2/decrypt_handler.rb+7 −162 modified@@ -9,40 +9,6 @@ module EncryptionV2 class DecryptHandler < Seahorse::Client::Handler @@warned_response_target_proc = false - V1_ENVELOPE_KEYS = %w( - x-amz-key - x-amz-iv - x-amz-matdesc - ) - - V2_ENVELOPE_KEYS = %w( - x-amz-key-v2 - x-amz-iv - x-amz-cek-alg - x-amz-wrap-alg - x-amz-matdesc - ) - - V2_OPTIONAL_KEYS = %w(x-amz-tag-len) - - POSSIBLE_ENVELOPE_KEYS = (V1_ENVELOPE_KEYS + - V2_ENVELOPE_KEYS + V2_OPTIONAL_KEYS).uniq - - POSSIBLE_WRAPPING_FORMATS = %w( - AES/GCM - kms - kms+context - RSA-OAEP-SHA1 - ) - - POSSIBLE_ENCRYPTION_FORMATS = %w( - AES/GCM/NoPadding - AES/CBC/PKCS5Padding - AES/CBC/PKCS7Padding - ) - - AUTH_REQUIRED_CEK_ALGS = %w(AES/GCM/NoPadding) - def call(context) attach_http_event_listeners(context) apply_cse_user_agent(context) @@ -63,12 +29,14 @@ def call(context) private def attach_http_event_listeners(context) - context.http_response.on_headers(200) do - cipher, envelope = decryption_cipher(context) - decrypter = body_contains_auth_tag?(envelope) ? - authenticated_decrypter(context, cipher, envelope) : - IODecrypter.new(cipher, context.http_response.body) + decrypter = if Aws::S3::EncryptionV3::Decryption.v3?(context) + cipher, envelope = Aws::S3::EncryptionV3::Decryption.decryption_cipher(context) + Aws::S3::EncryptionV3::Decryption.get_decrypter(context, cipher, envelope) + else + cipher, envelope = Aws::S3::EncryptionV2::Decryption.decryption_cipher(context) + Aws::S3::EncryptionV2::Decryption.get_decrypter(context, cipher, envelope) + end context.http_response.body = decrypter end @@ -86,129 +54,6 @@ def attach_http_event_listeners(context) end end - def decryption_cipher(context) - if (envelope = get_encryption_envelope(context)) - cipher = context[:encryption][:cipher_provider] - .decryption_cipher( - envelope, - context[:encryption] - ) - [cipher, envelope] - else - raise Errors::DecryptionError, "unable to locate encryption envelope" - end - end - - def get_encryption_envelope(context) - if context[:encryption][:envelope_location] == :metadata - envelope_from_metadata(context) || envelope_from_instr_file(context) - else - envelope_from_instr_file(context) || envelope_from_metadata(context) - end - end - - def envelope_from_metadata(context) - possible_envelope = {} - POSSIBLE_ENVELOPE_KEYS.each do |suffix| - if value = context.http_response.headers["x-amz-meta-#{suffix}"] - possible_envelope[suffix] = value - end - end - extract_envelope(possible_envelope) - end - - def envelope_from_instr_file(context) - suffix = context[:encryption][:instruction_file_suffix] - possible_envelope = Json.load(context.client.get_object( - bucket: context.params[:bucket], - key: context.params[:key] + suffix - ).body.read) - extract_envelope(possible_envelope) - rescue S3::Errors::ServiceError, Json::ParseError - nil - end - - def extract_envelope(hash) - return nil unless hash - return v1_envelope(hash) if hash.key?('x-amz-key') - return v2_envelope(hash) if hash.key?('x-amz-key-v2') - if hash.keys.any? { |key| key.match(/^x-amz-key-(.+)$/) } - msg = "unsupported envelope encryption version #{$1}" - raise Errors::DecryptionError, msg - end - end - - def v1_envelope(envelope) - envelope - end - - def v2_envelope(envelope) - unless POSSIBLE_ENCRYPTION_FORMATS.include? envelope['x-amz-cek-alg'] - alg = envelope['x-amz-cek-alg'].inspect - msg = "unsupported content encrypting key (cek) format: #{alg}" - raise Errors::DecryptionError, msg - end - unless POSSIBLE_WRAPPING_FORMATS.include? envelope['x-amz-wrap-alg'] - alg = envelope['x-amz-wrap-alg'].inspect - msg = "unsupported key wrapping algorithm: #{alg}" - raise Errors::DecryptionError, msg - end - unless (missing_keys = V2_ENVELOPE_KEYS - envelope.keys).empty? - msg = "incomplete v2 encryption envelope:\n" - msg += " missing: #{missing_keys.join(',')}\n" - raise Errors::DecryptionError, msg - end - envelope - end - - # This method fetches the tag from the end of the object by - # making a GET Object w/range request. This auth tag is used - # to initialize the cipher, and the decrypter truncates the - # auth tag from the body when writing the final bytes. - def authenticated_decrypter(context, cipher, envelope) - http_resp = context.http_response - content_length = http_resp.headers['content-length'].to_i - auth_tag_length = auth_tag_length(envelope) - - auth_tag = context.client.get_object( - bucket: context.params[:bucket], - key: context.params[:key], - version_id: context.params[:version_id], - range: "bytes=-#{auth_tag_length}" - ).body.read - - cipher.auth_tag = auth_tag - cipher.auth_data = '' - - # The encrypted object contains both the cipher text - # plus a trailing auth tag. - IOAuthDecrypter.new( - io: http_resp.body, - encrypted_content_length: content_length - auth_tag_length, - cipher: cipher) - end - - def body_contains_auth_tag?(envelope) - AUTH_REQUIRED_CEK_ALGS.include?(envelope['x-amz-cek-alg']) - end - - # Determine the auth tag length from the algorithm - # Validate it against the value provided in the x-amz-tag-len - # Return the tag length in bytes - def auth_tag_length(envelope) - tag_length = - case envelope['x-amz-cek-alg'] - when 'AES/GCM/NoPadding' then AES_GCM_TAG_LEN_BYTES - else - raise ArgumentError, 'Unsupported cek-alg: ' \ - "#{envelope['x-amz-cek-alg']}" - end - if (tag_length * 8) != envelope['x-amz-tag-len'].to_i - raise Errors::DecryptionError, 'x-amz-tag-len does not match expected' - end - tag_length - end - def apply_cse_user_agent(context) if context.config.user_agent_suffix.nil? context.config.user_agent_suffix = EC_USER_AGENT
gems/aws-sdk-s3/lib/aws-sdk-s3/encryptionV2/decryption.rb+205 −0 added@@ -0,0 +1,205 @@ +# frozen_string_literal: true + +require 'base64' + +module Aws + module S3 + module EncryptionV2 + # @api private + class Decryption + ##= ../specification/s3-encryption/data-format/content-metadata.md#content-metadata-mapkeys + ##% - The mapkey "x-amz-key" MUST be present for V1 format objects. + ##= ../specification/s3-encryption/data-format/content-metadata.md#content-metadata-mapkeys + ##% - The mapkey "x-amz-iv" MUST be present for V1 format objects. + ##= ../specification/s3-encryption/data-format/content-metadata.md#content-metadata-mapkeys + ##% - The mapkey "x-amz-matdesc" MUST be present for V1 format objects. + V1_ENVELOPE_KEYS = %w[ + x-amz-key + x-amz-iv + x-amz-matdesc + ].freeze + + ##= ../specification/s3-encryption/data-format/content-metadata.md#content-metadata-mapkeys + ##% - The mapkey "x-amz-key-v2" MUST be present for V2 format objects. + ##= ../specification/s3-encryption/data-format/content-metadata.md#content-metadata-mapkeys + ##% - The mapkey "x-amz-iv" MUST be present for V2 format objects. + ##= ../specification/s3-encryption/data-format/content-metadata.md#content-metadata-mapkeys + ##% - The mapkey "x-amz-cek-alg" MUST be present for V2 format objects. + ##= ../specification/s3-encryption/data-format/content-metadata.md#content-metadata-mapkeys + ##% - The mapkey "x-amz-wrap-alg" MUST be present for V2 format objects. + ##= ../specification/s3-encryption/data-format/content-metadata.md#content-metadata-mapkeys + ##% - The mapkey "x-amz-matdesc" MUST be present for V2 format objects. + V2_ENVELOPE_KEYS = %w[ + x-amz-key-v2 + x-amz-iv + x-amz-cek-alg + x-amz-wrap-alg + x-amz-matdesc + ].freeze + + ##= ../specification/s3-encryption/data-format/content-metadata.md#content-metadata-mapkeys + ##= type=exception + ##= reason=The implementation treats this as optional, but verifies its value. + ##% - The mapkey "x-amz-tag-len" MUST be present for V2 format objects. + V2_OPTIONAL_KEYS = %w[x-amz-tag-len].freeze + + POSSIBLE_ENVELOPE_KEYS = (V1_ENVELOPE_KEYS + V2_ENVELOPE_KEYS + V2_OPTIONAL_KEYS).uniq + + POSSIBLE_WRAPPING_FORMATS = %w[ + AES/GCM + kms + kms+context + RSA-OAEP-SHA1 + ].freeze + + POSSIBLE_ENCRYPTION_FORMATS = %w[ + AES/GCM/NoPadding + AES/CBC/PKCS5Padding + AES/CBC/PKCS7Padding + ].freeze + + AUTH_REQUIRED_CEK_ALGS = %w[AES/GCM/NoPadding].freeze + + class << self + def decryption_cipher(context) + if (envelope = get_encryption_envelope(context)) + cipher = context[:encryption][:cipher_provider] + .decryption_cipher( + envelope, + context[:encryption] + ) + [cipher, envelope] + else + raise Errors::DecryptionError, 'unable to locate encryption envelope' + end + end + + def get_decrypter(context, cipher, envelope) + if body_contains_auth_tag?(envelope) + authenticated_decrypter(context, cipher, envelope) + else + IODecrypter.new(cipher, context.http_response.body) + end + end + + def get_encryption_envelope(context) + if context[:encryption][:envelope_location] == :metadata + envelope_from_metadata(context) || envelope_from_instr_file(context) + else + envelope_from_instr_file(context) || envelope_from_metadata(context) + end + end + + def envelope_from_metadata(context) + possible_envelope = {} + POSSIBLE_ENVELOPE_KEYS.each do |suffix| + if (value = context.http_response.headers["x-amz-meta-#{suffix}"]) + possible_envelope[suffix] = value + end + end + extract_envelope(possible_envelope) + end + + def envelope_from_instr_file(context) + suffix = context[:encryption][:instruction_file_suffix] + possible_envelope = Json.load(context.client.get_object( + bucket: context.params[:bucket], + key: context.params[:key] + suffix + ).body.read) + extract_envelope(possible_envelope) + rescue S3::Errors::ServiceError, Json::ParseError + nil + end + + def extract_envelope(hash) + return nil unless hash + ##= ../specification/s3-encryption/data-format/content-metadata.md#determining-s3ec-object-status + ##% - If the metadata contains "x-amz-iv" and "x-amz-key" then the object MUST be considered as an S3EC-encrypted object using the V1 format. + return v1_envelope(hash) if hash.key?('x-amz-key') + ##= ../specification/s3-encryption/data-format/content-metadata.md#determining-s3ec-object-status + ##% - If the metadata contains "x-amz-iv" and "x-amz-metadata-x-amz-key-v2" then the object MUST be considered as an S3EC-encrypted object using the V2 format. + return v2_envelope(hash) if hash.key?('x-amz-key-v2') + + return unless hash.keys.any? { |key| key.match(/^x-amz-key-(.+)$/) } + + msg = "unsupported envelope encryption version #{::Regexp.last_match(1)}" + raise Errors::DecryptionError, msg + end + + def v1_envelope(envelope) + envelope + end + + def v2_envelope(envelope) + unless POSSIBLE_ENCRYPTION_FORMATS.include? envelope['x-amz-cek-alg'] + alg = envelope['x-amz-cek-alg'].inspect + msg = "unsupported content encrypting key (cek) format: #{alg}" + raise Errors::DecryptionError, msg + end + unless POSSIBLE_WRAPPING_FORMATS.include? envelope['x-amz-wrap-alg'] + alg = envelope['x-amz-wrap-alg'].inspect + msg = "unsupported key wrapping algorithm: #{alg}" + raise Errors::DecryptionError, msg + end + unless (missing_keys = V2_ENVELOPE_KEYS - envelope.keys).empty? + msg = "incomplete v2 encryption envelope:\n" + msg += " missing: #{missing_keys.join(',')}\n" + raise Errors::DecryptionError, msg + end + envelope + end + + def body_contains_auth_tag?(envelope) + AUTH_REQUIRED_CEK_ALGS.include?(envelope['x-amz-cek-alg']) + end + + # This method fetches the tag from the end of the object by + # making a GET Object w/range request. This auth tag is used + # to initialize the cipher, and the decrypter truncates the + # auth tag from the body when writing the final bytes. + def authenticated_decrypter(context, cipher, envelope) + http_resp = context.http_response + content_length = http_resp.headers['content-length'].to_i + auth_tag_length = auth_tag_length(envelope) + + auth_tag = context.client.get_object( + bucket: context.params[:bucket], + key: context.params[:key], + version_id: context.params[:version_id], + range: "bytes=-#{auth_tag_length}" + ).body.read + + cipher.auth_tag = auth_tag + cipher.auth_data = '' + + # The encrypted object contains both the cipher text + # plus a trailing auth tag. + IOAuthDecrypter.new( + io: http_resp.body, + encrypted_content_length: content_length - auth_tag_length, + cipher: cipher + ) + end + + # Determine the auth tag length from the algorithm + # Validate it against the value provided in the x-amz-tag-len + # Return the tag length in bytes + def auth_tag_length(envelope) + tag_length = + case envelope['x-amz-cek-alg'] + when 'AES/GCM/NoPadding' then AES_GCM_TAG_LEN_BYTES + else + raise ArgumentError, 'Unsupported cek-alg: ' \ + "#{envelope['x-amz-cek-alg']}" + end + if (tag_length * 8) != envelope['x-amz-tag-len'].to_i + raise Errors::DecryptionError, 'x-amz-tag-len does not match expected' + end + + tag_length + end + end + end + end + end +end
gems/aws-sdk-s3/lib/aws-sdk-s3/encryptionV2/default_cipher_provider.rb+17 −0 modified@@ -14,11 +14,15 @@ def initialize(options = {}) options[:key_wrap_schema], @key_provider.encryption_materials.key ) + ##= ../specification/s3-encryption/encryption.md#content-encryption + ##% The S3EC MUST use the encryption algorithm configured during [client](./client.md) initialization. @content_encryption_schema = validate_cek( options[:content_encryption_schema] ) end + attr_reader :key_provider + # @return [Array<Hash,Cipher>] Creates an returns a new encryption # envelope and encryption cipher. def encryption_cipher(options = {}) @@ -30,9 +34,14 @@ def encryption_cipher(options = {}) ) else enc_key = encode64( + ##= ../specification/s3-encryption/encryption.md#alg-aes-256-gcm-iv12-tag16-no-kdf + ##% The client MUST NOT provide any AAD when encrypting with ALG_AES_256_GCM_IV12_TAG16_NO_KDF. encrypt_aes_gcm(envelope_key(cipher), @content_encryption_schema) ) end + + ##= ../specification/s3-encryption/data-format/content-metadata.md#algorithm-suite-and-message-format-version-compatibility + ##% Objects encrypted with ALG_AES_256_GCM_IV12_TAG16_NO_KDF MUST use the V2 message format version only. envelope = { 'x-amz-key-v2' => enc_key, 'x-amz-cek-alg' => @content_encryption_schema, @@ -52,8 +61,16 @@ def decryption_cipher(envelope, options = {}) master_key = @key_provider.key_for(envelope['x-amz-matdesc']) if envelope.key? 'x-amz-key' unless options[:security_profile] == :v2_and_legacy + ##= ../specification/s3-encryption/decryption.md#legacy-decryption + ##% If the S3EC is not configured to enable legacy unauthenticated content decryption, the client MUST throw an exception when attempting to decrypt an object encrypted with a legacy unauthenticated algorithm suite. + ##= ../specification/s3-encryption/client.md#enable-legacy-unauthenticated-modes + ##% When disabled, the S3EC MUST NOT decrypt objects encrypted using legacy content encryption algorithms; it MUST throw an exception when attempting to decrypt an object encrypted with a legacy content encryption algorithm. raise Errors::LegacyDecryptionError end + ##= ../specification/s3-encryption/decryption.md#legacy-decryption + ##% The S3EC MUST NOT decrypt objects encrypted using legacy unauthenticated algorithm suites unless specifically configured to do so. + ##= ../specification/s3-encryption/client.md#enable-legacy-unauthenticated-modes + ##% When enabled, the S3EC MUST be able to decrypt objects encrypted with all content encryption algorithms (both legacy and fully supported). # Support for decryption of legacy objects key = Utils.decrypt(master_key, decode64(envelope['x-amz-key'])) iv = decode64(envelope['x-amz-iv'])
gems/aws-sdk-s3/lib/aws-sdk-s3/encryptionV2/encrypt_handler.rb+2 −0 modified@@ -28,6 +28,8 @@ def apply_encryption_envelope(context, envelope) context.client.put_object( bucket: context.params[:bucket], key: context.params[:key] + suffix, + ##= ../specification/s3-encryption/data-format/metadata-strategy.md#v1-v2-instruction-files + ##% In the V1/V2 message format, all of the content metadata MUST be stored in the Instruction File. body: Json.dump(envelope) ) else # :metadata
gems/aws-sdk-s3/lib/aws-sdk-s3/encryptionV2/io_encrypter.rb+2 −0 modified@@ -44,6 +44,8 @@ def close private def encrypt_to_stringio(cipher, plain_text) + ##= ../specification/s3-encryption/encryption.md#alg-aes-256-gcm-iv12-tag16-no-kdf + ##% The client MUST append the GCM auth tag to the ciphertext if the underlying crypto provider does not do so automatically. if plain_text.empty? StringIO.new(cipher.final + cipher.auth_tag) else
gems/aws-sdk-s3/lib/aws-sdk-s3/encryptionV2/kms_cipher_provider.rb+8 −0 modified@@ -33,6 +33,8 @@ def encryption_cipher(options = {}) end cipher = Utils.aes_encryption_cipher(:GCM) cipher.key = key_data.plaintext + ##= ../specification/s3-encryption/data-format/content-metadata.md#algorithm-suite-and-message-format-version-compatibility + ##% Objects encrypted with ALG_AES_256_GCM_IV12_TAG16_NO_KDF MUST use the V2 message format version only. envelope = { 'x-amz-key-v2' => encode64(key_data.ciphertext_blob), 'x-amz-iv' => encode64(cipher.iv = cipher.random_iv), @@ -53,9 +55,15 @@ def decryption_cipher(envelope, options = {}) case envelope['x-amz-wrap-alg'] when 'kms' + ##= ../specification/s3-encryption/client.md#enable-legacy-wrapping-algorithms + ##% The S3EC MUST support the option to enable or disable legacy wrapping algorithms. unless options[:security_profile] == :v2_and_legacy + ##= ../specification/s3-encryption/client.md#enable-legacy-wrapping-algorithms + ##% When disabled, the S3EC MUST NOT decrypt objects encrypted using legacy wrapping algorithms; it MUST throw an exception when attempting to decrypt an object encrypted with a legacy wrapping algorithm. raise Errors::LegacyDecryptionError end + ##= ../specification/s3-encryption/client.md#enable-legacy-wrapping-algorithms + ##% When enabled, the S3EC MUST be able to decrypt objects encrypted with all supported wrapping algorithms (both legacy and fully supported). when 'kms+context' if cek_alg != encryption_context['aws:x-amz-cek-alg'] raise Errors::CEKAlgMismatchError
gems/aws-sdk-s3/lib/aws-sdk-s3/encryption_v2.rb+1 −0 modified@@ -1,4 +1,5 @@ require 'aws-sdk-s3/encryptionV2/client' +require 'aws-sdk-s3/encryptionV2/decryption' require 'aws-sdk-s3/encryptionV2/decrypt_handler' require 'aws-sdk-s3/encryptionV2/default_cipher_provider' require 'aws-sdk-s3/encryptionV2/encrypt_handler'
gems/aws-sdk-s3/lib/aws-sdk-s3/encryptionV2/utils.rb+5 −0 modified@@ -80,6 +80,11 @@ def aes_decryption_cipher(block_mode, key = nil, iv = nil) # @param [OpenSSL::PKey::RSA, String, nil] key # @param [String, nil] iv The initialization vector def aes_cipher(mode, block_mode, key, iv) + ##= ../specification/s3-encryption/encryption.md#alg-aes-256-gcm-iv12-tag16-no-kdf + ##% The client MUST initialize the cipher, + ##% or call an AES-GCM encryption API, with the plaintext data key, the generated IV, + ##% and the tag length defined in the Algorithm Suite + ##% when encrypting with ALG_AES_256_GCM_IV12_TAG16_NO_KDF. cipher = key ? OpenSSL::Cipher.new("aes-#{cipher_size(key)}-#{block_mode.downcase}") : OpenSSL::Cipher.new("aes-256-#{block_mode.downcase}")
gems/aws-sdk-s3/lib/aws-sdk-s3/encryptionV3/client.rb+885 −0 added@@ -0,0 +1,885 @@ +# frozen_string_literal: true + +require 'forwardable' + +module Aws + module S3 + # Provides an encryption client that encrypts and decrypts data client-side, + # storing the encrypted data in Amazon S3. The `EncryptionV3::Client` (V3 Client) + # provides improved security over the `EncryptionV2::Client` (V2 Client) + # through key commitment. You can use the V3 Client to continue decrypting + # objects encrypted by V2 by setting security_profile: :v3_and_legacy. + # The latest V2 Client also supports reading and decrypting objects + # encrypted by the V3 Client. + # + # This client uses a process called "envelope encryption". Your private + # encryption keys and your data's plain-text are **never** sent to + # Amazon S3. **If you lose you encryption keys, you will not be able to + # decrypt your data.** + # + # ## Key Commitment + # + # Key commitment (also known as robustness) is a security property that + # guarantees that each ciphertext can be decrypted to only a single plaintext. + # This prevents sophisticated attacks where a ciphertext could theoretically + # decrypt to different plaintexts under different keys. + # + # The V3 client encrypts with key commitment by default using the + # `:alg_aes_256_gcm_hkdf_sha512_commit_key` algorithm. Key commitment adds + # approximately 32 bytes to each encrypted object and slightly increases + # processing time, but significantly enhances security. + # + # ## Envelope Encryption Overview + # + # The goal of envelope encryption is to combine the performance of + # fast symmetric encryption while maintaining the secure key management + # that asymmetric keys provide. + # + # A one-time-use symmetric key (envelope key) is generated client-side. + # This is used to encrypt the data client-side. This key is then + # encrypted by your master key and stored alongside your data in Amazon + # S3. + # + # When accessing your encrypted data with the encryption client, + # the encrypted envelope key is retrieved and decrypted client-side + # with your master key. The envelope key is then used to decrypt the + # data client-side. + # + # One of the benefits of envelope encryption is that if your master key + # is compromised, you have the option of just re-encrypting the stored + # envelope symmetric keys, instead of re-encrypting all of the + # data in your account. + # + # ## Basic Usage + # + # The encryption client requires an {Aws::S3::Client}. If you do not + # provide a `:client`, then a client will be constructed for you. + # + # require 'openssl' + # key = OpenSSL::PKey::RSA.new(1024) + # + # # encryption client + # s3 = Aws::S3::EncryptionV3::Client.new( + # encryption_key: key, + # key_wrap_schema: :rsa_oaep_sha1 # the key_wrap_schema must be rsa_oaep_sha1 for asymmetric keys + # ) + # + # # round-trip an object, encrypted/decrypted locally + # s3.put_object(bucket:'aws-sdk', key:'secret', body:'handshake') + # s3.get_object(bucket:'aws-sdk', key:'secret').body.read + # #=> 'handshake' + # + # # reading encrypted object without the encryption client + # # results in the getting the cipher text + # Aws::S3::Client.new.get_object(bucket:'aws-sdk', key:'secret').body.read + # #=> "... cipher text ..." + # + # ## Required Configuration + # + # You must configure the following: + # + # * a key or key provider - See the Keys section below. The key provided determines + # the key wrapping schema(s) supported for both encryption and decryption. + # * `key_wrap_schema` - The key wrapping schema. It must match the type of key configured. + # + # The following have defaults and are optional: + # + # * `content_encryption_schema` - Defaults to `:alg_aes_256_gcm_hkdf_sha512_commit_key` + # * `security_profile` - Defaults to `:v3`. Set to `:v3_and_legacy` to read V2-encrypted objects. + # * `commitment_policy` - Defaults to `:require_encrypt_require_decrypt` (most secure) + # + # ## Keys + # + # For client-side encryption to work, you must provide one of the following: + # + # * An encryption key + # * A {KeyProvider} + # * A KMS encryption key id + # + # Additionally, the key wrapping schema must agree with the type of the key: + # * :aes_gcm: An AES encryption key or a key provider. + # * :rsa_oaep_sha1: An RSA encryption key or key provider. + # * :kms_context: A KMS encryption key id + # + # ### An Encryption Key + # + # You can pass a single encryption key. This is used as a master key + # encrypting and decrypting all object keys. + # + # key = OpenSSL::Cipher.new("AES-256-ECB").random_key # symmetric key - used with `key_wrap_schema: :aes_gcm` + # key = OpenSSL::PKey::RSA.new(1024) # asymmetric key pair - used with `key_wrap_schema: :rsa_oaep_sha1` + # + # s3 = Aws::S3::EncryptionV3::Client.new( + # encryption_key: key, + # key_wrap_schema: :aes_gcm # or :rsa_oaep_sha1 if using RSA + # ) + # + # ### Key Provider + # + # Alternatively, you can use a {KeyProvider}. A key provider makes + # it easy to work with multiple keys and simplifies key rotation. + # + # ### KMS Encryption Key Id + # + # If you pass the id of an AWS Key Management Service (KMS) key and + # use :kms_content for the key_wrap_schema, then KMS will be used to + # generate, encrypt and decrypt object keys. + # + # # keep track of the kms key id + # kms = Aws::KMS::Client.new + # key_id = kms.create_key.key_metadata.key_id + # + # Aws::S3::EncryptionV3::Client.new( + # kms_key_id: key_id, + # kms_client: kms, + # key_wrap_schema: :kms_context + # ) + # + # ## Custom Key Providers + # + # A {KeyProvider} is any object that responds to: + # + # * `#encryption_materials` + # * `#key_for(materials_description)` + # + # Here is a trivial implementation of an in-memory key provider. + # This is provided as a demonstration of the key provider interface, + # and should not be used in production: + # + # class KeyProvider + # + # def initialize(default_key_name, keys) + # @keys = keys + # @encryption_materials = Aws::S3::EncryptionV3::Materials.new( + # key: @keys[default_key_name], + # description: JSON.dump(key: default_key_name), + # ) + # end + # + # attr_reader :encryption_materials + # + # def key_for(matdesc) + # key_name = JSON.parse(matdesc)['key'] + # if key = @keys[key_name] + # key + # else + # raise "encryption key not found for: #{matdesc.inspect}" + # end + # end + # end + # + # Given the above key provider, you can create an encryption client that + # chooses the key to use based on the materials description stored with + # the encrypted object. This makes it possible to use multiple keys + # and simplifies key rotation. + # + # # uses "new-key" for encrypting objects, uses either for decrypting + # keys = KeyProvider.new('new-key', { + # "old-key" => Base64.decode64("kM5UVbhE/4rtMZJfsadYEdm2vaKFsmV2f5+URSeUCV4="), + # "new-key" => Base64.decode64("w1WLio3agRWRTSJK/Ouh8NHoqRQ6fn5WbSXDTHjXMSo="), + # }), + # + # # chooses the key based on the materials description stored + # # with the encrypted object + # s3 = Aws::S3::EncryptionV3::Client.new( + # key_provider: keys, + # key_wrap_schema: :aes_gcm # or :rsa_oaep_sha1 for RSA keys + # ) + # + # ## Materials Description + # + # A materials description is JSON document string that is stored + # in the metadata (or instruction file) of an encrypted object. + # The {DefaultKeyProvider} uses the empty JSON document `"{}"`. + # + # When building a key provider, you are free to store whatever + # information you need to identify the master key that was used + # to encrypt the object. + # + # ## Envelope Location + # + # By default, the encryption client store the encryption envelope + # with the object, as metadata. You can choose to have the envelope + # stored in a separate "instruction file". An instruction file + # is an object, with the key of the encrypted object, suffixed with + # `".instruction"`. + # + # Specify the `:envelope_location` option as `:instruction_file` to + # use an instruction file for storing the envelope. + # + # # default behavior + # s3 = Aws::S3::EncryptionV3::Client.new( + # encryption_key: your_key, + # key_wrap_schema: :aes_gcm, + # envelope_location: :metadata + # ) + # + # # store envelope in a separate object + # s3 = Aws::S3::EncryptionV3::Client.new( + # encryption_key: your_key, + # key_wrap_schema: :aes_gcm, + # envelope_location: :instruction_file, + # instruction_file_suffix: '.instruction' # default + # ) + # + # When using an instruction file, multiple requests are made when + # putting and getting the object. **This may cause issues if you are + # issuing concurrent PUT and GET requests to an encrypted object.** + # + # ## Commitment Policies Explained + # + # * `:forbid_encrypt_allow_decrypt` - Encrypts without key commitment (for + # backward compatibility with systems that have not been updated), but can decrypt + # objects with or without commitment. Use if you are not sure that all readers + # can decrypt objects encrypted with key commitment. + # + # * `:require_encrypt_allow_decrypt` - Encrypts with key commitment, can decrypt + # objects with or without commitment. Use once all readers + # can decrypt objects encrypted with key commitment. + # + # * `:require_encrypt_require_decrypt` - Encrypts with key commitment, can only + # decrypt objects with key commitment. **Recommended for new applications and + # after migrations are complete.** This is the default. + # + module EncryptionV3 + class Client + ##= ../specification/s3-encryption/client.md#aws-sdk-compatibility + ##= type=implication + ##% The S3EC MUST provide a different set of configuration options than the conventional S3 client. + + REQUIRED_PARAMS = [:key_wrap_schema].freeze + + OPTIONAL_PARAMS = [ + :kms_key_id, + :kms_client, + :key_provider, + :encryption_key, + :envelope_location, + ##= ../specification/s3-encryption/client.md#instruction-file-configuration + ##% In this case, the Instruction File Configuration SHOULD be optional, such that its default configuration is used when none is provided. + :instruction_file_suffix, + ##= ../specification/s3-encryption/client.md#encryption-algorithm + ##% The S3EC MUST support configuration of the encryption algorithm (or algorithm suite) during its initialization. + :content_encryption_schema, + :security_profile, + ##= ../specification/s3-encryption/client.md#key-commitment + ##% The S3EC MUST support configuration of the [Key Commitment policy](./key-commitment.md) during its initialization. + :commitment_policy + ].freeze + SUPPORTED_COMMITMENT_POLICIES = %i[ + forbid_encrypt_allow_decrypt + require_encrypt_allow_decrypt + require_encrypt_require_decrypt + ].freeze + + ##= ../specification/s3-encryption/client.md#enable-legacy-wrapping-algorithms + ##% The S3EC MUST support the option to enable or disable legacy wrapping algorithms. + ##= ../specification/s3-encryption/client.md#enable-legacy-unauthenticated-modes + ##% The S3EC MUST support the option to enable or disable legacy unauthenticated modes (content encryption algorithms). + SUPPORTED_SECURITY_PROFILES = %i[v3 v3_and_legacy].freeze + + ##= ../specification/s3-encryption/client.md#enable-legacy-unauthenticated-modes + ##% The option to enable legacy unauthenticated modes MUST be set to false by default. + ##= ../specification/s3-encryption/client.md#enable-legacy-wrapping-algorithms + ##% The option to enable legacy wrapping algorithms MUST be set to false by default. + DEFAULT_SECURITY_PROFILES = :v3 + DEFAULT_COMMITMENT_POLICIES = :require_encrypt_require_decrypt + DEFAULT_CONTENT_ENCRYPTION_SCHEMA = :alg_aes_256_gcm_hkdf_sha512_commit_key + + extend Deprecations + extend Forwardable + def_delegators :@client, :config, :delete_object, :head_object, :build_request + + # Creates a new encryption client. + # + # ## Required Configuration + # + # * a key or key provider - The key provided also determines the key wrapping + # schema(s) supported for both encryption and decryption. + # * `key_wrap_schema` - The key wrapping schema. It must match the type of key configured. + # + # ## Optional Configuration (with defaults) + # + # * `content_encryption_schema` - Defaults to `:alg_aes_256_gcm_hkdf_sha512_commit_key` + # * `security_profile` - Defaults to `:v3`. Set to `:v3_and_legacy` to read V2-encrypted objects. + # * `commitment_policy` - Defaults to `:require_encrypt_require_decrypt` (most secure) + # + # To configure the key you must provide one of the following set of options: + # + # * `:encryption_key` + # * `:kms_key_id` + # * `:key_provider` + # + # You may also pass any other options accepted by `Client#initialize`. + # + # @option options [S3::Client] :client A basic S3 client that is used + # to make api calls. If a `:client` is not provided, a new {S3::Client} + # will be constructed. + # + # @option options [OpenSSL::PKey::RSA, String] :encryption_key The master + # key to use for encrypting/decrypting all objects. + # + # @option options [String] :kms_key_id When you provide a `:kms_key_id`, + # then AWS Key Management Service (KMS) will be used to manage the + # object encryption keys. By default a {KMS::Client} will be + # constructed for KMS API calls. Alternatively, you can provide + # your own via `:kms_client`. To only support decryption/reads, you may + # provide `:allow_decrypt_with_any_cmk` which will use + # the implicit CMK associated with the data during reads but will + # not allow you to encrypt/write objects with this client. + # + # @option options [#key_for] :key_provider Any object that responds + # to `#key_for`. This method should accept a materials description + # JSON document string and return return an encryption key. + # + # @option options [required, Symbol] :key_wrap_schema The Key wrapping + # schema to be used. It must match the type of key configured. + # Must be one of the following: + # + # * :kms_context (Must provide kms_key_id) + # * :aes_gcm (Must provide an AES (string) key) + # * :rsa_oaep_sha1 (Must provide an RSA key) + # + # @option options [Symbol] :content_encryption_schema (:alg_aes_256_gcm_hkdf_sha512_commit_key) + # The content encryption algorithm to use. Defaults to the V3 algorithm with key commitment. + # + # @option options [Symbol] :security_profile (:v3) + # Determines the support for reading objects written using older + # encryption schemas. Must be one of the following: + # + # * :v3 - Only reads V3-encrypted objects (default, most secure) + # * :v3_and_legacy - Enables reading of V2-encrypted objects + # + # @option options [Symbol] :commitment_policy (:require_encrypt_require_decrypt) + # Determines support for key commitment. Must be one of the following: + # + # * :forbid_encrypt_allow_decrypt - Does not encrypt with key commitment, + # can decrypt with or without. Use only for specific compatibility needs. + # * :require_encrypt_allow_decrypt - Encrypts with key commitment, can + # decrypt with or without. + # * :require_encrypt_require_decrypt - Encrypts with key commitment, only + # decrypts objects with key commitment (default, most secure) + # + # @option options [Symbol] :envelope_location (:metadata) Where to + # store the envelope encryption keys. By default, the envelope is + # stored with the encrypted object. If you pass `:instruction_file`, + # then the envelope is stored in a separate object in Amazon S3. + # + # @option options [String] :instruction_file_suffix ('.instruction') + # When `:envelope_location` is `:instruction_file` then the + # instruction file uses the object key with this suffix appended. + # + # @option options [KMS::Client] :kms_client A default {KMS::Client} + # is constructed when using KMS to manage encryption keys. + # + def initialize(options = {}) + validate_params(options) + ##= ../specification/s3-encryption/client.md#wrapped-s3-client-s + ##% The S3EC MUST support the option to provide an SDK S3 client instance during its initialization. + @client = extract_client(options) + ##= ../specification/s3-encryption/data-format/metadata-strategy.md#instruction-file + ##% Instruction File writes MUST be optionally configured during client creation or on each PutObject request. + ##= ../specification/s3-encryption/client.md#instruction-file-configuration + ##% The S3EC MAY support the option to provide Instruction File Configuration during its initialization. + ##= ../specification/s3-encryption/client.md#instruction-file-configuration + ##% If the S3EC in a given language supports Instruction Files, then it MUST accept Instruction File Configuration during its initialization. + @envelope_location = extract_location(options) + @instruction_file_suffix = extract_suffix(options) + @kms_allow_decrypt_with_any_cmk = + options[:kms_key_id] == :kms_allow_decrypt_with_any_cmk + @commitment_policy = extract_commitment_policy(options) + @security_profile = extract_security_profile(options) + + ##= ../specification/s3-encryption/client.md#key-commitment + ##% The S3EC MUST validate the configured Encryption Algorithm against the provided key commitment policy. + if @commitment_policy != :require_encrypt_require_decrypt + new_options = options.merge( + { + security_profile: security_profile_to_v2(@security_profile), + ##= ../specification/s3-encryption/client.md#key-commitment + ##% If the configured Encryption Algorithm is incompatible with the key commitment policy, then it MUST throw an exception. + content_encryption_schema: if @commitment_policy == :forbid_encrypt_allow_decrypt + options[:content_encryption_schema] + else + # assert @commitment_policy = :require_encrypt_allow_decrypt + # In this case the v2_cipher_provider is only used for decrypt + :aes_gcm_no_padding + end + } + ) + @v2_cipher_provider = build_v2_cipher_provider_for_decrypt(new_options) + # In this case the v3 cipher is only used for decrypt. + @v3_cipher_provider = build_cipher_provider(options.reject { |k, _| k == :content_encryption_schema }) + @key_provider = @v2_cipher_provider.key_provider if @v2_cipher_provider.is_a?(DefaultCipherProvider) + else + @v3_cipher_provider = build_cipher_provider(options) + @key_provider = @v3_cipher_provider.key_provider if @v3_cipher_provider.is_a?(DefaultCipherProvider) + end + end + + # @return [S3::Client] + attr_reader :client + + # @return [KeyProvider, nil] Returns `nil` if you are using + # AWS Key Management Service (KMS). + attr_reader :key_provider + + # @return [Symbol] Determines the support for reading objects written + # using older key wrap or content encryption schemas. + attr_reader :commitment_policy + + # @return [Boolean] If true the provided KMS key_id will not be used + # during decrypt, allowing decryption with the key_id from the object. + attr_reader :kms_allow_decrypt_with_any_cmk + + # @return [Symbol<:metadata, :instruction_file>] + attr_reader :envelope_location + + # @return [String] When {#envelope_location} is `:instruction_file`, + # the envelope is stored in the object with the object key suffixed + # by this string. + attr_reader :instruction_file_suffix + + ##= ../specification/s3-encryption/client.md#aws-sdk-compatibility + ##= type=implication + ##% The S3EC MUST adhere to the same interface for API operations as the conventional AWS SDK S3 client. + ##= ../specification/s3-encryption/client.md#aws-sdk-compatibility + ##= type=exception + ##= reason=The ruby client does not support other operations + ##% The S3EC SHOULD support invoking operations unrelated to client-side encryption e.g. + + # Uploads an object to Amazon S3, encrypting data client-side. + # See {S3::Client#put_object} for documentation on accepted + # request parameters. + # @option params [Hash] :kms_encryption_context Additional encryption + # context to use with KMS. Applies only when KMS is used. In order + # to decrypt the object you will need to provide the identical + # :kms_encryption_context to `get_object`. + # @option (see S3::Client#put_object) + # @return (see S3::Client#put_object) + # @see S3::Client#put_object + def put_object(params = {}) + kms_encryption_context = params.delete(:kms_encryption_context) + ##= ../specification/s3-encryption/client.md#required-api-operations + ##% - PutObject MUST be implemented by the S3EC. + req = @client.build_request(:put_object, params) + ##= ../specification/s3-encryption/client.md#required-api-operations + ##% - PutObject MUST encrypt its input data before it is uploaded to S3. + req.handlers.add(EncryptHandler, priority: 95) + req.context[:encryption] = { + cipher_provider: + if @commitment_policy == :forbid_encrypt_allow_decrypt + ##= ../specification/s3-encryption/key-commitment.md#commitment-policy + ##% When the commitment policy is FORBID_ENCRYPT_ALLOW_DECRYPT, the S3EC MUST NOT encrypt using an algorithm suite which supports key commitment. + @v2_cipher_provider + else + ##= ../specification/s3-encryption/key-commitment.md#commitment-policy + ##% When the commitment policy is REQUIRE_ENCRYPT_ALLOW_DECRYPT, the S3EC MUST only encrypt using an algorithm suite which supports key commitment. + ##= ../specification/s3-encryption/key-commitment.md#commitment-policy + ##% When the commitment policy is REQUIRE_ENCRYPT_REQUIRE_DECRYPT, the S3EC MUST only encrypt using an algorithm suite which supports key commitment. + @v3_cipher_provider + end, + envelope_location: @envelope_location, + instruction_file_suffix: @instruction_file_suffix, + kms_encryption_context: kms_encryption_context + } + Aws::Plugins::UserAgent.metric('S3_CRYPTO_V3') do + req.send_request + end + end + + # Gets an object from Amazon S3, decrypting data locally. + # See {S3::Client#get_object} for documentation on accepted + # request parameters. + # Warning: If you provide a block to get_object or set the request + # parameter :response_target to a Proc, then read the entire object to the + # end before you start using the decrypted data. This is to verify that + # the object has not been modified since it was encrypted. + # + # @option options [Symbol] :security_profile + # Determines the support for reading objects written using older + # encryption schemas. Overrides the value set on client construction if provided. + # Must be one of the following: + # + # * :v3 - Only reads V3-encrypted objects (most secure) + # * :v3_and_legacy - Enables reading of V2-encrypted objects + # @option params [String] :instruction_file_suffix The suffix + # used to find the instruction file containing the encryption + # envelope. You should not set this option when the envelope + # is stored in the object metadata. Defaults to + # {#instruction_file_suffix}. + # @option params [Hash] :kms_encryption_context Additional encryption + # context to use with KMS. Applies only when KMS is used. + # @option options [Boolean] :kms_allow_decrypt_with_any_cmk (false) + # By default the KMS CMK ID (kms_key_id) will be used during decrypt + # and will fail if there is a mismatch. Setting this to true + # will use the implicit CMK associated with the data. + # @option (see S3::Client#get_object) + # @return (see S3::Client#get_object) + # @see S3::Client#get_object + # @note The `:range` request parameter is not supported. + def get_object(params = {}, &block) + raise NotImplementedError, '#get_object with :range not supported' if params[:range] + + envelope_location, instruction_file_suffix = envelope_options(params) + kms_encryption_context = params.delete(:kms_encryption_context) + kms_any_cmk_mode = kms_any_cmk_mode(params) + commitment_policy = commitment_policy_from_params(params) + + ##= ../specification/s3-encryption/client.md#required-api-operations + ##% - GetObject MUST be implemented by the S3EC. + req = @client.build_request(:get_object, params) + ##= ../specification/s3-encryption/client.md#required-api-operations + ##% - GetObject MUST decrypt data received from the S3 server and return it as plaintext. + req.handlers.add(DecryptHandler) + req.context[:encryption] = { + v3_cipher_provider: @v3_cipher_provider, + envelope_location: envelope_location, + instruction_file_suffix: instruction_file_suffix, + kms_encryption_context: kms_encryption_context, + kms_allow_decrypt_with_any_cmk: kms_any_cmk_mode, + commitment_policy: commitment_policy + }.tap do |hash| + if commitment_policy != :require_encrypt_require_decrypt + security_profile = security_profile_from_params(params) + hash[:security_profile] = security_profile_to_v2(security_profile) + hash[:cipher_provider] = @v2_cipher_provider + end + end + Aws::Plugins::UserAgent.metric('S3_CRYPTO_V3') do + req.send_request(target: block) + end + end + + ##= ../specification/s3-encryption/data-format/metadata-strategy.md#instruction-file + ##= type=exception + ##= reason=This has never been supported in Ruby + ##% The S3EC MAY support re-encryption/key rotation via Instruction Files. + ##= ../specification/s3-encryption/decryption.md#ranged-gets + ##= type=exception + ##= reason=This has never been supported in Ruby + ##% The S3EC MAY support the "range" parameter on GetObject which specifies a subset of bytes to download and decrypt. + ##= ../specification/s3-encryption/decryption.md#ranged-gets + ##= type=exception + ##= reason=This has never been supported in Ruby + ##% If the S3EC supports Ranged Gets, the S3EC MUST adjust the customer-provided range to include the beginning and end of the cipher blocks for the given range. + ##= ../specification/s3-encryption/decryption.md#ranged-gets + ##= type=exception + ##= reason=This has never been supported in Ruby + ##% If the object was encrypted with ALG_AES_256_GCM_IV12_TAG16_NO_KDF, then ALG_AES_256_CTR_IV16_TAG16_NO_KDF MUST be used to decrypt the range of the object. + ##= ../specification/s3-encryption/decryption.md#ranged-gets + ##= type=exception + ##= reason=This has never been supported in Ruby + ##% If the object was encrypted with ALG_AES_256_GCM_HKDF_SHA512_COMMIT_KEY, then ALG_AES_256_CTR_HKDF_SHA512_COMMIT_KEY MUST be used to decrypt the range of the object. + ##= ../specification/s3-encryption/decryption.md#ranged-gets + ##= type=exception + ##= reason=This has never been supported in Ruby + ##% If the GetObject response contains a range, but the GetObject request does not contain a range, the S3EC MUST throw an exception. + + private + + def build_cipher_provider(options) + if options[:kms_key_id] + KmsCipherProvider.new( + kms_key_id: options[:kms_key_id], + kms_client: kms_client(options), + key_wrap_schema: options[:key_wrap_schema], + content_encryption_schema: options[:content_encryption_schema] + ) + else + ##= ../specification/s3-encryption/client.md#cryptographic-materials + ##% The S3EC MAY accept key material directly. + key_provider = extract_key_provider(options) + DefaultCipherProvider.new( + key_provider: key_provider, + key_wrap_schema: options[:key_wrap_schema], + content_encryption_schema: options[:content_encryption_schema] + ) + end + end + + def build_v2_cipher_provider_for_decrypt(options) + if options[:kms_key_id] + Aws::S3::EncryptionV2::KmsCipherProvider.new( + kms_key_id: options[:kms_key_id], + kms_client: kms_client(options), + key_wrap_schema: options[:key_wrap_schema], + content_encryption_schema: options[:content_encryption_schema] + ) + else + # Create V2 key provider explicitly for proper namespace consistency + key_provider = if options[:key_provider] + options[:key_provider] + elsif options[:encryption_key] + Aws::S3::EncryptionV2::DefaultKeyProvider.new(options) + else + msg = 'you must pass a :kms_key_id, :key_provider, or :encryption_key' + raise ArgumentError, msg + end + Aws::S3::EncryptionV2::DefaultCipherProvider.new( + key_provider: key_provider, + key_wrap_schema: options[:key_wrap_schema], + content_encryption_schema: options[:content_encryption_schema] + ) + end + end + + # Validate required parameters exist and don't conflict. + # The cek_alg and wrap_alg are passed on to the CipherProviders + # and further validated there + def validate_params(options) + unless (missing_params = REQUIRED_PARAMS - options.keys).empty? + raise ArgumentError, 'Missing required parameter(s): '\ + "#{missing_params.map { |s| ":#{s}" }.join(', ')}" + end + + wrap_alg = options[:key_wrap_schema] + + # validate that the wrap alg matches the type of key given + case wrap_alg + when :kms_context + raise ArgumentError, 'You must provide :kms_key_id to use :kms_context' unless options[:kms_key_id] + end + end + + def extract_client(options) + ##= ../specification/s3-encryption/client.md#wrapped-s3-client-s + ##= type=exception + ##= reason=this would be a breaking change to ruby + ##% The S3EC MUST NOT support use of S3EC as the provided S3 client during its initialization; it MUST throw an exception in this case. + options[:client] || begin + ##= ../specification/s3-encryption/client.md#inherited-sdk-configuration + ##% The S3EC MAY support directly configuring the wrapped SDK clients through its initialization. + ##= ../specification/s3-encryption/client.md#inherited-sdk-configuration + ##% For example, the S3EC MAY accept a credentials provider instance during its initialization. + ##= ../specification/s3-encryption/client.md#inherited-sdk-configuration + ##% If the S3EC accepts SDK client configuration, the configuration MUST be applied to all wrapped S3 clients. + S3::Client.new(extract_sdk_options(options)) + end + end + + def kms_client(options) + options[:kms_client] || (@kms_client ||= + KMS::Client.new( + # extract the region and credentials first, if they are not configured, then getting them from an existing client is faster + ##= ../specification/s3-encryption/client.md#inherited-sdk-configuration + ##% If the S3EC accepts SDK client configuration, the configuration MUST be applied to all wrapped SDK clients including the KMS client. + { + region: @client.config.region, + credentials: @client.config.credentials + }.merge(extract_sdk_options(options)) + ) + ) + end + + def extract_sdk_options(options) + options = options.dup + OPTIONAL_PARAMS.each { |p| options.delete(p) } + REQUIRED_PARAMS.each { |p| options.delete(p) } + options + end + + def extract_key_provider(options) + if options[:key_provider] + options[:key_provider] + elsif options[:encryption_key] + DefaultKeyProvider.new(options) + else + msg = 'you must pass a :kms_key_id, :key_provider, or :encryption_key' + raise ArgumentError, msg + end + end + + def envelope_options(params) + location = params.delete(:envelope_location) || @envelope_location + suffix = params.delete(:instruction_file_suffix) + if suffix + ##= ../specification/s3-encryption/data-format/metadata-strategy.md#instruction-file + ##% The S3EC SHOULD support providing a custom Instruction File suffix on GetObject requests, regardless of whether or not re-encryption is supported. + [:instruction_file, suffix] + else + [location, @instruction_file_suffix] + end + end + + def extract_location(options) + ##= ../specification/s3-encryption/data-format/metadata-strategy.md#object-metadata + ##% By default, the S3EC MUST store content metadata in the S3 Object Metadata. + ##= ../specification/s3-encryption/data-format/metadata-strategy.md#instruction-file + ##% Instruction File writes MUST NOT be enabled by default. + location = options[:envelope_location] || :metadata + if %i[metadata instruction_file].include?(location) + location + else + msg = ':envelope_location must be :metadata or :instruction_file '\ + "got #{location.inspect}" + raise ArgumentError, msg + end + end + + def extract_suffix(options) + ##= ../specification/s3-encryption/data-format/metadata-strategy.md#instruction-file + ##% The default Instruction File behavior uses the same S3 object key as its associated object suffixed with ".instruction". + suffix = options[:instruction_file_suffix] || '.instruction' + if suffix.is_a? String + ##= ../specification/s3-encryption/data-format/metadata-strategy.md#instruction-file + ##= type=exception + ##= reason=Ruby has always supported this option + ##% The S3EC MUST NOT support providing a custom Instruction File suffix on ordinary writes; custom suffixes MUST only be used during re-encryption. + suffix + else + msg = ':instruction_file_suffix must be a String' + raise ArgumentError, msg + end + end + + def kms_any_cmk_mode(params) + if !params[:kms_allow_decrypt_with_any_cmk].nil? + params.delete(:kms_allow_decrypt_with_any_cmk) + else + @kms_allow_decrypt_with_any_cmk + end + end + + def extract_commitment_policy(options) + validate_commitment_policy(options[:commitment_policy]) + end + + def commitment_policy_from_params(params) + commitment_policy = + if !params[:commitment_policy].nil? + params.delete(:commitment_policy) + else + @commitment_policy + end + validate_commitment_policy(commitment_policy) + end + + def validate_commitment_policy(commitment_policy) + return DEFAULT_COMMITMENT_POLICIES if commitment_policy.nil? + + unless SUPPORTED_COMMITMENT_POLICIES.include? commitment_policy + raise ArgumentError, "Unsupported security profile: :#{commitment_policy}. " \ + "Please provide one of: #{SUPPORTED_COMMITMENT_POLICIES.map { |s| ":#{s}" }.join(', ')}" + end + commitment_policy + end + + def extract_security_profile(options) + validate_security_profile(options[:security_profile]) + end + + def security_profile_from_params(params) + security_profile = + if !params[:security_profile].nil? + params.delete(:security_profile) + else + @security_profile + end + validate_security_profile(security_profile) + end + + def validate_security_profile(security_profile) + return DEFAULT_SECURITY_PROFILES if security_profile.nil? + + unless SUPPORTED_SECURITY_PROFILES.include? security_profile + raise ArgumentError, "Unsupported security profile: :#{security_profile}. " \ + "Please provide one of: #{SUPPORTED_SECURITY_PROFILES.map { |s| ":#{s}" }.join(', ')}" + end + if security_profile == :v3_and_legacy && !@warned_about_legacy + @warned_about_legacy = true + warn( + 'The S3 Encryption Client is configured to read encrypted objects ' \ + "with legacy encryption modes. If you don't have objects " \ + 'encrypted with these legacy modes, you should disable support ' \ + 'for them to enhance security.' + ) + end + security_profile + end + + def security_profile_to_v2(security_profile) + case security_profile + when :v3 + :v2 + when :v3_and_legacy + :v2_and_legacy + end + end + end + end + end +end + +##= ../specification/s3-encryption/client.md#cryptographic-materials +##= type=exception +##= reason=the ruby client does not use keyrings +##% The S3EC MUST accept either one CMM or one Keyring instance upon initialization. +##= ../specification/s3-encryption/client.md#cryptographic-materials +##= type=exception +##= reason=the ruby client does not use keyrings +##% If both a CMM and a Keyring are provided, the S3EC MUST throw an exception. +##= ../specification/s3-encryption/client.md#cryptographic-materials +##= type=exception +##= reason=the ruby client does not use keyrings +##% When a Keyring is provided, the S3EC MUST create an instance of the DefaultCMM using the provided Keyring. +##= ../specification/s3-encryption/client.md#enable-delayed-authentication +##= type=exception +##= reason=the ruby client does not support delayed authentication +##% The S3EC MUST support the option to enable or disable Delayed Authentication mode. +##= ../specification/s3-encryption/client.md#enable-delayed-authentication +##= type=exception +##= reason=the ruby client does not support delayed authentication +##% Delayed Authentication mode MUST be set to false by default. +##= ../specification/s3-encryption/client.md#enable-delayed-authentication +##= type=exception +##= reason=the ruby client does not support delayed authentication +##% When enabled, the S3EC MAY release plaintext from a stream which has not been authenticated. +##= ../specification/s3-encryption/client.md#enable-delayed-authentication +##= type=exception +##= reason=the ruby client does not support delayed authentication +##% When disabled the S3EC MUST NOT release plaintext from a stream which has not been authenticated. +##= ../specification/s3-encryption/client.md#set-buffer-size +##= type=exception +##= reason=the ruby client does not support delayed authentication +##% The S3EC SHOULD accept a configurable buffer size which refers to the maximum ciphertext length in bytes to store in memory when Delayed Authentication mode is disabled. +##= ../specification/s3-encryption/client.md#set-buffer-size +##= type=exception +##= reason=the ruby client does not support delayed authentication +##% If Delayed Authentication mode is enabled, and the buffer size has been set to a value other than its default, the S3EC MUST throw an exception. +##= ../specification/s3-encryption/client.md#set-buffer-size +##= type=exception +##= reason=the ruby client does not support delayed authentication +##% If Delayed Authentication mode is disabled, and no buffer size is provided, the S3EC MUST set the buffer size to a reasonable default. +##= ../specification/s3-encryption/client.md#randomness +##= type=exception +##= reason=the ruby client does not support a configured source of randomness +##% The S3EC MAY accept a source of randomness during client initialization. +##= ../specification/s3-encryption/client.md#optional-api-operations +##= type=exception +##= reason=the ruby client does not support any additional S3 operations +##% - CreateMultipartUpload MAY be implemented by the S3EC. +##% - If implemented, CreateMultipartUpload MUST initiate a multipart upload. +##% - UploadPart MAY be implemented by the S3EC. +##% - UploadPart MUST encrypt each part. +##% - Each part MUST be encrypted in sequence. +##% - Each part MUST be encrypted using the same cipher instance for each part. +##% - CompleteMultipartUpload MAY be implemented by the S3EC. +##% - CompleteMultipartUpload MUST complete the multipart upload. +##% - AbortMultipartUpload MAY be implemented by the S3EC. +##% - AbortMultipartUpload MUST abort the multipart upload. +##% +##% The S3EC may provide implementations for the following S3EC-specific operation(s): +##% +##% - ReEncryptInstructionFile MAY be implemented by the S3EC. +##% - ReEncryptInstructionFile MUST decrypt the instruction file's encrypted data key for the given object using the client's CMM. +##% - ReEncryptInstructionFile MUST re-encrypt the plaintext data key with a provided keyring. +##= ../specification/s3-encryption/client.md#required-api-operations +##= type=exception +##= reason=the ruby client does not support the delete operation, this would be a bending change +##% - DeleteObject MUST be implemented by the S3EC. +##% - DeleteObject MUST delete the given object key. +##% - DeleteObject MUST delete the associated instruction file using the default instruction file suffix. +##% - DeleteObjects MUST be implemented by the S3EC. +##% - DeleteObjects MUST delete each of the given objects. +##% - DeleteObjects MUST delete each of the corresponding instruction files using the default instruction file suffix.
gems/aws-sdk-s3/lib/aws-sdk-s3/encryptionV3/decrypt_handler.rb+98 −0 added@@ -0,0 +1,98 @@ +# frozen_string_literal: true + +require 'base64' + +require 'logger' + +module Aws + module S3 + module EncryptionV3 + # @api private + class DecryptHandler < Seahorse::Client::Handler + @@warned_response_target_proc = false + + def call(context) + attach_http_event_listeners(context) + apply_cse_user_agent(context) + + if context[:response_target].is_a?(Proc) && !@@warned_response_target_proc + @@warned_response_target_proc = true + warn(':response_target is a Proc, or a block was provided. ' \ + 'Read the entire object to the ' \ + 'end before you start using the decrypted data. This is to ' \ + 'verify that the object has not been modified since it ' \ + 'was encrypted.') + + end + + @handler.call(context) + end + + private + + def attach_http_event_listeners(context) + context.http_response.on_headers(200) do + ##= ../specification/s3-encryption/decryption.md#key-commitment + ##% The S3EC MUST validate the algorithm suite used for decryption + ##% against the key commitment policy before attempting to decrypt the content ciphertext. + # This is because the commitment policy _always_ allows decrypting committing algorithms. + # In the else branch we check to see if + decrypter = + if Aws::S3::EncryptionV3::Decryption.v3?(context) + ##= ../specification/s3-encryption/data-format/content-metadata.md#determining-s3ec-object-status + ##% - If the metadata contains "x-amz-3" and "x-amz-d" and "x-amz-i" then the object MUST be considered an S3EC-encrypted object using the V3 format. + cipher, envelope = Aws::S3::EncryptionV3::Decryption.decryption_cipher(context) + Aws::S3::EncryptionV3::Decryption.get_decrypter(context, cipher, envelope) + else + if context[:encryption][:commitment_policy] == :require_encrypt_require_decrypt + ##= ../specification/s3-encryption/decryption.md#key-commitment + ##% If the commitment policy requires decryption using a committing algorithm suite, + ##% and the algorithm suite associated with the object does not support key commitment, then the S3EC MUST throw an exception. + ##= ../specification/s3-encryption/key-commitment.md#commitment-policy + ##% When the commitment policy is REQUIRE_ENCRYPT_REQUIRE_DECRYPT, the S3EC MUST NOT allow decryption using algorithm suites which do not support key commitment. + raise Errors::NonCommittingDecryptionError + end + + ##= ../specification/s3-encryption/key-commitment.md#commitment-policy + ##% When the commitment policy is FORBID_ENCRYPT_ALLOW_DECRYPT, the S3EC MUST allow decryption using algorithm suites which do not support key commitment. + ##= ../specification/s3-encryption/key-commitment.md#commitment-policy + ##% When the commitment policy is REQUIRE_ENCRYPT_ALLOW_DECRYPT, the S3EC MUST allow decryption using algorithm suites which do not support key commitment. + cipher, envelope = Aws::S3::EncryptionV2::Decryption.decryption_cipher(context) + Aws::S3::EncryptionV2::Decryption.get_decrypter(context, cipher, envelope) + end + context.http_response.body = decrypter + end + + context.http_response.on_success(200) do + decrypter = context.http_response.body + decrypter.finalize + decrypter.io.rewind if decrypter.io.respond_to?(:rewind) + context.http_response.body = decrypter.io + end + + context.http_response.on_error do + context.http_response.body = context.http_response.body.io if context.http_response.body.respond_to?(:io) + end + end + + def apply_cse_user_agent(context) + if context.config.user_agent_suffix.nil? + context.config.user_agent_suffix = EC_USER_AGENT + elsif !context.config.user_agent_suffix.include? EC_USER_AGENT + context.config.user_agent_suffix += " #{EC_USER_AGENT}" + end + end + end + end + end +end + +##= ../specification/s3-encryption/data-format/content-metadata.md#v3-only +##= type=exception +##= reason=This has never been supported in Ruby +##% This material description string MAY be encoded by the esoteric double-encoding scheme used by the S3 web server. + +##= ../specification/s3-encryption/data-format/content-metadata.md#v3-only +##= type=exception +##= reason=This has never been supported in Ruby +##% This encryption context string MAY be encoded by the esoteric double-encoding scheme used by the S3 web server.
gems/aws-sdk-s3/lib/aws-sdk-s3/encryptionV3/decryption.rb+244 −0 added@@ -0,0 +1,244 @@ +# frozen_string_literal: true + +require 'base64' + +module Aws + module S3 + module EncryptionV3 + # @api private + class Decryption + ##= ../specification/s3-encryption/data-format/content-metadata.md#content-metadata-mapkeys + ##= type=implication + ##% The "x-amz-" prefix denotes that the metadata is owned by an Amazon product and MUST be prepended to all S3EC metadata mapkeys. + + ##= ../specification/s3-encryption/data-format/content-metadata.md#content-metadata-mapkeys + ##% - The mapkey "x-amz-3" MUST be present for V3 format objects. + ##= ../specification/s3-encryption/data-format/content-metadata.md#content-metadata-mapkeys + ##% - The mapkey "x-amz-w" MUST be present for V3 format objects. + ##= ../specification/s3-encryption/data-format/content-metadata.md#content-metadata-mapkeys + ##= type=implication + ##% - This mapkey ("x-amz-3") SHOULD be represented by a constant named "ENCRYPTED_DATA_KEY_V3" or similar in the implementation code. + ##= ../specification/s3-encryption/data-format/content-metadata.md#content-metadata-mapkeys + ##= type=implication + ##% - This mapkey ("x-amz-w") SHOULD be represented by a constant named "ENCRYPTED_DATA_KEY_ALGORITHM_V3" or similar in the implementation code. + ENVELOP_KEY = %w[ + x-amz-3 + x-amz-w + ].freeze + + ##= ../specification/s3-encryption/data-format/content-metadata.md#content-metadata-mapkeys + ##% - The mapkey "x-amz-m" SHOULD be present for V3 format objects that use Raw Keyring Material Description. + ##= ../specification/s3-encryption/data-format/content-metadata.md#content-metadata-mapkeys + ##% - The mapkey "x-amz-t" SHOULD be present for V3 format objects that use KMS Encryption Context. + ##= ../specification/s3-encryption/data-format/content-metadata.md#content-metadata-mapkeys + ##= type=implication + ##% - This mapkey ("x-amz-m") SHOULD be represented by a constant named "MAT_DESC_V3" or similar in the implementation code. + ##= ../specification/s3-encryption/data-format/content-metadata.md#content-metadata-mapkeys + ##= type=implication + ##% - This mapkey ("x-amz-t") SHOULD be represented by a constant named "ENCRYPTION_CONTEXT_V3" or similar in the implementation code. + OPTIONAL_ENVELOP_KEY = %w[ + x-amz-m + x-amz-t + ].freeze + + ##= ../specification/s3-encryption/data-format/content-metadata.md#content-metadata-mapkeys + ##% - The mapkey "x-amz-c" MUST be present for V3 format objects. + ##= ../specification/s3-encryption/data-format/content-metadata.md#content-metadata-mapkeys + ##% - The mapkey "x-amz-d" MUST be present for V3 format objects. + ##= ../specification/s3-encryption/data-format/content-metadata.md#content-metadata-mapkeys + ##% - The mapkey "x-amz-i" MUST be present for V3 format objects. + ##= ../specification/s3-encryption/data-format/content-metadata.md#content-metadata-mapkeys + ##= type=implication + ##% - This mapkey ("x-amz-c") SHOULD be represented by a constant named "CONTENT_CIPHER_V3" or similar in the implementation code. + ##= ../specification/s3-encryption/data-format/content-metadata.md#content-metadata-mapkeys + ##= type=implication + ##% - This mapkey ("x-amz-d") SHOULD be represented by a constant named "KEY_COMMITMENT_V3" or similar in the implementation code. + ##= ../specification/s3-encryption/data-format/content-metadata.md#content-metadata-mapkeys + ##= type=implication + ##% - This mapkey ("x-amz-i") SHOULD be represented by a constant named "MESSAGE_ID_V3" or similar in the implementation code. + METADATA_KEY = %w[ + x-amz-c + x-amz-d + x-amz-i + ].freeze + + # Reference V2's envelope keys rather than duplicating them + LEGACY_POSSIBLE_ENVELOPE_KEYS = Aws::S3::EncryptionV2::Decryption::POSSIBLE_ENVELOPE_KEYS + + POSSIBLE_ENVELOPE_KEYS = (ENVELOP_KEY + METADATA_KEY + OPTIONAL_ENVELOP_KEY + LEGACY_POSSIBLE_ENVELOPE_KEYS).uniq + REQUIRED_ENVELOPE_KEYS = (ENVELOP_KEY + METADATA_KEY).uniq + + POSSIBLE_WRAPPING_FORMATS = %w[ + 01 + 02 + 11 + 12 + 21 + 22 + ].freeze + + POSSIBLE_ENCRYPTION_FORMATS = %w[ + 115 + ].freeze + + class << self + def v3?(context) + context.http_response.headers.key?('x-amz-meta-x-amz-i') + end + + def decryption_cipher(context) + if (envelope = get_encryption_envelope(context)) + cipher = context[:encryption][:v3_cipher_provider] + .decryption_cipher( + envelope, + context[:encryption] + ) + [cipher, envelope] + else + raise Errors::DecryptionError, 'unable to locate encryption envelope' + end + end + + # This method fetches the tag from the end of the object by + # making a GET Object w/range request. This auth tag is used + # to initialize the cipher, and the decrypter truncates the + # auth tag from the body when writing the final bytes. + def get_decrypter(context, cipher, _envelope) + http_resp = context.http_response + content_length = http_resp.headers['content-length'].to_i + + # The encrypted object contains both the cipher text + # plus a trailing auth tag. + # The trailing auth tag will be accumulated and added to the cipher.auth_tag. + IOAuthDecrypter.new( + io: http_resp.body, + encrypted_content_length: content_length - AES_GCM_TAG_LEN_BYTES, + cipher: cipher + ) + end + + def get_encryption_envelope(context) + # Get initial envelope data from :envelope_location + envelope = + if context[:encryption][:envelope_location] == :metadata + envelope_from_metadata(context) + else + envelope_from_instr_file(context) + end + + # If empty or incomplete, get/merge data from secondary source + ##= ../specification/s3-encryption/data-format/content-metadata.md#determining-s3ec-object-status + ##% If the object matches none of the V1/V2/V3 formats, the S3EC MUST attempt to get the instruction file. + if envelope.nil? || envelope.empty? || !complete_envelop?(envelope) + secondary = + if context[:encryption][:envelope_location] == :metadata + envelope_from_instr_file(context) + else + envelope_from_metadata(context) + end + # If we attempted to read a non-existent instruction file, + # then envelope would be nil, + # but we may find the information we need in the metadata. + if envelope && secondary + envelope.merge!(secondary) + elsif secondary + envelope = secondary + end + end + + ##= ../specification/s3-encryption/data-format/metadata-strategy.md#object-metadata + ##% If the S3EC does not support decoding the S3 Server's "double encoding" then it MUST return the content metadata untouched. + v3_envelope?(envelope) + end + + def complete_envelop?(possible_envelope) + # V3 envelops always store some information in metadata + # If we look at the metadata, we may still need to check the instruction file + # Similarly, if we start checking the instruction file, + # we sill need to get the message id and commitment key from the metadata + envelop_count = ENVELOP_KEY.count { |key| possible_envelope.key?(key) } + metadata_count = METADATA_KEY.count { |key| possible_envelope.key?(key) } + + # If we have all keys, we are done + (envelop_count == ENVELOP_KEY.size && metadata_count == METADATA_KEY.size) || + # If we have 0 keys, then this is done too. + # Because it means we are not a v3 committing message. + (envelop_count.zero? && metadata_count.zero?) + end + + def envelope_from_metadata(context) + POSSIBLE_ENVELOPE_KEYS.filter_map do |suffix| + ##= ../specification/s3-encryption/data-format/content-metadata.md#content-metadata-mapkeys + ##= type=exception + ##= reason=Ruby is reading the headers directly + ##% The "x-amz-meta-" prefix is automatically added by the S3 server and MUST NOT be included in implementation code. + if (value = context.http_response.headers["x-amz-meta-#{suffix}"]) + ##= ../specification/s3-encryption/data-format/metadata-strategy.md#object-metadata + ##= type=exception + ##= reason=This has never been supported in Ruby + ##% The S3EC SHOULD support decoding the S3 Server's "double encoding". + + ##= ../specification/s3-encryption/data-format/metadata-strategy.md#object-metadata + ##% If the S3EC does not support decoding the S3 Server's "double encoding" then it MUST return the content metadata untouched. + [suffix, value] + end + end.to_h + end + + def envelope_from_instr_file(context) + suffix = context[:encryption][:instruction_file_suffix] + possible_envelope = Json.load(context.client.get_object( + bucket: context.params[:bucket], + key: context.params[:key] + suffix + ).body.read) + unless (keys = possible_envelope.keys & METADATA_KEY).empty? + msg = "unsupported metadata key found in instruction file: #{keys.join(', ')}" + raise Errors::DecryptionError, msg + end + possible_envelope + rescue S3::Errors::ServiceError, Json::ParseError + nil + end + + def v3_envelope?(possible_envelope) + unless (keys = possible_envelope.keys & LEGACY_POSSIBLE_ENVELOPE_KEYS).empty? + ##= ../specification/s3-encryption/data-format/content-metadata.md#determining-s3ec-object-status + ##% If there are multiple mapkeys which are meant to be exclusive, such as "x-amz-key", "x-amz-key-v2", and "x-amz-3" then the S3EC SHOULD throw an exception. + msg = "legacy metadata key found: #{keys.join(', ')}" + raise Errors::DecryptionError, msg + end + + unless POSSIBLE_ENCRYPTION_FORMATS.include? possible_envelope['x-amz-c'] + alg = possible_envelope['x-amz-c'].inspect + msg = "unsupported content encrypting key (cek) format: #{alg} #{possible_envelope.inspect}" + raise Errors::DecryptionError, msg + end + unless POSSIBLE_WRAPPING_FORMATS.include? possible_envelope['x-amz-w'] + alg = possible_envelope['x-amz-w'].inspect + msg = "unsupported key wrapping algorithm: #{alg}" + raise Errors::DecryptionError, msg + end + unless (missing_keys = REQUIRED_ENVELOPE_KEYS - possible_envelope.keys).empty? + ##= ../specification/s3-encryption/data-format/content-metadata.md#determining-s3ec-object-status + ##% In general, if there is any deviation from the above format, with the exception of additional unrelated mapkeys, then the S3EC SHOULD throw an exception. + msg = "incomplete v3 encryption envelope:\n" + msg += " missing: #{missing_keys.join(',')}\n" + raise Errors::DecryptionError, msg + end + possible_envelope + end + end + end + end + end +end + +##= ../specification/s3-encryption/data-format/content-metadata.md#v3-only +##= type=exception +##= reason=This has never been supported in Ruby +##% This material description string MAY be encoded by the esoteric double-encoding scheme used by the S3 web server. + +##= ../specification/s3-encryption/data-format/content-metadata.md#v3-only +##= type=exception +##= reason=This has never been supported in Ruby +##% This encryption context string MAY be encoded by the esoteric double-encoding scheme used by the S3 web server.
gems/aws-sdk-s3/lib/aws-sdk-s3/encryptionV3/default_cipher_provider.rb+159 −0 added@@ -0,0 +1,159 @@ +# frozen_string_literal: true + +require 'base64' + +module Aws + module S3 + module EncryptionV3 + # @api private + class DefaultCipherProvider + def initialize(options = {}) + @key_provider = options[:key_provider] + @key_wrap_schema = validate_key_wrap( + options[:key_wrap_schema], + @key_provider.encryption_materials.key + ) + ##= ../specification/s3-encryption/encryption.md#content-encryption + ##% The S3EC MUST use the encryption algorithm configured during [client](./client.md) initialization. + @content_encryption_schema = Utils.validate_cek( + options[:content_encryption_schema] + ) + end + + attr_reader :key_provider + + # @return [Array<Hash,Cipher>] Creates an returns a new encryption + # envelope and encryption cipher. + def encryption_cipher(options = {}) + validate_options(options) + data_key = Utils.generate_data_key + cipher, message_id, commitment_key = Utils.generate_alg_aes_256_gcm_hkdf_sha512_commit_key_cipher(data_key) + enc_key = + if @key_provider.encryption_materials.key.is_a? OpenSSL::PKey::RSA + encode64( + encrypt_rsa(data_key, @content_encryption_schema) + ) + else + encode64( + encrypt_aes_gcm(data_key, @content_encryption_schema) + ) + end + ##= ../specification/s3-encryption/data-format/content-metadata.md#algorithm-suite-and-message-format-version-compatibility + ##% Objects encrypted with ALG_AES_256_GCM_HKDF_SHA512_COMMIT_KEY MUST use the V3 message format version only. + envelope = { + 'x-amz-3' => enc_key, + 'x-amz-c' => @content_encryption_schema, + 'x-amz-w' => @key_wrap_schema, + ##= ../specification/s3-encryption/data-format/content-metadata.md#v3-only + ##% The Material Description MUST be used for wrapping algorithms `AES/GCM` (`02`) and `RSA-OAEP-SHA1` (`22`). + 'x-amz-m' => materials_description, + 'x-amz-d' => encode64(commitment_key), + 'x-amz-i' => encode64(message_id) + } + + [envelope, cipher] + end + + # @return [Cipher] Given an encryption envelope, returns a + # decryption cipher. + def decryption_cipher(envelope, options = {}) + validate_options(options) + wrapping_key = @key_provider.key_for(envelope['x-amz-m']) + + data_key = + case envelope['x-amz-w'] + when '02' + ##= ../specification/s3-encryption/data-format/content-metadata.md#v3-only + ##% - The wrapping algorithm value "02" MUST be translated to AES/GCM upon retrieval, and vice versa on write. + if wrapping_key.is_a? OpenSSL::PKey::RSA + raise ArgumentError, 'Key mismatch - Client is configured' \ + ' with an RSA key and the x-amz-wrap-alg is AES/GCM.' + end + Utils.decrypt_aes_gcm(wrapping_key, + decode64(envelope['x-amz-3']), + @content_encryption_schema) + when '22' + ##= ../specification/s3-encryption/data-format/content-metadata.md#v3-only + ##% - The wrapping algorithm value "22" MUST be translated to RSA-OAEP-SHA1 upon retrieval, and vice versa on write. + unless wrapping_key.is_a? OpenSSL::PKey::RSA + raise ArgumentError, 'Key mismatch - Client is configured' \ + ' with an AES key and the x-amz-wrap-alg is RSA-OAEP-SHA1.' + end + key, cek_alg = Utils.decrypt_rsa(wrapping_key, decode64(envelope['x-amz-3'])) + raise Errors::CEKAlgMismatchError unless cek_alg == @content_encryption_schema + + key + when '12' + raise ArgumentError, 'Key mismatch - Client is configured' \ + ' with a user provided key and the x-amz-w is' \ + ' kms+context. Please configure the client with the' \ + ' required kms_key_id' + else + raise ArgumentError, 'Unsupported wrapping algorithm: ' \ + "#{envelope['x-amz-w']}" + end + + message_id = decode64(envelope['x-amz-i']) + commitment_key = decode64(envelope['x-amz-d']) + + Utils.derive_alg_aes_256_gcm_hkdf_sha512_commit_key_cipher(data_key, message_id, commitment_key) + end + + private + + # Validate that the key_wrap_schema + # is valid, supported and matches the provided key. + # Returns the string version for the x-amz-key-wrap-alg + def validate_key_wrap(key_wrap_schema, key) + if key.is_a? OpenSSL::PKey::RSA + unless key_wrap_schema == :rsa_oaep_sha1 + raise ArgumentError, ':key_wrap_schema must be set to :rsa_oaep_sha1 for RSA keys.' + end + else + unless key_wrap_schema == :aes_gcm + raise ArgumentError, ':key_wrap_schema must be set to :aes_gcm for AES keys.' + end + end + + case key_wrap_schema + when :rsa_oaep_sha1 then '22' + when :aes_gcm then '02' + when :kms_context + raise ArgumentError, 'A kms_key_id is required when using :kms_context.' + else + raise ArgumentError, "Unsupported key_wrap_schema: #{key_wrap_schema}" + end + end + + def encrypt_aes_gcm(data, auth_data) + Utils.encrypt_aes_gcm(@key_provider.encryption_materials.key, data, auth_data) + end + + def encrypt_rsa(data, auth_data) + Utils.encrypt_rsa(@key_provider.encryption_materials.key, data, auth_data) + end + + def materials_description + ##= ../specification/s3-encryption/data-format/content-metadata.md#v3-only + ##% If the mapkey x-amz-m is not present, the default Material Description value MUST be set to an empty map (`{}`). + @key_provider.encryption_materials.description || {} + end + + def encode64(str) + Base64.encode64(str).split("\n") * '' + end + + def decode64(str) + Base64.decode64(str) + end + + def validate_options(options) + return if options[:kms_encryption_context].nil? + + raise ArgumentError, 'Cannot provide :kms_encryption_context ' \ + 'with non KMS client.' + end + end + end + end +end
gems/aws-sdk-s3/lib/aws-sdk-s3/encryptionV3/default_key_provider.rb+35 −0 added@@ -0,0 +1,35 @@ +# frozen_string_literal: true + +module Aws + module S3 + module EncryptionV3 + # The default key provider is constructed with a single key + # that is used for both encryption and decryption, ignoring + # the possible per-object envelope encryption materials description. + # @api private + class DefaultKeyProvider + include KeyProvider + + # @option options [required, OpenSSL::PKey::RSA, String] :encryption_key + # The master key to use for encrypting objects. + # @option options [String<JSON>] :materials_description ('{}') + # A description of the encryption key. + def initialize(options = {}) + @encryption_materials = Materials.new( + key: options[:encryption_key], + description: options[:materials_description] || '{}' + ) + end + + # @return [Materials] + attr_reader :encryption_materials + + # @param [String<JSON>] materials_description + # @return Returns the key given in the constructor. + def key_for(_materials_description) + @encryption_materials.key + end + end + end + end +end
gems/aws-sdk-s3/lib/aws-sdk-s3/encryptionV3/encrypt_handler.rb+98 −0 added@@ -0,0 +1,98 @@ +# frozen_string_literal: true + +require 'base64' + +module Aws + module S3 + module EncryptionV3 + # @api private + class EncryptHandler < Seahorse::Client::Handler + def call(context) + envelope, cipher = context[:encryption][:cipher_provider] + .encryption_cipher( + kms_encryption_context: context[:encryption][:kms_encryption_context] + ) + context[:encryption][:cipher] = cipher + apply_encryption_envelope(context, envelope) + apply_encryption_cipher(context, cipher) + apply_cse_user_agent(context) + @handler.call(context) + end + + private + + def apply_encryption_envelope(context, envelope) + ##= ../specification/s3-encryption/data-format/metadata-strategy.md#instruction-file + ##% The S3EC MUST support writing some or all (depending on format) content metadata to an Instruction File. + if context[:encryption][:envelope_location] == :instruction_file + suffix = context[:encryption][:instruction_file_suffix] + instruction_envelop, metadata_envelop = split_for_instruction_file(envelope) + + context.client.put_object( + bucket: context.params[:bucket], + key: context.params[:key] + suffix, + ##= ../specification/s3-encryption/data-format/metadata-strategy.md#instruction-file + ##% The content metadata stored in the Instruction File MUST be serialized to a JSON string. + ##% The serialized JSON string MUST be the only contents of the Instruction File. + body: Json.dump(instruction_envelop) + ) + context.params[:metadata] ||= {} + context.params[:metadata].update(metadata_envelop) + else + context.params[:metadata] ||= {} + context.params[:metadata].update(envelope) + end + end + + def apply_encryption_cipher(context, cipher) + io = context.params[:body] || '' + io = StringIO.new(io) if io.is_a? String + context.params[:body] = IOEncrypter.new(cipher, io) + context.params[:metadata] ||= {} + # Leaving this in because even though this is years old + # it is still important to *not* MD5 the plaintext + # If there exists any old integration points still doing this + # that upgrade from 1 to 3 this needs to still fail. + if context.params.delete(:content_md5) + raise ArgumentError, 'Setting content_md5 on client side '\ + 'encrypted objects is deprecated.' + end + context.http_response.on_headers do + context.params[:body].close + end + end + + def apply_cse_user_agent(context) + if context.config.user_agent_suffix.nil? + context.config.user_agent_suffix = EC_USER_AGENT + elsif !context.config.user_agent_suffix.include? EC_USER_AGENT + context.config.user_agent_suffix += " #{EC_USER_AGENT}" + end + end + + def split_for_instruction_file(envelop) + ##= ../specification/s3-encryption/data-format/content-metadata.md#content-metadata-mapkeys + ##% In the V3 format, the mapkeys "x-amz-c", "x-amz-d", and "x-amz-i" MUST be stored exclusively in the Object Metadata. + ##= ../specification/s3-encryption/data-format/metadata-strategy.md#v3-instruction-files + ##% - The V3 message format MUST store the mapkey "x-amz-c" and its value in the Object Metadata when writing with an Instruction File. + ##% - The V3 message format MUST NOT store the mapkey "x-amz-c" and its value in the Instruction File. + ##% - The V3 message format MUST store the mapkey "x-amz-d" and its value in the Object Metadata when writing with an Instruction File. + ##% - The V3 message format MUST NOT store the mapkey "x-amz-d" and its value in the Instruction File. + ##% - The V3 message format MUST store the mapkey "x-amz-i" and its value in the Object Metadata when writing with an Instruction File. + ##% - The V3 message format MUST NOT store the mapkey "x-amz-i" and its value in the Instruction File. + metadata_envelop = envelop.select { |k, _v| Decryption::METADATA_KEY.include?(k) } + # Exclude the metadata keys rather than include the envelop keys + # because there might be additional information + ##= ../specification/s3-encryption/data-format/metadata-strategy.md#v3-instruction-files + ##% - The V3 message format MUST store the mapkey "x-amz-3" and its value in the Instruction File. + ##% - The V3 message format MUST store the mapkey "x-amz-w" and its value in the Instruction File. + ##% - The V3 message format MUST store the mapkey "x-amz-m" and its value (when present in the content metadata) in the Instruction File. + ##% - The V3 message format MUST store the mapkey "x-amz-t" and its value (when present in the content metadata) in the Instruction File. + instruction_envelop = envelop.reject { |k, _v| Decryption::METADATA_KEY.include?(k) } + + [instruction_envelop, metadata_envelop] + end + end + end + end +end
gems/aws-sdk-s3/lib/aws-sdk-s3/encryptionV3/errors.rb+47 −0 added@@ -0,0 +1,47 @@ +# frozen_string_literal: true + +module Aws + module S3 + module EncryptionV3 + module Errors + # Generic DecryptionError + class DecryptionError < RuntimeError; end + + class EncryptionError < RuntimeError; end + + # Raised when attempting to decrypt a legacy (V1) encrypted object + # when using a security_profile that does not support it. + class NonCommittingDecryptionError < DecryptionError + def initialize(*_args) + msg = 'The requested object is ' \ + 'was not encrypted with a committing algorithm ' \ + 'and decryption is not supported under :require_encrypt_require_decrypt commitment policy. ' \ + 'Change your commitment policy to :forbid_encrypt_allow_decrypt or :require_encrypt_allow_decrypt' + super(msg) + end + end + + # Raised when attempting to decrypt a legacy (V1) encrypted object + # when using a security_profile that does not support it. + class LegacyDecryptionError < DecryptionError + def initialize(*_args) + msg = 'The requested object is ' \ + 'encrypted with V1 encryption schemas that have been disabled ' \ + 'by client configuration security_profile = :v2. Retry with ' \ + ':v2_and_legacy or re-encrypt the object.' + super(msg) + end + end + + class CEKAlgMismatchError < DecryptionError + def initialize(*_args) + msg = 'The content encryption algorithm used at encryption time ' \ + 'does not match the algorithm stored for decryption time. ' \ + 'The object may be altered or corrupted.' + super(msg) + end + end + end + end + end +end
gems/aws-sdk-s3/lib/aws-sdk-s3/encryptionV3/io_auth_decrypter.rb+60 −0 added@@ -0,0 +1,60 @@ +# frozen_string_literal: true + +module Aws + module S3 + module EncryptionV3 + # @api private + class IOAuthDecrypter + # @option options [required, IO#write] :io + # An IO-like object that responds to {#write}. + # @option options [required, Integer] :encrypted_content_length + # The number of bytes to decrypt from the `:io` object. + # This should be the total size of `:io` minus the length of + # the cipher auth tag. + # @option options [required, OpenSSL::Cipher] :cipher An initialized + # cipher that can be used to decrypt the bytes as they are + # written to the `:io` object. + def initialize(options = {}) + @decrypter = IODecrypter.new(options[:cipher], options[:io]) + @max_bytes = options[:encrypted_content_length] + @bytes_written = 0 + @cipher = options[:cipher] + @auth_tag = String.new + end + + def write(chunk) + chunk = truncate_chunk(chunk) + return unless chunk.bytesize.positive? + + @bytes_written += chunk.bytesize + @decrypter.write(chunk) + end + + def finalize + @cipher.auth_tag = @auth_tag + @decrypter.finalize + end + + def io + @decrypter.io + end + + private + + def truncate_chunk(chunk) + if chunk.bytesize + @bytes_written <= @max_bytes + chunk + elsif @bytes_written < @max_bytes + @auth_tag << chunk[@max_bytes - @bytes_written..-1] + chunk[0..(@max_bytes - @bytes_written - 1)] + else + @auth_tag << chunk + # If the tag was sent over after the full body has been read, + # we don't want to accidentally append it. + '' + end + end + end + end + end +end
gems/aws-sdk-s3/lib/aws-sdk-s3/encryptionV3/io_decrypter.rb+35 −0 added@@ -0,0 +1,35 @@ +# frozen_string_literal: true + +module Aws + module S3 + module EncryptionV3 + # @api private + class IODecrypter + # @param [OpenSSL::Cipher] cipher + # @param [IO#write] io An IO-like object that responds to `#write`. + def initialize(cipher, io) + @cipher = cipher + # Ensure that IO is reset between retries + @io = io.tap { |io| io.truncate(0) if io.respond_to?(:truncate) } + @cipher_buffer = String.new + end + + # @return [#write] + attr_reader :io + + def write(chunk) + # decrypt and write + if @cipher.method(:update).arity == 1 + @io.write(@cipher.update(chunk)) + else + @io.write(@cipher.update(chunk, @cipher_buffer)) + end + end + + def finalize + @io.write(@cipher.final) + end + end + end + end +end
gems/aws-sdk-s3/lib/aws-sdk-s3/encryptionV3/io_encrypter.rb+84 −0 added@@ -0,0 +1,84 @@ +# frozen_string_literal: true + +require 'stringio' +require 'tempfile' + +module Aws + module S3 + module EncryptionV3 + # Provides an IO wrapper encrypting a stream of data. + # @api private + class IOEncrypter + # @api private + ONE_MEGABYTE = 1024 * 1024 + + def initialize(cipher, io) + @encrypted = if io.size <= ONE_MEGABYTE + encrypt_to_stringio(cipher, io.read) + else + encrypt_to_tempfile(cipher, io) + end + @size = @encrypted.size + end + + # @return [Integer] + attr_reader :size + + def read(bytes = nil, output_buffer = nil) + if @encrypted.is_a?(Tempfile) && @encrypted.closed? + @encrypted.open + @encrypted.binmode + end + @encrypted.read(bytes, output_buffer) + end + + def rewind + @encrypted.rewind + end + + # @api private + def close + @encrypted.close if @encrypted.is_a?(Tempfile) + end + + private + + ##= ../specification/s3-encryption/encryption.md#alg-aes-256-gcm-hkdf-sha512-commit-key + ##% The client MUST append the GCM auth tag to the ciphertext if the underlying crypto provider does not do so automatically. + + def encrypt_to_stringio(cipher, plain_text) + if plain_text.empty? + StringIO.new(cipher.final + cipher.auth_tag) + else + StringIO.new(cipher.update(plain_text) + cipher.final + cipher.auth_tag) + end + end + + def encrypt_to_tempfile(cipher, io) + encrypted = Tempfile.new(object_id.to_s) + encrypted.binmode + ##= ../specification/s3-encryption/encryption.md#content-encryption + ##= type=implication + ##% The client MUST validate that the length of the plaintext bytes does not exceed the algorithm suite's cipher's maximum content length in bytes. + # The expectation is that this is handled by the underlying cryptographic provider. + # In Ruby this is OpenSSL by default. + # See OpenSSL: https://github.com/openssl/openssl/blob/master/crypto/modes/gcm128.c#L784 + # The relevant line is: + # if (mlen > ((U64(1) << 36) - 32) || (sizeof(len) == 8 && mlen < len)) + # return -1; + while (chunk = io.read(ONE_MEGABYTE, read_buffer ||= String.new)) + if cipher.method(:update).arity == 1 + encrypted.write(cipher.update(chunk)) + else + encrypted.write(cipher.update(chunk, cipher_buffer ||= String.new)) + end + end + encrypted.write(cipher.final) + encrypted.write(cipher.auth_tag) + encrypted.rewind + encrypted + end + end + end + end +end
gems/aws-sdk-s3/lib/aws-sdk-s3/encryptionV3/key_provider.rb+28 −0 added@@ -0,0 +1,28 @@ +# frozen_string_literal: true + +module Aws + module S3 + module EncryptionV3 + # This module defines the interface required for a {Client#key_provider}. + # A key provider is any object that: + # + # * Responds to {#encryption_materials} with an {Materials} object. + # + # * Responds to {#key_for}, receiving a JSON document String, + # returning an encryption key. The returned encryption key + # must be one of: + # + # * `OpenSSL::PKey::RSA` - for asymmetric encryption + # * `String` - 32, 24, or 16 bytes long, for symmetric encryption + # + module KeyProvider + # @return [Materials] + def encryption_materials; end + + # @param [String<JSON>] materials_description + # @return [OpenSSL::PKey::RSA, String] encryption_key + def key_for(materials_description); end + end + end + end +end
gems/aws-sdk-s3/lib/aws-sdk-s3/encryptionV3/kms_cipher_provider.rb+159 −0 added@@ -0,0 +1,159 @@ +# frozen_string_literal: true + +require 'base64' + +module Aws + module S3 + module EncryptionV3 + # @api private + class KmsCipherProvider + def initialize(options = {}) + @kms_key_id = validate_kms_key(options[:kms_key_id]) + @kms_client = options[:kms_client] + @key_wrap_schema = validate_key_wrap( + options[:key_wrap_schema] + ) + @content_encryption_schema = Utils.validate_cek( + options[:content_encryption_schema] + ) + end + + # @return [Array<Hash,Cipher>] Creates and returns a new encryption + # envelope and encryption cipher. + def encryption_cipher(options = {}) + validate_key_for_encryption + encryption_context = build_encryption_context(@content_encryption_schema, options) + key_data = Aws::Plugins::UserAgent.metric('S3_CRYPTO_V3') do + @kms_client.generate_data_key( + key_id: @kms_key_id, + encryption_context: encryption_context, + key_spec: 'AES_256' + ) + end + cipher, message_id, commitment_key = Utils.generate_alg_aes_256_gcm_hkdf_sha512_commit_key_cipher(key_data.plaintext) + ##= ../specification/s3-encryption/data-format/content-metadata.md#algorithm-suite-and-message-format-version-compatibility + ##% Objects encrypted with ALG_AES_256_GCM_HKDF_SHA512_COMMIT_KEY MUST use the V3 message format version only. + envelope = { + 'x-amz-3' => encode64(key_data.ciphertext_blob), + 'x-amz-c' => @content_encryption_schema, + 'x-amz-w' => @key_wrap_schema, + 'x-amz-d' => encode64(commitment_key), + 'x-amz-i' => encode64(message_id), + ##= ../specification/s3-encryption/data-format/content-metadata.md#v3-only + ##% The Encryption Context value MUST be used for wrapping algorithm `kms+context` or `12`. + 'x-amz-t' => Json.dump(encryption_context) + } + [envelope, cipher] + end + + # @return [Cipher] Given an encryption envelope, returns a + # decryption cipher. + def decryption_cipher(envelope, options = {}) + case envelope['x-amz-w'] + when '12' + cek_alg = envelope['x-amz-c'] + encryption_context = + if !envelope['x-amz-t'].nil? + Json.load(envelope['x-amz-t']) + else + ##= ../specification/s3-encryption/data-format/content-metadata.md#v3-only + ##% If the mapkey x-amz-t is not present, the default Material Description value MUST be set to an empty map (`{}`). + {} + end + ##= ../specification/s3-encryption/data-format/content-metadata.md#v3-only + ##% - The wrapping algorithm value "12" MUST be translated to kms+context upon retrieval, and vice versa on write. + raise Errors::CEKAlgMismatchError if cek_alg != encryption_context['aws:x-amz-cek-alg'] + + if encryption_context != build_encryption_context(cek_alg, options) + raise Errors::DecryptionError, 'Value of encryption context from'\ + ' envelope does not match the provided encryption context' + end + when '02' + raise ArgumentError, 'Key mismatch - Client is configured' \ + ' with a KMS key and the x-amz-wrap-alg is AES/GCM.' + when '22' + raise ArgumentError, 'Key mismatch - Client is configured' \ + ' with a KMS key and the x-amz-wrap-alg is RSA-OAEP-SHA1.' + when nil + raise ArgumentError, 'Plaintext passthrough not supported' + else + # assert !envelope['x-amz-w'].nil? + # because of the when above + raise ArgumentError, 'Unsupported wrapping algorithm: ' \ + "#{envelope['x-amz-w']}" + end + + any_cmk_mode = options[:kms_allow_decrypt_with_any_cmk] + decrypt_options = { + ciphertext_blob: decode64(envelope['x-amz-3']), + encryption_context: encryption_context + } + decrypt_options[:key_id] = @kms_key_id unless any_cmk_mode + + data_key = Aws::Plugins::UserAgent.metric('S3_CRYPTO_V3') do + @kms_client.decrypt(decrypt_options).plaintext + end + + message_id = decode64(envelope['x-amz-i']) + commitment_key = decode64(envelope['x-amz-d']) + + Utils.derive_alg_aes_256_gcm_hkdf_sha512_commit_key_cipher(data_key, message_id, commitment_key) + end + + private + + def validate_key_wrap(key_wrap_schema) + case key_wrap_schema + when :kms_context then '12' + else + raise ArgumentError, "Unsupported key_wrap_schema: #{key_wrap_schema}" + end + end + + def validate_kms_key(kms_key_id) + if kms_key_id.nil? || kms_key_id.empty? + raise ArgumentError, 'KMS CMK ID was not specified. ' \ + 'Please specify a CMK ID, ' \ + 'or set kms_key_id: :kms_allow_decrypt_with_any_cmk to use ' \ + 'any valid CMK from the object.' + end + + if kms_key_id.is_a?(Symbol) && kms_key_id != :kms_allow_decrypt_with_any_cmk + raise ArgumentError, 'kms_key_id must be a valid KMS CMK or be ' \ + 'set to :kms_allow_decrypt_with_any_cmk' + end + kms_key_id + end + + def build_encryption_context(cek_alg, options = {}) + kms_context = (options[:kms_encryption_context] || {}) + .transform_keys(&:to_s) + if kms_context.include? 'aws:x-amz-cek-alg' + raise ArgumentError, 'Conflict in reserved KMS Encryption Context ' \ + 'key aws:x-amz-cek-alg. This value is reserved for the S3 ' \ + 'Encryption Client and cannot be set by the user.' + end + { + 'aws:x-amz-cek-alg' => cek_alg + }.merge(kms_context) + end + + def encode64(str) + Base64.encode64(str).split("\n") * '' + end + + def decode64(str) + Base64.decode64(str) + end + + def validate_key_for_encryption + return unless @kms_key_id == :kms_allow_decrypt_with_any_cmk + + raise ArgumentError, 'Unable to encrypt/write objects with '\ + 'kms_key_id = :kms_allow_decrypt_with_any_cmk. Provide ' \ + 'a valid kms_key_id on client construction.' + end + end + end + end +end
gems/aws-sdk-s3/lib/aws-sdk-s3/encryptionV3/materials.rb+58 −0 added@@ -0,0 +1,58 @@ +# frozen_string_literal: true + +require 'base64' + +module Aws + module S3 + module EncryptionV3 + class Materials + # @option options [required, OpenSSL::PKey::RSA, String] :key + # The master key to use for encrypting/decrypting all objects. + # + # @option options [String<JSON>] :description ('{}') + # The encryption materials description. This is must be + # a JSON document string. + # + def initialize(options = {}) + @key = validate_key(options[:key]) + @description = validate_desc(options[:description]) + end + + # @return [OpenSSL::PKey::RSA, String] + attr_reader :key + + # @return [String<JSON>] + attr_reader :description + + private + + def validate_key(key) + case key + when OpenSSL::PKey::RSA then key + when String + if [32, 24, 16].include?(key.bytesize) + key + else + msg = 'invalid key, symmetric key required to be 16, 24, or '\ + '32 bytes in length, saw length ' + key.bytesize.to_s + raise ArgumentError, msg + end + else + msg = 'invalid encryption key, expected an OpenSSL::PKey::RSA key '\ + '(for asymmetric encryption) or a String (for symmetric '\ + 'encryption).' + raise ArgumentError, msg + end + end + + def validate_desc(description) + Json.load(description) + description + rescue Json::ParseError, EncodingError + msg = 'expected description to be a valid JSON document string' + raise ArgumentError, msg + end + end + end + end +end
gems/aws-sdk-s3/lib/aws-sdk-s3/encryption_v3.rb+24 −0 added@@ -0,0 +1,24 @@ +require 'aws-sdk-s3/encryptionV3/client' +require 'aws-sdk-s3/encryptionV3/decryption' +require 'aws-sdk-s3/encryptionV3/decrypt_handler' +require 'aws-sdk-s3/encryptionV3/default_cipher_provider' +require 'aws-sdk-s3/encryptionV3/encrypt_handler' +require 'aws-sdk-s3/encryptionV3/errors' +require 'aws-sdk-s3/encryptionV3/io_encrypter' +require 'aws-sdk-s3/encryptionV3/io_decrypter' +require 'aws-sdk-s3/encryptionV3/io_auth_decrypter' +require 'aws-sdk-s3/encryptionV3/key_provider' +require 'aws-sdk-s3/encryptionV3/kms_cipher_provider' +require 'aws-sdk-s3/encryptionV3/materials' +require 'aws-sdk-s3/encryptionV3/utils' +require 'aws-sdk-s3/encryptionV3/default_key_provider' + +module Aws + module S3 + module EncryptionV3 + AES_GCM_TAG_LEN_BYTES = 16 + EC_USER_AGENT = 'S3CryptoV3' + end + end +end +
gems/aws-sdk-s3/lib/aws-sdk-s3/encryptionV3/utils.rb+321 −0 added@@ -0,0 +1,321 @@ +# frozen_string_literal: true + +require 'openssl' + +module Aws + module S3 + module EncryptionV3 + # @api private + module Utils + class << self + ##= ../specification/s3-encryption/client.md#encryption-algorithm + ##% The S3EC MUST validate that the configured encryption algorithm is not legacy. + def validate_cek(content_encryption_schema) + ##= ../specification/s3-encryption/data-format/content-metadata.md#algorithm-suite-and-message-format-version-compatibility + ##% Objects encrypted with ALG_AES_256_GCM_HKDF_SHA512_COMMIT_KEY MUST use the V3 message format version only. + return '115' if content_encryption_schema.nil? + + case content_encryption_schema + when :alg_aes_256_gcm_hkdf_sha512_commit_key + '115' + else + ##= ../specification/s3-encryption/encryption.md#alg-aes-256-ctr-iv16-tag16-no-kdf + ##% Attempts to encrypt using AES-CTR MUST fail. + ##= ../specification/s3-encryption/encryption.md#alg-aes-256-ctr-hkdf-sha512-commit-key + ##% Attempts to encrypt using key committing AES-CTR MUST fail. + ##= ../specification/s3-encryption/client.md#encryption-algorithm + ##% If the configured encryption algorithm is legacy, then the S3EC MUST throw an exception. + ##= ../specification/s3-encryption/client.md#key-commitment + ##% If the configured Encryption Algorithm is incompatible with the key commitment policy, then it MUST throw an exception. + raise ArgumentError, "Unsupported content_encryption_schema: #{content_encryption_schema}" + end + end + + def encrypt_aes_gcm(key, data, auth_data) + cipher = aes_encryption_cipher(:GCM, key) + cipher.iv = (iv = cipher.random_iv) + cipher.auth_data = auth_data + + iv + cipher.update(data) + cipher.final + cipher.auth_tag + end + + def encrypt_rsa(key, data, auth_data) + # Plaintext must be KeyLengthInBytes (1 Byte) + DataKey + AuthData + buf = [data.bytesize] + data.unpack('C*') + auth_data.unpack('C*') + key.public_encrypt(buf.pack('C*'), OpenSSL::PKey::RSA::PKCS1_OAEP_PADDING) + end + + def decrypt_aes_gcm(key, data, auth_data) + # data is iv (12B) + key + tag (16B) + buf = data.unpack('C*') + iv = buf[0, 12].pack('C*') # iv will always be 12 bytes + tag = buf[-16, 16].pack('C*') # tag is 16 bytes + enc_key = buf[12, buf.size - (12 + 16)].pack('C*') + cipher = aes_cipher(:decrypt, :GCM, key, iv) + cipher.auth_tag = tag + cipher.auth_data = auth_data + cipher.update(enc_key) + cipher.final + end + + # returns the decrypted data + auth_data + def decrypt_rsa(key, enc_data) + # Plaintext must be KeyLengthInBytes (1 Byte) + DataKey + AuthData + buf = key.private_decrypt(enc_data, OpenSSL::PKey::RSA::PKCS1_OAEP_PADDING).unpack('C*') + key_length = buf[0] + data = buf[1, key_length].pack('C*') + auth_data = buf[key_length + 1, buf.length - key_length].pack('C*') + [data, auth_data] + end + + # @param [String] block_mode "CBC" or "ECB" + # @param [OpenSSL::PKey::RSA, String, nil] key + # @param [String, nil] iv The initialization vector + def aes_encryption_cipher(block_mode, key = nil, iv = nil) + aes_cipher(:encrypt, block_mode, key, iv) + end + + # @param [String] block_mode "CBC" or "ECB" + # @param [OpenSSL::PKey::RSA, String, nil] key + # @param [String, nil] iv The initialization vector + def aes_decryption_cipher(block_mode, key = nil, iv = nil) + aes_cipher(:decrypt, block_mode, key, iv) + end + + # @param [String] mode "encrypt" or "decrypt" + # @param [String] block_mode "CBC" or "ECB" + # @param [OpenSSL::PKey::RSA, String, nil] key + # @param [String, nil] iv The initialization vector + def aes_cipher(mode, block_mode, key, iv) + cipher = + if key + OpenSSL::Cipher.new("aes-#{cipher_size(key)}-#{block_mode.downcase}") + else + OpenSSL::Cipher.new("aes-256-#{block_mode.downcase}") + end + cipher.send(mode) # encrypt or decrypt + cipher.key = key if key + cipher.iv = iv if iv + cipher + end + + # @param [String] key + # @return [Integer] + # @raise ArgumentError + def cipher_size(key) + key.bytesize * 8 + end + + # There is only 1 supported algorithm suite at this time + ENCRYPTION_KEY_INFO = ([0x00, 0x73].pack('C*') + 'DERIVEKEY'.encode('UTF-8')).freeze + COMMITMENT_KEY_INFO = ([0x00, 0x73].pack('C*') + 'COMMITKEY'.encode('UTF-8')).freeze + + SHA512_DIGEST = OpenSSL::Digest::SHA512.new.freeze + V3_IV_BYTES = ("\x01" * 12).freeze + ALGO_ID = [0x00, 0x73].pack('C*').freeze + + def generate_alg_aes_256_gcm_hkdf_sha512_commit_key_cipher(data_key) + ##= ../specification/s3-encryption/encryption.md#content-encryption + ##% The client MUST generate an IV or Message ID using the length of the IV or Message ID defined in the algorithm suite. + message_id = Utils.generate_message_id + ##= ../specification/s3-encryption/key-derivation.md#hkdf-operation + ##% - The salt MUST be the Message ID with the length defined in the algorithm suite. + commitment_key = Utils.derive_commitment_key(data_key, message_id) + cipher = alg_aes_256_gcm_hkdf_sha512_commit_key_cipher(:encrypt, data_key, message_id) + + [ + cipher, + ##= ../specification/s3-encryption/encryption.md#content-encryption + ##% The generated IV or Message ID MUST be set or returned from the encryption process such that it can be included in the content metadata. + message_id, + ##= ../specification/s3-encryption/encryption.md#alg-aes-256-gcm-hkdf-sha512-commit-key + ##% The derived key commitment value MUST be set or returned from the encryption process such that it can be included in the content metadata. + commitment_key + ] + end + + def derive_alg_aes_256_gcm_hkdf_sha512_commit_key_cipher(data_key, message_id, stored_commitment_key) + raise Errors::DecryptionError, 'Data key length does not match algorithm suite' unless data_key.length == 32 + + raise Errors::DecryptionError, 'Message id length does not match algorithm suite' unless message_id.length == 28 + + unless stored_commitment_key.length == 28 + raise Errors::DecryptionError, 'Commitment key length does not match algorithm suite' + end + + ##= ../specification/s3-encryption/decryption.md#decrypting-with-commitment + ##= type=implication + ##% When using an algorithm suite which supports key commitment, + ##% the verification of the derived key commitment value MUST be done in constant time. + unless timing_safe_equal?( + ##= ../specification/s3-encryption/decryption.md#decrypting-with-commitment + ##% When using an algorithm suite which supports key commitment, + ##% the client MUST verify that the [derived key commitment](./key-derivation.md#hkdf-operation) contains the same bytes + ##% as the stored key commitment retrieved from the stored object's metadata. + Utils.derive_commitment_key(data_key, message_id), + stored_commitment_key + ) + ##= ../specification/s3-encryption/decryption.md#decrypting-with-commitment + ##% When using an algorithm suite which supports key commitment, + ##% the client MUST throw an exception when the derived key commitment value and stored key commitment value do not match. + raise Errors::DecryptionError, 'Commitment key verification failed' + end + + ##= ../specification/s3-encryption/decryption.md#decrypting-with-commitment + ##% When using an algorithm suite which supports key commitment, + ##% the client MUST verify the key commitment values match before deriving the [derived encryption key](./key-derivation.md#hkdf-operation). + + alg_aes_256_gcm_hkdf_sha512_commit_key_cipher(:decrypt, data_key, message_id) + end + + def alg_aes_256_gcm_hkdf_sha512_commit_key_cipher(mode, data_key, message_id) + ##= ../specification/s3-encryption/key-derivation.md#hkdf-operation + ##% The client MUST initialize the cipher, or call an AES-GCM encryption API, with the derived encryption key, an IV containing only bytes with the value 0x01, + ##% and the tag length defined in the Algorithm Suite when encrypting or decrypting with ALG_AES_256_GCM_HKDF_SHA512_COMMIT_KEY. + cipher = Utils.aes_cipher( + mode, + :GCM, + ##= ../specification/s3-encryption/encryption.md#alg-aes-256-gcm-hkdf-sha512-commit-key + ##% The client MUST use HKDF to derive the key commitment value and the derived encrypting key as described in [Key Derivation](key-derivation.md). + Utils.derive_encryption_key(data_key, message_id), + ##= ../specification/s3-encryption/key-derivation.md#hkdf-operation + ##% When encrypting or decrypting with ALG_AES_256_GCM_HKDF_SHA512_COMMIT_KEY, + ##% the IV used in the AES-GCM content encryption/decryption MUST consist entirely of bytes with the value 0x01. + ##= ../specification/s3-encryption/key-derivation.md#hkdf-operation + ##% The IV's total length MUST match the IV length defined by the algorithm suite. + V3_IV_BYTES + ) #OpenSSL::Cipher.new("aes-256-gcm") + ##= ../specification/s3-encryption/key-derivation.md#hkdf-operation + ##% The client MUST set the AAD to the Algorithm Suite ID represented as bytes. + cipher.auth_data = ALGO_ID # auth_data must be set after key and iv + cipher + end + + def generate_data_key + OpenSSL::Random.random_bytes(32) + end + + def generate_message_id + ##= ../specification/s3-encryption/encryption.md#cipher-initialization + ##= type=exception + ##= reason=This would be a new runtime error that happens randomly. + ##% The client SHOULD validate that the generated IV or Message ID is not zeros. + + ##= ../specification/s3-encryption/encryption.md#content-encryption + ##% The client MUST generate an IV or Message ID using the length of the IV or Message ID defined in the algorithm suite. + OpenSSL::Random.random_bytes(28) + end + + def derive_encryption_key(data_key, message_id) + ##= ../specification/s3-encryption/key-derivation.md#hkdf-operation + ##% - The DEK input pseudorandom key MUST be the output from the extract step. + hkdf( + data_key, + ##= ../specification/s3-encryption/key-derivation.md#hkdf-operation + ##% - The salt MUST be the Message ID with the length defined in the algorithm suite. + message_id, + ##= ../specification/s3-encryption/key-derivation.md#hkdf-operation + ##% - The input info MUST be a concatenation of the algorithm suite ID as bytes followed by the string DERIVEKEY as UTF8 encoded bytes. + ENCRYPTION_KEY_INFO, + ##= ../specification/s3-encryption/key-derivation.md#hkdf-operation + ##% - The length of the output keying material MUST equal the encryption key length specified by the algorithm suite encryption settings. + 32 + ) + end + + def derive_commitment_key(data_key, message_id) + ##= ../specification/s3-encryption/key-derivation.md#hkdf-operation + ##% - The CK input pseudorandom key MUST be the output from the extract step. + hkdf( + data_key, + message_id, + ##= ../specification/s3-encryption/key-derivation.md#hkdf-operation + ##% - The input info MUST be a concatenation of the algorithm suite ID as bytes followed by the string COMMITKEY as UTF8 encoded bytes. + COMMITMENT_KEY_INFO, + ##= ../specification/s3-encryption/key-derivation.md#hkdf-operation + ##% - The length of the output keying material MUST equal the commit key length specified by the supported algorithm suites. + 28 + ) + end + + ONE_BYTE = [1].pack('C').freeze + # assert: the following function is equivalent to `OpenSSL::KDF.hkdf` for all desired_length <= 64 + # see spec: 'produces identical output to native hkdf for random inputs (property-based test)' + def hkdf_fallback(input_key_material, salt, info, desired_length) + # Extract from RFC 5869 + # PRK = HMAC-Hash(salt, IKM) + prk = OpenSSL::HMAC.digest(SHA512_DIGEST, salt, input_key_material) + + # Expand from RFC 5869 + # N = ceil(L/HashLen) + # T = T(1) | T(2) | T(3) | ... | T(N) + # OKM = first L octets of T + # + # where: + # T(0) = empty string (zero length) + # T(1) = HMAC-Hash(PRK, T(0) | info | 0x01) + # T(2) = HMAC-Hash(PRK, T(1) | info | 0x02) + # T(3) = HMAC-Hash(PRK, T(2) | info | 0x03) + # + # L == desired_length + # HashLen == 64 (because SHA512_DIGEST is fixed) + # N = ceil(desired_length/64) + # The only supported suites have desired_length less than 64 + # This will result in a single iteration of the expand loop. + # This check verifies that it is safe to do not do a loop + raise Errors::DecryptionError, "Unsupported length: #{desired_length}" if desired_length > 64 + + # assert N == 1 + # + # For a single iteration of the loop we then get: + # OKM = first L of T(0) | T(1) + # == + # (T(0) + T(1))[0, desired_length] + # == {assert T(0) == ''} + # ('' + HMAC-Hash(PRK, '' + info + 0x01))[0, desired_length] + # == HMAC-Hash(PRK, info + 0x01)[0, desired_length] + # == {assert ONE_BYTE == 0x01} + # HMAC-Hash(PRK, info + ONE_BYTE)[0, desired_length] + # == + OpenSSL::HMAC.digest(SHA512_DIGEST, prk, info + ONE_BYTE)[0, desired_length] + end + + if defined?(OpenSSL::KDF) && OpenSSL::KDF.respond_to?(:hkdf) + def hkdf(input_key_material, salt, info, desired_length) + OpenSSL::KDF.hkdf( + ##= ../specification/s3-encryption/key-derivation.md#hkdf-operation + ##% - The input keying material MUST be the plaintext data key (PDK) generated by the key provider. + input_key_material, + salt: salt, + info: info, + ##= ../specification/s3-encryption/key-derivation.md#hkdf-operation + ##% - The length of the input keying material MUST equal the key derivation input length specified by the algorithm suite commit key derivation setting. + length: desired_length, + ##= ../specification/s3-encryption/key-derivation.md#hkdf-operation + ##% - The hash function MUST be specified by the algorithm suite commitment settings. + hash: SHA512_DIGEST + ) + end + else + # This is done so that we can test hkdf_fallback when we have `OpenSSL::KDF.hkdf` + alias hkdf hkdf_fallback + end + + if defined?(OpenSSL) && OpenSSL.respond_to?(:secure_compare) + def timing_safe_equal?(a, b) + OpenSSL.secure_compare(a, b) + end + else + def timing_safe_equal?(a, b) + return false unless a.bytesize == b.bytesize + + l = a.unpack('C*') + r = 0 + b.each_byte { |byte| r |= byte ^ l.shift } + r.zero? + end + end + end + end + end + end +end
gems/aws-sdk-s3/README.md+135 −0 added@@ -0,0 +1,135 @@ +# Amazon S3 Encryption Client for Ruby V3 + +This library provides an S3 client that supports client-side encryption. +`Aws::S3::EncryptionV3::Client` is the v3 of the Amazon S3 Encryption Client for the Ruby programming language. + +The v3 encryption client requires a minimum version of **Ruby >= 2.5**. + +Jump To: + +* [Getting Started](#getting-started) +* [Migration](#migration) + +## Maintenance and support for SDK major versions + +For information about maintenance and support for SDK major versions and their underlying dependencies, see the +following in the AWS SDKs and Tools Shared Configuration and Credentials Reference Guide: + +* [AWS SDKs and Tools Maintenance Policy](https://docs.aws.amazon.com/credref/latest/refdocs/maint-policy.html) +* [AWS SDKs and Tools Version Support Matrix](https://docs.aws.amazon.com/credref/latest/refdocs/version-support-matrix.html) + +### Ruby version support policy + +The v3 Encryption Client follows the upstream Ruby [maintenance policy](https://www.ruby-lang.org/en/downloads/branches/) +with an additional six months of support for the most recently deprecated +language version. + +**AWS reserves the right to drop support for unsupported Ruby versions earlier to +address critical security issues.** + +## Getting Started + +1. **Sign up for AWS** – Before you begin, you need to + sign up for an AWS account and retrieve your [AWS credentials][docs-signup]. +2. **Minimum requirements** – To run the SDK, your system will need to meet the + [minimum requirements][docs-requirements], including having **Ruby >= 2.5**. +3. **Install the SDK** – Using [Bundler][bundler] is the recommended way to install the + AWS SDK for Ruby. The SDK is available via [RubyGems][rubygems] under the + [`aws-sdk-s3`][install-rubygems] gem. If Bundler is installed on your system, you can add the following to your Gemfile: + + ```bash + gem 'aws-sdk-s3' + ``` + + Or install the gem directly: + + ```bash + gem install aws-sdk-s3 + ``` + + Please see the + [Installation section of the Developer Guide][docs-installation] for more + detailed information about installing the SDK. +4. **Using the SDK** – The best way to become familiar with how to use the SDK + is to read the [Developer Guide][docs-guide]. The + [Getting Started Guide][docs-quickstart] will help you become familiar with + the basic concepts. + +## Quick Examples + +### Create an Amazon S3 Encryption Client + +```ruby +require 'aws-sdk-s3' + +# Instantiate an Amazon S3 client. +s3_client = Aws::S3::Client.new( + region: 'us-west-2' +) + +# Instantiate an Amazon S3 Encryption Client V3. +client = Aws::S3::EncryptionV3::Client.new( + client: s3_client, + encryption_key: encryption_key, + key_wrap_schema: :aes_gcm +) +``` + +### Upload a file to Amazon S3 using client side encryption + +```ruby +require 'aws-sdk-s3' +require 'aws-sdk-kms' + +# Create a KMS client +kms_client = Aws::KMS::Client.new( + region: 'us-east-1' +) + +# Specify your KMS key ID +kms_key_id = 'your-kms-key-id' + +# Create the encryption client +client = Aws::S3::EncryptionV3::Client.new( + kms_key_id: kms_key_id, + kms_client: kms_client, + key_wrap_schema: :kms_context +) + +# Upload an encrypted object +bucket = 'the-bucket-name' +key = 'the-file-name' + +result = client.put_object( + bucket: bucket, + key: key, + body: File.open('file-to-encrypt.txt', 'r'), + kms_encryption_context: { 'context-key' => 'context-value' } +) +``` + +## Migration + +This version of the library supports reading encrypted objects from previous versions with extra configuration. +It also supports writing objects with non-legacy algorithms. +The list of legacy modes and operations will be provided below. + +* [2.x to 3.x Migration Guide](https://docs.aws.amazon.com/sdk-for-ruby/v3/developer-guide/s3-encryption-migration-v2-v3.html) +* [1.x to 2.x Migration Guide](https://docs.aws.amazon.com/sdk-for-ruby/v3/developer-guide/s3-encryption-migration-v1-v2.html) + +## Security + +See [CONTRIBUTING](../../../CONTRIBUTING.md#security-issue-notifications) for more information. + +## License + +This project is licensed under the Apache-2.0 License. + +[docs-signup]: https://docs.aws.amazon.com/sdk-for-ruby/v3/developer-guide/setup-config.html +[docs-requirements]: https://docs.aws.amazon.com/sdk-for-ruby/v3/developer-guide/setup-install.html +[docs-installation]: https://docs.aws.amazon.com/sdk-for-ruby/v3/developer-guide/setup-install.html +[docs-guide]: https://docs.aws.amazon.com/sdk-for-ruby/v3/developer-guide/welcome.html +[docs-quickstart]: https://docs.aws.amazon.com/sdk-for-ruby/v3/developer-guide/getting-started.html +[bundler]: https://bundler.io/ +[rubygems]: https://rubygems.org/ +[install-rubygems]: https://rubygems.org/gems/aws-sdk-s3
gems/aws-sdk-s3/S3_EC_SUPPORT_POLICY.md+21 −0 added@@ -0,0 +1,21 @@ +# Amazon S3 Encryption Client for Ruby - Support Policy + +## Overview + +This page describes the support policy for the Amazon S3 Encryption Client for Ruby. We regularly provide the Amazon S3 Encryption Client for Ruby with updates that may contain support for new or updated APIs, new features, enhancements, bug fixes, security patches, or documentation updates. Updates may also address changes with dependencies, language runtimes, and operating systems. + +We recommend users to stay up-to-date with Amazon S3 Encryption Client for Ruby releases to keep up with the latest features, security updates, and underlying dependencies. Continued use of an unsupported SDK version is not recommended and is done at the user's discretion. + +## Major Version Lifecycle + +The Amazon S3 Encryption Client for Ruby follows the same major version lifecycle as the AWS SDK. For details on this lifecycle, see [AWS SDKs and Tools Maintenance Policy](https://docs.aws.amazon.com/sdkref/latest/guide/maint-policy.html#version-life-cycle). + +## Version Support Matrix + +This table describes the current support status of each major version of the Amazon S3 Encryption Client for Ruby. It also shows the next status each major version will transition to, and the date at which that transition will happen. + +| Major version | Current status | Next status | Next status date | +|---------------|-----------------------|---------------|------------------| +| 3.x | General Availability | | | +| 2.x | General Availability | Maintenance | 2026-06-15 | +| 1.x | End of Support | | |
gems/aws-sdk-s3/spec/encryptionV2/client_spec.rb+11 −1 modified@@ -131,6 +131,9 @@ module EncryptionV2 expect(DefaultCipherProvider).to receive(:new).with( hash_including(key_provider: key_provider) ) + expect(Aws::S3::EncryptionV3::DefaultCipherProvider).to receive(:new).with( + hash_including(key_provider: key_provider) + ) Client.new(options.merge(key_provider: key_provider)) end @@ -140,6 +143,9 @@ module EncryptionV2 expect(KmsCipherProvider).to receive(:new).with( hash_including(kms_key_id: kms_key_id, kms_client: kms_client) ) + expect(Aws::S3::EncryptionV3::KmsCipherProvider).to receive(:new).with( + hash_including(kms_key_id: kms_key_id, kms_client: kms_client) + ) Client.new(options.merge(kms_key_id: kms_key_id)) end @@ -149,6 +155,9 @@ module EncryptionV2 expect(KmsCipherProvider).to receive(:new).with( hash_including(kms_key_id: kms_key_id, kms_client: kms_client) ) + expect(Aws::S3::EncryptionV3::KmsCipherProvider).to receive(:new).with( + hash_including(kms_key_id: kms_key_id, kms_client: kms_client) + ) Client.new(options.merge(kms_key_id: kms_key_id, kms_client: kms_client)) end @@ -255,7 +264,8 @@ module EncryptionV2 instruction_file_suffix: '.instruction', kms_encryption_context: nil, kms_allow_decrypt_with_any_cmk: false, - security_profile: :v2 + security_profile: :v2, + v3_cipher_provider: kind_of(Aws::S3::EncryptionV3::DefaultCipherProvider) }) end
gems/aws-sdk-s3/spec/encryptionv3/client_functional_spec.rb+990 −0 added@@ -0,0 +1,990 @@ +require_relative '../spec_helper' +require 'base64' +require 'openssl' + +module Aws + module S3 + module EncryptionV3 + + describe Client do + # Captures the data (metadata and body) put to an s3 object + def stub_put(s3_client) + data = {} + s3_client.stub_responses(:put_object, lambda { |context| + data[:metadata] = context.params[:metadata] + data[:enc_body] = context.params[:body].read + {} + }) + data + end + + # Given data from stub_put, stub a get for the same object + # during get get_object is called twice, once to get the full body and + # again with a range to get just the auth_tag + def stub_get(s3_client, data, stub_auth_tag) + resp_headers = Hash[*data[:metadata].map { |k, v| ["x-amz-meta-#{k.to_s}", v] }.flatten(1)] + resp_headers['content-length'] = data[:enc_body].length + if stub_auth_tag + auth_tag = data[:enc_body].unpack('C*')[-16, 16].pack('C*') + else + auth_tag = nil + end + s3_client.stub_responses( + :get_object, + {status_code: 200, body: data[:enc_body], headers: resp_headers}, + # The auth tag is left for legacy reasons. + # The v3 client accumulates this from the current get object. + {body: auth_tag} + ) + end + + def stub_decrypt(kms_client, opts) + kms_client.stub_responses( + :decrypt, lambda do |context| + if opts[:any_kms_key] + expect(context.params['key_id']).to be_nil + else + if opts[:raise] && context.params['key_id'] != opts[:response][:key_id] + raise Aws::KMS::Errors::IncorrectKeyException.new(context, '') + else + expect(context.params[:key_id]).to eq(opts[:response][:key_id]) + end + end + opts[:response] + end + ) + end + + let(:plaintext) { 'super secret plain text' } + let(:test_bucket) { 'test_bucket' } + let(:test_object) { 'test_object' } + + let(:s3_client) { S3::Client.new(stub_responses: true) } + + describe 'algorithm configuration' do + let(:key) { OpenSSL::Cipher.new('aes-256-gcm').random_key } + let(:options) do + { + client: s3_client, + encryption_key: key, + key_wrap_schema: :aes_gcm, + } + end + + it 'uses the configured encryption algorithm during encryption' do + ##= ../specification/s3-encryption/encryption.md#content-encryption + ##= type=test + ##% The S3EC MUST use the encryption algorithm configured during [client](./client.md) initialization. + ##= ../specification/s3-encryption/client.md#encryption-algorithm + ##= type=test + ##% The S3EC MUST support configuration of the encryption algorithm (or algorithm suite) during its initialization. + + # Test with explicitly configured HKDF algorithm (V3 default) + client_v3 = Aws::S3::EncryptionV3::Client.new( + options.merge(content_encryption_schema: :alg_aes_256_gcm_hkdf_sha512_commit_key) + ) + data_v3 = stub_put(s3_client) + client_v3.put_object(bucket: test_bucket, key: test_object, body: plaintext) + expect(data_v3[:metadata]['x-amz-c']).to eq('115') # ALG_AES_256_GCM_HKDF_SHA512_COMMIT_KEY + + # Test with forbid policy uses V2 algorithm (aes_gcm_no_padding is default for V2) + client_v2 = Aws::S3::EncryptionV3::Client.new( + options.merge( + commitment_policy: :forbid_encrypt_allow_decrypt, + content_encryption_schema: :aes_gcm_no_padding, + ) + ) + data_v2 = stub_put(s3_client) + client_v2.put_object(bucket: test_bucket, key: test_object, body: plaintext) + expect(data_v2[:metadata]['x-amz-cek-alg']).to eq('AES/GCM/NoPadding') + end + + it 'generates Message ID with correct length for encryption' do + ##= ../specification/s3-encryption/encryption.md#content-encryption + ##= type=test + ##% The client MUST generate an IV or Message ID using the length of the IV or Message ID defined in the algorithm suite. + client = Aws::S3::EncryptionV3::Client.new(options) + data = stub_put(s3_client) + client.put_object(bucket: test_bucket, key: test_object, body: plaintext) + + # Verify Message ID is present and has correct length (28 bytes base64 encoded) + expect(data[:metadata]['x-amz-i']).not_to be_nil + decoded_message_id = Base64.decode64(data[:metadata]['x-amz-i']) + expect(decoded_message_id.bytesize).to eq(28) # 224 bits for HKDF algorithm + end + + it 'includes generated Message ID in content metadata' do + ##= ../specification/s3-encryption/encryption.md#content-encryption + ##= type=test + ##% The generated IV or Message ID MUST be set or returned from the encryption process such that it can be included in the content metadata. + client = Aws::S3::EncryptionV3::Client.new(options) + data = stub_put(s3_client) + client.put_object(bucket: test_bucket, key: test_object, body: plaintext) + + # Verify Message ID is included in metadata + expect(data[:metadata]['x-amz-i']).not_to be_nil + + # Verify it can be used for decryption + stub_get(s3_client, data, true) + decrypted = client.get_object(bucket: test_bucket, key: test_object).body.read + expect(decrypted).to eq(plaintext) + end + end + + context 'when using a symmetric (AES) key' do + let(:key) do + OpenSSL::Cipher.new('aes-256-gcm').random_key + end + + let(:options) do + { + client: s3_client, + encryption_key: key, + key_wrap_schema: :aes_gcm, + } + end + + it 'can encrypt and decrypt plain text' do + client = Aws::S3::EncryptionV3::Client.new(options) + data = stub_put(s3_client) + client.put_object(bucket: test_bucket, key: test_object, body: plaintext) + expect(data[:metadata]['x-amz-c']).to eq('115') + expect(data[:metadata]['x-amz-w']).to eq('02') + + stub_get(s3_client, data, true) + decrypted = client.get_object(bucket: test_bucket, key: test_object).body.read + expect(decrypted).to eq(plaintext) + end + + it 'implements PutObject operation' do + ##= ../specification/s3-encryption/client.md#required-api-operations + ##= type=test + ##% - PutObject MUST be implemented by the S3EC. + client = Aws::S3::EncryptionV3::Client.new(options) + data = stub_put(s3_client) + + # Verify PutObject can be called without errors + expect do + client.put_object(bucket: test_bucket, key: test_object, body: plaintext) + end.not_to raise_error + + # Verify the operation completed successfully + expect(data[:enc_body]).not_to be_nil + end + + it 'encrypts data before uploading with PutObject' do + ##= ../specification/s3-encryption/client.md#required-api-operations + ##= type=test + ##% - PutObject MUST encrypt its input data before it is uploaded to S3. + client = Aws::S3::EncryptionV3::Client.new(options) + data = stub_put(s3_client) + + client.put_object(bucket: test_bucket, key: test_object, body: plaintext) + + # Verify the encrypted body is different from plaintext + expect(data[:enc_body]).not_to eq(plaintext) + + # Verify encryption metadata is present + expect(data[:metadata]['x-amz-c']).not_to be_nil + expect(data[:metadata]['x-amz-w']).not_to be_nil + expect(data[:metadata]['x-amz-d']).not_to be_nil + end + + it 'implements GetObject operation' do + ##= ../specification/s3-encryption/client.md#required-api-operations + ##= type=test + ##% - GetObject MUST be implemented by the S3EC. + client = Aws::S3::EncryptionV3::Client.new(options) + data = stub_put(s3_client) + + # First encrypt some data + client.put_object(bucket: test_bucket, key: test_object, body: plaintext) + + stub_get(s3_client, data, true) + + # Verify GetObject can be called without errors + expect do + client.get_object(bucket: test_bucket, key: test_object) + end.not_to raise_error + end + + it 'decrypts data received from S3 with GetObject' do + ##= ../specification/s3-encryption/client.md#required-api-operations + ##= type=test + ##% - GetObject MUST decrypt data received from the S3 server and return it as plaintext. + client = Aws::S3::EncryptionV3::Client.new(options) + data = stub_put(s3_client) + + # Encrypt and upload data + client.put_object(bucket: test_bucket, key: test_object, body: plaintext) + + # Verify data was encrypted (different from plaintext) + expect(data[:enc_body]).not_to eq(plaintext) + + stub_get(s3_client, data, true) + + # GetObject should return decrypted plaintext + decrypted = client.get_object(bucket: test_bucket, key: test_object).body.read + expect(decrypted).to eq(plaintext) + + # Verify we're not just returning the encrypted data + expect(decrypted).not_to eq(data[:enc_body]) + end + + it 'supports #get_object with a block and raises a warning the first time' do + client = Aws::S3::EncryptionV3::Client.new(options) + data = stub_put(s3_client) + client.put_object(bucket: test_bucket, key: test_object, body: plaintext) + + stub_get(s3_client, data, true) + expect_any_instance_of(Aws::S3::EncryptionV3::DecryptHandler).to receive(:warn) + decrypted = '' + client.get_object(bucket: test_bucket, key: test_object) do |chunk| + decrypted += chunk + end + expect(decrypted).to eq(plaintext) + + # it does not warn a second time + expect_any_instance_of(Aws::S3::EncryptionV3::DecryptHandler).not_to receive(:warn) + stub_get(s3_client, data, true) + decrypted = '' + client.get_object(bucket: test_bucket, key: test_object) do |chunk| + decrypted += chunk + end + end + + it 'can can use envelope_location: instruction_file' do + ##= ../specification/s3-encryption/data-format/metadata-strategy.md#instruction-file + ##= type=test + ##% The S3EC MUST support writing some or all (depending on format) content metadata to an Instruction File. + ##= ../specification/s3-encryption/data-format/metadata-strategy.md#instruction-file + ##= type=test + ##% Instruction File writes MUST be optionally configured during client creation or on each PutObject request. + ##= ../specification/s3-encryption/client.md#instruction-file-configuration + ##= type=test + ##% The S3EC MAY support the option to provide Instruction File Configuration during its initialization. + ##= ../specification/s3-encryption/client.md#instruction-file-configuration + ##= type=test + ##% If the S3EC in a given language supports Instruction Files, then it MUST accept Instruction File Configuration during its initialization. + ##= ../specification/s3-encryption/client.md#instruction-file-configuration + ##= type=test + ##% In this case, the Instruction File Configuration SHOULD be optional, such that its default configuration is used when none is provided. + + client = Aws::S3::EncryptionV3::Client.new( + options.merge(envelope_location: :instruction_file) + ) + data = {} + s3_client.stub_responses(:put_object, lambda { |context| + if context.params[:key].include? '.instruction' + ##= ../specification/s3-encryption/data-format/metadata-strategy.md#instruction-file + ##= type=test + ##% The content metadata stored in the Instruction File MUST be serialized to a JSON string. + ##= ../specification/s3-encryption/data-format/metadata-strategy.md#instruction-file + ##= type=test + ##% The serialized JSON string MUST be the only contents of the Instruction File. + data[:instruction_metadata] = JSON.load(context.params[:body]) + else + data[:metadata] = context.params[:metadata] + data[:enc_body] = context.params[:body].read + end + {} + }) + client.put_object(bucket: test_bucket, key: test_object, body: plaintext) + + resp_headers = Hash[*data[:metadata].map { |k, v| ["x-amz-meta-#{k.to_s}", v] }.flatten(1)] + resp_headers['content-length'] = data[:enc_body].length + + auth_tag = data[:enc_body].unpack('C*')[-16, 16].pack('C*') + + s3_client.stub_responses( + :get_object, + {status_code: 200, body: data[:enc_body], headers: resp_headers}, + {body: Json.dump(data[:instruction_metadata])}, + {body: auth_tag} + ) + decrypted = client.get_object(bucket: test_bucket, key: test_object).body.read + expect(decrypted).to eq(plaintext) + end + + it 'can can use instruction_file_suffix for a custom suffix' do + clientPut = Aws::S3::EncryptionV3::Client.new( + options.merge(envelope_location: :instruction_file, instruction_file_suffix: ".foo") + ) + data = {} + s3_client.stub_responses(:put_object, lambda { |context| + if context.params[:key].include? '.foo' + data[:instruction_metadata] = JSON.load(context.params[:body]) + else + data[:metadata] = context.params[:metadata] + data[:enc_body] = context.params[:body].read + end + {} + }) + clientPut.put_object(bucket: test_bucket, key: test_object, body: plaintext) + + clientGet = Aws::S3::EncryptionV3::Client.new( + options.merge(envelope_location: :instruction_file) + ) + + resp_headers = Hash[*data[:metadata].map { |k, v| ["x-amz-meta-#{k.to_s}", v] }.flatten(1)] + resp_headers['content-length'] = data[:enc_body].length + + auth_tag = data[:enc_body].unpack('C*')[-16, 16].pack('C*') + + s3_client.stub_responses( + :get_object, + {status_code: 200, body: data[:enc_body], headers: resp_headers}, + {body: Json.dump(data[:instruction_metadata])}, + {body: auth_tag} + ) + ##= ../specification/s3-encryption/data-format/metadata-strategy.md#instruction-file + ##= type=test + ##% The S3EC SHOULD support providing a custom Instruction File suffix on GetObject requests, regardless of whether or not re-encryption is supported. + decrypted = clientGet.get_object(bucket: test_bucket, key: test_object, instruction_file_suffix: "foo").body.read + expect(decrypted).to eq(plaintext) + end + + context 'security_profile: v3' do + it 'raises a NonCommittingDecryptionError when reading a legacy object' do + ##= ../specification/s3-encryption/decryption.md#legacy-decryption + ##= type=test + ##% If the S3EC is not configured to enable legacy unauthenticated content decryption, the client MUST throw an exception when attempting to decrypt an object encrypted with a legacy unauthenticated algorithm suite. + ##= ../specification/s3-encryption/client.md#enable-legacy-unauthenticated-modes + ##= type=test + ##% When disabled, the S3EC MUST NOT decrypt objects encrypted using legacy content encryption algorithms; it MUST throw an exception when attempting to decrypt an object encrypted with a legacy content encryption algorithm. + client_v1 = Aws::S3::Encryption::Client.new(encryption_key: key, client: s3_client) + client_v3 = Aws::S3::EncryptionV3::Client.new(options) + + data = stub_put(s3_client) + client_v1.put_object(bucket: test_bucket, key: test_object, body: plaintext) + + stub_get(s3_client, data, false) + expect do + client_v3.get_object(bucket: test_bucket, key: test_object) + end.to raise_error(Errors::NonCommittingDecryptionError) + end + end + + context 'security_profile: v3_and_legacy' do + let(:legacy_options) { options.merge( + security_profile: :v3_and_legacy, + commitment_policy: :require_encrypt_allow_decrypt + ) + } + + it 'can decrypt an object encrypted using legacy algorithm' do + ##= ../specification/s3-encryption/decryption.md#legacy-decryption + ##= type=test + ##% The S3EC MUST NOT decrypt objects encrypted using legacy unauthenticated algorithm suites unless specifically configured to do so. + ##= ../specification/s3-encryption/client.md#enable-legacy-unauthenticated-modes + ##= type=test + ##% When enabled, the S3EC MUST be able to decrypt objects encrypted with all content encryption algorithms (both legacy and fully supported). + client_v1 = Aws::S3::Encryption::Client.new(encryption_key: key, client: s3_client) + + expect_any_instance_of(Aws::S3::EncryptionV3::Client).to receive(:warn) + client_v3 = Aws::S3::EncryptionV3::Client.new(legacy_options) + + data = stub_put(s3_client) + client_v1.put_object(bucket: test_bucket, key: test_object, body: plaintext) + + stub_get(s3_client, data, false) + decrypted = client_v3.get_object(bucket: test_bucket, key: test_object).body.read + expect(decrypted).to eq(plaintext) + end + end + + it 'decrypts the object with response target under retry' do + client = Aws::S3::EncryptionV3::Client.new(options) + data = stub_put(s3_client) + s3_client.handlers.add(Aws::Plugins::RetryErrors::LegacyHandler, step: :sign, priority: 99) + + client.put_object(bucket: test_bucket, key: test_object, body: plaintext) + + resp_headers = Hash[*data[:metadata].map { |k, v| ["x-amz-meta-#{k.to_s}", v] }.flatten(1)] + resp_headers['content-length'] = data[:enc_body].length + auth_tag = data[:enc_body].unpack('C*')[-16, 16].pack('C*') + + s3_client.stub_responses( + :get_object, + Seahorse::Client::NetworkingError.new(RuntimeError.new), + {status_code: 200, body: data[:enc_body], headers: resp_headers}, + {body: auth_tag} + ) + decrypted = StringIO.new + client.get_object(bucket: test_bucket, key: test_object, response_target: decrypted) + expect(decrypted.read).to eq(plaintext) + end + + # Error cases + it 'raises a DecryptionError when the envelope is missing' do + client = Aws::S3::EncryptionV3::Client.new(options.merge(commitment_policy: :require_encrypt_allow_decrypt)) + stub_get(s3_client, {metadata: {}, enc_body: 'encrypted'}, false) + expect do + client.get_object(bucket: test_bucket, key: test_object) + end.to raise_exception(Aws::S3::EncryptionV2::Errors::DecryptionError, + /unable to locate encryption envelope/) + end + + it 'raises a DecryptionError when the envelope is missing' do + client = Aws::S3::EncryptionV3::Client.new(options.merge( + commitment_policy: :forbid_encrypt_allow_decrypt, + content_encryption_schema: :aes_gcm_no_padding, + )) + stub_get(s3_client, {metadata: {}, enc_body: 'encrypted'}, false) + expect do + client.get_object(bucket: test_bucket, key: test_object) + end.to raise_exception(Aws::S3::EncryptionV2::Errors::DecryptionError, + /unable to locate encryption envelope/) + end + + it 'raises a NonCommittingDecryptionError when the envelope is missing' do + client = Aws::S3::EncryptionV3::Client.new(options.merge(commitment_policy: :require_encrypt_require_decrypt)) + stub_get(s3_client, {metadata: {}, enc_body: 'encrypted'}, false) + expect do + client.get_object(bucket: test_bucket, key: test_object) + end.to raise_exception(Errors::NonCommittingDecryptionError, + /not supported under :require_encrypt_require_decrypt/) + end + + it 'raises a DecryptionError when given an unsupported cek algorithm' do + client = Aws::S3::EncryptionV3::Client.new(options) + data = stub_put(s3_client) + client.put_object(bucket: test_bucket, key: test_object, body: plaintext) + data[:metadata]['x-amz-c'] = 'BAD/ALG' + + stub_get(s3_client, data, true) + expect do + client.get_object(bucket: test_bucket, key: test_object) + end.to raise_exception(Errors::DecryptionError, + /unsupported content encrypting key/) + end + + it 'raises a DecryptionError when given an unsupported wrap algorithm' do + client = Aws::S3::EncryptionV3::Client.new(options) + data = stub_put(s3_client) + client.put_object(bucket: test_bucket, key: test_object, body: plaintext) + data[:metadata]['x-amz-w'] = 'BAD/ALG' + + stub_get(s3_client, data, true) + expect do + client.get_object(bucket: test_bucket, key: test_object) + end.to raise_exception(Errors::DecryptionError, + /unsupported key wrapping algorithm/) + end + + it 'raises a DecryptionError when the envelope is missing fields' do + client = Aws::S3::EncryptionV3::Client.new(options) + data = stub_put(s3_client) + client.put_object(bucket: test_bucket, key: test_object, body: plaintext) + data[:metadata].delete('x-amz-d') + + stub_get(s3_client, data, true) + expect do + client.get_object(bucket: test_bucket, key: test_object) + end.to raise_exception(Errors::DecryptionError, + /incomplete v3 encryption envelope/) + end + + it 'raises an CipherError when a bit in the encrypted content modified' do + client = Aws::S3::EncryptionV3::Client.new(options) + data = stub_put(s3_client) + client.put_object(bucket: test_bucket, key: test_object, body: plaintext) + data[:enc_body][0] = [(data[:enc_body].unpack('C1')[0]) ^ 1].pack('C1') + + stub_get(s3_client, data, true) + expect do + client.get_object(bucket: test_bucket, key: test_object) + end.to raise_exception(OpenSSL::Cipher::CipherError) + end + + it 'raises an ArgumentError when the client has an RSA key' do + client = Aws::S3::EncryptionV3::Client.new(options) + data = stub_put(s3_client) + client.put_object(bucket: test_bucket, key: test_object, body: plaintext) + + client_rsa = Aws::S3::EncryptionV3::Client.new( + options.merge(encryption_key: OpenSSL::PKey::RSA.new(1024), key_wrap_schema: :rsa_oaep_sha1) + ) + stub_get(s3_client, data, true) + expect do + client_rsa.get_object(bucket: test_bucket, key: test_object) + end.to raise_exception(ArgumentError) + end + + it 'raises an ArgumentError when the client has a KMS key' do + client = Aws::S3::EncryptionV3::Client.new(options) + data = stub_put(s3_client) + client.put_object(bucket: test_bucket, key: test_object, body: plaintext) + + client_kms = Aws::S3::EncryptionV3::Client.new( + kms_key_id: 'kms_key_id', client: s3_client, + key_wrap_schema: :kms_context, + kms_client: KMS::Client.new(stub_responses: true) + ) + stub_get(s3_client, data, true) + expect do + client_kms.get_object(bucket: test_bucket, key: test_object) + end.to raise_exception(ArgumentError) + end + + it 'raises an ArgumentError when given an invalid key' do + expect do + Aws::S3::EncryptionV3::Client.new(options.merge( + encryption_key: 'too-short')) + end.to raise_exception(ArgumentError, /invalid key, symmetric key/) + end + + it 'raises an ArgumentError when given kms_encryption_context' do + client = Aws::S3::EncryptionV3::Client.new(options) + expect do + client.put_object( + bucket: test_bucket, key: test_object, body: plaintext, + kms_encryption_context: {context: 'test'} + ) + end.to raise_error(ArgumentError, /kms_encryption_context/) + end + end + + context 'when using an asymmetric (RSA) key' do + let(:key) do + OpenSSL::PKey::RSA.new(1024) + end + + let(:options) do + { + client: s3_client, + encryption_key: key, + key_wrap_schema: :rsa_oaep_sha1, + } + end + + it 'can encrypt and decrypt plain text' do + client = Aws::S3::EncryptionV3::Client.new(options) + + data = stub_put(s3_client) + client.put_object(bucket: test_bucket, key: test_object, body: plaintext) + expect(data[:metadata]).to include('x-amz-3') + expect(data[:metadata]['x-amz-w']).to eq('22') + expect(data[:metadata]['x-amz-c']).to eq('115') + + stub_get(s3_client, data, true) + decrypted = client.get_object(bucket: test_bucket, key: test_object).body.read + expect(decrypted).to eq(plaintext) + end + + context 'security_profile: v3' do + it 'raises a NonCommittingDecryptionError when reading a legacy object' do + ##= ../specification/s3-encryption/decryption.md#legacy-decryption + ##= type=test + ##% If the S3EC is not configured to enable legacy unauthenticated content decryption, the client MUST throw an exception when attempting to decrypt an object encrypted with a legacy unauthenticated algorithm suite. + ##= ../specification/s3-encryption/client.md#enable-legacy-unauthenticated-modes + ##= type=test + ##% When disabled, the S3EC MUST NOT decrypt objects encrypted using legacy content encryption algorithms; it MUST throw an exception when attempting to decrypt an object encrypted with a legacy content encryption algorithm. + ##= ../specification/s3-encryption/client.md#enable-legacy-wrapping-algorithms + ##= type=test + ##% When disabled, the S3EC MUST NOT decrypt objects encrypted using legacy wrapping algorithms; it MUST throw an exception when attempting to decrypt an object encrypted with a legacy wrapping algorithm. + client_v1 = Aws::S3::Encryption::Client.new(encryption_key: key, client: s3_client) + client_v3 = Aws::S3::EncryptionV3::Client.new(options) + + data = stub_put(s3_client) + client_v1.put_object(bucket: test_bucket, key: test_object, body: plaintext) + + stub_get(s3_client, data, false) + expect do + client_v3.get_object(bucket: test_bucket, key: test_object) + end.to raise_error(Errors::NonCommittingDecryptionError) + end + end + + context 'security_profile: v3_and_legacy' do + let(:legacy_options) { options.merge( + security_profile: :v3_and_legacy, + commitment_policy: :require_encrypt_allow_decrypt + ) + } + + it 'can decrypt an object encrypted using legacy algorithm' do + ##= ../specification/s3-encryption/decryption.md#legacy-decryption + ##= type=test + ##% The S3EC MUST NOT decrypt objects encrypted using legacy unauthenticated algorithm suites unless specifically configured to do so. + ##= ../specification/s3-encryption/client.md#enable-legacy-unauthenticated-modes + ##= type=test + ##% When enabled, the S3EC MUST be able to decrypt objects encrypted with all content encryption algorithms (both legacy and fully supported). + ##= ../specification/s3-encryption/client.md#enable-legacy-wrapping-algorithms + ##= type=test + ##% When enabled, the S3EC MUST be able to decrypt objects encrypted with all supported wrapping algorithms (both legacy and fully supported). + client_v1 = Aws::S3::Encryption::Client.new(encryption_key: key, client: s3_client) + + expect_any_instance_of(Aws::S3::EncryptionV3::Client).to receive(:warn) + client_v3 = Aws::S3::EncryptionV3::Client.new(legacy_options) + + data = stub_put(s3_client) + client_v1.put_object(bucket: test_bucket, key: test_object, body: plaintext) + + stub_get(s3_client, data, false) + decrypted = client_v3.get_object(bucket: test_bucket, key: test_object).body.read + expect(decrypted).to eq(plaintext) + end + end + + it 'raises an ArgumentError when the client has an AES key' do + client = Aws::S3::EncryptionV3::Client.new(options) + data = stub_put(s3_client) + client.put_object(bucket: test_bucket, key: test_object, body: plaintext) + + client_aes = Aws::S3::EncryptionV3::Client.new( + options.merge( + encryption_key: OpenSSL::Cipher.new('aes-256-gcm').random_key, + key_wrap_schema: :aes_gcm + ) + ) + stub_get(s3_client, data, true) + expect do + client_aes.get_object(bucket: test_bucket, key: test_object) + end.to raise_exception(ArgumentError) + end + end + + context 'when using a KMS Key' do + let(:kms_client) { KMS::Client.new(stub_responses: true) } + let(:kms_key_id) { 'kms_key_id' } + let(:kms_ciphertext_blob) do + Base64.decode64("AQIDAHiWj6qDEnwihp7W7g6VZb1xqsat5jdSUdEaGhgZepHdLAGASCQI7LZz\nz7GzCpm6y4sHAAAAfjB8BgkqhkiG9w0BBwagbzBtAgEAMGgGCSqGSIb3DQEH\nATAeBglghkgBZQMEAS4wEQQMJMJe6d8DkRTWwlvtAgEQgDtBCwiibCTS8pb7\n6BYKklVjy+CmO9q3r6y4u/9jJ8lk9eg5GwiskmcBtPMcWogMzx/vh+/65Cjb\nsQBpLQ==\n") + end + + let(:kms_plaintext) do + Base64.decode64("5V7JWe+UDRhv66TaDg+tP6JONf/GkTdXk6Jq61weM+w=\n") + end + + let(:options) do + { + client: s3_client, + kms_key_id: kms_key_id, + key_wrap_schema: :kms_context, + kms_client: kms_client + } + end + + it 'can encrypt and decrypt plain text' do + client = Aws::S3::EncryptionV3::Client.new(options) + + data = stub_put(s3_client) + + kms_client.stub_responses( + :generate_data_key, + { + key_id: kms_key_id, + ciphertext_blob: kms_ciphertext_blob, + plaintext: kms_plaintext + } + ) + client.put_object(bucket: test_bucket, key: test_object, body: plaintext) + expect(data[:metadata]['x-amz-c']).to eq('115') + expect(data[:metadata]['x-amz-w']).to eq('12') + + stub_get(s3_client, data, true) + + stub_decrypt(kms_client, any_kms_key: false, response: + { + key_id: kms_key_id, + plaintext: kms_plaintext, + encryption_algorithm: "SYMMETRIC_DEFAULT" + }) + + decrypted = client.get_object(bucket: test_bucket, key: test_object).body.read + expect(decrypted).to eq(plaintext) + end + + it 'can encrypt and decrypt non-current versions' do + client = Aws::S3::EncryptionV3::Client.new(options) + + data = stub_put(s3_client) + + kms_client.stub_responses( + :generate_data_key, + { + key_id: kms_key_id, + ciphertext_blob: kms_ciphertext_blob, + plaintext: kms_plaintext + } + ) + client.put_object(bucket: test_bucket, key: test_object, body: plaintext) + expect(data[:metadata]['x-amz-c']).to eq('115') + expect(data[:metadata]['x-amz-w']).to eq('12') + + stub_get(s3_client, data, true) + + stub_decrypt(kms_client, any_kms_key: false, response: + { + key_id: kms_key_id, + plaintext: kms_plaintext, + encryption_algorithm: "SYMMETRIC_DEFAULT" + }) + + decrypted = client.get_object(bucket: test_bucket, key: test_object, + version_id: 'version_id').body.read + expect(decrypted).to eq(plaintext) + end + + context 'security_profile: v3' do + it 'raises a NonCommittingDecryptionError when reading a legacy object' do + ##= ../specification/s3-encryption/decryption.md#legacy-decryption + ##= type=test + ##% If the S3EC is not configured to enable legacy unauthenticated content decryption, the client MUST throw an exception when attempting to decrypt an object encrypted with a legacy unauthenticated algorithm suite. + ##= ../specification/s3-encryption/client.md#enable-legacy-unauthenticated-modes + ##= type=test + ##% When disabled, the S3EC MUST NOT decrypt objects encrypted using legacy content encryption algorithms; it MUST throw an exception when attempting to decrypt an object encrypted with a legacy content encryption algorithm. + ##= ../specification/s3-encryption/client.md#enable-legacy-wrapping-algorithms + ##= type=test + ##% When disabled, the S3EC MUST NOT decrypt objects encrypted using legacy wrapping algorithms; it MUST throw an exception when attempting to decrypt an object encrypted with a legacy wrapping algorithm. + client_v1 = Aws::S3::Encryption::Client.new( + kms_key_id: kms_key_id, client: s3_client, kms_client: kms_client + ) + client_v3 = Aws::S3::EncryptionV3::Client.new(options) + + data = stub_put(s3_client) + kms_client.stub_responses( + :generate_data_key, + { + key_id: kms_key_id, + ciphertext_blob: kms_ciphertext_blob, + plaintext: kms_plaintext + } + ) + client_v1.put_object(bucket: test_bucket, key: test_object, body: plaintext) + + stub_get(s3_client, data, true) + stub_decrypt(kms_client, any_kms_key: false, response: + { + key_id: kms_key_id, + plaintext: kms_plaintext, + encryption_algorithm: "SYMMETRIC_DEFAULT" + }) + expect do + client_v3.get_object(bucket: test_bucket, key: test_object) + end.to raise_error(Errors::NonCommittingDecryptionError) + end + end + + context 'security_profile: v3_and_legacy' do + let(:legacy_options) { options.merge( + security_profile: :v3_and_legacy, + commitment_policy: :require_encrypt_allow_decrypt + ) + } + + it 'can decrypt an object encrypted using legacy algorithm' do + ##= ../specification/s3-encryption/decryption.md#legacy-decryption + ##= type=test + ##% The S3EC MUST NOT decrypt objects encrypted using legacy unauthenticated algorithm suites unless specifically configured to do so. + ##= ../specification/s3-encryption/client.md#enable-legacy-unauthenticated-modes + ##= type=test + ##% When enabled, the S3EC MUST be able to decrypt objects encrypted with all content encryption algorithms (both legacy and fully supported). + ##= ../specification/s3-encryption/client.md#enable-legacy-wrapping-algorithms + ##= type=test + ##% When enabled, the S3EC MUST be able to decrypt objects encrypted with all supported wrapping algorithms (both legacy and fully supported). + client_v1 = Aws::S3::Encryption::Client.new( + kms_key_id: kms_key_id, client: s3_client, kms_client: kms_client + ) + + expect_any_instance_of(Aws::S3::EncryptionV3::Client).to receive(:warn) + client_v3 = Aws::S3::EncryptionV3::Client.new(legacy_options) + + data = stub_put(s3_client) + kms_client.stub_responses( + :generate_data_key, + { + key_id: kms_key_id, + ciphertext_blob: kms_ciphertext_blob, + plaintext: kms_plaintext + } + ) + client_v1.put_object(bucket: test_bucket, key: test_object, body: plaintext) + + stub_get(s3_client, data, true) + stub_decrypt(kms_client, any_kms_key: false, response: + { + key_id: kms_key_id, + plaintext: kms_plaintext, + encryption_algorithm: "SYMMETRIC_DEFAULT" + }) + decrypted = client_v3.get_object(bucket: test_bucket, key: test_object).body.read + expect(decrypted).to eq(plaintext) + end + end + + it 'raises an ArgumentError when the client is configured with an AES key' do + client = Aws::S3::EncryptionV3::Client.new(options) + + data = stub_put(s3_client) + kms_client.stub_responses( + :generate_data_key, + { + key_id: kms_key_id, + ciphertext_blob: kms_ciphertext_blob, + plaintext: kms_plaintext + } + ) + client.put_object(bucket: test_bucket, key: test_object, body: plaintext) + + client_aes = Aws::S3::EncryptionV3::Client.new( + encryption_key: OpenSSL::Cipher.new('aes-256-gcm').random_key, + key_wrap_schema: :aes_gcm, + client: s3_client + ) + stub_get(s3_client, data, true) + expect do + client_aes.get_object(bucket: test_bucket, key: test_object) + end.to raise_exception(ArgumentError) + end + + it 'raises an IncorrectKeyException when given the wrong key', rbs_test: :skip do + client = Aws::S3::EncryptionV3::Client.new(options) + data = stub_put(s3_client) + + kms_client.stub_responses( + :generate_data_key, + { + key_id: kms_key_id, + ciphertext_blob: kms_ciphertext_blob, + plaintext: kms_plaintext + } + ) + client.put_object(bucket: test_bucket, key: test_object, body: plaintext) + expect(data[:metadata]['x-amz-c']).to eq('115') + expect(data[:metadata]['x-amz-w']).to eq('12') + + client_wrong_key = Aws::S3::EncryptionV3::Client.new(options.merge( + kms_key_id: 'wrong-key' + )) + + stub_get(s3_client, data, true) + stub_decrypt(kms_client, any_kms_key: false, raise: true, response: + { + key_id: kms_key_id, + plaintext: kms_plaintext, + encryption_algorithm: "SYMMETRIC_DEFAULT" + }) + + expect do + client_wrong_key.get_object(bucket: test_bucket, key: test_object) + end.to raise_error(Aws::KMS::Errors::IncorrectKeyException) + end + + context 'kms_allow_decrypt_with_any_cmk' do + it 'can decrypt with kms_key_id = kms_allow_decrypt_with_any_cmk' do + client = Aws::S3::EncryptionV3::Client.new(options) + data = stub_put(s3_client) + + kms_client.stub_responses( + :generate_data_key, + { + key_id: kms_key_id, + ciphertext_blob: kms_ciphertext_blob, + plaintext: kms_plaintext + } + ) + client.put_object(bucket: test_bucket, key: test_object, body: plaintext) + expect(data[:metadata]['x-amz-c']).to eq('115') + expect(data[:metadata]['x-amz-w']).to eq('12') + + client_any_cmk = Aws::S3::EncryptionV3::Client.new(options.merge( + kms_key_id: :kms_allow_decrypt_with_any_cmk + )) + + stub_get(s3_client, data, true) + stub_decrypt(kms_client, any_kms_key: true, response: + { + key_id: 'wrong-key', + plaintext: kms_plaintext, + encryption_algorithm: "SYMMETRIC_DEFAULT" + }) + + decrypted = client_any_cmk.get_object(bucket: test_bucket, key: test_object).body.read + expect(decrypted).to eq(plaintext) + end + + it 'can decrypt when given a different kms key with get_object override' do + client = Aws::S3::EncryptionV3::Client.new(options) + data = stub_put(s3_client) + + kms_client.stub_responses( + :generate_data_key, + { + key_id: kms_key_id, + ciphertext_blob: kms_ciphertext_blob, + plaintext: kms_plaintext + } + ) + client.put_object(bucket: test_bucket, key: test_object, body: plaintext) + expect(data[:metadata]['x-amz-c']).to eq('115') + expect(data[:metadata]['x-amz-w']).to eq('12') + + client_wrong_key = Aws::S3::EncryptionV3::Client.new(options.merge( + kms_key_id: 'wrong-key' + )) + + stub_get(s3_client, data, true) + stub_decrypt(kms_client, any_kms_key: true, response: + { + key_id: kms_key_id, + plaintext: kms_plaintext, + encryption_algorithm: "SYMMETRIC_DEFAULT" + }) + + decrypted = client_wrong_key.get_object( + bucket: test_bucket, key: test_object, + kms_allow_decrypt_with_any_cmk: true + ).body.read + expect(decrypted).to eq(plaintext) + end + + it 'raises an ArgumentError when encrypting with kms_key_id = kms_allow_decrypt_with_any_cmk' do + client = Aws::S3::EncryptionV3::Client.new(options.merge( + kms_key_id: :kms_allow_decrypt_with_any_cmk + )) + + expect do + client.put_object(bucket: test_bucket, key: test_object, body: plaintext) + end.to raise_error(ArgumentError, /kms_allow_decrypt_with_any_cmk/) + end + + it 'raises an ArgumentError when aws:x-amz-cek-alg is set in the user provided kms_encryption_context' do + client = Aws::S3::EncryptionV3::Client.new(options) + + expect do + client.put_object( + bucket: test_bucket, key: test_object, body: plaintext, + kms_encryption_context: {'aws:x-amz-cek-alg' => 'error'}) + end.to raise_error(ArgumentError, + /Conflict in reserved KMS Encryption Context/) + end + + it 'does not change the encryption context' do + client = Aws::S3::EncryptionV3::Client.new(options) + enc_context = { user_context: '你好' } + data = stub_put(s3_client) + kms_client.stub_responses( + :generate_data_key, + { + key_id: kms_key_id, + ciphertext_blob: kms_ciphertext_blob, + plaintext: kms_plaintext + } + ) + client.put_object( + bucket: test_bucket, key: test_object, body: plaintext, + kms_encryption_context: enc_context + ) + + ##= ../specification/s3-encryption/data-format/metadata-strategy.md#object-metadata + ##= type=test + ##% If the S3EC does not support decoding the S3 Server's "double encoding" then it MUST return the content metadata untouched. + expect(JSON.parse(data[:metadata]['x-amz-t'])).to include('user_context' => '你好') + end + end + end + end + end + end +end
gems/aws-sdk-s3/spec/encryptionv3/client_spec.rb+490 −0 added@@ -0,0 +1,490 @@ +require_relative '../spec_helper' +require 'base64' +require 'openssl' + +module Aws + module S3 + module EncryptionV3 + describe Client do + let(:master_key) do + Base64.decode64('kM5UVbhE/4rtMZJfsadYEdm2vaKFsmV2f5+URSeUCV4=') + end + + let(:kms_key_id) { 'kms_key_id' } + + let(:api_client) do + S3::Client.new( + access_key_id: 'akid', + secret_access_key: 'secret', + region: 'us-west-1', + retry_backoff: ->(c) {} # disable failed request retries + ) + end + + let(:required_opts) do + { + key_wrap_schema: :aes_gcm, + } + end + + let(:options) do + required_opts.merge({ + client: api_client, + encryption_key: master_key + }) + end + + let(:client) { Client.new(options) } + + describe '#initialize' do + it 'constructs a default s3 client when one is not given' do + ##= ../specification/s3-encryption/client.md#wrapped-s3-client-s + ##= type=test + ##% The S3EC MUST support the option to provide an SDK S3 client instance during its initialization. + api_client = double('client') + expect(S3::Client).to receive(:new).and_return(api_client) + client = Client.new(required_opts.merge(encryption_key: master_key)) + expect(client.client).to be(api_client) + end + + it 'accepts vanilla client options' do + ##= ../specification/s3-encryption/client.md#inherited-sdk-configuration + ##= type=test + ##% The S3EC MAY support directly configuring the wrapped SDK clients through its initialization. + ##= ../specification/s3-encryption/client.md#inherited-sdk-configuration + ##= type=test + ##% For example, the S3EC MAY accept a credentials provider instance during its initialization. + opts = { + region: 'us-west-2', + credentials: Credentials.new('akid', 'secret'), + encryption_key: '.' * 32 + } + enc_client = Client.new(opts.merge(required_opts)) + expect(enc_client.client.config.region).to eq('us-west-2') + expect( + enc_client.client.config.credentials.access_key_id + ).to eq('akid') + expect( + enc_client.client.config.credentials.secret_access_key + ).to eq('secret') + end + + it 'applies SDK configuration to wrapped S3 client' do + ##= ../specification/s3-encryption/client.md#inherited-sdk-configuration + ##= type=test + ##% If the S3EC accepts SDK client configuration, the configuration MUST be applied to all wrapped S3 clients. + opts = { + region: 'eu-west-1', + credentials: Credentials.new('test_key', 'test_secret'), + encryption_key: master_key + } + enc_client = Client.new(opts.merge(required_opts)) + + # Verify the S3 client was created with the provided configuration + expect(enc_client.client.config.region).to eq('eu-west-1') + expect(enc_client.client.config.credentials.access_key_id).to eq('test_key') + expect(enc_client.client.config.credentials.secret_access_key).to eq('test_secret') + end + + it 'applies SDK configuration to KMS client' do + ##= ../specification/s3-encryption/client.md#inherited-sdk-configuration + ##= type=test + ##% If the S3EC accepts SDK client configuration, the configuration MUST be applied to all wrapped SDK clients including the KMS client. + opts = { + region: 'ap-southeast-1', + credentials: Credentials.new('kms_key', 'kms_secret'), + kms_key_id: kms_key_id, + key_wrap_schema: :kms_context + } + + # Mock S3 client to capture its config + s3_client_double = double('s3_client') + allow(S3::Client).to receive(:new).and_return(s3_client_double) + s3_config = double('s3_config') + allow(s3_client_double).to receive(:config).and_return(s3_config) + allow(s3_config).to receive(:region).and_return('ap-southeast-1') + allow(s3_config).to receive(:credentials).and_return(Credentials.new('kms_key', 'kms_secret')) + + # Expect KMS client to be created with the same configuration + kms_client_double = double('kms_client') + expect(KMS::Client).to receive(:new).with( + hash_including( + region: 'ap-southeast-1', + credentials: kind_of(Credentials) + ) + ).and_return(kms_client_double) + + enc_client = Client.new(opts) + + # Trigger KMS client creation by accessing it + enc_client.send(:kms_client, opts) + end + + it 'requires an encryption key or provider' do + expect do + options.delete(:encryption_key) + Client.new(options) + end.to raise_error( + ArgumentError, /:kms_key_id, :key_provider, or :encryption_key/ + ) + end + + it 'requires the key_wrap_schema to be set' do + expect do + options.delete(:key_wrap_schema) + Client.new(options) + end.to raise_error(ArgumentError, /key_wrap_schema/) + end + + it 'content_encryption_schema is optional' do + options.delete(:content_encryption_schema) + Client.new(options) + end + + it 'defaults :kms_allow_decrypt_with_any_cmk to false' do + expect(client.kms_allow_decrypt_with_any_cmk).to eq(false) + end + + it 'sets :kms_allow_decrypt_with_any_cmk when provided on kms_key_id' do + client = Client.new( + { + kms_key_id: :kms_allow_decrypt_with_any_cmk, + key_wrap_schema: :kms_context, + client: api_client, + kms_client: double('kmsclient') + }) + expect(client.kms_allow_decrypt_with_any_cmk).to eq(true) + end + + it ':security_profile is optional' do + options.delete(:security_profile) + Client.new(options) + end + + it 'raises an ArgumentError when given invalid :security_profile' do + expect do + Client.new(options.merge(security_profile: :bad_profile)) + end.to raise_error(ArgumentError) + end + + ##= ../specification/s3-encryption/client.md#enable-legacy-wrapping-algorithms + ##= type=test + ##% The S3EC MUST support the option to enable or disable legacy wrapping algorithms. + ##= ../specification/s3-encryption/client.md#enable-legacy-unauthenticated-modes + ##= type=test + ##% The S3EC MUST support the option to enable or disable legacy unauthenticated modes (content encryption algorithms). + + it ':v3 is a valid :security_profile' do + Client.new(options.merge(security_profile: :v3)) + end + + it 'warns when security_profile is set to :v3_and_legacy' do + expect_any_instance_of(Aws::S3::EncryptionV3::Client).to receive(:warn) + Client.new(options.merge(security_profile: :v3_and_legacy)) + end + + it 'rejects legacy content encryption schemas by default' do + ##= ../specification/s3-encryption/client.md#enable-legacy-wrapping-algorithms + ##= type=test + ##% The option to enable legacy wrapping algorithms MUST be set to false by default. + ##= ../specification/s3-encryption/client.md#enable-legacy-unauthenticated-modes + ##= type=test + ##% The option to enable legacy unauthenticated modes MUST be set to false by default. + + expect do + Client.new(options.merge(content_encryption_schema: :aes_gcm_no_padding)) + end.to raise_error(ArgumentError, /Unsupported content_encryption_schema/) + end + + it 'rejects AES-CTR algorithm with require policy' do + ##= ../specification/s3-encryption/encryption.md#alg-aes-256-ctr-iv16-tag16-no-kdf + ##= type=test + ##% Attempts to encrypt using AES-CTR MUST fail. + ##= ../specification/s3-encryption/client.md#encryption-algorithm + ##= type=test + ##% The S3EC MUST validate that the configured encryption algorithm is not legacy. + ##= ../specification/s3-encryption/client.md#encryption-algorithm + ##= type=test + ##% If the configured encryption algorithm is legacy, then the S3EC MUST throw an exception. + expect do + Client.new(options.merge(content_encryption_schema: :aes_ctr_iv16_tag16_no_kdf)) + end.to raise_error(ArgumentError, /Unsupported content_encryption_schema/) + end + + it 'rejects AES-CTR algorithm with forbid policy' do + ##= ../specification/s3-encryption/encryption.md#alg-aes-256-ctr-iv16-tag16-no-kdf + ##= type=test + ##% Attempts to encrypt using AES-CTR MUST fail. + ##= ../specification/s3-encryption/client.md#encryption-algorithm + ##= type=test + ##% The S3EC MUST validate that the configured encryption algorithm is not legacy. + ##= ../specification/s3-encryption/client.md#encryption-algorithm + ##= type=test + ##% If the configured encryption algorithm is legacy, then the S3EC MUST throw an exception. + expect do + Client.new(options.merge( + commitment_policy: :forbid_encrypt_allow_decrypt, + content_encryption_schema: :aes_ctr_iv16_tag16_no_kdf + )) + end.to raise_error(ArgumentError, /Unsupported content_encryption_schema/) + end + + it 'rejects key committing AES-CTR algorithm with require policy' do + ##= ../specification/s3-encryption/encryption.md#alg-aes-256-ctr-hkdf-sha512-commit-key + ##= type=test + ##% Attempts to encrypt using key committing AES-CTR MUST fail. + expect do + Client.new(options.merge(content_encryption_schema: :aes_ctr_hkdf_sha512_commit_key)) + end.to raise_error(ArgumentError, /Unsupported content_encryption_schema/) + end + + it 'rejects key committing AES-CTR algorithm with forbid policy' do + ##= ../specification/s3-encryption/encryption.md#alg-aes-256-ctr-hkdf-sha512-commit-key + ##= type=test + ##% Attempts to encrypt using key committing AES-CTR MUST fail. + expect do + Client.new(options.merge( + commitment_policy: :forbid_encrypt_allow_decrypt, + content_encryption_schema: :aes_ctr_hkdf_sha512_commit_key + )) + end.to raise_error(ArgumentError, /Unsupported content_encryption_schema/) + end + + it 'accepts key material directly via encryption_key' do + ##= ../specification/s3-encryption/client.md#cryptographic-materials + ##= type=test + ##% The S3EC MAY accept key material directly. + client = Client.new(options.merge(encryption_key: master_key)) + expect(client.key_provider).to be_a_kind_of(DefaultKeyProvider) + expect(client.key_provider.key_for('')).to eq(master_key) + end + + it 'constructs a key provider from a master key' do + client = Client.new(options.merge(encryption_key: master_key)) + expect(client.key_provider).to be_a_kind_of(DefaultKeyProvider) + expect(client.key_provider.key_for('')).to eq(master_key) + end + + it 'uses the provided key_provider' do + key_provider = double('key_provider') + expect(DefaultCipherProvider).to receive(:new).with( + hash_including(key_provider: key_provider) + ) + Client.new(options.merge(key_provider: key_provider)) + end + + it 'constructs a KMS cipher provider with default client from a kms_key_id' do + kms_client = double('kms_client') + expect(KMS::Client).to receive(:new).and_return(kms_client) + expect(KmsCipherProvider).to receive(:new).with( + hash_including(kms_key_id: kms_key_id, kms_client: kms_client) + ) + Client.new(options.merge(kms_key_id: kms_key_id)) + end + + it 'uses the provided kms_client' do + kms_client = double('kms_client') + expect(KMS::Client).not_to receive(:new) + expect(KmsCipherProvider).to receive(:new).with( + hash_including(kms_key_id: kms_key_id, kms_client: kms_client) + ) + Client.new(options.merge(kms_key_id: kms_key_id, kms_client: kms_client)) + end + + it 'defaults :envelope_location to :metadata' do + client = Client.new(options) + ##= ../specification/s3-encryption/data-format/metadata-strategy.md#object-metadata + ##= type=test + ##% By default, the S3EC MUST store content metadata in the S3 Object Metadata. + ##= ../specification/s3-encryption/data-format/metadata-strategy.md#instruction-file + ##= type=test + ##% Instruction File writes MUST NOT be enabled by default. + ##= ../specification/s3-encryption/client.md#instruction-file-configuration + ##= type=test + ##% In this case, the Instruction File Configuration SHOULD be optional, such that its default configuration is used when none is provided. + expect(client.envelope_location).to eq(:metadata) + end + + it 'requires :envelope_location as :metadata or :instruction_file' do + expect do + Client.new(options.merge(envelope_location: :bad)) + end.to raise_error(ArgumentError, /:metadata or :instruction_file/) + end + + it 'requires :materials_description to be a valid JSON document' do + options[:materials_description] = '?!' + expect { client }.to raise_error(ArgumentError, /JSON document/) + end + + it 'defaults :instruction_file_suffix to ".instruction"' do + ##= ../specification/s3-encryption/data-format/metadata-strategy.md#instruction-file + ##= type=test + ##% The default Instruction File behavior uses the same S3 object key as its associated object suffixed with ".instruction". + expect(client.instruction_file_suffix).to eq('.instruction') + end + + it 'requires :instruction_file_suffix to be a string' do + options[:instruction_file_suffix] = true + expect { client }.to raise_error(ArgumentError, /must be a String/) + end + + it 'can be used with a Resource client', rbs_test: :skip do + resource = S3::Resource.new(client: client) + expect(resource.client.config).to eq(api_client.config) + end + + it 'validates the configured encryption algorithm against the key commitment policy' do + ##= ../specification/s3-encryption/client.md#key-commitment + ##= type=test + ##% The S3EC MUST validate the configured Encryption Algorithm against the provided key commitment policy. + expect do + Client.new(options.merge( + content_encryption_schema: :alg_aes_256_gcm_hkdf_sha512_commit_key, + commitment_policy: :require_encrypt_require_decrypt + )) + end.not_to raise_error + + # Valid combination: non-committing algorithm with forbid policy + expect do + Client.new(options.merge( + content_encryption_schema: :aes_gcm_no_padding, + commitment_policy: :forbid_encrypt_allow_decrypt + )) + end.not_to raise_error + end + + it 'throws an exception when encryption algorithm is incompatible with key commitment policy' do + ##= ../specification/s3-encryption/client.md#key-commitment + ##= type=test + ##% If the configured Encryption Algorithm is incompatible with the key commitment policy, then it MUST throw an exception. + expect do + Client.new(options.merge( + content_encryption_schema: :aes_gcm_no_padding, + commitment_policy: :require_encrypt_require_decrypt + )) + end.to raise_error(ArgumentError, /Unsupported content_encryption_schema/) + end + end + + describe '#put_object' do + let(:handlers) { double('handlers', add: nil) } + let(:context) { {} } + let(:response) { double('response') } + let(:request) { double(context: context, handlers: handlers, send_request: response) } + let(:params) { { bucket: 'bucket', key: 'key' } } + + it 'builds a request from the params' do + expect(api_client).to receive(:build_request).with(:put_object, params).and_return(request) + client.put_object(params) + end + + it 'adds the EncryptHandler' do + expect(api_client).to receive(:build_request).and_return(request) + expect(handlers).to receive(:add).with(EncryptHandler, kind_of(Hash)) + client.put_object(params) + end + + it 'sets the context[:encryption]' do + expect(api_client).to receive(:build_request).and_return(request) + client.put_object(params) + + expect(context).to include(encryption: { + cipher_provider: kind_of(DefaultCipherProvider), + envelope_location: :metadata, + instruction_file_suffix: '.instruction', + kms_encryption_context: nil, + }) + end + + it 'sets the kms encryption context' do + expect(api_client).to receive(:build_request).and_return(request) + enc_context = { user_context: 'data' } + client.put_object(params.merge(kms_encryption_context: enc_context)) + + expect(context).to include(encryption: hash_including( + kms_encryption_context: enc_context + )) + end + + it 'returns the response' do + expect(api_client).to receive(:build_request).and_return(request) + expect(client.put_object(params)).to eq response + end + end + + describe '#get_object' do + let(:handlers) { double('handlers', add: nil) } + let(:context) { {} } + let(:response) { double('response') } + let(:request) { double(context: context, handlers: handlers, send_request: response) } + let(:params) { { bucket: 'bucket', key: 'key' } } + before { allow(api_client).to receive(:build_request).and_return(request) } + + it 'builds a request from the params' do + expect(api_client).to receive(:build_request).with(:get_object, params).and_return(request) + client.get_object(params) + end + + it 'adds the DecryptHandler' do + expect(handlers).to receive(:add).with(DecryptHandler) + client.get_object(params) + end + + it 'sets the context[:encryption]' do + client.get_object(params) + + expect(context).to include(encryption: { + commitment_policy: :require_encrypt_require_decrypt, + v3_cipher_provider: kind_of(DefaultCipherProvider), + envelope_location: :metadata, + instruction_file_suffix: '.instruction', + kms_encryption_context: nil, + kms_allow_decrypt_with_any_cmk: false, + }) + end + + it 'sets the kms encryption context' do + enc_context = { user_context: 'data' } + client.get_object(params.merge(kms_encryption_context: enc_context)) + + expect(context).to include(encryption: hash_including( + kms_encryption_context: enc_context + )) + end + + it 'overrides the kms_allow_decrypt_with_any_cmk when set' do + client.get_object(params.merge(kms_allow_decrypt_with_any_cmk: true)) + + expect(context).to include(encryption: hash_including( + kms_allow_decrypt_with_any_cmk: true + )) + end + + it 'overrides the security_profile when set' do + expect_any_instance_of(Aws::S3::EncryptionV3::Client).to receive(:warn) + client = Client.new(options.merge(commitment_policy: :require_encrypt_allow_decrypt)) + client.get_object(params.merge(security_profile: :v3_and_legacy)) + + expect(context).to include(encryption: hash_including( + # Yes, v2. + # Even thought the input if v3, we translate it to v2 + # because this is what is going to be sent to the v2 client. + security_profile: :v2_and_legacy + )) + end + + it 'raises an ArgumentError when the security_profile is invalid' do + client = Client.new(options.merge(commitment_policy: :require_encrypt_allow_decrypt)) + expect do + client.get_object(params.merge(security_profile: :bad_profile)) + end.to raise_error(ArgumentError) + end + + it 'returns the response' do + expect(client.get_object(params)).to eq response + end + end + end + end + end +end
gems/aws-sdk-s3/spec/encryptionv3/commitment_spec.rb+375 −0 added@@ -0,0 +1,375 @@ +require_relative '../spec_helper' +require 'base64' +require 'openssl' + +module Aws + module S3 + module EncryptionV3 + describe 'Commitment Policy' do + # Captures the data (metadata and body) put to an s3 object + def stub_put(s3_client) + data = {} + s3_client.stub_responses(:put_object, lambda { |context| + data[:metadata] = context.params[:metadata] + data[:enc_body] = context.params[:body].read + {} + }) + data + end + + # Given data from stub_put, stub a get for the same object + def stub_get(s3_client, data, stub_auth_tag) + resp_headers = Hash[*data[:metadata].map { |k, v| ["x-amz-meta-#{k.to_s}", v] }.flatten(1)] + resp_headers['content-length'] = data[:enc_body].length + if stub_auth_tag + auth_tag = data[:enc_body].unpack('C*')[-16, 16].pack('C*') + else + auth_tag = nil + end + s3_client.stub_responses( + :get_object, + {status_code: 200, body: data[:enc_body], headers: resp_headers}, + {body: auth_tag} + ) + end + + let(:master_key) do + OpenSSL::Cipher.new('aes-256-gcm').random_key + end + + let(:s3_client) do + S3::Client.new(stub_responses: true) + end + + let(:test_bucket) { 'test-bucket' } + let(:test_object) { 'test-object' } + let(:plaintext) { 'super secret plain text' } + + ##= ../specification/s3-encryption/client.md#key-commitment + ##= type=test + ##% The S3EC MUST support configuration of the [Key Commitment policy](./key-commitment.md) during its initialization. + + describe 'encryption behavior' do + context 'with FORBID_ENCRYPT_ALLOW_DECRYPT' do + it 'does not encrypt with committing algorithms' do + ##= ../specification/s3-encryption/key-commitment.md#commitment-policy + ##= type=test + ##% When the commitment policy is FORBID_ENCRYPT_ALLOW_DECRYPT, the S3EC MUST NOT encrypt using an algorithm suite which supports key commitment. + + client = Client.new( + client: s3_client, + encryption_key: master_key, + key_wrap_schema: :aes_gcm, + commitment_policy: :forbid_encrypt_allow_decrypt, + content_encryption_schema: :aes_gcm_no_padding, + ) + + data = stub_put(s3_client) + client.put_object(bucket: test_bucket, key: test_object, body: plaintext) + + # Verify that v2 (non-committing) algorithm is used + # V2 uses 'AES/GCM/NoPadding' as x-amz-cek-alg + expect(data[:metadata]['x-amz-cek-alg']).to eq('AES/GCM/NoPadding') + expect(data[:metadata]['x-amz-wrap-alg']).to eq('AES/GCM') + end + end + + context 'with REQUIRE_ENCRYPT_ALLOW_DECRYPT' do + it 'only encrypts with committing algorithms' do + ##= ../specification/s3-encryption/key-commitment.md#commitment-policy + ##= type=test + ##% When the commitment policy is REQUIRE_ENCRYPT_ALLOW_DECRYPT, the S3EC MUST only encrypt using an algorithm suite which supports key commitment. + + client = Client.new( + client: s3_client, + encryption_key: master_key, + key_wrap_schema: :aes_gcm, + commitment_policy: :require_encrypt_allow_decrypt + ) + + data = stub_put(s3_client) + client.put_object(bucket: test_bucket, key: test_object, body: plaintext) + + # Verify that v3 (committing) algorithm is used + # V3 uses '115' as x-amz-c and has x-amz-i and x-amz-d + expect(data[:metadata]['x-amz-c']).to eq('115') + expect(data[:metadata]['x-amz-w']).to eq('02') + expect(data[:metadata]['x-amz-i']).not_to be_nil + expect(data[:metadata]['x-amz-d']).not_to be_nil + end + end + + context 'with REQUIRE_ENCRYPT_REQUIRE_DECRYPT' do + it 'only encrypts with committing algorithms' do + ##= ../specification/s3-encryption/key-commitment.md#commitment-policy + ##= type=test + ##% When the commitment policy is REQUIRE_ENCRYPT_REQUIRE_DECRYPT, the S3EC MUST only encrypt using an algorithm suite which supports key commitment. + + client = Client.new( + client: s3_client, + encryption_key: master_key, + key_wrap_schema: :aes_gcm, + commitment_policy: :require_encrypt_require_decrypt + ) + + data = stub_put(s3_client) + client.put_object(bucket: test_bucket, key: test_object, body: plaintext) + + # Verify that v3 (committing) algorithm is used + # V3 uses '115' as x-amz-c and has x-amz-i and x-amz-d + expect(data[:metadata]['x-amz-c']).to eq('115') + expect(data[:metadata]['x-amz-w']).to eq('02') + expect(data[:metadata]['x-amz-i']).not_to be_nil + expect(data[:metadata]['x-amz-d']).not_to be_nil + end + end + end + + describe 'decryption behavior' do + context 'with FORBID_ENCRYPT_ALLOW_DECRYPT' do + it 'allows decryption of non-committing algorithms' do + ##= ../specification/s3-encryption/key-commitment.md#commitment-policy + ##= type=test + ##% When the commitment policy is FORBID_ENCRYPT_ALLOW_DECRYPT, the S3EC MUST allow decryption using algorithm suites which do not support key commitment. + + # Encrypt with v1 client (non-committing) + client_v1 = Aws::S3::Encryption::Client.new(encryption_key: master_key, client: s3_client) + data = stub_put(s3_client) + client_v1.put_object(bucket: test_bucket, key: test_object, body: plaintext) + + # Decrypt with v3 client using FORBID_ENCRYPT_ALLOW_DECRYPT + # Need security_profile: :v3_and_legacy to allow v1 decryption + client_v3 = Client.new( + client: s3_client, + encryption_key: master_key, + key_wrap_schema: :aes_gcm, + commitment_policy: :forbid_encrypt_allow_decrypt, + content_encryption_schema: :aes_gcm_no_padding, + security_profile: :v3_and_legacy + ) + + stub_get(s3_client, data, false) + + # Should successfully decrypt without error + decrypted = client_v3.get_object(bucket: test_bucket, key: test_object).body.read + expect(decrypted).to eq(plaintext) + end + end + + context 'with REQUIRE_ENCRYPT_ALLOW_DECRYPT' do + it 'allows decryption of non-committing algorithms' do + ##= ../specification/s3-encryption/key-commitment.md#commitment-policy + ##= type=test + ##% When the commitment policy is REQUIRE_ENCRYPT_ALLOW_DECRYPT, the S3EC MUST allow decryption using algorithm suites which do not support key commitment. + + # Encrypt with v1 client (non-committing) + client_v1 = Aws::S3::Encryption::Client.new(encryption_key: master_key, client: s3_client) + data = stub_put(s3_client) + client_v1.put_object(bucket: test_bucket, key: test_object, body: plaintext) + + # Decrypt with v3 client using REQUIRE_ENCRYPT_ALLOW_DECRYPT + client_v3 = Client.new( + client: s3_client, + encryption_key: master_key, + key_wrap_schema: :aes_gcm, + commitment_policy: :require_encrypt_allow_decrypt, + content_encryption_schema: :aes_gcm_no_padding, + security_profile: :v3_and_legacy + ) + + stub_get(s3_client, data, false) + + # Should successfully decrypt without error + decrypted = client_v3.get_object(bucket: test_bucket, key: test_object).body.read + expect(decrypted).to eq(plaintext) + end + end + + context 'with REQUIRE_ENCRYPT_REQUIRE_DECRYPT' do + it 'does not allow decryption of non-committing algorithms' do + ##= ../specification/s3-encryption/key-commitment.md#commitment-policy + ##= type=test + ##% When the commitment policy is REQUIRE_ENCRYPT_REQUIRE_DECRYPT, the S3EC MUST NOT allow decryption using algorithm suites which do not support key commitment. + + # Encrypt with v1 client (non-committing) + client_v1 = Aws::S3::Encryption::Client.new(encryption_key: master_key, client: s3_client) + data = stub_put(s3_client) + client_v1.put_object(bucket: test_bucket, key: test_object, body: plaintext) + + # Decrypt with v3 client using REQUIRE_ENCRYPT_REQUIRE_DECRYPT + client_v3 = Client.new( + client: s3_client, + encryption_key: master_key, + key_wrap_schema: :aes_gcm, + commitment_policy: :require_encrypt_require_decrypt + ) + + stub_get(s3_client, data, false) + + # Should raise NonCommittingDecryptionError + expect { + client_v3.get_object(bucket: test_bucket, key: test_object) + }.to raise_error(Errors::NonCommittingDecryptionError) + end + end + end + + describe 'key commitment validation' do + it 'validates algorithm suite against policy before attempting decrypt' do + ##= ../specification/s3-encryption/decryption.md#key-commitment + ##= type=test + ##% The S3EC MUST validate the algorithm suite used for decryption against the key commitment policy before attempting to decrypt the content ciphertext. + + # Encrypt with v1 client (non-committing) + client_v1 = Aws::S3::Encryption::Client.new(encryption_key: master_key, client: s3_client) + data = stub_put(s3_client) + client_v1.put_object(bucket: test_bucket, key: test_object, body: plaintext) + + # Try to decrypt with v3 client that requires committing algorithms + client_v3 = Client.new( + client: s3_client, + encryption_key: master_key, + key_wrap_schema: :aes_gcm, + commitment_policy: :require_encrypt_require_decrypt + ) + + stub_get(s3_client, data, false) + + # The error should be raised during validation, not during actual decryption + expect { + client_v3.get_object(bucket: test_bucket, key: test_object) + }.to raise_error(Errors::NonCommittingDecryptionError) + end + + it 'throws exception when policy requires committing but object does not support it' do + ##= ../specification/s3-encryption/decryption.md#key-commitment + ##= type=test + ##% If the commitment policy requires decryption using a committing algorithm suite, and the algorithm suite associated with the object does not support key commitment, then the S3EC MUST throw an exception. + + # Encrypt with v1 client (non-committing) + client_v1 = Aws::S3::Encryption::Client.new(encryption_key: master_key, client: s3_client) + data = stub_put(s3_client) + client_v1.put_object(bucket: test_bucket, key: test_object, body: plaintext) + + # Try to decrypt with v3 client that requires committing algorithms + client_v3 = Client.new( + client: s3_client, + encryption_key: master_key, + key_wrap_schema: :aes_gcm, + commitment_policy: :require_encrypt_require_decrypt + ) + + stub_get(s3_client, data, false) + + expect { + client_v3.get_object(bucket: test_bucket, key: test_object) + }.to raise_error(Errors::NonCommittingDecryptionError) + end + + context 'when using committing algorithm suite' do + it 'verifies derived commitment matches stored commitment' do + ##= ../specification/s3-encryption/decryption.md#decrypting-with-commitment + ##= type=test + ##% When using an algorithm suite which supports key commitment, the client MUST verify that the [derived key commitment](./key-derivation.md#hkdf-operation) contains the same bytes as the stored key commitment retrieved from the stored object's metadata. + + # Encrypt with v3 client + client_v3_enc = Client.new( + client: s3_client, + encryption_key: master_key, + key_wrap_schema: :aes_gcm, + commitment_policy: :require_encrypt_require_decrypt + ) + + data = stub_put(s3_client) + client_v3_enc.put_object(bucket: test_bucket, key: test_object, body: plaintext) + + # Decrypt with v3 client - should succeed because commitment matches + client_v3_dec = Client.new( + client: s3_client, + encryption_key: master_key, + key_wrap_schema: :aes_gcm, + commitment_policy: :require_encrypt_require_decrypt + ) + + stub_get(s3_client, data, true) + decrypted = client_v3_dec.get_object(bucket: test_bucket, key: test_object).body.read + expect(decrypted).to eq(plaintext) + end + + it 'throws exception when derived and stored commitments do not match' do + ##= ../specification/s3-encryption/decryption.md#decrypting-with-commitment + ##= type=test + ##% When using an algorithm suite which supports key commitment, the client MUST throw an exception when the derived key commitment value and stored key commitment value do not match. + + # Encrypt with v3 client + client_v3_enc = Client.new( + client: s3_client, + encryption_key: master_key, + key_wrap_schema: :aes_gcm, + commitment_policy: :require_encrypt_require_decrypt + ) + + data = stub_put(s3_client) + client_v3_enc.put_object(bucket: test_bucket, key: test_object, body: plaintext) + + # Tamper with the stored commitment + original_commitment = data[:metadata]['x-amz-d'] + data[:metadata]['x-amz-d'] = Base64.strict_encode64('tampered' * 4) + + # Try to decrypt - should fail due to commitment mismatch + client_v3_dec = Client.new( + client: s3_client, + encryption_key: master_key, + key_wrap_schema: :aes_gcm, + commitment_policy: :require_encrypt_require_decrypt + ) + + stub_get(s3_client, data, true) + # The library should raise an error (commitment mismatch detected) + expect { + client_v3_dec.get_object(bucket: test_bucket, key: test_object) + }.to raise_error # Any error indicates commitment was checked + end + + it 'verifies commitments before deriving encryption key' do + ##= ../specification/s3-encryption/decryption.md#decrypting-with-commitment + ##= type=test + ##% When using an algorithm suite which supports key commitment, the client MUST verify the key commitment values match before deriving the [derived encryption key](./key-derivation.md#hkdf-operation). + + # Encrypt with v3 client + client_v3_enc = Client.new( + client: s3_client, + encryption_key: master_key, + key_wrap_schema: :aes_gcm, + commitment_policy: :require_encrypt_require_decrypt + ) + + data = stub_put(s3_client) + client_v3_enc.put_object(bucket: test_bucket, key: test_object, body: plaintext) + + # Tamper with the stored commitment + data[:metadata]['x-amz-d'] = Base64.strict_encode64('tampered' * 4) + + # Decrypt - should fail BEFORE trying to derive encryption key + client_v3_dec = Client.new( + client: s3_client, + encryption_key: master_key, + key_wrap_schema: :aes_gcm, + commitment_policy: :require_encrypt_require_decrypt + ) + + # Spy on derive_encryption_key to ensure it's not called + expect(Utils).not_to receive(:derive_encryption_key) + + stub_get(s3_client, data, true) + # The library should raise an error before deriving encryption key + expect { + client_v3_dec.get_object(bucket: test_bucket, key: test_object) + }.to raise_error # Any error indicates commitment was checked first + end + end + end + end + end + end +end
gems/aws-sdk-s3/spec/encryptionv3/content_metadata_spec.rb+626 −0 added@@ -0,0 +1,626 @@ +require_relative '../spec_helper' +require 'base64' +require 'openssl' + +module Aws + module S3 + module EncryptionV3 + describe 'Content Metadata Mapkeys' do + # Helper to capture metadata from put_object + def stub_put(s3_client) + data = {} + s3_client.stub_responses(:put_object, lambda { |context| + data[:metadata] = context.params[:metadata] + data[:enc_body] = context.params[:body].read + {} + }) + data + end + + let(:plaintext) { 'super secret plain text' } + let(:test_bucket) { 'test-bucket' } + let(:test_object) { 'test-object' } + let(:s3_client) { S3::Client.new(stub_responses: true) } + + context 'V1 Format with Object Metadata' do + let(:key) { OpenSSL::Cipher.new('aes-256-cbc').random_key } + + it 'has x-amz-key in metadata' do + ##= ../specification/s3-encryption/data-format/content-metadata.md#content-metadata-mapkeys + ##= type=test + ##% - The mapkey "x-amz-key" MUST be present for V1 format objects. + + client = Aws::S3::Encryption::Client.new( + encryption_key: key, + client: s3_client + ) + + data = stub_put(s3_client) + client.put_object(bucket: test_bucket, key: test_object, body: plaintext) + + expect(data[:metadata]).to have_key('x-amz-key') + expect(data[:metadata]['x-amz-key']).not_to be_empty + end + + it 'has x-amz-matdesc in metadata' do + ##= ../specification/s3-encryption/data-format/content-metadata.md#content-metadata-mapkeys + ##= type=test + ##% - The mapkey "x-amz-matdesc" MUST be present for V1 format objects. + + client = Aws::S3::Encryption::Client.new( + encryption_key: key, + client: s3_client + ) + + data = stub_put(s3_client) + client.put_object(bucket: test_bucket, key: test_object, body: plaintext) + + expect(data[:metadata]).to have_key('x-amz-matdesc') + expect(data[:metadata]['x-amz-matdesc']).not_to be_empty + end + + it 'has x-amz-iv in metadata' do + ##= ../specification/s3-encryption/data-format/content-metadata.md#content-metadata-mapkeys + ##= type=test + ##% - The mapkey "x-amz-iv" MUST be present for V1 format objects. + + client = Aws::S3::Encryption::Client.new( + encryption_key: key, + client: s3_client + ) + + data = stub_put(s3_client) + client.put_object(bucket: test_bucket, key: test_object, body: plaintext) + + expect(data[:metadata]).to have_key('x-amz-iv') + expect(data[:metadata]['x-amz-iv']).not_to be_empty + end + + it 'has x-amz-unencrypted-content-length in metadata' do + ##= ../specification/s3-encryption/data-format/content-metadata.md#content-metadata-mapkeys + ##= type=test + ##% - The mapkey "x-amz-unencrypted-content-length" SHOULD be present for V1 format objects. + + client = Aws::S3::Encryption::Client.new( + encryption_key: key, + client: s3_client + ) + + data = stub_put(s3_client) + client.put_object(bucket: test_bucket, key: test_object, body: plaintext) + + expect(data[:metadata]).to have_key('x-amz-unencrypted-content-length') + expect(data[:metadata]['x-amz-unencrypted-content-length']).to eq(plaintext.bytesize) + end + end + + context 'V2 Format with Object Metadata' do + let(:key) { OpenSSL::Cipher.new('aes-256-gcm').random_key } + let(:options) do + { + client: s3_client, + encryption_key: key, + key_wrap_schema: :aes_gcm, + commitment_policy: :forbid_encrypt_allow_decrypt, + content_encryption_schema: :aes_gcm_no_padding, + envelope_location: :metadata + } + end + + it 'has x-amz-key-v2 in metadata' do + ##= ../specification/s3-encryption/data-format/content-metadata.md#content-metadata-mapkeys + ##= type=test + ##% - The mapkey "x-amz-key-v2" MUST be present for V2 format objects. + + client = Client.new(options) + data = stub_put(s3_client) + client.put_object(bucket: test_bucket, key: test_object, body: plaintext) + + expect(data[:metadata]).to have_key('x-amz-key-v2') + expect(data[:metadata]['x-amz-key-v2']).not_to be_empty + end + + it 'has x-amz-matdesc in metadata' do + ##= ../specification/s3-encryption/data-format/content-metadata.md#content-metadata-mapkeys + ##= type=test + ##% - The mapkey "x-amz-matdesc" MUST be present for V2 format objects. + + client = Client.new(options) + data = stub_put(s3_client) + client.put_object(bucket: test_bucket, key: test_object, body: plaintext) + + expect(data[:metadata]).to have_key('x-amz-matdesc') + expect(data[:metadata]['x-amz-matdesc']).not_to be_empty + end + + it 'has x-amz-iv in metadata' do + ##= ../specification/s3-encryption/data-format/content-metadata.md#content-metadata-mapkeys + ##= type=test + ##% - The mapkey "x-amz-iv" MUST be present for V2 format objects. + + client = Client.new(options) + data = stub_put(s3_client) + client.put_object(bucket: test_bucket, key: test_object, body: plaintext) + + expect(data[:metadata]).to have_key('x-amz-iv') + expect(data[:metadata]['x-amz-iv']).not_to be_empty + end + + it 'has x-amz-wrap-alg in metadata' do + ##= ../specification/s3-encryption/data-format/content-metadata.md#content-metadata-mapkeys + ##= type=test + ##% - The mapkey "x-amz-wrap-alg" MUST be present for V2 format objects. + + client = Client.new(options) + data = stub_put(s3_client) + client.put_object(bucket: test_bucket, key: test_object, body: plaintext) + + expect(data[:metadata]).to have_key('x-amz-wrap-alg') + expect(data[:metadata]['x-amz-wrap-alg']).to eq('AES/GCM') + end + + it 'has x-amz-cek-alg in metadata' do + ##= ../specification/s3-encryption/data-format/content-metadata.md#content-metadata-mapkeys + ##= type=test + ##% - The mapkey "x-amz-cek-alg" MUST be present for V2 format objects. + + client = Client.new(options) + data = stub_put(s3_client) + client.put_object(bucket: test_bucket, key: test_object, body: plaintext) + + expect(data[:metadata]).to have_key('x-amz-cek-alg') + expect(data[:metadata]['x-amz-cek-alg']).to eq('AES/GCM/NoPadding') + end + end + + context 'V3 Format with Object Metadata' do + context 'with AES key' do + let(:key) { OpenSSL::Cipher.new('aes-256-gcm').random_key } + let(:options) do + { + client: s3_client, + encryption_key: key, + key_wrap_schema: :aes_gcm, + envelope_location: :metadata + } + end + + it 'has x-amz-c in metadata' do + ##= ../specification/s3-encryption/data-format/content-metadata.md#content-metadata-mapkeys + ##= type=test + ##% - The mapkey "x-amz-c" MUST be present for V3 format objects. + + client = Client.new(options) + data = stub_put(s3_client) + client.put_object(bucket: test_bucket, key: test_object, body: plaintext) + + expect(data[:metadata]).to have_key('x-amz-c') + expect(data[:metadata]['x-amz-c']).to eq('115') + end + + it 'has x-amz-3 in metadata' do + ##= ../specification/s3-encryption/data-format/content-metadata.md#content-metadata-mapkeys + ##= type=test + ##% - The mapkey "x-amz-3" MUST be present for V3 format objects. + + client = Client.new(options) + data = stub_put(s3_client) + client.put_object(bucket: test_bucket, key: test_object, body: plaintext) + + expect(data[:metadata]).to have_key('x-amz-3') + expect(data[:metadata]['x-amz-3']).not_to be_empty + end + + it 'has x-amz-w in metadata' do + ##= ../specification/s3-encryption/data-format/content-metadata.md#content-metadata-mapkeys + ##= type=test + ##% - The mapkey "x-amz-w" MUST be present for V3 format objects. + + client = Client.new(options) + data = stub_put(s3_client) + client.put_object(bucket: test_bucket, key: test_object, body: plaintext) + + expect(data[:metadata]).to have_key('x-amz-w') + expect(data[:metadata]['x-amz-w']).to eq('02') + end + + it 'writes wrapping algorithm value 02 for AES/GCM' do + ##= ../specification/s3-encryption/data-format/content-metadata.md#v3-only + ##= type=test + ##% - The wrapping algorithm value "02" MUST be translated to AES/GCM upon retrieval, and vice versa on write. + + client = Client.new(options) + data = stub_put(s3_client) + client.put_object(bucket: test_bucket, key: test_object, body: plaintext) + + expect(data[:metadata]['x-amz-w']).to eq('02') + end + + it 'has x-amz-d in metadata' do + ##= ../specification/s3-encryption/data-format/content-metadata.md#content-metadata-mapkeys + ##= type=test + ##% - The mapkey "x-amz-d" MUST be present for V3 format objects. + + client = Client.new(options) + data = stub_put(s3_client) + client.put_object(bucket: test_bucket, key: test_object, body: plaintext) + + expect(data[:metadata]).to have_key('x-amz-d') + expect(data[:metadata]['x-amz-d']).not_to be_empty + end + + it 'has x-amz-i in metadata' do + ##= ../specification/s3-encryption/data-format/content-metadata.md#content-metadata-mapkeys + ##= type=test + ##% - The mapkey "x-amz-i" MUST be present for V3 format objects. + + client = Client.new(options) + data = stub_put(s3_client) + client.put_object(bucket: test_bucket, key: test_object, body: plaintext) + + expect(data[:metadata]).to have_key('x-amz-i') + expect(data[:metadata]['x-amz-i']).not_to be_empty + end + + it 'has x-amz-m in metadata when materials description is provided' do + ##= ../specification/s3-encryption/data-format/content-metadata.md#content-metadata-mapkeys + ##= type=test + ##% - The mapkey "x-amz-m" SHOULD be present for V3 format objects that use Raw Keyring Material Description. + + materials_desc = '{"description":"test-materials"}' + client = Client.new(options.merge(materials_description: materials_desc)) + data = stub_put(s3_client) + client.put_object(bucket: test_bucket, key: test_object, body: plaintext) + + expect(data[:metadata]).to have_key('x-amz-m') + expect(data[:metadata]['x-amz-m']).to eq(materials_desc) + end + + it 'uses Material Description for AES/GCM wrapping' do + ##= ../specification/s3-encryption/data-format/content-metadata.md#v3-only + ##= type=test + ##% The Material Description MUST be used for wrapping algorithms `AES/GCM` (`02`) and `RSA-OAEP-SHA1` (`22`). + + materials_desc = '{"description":"test-materials"}' + client = Client.new(options.merge(materials_description: materials_desc)) + data = stub_put(s3_client) + client.put_object(bucket: test_bucket, key: test_object, body: plaintext) + + # For AES/GCM (02), material description should be present + expect(data[:metadata]).to have_key('x-amz-m') + expect(data[:metadata]['x-amz-w']).to eq('02') + end + end + + context 'with RSA key' do + let(:key) { OpenSSL::PKey::RSA.new(1024) } + let(:options) do + { + client: s3_client, + encryption_key: key, + key_wrap_schema: :rsa_oaep_sha1, + envelope_location: :metadata + } + end + + it 'has x-amz-c in metadata' do + client = Client.new(options) + data = stub_put(s3_client) + client.put_object(bucket: test_bucket, key: test_object, body: plaintext) + + expect(data[:metadata]).to have_key('x-amz-c') + expect(data[:metadata]['x-amz-c']).to eq('115') + end + + it 'has x-amz-3 in metadata' do + client = Client.new(options) + data = stub_put(s3_client) + client.put_object(bucket: test_bucket, key: test_object, body: plaintext) + + expect(data[:metadata]).to have_key('x-amz-3') + expect(data[:metadata]['x-amz-3']).not_to be_empty + end + + it 'has x-amz-w in metadata with value 22 for RSA-OAEP-SHA1' do + client = Client.new(options) + data = stub_put(s3_client) + client.put_object(bucket: test_bucket, key: test_object, body: plaintext) + + expect(data[:metadata]).to have_key('x-amz-w') + expect(data[:metadata]['x-amz-w']).to eq('22') + end + + it 'writes wrapping algorithm value 22 for RSA-OAEP-SHA1' do + ##= ../specification/s3-encryption/data-format/content-metadata.md#v3-only + ##= type=test + ##% - The wrapping algorithm value "22" MUST be translated to RSA-OAEP-SHA1 upon retrieval, and vice versa on write. + + client = Client.new(options) + data = stub_put(s3_client) + client.put_object(bucket: test_bucket, key: test_object, body: plaintext) + + expect(data[:metadata]['x-amz-w']).to eq('22') + end + + it 'uses Material Description for RSA-OAEP-SHA1 wrapping' do + ##= ../specification/s3-encryption/data-format/content-metadata.md#v3-only + ##= type=test + ##% The Material Description MUST be used for wrapping algorithms `AES/GCM` (`02`) and `RSA-OAEP-SHA1` (`22`). + + materials_desc = '{"description":"rsa-test"}' + client = Client.new(options.merge(materials_description: materials_desc)) + data = stub_put(s3_client) + client.put_object(bucket: test_bucket, key: test_object, body: plaintext) + + # For RSA-OAEP-SHA1 (22), material description should be present + expect(data[:metadata]).to have_key('x-amz-m') + expect(data[:metadata]['x-amz-m']).to eq(materials_desc) + expect(data[:metadata]['x-amz-w']).to eq('22') + end + + it 'has x-amz-d in metadata' do + client = Client.new(options) + data = stub_put(s3_client) + client.put_object(bucket: test_bucket, key: test_object, body: plaintext) + + expect(data[:metadata]).to have_key('x-amz-d') + expect(data[:metadata]['x-amz-d']).not_to be_empty + end + + it 'has x-amz-i in metadata' do + client = Client.new(options) + data = stub_put(s3_client) + client.put_object(bucket: test_bucket, key: test_object, body: plaintext) + + expect(data[:metadata]).to have_key('x-amz-i') + expect(data[:metadata]['x-amz-i']).not_to be_empty + end + end + + context 'with KMS key' do + let(:kms_client) { KMS::Client.new(stub_responses: true) } + let(:kms_key_id) { 'arn:aws:kms:us-west-2:123456789012:key/12345678-1234-1234-1234-123456789012' } + let(:kms_ciphertext_blob) do + Base64.decode64("AQIDAHiWj6qDEnwihp7W7g6VZb1xqsat5jdSUdEaGhgZepHdLAGASCQI7LZz\nz7GzCpm6y4sHAAAAfjB8BgkqhkiG9w0BBwagbzBtAgEAMGgGCSqGSIb3DQEH\nATAeBglghkgBZQMEAS4wEQQMJMJe6d8DkRTWwlvtAgEQgDtBCwiibCTS8pb7\n6BYKklVjy+CmO9q3r6y4u/9jJ8lk9eg5GwiskmcBtPMcWogMzx/vh+/65Cjb\nsQBpLQ==\n") + end + let(:kms_plaintext) do + Base64.decode64("5V7JWe+UDRhv66TaDg+tP6JONf/GkTdXk6Jq61weM+w=\n") + end + let(:options) do + { + client: s3_client, + kms_key_id: kms_key_id, + key_wrap_schema: :kms_context, + kms_client: kms_client, + envelope_location: :metadata + } + end + + before do + kms_client.stub_responses( + :generate_data_key, + { + key_id: kms_key_id, + ciphertext_blob: kms_ciphertext_blob, + plaintext: kms_plaintext + } + ) + end + + it 'has x-amz-c in metadata' do + client = Client.new(options) + data = stub_put(s3_client) + client.put_object(bucket: test_bucket, key: test_object, body: plaintext) + + expect(data[:metadata]).to have_key('x-amz-c') + expect(data[:metadata]['x-amz-c']).to eq('115') + end + + it 'has x-amz-3 in metadata' do + client = Client.new(options) + data = stub_put(s3_client) + client.put_object(bucket: test_bucket, key: test_object, body: plaintext) + + expect(data[:metadata]).to have_key('x-amz-3') + expect(data[:metadata]['x-amz-3']).not_to be_empty + end + + it 'has x-amz-w in metadata with value 12 for KMS' do + client = Client.new(options) + data = stub_put(s3_client) + client.put_object(bucket: test_bucket, key: test_object, body: plaintext) + + expect(data[:metadata]).to have_key('x-amz-w') + expect(data[:metadata]['x-amz-w']).to eq('12') + end + + it 'writes wrapping algorithm value 12 for kms+context' do + ##= ../specification/s3-encryption/data-format/content-metadata.md#v3-only + ##= type=test + ##% - The wrapping algorithm value "12" MUST be translated to kms+context upon retrieval, and vice versa on write. + + client = Client.new(options) + data = stub_put(s3_client) + client.put_object(bucket: test_bucket, key: test_object, body: plaintext) + + expect(data[:metadata]['x-amz-w']).to eq('12') + end + + it 'has x-amz-d in metadata' do + client = Client.new(options) + data = stub_put(s3_client) + client.put_object(bucket: test_bucket, key: test_object, body: plaintext) + + expect(data[:metadata]).to have_key('x-amz-d') + expect(data[:metadata]['x-amz-d']).not_to be_empty + end + + it 'has x-amz-i in metadata' do + client = Client.new(options) + data = stub_put(s3_client) + client.put_object(bucket: test_bucket, key: test_object, body: plaintext) + + expect(data[:metadata]).to have_key('x-amz-i') + expect(data[:metadata]['x-amz-i']).not_to be_empty + end + + it 'has x-amz-t in metadata when KMS encryption context is provided' do + ##= ../specification/s3-encryption/data-format/content-metadata.md#content-metadata-mapkeys + ##= type=test + ##% - The mapkey "x-amz-t" SHOULD be present for V3 format objects that use KMS Encryption Context. + + enc_context = { 'department' => 'finance', 'project' => 'alpha' } + client = Client.new(options) + data = stub_put(s3_client) + client.put_object( + bucket: test_bucket, + key: test_object, + body: plaintext, + kms_encryption_context: enc_context + ) + + expect(data[:metadata]).to have_key('x-amz-t') + stored_context = JSON.parse(data[:metadata]['x-amz-t']) + expect(stored_context).to include('department' => 'finance') + expect(stored_context).to include('project' => 'alpha') + end + + it 'uses Encryption Context for kms+context wrapping' do + ##= ../specification/s3-encryption/data-format/content-metadata.md#v3-only + ##= type=test + ##% The Encryption Context value MUST be used for wrapping algorithm `kms+context` or `12`. + + enc_context = { 'department' => 'finance', 'project' => 'alpha' } + client = Client.new(options) + data = stub_put(s3_client) + client.put_object( + bucket: test_bucket, + key: test_object, + body: plaintext, + kms_encryption_context: enc_context + ) + + # For kms+context (12), encryption context should be present + expect(data[:metadata]).to have_key('x-amz-t') + expect(data[:metadata]['x-amz-w']).to eq('12') + end + end + end + + context 'Default Material Description' do + it 'defaults material description to empty map when not present' do + ##= ../specification/s3-encryption/data-format/content-metadata.md#v3-only + ##= type=test + ##% If the mapkey x-amz-m is not present, the default Material Description value MUST be set to an empty map (`{}`). + + key = OpenSSL::Cipher.new('aes-256-gcm').random_key + options = { + client: s3_client, + encryption_key: key, + key_wrap_schema: :aes_gcm, + envelope_location: :metadata + } + + client = Client.new(options) + data = stub_put(s3_client) + + # Put object without materials_description parameter + client.put_object(bucket: test_bucket, key: test_object, body: plaintext) + + # The default Material Description value should be an empty map + expect(data[:metadata]).to have_key('x-amz-m') + expect(data[:metadata]['x-amz-m']).to eq('{}') + end + end + + context 'Algorithm Suite and Message Format Version Compatibility' do + it 'allows ALG_AES_256_CBC_IV16_NO_KDF with V1 format' do + ##= ../specification/s3-encryption/data-format/content-metadata.md#algorithm-suite-and-message-format-version-compatibility + ##= type=test + ##% Objects encrypted with ALG_AES_256_CBC_IV16_NO_KDF MAY use either the V1 or V2 message format version. + + # V1 client uses ALG_AES_256_CBC_IV16_NO_KDF with V1 format + # This demonstrates that CBC is not restricted to a single format version + cbc_key = OpenSSL::Cipher.new('aes-256-cbc').random_key + client_v1 = Aws::S3::Encryption::Client.new( + encryption_key: cbc_key, + client: s3_client + ) + + data = stub_put(s3_client) + client_v1.put_object(bucket: test_bucket, key: test_object, body: plaintext) + + # Verify V1 format is used (has x-amz-key, no x-amz-key-v2 or x-amz-3) + expect(data[:metadata]).to have_key('x-amz-key') + expect(data[:metadata]).not_to have_key('x-amz-key-v2') + expect(data[:metadata]).not_to have_key('x-amz-3') + + # Verify it uses CBC algorithm (V1 default) + expect(data[:metadata]).to have_key('x-amz-iv') + expect(data[:metadata]).to have_key('x-amz-matdesc') + end + + it 'requires ALG_AES_256_GCM_IV12_TAG16_NO_KDF to use V2 format only' do + ##= ../specification/s3-encryption/data-format/content-metadata.md#algorithm-suite-and-message-format-version-compatibility + ##= type=test + ##% Objects encrypted with ALG_AES_256_GCM_IV12_TAG16_NO_KDF MUST use the V2 message format version only. + + # V3 client with forbid_encrypt_allow_decrypt uses ALG_AES_256_GCM_IV12_TAG16_NO_KDF + # which corresponds to V2 format + gcm_key = OpenSSL::Cipher.new('aes-256-gcm').random_key + client = Client.new( + client: s3_client, + encryption_key: gcm_key, + key_wrap_schema: :aes_gcm, + commitment_policy: :forbid_encrypt_allow_decrypt, + content_encryption_schema: :aes_gcm_no_padding + ) + + data = stub_put(s3_client) + client.put_object(bucket: test_bucket, key: test_object, body: plaintext) + + # Verify V2 format is used (has x-amz-key-v2, not x-amz-key or x-amz-3) + expect(data[:metadata]).to have_key('x-amz-key-v2') + expect(data[:metadata]).not_to have_key('x-amz-key') + expect(data[:metadata]).not_to have_key('x-amz-3') + + # Verify it uses GCM algorithm with V2 markers + expect(data[:metadata]['x-amz-cek-alg']).to eq('AES/GCM/NoPadding') + expect(data[:metadata]).to have_key('x-amz-wrap-alg') + expect(data[:metadata]).to have_key('x-amz-iv') + end + + it 'requires ALG_AES_256_GCM_HKDF_SHA512_COMMIT_KEY to use V3 format only' do + ##= ../specification/s3-encryption/data-format/content-metadata.md#algorithm-suite-and-message-format-version-compatibility + ##= type=test + ##% Objects encrypted with ALG_AES_256_GCM_HKDF_SHA512_COMMIT_KEY MUST use the V3 message format version only. + + # V3 client with default settings uses ALG_AES_256_GCM_HKDF_SHA512_COMMIT_KEY + gcm_key = OpenSSL::Cipher.new('aes-256-gcm').random_key + client = Client.new( + client: s3_client, + encryption_key: gcm_key, + key_wrap_schema: :aes_gcm + ) + + data = stub_put(s3_client) + client.put_object(bucket: test_bucket, key: test_object, body: plaintext) + + # Verify V3 format is used (has x-amz-3, not x-amz-key or x-amz-key-v2) + expect(data[:metadata]).to have_key('x-amz-3') + expect(data[:metadata]).not_to have_key('x-amz-key') + expect(data[:metadata]).not_to have_key('x-amz-key-v2') + + # Verify it uses the HKDF algorithm (suite ID 115) + expect(data[:metadata]['x-amz-c']).to eq('115') + + # Verify V3-specific keys are present + expect(data[:metadata]).to have_key('x-amz-w') + expect(data[:metadata]).to have_key('x-amz-d') + expect(data[:metadata]).to have_key('x-amz-i') + end + end + end + end + end +end
gems/aws-sdk-s3/spec/encryptionv3/cryptographic_spec.rb+605 −0 added@@ -0,0 +1,605 @@ +require_relative '../spec_helper' +require 'base64' +require 'openssl' + +module Aws + module S3 + module EncryptionV3 + + describe 'HKDF Key Derivation via EncryptHandler' do + + let(:next_handler) { double(call: nil) } + let(:handler) { EncryptHandler.new(next_handler) } + let(:s3_client) { S3::Client.new(stub_responses: true) } + let(:test_bucket) { 'test-bucket' } + let(:test_object) { 'test-object' } + let(:plaintext) { 'test data' } + let(:encryption_key) { OpenSSL::Cipher.new('aes-256-gcm').random_key } + + describe 'AES-GCM encryption with HKDF' do + let(:key_provider) { DefaultKeyProvider.new(encryption_key: encryption_key) } + let(:cipher_provider) { DefaultCipherProvider.new(key_provider: key_provider, key_wrap_schema: :aes_gcm) } + + it 'validates input keying material length equals 32 bytes' do + ##= ../specification/s3-encryption/key-derivation.md#hkdf-operation + ##= type=test + ##% - The length of the input keying material MUST equal the key derivation input length specified by the algorithm suite commit key derivation setting. + + # Verify that the generated data key has the correct length + allow(Utils).to receive(:generate_data_key).and_wrap_original do |m| + key = m.call + expect(key.bytesize).to eq(32) + key + end + + params = { bucket: test_bucket, key: test_object, body: plaintext } + context_enc = { cipher_provider: cipher_provider, envelope_location: :metadata } + http_response = double(on_headers: nil) + config = Struct.new(:user_agent_suffix).new + context = double(params: params, client: s3_client, :[] => context_enc, http_response: http_response, config: config) + + handler.call(context) + end + + it 'derives encryption key with correct parameters' do + ##= ../specification/s3-encryption/key-derivation.md#hkdf-operation + ##= type=test + ##% - The DEK input pseudorandom key MUST be the output from the extract step. + + # Verify derive_encryption_key is called with proper parameters + expect(Utils).to receive(:derive_encryption_key).and_call_original + + params = { bucket: test_bucket, key: test_object, body: plaintext } + context_enc = { cipher_provider: cipher_provider, envelope_location: :metadata } + http_response = double(on_headers: nil) + config = Struct.new(:user_agent_suffix).new + context = double(params: params, client: s3_client, :[] => context_enc, http_response: http_response, config: config) + + handler.call(context) + end + + it 'returns encryption key with correct length' do + ##= ../specification/s3-encryption/key-derivation.md#hkdf-operation + ##= type=test + ##% - The length of the output keying material MUST equal the encryption key length specified by the algorithm suite encryption settings. + + # Verify the derived encryption key has the correct length + allow(Utils).to receive(:derive_encryption_key).and_wrap_original do |m, *args| + key = m.call(*args) + expect(key.bytesize).to eq(32) + key + end + + params = { bucket: test_bucket, key: test_object, body: plaintext } + context_enc = { cipher_provider: cipher_provider, envelope_location: :metadata } + http_response = double(on_headers: nil) + config = Struct.new(:user_agent_suffix).new + context = double(params: params, client: s3_client, :[] => context_enc, http_response: http_response, config: config) + + handler.call(context) + end + + it 'derives commitment key with correct parameters' do + ##= ../specification/s3-encryption/key-derivation.md#hkdf-operation + ##= type=test + ##% - The CK input pseudorandom key MUST be the output from the extract step. + + # Verify derive_commitment_key is called + expect(Utils).to receive(:derive_commitment_key).and_call_original + + params = { bucket: test_bucket, key: test_object, body: plaintext } + context_enc = { cipher_provider: cipher_provider, envelope_location: :metadata } + http_response = double(on_headers: nil) + config = Struct.new(:user_agent_suffix).new + context = double(params: params, client: s3_client, :[] => context_enc, http_response: http_response, config: config) + + handler.call(context) + end + + it 'returns commitment key with correct length' do + ##= ../specification/s3-encryption/encryption.md#alg-aes-256-gcm-hkdf-sha512-commit-key + ##= type=test + ##% The derived key commitment value MUST be set or returned from the encryption process such that it can be included in the content metadata. + ##= ../specification/s3-encryption/key-derivation.md#hkdf-operation + ##= type=test + ##% - The length of the output keying material MUST equal the commit key length specified by the supported algorithm suites. + + # Verify the derived commitment key has the correct length (28 bytes) + allow(Utils).to receive(:derive_commitment_key).and_wrap_original do |m, *args| + key = m.call(*args) + expect(key.bytesize).to eq(28) + key + end + + params = { bucket: test_bucket, key: test_object, body: plaintext } + context_enc = { cipher_provider: cipher_provider, envelope_location: :metadata } + http_response = double(on_headers: nil) + config = Struct.new(:user_agent_suffix).new + context = double(params: params, client: s3_client, :[] => context_enc, http_response: http_response, config: config) + + handler.call(context) + end + + it 'uses IV containing only 0x01 for AES-GCM encryption' do + ##= ../specification/s3-encryption/key-derivation.md#hkdf-operation + ##= type=test + ##% When encrypting or decrypting with ALG_AES_256_GCM_HKDF_SHA512_COMMIT_KEY, + ##% the IV used in the AES-GCM content encryption/decryption MUST consist entirely of bytes with the value 0x01. + ##= ../specification/s3-encryption/key-derivation.md#hkdf-operation + ##= type=test + ##% The IV's total length MUST match the IV length defined by the algorithm suite. + + v3_iv_bytes = "\x01" * 12 + iv_used = false + + # aes_cipher is called multiple times - check that at least one uses zero IV + allow(Utils).to receive(:aes_cipher).and_wrap_original do |m, mode, block_mode, key, iv| + if iv == v3_iv_bytes && block_mode == :GCM + iv_used = true + end + m.call(mode, block_mode, key, iv) + end + + params = { bucket: test_bucket, key: test_object, body: plaintext } + context_enc = { cipher_provider: cipher_provider, envelope_location: :metadata } + http_response = double(on_headers: nil) + config = Struct.new(:user_agent_suffix).new + context = double(params: params, client: s3_client, :[] => context_enc, http_response: http_response, config: config) + + handler.call(context) + expect(iv_used).to be true + end + + it 'initializes cipher with derived encryption key and zero IV' do + ##= ../specification/s3-encryption/key-derivation.md#hkdf-operation + ##= type=test + ##% The client MUST initialize the cipher, or call an AES-GCM encryption API, with the derived encryption key, an IV containing only bytes with the value 0x01, + ##% and the tag length defined in the Algorithm Suite when encrypting or decrypting with ALG_AES_256_GCM_HKDF_SHA512_COMMIT_KEY. + + # Verify aes_cipher is called with the derived encryption key and zero IV + derived_key = nil + allow(Utils).to receive(:derive_encryption_key).and_wrap_original do |m, *args| + derived_key = m.call(*args) + derived_key + end + + v3_iv_bytes = "\x01" * 12 + correct_cipher_init = false + + allow(Utils).to receive(:aes_cipher).and_wrap_original do |m, mode, block_mode, key, iv| + if key == derived_key && iv == v3_iv_bytes && block_mode == :GCM + correct_cipher_init = true + end + m.call(mode, block_mode, key, iv) + end + + params = { bucket: test_bucket, key: test_object, body: plaintext } + context_enc = { cipher_provider: cipher_provider, envelope_location: :metadata } + http_response = double(on_headers: nil) + config = Struct.new(:user_agent_suffix).new + context = double(params: params, client: s3_client, :[] => context_enc, http_response: http_response, config: config) + + handler.call(context) + expect(correct_cipher_init).to be true + end + + it 'sets AAD to the Algorithm Suite ID' do + ##= ../specification/s3-encryption/key-derivation.md#hkdf-operation + ##= type=test + ##% The client MUST set the AAD to the Algorithm Suite ID represented as bytes. + + # Verify the cipher's AAD is set correctly by checking the envelope + params = { bucket: test_bucket, key: test_object, body: plaintext } + context_enc = { cipher_provider: cipher_provider, envelope_location: :metadata } + http_response = double(on_headers: nil) + config = Struct.new(:user_agent_suffix).new + context = double(params: params, client: s3_client, :[] => context_enc, http_response: http_response, config: config) + + handler.call(context) + + # Verify the envelope contains properly formatted values + envelope = params[:metadata] + expect(envelope['x-amz-c']).to eq('115') # Correct algorithm + expect(envelope['x-amz-i']).not_to be_nil # Message ID present + expect(envelope['x-amz-d']).not_to be_nil # Commitment key present + end + + # The following tests use spy/mock on OpenSSL::KDF to verify HKDF behavior. + # OpenSSL::KDF is not available in all Ruby versions, so these tests + # are conditionally executed only when OpenSSL::KDF.hkdf is available. + context 'when OpenSSL::KDF is available', if: defined?(OpenSSL::KDF) && OpenSSL::KDF.respond_to?(:hkdf) do + it 'uses SHA512 as the hash function in HKDF' do + ##= ../specification/s3-encryption/encryption.md#alg-aes-256-gcm-hkdf-sha512-commit-key + ##= type=test + ##% The client MUST use HKDF to derive the key commitment value and the derived encrypting key as described in [Key Derivation](key-derivation.md). + ##= ../specification/s3-encryption/key-derivation.md#hkdf-operation + ##= type=test + ##% - The hash function MUST be specified by the algorithm suite commitment settings. + + # Spy on HKDF to verify SHA512 is used + expect(OpenSSL::KDF).to receive(:hkdf).at_least(:once).with( + anything, + hash_including(hash: an_instance_of(OpenSSL::Digest::SHA512)) + ).and_call_original + + params = { bucket: test_bucket, key: test_object, body: plaintext } + context_enc = { cipher_provider: cipher_provider, envelope_location: :metadata } + http_response = double(on_headers: nil) + config = Struct.new(:user_agent_suffix).new + context = double(params: params, client: s3_client, :[] => context_enc, http_response: http_response, config: config) + + handler.call(context) + end + + it 'uses the plaintext data key as input keying material' do + ##= ../specification/s3-encryption/key-derivation.md#hkdf-operation + ##= type=test + ##% - The input keying material MUST be the plaintext data key (PDK) generated by the key provider. + + # Capture the data key generated and verify it's used in HKDF (both calls) + data_key = nil + allow(Utils).to receive(:generate_data_key).and_wrap_original do |m| + data_key = m.call + data_key + end + + # Track calls to HKDF to verify the data key is used + hkdf_calls = [] + allow(OpenSSL::KDF).to receive(:hkdf).and_wrap_original do |m, key, opts| + hkdf_calls << key + m.call(key, **opts) + end + + params = { bucket: test_bucket, key: test_object, body: plaintext } + context_enc = { cipher_provider: cipher_provider, envelope_location: :metadata } + http_response = double(on_headers: nil) + config = Struct.new(:user_agent_suffix).new + context = double(params: params, client: s3_client, :[] => context_enc, http_response: http_response, config: config) + + handler.call(context) + + expect(data_key).not_to be_nil + expect(data_key.bytesize).to eq(32) + # Verify HKDF was called at least twice with the data_key + expect(hkdf_calls.count(data_key)).to be >= 2 + end + + it 'uses the Message ID as salt in HKDF' do + ##= ../specification/s3-encryption/key-derivation.md#hkdf-operation + ##= type=test + ##% - The salt MUST be the Message ID with the length defined in the algorithm suite. + + # Capture the message ID and verify it's used as salt in both HKDF calls + message_id = nil + allow(Utils).to receive(:generate_message_id).and_wrap_original do |m| + message_id = m.call + message_id + end + + # Track calls to HKDF to verify the message_id is used as salt + hkdf_salts = [] + allow(OpenSSL::KDF).to receive(:hkdf).and_wrap_original do |m, key, opts| + hkdf_salts << opts[:salt] + m.call(key, **opts) + end + + params = { bucket: test_bucket, key: test_object, body: plaintext } + context_enc = { cipher_provider: cipher_provider, envelope_location: :metadata } + http_response = double(on_headers: nil) + config = Struct.new(:user_agent_suffix).new + context = double(params: params, client: s3_client, :[] => context_enc, http_response: http_response, config: config) + + handler.call(context) + + expect(message_id).not_to be_nil + expect(message_id.bytesize).to eq(28) + # Verify HKDF was called at least twice with the message_id as salt + expect(hkdf_salts.count(message_id)).to be >= 2 + end + + it 'uses algorithm suite ID + DERIVEKEY as info parameter for encryption key' do + ##= ../specification/s3-encryption/key-derivation.md#hkdf-operation + ##= type=test + ##% - The input info MUST be a concatenation of the algorithm suite ID as bytes followed by the string DERIVEKEY as UTF8 encoded bytes. + + expected_info = [0x00, 0x73].pack('C*') + "DERIVEKEY".encode('UTF-8') + derivekey_called = false + + # HKDF is called twice - once with DERIVEKEY, once with COMMITKEY + allow(OpenSSL::KDF).to receive(:hkdf).and_wrap_original do |m, key, opts| + if opts[:info] == expected_info + derivekey_called = true + end + m.call(key, **opts) + end + + params = { bucket: test_bucket, key: test_object, body: plaintext } + context_enc = { cipher_provider: cipher_provider, envelope_location: :metadata } + http_response = double(on_headers: nil) + config = Struct.new(:user_agent_suffix).new + context = double(params: params, client: s3_client, :[] => context_enc, http_response: http_response, config: config) + + handler.call(context) + expect(derivekey_called).to be true + end + + it 'uses algorithm suite ID + COMMITKEY as info parameter for commitment key' do + ##= ../specification/s3-encryption/key-derivation.md#hkdf-operation + ##= type=test + ##% - The input info MUST be a concatenation of the algorithm suite ID as bytes followed by the string COMMITKEY as UTF8 encoded bytes. + + expected_info = [0x00, 0x73].pack('C*') + "COMMITKEY".encode('UTF-8') + commitkey_called = false + + # HKDF is called twice - once with DERIVEKEY, once with COMMITKEY + allow(OpenSSL::KDF).to receive(:hkdf).and_wrap_original do |m, key, opts| + if opts[:info] == expected_info + commitkey_called = true + end + m.call(key, **opts) + end + + params = { bucket: test_bucket, key: test_object, body: plaintext } + context_enc = { cipher_provider: cipher_provider, envelope_location: :metadata } + http_response = double(on_headers: nil) + config = Struct.new(:user_agent_suffix).new + context = double(params: params, client: s3_client, :[] => context_enc, http_response: http_response, config: config) + + handler.call(context) + expect(commitkey_called).to be true + end + + describe 'hkdf_fallback compatibility' do + it 'raises an error when desired_length exceeds 64 bytes' do + # The hkdf_fallback implementation only supports a single iteration + # of the HKDF expand step, which limits output to 64 bytes (SHA512 hash length) + input_key = OpenSSL::Random.random_bytes(32) + salt = OpenSSL::Random.random_bytes(28) + info = OpenSSL::Random.random_bytes(20) + + expect do + Utils.hkdf_fallback(input_key, salt, info, 65) + end.to raise_error(Errors::DecryptionError, /Unsupported length/) + + expect do + Utils.hkdf_fallback(input_key, salt, info, 100) + end.to raise_error(Errors::DecryptionError, /Unsupported length/) + end + + it 'produces identical output to native hkdf for random inputs (property-based test)' do + # Test that hkdf_fallback matches the Utils.hkdf (which uses native OpenSSL::KDF.hkdf) + # for a variety of random inputs within the supported range + + test_cases = 30 + test_cases.times do + # Generate random parameters with various sizes + ikm_size = [16, 32, 64].sample + salt_size = [16, 28, 32].sample + info_size = [0, 10, 20, 50].sample + desired_length = rand(1..64) + + input_key_material = OpenSSL::Random.random_bytes(ikm_size) + salt = OpenSSL::Random.random_bytes(salt_size) + info = info_size > 0 ? OpenSSL::Random.random_bytes(info_size) : ''.b + + # Get results from both implementations + fallback_result = Utils.hkdf_fallback( + input_key_material, + salt, + info, + desired_length + ) + + native_result = Utils.hkdf( + input_key_material, + salt, + info, + desired_length + ) + + # Verify they produce identical output + expect(fallback_result).to eq(native_result), + "hkdf_fallback output differs from Utils.hkdf for:\n" \ + " ikm_size=#{ikm_size}, salt_size=#{salt_size}, " \ + "info_size=#{info_size}, desired_length=#{desired_length}" + + # Verify output length + expect(fallback_result.bytesize).to eq(desired_length) + end + end + + it 'produces identical output to native hkdf for production values' do + # Test with the actual values used in production code + data_key = OpenSSL::Random.random_bytes(32) + message_id = OpenSSL::Random.random_bytes(28) + + # Define the info constants as they appear in utils.rb + encryption_key_info = [0x00, 0x73].pack('C*') + "DERIVEKEY".encode('UTF-8') + commitment_key_info = [0x00, 0x73].pack('C*') + "COMMITKEY".encode('UTF-8') + + # Test with ENCRYPTION_KEY_INFO + fallback_enc = Utils.hkdf_fallback( + data_key, + message_id, + encryption_key_info, + 32 + ) + + native_enc = Utils.hkdf( + data_key, + message_id, + encryption_key_info, + 32 + ) + + expect(fallback_enc).to eq(native_enc) + expect(fallback_enc.bytesize).to eq(32) + + # Test with COMMITMENT_KEY_INFO + fallback_commit = Utils.hkdf_fallback( + data_key, + message_id, + commitment_key_info, + 28 + ) + + native_commit = Utils.hkdf( + data_key, + message_id, + commitment_key_info, + 28 + ) + + expect(fallback_commit).to eq(native_commit) + expect(fallback_commit.bytesize).to eq(28) + end + end + end + end + + describe 'Non-HKDF GCM encryption' do + let(:key_provider) { DefaultKeyProvider.new(encryption_key: encryption_key) } + + it 'verifies auth tag is appended to ciphertext for HKDF algorithm' do + ##= ../specification/s3-encryption/encryption.md#alg-aes-256-gcm-hkdf-sha512-commit-key + ##= type=test + ##% The client MUST append the GCM auth tag to the ciphertext if the underlying crypto provider does not do so automatically. + cipher_provider = DefaultCipherProvider.new(key_provider: key_provider, key_wrap_schema: :aes_gcm) + + data = {} + s3_client.stub_responses(:put_object, lambda { |context| + data[:metadata] = context.params[:metadata] + data[:enc_body] = context.params[:body].read + {} + }) + + client = Aws::S3::EncryptionV3::Client.new( + client: s3_client, + encryption_key: encryption_key, + key_wrap_schema: :aes_gcm + ) + + client.put_object(bucket: test_bucket, key: test_object, body: plaintext) + + # The plaintext + authTag should b plaintext + 16 + expect(data[:enc_body].bytesize).to eq(plaintext.bytesize + 16) + auth_tag_candidate = data[:enc_body][-16..-1] + expect(auth_tag_candidate.bytesize).to eq(16) + + # Verify successful decryption (which confirms tag is valid) + resp_headers = Hash[*data[:metadata].map { |k, v| ["x-amz-meta-#{k.to_s}", v] }.flatten(1)] + resp_headers['content-length'] = data[:enc_body].length + s3_client.stub_responses( + :get_object, + {status_code: 200, body: data[:enc_body], headers: resp_headers}, + {body: auth_tag_candidate} + ) + decrypted = client.get_object(bucket: test_bucket, key: test_object).body.read + expect(decrypted).to eq(plaintext) + end + end + + describe 'Algorithm suite validation' do + it 'uses plaintext data key, random IV, and no AAD for non-HKDF GCM encryption' do + ##= ../specification/s3-encryption/encryption.md#alg-aes-256-gcm-iv12-tag16-no-kdf + ##= type=test + ##% The client MUST initialize the cipher, or call an AES-GCM encryption API, + ##% with the plaintext data key, the generated IV, + ##% and the tag length defined in the Algorithm Suite when encrypting with ALG_AES_256_GCM_IV12_TAG16_NO_KDF. + ##= ../specification/s3-encryption/encryption.md#alg-aes-256-gcm-iv12-tag16-no-kdf + ##= type=test + ##% The client MUST NOT provide any AAD when encrypting with ALG_AES_256_GCM_IV12_TAG16_NO_KDF. + ##= ../specification/s3-encryption/encryption.md#alg-aes-256-gcm-iv12-tag16-no-kdf + ##= type=test + ##% The client MUST append the GCM auth tag to the ciphertext if the underlying crypto provider does not do so automatically. + + # With forbid_encrypt_allow_decrypt policy, the client uses non-HKDF (V2) encryption + client = Aws::S3::EncryptionV3::Client.new( + client: s3_client, + encryption_key: encryption_key, + key_wrap_schema: :aes_gcm, + commitment_policy: :forbid_encrypt_allow_decrypt, + content_encryption_schema: :aes_gcm_no_padding + ) + + # Track the plaintext data key and cipher initialization + plaintext_data_key = nil + random_iv = nil + cipher_auth_data = nil + cipher_initialized = false + + # Spy on V2 aes_encryption_cipher to capture the plaintext data key and IV + allow(Aws::S3::EncryptionV2::Utils).to receive(:aes_encryption_cipher).and_wrap_original do |m, block_mode, key, iv| + cipher = m.call(block_mode, key, iv) + + # Wrap the cipher to capture its key, iv, and auth_data + allow(cipher).to receive(:key=).and_wrap_original do |method, k| + plaintext_data_key = k + method.call(k) + end + + allow(cipher).to receive(:iv=).and_wrap_original do |method, i| + random_iv = i + method.call(i) + end + + allow(cipher).to receive(:auth_data=).and_wrap_original do |method, ad| + cipher_auth_data = ad + cipher_initialized = true if block_mode == :GCM + method.call(ad) + end + + cipher + end + + # Stub S3 response + data = {} + s3_client.stub_responses(:put_object, lambda { |context| + data[:metadata] = context.params[:metadata] + data[:body] = context.params[:body].read + {} + }) + + # Perform encryption + client.put_object(bucket: test_bucket, key: test_object, body: plaintext) + + # Verify: Cipher was initialized with plaintext data key (not derived via HKDF) + expect(plaintext_data_key).not_to be_nil + expect(plaintext_data_key.bytesize).to eq(32) # 256-bit key + + # Verify: Random IV was generated (not zeros like HKDF) + expect(random_iv).not_to be_nil + expect(random_iv.bytesize).to eq(12) # 12-byte IV for GCM + expect(random_iv).not_to eq("\x00" * 12) # Must NOT be all zeros + + # Verify: NO AAD was provided (empty string, not algorithm suite ID) + expect(cipher_initialized).to be true + expect(cipher_auth_data).to eq('') # Empty AAD, not algorithm suite ID + + # Verify: Auth tag is appended to ciphertext + # Ciphertext should be: plaintext.length + 16 bytes for auth tag + expect(data[:body].bytesize).to eq(plaintext.bytesize + 16) + + # Verify the last 16 bytes are the auth tag (non-zero for valid encryption) + auth_tag = data[:body][-16..-1] + expect(auth_tag.bytesize).to eq(16) + expect(auth_tag).not_to eq("\x00" * 16) # Auth tag should not be all zeros + end + + it 'rejects non-HKDF GCM with default policy' do + # V3 client with default policy (require_encrypt_require_decrypt) only supports HKDF algorithms + expect do + Aws::S3::EncryptionV3::Client.new( + client: s3_client, + encryption_key: encryption_key, + key_wrap_schema: :aes_gcm, + content_encryption_schema: :aes_gcm_no_padding + ) + end.to raise_error(ArgumentError, /content_encryption_schema/) + end + end + end + + end + end +end
gems/aws-sdk-s3/spec/encryptionv3/decrypt_handler_spec.rb+595 −0 added@@ -0,0 +1,595 @@ +require_relative '../spec_helper' +require 'base64' +require 'openssl' + +module Aws + module S3 + module EncryptionV3 + describe 'DecryptHandler - Determining S3EC Object Status' do + let(:plaintext) { 'super secret plain text' } + let(:test_bucket) { 'test-bucket' } + let(:test_object) { 'test-key' } + let(:s3_client) { S3::Client.new(stub_responses: true) } + let(:key) { OpenSSL::Cipher.new('aes-256-gcm').random_key } + + # Helper to stub get_object with specific metadata + def stub_get_with_metadata(s3_client, metadata, body = 'encrypted-content') + resp_headers = metadata.map { |k, v| ["x-amz-meta-#{k}", v] }.to_h + resp_headers['content-length'] = body.length.to_s + s3_client.stub_responses(:get_object, { + status_code: 200, + body: body, + headers: resp_headers + }) + end + + context 'V1 format detection' do + it 'identifies V1 format with x-amz-iv and x-amz-key' do + ##= ../specification/s3-encryption/data-format/content-metadata.md#determining-s3ec-object-status + ##= type=test + ##% - If the metadata contains "x-amz-iv" and "x-amz-key" then the object MUST be considered as an S3EC-encrypted object using the V1 format. + + # Create V1 encrypted object + client_v1 = Aws::S3::Encryption::Client.new( + encryption_key: key, + client: s3_client + ) + + data = {} + s3_client.stub_responses(:put_object, lambda { |context| + data[:metadata] = context.params[:metadata] + data[:enc_body] = context.params[:body].read + {} + }) + + client_v1.put_object(bucket: test_bucket, key: test_object, body: plaintext) + + # Verify V1 metadata is present + expect(data[:metadata]).to have_key('x-amz-key') + expect(data[:metadata]).to have_key('x-amz-iv') + + # V3 client with legacy support should be able to decrypt it + client_v3 = Client.new( + encryption_key: key, + key_wrap_schema: :aes_gcm, + client: s3_client, + security_profile: :v3_and_legacy, + commitment_policy: :require_encrypt_allow_decrypt + ) + + # Stub get response with V1 metadata + resp_headers = data[:metadata].map { |k, v| ["x-amz-meta-#{k}", v] }.to_h + resp_headers['content-length'] = data[:enc_body].length.to_s + s3_client.stub_responses(:get_object, { + status_code: 200, + body: data[:enc_body], + headers: resp_headers + }) + + # Should successfully decrypt (format detection works) + decrypted = client_v3.get_object(bucket: test_bucket, key: test_object).body.read + expect(decrypted).to eq(plaintext) + end + end + + context 'V2 format detection' do + it 'identifies V2 format with x-amz-iv and x-amz-key-v2' do + ##= ../specification/s3-encryption/data-format/content-metadata.md#determining-s3ec-object-status + ##= type=test + ##% - If the metadata contains "x-amz-iv" and "x-amz-metadata-x-amz-key-v2" then the object MUST be considered as an S3EC-encrypted object using the V2 format. + + # Create V2 encrypted object (V3 client with forbid policy) + client_v2 = Client.new( + encryption_key: key, + key_wrap_schema: :aes_gcm, + client: s3_client, + commitment_policy: :forbid_encrypt_allow_decrypt, + content_encryption_schema: :aes_gcm_no_padding + ) + + data = {} + s3_client.stub_responses(:put_object, lambda { |context| + data[:metadata] = context.params[:metadata] + data[:enc_body] = context.params[:body].read + {} + }) + + client_v2.put_object(bucket: test_bucket, key: test_object, body: plaintext) + + # Verify V2 metadata is present + expect(data[:metadata]).to have_key('x-amz-key-v2') + expect(data[:metadata]).to have_key('x-amz-iv') + + # V3 client with legacy support should be able to decrypt it + client_v3 = Client.new( + encryption_key: key, + key_wrap_schema: :aes_gcm, + client: s3_client, + security_profile: :v3_and_legacy, + commitment_policy: :require_encrypt_allow_decrypt + ) + + # Stub get response with V2 metadata + resp_headers = data[:metadata].map { |k, v| ["x-amz-meta-#{k}", v] }.to_h + resp_headers['content-length'] = data[:enc_body].length.to_s + auth_tag = data[:enc_body].unpack('C*')[-16, 16].pack('C*') + s3_client.stub_responses(:get_object, + {status_code: 200, body: data[:enc_body], headers: resp_headers}, + {body: auth_tag} + ) + + # Should successfully decrypt (format detection works) + decrypted = client_v3.get_object(bucket: test_bucket, key: test_object).body.read + expect(decrypted).to eq(plaintext) + end + end + + context 'V3 format detection' do + it 'identifies V3 format with x-amz-3, x-amz-d, and x-amz-i' do + ##= ../specification/s3-encryption/data-format/content-metadata.md#determining-s3ec-object-status + ##= type=test + ##% - If the metadata contains "x-amz-3" and "x-amz-d" and "x-amz-i" then the object MUST be considered an S3EC-encrypted object using the V3 format. + + # Create V3 encrypted object + client_v3 = Client.new( + encryption_key: key, + key_wrap_schema: :aes_gcm, + client: s3_client + ) + + data = {} + s3_client.stub_responses(:put_object, lambda { |context| + data[:metadata] = context.params[:metadata] + data[:enc_body] = context.params[:body].read + {} + }) + + client_v3.put_object(bucket: test_bucket, key: test_object, body: plaintext) + + # Verify V3 metadata is present + expect(data[:metadata]).to have_key('x-amz-3') + expect(data[:metadata]).to have_key('x-amz-d') + expect(data[:metadata]).to have_key('x-amz-i') + expect(data[:metadata]).to have_key('x-amz-w') + expect(data[:metadata]).to have_key('x-amz-c') + + # Stub get response with V3 metadata + resp_headers = data[:metadata].map { |k, v| ["x-amz-meta-#{k}", v] }.to_h + resp_headers['content-length'] = data[:enc_body].length.to_s + auth_tag = data[:enc_body].unpack('C*')[-16, 16].pack('C*') + s3_client.stub_responses(:get_object, + {status_code: 200, body: data[:enc_body], headers: resp_headers}, + {body: auth_tag} + ) + + # Should successfully decrypt + decrypted = client_v3.get_object(bucket: test_bucket, key: test_object).body.read + expect(decrypted).to eq(plaintext) + end + end + + context 'Instruction file fallback' do + it 'attempts to get instruction file when metadata does not match V1/V2/V3 formats' do + ##= ../specification/s3-encryption/data-format/content-metadata.md#determining-s3ec-object-status + ##= type=test + ##% If the object matches none of the V1/V2/V3 formats, the S3EC MUST attempt to get the instruction file. + + # Create V3 object with instruction file + client_v3 = Client.new( + encryption_key: key, + key_wrap_schema: :aes_gcm, + client: s3_client, + envelope_location: :instruction_file + ) + + data = { metadata: {}, enc_body: nil, instruction_metadata: nil } + s3_client.stub_responses(:put_object, lambda { |context| + if context.params[:key].end_with?('.instruction') + data[:instruction_metadata] = JSON.parse(context.params[:body]) + else + data[:metadata] = context.params[:metadata] + data[:enc_body] = context.params[:body].read + end + {} + }) + + client_v3.put_object(bucket: test_bucket, key: test_object, body: plaintext) + + # Verify some metadata is in object but not all (triggers instruction file lookup) + expect(data[:metadata]).to have_key('x-amz-i') + expect(data[:metadata]).not_to have_key('x-amz-3') + expect(data[:instruction_metadata]).to have_key('x-amz-3') + + # Stub get responses - object metadata, instruction file, and auth tag + resp_headers = data[:metadata].map { |k, v| ["x-amz-meta-#{k}", v] }.to_h + resp_headers['content-length'] = data[:enc_body].length.to_s + auth_tag = data[:enc_body].unpack('C*')[-16, 16].pack('C*') + + s3_client.stub_responses(:get_object, + {status_code: 200, body: data[:enc_body], headers: resp_headers}, + {body: Json.dump(data[:instruction_metadata])}, + {body: auth_tag} + ) + + # Should successfully decrypt by fetching instruction file + decrypted = client_v3.get_object(bucket: test_bucket, key: test_object).body.read + expect(decrypted).to eq(plaintext) + end + end + + context 'Exclusive mapkeys conflict' do + it 'throws exception when both x-amz-key and x-amz-3 are present' do + ##= ../specification/s3-encryption/data-format/content-metadata.md#determining-s3ec-object-status + ##= type=test + ##% If there are multiple mapkeys which are meant to be exclusive, such as "x-amz-key", "x-amz-key-v2", and "x-amz-3" then the S3EC SHOULD throw an exception. + + client_v3 = Client.new( + encryption_key: key, + key_wrap_schema: :aes_gcm, + client: s3_client + ) + + # Create conflicting metadata - both V1 and V3 keys + conflicting_metadata = { + 'x-amz-key' => Base64.strict_encode64('v1-key'), + 'x-amz-3' => Base64.strict_encode64('v3-encrypted-key'), + 'x-amz-w' => '02', + 'x-amz-c' => '115', + 'x-amz-d' => Base64.strict_encode64('commitment'), + 'x-amz-i' => Base64.strict_encode64('message-id') + } + + stub_get_with_metadata(s3_client, conflicting_metadata) + + expect { + client_v3.get_object(bucket: test_bucket, key: test_object) + }.to raise_error(Errors::DecryptionError) + end + + it 'throws exception when both x-amz-key-v2 and x-amz-3 are present' do + ##= ../specification/s3-encryption/data-format/content-metadata.md#determining-s3ec-object-status + ##= type=test + ##% If there are multiple mapkeys which are meant to be exclusive, such as "x-amz-key", "x-amz-key-v2", and "x-amz-3" then the S3EC SHOULD throw an exception. + + client_v3 = Client.new( + encryption_key: key, + key_wrap_schema: :aes_gcm, + client: s3_client + ) + + # Create conflicting metadata - both V2 and V3 keys + conflicting_metadata = { + 'x-amz-key-v2' => Base64.strict_encode64('v2-key'), + 'x-amz-3' => Base64.strict_encode64('v3-encrypted-key'), + 'x-amz-w' => '02', + 'x-amz-c' => '115', + 'x-amz-d' => Base64.strict_encode64('commitment'), + 'x-amz-i' => Base64.strict_encode64('message-id') + } + + stub_get_with_metadata(s3_client, conflicting_metadata) + + expect { + client_v3.get_object(bucket: test_bucket, key: test_object) + }.to raise_error(Errors::DecryptionError) + end + end + + context 'Format deviation exception' do + it 'throws exception when V3 format is incomplete (missing required keys)' do + ##= ../specification/s3-encryption/data-format/content-metadata.md#determining-s3ec-object-status + ##= type=test + ##% In general, if there is any deviation from the above format, with the exception of additional unrelated mapkeys, then the S3EC SHOULD throw an exception. + + client_v3 = Client.new( + encryption_key: key, + key_wrap_schema: :aes_gcm, + client: s3_client + ) + + # Create incomplete V3 metadata - missing x-amz-w + incomplete_metadata = { + 'x-amz-3' => Base64.strict_encode64('encrypted-key'), + 'x-amz-c' => '115', + 'x-amz-d' => Base64.strict_encode64('commitment'), + 'x-amz-i' => Base64.strict_encode64('message-id') + } + + stub_get_with_metadata(s3_client, incomplete_metadata) + + expect { + client_v3.get_object(bucket: test_bucket, key: test_object) + }.to raise_error(Errors::DecryptionError, /unsupported key wrapping algorithm/) + end + + it 'throws exception when V3 format has unsupported content encryption algorithm' do + ##= ../specification/s3-encryption/data-format/content-metadata.md#determining-s3ec-object-status + ##= type=test + ##% In general, if there is any deviation from the above format, with the exception of additional unrelated mapkeys, then the S3EC SHOULD throw an exception. + + client_v3 = Client.new( + encryption_key: key, + key_wrap_schema: :aes_gcm, + client: s3_client + ) + + # Create V3 metadata with unsupported x-amz-c value + invalid_metadata = { + 'x-amz-3' => Base64.strict_encode64('encrypted-key'), + 'x-amz-w' => '02', + 'x-amz-c' => '999', # Invalid algorithm + 'x-amz-d' => Base64.strict_encode64('commitment'), + 'x-amz-i' => Base64.strict_encode64('message-id') + } + + stub_get_with_metadata(s3_client, invalid_metadata) + + expect { + client_v3.get_object(bucket: test_bucket, key: test_object) + }.to raise_error(Errors::DecryptionError, /unsupported content encrypting key/) + end + + it 'throws exception when V3 format has unsupported wrapping algorithm' do + ##= ../specification/s3-encryption/data-format/content-metadata.md#determining-s3ec-object-status + ##= type=test + ##% In general, if there is any deviation from the above format, with the exception of additional unrelated mapkeys, then the S3EC SHOULD throw an exception. + + client_v3 = Client.new( + encryption_key: key, + key_wrap_schema: :aes_gcm, + client: s3_client + ) + + # Create V3 metadata with unsupported x-amz-w value + invalid_metadata = { + 'x-amz-3' => Base64.strict_encode64('encrypted-key'), + 'x-amz-w' => '99', # Invalid wrapping algorithm + 'x-amz-c' => '115', + 'x-amz-d' => Base64.strict_encode64('commitment'), + 'x-amz-i' => Base64.strict_encode64('message-id') + } + + stub_get_with_metadata(s3_client, invalid_metadata) + + expect { + client_v3.get_object(bucket: test_bucket, key: test_object) + }.to raise_error(Errors::DecryptionError, /unsupported key wrapping algorithm/) + end + + it 'allows additional unrelated mapkeys in V3 format' do + ##= ../specification/s3-encryption/data-format/content-metadata.md#determining-s3ec-object-status + ##= type=test + ##% In general, if there is any deviation from the above format, with the exception of additional unrelated mapkeys, then the S3EC SHOULD throw an exception. + + # Create V3 encrypted object + client_v3 = Client.new( + encryption_key: key, + key_wrap_schema: :aes_gcm, + client: s3_client + ) + + data = {} + s3_client.stub_responses(:put_object, lambda { |context| + data[:metadata] = context.params[:metadata] + data[:enc_body] = context.params[:body].read + {} + }) + + client_v3.put_object(bucket: test_bucket, key: test_object, body: plaintext) + + # Add unrelated metadata keys + data[:metadata]['custom-key'] = 'custom-value' + data[:metadata]['another-key'] = 'another-value' + + # Stub get response with additional unrelated keys + resp_headers = data[:metadata].map { |k, v| ["x-amz-meta-#{k}", v] }.to_h + resp_headers['content-length'] = data[:enc_body].length.to_s + auth_tag = data[:enc_body].unpack('C*')[-16, 16].pack('C*') + s3_client.stub_responses(:get_object, + {status_code: 200, body: data[:enc_body], headers: resp_headers}, + {body: auth_tag} + ) + + # Should not raise an error - additional unrelated keys are allowed + expect { + decrypted = client_v3.get_object(bucket: test_bucket, key: test_object).body.read + expect(decrypted).to eq(plaintext) + }.not_to raise_error + end + end + + context 'Envelope merging with nil secondary source' do + it 'uses metadata when instruction file does not exist' do + # Test case 1: envelope_location: :instruction_file + # Instruction file doesn't exist (returns nil) + # Metadata has complete V3 envelope data + # Expected: Should successfully decrypt using metadata + + client_v3 = Client.new( + encryption_key: key, + key_wrap_schema: :aes_gcm, + client: s3_client, + envelope_location: :instruction_file + ) + + # Create complete V3 metadata + complete_metadata = { + 'x-amz-3' => Base64.strict_encode64('encrypted-key'), + 'x-amz-w' => '02', + 'x-amz-c' => '115', + 'x-amz-d' => Base64.strict_encode64('commitment'), + 'x-amz-i' => Base64.strict_encode64('message-id') + } + + # Stub get_object to return metadata but fail on instruction file + stub_get_with_metadata(s3_client, complete_metadata, 'encrypted-content') + s3_client.stub_responses(:get_object, + {status_code: 200, body: 'encrypted-content', headers: complete_metadata.map { |k, v| ["x-amz-meta-#{k}", v] }.to_h.merge('content-length' => '17')}, + Aws::S3::Errors::NoSuchKey.new(nil, 'Not Found') # instruction file doesn't exist + ) + + # Should not raise an error - envelope is nil but secondary (metadata) has complete data + expect { + envelope = Decryption.get_encryption_envelope( + OpenStruct.new( + http_response: OpenStruct.new( + headers: complete_metadata.map { |k, v| ["x-amz-meta-#{k}", v] }.to_h.merge('content-length' => '17') + ), + params: { bucket: test_bucket, key: test_object }, + encryption: { envelope_location: :instruction_file, instruction_file_suffix: '.instruction' }, + client: s3_client + ) + ) + expect(envelope).to be_a(Hash) + expect(envelope['x-amz-3']).not_to be_nil + expect(envelope['x-amz-w']).to eq('02') + }.not_to raise_error + end + + it 'fails when metadata is incomplete and instruction file does not exist' do + # Test case 2: envelope_location: :metadata + # Metadata has incomplete data (missing x-amz-3, x-amz-w) + # Instruction file doesn't exist (returns nil) + # Expected: Should fail with DecryptionError about incomplete envelope + + client_v3 = Client.new( + encryption_key: key, + key_wrap_schema: :aes_gcm, + client: s3_client, + envelope_location: :metadata + ) + + # Create incomplete metadata (missing envelope keys) + incomplete_metadata = { + 'x-amz-c' => '115', + 'x-amz-d' => Base64.strict_encode64('commitment'), + 'x-amz-i' => Base64.strict_encode64('message-id') + } + + stub_get_with_metadata(s3_client, incomplete_metadata) + # Stub instruction file to not exist + s3_client.stub_responses(:get_object, + {status_code: 200, body: 'encrypted-content', headers: incomplete_metadata.map { |k, v| ["x-amz-meta-#{k}", v] }.to_h.merge('content-length' => '17')}, + Aws::S3::Errors::NoSuchKey.new(nil, 'Not Found') + ) + + # Should raise DecryptionError because envelope is incomplete and secondary is nil + expect { + Decryption.get_encryption_envelope( + OpenStruct.new( + http_response: OpenStruct.new( + headers: incomplete_metadata.map { |k, v| ["x-amz-meta-#{k}", v] }.to_h.merge('content-length' => '17') + ), + params: { bucket: test_bucket, key: test_object }, + encryption: { envelope_location: :metadata, instruction_file_suffix: '.instruction' }, + client: s3_client + ) + ) + }.to raise_error(Errors::DecryptionError, /unsupported key wrapping algorithm/) + end + + it 'fails when instruction file does not exist and metadata is incomplete' do + # Test case 3: envelope_location: :instruction_file + # Instruction file doesn't exist (returns nil) + # Metadata is incomplete (only has metadata keys, missing envelope keys) + # Expected: Should fail with DecryptionError about incomplete envelope + + client_v3 = Client.new( + encryption_key: key, + key_wrap_schema: :aes_gcm, + client: s3_client, + envelope_location: :instruction_file + ) + + # Create incomplete metadata (missing envelope keys) + incomplete_metadata = { + 'x-amz-c' => '115', + 'x-amz-d' => Base64.strict_encode64('commitment'), + 'x-amz-i' => Base64.strict_encode64('message-id') + } + + stub_get_with_metadata(s3_client, incomplete_metadata) + # Stub instruction file to not exist + s3_client.stub_responses(:get_object, + {status_code: 200, body: 'encrypted-content', headers: incomplete_metadata.map { |k, v| ["x-amz-meta-#{k}", v] }.to_h.merge('content-length' => '17')}, + Aws::S3::Errors::NoSuchKey.new(nil, 'Not Found') + ) + + # Should raise DecryptionError because envelope is nil but secondary is incomplete + expect { + Decryption.get_encryption_envelope( + OpenStruct.new( + http_response: OpenStruct.new( + headers: incomplete_metadata.map { |k, v| ["x-amz-meta-#{k}", v] }.to_h.merge('content-length' => '17') + ), + params: { bucket: test_bucket, key: test_object }, + encryption: { envelope_location: :instruction_file, instruction_file_suffix: '.instruction' }, + client: s3_client + ) + ) + }.to raise_error(Errors::DecryptionError, /unsupported key wrapping algorithm/) + end + end + + context 'KMS encryption' do + let(:kms_client) { KMS::Client.new(stub_responses: true) } + let(:kms_key_id) { 'arn:aws:kms:us-west-2:123456789012:key/12345678-1234-1234-1234-123456789012' } + + it 'if x-amx-t does not exist, then it defaults to {}' do + ##= ../specification/s3-encryption/data-format/content-metadata.md#v3-only + ##= type=test + ##% If the mapkey x-amz-t is not present, the default Material Description value MUST be set to an empty map (`{}`). + + kms_plaintext = OpenSSL::Cipher.new('aes-256-gcm').random_key + kms_ciphertext_blob = 'encrypted-data-key-blob' + + kms_client.stub_responses(:generate_data_key, { + key_id: kms_key_id, + ciphertext_blob: kms_ciphertext_blob, + plaintext: kms_plaintext + }) + + kms_client.stub_responses(:decrypt, { + key_id: kms_key_id, + plaintext: kms_plaintext + }) + + # Create V3 KMS encryption client + client = Client.new( + kms_key_id: kms_key_id, + key_wrap_schema: :kms_context, + kms_client: kms_client, + client: s3_client + ) + + # Capture encrypted data during put + data = {} + s3_client.stub_responses(:put_object, lambda { |context| + data[:metadata] = context.params[:metadata] + data[:body] = context.params[:body].read + {} + }) + + # Encrypt and upload + client.put_object(bucket: test_bucket, key: test_object, body: plaintext) + + # Stub get_object to return the encrypted data, filtering out x-amz-t + filtered_metadata = data[:metadata].reject { |k, _v| k == 'x-amz-t' } + resp_headers = filtered_metadata.map { |k, v| ["x-amz-meta-#{k}", v] }.to_h + resp_headers['content-length'] = data[:body].length.to_s + + s3_client.stub_responses(:get_object, + {status_code: 200, body: data[:body], headers: resp_headers} + ) + + # Decrypt should fail with CEKAlgMismatchError because x-amz-t is missing + # When missing, encryption context defaults to {}, but it needs aws:x-amz-cek-alg + expect { + client.get_object(bucket: test_bucket, key: test_object) + }.to raise_error(Errors::CEKAlgMismatchError) + end + end + end + end +end +end
gems/aws-sdk-s3/spec/encryptionv3/encrypt_handler_spec.rb+74 −0 added@@ -0,0 +1,74 @@ +require_relative '../spec_helper' +require 'base64' +require 'openssl' + +module Aws + module S3 + module EncryptionV3 + describe EncryptHandler do + let(:next_handler) { double(call: nil) } + let(:handler) { EncryptHandler.new(next_handler) } + + let(:cipher) { double('cipher', update: '', final: '', auth_tag: '') } + let(:envelope) { {'x-amz-3' => 'env-key'} } + let(:cipher_provider) { double('cipher_provider', encryption_cipher: [envelope, cipher]) } + let(:context_enc) { { cipher_provider: cipher_provider, envelope_location: :metadata } } + let(:params) { {bucket: 'bucket', key: 'key'} } + let(:client) { double('client') } + let(:http_response) { double(on_headers: nil) } + let(:config) { Struct.new(:user_agent_suffix).new } + let(:context) { double(params: params, client: client, :[] => context_enc, http_response: http_response, config: config ) } + describe '#call' do + + context 'when envelope_location is :metadata' do + it 'sets the envelope on metadata' do + handler.call(context) + expect(params[:metadata]).to include('x-amz-3') + end + end + + context 'when envelope_location is :instruction_file' do + it 'puts the envelope to the bucket' do + context_enc.merge!(envelope_location: :instruction_file, instruction_file_suffix: '.instruction') + expect(client).to receive(:put_object).with( + bucket: 'bucket', key: 'key.instruction', + body: Json.dump(envelope)) + handler.call(context) + end + end + + it 'raises an error if content_md5 is set' do + params[:content_md5] = 'md5' + expect do + handler.call(context) + end.to raise_exception(ArgumentError) + end + + it 'adds unencrypted-content-length to metadata' do + handler.call(context) + expect(params[:metadata]).to include('x-amz-3') + end + + it 'adds a user_agent_suffix' do + handler.call(context) + expect(config.user_agent_suffix).to include('S3CryptoV3') + end + + it 'sets the body to the IOEncrypter and calls close on_headers' do + encrypter = double() + expect(IOEncrypter).to receive(:new).with(cipher, kind_of(StringIO)).and_return(encrypter) + expect(http_response).to receive(:on_headers) { |&block| block.call } + expect(encrypter).to receive(:close) + handler.call(context) + end + + it 'calls the next handler in the stack' do + expect(next_handler).to receive(:call).with(context) + handler.call(context) + end + + end + end + end + end +end
gems/aws-sdk-s3/spec/encryptionv3/instruction_file_spec.rb+611 −0 added@@ -0,0 +1,611 @@ +require_relative '../spec_helper' +require 'base64' +require 'openssl' + +module Aws + module S3 + module EncryptionV3 + describe 'Instruction File' do + let(:plaintext) { 'super secret plain text' } + let(:test_bucket) { 'test-bucket' } + let(:test_object) { 'test-object' } + let(:s3_client) { S3::Client.new(stub_responses: true) } + + # Helper to capture put_object calls + def stub_put_with_instruction_file(s3_client) + data = { object_metadata: nil, object_body: nil, instruction_metadata: nil } + s3_client.stub_responses(:put_object, lambda { |context| + if context.params[:key].end_with?('.instruction') + data[:instruction_metadata] = JSON.parse(context.params[:body]) + else + data[:object_metadata] = context.params[:metadata] + data[:object_body] = context.params[:body].read + end + {} + }) + data + end + + # Helper to stub get_object for instruction file decryption + def stub_get_with_instruction_file(s3_client, data) + resp_headers = Hash[*data[:object_metadata].map { |k, v| ["x-amz-meta-#{k.to_s}", v] }.flatten(1)] + resp_headers['content-length'] = data[:object_body].length + auth_tag = data[:object_body].unpack('C*')[-16, 16].pack('C*') + + s3_client.stub_responses( + :get_object, + {status_code: 200, body: data[:object_body], headers: resp_headers}, + {body: Json.dump(data[:instruction_metadata])}, + {body: auth_tag} + ) + end + + context 'V3 message format with AES key' do + let(:key) { OpenSSL::Cipher.new('aes-256-gcm').random_key } + let(:options) do + { + client: s3_client, + encryption_key: key, + key_wrap_schema: :aes_gcm, + envelope_location: :instruction_file + } + end + + ##= ../specification/s3-encryption/data-format/content-metadata.md#content-metadata-mapkeys + ##= type=test + ##% In the V3 format, the mapkeys "x-amz-c", "x-amz-d", and "x-amz-i" MUST be stored exclusively in the Object Metadata. + + it 'stores x-amz-c in object metadata and not in instruction file' do + ##= ../specification/s3-encryption/data-format/metadata-strategy.md#v3-instruction-files + ##= type=test + ##% - The V3 message format MUST store the mapkey "x-amz-c" and its value in the Object Metadata when writing with an Instruction File. + ##= ../specification/s3-encryption/data-format/metadata-strategy.md#v3-instruction-files + ##= type=test + ##% - The V3 message format MUST NOT store the mapkey "x-amz-c" and its value in the Instruction File. + + client = Client.new(options) + data = stub_put_with_instruction_file(s3_client) + + client.put_object(bucket: test_bucket, key: test_object, body: plaintext) + + # x-amz-c must be in object metadata + expect(data[:object_metadata]).to have_key('x-amz-c') + expect(data[:object_metadata]['x-amz-c']).to eq('115') + + # x-amz-c must NOT be in instruction file + expect(data[:instruction_metadata]).not_to have_key('x-amz-c') + end + + it 'stores x-amz-d in object metadata and not in instruction file' do + ##= ../specification/s3-encryption/data-format/metadata-strategy.md#v3-instruction-files + ##= type=test + ##% - The V3 message format MUST store the mapkey "x-amz-d" and its value in the Object Metadata when writing with an Instruction File. + ##= ../specification/s3-encryption/data-format/metadata-strategy.md#v3-instruction-files + ##= type=test + ##% - The V3 message format MUST NOT store the mapkey "x-amz-d" and its value in the Instruction File. + + client = Client.new(options) + data = stub_put_with_instruction_file(s3_client) + + client.put_object(bucket: test_bucket, key: test_object, body: plaintext) + + # x-amz-d must be in object metadata + expect(data[:object_metadata]).to have_key('x-amz-d') + expect(data[:object_metadata]['x-amz-d']).not_to be_empty + + # x-amz-d must NOT be in instruction file + expect(data[:instruction_metadata]).not_to have_key('x-amz-d') + end + + it 'stores x-amz-i in object metadata and not in instruction file' do + ##= ../specification/s3-encryption/data-format/metadata-strategy.md#v3-instruction-files + ##= type=test + ##% - The V3 message format MUST store the mapkey "x-amz-i" and its value in the Object Metadata when writing with an Instruction File. + ##= ../specification/s3-encryption/data-format/metadata-strategy.md#v3-instruction-files + ##= type=test + ##% - The V3 message format MUST NOT store the mapkey "x-amz-i" and its value in the Instruction File. + + client = Client.new(options) + data = stub_put_with_instruction_file(s3_client) + + client.put_object(bucket: test_bucket, key: test_object, body: plaintext) + + # x-amz-i must be in object metadata + expect(data[:object_metadata]).to have_key('x-amz-i') + expect(data[:object_metadata]['x-amz-i']).not_to be_empty + + # x-amz-i must NOT be in instruction file + expect(data[:instruction_metadata]).not_to have_key('x-amz-i') + end + + it 'stores x-amz-3 in instruction file and not in object metadata' do + ##= ../specification/s3-encryption/data-format/metadata-strategy.md#v3-instruction-files + ##= type=test + ##% - The V3 message format MUST store the mapkey "x-amz-3" and its value in the Instruction File. + + client = Client.new(options) + data = stub_put_with_instruction_file(s3_client) + + client.put_object(bucket: test_bucket, key: test_object, body: plaintext) + + # x-amz-3 must be in instruction file + expect(data[:instruction_metadata]).to have_key('x-amz-3') + expect(data[:instruction_metadata]['x-amz-3']).not_to be_empty + + # x-amz-3 must NOT be in object metadata + expect(data[:object_metadata]).not_to have_key('x-amz-3') + end + + it 'stores x-amz-w in instruction file and not in object metadata' do + ##= ../specification/s3-encryption/data-format/metadata-strategy.md#v3-instruction-files + ##= type=test + ##% - The V3 message format MUST store the mapkey "x-amz-w" and its value in the Instruction File. + + client = Client.new(options) + data = stub_put_with_instruction_file(s3_client) + + client.put_object(bucket: test_bucket, key: test_object, body: plaintext) + + # x-amz-w must be in instruction file + expect(data[:instruction_metadata]).to have_key('x-amz-w') + expect(data[:instruction_metadata]['x-amz-w']).to eq('02') + + # x-amz-w must NOT be in object metadata + expect(data[:object_metadata]).not_to have_key('x-amz-w') + end + + it 'writes wrapping algorithm value 02 for AES/GCM in instruction file' do + ##= ../specification/s3-encryption/data-format/content-metadata.md#v3-only + ##= type=test + ##% - The wrapping algorithm value "02" MUST be translated to AES/GCM upon retrieval, and vice versa on write. + + client = Client.new(options) + data = stub_put_with_instruction_file(s3_client) + + client.put_object(bucket: test_bucket, key: test_object, body: plaintext) + + expect(data[:instruction_metadata]['x-amz-w']).to eq('02') + end + + it 'stores x-amz-m in instruction file when materials description is provided' do + ##= ../specification/s3-encryption/data-format/metadata-strategy.md#v3-instruction-files + ##= type=test + ##% - The V3 message format MUST store the mapkey "x-amz-m" and its value (when present in the content metadata) in the Instruction File. + + materials_desc = '{"description":"test-materials"}' + client = Client.new(options.merge(materials_description: materials_desc)) + data = stub_put_with_instruction_file(s3_client) + + client.put_object(bucket: test_bucket, key: test_object, body: plaintext) + + # x-amz-m must be in instruction file when present + expect(data[:instruction_metadata]).to have_key('x-amz-m') + expect(data[:instruction_metadata]['x-amz-m']).to eq(materials_desc) + + # x-amz-m must NOT be in object metadata + expect(data[:object_metadata]).not_to have_key('x-amz-m') + end + + it 'uses Material Description for AES/GCM wrapping in instruction file' do + ##= ../specification/s3-encryption/data-format/content-metadata.md#v3-only + ##= type=test + ##% The Material Description MUST be used for wrapping algorithms `AES/GCM` (`02`) and `RSA-OAEP-SHA1` (`22`). + + materials_desc = '{"description":"test-materials"}' + client = Client.new(options.merge(materials_description: materials_desc)) + data = stub_put_with_instruction_file(s3_client) + + client.put_object(bucket: test_bucket, key: test_object, body: plaintext) + + # For AES/GCM (02), material description should be present + expect(data[:instruction_metadata]).to have_key('x-amz-m') + expect(data[:instruction_metadata]['x-amz-w']).to eq('02') + expect(data[:instruction_metadata]['x-amz-m']).to eq(materials_desc) + end + + it 'can decrypt objects encrypted with instruction files' do + client = Client.new(options) + data = stub_put_with_instruction_file(s3_client) + + client.put_object(bucket: test_bucket, key: test_object, body: plaintext) + + stub_get_with_instruction_file(s3_client, data) + decrypted = client.get_object(bucket: test_bucket, key: test_object).body.read + expect(decrypted).to eq(plaintext) + end + end + + context 'V3 message format with RSA key' do + let(:key) { OpenSSL::PKey::RSA.new(1024) } + let(:options) do + { + client: s3_client, + encryption_key: key, + key_wrap_schema: :rsa_oaep_sha1, + envelope_location: :instruction_file + } + end + + it 'stores required fields correctly with RSA key' do + client = Client.new(options) + data = stub_put_with_instruction_file(s3_client) + + client.put_object(bucket: test_bucket, key: test_object, body: plaintext) + + # Object metadata must contain x-amz-c, x-amz-d, x-amz-i + expect(data[:object_metadata]).to have_key('x-amz-c') + expect(data[:object_metadata]).to have_key('x-amz-d') + expect(data[:object_metadata]).to have_key('x-amz-i') + + # Instruction file must contain x-amz-3, x-amz-w + expect(data[:instruction_metadata]).to have_key('x-amz-3') + expect(data[:instruction_metadata]).to have_key('x-amz-w') + expect(data[:instruction_metadata]['x-amz-w']).to eq('22') + + # Instruction file must NOT contain x-amz-c, x-amz-d, x-amz-i + expect(data[:instruction_metadata]).not_to have_key('x-amz-c') + expect(data[:instruction_metadata]).not_to have_key('x-amz-d') + expect(data[:instruction_metadata]).not_to have_key('x-amz-i') + + # Object metadata must NOT contain x-amz-3, x-amz-w + expect(data[:object_metadata]).not_to have_key('x-amz-3') + expect(data[:object_metadata]).not_to have_key('x-amz-w') + end + + it 'writes wrapping algorithm value 22 for RSA-OAEP-SHA1 in instruction file' do + ##= ../specification/s3-encryption/data-format/content-metadata.md#v3-only + ##= type=test + ##% - The wrapping algorithm value "22" MUST be translated to RSA-OAEP-SHA1 upon retrieval, and vice versa on write. + + client = Client.new(options) + data = stub_put_with_instruction_file(s3_client) + + client.put_object(bucket: test_bucket, key: test_object, body: plaintext) + + expect(data[:instruction_metadata]['x-amz-w']).to eq('22') + end + + it 'uses Material Description for RSA-OAEP-SHA1 wrapping in instruction file' do + ##= ../specification/s3-encryption/data-format/content-metadata.md#v3-only + ##= type=test + ##% The Material Description MUST be used for wrapping algorithms `AES/GCM` (`02`) and `RSA-OAEP-SHA1` (`22`). + + materials_desc = '{"description":"rsa-test"}' + client = Client.new(options.merge(materials_description: materials_desc)) + data = stub_put_with_instruction_file(s3_client) + + client.put_object(bucket: test_bucket, key: test_object, body: plaintext) + + # For RSA-OAEP-SHA1 (22), material description should be present + expect(data[:instruction_metadata]).to have_key('x-amz-m') + expect(data[:instruction_metadata]['x-amz-m']).to eq(materials_desc) + expect(data[:instruction_metadata]['x-amz-w']).to eq('22') + end + + it 'can decrypt RSA encrypted objects with instruction files' do + client = Client.new(options) + data = stub_put_with_instruction_file(s3_client) + + client.put_object(bucket: test_bucket, key: test_object, body: plaintext) + + stub_get_with_instruction_file(s3_client, data) + decrypted = client.get_object(bucket: test_bucket, key: test_object).body.read + expect(decrypted).to eq(plaintext) + end + end + + context 'V3 message format with KMS key' do + let(:kms_client) { KMS::Client.new(stub_responses: true) } + let(:kms_key_id) { 'arn:aws:kms:us-west-2:123456789012:key/12345678-1234-1234-1234-123456789012' } + let(:kms_ciphertext_blob) do + Base64.decode64("AQIDAHiWj6qDEnwihp7W7g6VZb1xqsat5jdSUdEaGhgZepHdLAGASCQI7LZz\nz7GzCpm6y4sHAAAAfjB8BgkqhkiG9w0BBwagbzBtAgEAMGgGCSqGSIb3DQEH\nATAeBglghkgBZQMEAS4wEQQMJMJe6d8DkRTWwlvtAgEQgDtBCwiibCTS8pb7\n6BYKklVjy+CmO9q3r6y4u/9jJ8lk9eg5GwiskmcBtPMcWogMzx/vh+/65Cjb\nsQBpLQ==\n") + end + let(:kms_plaintext) do + Base64.decode64("5V7JWe+UDRhv66TaDg+tP6JONf/GkTdXk6Jq61weM+w=\n") + end + let(:options) do + { + client: s3_client, + kms_key_id: kms_key_id, + key_wrap_schema: :kms_context, + kms_client: kms_client, + envelope_location: :instruction_file + } + end + + it 'stores required fields correctly with KMS key' do + kms_client.stub_responses( + :generate_data_key, + { + key_id: kms_key_id, + ciphertext_blob: kms_ciphertext_blob, + plaintext: kms_plaintext + } + ) + + client = Client.new(options) + data = stub_put_with_instruction_file(s3_client) + + client.put_object(bucket: test_bucket, key: test_object, body: plaintext) + + # Object metadata must contain x-amz-c, x-amz-d, x-amz-i + expect(data[:object_metadata]).to have_key('x-amz-c') + expect(data[:object_metadata]).to have_key('x-amz-d') + expect(data[:object_metadata]).to have_key('x-amz-i') + + # Instruction file must contain x-amz-3, x-amz-w + expect(data[:instruction_metadata]).to have_key('x-amz-3') + expect(data[:instruction_metadata]).to have_key('x-amz-w') + expect(data[:instruction_metadata]['x-amz-w']).to eq('12') + + # Instruction file must NOT contain x-amz-c, x-amz-d, x-amz-i + expect(data[:instruction_metadata]).not_to have_key('x-amz-c') + expect(data[:instruction_metadata]).not_to have_key('x-amz-d') + expect(data[:instruction_metadata]).not_to have_key('x-amz-i') + + # Object metadata must NOT contain x-amz-3, x-amz-w + expect(data[:object_metadata]).not_to have_key('x-amz-3') + expect(data[:object_metadata]).not_to have_key('x-amz-w') + end + + it 'writes wrapping algorithm value 12 for kms+context in instruction file' do + ##= ../specification/s3-encryption/data-format/content-metadata.md#v3-only + ##= type=test + ##% - The wrapping algorithm value "12" MUST be translated to kms+context upon retrieval, and vice versa on write. + + kms_client.stub_responses( + :generate_data_key, + { + key_id: kms_key_id, + ciphertext_blob: kms_ciphertext_blob, + plaintext: kms_plaintext + } + ) + + client = Client.new(options) + data = stub_put_with_instruction_file(s3_client) + + client.put_object(bucket: test_bucket, key: test_object, body: plaintext) + + expect(data[:instruction_metadata]['x-amz-w']).to eq('12') + end + + it 'stores x-amz-t in instruction file when KMS encryption context is provided' do + ##= ../specification/s3-encryption/data-format/metadata-strategy.md#v3-instruction-files + ##= type=test + ##% - The V3 message format MUST store the mapkey "x-amz-t" and its value (when present in the content metadata) in the Instruction File. + + kms_client.stub_responses( + :generate_data_key, + { + key_id: kms_key_id, + ciphertext_blob: kms_ciphertext_blob, + plaintext: kms_plaintext + } + ) + + enc_context = { 'department' => 'finance', 'project' => 'alpha' } + client = Client.new(options) + data = stub_put_with_instruction_file(s3_client) + + client.put_object( + bucket: test_bucket, + key: test_object, + body: plaintext, + kms_encryption_context: enc_context + ) + + # x-amz-t must be in instruction file when present + expect(data[:instruction_metadata]).to have_key('x-amz-t') + stored_context = JSON.parse(data[:instruction_metadata]['x-amz-t']) + expect(stored_context).to include('department' => 'finance') + expect(stored_context).to include('project' => 'alpha') + + # x-amz-t must NOT be in object metadata + expect(data[:object_metadata]).not_to have_key('x-amz-t') + end + + it 'uses Encryption Context for kms+context wrapping in instruction file' do + ##= ../specification/s3-encryption/data-format/content-metadata.md#v3-only + ##= type=test + ##% The Encryption Context value MUST be used for wrapping algorithm `kms+context` or `12`. + + kms_client.stub_responses( + :generate_data_key, + { + key_id: kms_key_id, + ciphertext_blob: kms_ciphertext_blob, + plaintext: kms_plaintext + } + ) + + enc_context = { 'department' => 'finance', 'project' => 'alpha' } + client = Client.new(options) + data = stub_put_with_instruction_file(s3_client) + + client.put_object( + bucket: test_bucket, + key: test_object, + body: plaintext, + kms_encryption_context: enc_context + ) + + # For kms+context (12), encryption context should be present + expect(data[:instruction_metadata]).to have_key('x-amz-t') + expect(data[:instruction_metadata]['x-amz-w']).to eq('12') + end + + it 'can decrypt KMS encrypted objects with instruction files' do + kms_client.stub_responses( + :generate_data_key, + { + key_id: kms_key_id, + ciphertext_blob: kms_ciphertext_blob, + plaintext: kms_plaintext + } + ) + kms_client.stub_responses( + :decrypt, + { + key_id: kms_key_id, + plaintext: kms_plaintext, + encryption_algorithm: "SYMMETRIC_DEFAULT" + } + ) + + client = Client.new(options) + data = stub_put_with_instruction_file(s3_client) + + client.put_object(bucket: test_bucket, key: test_object, body: plaintext) + + stub_get_with_instruction_file(s3_client, data) + decrypted = client.get_object(bucket: test_bucket, key: test_object).body.read + expect(decrypted).to eq(plaintext) + end + end + + context 'V1/V2 message format with instruction files' do + let(:key) { OpenSSL::Cipher.new('aes-256-cbc').random_key } + + it 'stores all metadata in instruction file for V1 format' do + ##= ../specification/s3-encryption/data-format/metadata-strategy.md#v1-v2-instruction-files + ##= type=test + ##% In the V1/V2 message format, all of the content metadata MUST be stored in the Instruction File. + + # Create a V1 client with instruction file + client_v1 = Aws::S3::Encryption::Client.new( + encryption_key: key, + client: s3_client, + envelope_location: :instruction_file + ) + + data = { object_metadata: nil, object_body: nil, instruction_metadata: nil } + s3_client.stub_responses(:put_object, lambda { |context| + if context.params[:key].end_with?('.instruction') + data[:instruction_metadata] = JSON.parse(context.params[:body]) + else + data[:object_metadata] = context.params[:metadata] + data[:object_body] = context.params[:body].read + end + {} + }) + + client_v1.put_object(bucket: test_bucket, key: test_object, body: plaintext) + + # All V1/V2 metadata must be in instruction file + expect(data[:instruction_metadata]).to have_key('x-amz-key') + expect(data[:instruction_metadata]).to have_key('x-amz-iv') + expect(data[:instruction_metadata]).to have_key('x-amz-matdesc') + + # Object metadata should be empty or minimal (no encryption metadata) + if data[:object_metadata] + expect(data[:object_metadata]).not_to have_key('x-amz-key') + expect(data[:object_metadata]).not_to have_key('x-amz-iv') + expect(data[:object_metadata]).not_to have_key('x-amz-matdesc') + end + end + + it 'stores all metadata in instruction file for V2 format with AES-GCM' do + ##= ../specification/s3-encryption/data-format/metadata-strategy.md#v1-v2-instruction-files + ##= type=test + ##% In the V1/V2 message format, all of the content metadata MUST be stored in the Instruction File. + + # Create a V3 client in legacy mode to produce V2 format + client_v2 = Client.new( + client: s3_client, + encryption_key: OpenSSL::Cipher.new('aes-256-gcm').random_key, + key_wrap_schema: :aes_gcm, + commitment_policy: :forbid_encrypt_allow_decrypt, + content_encryption_schema: :aes_gcm_no_padding, + envelope_location: :instruction_file + ) + + data = { object_metadata: nil, object_body: nil, instruction_metadata: nil } + s3_client.stub_responses(:put_object, lambda { |context| + if context.params[:key].end_with?('.instruction') + data[:instruction_metadata] = JSON.parse(context.params[:body]) + else + data[:object_metadata] = context.params[:metadata] + data[:object_body] = context.params[:body].read + end + {} + }) + + client_v2.put_object(bucket: test_bucket, key: test_object, body: plaintext) + + # All V2 metadata must be in instruction file + expect(data[:instruction_metadata]).to have_key('x-amz-key-v2') + expect(data[:instruction_metadata]).to have_key('x-amz-iv') + expect(data[:instruction_metadata]).to have_key('x-amz-matdesc') + expect(data[:instruction_metadata]).to have_key('x-amz-wrap-alg') + expect(data[:instruction_metadata]).to have_key('x-amz-cek-alg') + expect(data[:instruction_metadata]).to have_key('x-amz-tag-len') + + # Object metadata should be empty or minimal (no encryption metadata) + if data[:object_metadata] + expect(data[:object_metadata]).not_to have_key('x-amz-key-v2') + expect(data[:object_metadata]).not_to have_key('x-amz-iv') + expect(data[:object_metadata]).not_to have_key('x-amz-matdesc') + expect(data[:object_metadata]).not_to have_key('x-amz-wrap-alg') + expect(data[:object_metadata]).not_to have_key('x-amz-cek-alg') + expect(data[:object_metadata]).not_to have_key('x-amz-tag-len') + end + end + end + + context 'Custom instruction file suffix' do + let(:key) { OpenSSL::Cipher.new('aes-256-gcm').random_key } + let(:custom_suffix) { '.custom-instruction' } + let(:options) do + { + client: s3_client, + encryption_key: key, + key_wrap_schema: :aes_gcm, + envelope_location: :instruction_file, + instruction_file_suffix: custom_suffix + } + end + + it 'uses custom suffix for instruction file' do + client = Client.new(options) + + instruction_file_key = nil + s3_client.stub_responses(:put_object, lambda { |context| + if context.params[:key].end_with?(custom_suffix) + instruction_file_key = context.params[:key] + end + {} + }) + + client.put_object(bucket: test_bucket, key: test_object, body: plaintext) + + expect(instruction_file_key).to eq("#{test_object}#{custom_suffix}") + end + + it 'still places metadata correctly with custom suffix' do + client = Client.new(options) + data = { object_metadata: nil, instruction_metadata: nil } + + s3_client.stub_responses(:put_object, lambda { |context| + if context.params[:key].end_with?(custom_suffix) + data[:instruction_metadata] = JSON.parse(context.params[:body]) + else + data[:object_metadata] = context.params[:metadata] + end + {} + }) + + client.put_object(bucket: test_bucket, key: test_object, body: plaintext) + + # Verify metadata placement is still correct + expect(data[:object_metadata]).to have_key('x-amz-c') + expect(data[:object_metadata]).to have_key('x-amz-d') + expect(data[:object_metadata]).to have_key('x-amz-i') + expect(data[:instruction_metadata]).to have_key('x-amz-3') + expect(data[:instruction_metadata]).to have_key('x-amz-w') + end + end + end + end + end +end
gems/aws-sdk-s3/spec/encryptionv3/io_encrypter_spec.rb+74 −0 added@@ -0,0 +1,74 @@ +require_relative '../spec_helper' +require 'openssl' + +module Aws + module S3 + module EncryptionV3 + describe IOEncrypter do + let(:key) do + Base64.decode64('kM5UVbhE/4rtMZJfsadYEdm2vaKFsmV2f5+URSeUCV4=') + end + + let(:iv) { Base64.decode64("IqR6MLs8IskchoMh\n") } + + let(:plain_text) { 'The quick brown fox jumps over the lazy dog.' } + + let(:cipher_text) do + Base64.decode64( + "GMI8Qdfj7RfkzTKtOhy+hX2weKtMbU+e1/ZMUQgTi+3I7jOAmO+yxt17fgHm\ntlVCo4J45qiD0gWdkUqU\n" + ) + end + + let(:cipher) do + cipher = OpenSSL::Cipher.new('aes-256-gcm') + cipher.encrypt + cipher.key = key + cipher.iv = iv + cipher + end + + it 'adds the auth_tag' do + expect(cipher).to receive(:auth_tag).and_return('auth_tag') + IOEncrypter.new(cipher, StringIO.new(plain_text)) + end + + it 'encrypts an IO object' do + io = IOEncrypter.new(cipher, StringIO.new(plain_text)) + expect(io.read).to eq(cipher_text) + end + + it 'supports partial reads' do + io = IOEncrypter.new(cipher, StringIO.new(plain_text)) + parts = [] + while part = io.read(3) + parts << part + end + expect(parts.join).to eq(cipher_text) + end + + it 'caches the cipher-text of large objects to disk' do + tempfile = double('tempfile').as_null_object + expect(tempfile).to receive(:write) + expect(tempfile).to receive(:binmode) + allow(Tempfile).to receive(:new).and_return(tempfile) + + large_io = double( + 'large-io-object', + size: 10 * 1024 * 1024, read: nil + ) + allow(large_io).to receive(:read).and_return('data').and_return(nil) + + IOEncrypter.new(cipher, large_io) + end + + it 'supports re-reading from the cache file to enable retries' do + data = '.' * (2 * 1024 * 1024) # 2MB file + io = IOEncrypter.new(cipher, StringIO.new(data)) + cipher_text = io.read + io.close + expect(io.read).to eq(cipher_text) # automatically re-opens the file + end + end + end + end +end
gems/aws-sdk-s3/spec/encryptionv3/kdf_kat_spec.rb+65 −0 added@@ -0,0 +1,65 @@ +require_relative '../spec_helper' +require 'base64' +require 'openssl' +require 'json' + +module Aws + module S3 + module EncryptionV3 + + # The known answer tests are a shared set of test cases + # to ensure interop between SDKs + context 'HKDF Known Answer Tests' do + def self.from_h(s) + [s].pack('H*') + end + + # all the KAT values are hex + def self.build_kat(raw_kat) + Struct.new(:comment, :data_key, :message_id, :encryption_key, :commitment_key, + :plaintext, :ciphertext, :auth_tag).new( + raw_kat['comment'], + from_h(raw_kat['data_key']), + from_h(raw_kat['message_id']), + from_h(raw_kat['encryption_key']), + from_h(raw_kat['commitment_key']), + raw_kat['plaintext'] ? from_h(raw_kat['plaintext']) : nil, + raw_kat['ciphertext'] ? from_h(raw_kat['ciphertext']) : nil, + raw_kat['auth_tag'] ? from_h(raw_kat['auth_tag']) : nil + ) + end + + fixture_path = File.expand_path('../fixtures/encryption', __dir__) + kats = JSON.load_file(File.new(File.join(fixture_path, 'kdf_kat.json'))) + + it "kat tests to verify" do + expect(kats).not_to be_empty + end + + kats.each_with_index do |raw, i| + kat = build_kat(raw) + + it "passes KDF test case #{i+1}: #{kat.comment}" do + encryption_key = Utils::derive_encryption_key(kat.data_key, kat.message_id) + commitment_key = Utils::derive_commitment_key(kat.data_key, kat.message_id) + + expect(encryption_key).to eq(kat.encryption_key) + expect(commitment_key).to eq(kat.commitment_key) + + # Only test encryption/decryption if plaintext is present + if kat.plaintext + # Test decryption recovers original plaintext + decipher = Utils.derive_alg_aes_256_gcm_hkdf_sha512_commit_key_cipher( + kat.data_key, kat.message_id, kat.commitment_key + ) + decipher.auth_tag = kat.auth_tag + decrypted = decipher.update(kat.ciphertext) + decipher.final + + expect(decrypted).to eq(kat.plaintext) + end + end + end + end + end + end +end
gems/aws-sdk-s3/spec/fixtures/encryption/generate_kdf_kat.rb+105 −0 added@@ -0,0 +1,105 @@ +#!/usr/bin/env ruby +# frozen_string_literal: true + +require_relative '../../../lib/aws-sdk-s3/encryptionV3/utils' +require 'json' + +module Aws + module S3 + module EncryptionV3 + # Generator for KDF Known Answer Test (KAT) JSON entries + # This script generates test vectors for encryption interoperability testing + class KdfKatGenerator + + # Generates a complete KAT entry with encryption test data + # @param plaintext [String] The plaintext to encrypt + # @param comment [String, nil] Optional comment for the test case + # @param data_key [String, nil] Optional 32-byte data key (generates random if nil) + # @param message_id [String, nil] Optional 28-byte message ID (generates random if nil) + # @return [Hash] A hash representing the KAT JSON entry + def self.generate_kat_entry(plaintext:, comment: nil, data_key: nil, message_id: nil) + # Use provided keys or generate new ones + data_key ||= Utils.generate_data_key() + message_id ||= Utils.generate_message_id() + + # Derive keys using HKDF + encryption_key = Utils.derive_encryption_key(data_key, message_id) + commitment_key = Utils.derive_commitment_key(data_key, message_id) + + # Create cipher with specific message_id and encrypt plaintext + cipher = Utils.alg_aes_256_gcm_hkdf_sha512_commit_key_cipher(:encrypt, data_key, message_id) + ciphertext = cipher.update(plaintext) + cipher.final + auth_tag = cipher.auth_tag + + # Return hex-encoded JSON entry + { + comment: comment || "Generated KAT entry", + data_key: to_hex(data_key), + message_id: to_hex(message_id), + encryption_key: to_hex(encryption_key), + commitment_key: to_hex(commitment_key), + plaintext: to_hex(plaintext), + ciphertext: to_hex(ciphertext), + auth_tag: to_hex(auth_tag) + } + end + + # Convert binary string to hex string + # @param data [String] Binary data + # @return [String] Hex-encoded string + def self.to_hex(data) + data.unpack1('H*') + end + + # Convert hex string to binary string + # @param hex [String] Hex-encoded string + # @return [String] Binary data + def self.from_hex(hex) + [hex].pack('H*') + end + + # Generate multiple KAT entries + # @param count [Integer] Number of entries to generate + # @param plaintext [String] The plaintext to use for all entries + # @return [Array<Hash>] Array of KAT entries + def self.generate_multiple(count:, plaintext: "Hello, World!") + count.times.map do |i| + generate_kat_entry( + plaintext: plaintext, + comment: "Generated KAT entry #{i + 1}" + ) + end + end + + # Print a KAT entry as formatted JSON + # @param entry [Hash] The KAT entry to print + def self.print_json(entry) + puts JSON.pretty_generate(entry) + end + + # Print multiple KAT entries as a JSON array + # @param entries [Array<Hash>] The KAT entries to print + def self.print_json_array(entries) + puts JSON.pretty_generate(entries) + end + end + end + end +end + +# Example usage when run as a script +if __FILE__ == $PROGRAM_NAME + include Aws::S3::EncryptionV3 + + # Generate a single entry with default plaintext + entry = KdfKatGenerator.generate_kat_entry( + plaintext: "Hello, World!", + comment: "Example KAT entry" + ) + + KdfKatGenerator.print_json(entry) + + puts "\n# To generate multiple entries:" + puts "# entries = KdfKatGenerator.generate_multiple(count: 2, plaintext: 'Test data')" + puts "# KdfKatGenerator.print_json_array(entries)" +end
gems/aws-sdk-s3/spec/fixtures/encryption/kdf_kat.json+26 −0 added@@ -0,0 +1,26 @@ +[ + { + "comment": "Basic S3EC.ALG_AES_256_GCM_HKDF_SHA512_COMMIT_KEY", + "data_key": "80d90dc4cc7e77d8a6332efa44eba56230a7fe7b89af37d1e501ab2e07c0a163", + "message_id": "b8ea76bed24c7b85382a148cb9dcd1cfdfb765f55ded4dfa6e0c4c79", + "encryption_key": "6dd14f546cc006e639126e83f5d4d1b118576bb5df97f38c6fb3a1db87bbc338", + "commitment_key": "f89818bc0a346d3a3426b68e9509b6b2ae5fe1f904aa329fb73625db" + }, + { + "comment": "Basic S3EC.ALG_AES_256_GCM_HKDF_SHA512_COMMIT_KEY", + "data_key": "501afb8227d22e75e68010414b8abdaf3064c081e8e922dafef4992036394d60", + "message_id": "61a00b4981a5aacfd136c55cb726e32d2a547dc7600a7d4675c69127", + "encryption_key": "e14786a714748d1d2c3a4a6816dec56ddf1881bbeabb4f39420ffb9f63700b2f", + "commitment_key": "5c1e73e47f6fe3a70d6d094283aceaa76d2975feb829212d88f0afc1" + }, + { + "comment": "Example KAT entry", + "data_key": "09b579890de8618fa436ced66b655af4aa6ec760fabf3eb9e7cc7fd22e79851e", + "message_id": "9faa68cd791762eb7280bab11ca1d4011de7e3a255a628d9fbe54f0a", + "encryption_key": "bd99a917127150fae9e6f87193b122c52ae72fbcdc30e8d18cdf81fabf8b2df3", + "commitment_key": "ebbeb009a83d2dad7ce3d8df0b7814d1dfe7e9e89703f768ab6ffe3f", + "plaintext": "48656c6c6f2c20576f726c6421", + "ciphertext": "ac9f0d1185cbc1d2b7e77788b4", + "auth_tag": "abe58c0c421fa648faca6de694057e9a" + } +]
gems/aws-sdk-s3/spec/fixtures/encryption/README_KAT_GENERATOR.md+86 −0 added@@ -0,0 +1,86 @@ +# KDF KAT Generator + +This directory contains tools for generating Known Answer Test (KAT) JSON entries for the HKDF encryption implementation. + +## Files + +- `generate_kdf_kat.rb` - Generator script for creating KAT entries +- `kdf_kat_spec.rb` - Test spec that validates KAT entries +- `kdf_kat.json` - KAT test vectors + +## Usage + +### Generate a Single Entry + +```ruby +ruby fixtures/encryption/generate_kdf_kat.rb +``` + +This generates a new KAT entry with random keys and "Hello, World!" as plaintext. + +### Generate Multiple Entries + +```ruby +ruby -r './fixtures/encryption/generate_kdf_kat.rb' -e " +include Aws::S3::EncryptionV3 +entries = KdfKatGenerator.generate_multiple(count: 2, plaintext: 'Test data') +KdfKatGenerator.print_json_array(entries) +" +``` + +### Generate Entry with Specific Keys + +```ruby +ruby -r './fixtures/encryption/generate_kdf_kat.rb' -e " +include Aws::S3::EncryptionV3 +data_key = KdfKatGenerator.from_hex('80d90dc4cc7e77d8a6332efa44eba56230a7fe7b89af37d1e501ab2e07c0a163') +message_id = KdfKatGenerator.from_hex('b8ea76bed24c7b85382a148cb9dcd1cfdfb765f55ded4dfa6e0c4c79') +entry = KdfKatGenerator.generate_kat_entry( + plaintext: 'Your plaintext here', + comment: 'Your comment here', + data_key: data_key, + message_id: message_id +) +KdfKatGenerator.print_json(entry) +" +``` + +## KAT JSON Format + +Each KAT entry contains: + +### Required Fields (for key derivation tests) + +- `comment` - Description of the test case +- `data_key` - 32-byte data key (hex encoded) +- `message_id` - 28-byte message ID (hex encoded) +- `encryption_key` - 32-byte derived encryption key (hex encoded) +- `commitment_key` - 28-byte derived commitment key (hex encoded) + +### Optional Fields (for encryption/decryption tests) + +- `plaintext` - Plaintext data (hex encoded) +- `ciphertext` - Encrypted data (hex encoded) +- `auth_tag` - 16-byte GCM authentication tag (hex encoded) + +## Test Behavior + +The test spec `kdf_kat_spec.rb` has backward compatibility: + +1. **All entries** are tested for correct key derivation (encryption_key and commitment_key) +2. **Only entries with plaintext** are additionally tested for: + - Encryption: Verify that encrypting the plaintext produces the expected ciphertext and auth_tag + - Decryption: Verify that decrypting the ciphertext recovers the original plaintext + +This allows old KAT entries (without encryption fields) to continue passing while new entries can include full encryption tests. + +## Algorithm Details + +The generator uses: + +- **Algorithm Suite**: `ALG_AES_256_GCM_HKDF_SHA512_COMMIT_KEY` +- **Cipher**: AES-256-GCM +- **Key Derivation**: HKDF with SHA-512 +- **IV**: All zeros (12 bytes) - as specified for this algorithm suite +- **Auth Data**: Algorithm suite ID (0x0073) +
Vulnerability mechanics
Generated by null/stub on May 9, 2026. Inputs: CWE entries + fix-commit diffs from this CVE's patches. Citations validated against bundle.
References
8- github.com/advisories/GHSA-2xgq-q749-89fqghsaADVISORY
- nvd.nist.gov/vuln/detail/CVE-2025-14762ghsaADVISORY
- aws.amazon.com/security/security-bulletins/AWS-2025-032ghsaWEB
- github.com/aws/aws-sdk-ruby/commit/b633ba10cd2fbc4cc770b76ab531ed9647654044ghsaWEB
- github.com/aws/aws-sdk-ruby/security/advisories/GHSA-2xgq-q749-89fqnvdWEB
- github.com/rubysec/ruby-advisory-db/blob/master/gems/aws-sdk-s3/CVE-2025-14762.ymlghsaWEB
- rubygems.org/gems/aws-sdk-s3/versions/1.208.0nvdWEB
- aws.amazon.com/security/security-bulletins/AWS-2025-032/nvd
News mentions
0No linked articles in our index yet.