K i]JddlZddlmZddlZ ddlmZddlmcm Z ddl m Z m Z dZdZdZdZdZej(j*ej(j gZej.d d Zej.d d Zd ZdZdZdZdZdZdZ dZ!dZ"dZ#dZ$dZ%dZ&dZ'dZ(dZ)dZ*dZ+dZ,ej(j[d !d"Z.ej(j[d#!d$Z/d%Z0d&Z1d'Z2y#e$rdZdZ YwxYw)(N) timedelta)InMemoryKmsClientverify_file_encryptedzencrypted_table.in_mem.parquets0123456789112345 footer_keys1234567890123450col_keymodule)scopectjjtjgdtjgdtjgdd}|S)N))abc)xyz)paTable from_pydictarray) data_tables k/mnt/ssd/data/python-lab/Trading/venv/lib/python3.12/site-packages/pyarrow/tests/parquet/test_encryption.pyrr0sG%% XXi XXo & XXo &'J cLtjttddgi}|S)Nrr)r column_keys)peEncryptionConfigurationFOOTER_KEY_NAME COL_KEY_NAME)basic_encryption_configs rr!r!:s, 88" 3*   #"rcftj|}d}tj|}||fS)z Sets up and returns the KMS connection configuration and crypto factory based on provided KMS configuration parameters. custom_kms_confct|SNrkms_connection_configurations r kms_factoryz1setup_encryption_environment..kms_factoryK !=>>r)rKmsConnectionConfig CryptoFactory)r$kms_connection_configr*crypto_factorys rsetup_encryption_environmentr0Ds7 22?S?%%k2N . 00rc||jd||jdi}t|\}} t||||| || fS)zL Writes an encrypted parquet file based on the provided parameters. UTF-8)decoder0write_encrypted_parquet) pathrfooter_key_name col_key_namerrencryption_configr$r.r/s rwrite_encrypted_filer9Tsc **73gnnW-O -I-)>D*.?1>C !. 00rc v|tz }tjttddgidt dd}|j dusJt||tttt|\}}t|tjt d }t||||}|j|sJy ) DWrite an encrypted parquet, verify it's encrypted, and then read it.rr AES_GCM_V1@minutesrrencryption_algorithmcache_lifetimedata_key_length_bitsFrCN) PARQUET_NAMErrrr runiform_encryptionr9 FOOTER_KEYCOL_KEYrDecryptionConfigurationread_encrypted_parquetequalstempdirrr5r8r.r/decryption_config result_tables r!test_encrypted_parquet_write_readrQjs \ !D 22" 3* * - "  / /5 88 8,@ j/<W-)>$22 -/) !6HL   \ ** *rc ^|tz }tjtddt dd}|j dusJt ||tttd|\}}t|tjt d}t||||}|j|sJy ) r;Tr<r=r>r@)rrGrBrCrDrrEN) rFrrrrrGr9r rHrrJrKrLrMs r)test_uniform_encrypted_parquet_write_readrSs \ !D22") - "  / /4 77 7,@ j/<S-)>$22 -/) !6HL   \ ** *rc|j||}|Jtj||j|5}|j |dddy#1swYyxYw)N)encryption_properties)file_encryption_propertiespq ParquetWriterschema write_table)r5tabler8r.r/rVwriters rr4r4se!/!J!J0"2 % 11 1   %,,"< >"AG5!"""s AAc2|j||}|Jtj||}|jdk(sJtj||}t |j dk(sJtj||}|jdS)Ndecryption_propertiesr T use_threads) file_decryption_propertiesrW read_metadata num_columns read_schemalennames ParquetFileread)r5rOr.r/rbmetarYresults rrKrKs!/!J!J0"2 % 11 1   $> @D   q  ^^ $>@F v||  !! ! ^^ $>@F ;;4; ((rc  |tz }tjttddgidt dd}t ||tttt|t|tttjdttjdi\}}tjt d }tjtd 5t!||||d d d y #1swYy xYw) zYWrite an encrypted parquet, verify it's encrypted, and then read it using wrong keys.rrr<r=r>r@rAr2rEzIncorrect master key usedmatchN)rFrrrr rr9rHrIrr0r3rJpytestraises ValueErrorrK)rNrr5r8wrong_kms_connection_configwrong_crypto_factoryrOs r+test_encrypted_parquet_write_read_wrong_keyrts \ !D 22" 3* * - "z?L#W.?A$8T0j''0V95!5 22 -/ z)E F" #%@  """"s !C99Dct||tjtd5t j |t z jdddy#1swYyxYw)zmWrite an encrypted parquet, verify it's encrypted, but then try to read it without decryption properties. no decryptionrmN)rQrorpIOErrorrWrhrFrirNrs r0test_encrypted_parquet_read_no_decryption_configrysL&gz: w&6 76 w-.335666s +AA%ct||tjtd5t j |t z dddy#1swYyxYw)zwWrite an encrypted parquet, verify it's encrypted, but then try to read its metadata without decryption properties.rvrmN)rQrorprwrWrcrFrxs r9test_encrypted_parquet_read_metadata_no_decryption_configr{sE&gz: w&6 71 </0111 AAct||tjtd5t j |t z dddy#1swYyxYw)zuWrite an encrypted parquet, verify it's encrypted, but then try to read its schema without decryption properties.rvrmN)rQrorprwrWrerFrxs r7test_encrypted_parquet_read_schema_no_decryption_configr~sC&gz: w&6 7/ w-.///r|c |dz }tjt}tjt d5t ||tttd|dddy#1swYyxYw)MWrite an encrypted parquet, but give only footer key, without column key.z)encrypted_table_no_col_key.in_mem.parquetrz4Either column_keys or uniform_encryption must be setrmrN) rrrrorpOSErrorr9r rHrNrr5r8s r'test_encrypted_parquet_write_no_col_keyrsn @ @D22"$ w% &A T: '.? A AAAs A""A+c |dz }tjttddgid}t j t d5t||tttd|d d d y #1swYy xYw) rz=encrypted_table_col_key_and_uniform_encryption.in_mem.parquetrrT)rrrGz2Cannot set both column_keys and uniform_encryptionrmrN) rrrr rorprr9rHrs r;test_encrypted_parquet_write_col_key_and_uniform_encryptionrs T TD22" 3*  ! wR TA T: '.? AAAAs A,,A5c|dz }|}tj}d}tj|}tjt d5t |||||dddy#1swYyxYw).kms_factory-s!!=>>rrrmN)rr,r-rorpKeyErrorr4rNrr!r5r8r.r*r/s r&test_encrypted_parquet_write_kms_errorr$sy ? ?D/224? %%k2N x| 4Gj2C 5~ GGGG A((A1c |dz }|}tj}Gddtjfd}tj|}t j t d5t|||||dddy#1swYyxYw)rrc"eZdZdZdZdZdZy)Jtest_encrypted_parquet_write_kms_specific_error..ThrowingKmsClientzVA KmsClient implementation that throws exception in wrap/unwrap calls cPtjj|||_y)z%Create an InMemoryKmsClient instance.N)r KmsClient__init__configselfrs rrzStest_encrypted_parquet_write_kms_specific_error..ThrowingKmsClient.__init__Gs LL ! !$ ' DKrctd)NCannot Wrap Keyrqr key_bytesmaster_key_identifiers rwrap_keyzStest_encrypted_parquet_write_kms_specific_error..ThrowingKmsClient.wrap_keyLs./ /rctd)NzCannot Unwrap Keyrr wrapped_keyrs r unwrap_keyzUtest_encrypted_parquet_write_kms_specific_error..ThrowingKmsClient.unwrap_keyOs01 1rN__name__ __module__ __qualname____doc__rrrrrThrowingKmsClientrBs  !  0 2rrc|Sr&r)r)rs rr*zDtest_encrypted_parquet_write_kms_specific_error..kms_factoryRs !=>>rrrmN)rr,rr-rorprqr4) rNrr!r5r8r.r*r/rs @r/test_encrypted_parquet_write_kms_specific_errorr9s ? ?D/2242BLL2 ?%%k2N z): ;Gj2C 5~ GGGGs +BB c|dz }|}tj}d}tj|}tjt d5t |||||dddy#1swYyxYw)z@Write an encrypted parquet, but raise ValueError in kms_factory.0encrypted_table_kms_factory_error.in_mem.parquetctd)NCannot create KmsClientrr(s rr*zCtest_encrypted_parquet_write_kms_factory_error..kms_factoryfs233rrrmN)rr,r-rorprqr4rs r.test_encrypted_parquet_write_kms_factory_errorr]s~ G GD/2244%%k2N z6 8G j2C 5~ GGGGrc|dz }|}tj}Gddfd}tj|}tjt 5t |||||dddy#1swYyxYw)z_Write an encrypted parquet, but use wrong KMS client type that doesn't implement KmsClient.rc"eZdZdZdZdZdZy)Otest_encrypted_parquet_write_kms_factory_type_error..WrongTypeKmsClientz4This is not an implementation of KmsClient. c&|j|_yr&)r$master_keys_maprs rrzXtest_encrypted_parquet_write_kms_factory_type_error..WrongTypeKmsClient.__init__s#)#9#9D rcyr&rrs rrzXtest_encrypted_parquet_write_kms_factory_type_error..WrongTypeKmsClient.wrap_keyrcyr&rrs rrzZtest_encrypted_parquet_write_kms_factory_type_error..WrongTypeKmsClient.unwrap_keyrrNrrrrWrongTypeKmsClientr{s  :  rrc|Sr&r)r)rs rr*zHtest_encrypted_parquet_write_kms_factory_type_error..kms_factorys!">??rN)rr,r-rorp TypeErrorr4) rNrr!r5r8r.r*r/rs @r3test_encrypted_parquet_write_kms_factory_type_errorrqs G GD/224  @%%k2N y !Gj2C 5~ GGGGs A33A<c Jd}tjttddgidddt ddd }||tjt }tddgi|_d|_d|_d|_t d|_ d|_ d |_ ||y) Nc0t|jk(sJddg|jtk(sJd|jk(sJ|j sJ|j rJtd|jk(sJ|jrJd|jk(sJy)NrrAES_GCM_CTR_V1$@r>) rrrr rBplaintext_footerdouble_wrappingrrCinternal_key_materialrD)r8s r!validate_encryption_configurationzZtest_encrypted_parquet_encryption_configuration..validate_encryption_configurations"3">">>>>Sz.::<HHHH#4#I#IIII 1111$4444&*;*J*JJJJ$::::'<<<<r)rrrBrrrCrrDr) rrrr rrrBrrrCrrD)rr8encryption_config_1s r/test_encrypted_parquet_encryption_configurationrs=22"!C:0- .#  &&7844"$'3c3Z&B#/?,+/(*/')24)@&05-/2,%&9:rctjtd}td|jk(sJtj}td|_td|jk(sJy)Nrr>rE)rrJrrC)rOdecryption_config_1s r/test_encrypted_parquet_decryption_configurationrsl22 .0 T "&7&F&F FF F446)24)@& T "&9&H&H HH Hrcd}tjdddddd}||tj}d|_d|_d|_ddd|_||y) Ncd|jk(sJd|jk(sJd|jk(sJddd|jk(sJy)N Instance1URL1MyTokenkey_material_1key_material_2key1key2kms_instance_idkms_instance_urlkey_access_tokenr$)r.s rvalidate_kms_connection_configzPtest_encrypted_parquet_kms_configuration..validate_kms_connection_configsb3CCCCC.?????1BBBBB)3CD%556 76rrrrrrrr)rr,rrrr$)rr.kms_connection_config_1s r(test_encrypted_parquet_kms_configurationrs722#"$$  ##89 446.9+/5,/8,  /+##:;rzNPlaintext footer - reading plaintext column subset reads encrypted columns too)reasonc>|tz }tjttddgidd}tj tt jdttjdi}d}tj|}t|||||y ) zWrite an encrypted parquet, with plaintext footer and with single wrapping, verify it's encrypted, and then read plaintext columns.rrTF)rrrrr2r#ct|Sr&r'r(s rr*zStest_encrypted_parquet_write_read_plain_footer_single_wrapping..kms_factoryr+rN) rFrrrr r,rHr3rIr-r4rNrr5r8r.r*r/s r>test_encrypted_parquet_write_read_plain_footer_single_wrappingrs \ !D 22" 3*  22 Z..w7 '..1 ?%%k2ND*.?1>Crz'External key material not supported yetc|tz }tjtid}tjtt j di}d}tj|}t|||||y)zWrite an encrypted parquet, with external key material. Currently it's not implemented, so should throw an exceptionF)rrrr2r#ct|Sr&r'r(s rr*z:test_encrypted_parquet_write_external..kms_factoryr+rN) rFrrrr,rHr3r-r4rs r%test_encrypted_parquet_write_externalrs| \ !D22"#% 22(**;*;G*DE?%%k2ND*.?1>Crc x|tz }t||tttt |\}}t |tjtd}tdD]T}|j||}|Jtj||} | jd} |j| rTJy) z`Write an encrypted parquet, verify it's encrypted, and then read it multithreaded in a loop.r=r>rE2Nr^Tr`)rFr9rr rHrIrrrJrrangerbrWrhrirL) rNrr!r5r.r/rOirbrkrPs rtest_encrypted_parquet_loopr s \ !D -A j/<W-!)>$22 -/2Y /%3%N%N !#4&6")555 (BD{{t{4   ... /rc R|tz }t||tttt |\}}t |tjtd}|j||}~tj||}|jd} |j| sJy)z^ Test that decryption properties can be used if the crypto factory is no longer alive r=r>rEr^Tr`N)rFr9rr rHrIrrrJrrbrWrhrirL) rNrr!r5r.r/rOrbrkrPs r%test_read_with_deleted_crypto_factoryr=s \ !D,@ j/<W-!)>$22 -/!/!J!J0"2 ^^ $>@F;;4;0L   \ ** *rc j|tz }t||tttt |\}}t jtd}|j||}tj||}|j|sJtj||}|j|sJy)z>Write an encrypted parquet then read it back using read_table.r=r>rEr^N) rFr9rr rHrIrrJrrbrW read_tablerL) rNrr!r5r.r/rOrbrPs r!test_encrypted_parquet_read_tablerUs \ !D-A j/<W-!)>22 -/!/!J!J0"2===WXL   \ ** *=='ACL   \ ** *r)3rodatetimerpyarrowrpyarrow.parquetparquetrWpyarrow.parquet.encryption encryptionr pyarrow.tests.parquet.encryptionrr ImportErrorrFrHrrIr markparquet_encryption pytestmarkfixturerr!r0r9rQrSr4rKrtryr{r~rrrrrrrrrxfailrrrrrrrrrs"2 ++ 20    KK"" KK h h# # 1 1,+>+6")" "F61/A"A(G*!GHG(GB ;FI<:23C3CNCDCEC4/:+0+{ B BsD D"!D"