For [RFC1951] zipped safe-base64 [RFC3548] output the input to base64 encoding is
LLSSSSZZZZZZZZZZZZZZ -- RFC1951 zipped safe-base64
For encrypted modes the input to AES (or other symmetric cipher) is
NNNNLLSSSSZZZZZZZZZZ -- Note how nonce is prepended
The NNNN is used as initialization vector and actual encryption encompasses LL, SSSS, and ZZZZ.
In RSA-AES the session key is encrypted using RSA and prepended to the input for base64 encoding.
KKEEEECCCCCCCCCCCCCC -- RSA-AES: note prepended session key
16 bytes of nonce. This is used as initialization vector
for AES or 3DES cipher operated in CBC mode.
Bigendian integer representing signature length in bytes.
0 means none. Negative values reserved for future use.
The signature in binary
Bigendian integer representing encrypted session key
length in bytes. Negative values are reserved for future use.
RSA encrypted session key in binary
Ciphertext from the symmetric cipher, including nonce.
In RSA operations RSA_PKCS1_OAEP_PADDING padding is used (PKCS #1 v2.0).