**2. Investigation of Shamir's protocol based on commutative cryptography**

Shamir's protocol was described in the book [6]. This protocol can be executed for any cryptosystem (both symmetric and asymmetric) if they satisfy the following condition:

$$f\_{K\_{\mathcal{A}}}\left(f\_{K\_{\mathcal{B}}}(\mathcal{M})\right) = f\_{K\_{\mathcal{B}}}\left(f\_{K\_{\mathcal{A}}}(\mathcal{M})\right),\tag{1}$$

where *<sup>f</sup> KA*ð Þ *KB* ð Þ *<sup>M</sup>* is encryption algorithm using the keys *KA* and *KB*, respectively and *<sup>M</sup>* is any message. If relation (1) is true for some encryption algorithm *<sup>f</sup> KA*ð Þ *KB* ð Þ *<sup>M</sup>* then can be performed protocol presented in **Figure 1**.

If the relation (1) is valid, then we get:

$$\mathbf{C}'\_{A} = f\_{K\_{A}}^{-1}(\mathbf{C}\_{\mathcal{B}}) = f\_{K\_{A}}^{-1}\left(f\_{K\_{\mathcal{B}}}\left(f\_{K\_{A}}(\mathbf{M})\right)\right) = f\_{K\_{A}}^{-1}\left(f\_{K\_{A}}\left(f\_{K\_{\mathcal{B}}}(\mathbf{M})\right)\right) = f\_{K\_{\mathcal{B}}}(\mathbf{M}) \tag{2}$$

On the next step user B decrypts *f KB* ð Þ *M* by the algorithm *f* �1 *KB* ð Þ� and gets the message *M*. But the following question arises—for which cryptosystems the relation (1) is valid? Let us call such encryption algorithms, for which (1) is true, *commutative encryption* (CE). It is easy to verify that for such cryptographic standard as AES, 3DES, GOST [7] the relation (1) is not fulfilled for all messages. Let us consider public key cryptosystems and, first of all, Rivest-Shamir-Adleman system (RSA). It is well known that such cryptosystem is determined by the following parameters: *p*, *q* different primes, *n* ¼ *pq modulo* ð Þ, *φ* ¼ ð Þ *p* � 1 ð Þ *q* � 1 , *e*—encryption key, 1≤ *e*≤*φ*, *<sup>d</sup>*—decryption key, *<sup>d</sup>* <sup>¼</sup> *<sup>e</sup>*�<sup>1</sup>*mod <sup>φ</sup>*, gcdð Þ¼ *<sup>e</sup>*, *<sup>φ</sup>* 1, 1<sup>≤</sup> *<sup>M</sup>* <sup>≤</sup>*<sup>n</sup>* � <sup>1</sup>—integers. Encryption algorithm for RSA is *<sup>E</sup>* <sup>¼</sup> *Me mod n*; decryption algorithm is *<sup>M</sup>* <sup>¼</sup> *Edmod n*. Then for RSA condition (1) takes the form:

$$(\mathcal{M}^{\epsilon\_A} \bmod n)^{\epsilon\_B} \bmod n = (\mathcal{M}^{\epsilon\_B} \bmod n)^{\epsilon\_A} \bmod n \tag{3}$$

Since exponentiation by modulo *n* is a commutative operation, the relation (3) becomes trivially true. In fact

$$(\mathsf{M}^{\epsilon\_A} \bmod n)^{\epsilon\_B} \bmod n = \mathsf{M}^{\epsilon\_A \epsilon\_B} \bmod n = (\mathsf{M}^{\epsilon\_B} \bmod n)^{\epsilon\_A} \bmod n = \mathsf{M}^{\epsilon\_B \epsilon\_A} \bmod n.$$

On the one hand RSA system is self-sufficient one but on the other hand there may be situations where its public key *eB* is unknown for the user *A* or should be secure. Then protocol shown in **Figure 1** could have a reason.

*Example*. Let *p* ¼ 3, *q* ¼ 5, *M* ¼ 2, *eA* ¼ 3, *eB* ¼ 5. Then it is easy to verify that both parts of (3) are equal to 8.

Let us consider as next public key cryptosystem, Rabin's one, that is determined by the following parameters [7]: *p*, *q* are distinct primes (secret key), *nA B*ð Þ ¼ *pq modulo* ð Þ is public key. 0 <sup>≤</sup> *<sup>M</sup>* <sup>≤</sup>*<sup>n</sup>* � 1, *<sup>E</sup>* <sup>¼</sup> *<sup>M</sup>*<sup>2</sup> *mod n*. Then relation (1) takes the form:

$$\left(\mathrm{M}^2 \mathrm{mod}\, n\_{\mathrm{A}}\right)^2 \mathrm{mod}\, n\_{\mathrm{B}} = \left(\mathrm{M}^2 \mathrm{mod}\, n\_{\mathrm{B}}\right)^2 \mathrm{mod}\, n\_{\mathrm{A}}\tag{4}$$

It is easy to make sure that (4) is not satisfied, generally speaking. In fact, let us consider an example: *pA* ¼ 3, *qA* ¼ 5, *nA* ¼ 15, *pB* ¼ 11, *qB* ¼ 7, *nB* ¼ 77, *M* ¼ 5.

**Figure 1.**

*Protocol based on relation (1) where f* �<sup>1</sup> *KA* ð Þ� *is decryption algorithm given the key KA.* Then it is easy to check that the left side of (4) is equal to 23, while the right side is equal to 10.

In order to find out the reason of that fact, let us present both sides of (4) as the following values, respectively:

$$\begin{aligned} \left(\mathsf{M}^2 \boldsymbol{mod} \ n\_A\right)^2 \boldsymbol{mod} \ n\_B &= \left(\mathsf{M}^2 - n\_A \boldsymbol{l}\right)^2 \boldsymbol{mod} \ n\_B = \\ &= \left(\mathsf{M}^4 - 2\mathsf{M}^2 n\_A \boldsymbol{l} + n\_A^2 \boldsymbol{l}^2\right) \boldsymbol{mod} \ n\_B, \end{aligned} \tag{5}$$

where *l* is some integer.

$$\begin{aligned} \left(M^2 \bmod n\_B\right)^2 \bmod n\_A &= \left(M^2 - n\_B m\right)^2 \bmod n\_A = \\ &= \left(M^4 - 2M^2 n\_B m + n\_B^2 m^2\right) \bmod n\_A,\end{aligned} \tag{6}$$

where *m* is some integer too. We can see that in general case *nA* 6¼ *nB* the right sides of (5) and (6) are different values. It was mentioned in Section 1 that some PKC is vulnerable against QC's. Such of them that are resistant against QC's attacks were called *post quantum cryptosystems*. The most known among them is McEliece PKC [7]. The matrices that determine it are: *<sup>k</sup>* � *<sup>n</sup>* matrix *<sup>G</sup>*<sup>~</sup> *A B*ð Þ <sup>¼</sup> *SA B*ð Þ*GA B*ð Þ*PA B*ð Þ—public key where *SA B*ð Þ is nonsingular random *k* � *k* matrix, *PA B*ð Þ is permutation random *n* � *n* matrix, *GA B*ð Þ is random Goppa code generating matrix. Matrices *SA B*ð Þ, *GA B*ð Þ, *PA B*ð Þ are believed as secret key jointly with a binary random vector *ZA B*ð Þ of the known length *n*, and with given weight *tA B*ð Þ. Encryption procedure is performed as follows:

$$E\_{A(B)} = M\tilde{G}\_{A(B)} \oplus Z\_{A(B)},\tag{7}$$

where " ⊕ " is bitwise modulo two addition. It follows from (7) that commutative property (1) for McEliece CS was looking as:

$$\mathcal{G}\_{K\_{\mathcal{A}}}\left(f\_{K\_{\mathcal{B}}}(\mathcal{M})\right) = \left(\mathcal{M}\breve{\mathcal{G}}\_{\mathcal{B}}\oplus Z\_{\mathcal{B}}\right)\breve{\mathcal{G}}\_{\mathcal{A}}\oplus Z\_{\mathcal{A}} = \mathcal{M}\breve{\mathcal{G}}\_{\mathcal{B}}\breve{\mathcal{G}}\_{\mathcal{A}}\oplus Z\_{\mathcal{B}}\breve{\mathcal{G}}\_{\mathcal{A}}\oplus Z\_{\mathcal{A}}\tag{8}$$

$$\mathcal{G}\_{K\_{\mathcal{B}}}\left(f\_{K\_{\mathcal{A}}}(\mathcal{M})\right) = \left(\mathbf{M}\mathbf{\tilde{G}}\_{A}\oplus \mathbf{Z}\_{A}\right)\mathbf{\tilde{G}}\_{B}\oplus \mathbf{Z}\_{B} = \mathbf{M}\mathbf{\tilde{G}}\_{A}\mathbf{\tilde{G}}\_{B}\oplus \mathbf{Z}\_{A}\mathbf{\tilde{G}}\_{B}\oplus \mathbf{Z}\_{B} \tag{9}$$

It is easy to see that the values in the right sides of (8) and (9) are, generally speaking, unequal one to another owing at least different vectors *ZA* and *ZB*. Hence McEliece CS does not correspond to CE algorithm. At a single glance stream cipher is looking as CE algorithm. In fact, the encryption procedure for this cipher is:

$$E(M, K) = M \oplus \chi(K),\tag{10}$$

where *E M*ð Þ ,*K* is a cipher text given secret key *K*, *γ*ð Þ *K* is an encrypting binary sequence (gamma) given the key *K*, " ⊕" is bitwise modulo 2 addition, *M* is the binary message to be encrypted. It follows from (10) that the condition (1) takes the following form for *γ*-based stream ciphers:

$$\mathbf{M} \oplus \boldsymbol{\chi}(\mathbf{K}\_A) \oplus \boldsymbol{\chi}(\mathbf{K}\_B) = \mathbf{M} \oplus \boldsymbol{\chi}(\mathbf{K}\_B) \oplus \boldsymbol{\chi}(\mathbf{K}\_A) \tag{11}$$

that is valid trivially. The use of stream ciphers as CE in protocol shown in **Figure 1** was looking as it is presented in **Figure 2**.

**Figure 2.** *Protocol of secret message transmission based on stream ciphers.*

However we can see from **Figure 2** that eavesdropper receiving *CB* and *C*<sup>0</sup> *<sup>A</sup>* be able to bitwise add them by modulo 2 that gives the following binary sequence: *M* ⊕ *γ*ð Þ *KB* ⊕ *γ*ð Þ *KB* ⊕ *M* ⊕ *γ*ð Þ¼ *K*A *γ*ð Þ *KA* . Next, attacker add bitwise modulo 2 the last sequence *γ*ð Þ *KA* with the sequence *CA* intercepted during the first transmission session that gives *M* ⊕ *γ*ð Þ *KA* ⊕ *γ*ð Þ¼ *KA M* that is exactly the desired open sequence *M*. (It is worth to note that similar conclusion was mentioned in [6].)
