**2. Histone methyltransferases in cancer**

As we mentioned above, epigenetic enzymes, including histone methyltransferases, are novel targets for cancer therapy. In this section, we will review the role of histone methyltransferases in cancer.

#### **2.1 Lysine methyltransferases**

Lysine methylation of histones was first characterized in the 1960s [33, 34] and was initially described as an "irreversible" post-translation modification. This dogma was challenged by the discovery of histone demethylases by Shi et al*.* indicating a dynamic function of methylation allowing the addition and removal of methyl groups [35]. The proteins responsible for the addition of methyl groups to histones are known as lysine methyltransferases (KMTs). KMTs are broadly defined as SET [Su(var)3-9, Enhancer of Zeste, Trithorax] domain-containing proteins and non-SET domain-containing proteins [36]. The only non-SET domain-containing KMTs identified so far are DOT1 (disrupter of telomeric silencing 1) family members, which methylate K79 of histone H3 and which also do not share structural similarities with SET-containing proteins [37–39]. KMTs act by catalyzing the transfer of 1~3 methyl groups to lysine residues of histone and non-histone proteins through the addition of the cofactor S-adenosyl methionine (SAM), which acts as a methyl donor group [40]. In the case of SET domain-containing proteins, SAM interacts and orients with a lysine residue of the substrate histone tail within the catalytic pocket of the SET domain. Then, a tyrosine residue acts to deprotonate the ε-amino group of the lysine residue, which allows the lysine chain to nucleophilically attack the methyl group of the SAM molecule, transferring the methyl group to the lysine side chain [41]. In the case of non-SET domain-containing KMTs,

**5**

*Discovery of Small Molecule Inhibitors for Histone Methyltransferases in Cancer*

the enzyme DOT1 acts to methylate a lysine residue in the histone core [40]. As we described above, histone methylation is a critical epigenetic modification of chromatin that can impact genomic stability, alter expression of different genes, determine cell lineage, alter DNA methylation, as well as control cell mitosis [42]. SET-containing proteins have been characterized in greater detail than non-SET-containing proteins. SET domain proteins are characterized as seven families of the superfamily of KMTs: SUV39, SET1, SET2, EZ, RIZ, SMYD, and SUV4-20 [36]. There are also several SET domain proteins that do not fall into these groups including SET7/9 and SET8 [36]. The SUV39 family has been characterized the most out of these groups of KMTs. Specifically, *Schizosaccharomyces pombe* Cryptic loci regulator 4 (CLR4), human SUV39H1, and murine SUV39H2 were among the first identified SET domain protein lysine methyltransferases characterized when their sequence homology between their SET domains was discovered. These proteins methylate lysine 9 of histone H3 (H3K9) [43]. SET1 and SET2 complexes are involved in the RNA polymerase II holoenzyme [44, 45]. TSET1 acts to trimethylate H3K4 and is associated with early gene transcription as opposed to SET2-mediated methylation of H3K36, which is associated with later transcriptional elongation of downstream genes. The mammalian nuclear-receptor-binding SET domain-containing protein (NSD1, a member of the SET2 family) has an important function in methylation of H3K36 and H4K20 in development [46]. Lu et al*.* in our research group also reported that NSD1 could methylate non-histone protein, NF-κB, at K218/221 of its p65 subunit, leading to the activation of NF-κB and its downstream target gene expression [47–50]. Another example is SETDB1, a H3K9 methyltransferase, which is amplified in primary tumors of lung cancer patients and contributes to the invasive phenotype of tumor cells [51]. Additionally, SETDB1 methylates H3K9 in euchromatin, and is also required for development [52]. Human SET7/9 acts to mono-methylate H3K4 [53]. Furthermore, SET domain enzymes' functions are not specifically tied to histone methylation. Human SET7/9 methylates K189 on transcription factor TAF10, which increases RNA polymerase II affinity and transcriptional activity, thereby expression of TAF10-dependent genes [54]. SET7/9 is also involved in p53 methylation, which increases its stability [55]. These examples described are only a few of the many SET-containing proteins of which over 50 different proteins have been discovered [56]. While SET domain proteins are typically referred to as histone lysine methyltransferases, it may be more accurate to refer to them as protein lysine methyltransferases due to their

*DOI: http://dx.doi.org/10.5772/intechopen.92830*

identification of non-histone targets for these KMTs.

with tumorigenesis of several different cancers.

These examples of SET and non-SET domain-containing proteins exhibit the importance of KMTs in the regulation of histone and non-histone proteins. Therefore, it is not surprising that dysregulation of KMT function can result in dysregulation of cellular functions. Specifically several KMTs have been associated

H3K9 methyltransferases SETDB1 and G9a are both known to have roles in gene silencing and embryo development [57]. G9a regulates cancer metabolism in several types of cancer [58]. Overexpression of G9a is associated with worse prognosis in patients with prostate cancer. Intriguingly, G9a knockdown in breast and lung cancer has been shown to promote E-cadherin expression and thus tumor metastasis [59, 60]. Moreover, G9a's higher expression predicts higher mortality of ovarian cancer patients [61] and is reported to be associated with cell growth and proliferation in neuroblastoma [57]. Furthermore, G9a has been shown to have a critical role in the development of pancreatic carcinoma and acute myeloid squamous cell carcinoma. G9a-dependent repression of genes is also associated with development of leukemia as well as squamous cell carcinomas [57]. SETDB1 is another H3K9 methyltransferase reported to play a role in numerous types of human cancer.

#### *Discovery of Small Molecule Inhibitors for Histone Methyltransferases in Cancer DOI: http://dx.doi.org/10.5772/intechopen.92830*

the enzyme DOT1 acts to methylate a lysine residue in the histone core [40]. As we described above, histone methylation is a critical epigenetic modification of chromatin that can impact genomic stability, alter expression of different genes, determine cell lineage, alter DNA methylation, as well as control cell mitosis [42].

SET-containing proteins have been characterized in greater detail than non-SET-containing proteins. SET domain proteins are characterized as seven families of the superfamily of KMTs: SUV39, SET1, SET2, EZ, RIZ, SMYD, and SUV4-20 [36]. There are also several SET domain proteins that do not fall into these groups including SET7/9 and SET8 [36]. The SUV39 family has been characterized the most out of these groups of KMTs. Specifically, *Schizosaccharomyces pombe* Cryptic loci regulator 4 (CLR4), human SUV39H1, and murine SUV39H2 were among the first identified SET domain protein lysine methyltransferases characterized when their sequence homology between their SET domains was discovered. These proteins methylate lysine 9 of histone H3 (H3K9) [43]. SET1 and SET2 complexes are involved in the RNA polymerase II holoenzyme [44, 45]. TSET1 acts to trimethylate H3K4 and is associated with early gene transcription as opposed to SET2-mediated methylation of H3K36, which is associated with later transcriptional elongation of downstream genes. The mammalian nuclear-receptor-binding SET domain-containing protein (NSD1, a member of the SET2 family) has an important function in methylation of H3K36 and H4K20 in development [46]. Lu et al*.* in our research group also reported that NSD1 could methylate non-histone protein, NF-κB, at K218/221 of its p65 subunit, leading to the activation of NF-κB and its downstream target gene expression [47–50]. Another example is SETDB1, a H3K9 methyltransferase, which is amplified in primary tumors of lung cancer patients and contributes to the invasive phenotype of tumor cells [51]. Additionally, SETDB1 methylates H3K9 in euchromatin, and is also required for development [52]. Human SET7/9 acts to mono-methylate H3K4 [53]. Furthermore, SET domain enzymes' functions are not specifically tied to histone methylation. Human SET7/9 methylates K189 on transcription factor TAF10, which increases RNA polymerase II affinity and transcriptional activity, thereby expression of TAF10-dependent genes [54]. SET7/9 is also involved in p53 methylation, which increases its stability [55]. These examples described are only a few of the many SET-containing proteins of which over 50 different proteins have been discovered [56]. While SET domain proteins are typically referred to as histone lysine methyltransferases, it may be more accurate to refer to them as protein lysine methyltransferases due to their identification of non-histone targets for these KMTs.

These examples of SET and non-SET domain-containing proteins exhibit the importance of KMTs in the regulation of histone and non-histone proteins. Therefore, it is not surprising that dysregulation of KMT function can result in dysregulation of cellular functions. Specifically several KMTs have been associated with tumorigenesis of several different cancers.

H3K9 methyltransferases SETDB1 and G9a are both known to have roles in gene silencing and embryo development [57]. G9a regulates cancer metabolism in several types of cancer [58]. Overexpression of G9a is associated with worse prognosis in patients with prostate cancer. Intriguingly, G9a knockdown in breast and lung cancer has been shown to promote E-cadherin expression and thus tumor metastasis [59, 60]. Moreover, G9a's higher expression predicts higher mortality of ovarian cancer patients [61] and is reported to be associated with cell growth and proliferation in neuroblastoma [57]. Furthermore, G9a has been shown to have a critical role in the development of pancreatic carcinoma and acute myeloid squamous cell carcinoma. G9a-dependent repression of genes is also associated with development of leukemia as well as squamous cell carcinomas [57]. SETDB1 is another H3K9 methyltransferase reported to play a role in numerous types of human cancer.

*Translational Research in Cancer*

development of resistance by cancer cells [29].

**2. Histone methyltransferases in cancer**

histone methyltransferases in cancer.

**2.1 Lysine methyltransferases**

due to their small size, which enables easy permeability into the cell, but also their desirable pharmacokinetics, pharmacodynamics, longer shelf life, and easy synthesis [28]. Several small molecule modulators have been developed into drugs to treat various types of cancers. The range of small molecule inhibitors developed to enable tumor regression can be broadly categorized into small molecule kinase inhibitors, proteasome inhibitors, metalloproteinases and heat shock protein inhibitors, and apoptosis targeting inhibitors [29]. The most common small molecule inhibitors, kinase inhibitors, have been used to inhibit the several protein kinases whose activity is dysregulated in cancers. The first tyrosine kinase inhibitor drug, Imatinib, is a small molecule ATP analog that competitively inhibits Bcr-Abl fusion protein kinase activity in chronic myeloid leukemia patients [30]. Similarly, a number of small molecule inhibitors targeting epigenetic enzymes implicated in tumorigenesis are in development or have been FDA approved for cancer treatment. For instance, drugs like belinostat and romidepsin are HDAC inhibitors that are FDA approved for the treatment of lymphoma [31]. After the first clinical trial in 2014, tazemetostat, an EZH2 small molecule inhibitor, moved to phase 2 clinical trial and was fast tracked by FDA in 2017 for the treatment of follicular lymphoma [32]. The use of tazemetostat for treatment of epithelioid sarcoma in adults 16 years and above was also granted accelerated approval by the FDA. These examples, among many others, show the potentials of epigenetic modifiers as a druggable target for cancer treatment. More of these small molecule inhibitors for histone methyltransferases will be explored later on in this chapter. However, challenges in the clinical application of certain small molecule inhibitors as drugs remain due to their off-target effects or

As we mentioned above, epigenetic enzymes, including histone methyltransferases, are novel targets for cancer therapy. In this section, we will review the role of

Lysine methylation of histones was first characterized in the 1960s [33, 34] and was initially described as an "irreversible" post-translation modification. This dogma was challenged by the discovery of histone demethylases by Shi et al*.* indicating a dynamic function of methylation allowing the addition and removal of methyl groups [35]. The proteins responsible for the addition of methyl groups to histones are known as lysine methyltransferases (KMTs). KMTs are broadly defined as SET [Su(var)3-9, Enhancer of Zeste, Trithorax] domain-containing proteins and non-SET domain-containing proteins [36]. The only non-SET domain-containing KMTs identified so far are DOT1 (disrupter of telomeric silencing 1) family members, which methylate K79 of histone H3 and which also do not share structural similarities with SET-containing proteins [37–39]. KMTs act by catalyzing the transfer of 1~3 methyl groups to lysine residues of histone and non-histone proteins through the addition of the cofactor S-adenosyl methionine (SAM), which acts as a methyl donor group [40]. In the case of SET domain-containing proteins, SAM interacts and orients with a lysine residue of the substrate histone tail within the catalytic pocket of the SET domain. Then, a tyrosine residue acts to deprotonate the ε-amino group of the lysine residue, which allows the lysine chain to nucleophilically attack the methyl group of the SAM molecule, transferring the methyl group to the lysine side chain [41]. In the case of non-SET domain-containing KMTs,

**4**

SETDB1 is involved in regulation of several cellular processes, including apoptosis, DNA damage repair, and regulation of transcription factors [62]. SETB1 is associated with oncogenic activity and is upregulated in several cancers including lung cancers, gliomas, and prostate cancer [63–66]. For instance, in lung cancer, overexpression of SETDB1 promotes invasiveness and knockdown reduced lung cancer cell growth. In gliomas, cell proliferation was reduced by suppression of SETDB1. In prostate cancer, downregulation of SETDB1 led to reduced cellular proliferation, migratory ability, and invasive ability.

EZH2 (Enhancer of zeste homolog 2), a H3K27 methyltransferase, plays an important role in transcriptional repression [36]. EZH2 plays a critical regulatory role in as many as 46 types of human cancer [57, 67]. Typically, EZH2 is overexpressed in cancers, and its high expression has been linked to worse patient survival. For instance, EZH2 downregulation in breast cancer can block cell growth and survival [68]. Moreover, EZH2 knockdown inhibits invasive ability and cellular proliferation in prostate cancer [69]. Additionally, EZH2 has also been shown to have roles in gliomas and renal cell carcinoma, wherein decreased EZH2 expression reduces cellular proliferative ability and promotes apoptosis [70].

Another example is SMYD3 (SET and MYND domain-containing 3). It is a KMT that methylates H3K4 and is extremely important for initiation of transcription. High SMYD3 activity is indicative of an epigenetic signature of active enhancers [71]. SMYD3 knockdown in colorectal cancer can impair cellular proliferation [72]. In breast cancer, knockdown of SMYD3 also inhibits cell growth, and overexpression can promote carcinogenesis by regulation of the Wnt signaling pathways [73]. Additionally, reduction of SMYD3 expression in prostate cancer inhibits cellular proliferation, migration, and colony formation [74].

H3K36 methylation is critical for transcriptional elongation, and H3K36 methyltransferases have been identified to play important regulatory roles in several types of cancer. NSD1, a H3K36 KMT, is involved in prostate cancer androgen receptor transactivation, which results in prostate tumorigenesis [75]. Overexpression of NSD1 in neuroblastoma reduces cellular growth ability and colony formation [76]. NSD1 has also been reported to play a role in myelomas and lung cancers [57]. As we described above, our lab also found that NSD1 activates NF-κB to induce its target gene expression. The function of these genes is frequently involved in oncogenic phenotype, or cytokine, chemokine secretion [47]. Another example is H4K20 methylation, which is reported to be important for gene transcription. For example, overexpression of methyltransferases of H4K20 SUV420H1 and SUV420H2 is associated with decreased cell invasiveness in breast cancer, and knockdown increases epithelial to mesenchymal transition [77].

DOT1, the only known H3K79 methyltransferase and non-SET containing KMT, also plays an important role in cancer development. DOT1 is involved in cell survival and colony formation in several forms of leukemia [78]. The activity of human DOT1L (hDOT1 like) methyltransferase is compromised in mixed lineage leukemia (MLL) and is required for maintaining proliferative state of transformed cells [79]. Also, downregulation of DOT1 in lung cancer can cause cell cycle arrest and reduce cellular proliferation [80].

Together, these examples show several of the pathways regulated by KMTs and also highlight the critical importance of tight regulation of these pathways wherein dysregulation of KMTs can result in promotion of tumorigenesis.

#### **2.2 Protein arginine methyltransferases**

In contrast to KTMs, there is another important family of protein methyltransferases, namely, protein arginine methyltransferases (PRMTs). It is a family of nine

**7**

pathways.

*Discovery of Small Molecule Inhibitors for Histone Methyltransferases in Cancer*

members (PRMT1-9) and specifically catalyzes the methyl transfer from SAM to the guanidine nitrogen (ω-NG) of the arginine residues of protein substrates [81]. After the donation of methyl groups, SAM forms S-adenosyl-L-homocysteine (AdoHcy, SAH) and methylarginine is produced [81]. There are three forms of methylarginine recognized in mammalian cells: ω-NG-monomethylarginine (MMA), ω-NG, NG-asymmetric dimethylarginine (ADMA), and ω-NG, N'Gsymmetric dimethylarginine (SDMA) [82]. The family of PRMTs is categorized into three major types: type I, II, and III. Type I and II PRMTs first catalyze the formation of MMA, and then type I PRMTs (PRMT1, 2, 3, 4, 6, and 8) further catalyze the formation of ADMA, while type II (PRMT5 and 9) would instead catalyze the production of SDMA. For type III PRMTs (PRMT7), they only catalyze the formation of MMA [83]. These three types of PRMTs methylated similar substrates, such as histone, but they also can catalyze different non-histone substrate proteins. Additionally, although the majority of PRMTs methylate glycine-arginine-rich (GAR) motifs in their substrates [84, 85], some of them display a preference of methylation on the proline-glycine-methionine rich (PGM) motifs in substrate proteins such as PRMT4 [86, 87], while PRMT5 is characterized with the methyla-

PRMTs are widely expressed in mammalian cells and regulate primary cellular processes, including cell proliferation and differentiation. The abnormal expression of some types of PRMTs such as PRMT1, 4, 5 and 6 frequently leads to tumorigen-

PRMT1 was the first protein arginine transferase recognized in mammals and assumes the large fraction of arginine transferases activity in mammalian cells [89, 90]. It has been reported that PRMT1 is related to various kinds of cancer. Le Romancer et al. suggested that PRMT1 governed the interaction of estrogen receptor α (ERα) with steroid receptor coactivator proteins (Src), the p85 subunit of phosphatidylinositide 3-kinases (PI3K) and focal adhesion kinase (FAK) [91]. PRMT1-mediated ERα methylation is integral for the activation of the Src-PI3K-FAK signaling pathway [91]. In their subsequent report, the authors further demonstrated arginine methylation of ERα by PRMT1 might remarkably activate protein kinase B (PKB, also known as AKT) [92]. Another example is methylation of Axin by PRMT1. Cha et al. showed that arginine methylation of Axin by PRMT1 may activate the WNT pathway by destabilizing Axin and promote tumorigenesis [93, 94]. Importantly, it is reported that methylation of meiotic recombination 11 (MRE11, also known as MRE11A) and p53 binding protein 1 (53BP1) by PRMT1 can block the DNA repair pathway, contributing to cancer progression [95, 96]. MRE11 combines with DNA repair protein RAD50 and Nijmegen breakage syndrome 1 (NBS1) to form MRE11-RAD50-NBS1 complex (MRN complex). The mammalian MRN complex plays significant role in repairing DNA double-strand breaks (DSBs). Yu et al. reported that the deficiency of arginine methylation of MRE11 in its GAR motif resulted in exonuclease and DNA-binding defects and finally failing to repair DNA damage [96]. Interestingly, Boisvert et al*.* found that 53BP1 could not relocalize to DNA damage sites and γ-H2AX formation was decreased in fibroblasts treated with methylase inhibitors [95, 97]. Moreover, Mitchell et al. discovered that depletion of PRMT1 affected the length and stability of telomere [98]. Since dysfunction of both DNA repair pathway and telomere maintenance is known to be the cause of cancer, deregulation of PRMT1 may lead to tumorigenesis by these

In addition to PRMT1, another PRMT member, PRMT4 (also known as CARM1), is reported to be tightly associated with estrogen-mediated oncogenesis of breast cancer through the upregulation of transcription factor E2F1 expression [99]. Moreover, CARM1 is suggested to be overexpressed in human colon

*DOI: http://dx.doi.org/10.5772/intechopen.92830*

tion of both types of motifs in proteins [86, 88].

esis and malignancy.

#### *Discovery of Small Molecule Inhibitors for Histone Methyltransferases in Cancer DOI: http://dx.doi.org/10.5772/intechopen.92830*

members (PRMT1-9) and specifically catalyzes the methyl transfer from SAM to the guanidine nitrogen (ω-NG) of the arginine residues of protein substrates [81]. After the donation of methyl groups, SAM forms S-adenosyl-L-homocysteine (AdoHcy, SAH) and methylarginine is produced [81]. There are three forms of methylarginine recognized in mammalian cells: ω-NG-monomethylarginine (MMA), ω-NG, NG-asymmetric dimethylarginine (ADMA), and ω-NG, N'Gsymmetric dimethylarginine (SDMA) [82]. The family of PRMTs is categorized into three major types: type I, II, and III. Type I and II PRMTs first catalyze the formation of MMA, and then type I PRMTs (PRMT1, 2, 3, 4, 6, and 8) further catalyze the formation of ADMA, while type II (PRMT5 and 9) would instead catalyze the production of SDMA. For type III PRMTs (PRMT7), they only catalyze the formation of MMA [83]. These three types of PRMTs methylated similar substrates, such as histone, but they also can catalyze different non-histone substrate proteins. Additionally, although the majority of PRMTs methylate glycine-arginine-rich (GAR) motifs in their substrates [84, 85], some of them display a preference of methylation on the proline-glycine-methionine rich (PGM) motifs in substrate proteins such as PRMT4 [86, 87], while PRMT5 is characterized with the methylation of both types of motifs in proteins [86, 88].

PRMTs are widely expressed in mammalian cells and regulate primary cellular processes, including cell proliferation and differentiation. The abnormal expression of some types of PRMTs such as PRMT1, 4, 5 and 6 frequently leads to tumorigenesis and malignancy.

PRMT1 was the first protein arginine transferase recognized in mammals and assumes the large fraction of arginine transferases activity in mammalian cells [89, 90]. It has been reported that PRMT1 is related to various kinds of cancer. Le Romancer et al. suggested that PRMT1 governed the interaction of estrogen receptor α (ERα) with steroid receptor coactivator proteins (Src), the p85 subunit of phosphatidylinositide 3-kinases (PI3K) and focal adhesion kinase (FAK) [91]. PRMT1-mediated ERα methylation is integral for the activation of the Src-PI3K-FAK signaling pathway [91]. In their subsequent report, the authors further demonstrated arginine methylation of ERα by PRMT1 might remarkably activate protein kinase B (PKB, also known as AKT) [92]. Another example is methylation of Axin by PRMT1. Cha et al. showed that arginine methylation of Axin by PRMT1 may activate the WNT pathway by destabilizing Axin and promote tumorigenesis [93, 94]. Importantly, it is reported that methylation of meiotic recombination 11 (MRE11, also known as MRE11A) and p53 binding protein 1 (53BP1) by PRMT1 can block the DNA repair pathway, contributing to cancer progression [95, 96]. MRE11 combines with DNA repair protein RAD50 and Nijmegen breakage syndrome 1 (NBS1) to form MRE11-RAD50-NBS1 complex (MRN complex). The mammalian MRN complex plays significant role in repairing DNA double-strand breaks (DSBs). Yu et al. reported that the deficiency of arginine methylation of MRE11 in its GAR motif resulted in exonuclease and DNA-binding defects and finally failing to repair DNA damage [96]. Interestingly, Boisvert et al*.* found that 53BP1 could not relocalize to DNA damage sites and γ-H2AX formation was decreased in fibroblasts treated with methylase inhibitors [95, 97]. Moreover, Mitchell et al. discovered that depletion of PRMT1 affected the length and stability of telomere [98]. Since dysfunction of both DNA repair pathway and telomere maintenance is known to be the cause of cancer, deregulation of PRMT1 may lead to tumorigenesis by these pathways.

In addition to PRMT1, another PRMT member, PRMT4 (also known as CARM1), is reported to be tightly associated with estrogen-mediated oncogenesis of breast cancer through the upregulation of transcription factor E2F1 expression [99]. Moreover, CARM1 is suggested to be overexpressed in human colon

*Translational Research in Cancer*

migratory ability, and invasive ability.

SETDB1 is involved in regulation of several cellular processes, including apoptosis, DNA damage repair, and regulation of transcription factors [62]. SETB1 is associated with oncogenic activity and is upregulated in several cancers including lung cancers, gliomas, and prostate cancer [63–66]. For instance, in lung cancer, overexpression of SETDB1 promotes invasiveness and knockdown reduced lung cancer cell growth. In gliomas, cell proliferation was reduced by suppression of SETDB1. In prostate cancer, downregulation of SETDB1 led to reduced cellular proliferation,

EZH2 (Enhancer of zeste homolog 2), a H3K27 methyltransferase, plays an important role in transcriptional repression [36]. EZH2 plays a critical regulatory role in as many as 46 types of human cancer [57, 67]. Typically, EZH2 is overexpressed in cancers, and its high expression has been linked to worse patient survival. For instance, EZH2 downregulation in breast cancer can block cell growth and survival [68]. Moreover, EZH2 knockdown inhibits invasive ability and cellular proliferation in prostate cancer [69]. Additionally, EZH2 has also been shown to have roles in gliomas and renal cell carcinoma, wherein decreased EZH2 expression

Another example is SMYD3 (SET and MYND domain-containing 3). It is a KMT that methylates H3K4 and is extremely important for initiation of transcription. High SMYD3 activity is indicative of an epigenetic signature of active enhancers [71]. SMYD3 knockdown in colorectal cancer can impair cellular proliferation [72]. In breast cancer, knockdown of SMYD3 also inhibits cell growth, and overexpression can promote carcinogenesis by regulation of the Wnt signaling pathways [73]. Additionally, reduction of SMYD3 expression in prostate cancer inhibits cellular

H3K36 methylation is critical for transcriptional elongation, and H3K36 methyltransferases have been identified to play important regulatory roles in several types of cancer. NSD1, a H3K36 KMT, is involved in prostate cancer androgen receptor transactivation, which results in prostate tumorigenesis [75]. Overexpression of NSD1 in neuroblastoma reduces cellular growth ability and colony formation [76]. NSD1 has also been reported to play a role in myelomas and lung cancers [57]. As we described above, our lab also found that NSD1 activates NF-κB to induce its target gene expression. The function of these genes is frequently involved in oncogenic phenotype, or cytokine, chemokine secretion [47]. Another example is H4K20 methylation, which is reported to be important for gene transcription. For example, overexpression of methyltransferases of H4K20 SUV420H1 and SUV420H2 is associated with decreased cell invasiveness in breast cancer, and knockdown increases

DOT1, the only known H3K79 methyltransferase and non-SET containing KMT, also plays an important role in cancer development. DOT1 is involved in cell survival and colony formation in several forms of leukemia [78]. The activity of human DOT1L (hDOT1 like) methyltransferase is compromised in mixed lineage leukemia (MLL) and is required for maintaining proliferative state of transformed cells [79]. Also, downregulation of DOT1 in lung cancer can cause cell cycle arrest and reduce

Together, these examples show several of the pathways regulated by KMTs and also highlight the critical importance of tight regulation of these pathways wherein

In contrast to KTMs, there is another important family of protein methyltransferases, namely, protein arginine methyltransferases (PRMTs). It is a family of nine

dysregulation of KMTs can result in promotion of tumorigenesis.

reduces cellular proliferative ability and promotes apoptosis [70].

proliferation, migration, and colony formation [74].

epithelial to mesenchymal transition [77].

**2.2 Protein arginine methyltransferases**

cellular proliferation [80].

**6**

cancer and exert a crucial role in Wnt signaling through mediating the action of β-catenin on Wnt target genes as a transcriptional coactivator [100]. Moreover, c-fos is a proto-oncogene and overexpressed in a set of cancers. Some groups reveal that PRMT4 regulates transcriptional activation of c-fos, and that matrix metalloproteinases (MMPs), c-fos target genes, are significantly downregulated in CARM1-deficient cells [101]. Therefore, arginine methylation by PRMT4 is related to multiple oncogenic signaling pathways.

An important PRMT member that plays a critical role in cancer is PRMT5. As a primary type II PRMT, PRMT5 functions in the presence of other binding partners, such as MEP50. Hou et al. discovered that E-cadherin expression was remarkably repressed by SNAIL and PRMT5 recruited by bridge molecule AJUBA, which was favorable for tumor metastasis [102]. It was also reported that p53 can be methylated on multiple arginine sites by PRMT5 in response to DNA damage [103]. Scoumanne et al. proposed that PRMT5 inhibition may promote cancer cells to progress toward apoptosis under chemotherapy/radiotherapy [104]. Moreover, Cho et al*.* found methylation of E2F1 by PRMT5 weakened its ability to promote apoptosis and repress proliferation, indicating PRMT5 overexpression may enhance cancer cells' growth and survival [105]. It is well known that continuous NF-κB activation exists in most cancers. Our group uncovered that PRMT5 dimethylated the p65 subunit of NF-κB at arginine 30 (R30) to activate NF-κB pathway [106], an important transcription factor that is involved in the progression of many cancers. Also, PRMT5 is reported to promote its own overexpression in several cancers through a feedback loop involving NF-κB signaling [107].

Besides the PRMTs we discussed above, PRMT6 is widely taken as a transcriptional repressor. Neault et al*.* reported that embryonic fibroblast cells from the PRMT6 knockout mouse were subjected to a premature senescence, while the cellular senescence can be rescued in PRMT6 and p53 double knockout mouse embryonic fibroblast (MEF) cells [108], affirming growth suppression effect of excess p53 due to PRMT6 deficiency. Thus, PRMT6 actively suppresses p53 cascade to promote tumorigenesis in cancer.

Taken together, many members in the PRMT family have shown essential role in cancer development and progression. Thus, it is unsurprising that these PRMTs have become the rising targets in cancer therapeutics in recent years.

### **3. Discovery of small molecule inhibitors for histone methyltransferases in cancer treatment**

#### **3.1 Screening assays for epigenetic drug discovery**

The timeline of drug development from conception of the idea to a feasible drug available in the market for treatment of diseases takes between 12 and 15 years and can cost up to \$1billion [109]. Drug discovery process begins with identifying a druggable biological target that contributes to a disease progression. These targets can be identified through text mining from online databases, microarray data mining using bioinformatic tools, proteomic data mining from proteomic databases, and chemogenomic data mining, which involves simultaneous exploration of multiple cell phenotypes by screening small molecules from chemical libraries to a number of biological targets [110]. Then, promising or known targets can be validated *in vitro* and *in vivo* to confirm that their activity influences phenotype associated with a disease. This step is followed by screening or high throughput screening (HTS), which describes the process of sifting through compound libraries for molecules with high affinity for a target of interest [111]. The two approaches

**9**

drug development purposes [119].

**3.3 FRET (Förster/fluorescent resonance energy transfer) assay**

Discovered in 1946 by Theodor Förster, FRET is cell-based assay that enables real-time observance of molecular interactions within cells [120]. This phenomenon

*Discovery of Small Molecule Inhibitors for Histone Methyltransferases in Cancer*

of developing assays for compound library screening are the biochemical targetbased approach and the cell-based approach. Biochemical target-based approach is often employed as the primary screening assay in epigenetic compound screen because it allows the direct monitoring of ability of a target activity as opposed to cell-based assays wherein changes in cell phenotypes are measured [112]. Listed below are some of screening assays used in the preclinical development of drugs for

**3.2 AlphaLISA screen (amplified luminescent proximity homogeneous** 

AlphaLISA is a high throughput screening approach used to analyze and measure post-translational modifications, protein-protein interactions, and concentrations of analytes. The robust, highly sensitive, reproducible, miniaturized, scalable, and automated nature of AlphaLISA assays earned them their widespread application in research and drug discovery. The principle behind AlphaLISA technology is based on the mechanism of another methodology, Luminescent oxygen channeling immunoassay (LOCI). LOCI involves chemiluminescent reaction of a singlet oxygen transfer and was developed in 1994 by Ullman et al*.* to quantify latex particle binding [113]. Similarly, AlphaLISA assays employ biotinylated antibody bound to streptavidin donor beads and an acceptor bead bound to a second antibody. These antibodies are specific to different epitope on a protein (could also be product of a bimolecular interaction or a modified protein). The binding of these antibodies to the proteins brings the donor and the acceptor bead into proximity of at most 200nm. Upon excitation of donor beads at 680nm, a singlet oxygen is excited from the donor bead and this triggers singlet oxygen reaction with the chemical dyes (thioxene and europium) on the acceptor bead, which results in chemiluminescent emission at 615nm [114, 115]. Multiplate readers equipped with AlphaLISA screen detections can be used to record signals. A systematic method used to validate the quality of HTS assay output like AlphaLISA is called the Z-factor. The Z-factor is calculated by accounting for the positive and negative controls' mean signal-tomean background ratio. Hence, in the design of AlphaLISA assay, to ensure quality control, wells containing maximum signal and no signal solution mix must be included [116]. This assay is widely optimized to screen for small molecule modulators for different epigenetic enzymes. For example, in our lab, Prabhu et al. used an optimized AlphaLISA screen protocol to identify a lead compound capable of targeting PRMT5, and subsequently, the compound was shown to be more potent in reducing pancreatic and colon cancer cells' proliferation compared to another commercially available compound EPZ015666 [26]. Not only is this assay used to screen for compounds for epigenetic targeted therapy, but also they uncover new roles of different epigenetic enzymes in cancer. Consequently, together with other assays, it was uncovered using AlphaLISA HTS assay that an inhibitor of G9a lysine methyltransferase, A-366, limits cell growth and differentiation in leukemic cells [117]. Potent small molecule inhibitors for EZH2, a methyltransferase known to silence tumor suppressor genes, are also being identified using this particular HTS assay [118]. AlphaLISA kits specific for histone methyltransferase modifications, among many other epigenetic enzymes, are commercially available for research and

*DOI: http://dx.doi.org/10.5772/intechopen.92830*

epigenetic enzymes.

**assay-linked assay)**

#### *Discovery of Small Molecule Inhibitors for Histone Methyltransferases in Cancer DOI: http://dx.doi.org/10.5772/intechopen.92830*

of developing assays for compound library screening are the biochemical targetbased approach and the cell-based approach. Biochemical target-based approach is often employed as the primary screening assay in epigenetic compound screen because it allows the direct monitoring of ability of a target activity as opposed to cell-based assays wherein changes in cell phenotypes are measured [112]. Listed below are some of screening assays used in the preclinical development of drugs for epigenetic enzymes.
