**Abstract**

Cancer is the second leading cause of mortality in the United States. There are several therapeutic regimens employed to mitigate the mortality rate of cancer. This includes the use of chemotherapy, radiation, immunotherapy, and precision medicine/targeted therapy. Targeted therapy involves the use of drugs that target a specific pathway or biomolecule compromised in cancer for cancer treatment. Aberrant expression of epigenetic enzymes has been well documented for their contribution in driving tumorigenesis and other cancer hallmarks. Hence, there is an urgent need for novel drug discovery and development in epigenetics to help combat various cancer morbidities. Herein, we review the roles and consequences of dysregulated function of several epigenetic enzymes, with a focus on histone methyltransferases (HMTs). Additionally, we discussed the current efforts made in the development of small molecule inhibitors for a few representative HMTs implicated in different cancers. Furthermore, the common screening assays used in discovering potent small molecule inhibitors were also detailed in this chapter. Overall, this book chapter highlights the significance of targeting HMTs in different cancers and the clinical application potentials/limitations faced by the developed or emerging small molecule inhibitors of HMTs for the purpose of cancer therapy.

**Keywords:** cancer, drug discovery, epigenetics, histone methyltransferases, small molecule inhibitors

### **1. Introduction**

Since the conception of the term "epigenetic landscape" by Conrad Waddington in 1940, the field of epigenetics has rapidly evolved with technological advances. In the study of embryonic development, it was observed that a single gene has the ability to produce different phenotypes, so epigenetics was used to describe the mechanisms through which that happens [1]. Today, epigenetics is defined as the study of changes in organisms caused by modification of gene expression through addition and removal of chemical groups to nucleotides and proteins rather than the alteration of the genetic code itself [2]. The human genome contains approximately 3 billion bases of nucleotides and they are compacted into chromosomes in the nucleus via histone proteins. About 146 base pairs of nucleotides are wound around core histone octamers and are sealed with the linker histone (H1) to form a nucleosome.

The linker histone connects multiple nucleosomes in the chromatin. The core histone octamers consist of two dimers of H2A-H2B and a tetramer of H3-H4 proteins [3]. These core histones contain two regions namely: the histone fold and the histone tails. The C-termini of H2A and N-termini of other core histones protrude out of the fold to form histone tails and are commonly subjected to epigenetic modifications [4]. DNA and RNA also undergo epigenetic modifications, and these modifications control gene expression and maintain genomic integrity. Epigenetic enzymes can be broadly categorized into three components: the writers, the erasers, and the readers. Writers are enzymes responsible for adding the modifications, erasers remove it, and readers recognize it. These modifications include but are not limited to methylation, acetylation, phosphorylation, ubiquitination, GlcNAcylation, and SUMOylation [5].

DNA mainly undergoes methylation, and this occurs through the action of a family of DNA methyltransferases (DNMT1, DNMT2, DNMT3a, and DNMT3b). DNMTs covalently modify DNA by catalyzing the transfer of methyl group from S-adenosylmethionine (SAM) to the C5 position on a cytosine ring. DNA methylation mostly occurs in CpG islands, a region of the DNA rich in cytosine and guanine base repeats [6]. This modification to the DNA functions to repress transcription when it occurs in gene promoters and regulates splicing when it occurs in gene bodies [7]. DNA methylation is a reversible mechanism, which can be either passive through reduced DNMT1 activity during DNA replication or active through activity of its erasers, DNA demethylases. For instance, the ten-eleven translocation proteins (TET 1/2/3) are human demethylases that catalyze the oxidation of 5-methyl-cytosine to 5-hydroxymethylcytosine [8]. Following DNA methylation, often readers such as the family of methyl binding domain proteins recognize the methylation marks to drive transcriptional repression [9].

RNA is also methylated on C5 position of cytosine (m<sup>5</sup> C) and N6 position of adenosine ring(m6 A) by family of RNA methyltransferases such as Dnmt2, NOP2/Sun, Mettl3, and Mettl14. RNA methylation can be reversed by these RNA demethylases: fat mass and obesity associated protein (Fto) and α-ketoglutaratedependent dioxygenase alkB homolog 5 (AlkBH5) [10]. Methylation modification on RNA is interpreted by readers such as the YTH domain family, and they mediate RNA splicing, export, stability, maturation, decay, secondary structure switch, and translation [11]. There have also been reports of RNA acetylation by NAT10 acetyltransferase, which functions to promote mRNA stability and efficiency in translation [12].

Moving up the central dogma, lysine and arginine residues on histone proteins are mostly subjected to various post-translational modifications by their respective epigenetic enzymes to either activate or repress transcription. Although the focus of this chapter is histone methylation, histone acetylation will be briefly discussed. Histone methyltransferases (HMTs) and histone acetylases (HATs) are the writers of histone methylation and acetylation, respectively. HMTs can be further subdivided into lysine methyltransferase (KMT) and protein arginine methyltransferase (PRMT) [13]. The families and functions of HMTs will be further explored in this chapter. On the other hand, several HATs have been discovered, with the major ones being the GNATs (Gcn5-N-acyltransferases), the MYST families, and p300/ CBP [5]. These enzymes catalyze the transfer of acetyl group from acetyl Co A to the side chain amino group on histone lysine residues, inducing transcriptionally active chromatin [13]. Histone deacetylases and histone demethylases are involved in reversing the histone modifications discussed above. The families of lysine-specific histone demethylase 1 (LSD1) and Jumonji histone demethylases (JMJD) mediate the removal of methyl groups from histone through respective mechanisms [13, 14]. Readers of histone methylation include protein containing

**3**

*Discovery of Small Molecule Inhibitors for Histone Methyltransferases in Cancer*

in detail in *Section II* "Histone Methyltransferases in Cancer."

can undergo mutation, making drug accessibility to target difficult [27].

Given the myriad of biological targets that have been discovered to mediate cancer progression, there has been increasing interest in the development of small molecule inhibitors capable of decreasing the activity of those targets. Small molecules are intracellular targeting compounds with low molecular weight of less than 900Da. They can modulate their target activity as an agonist or antagonist [23]. The growing interest in the use of small molecules for drug development is not only

the MBT, PHD, chromo, Tudor, double/tandem Tudor, Ankyrin Repeats, zf-CW,

Given this array of epigenetic enzymes and their broad spectrum of function in regulating several gene expression in humans, the roles of epigenetic enzymes have been implicated in tumorigenesis. The epigenome of cancerous cells has widespread changes in DNA methylation and histone modification patterns [16]. For instance, hypermethylation of CpG islands in the promoter of chromodomain helicase DNAbinding protein 5 (CHD5), a chromatin remodeler, was observed in colon, breast, hepatocarcinoma, cervix, and glioma cell lines [17]. This results in the downregulation of CHD5, which plays a tumor suppressive role in cells. Similarly, hypomethylation of DNA at promoters of oncogenes such as insulin-like growth factor 2 (IGF2) has been observed in breast and colon cancers. The differential methylation patterns on promoters of tumor suppressors and oncogenes mediated by increased/reduced activity of DNMTs and TETs enzymes have been used as biomarkers to predict the predisposition of individuals to cancer [18]. Also, aberrant expression of histone acetyl transferases (HATs) and histone deacetylases (HDACs) has been linked to tumor development. Studies have shown that that p300/CBP acts as a coactivator with c-Myb to activate the transformation of fusion oncoproteins in myeloid leukemia [19]. Increased expression of HDACs was reported in gastric, prostate, colon, and breast carcinomas, and this results in repression of tumor suppressor genes like cyclin-dependent kinase inhibitor, see p21 [20]. Of all the histone modifications, histone methylation dysregulation is mostly attributed to poor prognosis in several cancers, which we will further elaborate

As these epigenetic enzymes' activity has been altered in cancer and contributes to the genomic instability in cancer cells, it is crucial to develop targeted therapeutic treatments to restore their normal function. Aside from surgery, some common treatment options for cancer patients can be broadly categorized as thus: chemotherapy, immunotherapy, radiotherapy, and precision medicine/targeted therapy [21]. These classes of treatments are not mutually exclusive and can be used in concert for treating cancer patients. Among these different therapeutic approaches, targeted therapy is the future for cancer treatment. Targeted therapy involves the use of drugs that target a specific biological molecule/pathway or drug treatment that requires genome profiling of an individual before it can be administered [22]. For optimal development of drugs for targeted therapies, it is important to identify a well-defined biological target whose activity contributes to one to several hallmarks of cancer including propagating growth signals, evading immunosurveillance, cell death resistance, activating metastatic programs, suppressing antigrowth signals, inducing angiogenesis, and enabling immortal replication of cells [23, 24]. For example, cancer patients whose tumors are driven by high activity of epidermal growth factor receptor (EGFR) signaling can be treated with specific monoclonal antibodies or small molecule kinase inhibitors antagonizing the aberrant signaling, and thereby reducing tumor proliferation [25]. Similarly, targeting an epigenetic enzyme, PRMT5, which is highly expressed in gastrointestinal cancers, with small molecule inhibitor, PR5-LL-CM01, was shown to slow down cancer cell growth and invasion *in vitro* [26]. The limitations of targeted therapies include cancer cell resistance to drug treatment by activating a parallel pathway or sometimes targets

*DOI: http://dx.doi.org/10.5772/intechopen.92830*

WD40, and PWWP domains [15].

*Discovery of Small Molecule Inhibitors for Histone Methyltransferases in Cancer DOI: http://dx.doi.org/10.5772/intechopen.92830*

the MBT, PHD, chromo, Tudor, double/tandem Tudor, Ankyrin Repeats, zf-CW, WD40, and PWWP domains [15].

Given this array of epigenetic enzymes and their broad spectrum of function in regulating several gene expression in humans, the roles of epigenetic enzymes have been implicated in tumorigenesis. The epigenome of cancerous cells has widespread changes in DNA methylation and histone modification patterns [16]. For instance, hypermethylation of CpG islands in the promoter of chromodomain helicase DNAbinding protein 5 (CHD5), a chromatin remodeler, was observed in colon, breast, hepatocarcinoma, cervix, and glioma cell lines [17]. This results in the downregulation of CHD5, which plays a tumor suppressive role in cells. Similarly, hypomethylation of DNA at promoters of oncogenes such as insulin-like growth factor 2 (IGF2) has been observed in breast and colon cancers. The differential methylation patterns on promoters of tumor suppressors and oncogenes mediated by increased/reduced activity of DNMTs and TETs enzymes have been used as biomarkers to predict the predisposition of individuals to cancer [18]. Also, aberrant expression of histone acetyl transferases (HATs) and histone deacetylases (HDACs) has been linked to tumor development. Studies have shown that that p300/CBP acts as a coactivator with c-Myb to activate the transformation of fusion oncoproteins in myeloid leukemia [19]. Increased expression of HDACs was reported in gastric, prostate, colon, and breast carcinomas, and this results in repression of tumor suppressor genes like cyclin-dependent kinase inhibitor, see p21 [20]. Of all the histone modifications, histone methylation dysregulation is mostly attributed to poor prognosis in several cancers, which we will further elaborate in detail in *Section II* "Histone Methyltransferases in Cancer."

As these epigenetic enzymes' activity has been altered in cancer and contributes to the genomic instability in cancer cells, it is crucial to develop targeted therapeutic treatments to restore their normal function. Aside from surgery, some common treatment options for cancer patients can be broadly categorized as thus: chemotherapy, immunotherapy, radiotherapy, and precision medicine/targeted therapy [21]. These classes of treatments are not mutually exclusive and can be used in concert for treating cancer patients. Among these different therapeutic approaches, targeted therapy is the future for cancer treatment. Targeted therapy involves the use of drugs that target a specific biological molecule/pathway or drug treatment that requires genome profiling of an individual before it can be administered [22]. For optimal development of drugs for targeted therapies, it is important to identify a well-defined biological target whose activity contributes to one to several hallmarks of cancer including propagating growth signals, evading immunosurveillance, cell death resistance, activating metastatic programs, suppressing antigrowth signals, inducing angiogenesis, and enabling immortal replication of cells [23, 24]. For example, cancer patients whose tumors are driven by high activity of epidermal growth factor receptor (EGFR) signaling can be treated with specific monoclonal antibodies or small molecule kinase inhibitors antagonizing the aberrant signaling, and thereby reducing tumor proliferation [25]. Similarly, targeting an epigenetic enzyme, PRMT5, which is highly expressed in gastrointestinal cancers, with small molecule inhibitor, PR5-LL-CM01, was shown to slow down cancer cell growth and invasion *in vitro* [26]. The limitations of targeted therapies include cancer cell resistance to drug treatment by activating a parallel pathway or sometimes targets can undergo mutation, making drug accessibility to target difficult [27].

Given the myriad of biological targets that have been discovered to mediate cancer progression, there has been increasing interest in the development of small molecule inhibitors capable of decreasing the activity of those targets. Small molecules are intracellular targeting compounds with low molecular weight of less than 900Da. They can modulate their target activity as an agonist or antagonist [23]. The growing interest in the use of small molecules for drug development is not only

*Translational Research in Cancer*

GlcNAcylation, and SUMOylation [5].

of adenosine ring(m6

translation [12].

methylation marks to drive transcriptional repression [9]. RNA is also methylated on C5 position of cytosine (m<sup>5</sup>

The linker histone connects multiple nucleosomes in the chromatin. The core histone octamers consist of two dimers of H2A-H2B and a tetramer of H3-H4 proteins [3]. These core histones contain two regions namely: the histone fold and the histone tails. The C-termini of H2A and N-termini of other core histones protrude out of the fold to form histone tails and are commonly subjected to

epigenetic modifications [4]. DNA and RNA also undergo epigenetic modifications, and these modifications control gene expression and maintain genomic integrity. Epigenetic enzymes can be broadly categorized into three components: the writers, the erasers, and the readers. Writers are enzymes responsible for adding the modifications, erasers remove it, and readers recognize it. These modifications include but are not limited to methylation, acetylation, phosphorylation, ubiquitination,

DNA mainly undergoes methylation, and this occurs through the action of a family of DNA methyltransferases (DNMT1, DNMT2, DNMT3a, and DNMT3b). DNMTs covalently modify DNA by catalyzing the transfer of methyl group from S-adenosylmethionine (SAM) to the C5 position on a cytosine ring. DNA methylation mostly occurs in CpG islands, a region of the DNA rich in cytosine and guanine base repeats [6]. This modification to the DNA functions to repress transcription when it occurs in gene promoters and regulates splicing when it occurs in gene bodies [7]. DNA methylation is a reversible mechanism, which can be either passive through reduced DNMT1 activity during DNA replication or active through activity of its erasers, DNA demethylases. For instance, the ten-eleven translocation proteins (TET 1/2/3) are human demethylases that catalyze the oxidation of 5-methyl-cytosine to 5-hydroxymethylcytosine [8]. Following DNA methylation, often readers such as the family of methyl binding domain proteins recognize the

A) by family of RNA methyltransferases such as Dnmt2,

NOP2/Sun, Mettl3, and Mettl14. RNA methylation can be reversed by these RNA demethylases: fat mass and obesity associated protein (Fto) and α-ketoglutaratedependent dioxygenase alkB homolog 5 (AlkBH5) [10]. Methylation modification on RNA is interpreted by readers such as the YTH domain family, and they mediate RNA splicing, export, stability, maturation, decay, secondary structure switch, and translation [11]. There have also been reports of RNA acetylation by NAT10 acetyltransferase, which functions to promote mRNA stability and efficiency in

Moving up the central dogma, lysine and arginine residues on histone proteins are mostly subjected to various post-translational modifications by their respective epigenetic enzymes to either activate or repress transcription. Although the focus of this chapter is histone methylation, histone acetylation will be briefly discussed. Histone methyltransferases (HMTs) and histone acetylases (HATs) are the writers of histone methylation and acetylation, respectively. HMTs can be further subdivided into lysine methyltransferase (KMT) and protein arginine methyltransferase (PRMT) [13]. The families and functions of HMTs will be further explored in this chapter. On the other hand, several HATs have been discovered, with the major ones being the GNATs (Gcn5-N-acyltransferases), the MYST families, and p300/ CBP [5]. These enzymes catalyze the transfer of acetyl group from acetyl Co A to the side chain amino group on histone lysine residues, inducing transcriptionally active chromatin [13]. Histone deacetylases and histone demethylases are involved in reversing the histone modifications discussed above. The families of lysine-specific histone demethylase 1 (LSD1) and Jumonji histone demethylases (JMJD) mediate the removal of methyl groups from histone through respective mechanisms [13, 14]. Readers of histone methylation include protein containing

C) and N6 position

**2**

due to their small size, which enables easy permeability into the cell, but also their desirable pharmacokinetics, pharmacodynamics, longer shelf life, and easy synthesis [28]. Several small molecule modulators have been developed into drugs to treat various types of cancers. The range of small molecule inhibitors developed to enable tumor regression can be broadly categorized into small molecule kinase inhibitors, proteasome inhibitors, metalloproteinases and heat shock protein inhibitors, and apoptosis targeting inhibitors [29]. The most common small molecule inhibitors, kinase inhibitors, have been used to inhibit the several protein kinases whose activity is dysregulated in cancers. The first tyrosine kinase inhibitor drug, Imatinib, is a small molecule ATP analog that competitively inhibits Bcr-Abl fusion protein kinase activity in chronic myeloid leukemia patients [30]. Similarly, a number of small molecule inhibitors targeting epigenetic enzymes implicated in tumorigenesis are in development or have been FDA approved for cancer treatment. For instance, drugs like belinostat and romidepsin are HDAC inhibitors that are FDA approved for the treatment of lymphoma [31]. After the first clinical trial in 2014, tazemetostat, an EZH2 small molecule inhibitor, moved to phase 2 clinical trial and was fast tracked by FDA in 2017 for the treatment of follicular lymphoma [32]. The use of tazemetostat for treatment of epithelioid sarcoma in adults 16 years and above was also granted accelerated approval by the FDA. These examples, among many others, show the potentials of epigenetic modifiers as a druggable target for cancer treatment. More of these small molecule inhibitors for histone methyltransferases will be explored later on in this chapter. However, challenges in the clinical application of certain small molecule inhibitors as drugs remain due to their off-target effects or development of resistance by cancer cells [29].
