**6.1 Fault-tolerant fiber topology design**

BGP routing policies are usually made to avoid asymmetric or useless AS paths by setting the appropriate local preference values. However, these alternative AS paths worked as backup paths. Before the network failures from earthquake, Asia Pacific R&E network operators thought that removing the useless routes was urgent, because routing became too complicated after TEIN2 started. However, this complicated routing was able to provide valuable connections during network failures. This shows that maintaining full-mesh style routing information is very important for fault-tolerant routing.

Though BGP re-routing over the redundant AS paths was successful for the first step in restoration, it was not sufficient to provide full backup service without congestion by considering the traffic load. Since BGP routing does not carry QoS information, such as link capacity, link utilization, or available bandwidth, traffic re-routed to the backup AS path had experienced poor QoS, such as long delays. Therefore, QoS-aware BGP routing or traffic engineering-aware BGP routing is necessary.

### **6.2 Integrated network management**

During the restoration process, we used various network monitoring tools such as an MRTG (Oetiker, n.d.a), a network weather map, a BGP routing table visualizer, and a flow monitor. At first, the link outage was noticed on the network weather map, and the abrupt change of traffic load was noticed on the MRTG. However, the fast fault detection method that encompasses physical, link, routing, and application layers is necessary because it was able to identify the exact failure points and visualize their impacts on the network. In addition, a simulator or emulator that could show the results with the network topology and the traffic load before and after failures would be very useful in predicting the effects of faultmanagement decisions. While we took various restoration steps, we had to process the information collected by each different network-monitoring tool. Finally, the operators interpreted the situation and implemented recovery decisions manually. If the iperf (GOOGLE, n.d) or bwctl (INTERNET2, n.d.) is available throughout the network, the endto-end available bandwidth between ASs can be easily estimated. For example, to access Sydney from Tokyo, there are two possible routes. One is Tokyo – Seattle - Sydney, and the other is Tokyo – Honolulu - Sydney. The former provides 10 Gbps but has a long RTT. The latter route includes a bottleneck along the 155 Mbps path but has a short RTT. In addition, to make the final decision, we had to check the flow data, because MRTG or RRDTool (Oetiker, n.d.b) do not classify traffic breakdowns by their source/destination ASs. When the traffic from KR increased suddenly, the operators could not understand the reasons. This shows that integrated network monitoring or management systems would be very

Experience with Restoration of Asia Pacific Network Failures from Taiwan Earthquake 379

Yutaka Watanabe, OTC's director, took the MRTG snapshot of Figs. 8 and 9. AARNET NOC offered us the backup routes to access the TEIN2. KOREN NOC worked very hard to keep communication open between CERNET and ASNET even during the holiday. Hawaii University NOC worked very hard to keep control of this complicated routing. KDDI investigated the reason for the communication failures in the APAN area and gave useful advice to APAN Tokyo XP. The staff of Genkai XP NOC accepted CERNET's traffic from KOREN by upgrading the JP-KR link bandwidth. Due to these great efforts and the collaboration among the network engineers, we were able to quickly restore the Asia Pacific

This research was supported by the MIC (Ministry of Information and Communication), Korea, under the ITRC (Information Technology Research Center) support program supervised by the IITA (Institute of Information Technology Assessment) (NIPA-2011-

R&E networks.

C1090-1131-0005).

Country code Name JP Japan TH Thailand MY Malaysia HK Hong Kong ID Indonesia SG Singapore TW Taiwan CN China

KR Korea VN Vietnam

NP Nepal AU Australia BD Bangladesh

IN India LK Sri Lanka BT Bhutan KH Cambodia LA Laos PK Pakistan

US United States of America

Table 3. ISO 3166 Country Code (International Organization for Standardization [ISO], n.d.)

PH The Philipppines

**9. Appendix** 

in this chapter

useful for collecting information from several independent monitoring systems and for providing the correct information in an integrated wide view in case of significant network failures.
