**4.5 AS-level topology changes**

370 Earthquake Research and Analysis – Statistical Studies, Observations and Planning

D direct direct U C U direct D U US US

CERNET D D direct U N.A. TEIN2 TEIN2 TEIN2 TEIN2 TEIN2 TEIN2

ASTI U U U U U U U U U U U CSTNET C N.A. U U U N.A. N.A. N.A. N.A. U N.A. TEIN2 U TEIN2 N.A. TEIN2 U N.A. TEIN2 TEIN2 TEIN2 KOREN TEIN2 D: Detoured path through US U: unreachable C: Commodity link connection N.A. Non applicable Table 1. The monitored routing table for each R&E network on Dec.27 2006 (Kurokawa,

TEIN2 SingAREN (SG)

UniNet (TH)

U U N.A. N.A. N.A. N.A. APAN-JP APAN-

U U TEIN2 TEIN2 TEIN2 TEIN2 direct TEIN2

ThaiSARN (TH)

KREONET2 (KR)

AARNET (AU)

Hawaii

JP

CSTNET (CN)

Src\Dst APAN-JP (JP)

APAN-JP

APAN-TW

2006)

CERNET (CN)

KOREN direct direct APAN-

direct D APAN-

JP

APAN-TW (TW)

KOREN (KR)

JP

Fig. 10. BGP route changes of APAN before earthquake (Dec. 2006)

Fig. 11. BGP route changes of APAN after earthquake (Dec. 2006)

ASTI (PH)

To investigate the BGP route changes in detail, we used an AS-topology visualization tool called "ABEL2" (Sakiyama et al., 2006) that utilizes BGP routing tables that are stored every 10 minutes.

Fig. 12. After the earthquake at 19:30, Dec. 27 (UTC+9) 2006, the route from APAN-KR to GÉANT was diverted to a long AS path of (APAN-KR, APAN-JP, TransPAC, Abilene, GÉANT)

Fig. 13. Reconfigured AS path from APAN-KR to GÉANT (APAN-KR, CERNET, TEIN2-N, GÉANT) at 20:30, Dec. 27 (UTC+9) 2006

Figures 12, 13, and 14 show the changes between APAN-KR (AS9270) and GÉANT (AS20965) by the number of IP prefixes. It can be observed in Fig. 12, that just after the earthquake, at 19:30 on December 27th (UTC+9), 2006, the route from APAN-KR to GÉANT was diverted to the long AS path APAN-KR - APAN-JP - TransPAC (US) -

Experience with Restoration of Asia Pacific Network Failures from Taiwan Earthquake 373

On December 27th after the fiber cut, automatic BGP re-routing has been carried out between SG and JP. The route from SG to JP became SG-AU-Hawaii-JP instead of the direct

When the link between SG and KR was temporarily recovered with the backup fiber, the

Finally, on January 12th when the link between JP and SG was recovered with the direct

fiber, the RTT was reduced to 113 ms, which is slightly increased than the usual case.

link and its RTT was increased to 426 ms, while its normal RTT is around 88 ms.

RTT values was 240 ms between SG and JP via KR on December 28th.

Fig. 15. TEIN2 traffic was carried through Hawaii.

Fig. 16. TEIN2 traffic was routed through KR.

Abilene (US) – GÉANT (Tbl. 4) because of the TEIN2 link outage. Therefore, to connect APAN-KR to GÉANT with a shorter AS path, the operator configured the BGP routing policy to make CERNET (CN) announce GÉANT prefixes. As shown in Fig. 13, the route from APAN-KR to GÉANT was switched through CERNET at 20:30 on Dec. 27th (UTC+9), 2006. At 20:50 on Dec. 17th, 2006 (UTC+9), the TEIN2-SG NOC announced the route to EU, too. Since the link bandwidth between SG and KR is larger than that between CN and KR, the operator made a configuration for BGP routers to choose the AS path with the KR-SG link (Fig. 14).

Fig. 14. Reconfigured AS path from APAN-KR to GÉANT through TEIN2-SG at 20:50, Dec. 27 (UTC+9) 2006

### **4.6 Delay changes**

When Australia's Research and Education Network (AARNET) proposed that it would transit all the traffic of TEIN2 members (CN, SG, VN, MY, TH, ID), a flow data analysis was done and most of the traffic was found to be for Eastern Asia.

The APAN Tokyo XP and the TEIN2 NOC asked the AARNET NOC to route the TEIN2 traffic through Honolulu rather than through Seattle, because the traffic was between the Eastern Asia and TEIN2 members. The shorter Round Trip Time (RTT) worked better, especially with TCP-based applications (Tbl. 5). However, this operation occupied the link between Honolulu and Tokyo, because the bandwidth between Tokyo and Honolulu is 155 Mbps and the one between the Tokyo and TEIN2 NOCs is 622 Mbps and the one between AARNET and TEIN2 is 1 Gbps (Fig. 15).

The link between KR and JP, whose bandwidth was 10 Gbps, was up. According to the analysis, the KR-JP link still had space to carry the traffic, and this route is much shorter than the route through Australia.

The TEIN2 and AARNET NOCs and the Tokyo XP stopped routing the TEIN2 traffic through Australia, and the KR transit policy was initiated. But the bandwidth between CN and KR was also 155 Mbps, too, so the link was occupied again (Fig. 16).

At that moment, the JGNII JP-SG link (Tbl. 4) was restored, but after precise check by the engineers, it looked as though the telecommunication company had switched the route and provided connectivity. TEIN2 NOC and the Tokyo XP decided to separate the traffic to Tokyo into two routes. That is, CN traffic was transferred through KR, and other TEIN2 traffic was transferred through SG (Fig. 17).

Fig. 15. TEIN2 traffic was carried through Hawaii.

Abilene (US) – GÉANT (Tbl. 4) because of the TEIN2 link outage. Therefore, to connect APAN-KR to GÉANT with a shorter AS path, the operator configured the BGP routing policy to make CERNET (CN) announce GÉANT prefixes. As shown in Fig. 13, the route from APAN-KR to GÉANT was switched through CERNET at 20:30 on Dec. 27th (UTC+9), 2006. At 20:50 on Dec. 17th, 2006 (UTC+9), the TEIN2-SG NOC announced the route to EU, too. Since the link bandwidth between SG and KR is larger than that between CN and KR, the operator made a configuration for BGP routers to choose the AS path

Fig. 14. Reconfigured AS path from APAN-KR to GÉANT through TEIN2-SG at 20:50, Dec.

When Australia's Research and Education Network (AARNET) proposed that it would transit all the traffic of TEIN2 members (CN, SG, VN, MY, TH, ID), a flow data analysis was

The APAN Tokyo XP and the TEIN2 NOC asked the AARNET NOC to route the TEIN2 traffic through Honolulu rather than through Seattle, because the traffic was between the Eastern Asia and TEIN2 members. The shorter Round Trip Time (RTT) worked better, especially with TCP-based applications (Tbl. 5). However, this operation occupied the link between Honolulu and Tokyo, because the bandwidth between Tokyo and Honolulu is 155 Mbps and the one between the Tokyo and TEIN2 NOCs is 622 Mbps and the one between

The link between KR and JP, whose bandwidth was 10 Gbps, was up. According to the analysis, the KR-JP link still had space to carry the traffic, and this route is much shorter

The TEIN2 and AARNET NOCs and the Tokyo XP stopped routing the TEIN2 traffic through Australia, and the KR transit policy was initiated. But the bandwidth between CN

At that moment, the JGNII JP-SG link (Tbl. 4) was restored, but after precise check by the engineers, it looked as though the telecommunication company had switched the route and provided connectivity. TEIN2 NOC and the Tokyo XP decided to separate the traffic to Tokyo into two routes. That is, CN traffic was transferred through KR, and other TEIN2

and KR was also 155 Mbps, too, so the link was occupied again (Fig. 16).

done and most of the traffic was found to be for Eastern Asia.

AARNET and TEIN2 is 1 Gbps (Fig. 15).

traffic was transferred through SG (Fig. 17).

than the route through Australia.

with the KR-SG link (Fig. 14).

27 (UTC+9) 2006

**4.6 Delay changes** 

Fig. 16. TEIN2 traffic was routed through KR.

On December 27th after the fiber cut, automatic BGP re-routing has been carried out between SG and JP. The route from SG to JP became SG-AU-Hawaii-JP instead of the direct link and its RTT was increased to 426 ms, while its normal RTT is around 88 ms.

When the link between SG and KR was temporarily recovered with the backup fiber, the RTT values was 240 ms between SG and JP via KR on December 28th.

Finally, on January 12th when the link between JP and SG was recovered with the direct fiber, the RTT was reduced to 113 ms, which is slightly increased than the usual case.

Experience with Restoration of Asia Pacific Network Failures from Taiwan Earthquake 375

local preference to ignore them. However, after the earthquake, these useless BGP routes worked automatically as backup paths. In the Asia Pacific R&E networks, the routes became very complicated after TEIN2 started because TEIN2 provided a few unexpected routes around the world. Because there were backup AS paths, automatic BGP re-routing could be used for first aid to provide the connectivity to the ASs that lost the primary paths. However, automatic BGP re-routing did not consider the traffic engineering parameters of

the available bandwidth and the backup traffic load.

Table 2. Examples of the "useless routes"

**5.2 Traffic engineering with BGP policy change** 

(Src, Dst) Usual Path Useless Path

changed the BGP routing policy as shown in Fig. 13 and Fig. 14.

and Hawaii-JP links. Besides, CN traffic took an asymmetrical path.

part of the CN traffic was from CERNET but the other part was from TEIN2.

Fig. 19. Traffic from KR monitored at APAN JP (Dec. 26 2006 - Jan. 26 2007)}

(JP, TEIN2) JP-TEIN2 JP-Hawaii/Seattle-AU-TEIN2 (TEIN2, JP) TEIN2-JP TEIN2-GÉANT2-Abilene-JP

(US, US) Abilene US-KREONET2-JP-TransPAC2-US (JP, KR) JP-KR link JP-TEIN2-JP- TEIN2-SG -AU-Seattle-KR

BGP by itself does not provide any information regarding link capacity or available bandwidth. Moreover, due to recent VLAN (Varadaraja, 1997) technology, the distance between two ASs has no relation to physical distance. Thus, QoS information of the detour routes must be examined by the operators. This makes systems reliant on human knowledge of traffic engineering. To remove the congestion due to the long detour AS path, we

The members of TEIN2 (VN, MY, SG, ID, PH) lost their connections to APAN Tokyo XP because the fiber broke. AARNET NOC proposed backup routes for accessing APAN Tokyo XP through AU and Hawaii. However, this solution caused congestion on both the CN-KR

Therefore, to solve the traffic engineering issue, Tokyo XP made a decision to divide CN traffic by announcing CN IP prefixes through KR NOC and grouping CN prefixes at Tokyo XP. The results were monitored by Cisco NetFlow (CISCO, n.d.). The operators found out that half of the KR traffic was from CN. After a careful examination, it was discovered that a

Fig. 17. TEIN2 traffic was routed through SG and KR.

Figure 18 shows the RTT between SG and JP just after the earthquake and the recovery process.

Fig. 18. RTT between SG and JP during the restoration process
