Mux is sending broadcast

Two days spent in troubleshooting a point to point link with lot of hops between them. I stumble to see one weird thing that whenever the mux sfp port is connected to the router, the router card starts rebooting. This is the case not with a single router but with the multiple ones. We changed everything but all in vein. At last we able to catch a little point about mux which is using layer 2 switching and forwarding vlans traffic to the router directly. The broadcast was so huge which forces routes card to halt.
Click Here To Read Rest Of The Post...

BGP Vulnerability Found In IOS-XR

Cisco IOS XR Software contains a vulnerability in the Border Gateway Protocol (BGP) feature. The vulnerability manifests itself when a BGP peer announces a prefix with a specific, valid but unrecognized transitive attribute. On receipt of this prefix, the Cisco IOS XR device will corrupt the attribute before sending it to the neighboring devices. Neighboring devices that receive this corrupted update may reset the BGP peering session.

Affected devices running Cisco IOS XR Software corrupt the unrecognized attribute before sending to neighboring devices, but neighboring devices may be running operating systems other than Cisco IOS XR Software and may still reset the BGP peering session after receiving the corrupted update. This is per standards defining the operation of BGP.

Cisco developed a fix that addresses this vulnerability and will be releasing free software maintenance upgrades (SMU) progressively starting 28 August 2010. This advisory will be updated accordingly as fixes become available.

Source Cisco
Click Here To Read Rest Of The Post...

Chrome and Webmynd Web Browsing Issue

Since morning, I am facing a weird issue about opening of sites in chrome explorer. If I perform any of the new search in google, I get the results but after clicking on the any of the webpage, chrome shows that "Oops! Google Chrome could not find go.webmynd.com". But at the same time internet explorer is working fine. I am thinking that google has signed up webmynd.com.

Click Here To Read Rest Of The Post...

Upgrade IOS:TCP Packet Denial of Service Vulnerability

Cisco has raised a alarm about denial of services vulnerability that may allow remote unauthenticated attacker to cause an affected device to reload. Only IOS XR and IOS XE are not affected. According to Cisco no other products are currently known to be affected. So a best is to save your downtime by fixing the issues with the help of patches. Being its friendly nature make CISCO more popular. See the list given below:-

Vulnerable devices are running an affected version of Cisco IOS Software, and are configured for any of the following:
1. A specific TCP window size
2. TCP path MTU discovery (PMTUD)
3. Stateful Network Address Translation (SNAT) with TCP as the transport protocol
For detailed information refer this link.



Click Here To Read Rest Of The Post...

VPDN is not working on SB13

A weird issue of VPDN was observed in c7200-js-mz.122-31.SB13.bin. The VPDN was working fine but during the new installation of ios in router make all the vpdn session down. Still not able to find the exact route cause analysis.

Click Here To Read Rest Of The Post...

L2TPv3 Not Coming Down

The team stuck up in imbroglio where the l2tpv3 tunnel never torn down even after shutting down of physical sub-interface. There after lot of test were made but all in vein. The problem came with the SB13 ios but the opposite side was using different vendor equipment. The problem states that when the interface was shut down, no stop ack was sent to the remote peer, it is same as keepalives used in gre tunnel to get it torn. Still we are not able to find exact RCA except the IOS bug.

Click Here To Read Rest Of The Post...

No Show Running Command

Yesterday during testing we found no show running config command on given IOS. May be a IOS bug or something else.

c1841-adventerprisek9-mz.124-24.T.bin


regards
shivlu jain
Click Here To Read Rest Of The Post...

Sparse Mode Made Clients Down

Yesterday during multicast testing we enabled pim on lan interface of cisco router as well as on serial interfaces. In the lab end to end customers were working over l2tpv3 and tunnel was establishing successfully. The moment pim sparse mode was enabled on lan interface end to end customer was not able to reach. But if the customer was conifgured as layer vpn it worked fine end to end. In layer 2 circuit l2tunnel was up but no data flow works on it. As soon as pim sparse mode was disabled from lan interface data flow started on l2 tunnel. Issues faced only with l2tpv3 protocol after enabling pim sparse mode on lan interface.

Cisco IOS used during testing:- c1841-spservicesk9-mz.123-14.YT1.bin


regards
shivlu jain
Click Here To Read Rest Of The Post...

Physical Interface Down On Deletion Of Subinterface

Yesterday when logival interface of NPE-G2 router was deleted, at the same instant physical interface got down. The ios which was using 12.2 31 SB13.

regards
shivlu jain
Click Here To Read Rest Of The Post...

Internet VRF Leaking Bug Declared

From the last one month we are looking for a internet vrf leaking solution in SB13 ios but today finally cisco declared it as a new bug CSCsy29604 which is hampering SRC and SB series.


regards
shivlu jain
Click Here To Read Rest Of The Post...

Save your network...Reliability low bug on NPE-G2

Yerterday, a cisco bug is hitting mainly service provider network. Symptoms of the bug: All frames received on gigabit ethernet interface are dropped. All drops are reported as overruns in the output of show interfaces and show controllers. This is mainly hitting Cisco 7206 NPE-G2 router.
So if the problem is being faced first checked the bug CSCsk65796.

Bug was found in 12.2SB, 12.4T and 12.4XD.

NPE-G2: all rx frames counted as overruns on built-in gige
Symptoms: All frames received on gigabit ethernet interface are dropped. All drops are reported as
overruns in the output of show interfaces and show
controllers.

Conditions: Symptom is observed on gigabit ethernet interfaces on NPE-G2 network processor of Cisco 7200 Series Routers. All IOS trains that support NPE-G2 are affected.

Workaround: There is no workaround. When the gigabit controller falls into this condition, the only way to recover is to power-cycle the router. Soft reload does not clear the problem.

Further Problem Description:

Ethernet controller goes into promiscuous mode under two conditions:
- bridging is configured on the interface
- number of MAC addresses that have to be stored in its MAC address filter
table exceed the capacity of the table.

The latter case may happen when a large number of HSRP groups is configured or a
large number of IP multicast groups are to be received on the interface.


regards
shivlu jain
Click Here To Read Rest Of The Post...

12.2 31 SB13 Internet VRF Issue..RCA

Reminiscent 5th February when we faced a issue with SB13 and consequence faced downtime for internet customers. Why I was so upset and working continously on to the problem because the IOS was being tested arduous.

Continued with my previous post of SB 13 internet vrf problem . In this post we are able to know where the problem is. Now waiting for the cisco team how they are going to announce it. The problem was reported first by us to cisco with proper findings and results.


Logs are given below:-

a) Logs taken during that time when the PE was working for both INTERNET vrf and CUST vrf with IOS SB13.
Results:- Customers were not able to access internet.

Command a.1

INTERNET_MPLS#show ip cef vrf INTERNET 0.0.0.0 0.0.0.0 internal
0.0.0.0/0, epoch 0, RIB[S], refcount 6, per-destination sharing
sources: RIB, D/N, DRH
feature space:
LFD: 0.0.0.0/0 1 local label
local label info: other/17
contains path extension list
disposition chain 0x658A91B8
IPRM: 0x00058000
subblocks:
DefNet source: 0.0.0.0/0
ifnums:
FastEthernet0/0(3): 1.1.1.1
path_list contains at least one resolved destination(s). HW not notified
path 64BAF9A4, path list 64BA36B8, share 1/1, type recursive nexthop, for IPv4, flags resolved
MPLS short path extensions: MOI flags = 0x5
recursive via 1.1.1.1[IPv4:Default], fib 64BF6884, 1 terminal fib
path 64BB01E0, path list 64BA4048, share 1/1, type adjacency prefix, for IPv4
attached to FastEthernet0/0, adjacency IP adj out of FastEthernet0/0, addr 1.1.1.1 64E9CFA0
output chain: IP adj out of FastEthernet0/0, addr 1.1.1.1 64E9CFA0

In the above output one can see the value of ifnum: Actually it is showing the outgoing interface with next hop ip address and in Output chain clearly adjacency is showing.


Command a.2

IOS With SB13
INTERNET_MPLS#show ip cef vrf CUST 0.0.0.0 0.0.0.0 internal
0.0.0.0/0, epoch 0, RIB[B], refcount 6, per-destination sharing
sources: RIB, D/N, DRH
feature space:
IPRM: 0x00018000
subblocks:
DefNet source: 0.0.0.0/0
ifnums: (none)
path_list contains at least one resolved destination(s). HW not notified
path 64BAF928, path list 64BA3628, share 1/1, type attached nexthop, for IPv4, flags must-be-labelled
nexthop 1.1.1.1 FastEthernet0/0 unusable: no label, adjacency IP adj out of FastEthernet0/0, addr 1.1.1.1 64E9CFA0
output chain: unresolved


In the above output one can see the value of ifnum: Actually it is showing none which lucidly says that no outgoing interface and in Output chain clearly no adjacency is showing only unresolved is there which means next hop adjacency is unable to built.

This is the primary reason for not working because CUST vrf is not able to know whats its outgoing interface along its adjacency table.
But is the customers direct come in INTERNET vrf then they will work becasue INTERNET vrf is having information of outgoing interface with valid next hop address and adjacency.

So the next time if you are facing the same issues try to use the above mentioned commands.


regards
shivlu jain
Click Here To Read Rest Of The Post...

12.2 31 SB13 Internet VRF Issue...Continued

Finally I got time to write findings on 12.2 31 SB12. Findings covered the problem faced in Inter VRF Leaking.

Introduction
A weird problem faced with 12.2(31)SB13 series. I made a test best in which R2 is working as upstream service provider which is providing internet services to other service providers. R1 router is another service provider router which is injecting a default route towards the R2. R1 router service provider is having MPLSVPN network and also serving internet services to the customers. R1 is having two number of vrfs one is INTERNET and another is CUST. RT of INTERNET vrf is imported in CUST vrf so that CUST vrf is able to access the internet cloud. But when a ping is initiated from R0 which is working as CE I found the given results:-



Results After Testing (See Figure 1 For Setup)Results of CE Ping
CE# ping 4.2.2.2 source loopback 0
Sending 5, 100-byte ICMP Echos to 4.2.2.2, timeout is 2 seconds:
Packet sent with a source address of 3.1.1.1
.....
Success rate is 0 percent (0/5)

INTERNET_MPLS# show ip bgp vpnv4 vrf CUST
Network Next Hop Metric LocPrf Weight Path
Route Distinguisher: 1:1 (default for vrf CUST)
*> 0.0.0.0 1.1.1.1 0 32768 i
*> 2.2.2.0/30 0.0.0.0 0 32768 ?

INTERNET_MPLS# show mpls forwarding-table vrf INTERNET 0.0.0.0
Local Outgoing Prefix Bytes Label Outgoing Next Hop
Label Label or VC or Tunnel Id Switched interface
None No Label 0.0.0.0/32[V] 0 aggr-punt/INTERNET


Customer is not able to reach to the internet. Thereafter for test my scenario I simple remove the INTERNET RT from CUST vrf and leak the default route in CUST vrf instead of vrf INTERNET.

Given Route Added
ip route vrf CUST 0.0.0.0 0.0.0.0 1.1.1.1 global

CE# ping 4.2.2.2 source loopback 0
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 4.2.2.2, timeout is 2 seconds:
Packet sent with a source address of 3.1.1.1
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 116/315/544 ms

Note:- Need to add 0.0.0.0 with the help of network command under address family of vrf else the route won’t come in the vrf routing table because the next hop is available in global routing table not in vrf table.

INTERNET_MPLS# show ip bgp vpnv4 vrf CUST
Network Next Hop Metric LocPrf Weight Path
Route Distinguisher: 1:1 (default for vrf CUST)
*> 0.0.0.0 1.1.1.1 0 32768 i
*> 2.2.2.0/30 0.0.0.0 0 32768 ?

INTERNET_MPLS# show mpls forwarding-table vrf CUST 0.0.0.0
Local Outgoing Prefix Bytes Label Outgoing Next Hop
Label Label or VC or Tunnel Id Switched interface
None No Label 0.0.0.0/32[V] 0 aggr-punt/CUST


IOS Changed To 12.4 15 T1



Now I changed the IOS of R1 to 12.4 15 T1 and was able to serve internet to esteemed customers. Below are the findings after adding 12.4 15T1 to R1

Results of CE Ping
CE# ping 4.2.2.2 source loopback 0
Sending 5, 100-byte ICMP Echos to 4.2.2.2, timeout is 2 seconds:
Packet sent with a source address of 3.1.1.1
!!!!!
Success rate is 100 percent (5/5)

INTERNET_MPLS# show ip bgp vpnv4 vrf CUST
Network Next Hop Metric LocPrf Weight Path
Route Distinguisher: 1:1 (default for vrf CUST)
*> 0.0.0.0 1.1.1.1 0 32768 i
*> 2.2.2.0/30 0.0.0.0 0 32768 ?

INTERNET_MPLS#sh mpls forwarding-table vrf CUST 0.0.0.0 0
Local Outgoing Prefix Bytes tag Outgoing Next Hop
tag tag or VC or Tunnel Id switched interface
None Untagged 0.0.0.0/0 0 Fa0/0 1.1.1.1

Findings After Adding 12.2 31 SB13a) If the same router is advertising a default route and customers vrf are coming onto the same router in that case SB 13 is not able to serve internet to customers.
b) If default route is leaked in customer vrf then customer is able to surf internet.
c) If the default route is announced on another router except SB13, In that case internet works fine. See Figure 3



What My Thought Process Says
SB13 is not able to convert vpn traffic to ip traffic on the same router. Because in my scenario customer is forwarding ip traffic and R1 is receiving in the vrf and on the same router it converts the vpnv4 traffic to ip traffic and consequence customer is not able to surf internet. But if the default route shifted to another PE and SB13 route forwards the VPNv4 traffic till that router and thereafter traffic is converted to ip traffic and everything works fine.

Workaround
Either change the IOS or shift the default route to somewhere else so that till that VPN label will be swapped and thereafter traffic will be converted to ip traffic.


regards
shivlu jain
Click Here To Read Rest Of The Post...

Weird Issue With 12.2 (31)SB13 Series:Internet VRF Leaking

In my last post (12.2 31 SB13 Internet VRF Issue) I talked about the SB series which was creating problem in term of route leaking in global table from anotehr vrf. I have tested the scenario in which if you are having default route and customer vrf on the same pe and that default route is being used by internet vrf in this case the tarffic stops flowing. I have checked the cisco bug tool to fectch the information but did not find any relevant bug which can show this. So next time if you will upgrade your router to 12.2(31)SB13 series then care should be taken if you are serving internet customers from internet vrf.

Workarounds:- Try to shift the default route along with vrf to another router so that customers can run smootly.

regards
shivlu jain
Click Here To Read Rest Of The Post...

Weird Issue With 12.2 (31)SB13 Series

Since now it is assumed that SB series is one of the trusted series for service provider network. But as per my experience this is one of the series which is having only 56 number of bugs in cisco site but actually is affected with those problem which one cannot think about. I am along with my colleages working on to the same and hopefully with in a day or two come up with the new issues.

Currently it is affected with mdt bug.

Workaround:- Simply remove the mdt configuration from vrf and add it again.

How to check:- Use the command "show ip pim mdt"

regards
shivlu jain
Click Here To Read Rest Of The Post...

Double Traceroute In MPLS

If you are getting double traceroutes in MPLS cloud then definately you are running with cisco bug CSCef16357. From one of the thread Click Here I come to know about that bug.

Cisco has clearly mentioned that this is the hardware issue of sup720 in 6500 chassis.

CSCef16357 Bug Details

MPLS/SUP720 : egress CE appears twice in traceroute

Symptoms
In a MPLS/VPN environment, egress CE is seen twice in traceroute.

Conditions
This problem is seen if both following conditions are met :
- egress PE is a cat6k with sup720
- MPLS core is hidden (no tag-switching ip propagate-ttl
is configured on
ingress PE)

N.B. Same problem is seen in a simple MPLS environment (without VPN)

Workaround
none

Further Problem Description
This is an hardware problem and cannot be fixed by software.
It impacts all software versions
Click Here To Read Rest Of The Post...

L2TP Vulnerability

On 24th september, 2008 Cisco has official announced the l2tp vulnerability. A vulnerability exists in the Cisco IOS software implementation of Layer 2 Tunneling Protocol (L2TP), which affects limited Cisco IOS software releases.
Several features enable the L2TP mgmt daemon process within Cisco IOS software, including but not limited to Layer 2 virtual private networks (L2VPN), Layer 2 Tunnel Protocol Version 3 (L2TPv3), Stack Group Bidding Protocol (SGBP) and Cisco Virtual Private Dial-Up Networks (VPDN). Once this process is enabled the device is vulnerable.
This vulnerability will result in a reload of the device when processing a specially crafted L2TP packet.

Recent Post
Click Here

Work Around
Note: L2TP implementations will need to allow UDP 1701, from trusted addresses to infrastructure addresses. This does not provide for a full mitigation as the source addresses may be spoofed.

Note: L2TPv3 over IP only implementations need to deny all UDP 1701 from anywhere to the infrastructure addresses.

Create an iACL

access-list 101 permit udp trusted-address wcm trusted-address wcm eq 1701
access-list 101 deny udp any any
access-list 101 permit 115 trusted-address wcm trusted address wcm
access-list 101 permit ip any any

As shown in picture apply access-list to fa0/0 in direction of Delhi-PE

int fa0/0
ip access-group 101 in
Click Here To Read Rest Of The Post...

Cisco IOS MPLS VPN May Leak Information

Devices running Cisco IOS versions 12.0S, 12.2, 12.3 or 12.4 and configured for Multiprotocol Label Switching (MPLS) Virtual Private Networks (VPNs) or VPN Routing and Forwarding Lite (VRF Lite) and using Border Gateway Protocol (BGP) between Customer Edge (CE) and Provider Edge (PE) devices may permit information to propagate between VPNs.

Workarounds are available to help mitigate this vulnerability.

This issue is triggered by a logic error when processing extended communities on the PE device.

This issue cannot be deterministically exploited by an attacker.

Cisco has released free software updates that address these vulnerabilities. Workarounds that mitigate these vulnerabilities are available.

Workarounds

Customers running versions of Cisco IOS that support filtering of extended communities can prevent the corruption of the route target (RT) by applying a BGP route-map that removes RT entries on inbound BGP sessions.

The following configuration example applied in the ipv4 address family of a PE device removes extended communities from the CE router:

router bgp
address-family ipv4 vrf one
neighbor
remote-as
neighbor
activate neighbor
route-map FILTER in exit-address-family
!
ip extcommunity-list 100 permit _RT.*_
!
!
route-map FILTER permit 10
set extcomm-list 100 delete
!
The following configuration example applied in the ipv6 address family of a PE device removes extended communities from the CE router:

router bgp
address-family ipv6 vrf one
neighbor
remote-as
neighbor
activate neighbor
route-map FILTER in exit-address-family
!
ip extcommunity-list 100 permit _RT.*_
!
!
route-map FILTER permit 10
set extcomm-list 100 delete
!
Note: The capability of filtering extended communities is only available in certain 12.0S and 12.2S based Cisco IOS releases.

BGP session between the PE and the CE needs to cleared to make this configuration change effective.
Click Here To Read Rest Of The Post...