路 Brown Walrus
We kept trying, and the fix that shouldn't have worked did
A client lost power for a moment, and when it came back, their internet stayed down. We kept testing, and the fix was something that shouldn't have changed anything. It got the client back online without waiting on Rogers, whose own fix still hadn't arrived a week later.
Nothing on their network could reach the outside world, and that included the guest Wi-Fi. The client's fibre service comes from Rogers, and so does the router that connects their office to it, a Cisco ASR 920. Rogers provides and manages it; the client didn't choose it.
The hunt
We spent many hours in that rack. We went through it piece by piece, testing each part and ruling things out.
The trail ended at the ASR 920. The router's LAN port, the one that connects to the client's own network, had no connection and no link light (the small light beside a port that shows the connection is live). With that port dark, the client's whole network had no road out.
Knowing the cause helped less than you'd hope, since the router is Rogers' to manage.
The twist
We could have stopped there and waited on Rogers. We kept trying things. We plugged an SFP module, a small swappable part that adds a network connection to a router, into an unused port on the router. The module had nothing to do with the client's connection. The LAN port's link light came on, and all internet service came back.
We can't prove why that worked. Our best guess is that adding the module prompted the router's software to check its ports again, and the stuck LAN port came up during that second look. Until Rogers tells us otherwise, we're filing it under "it works, don't touch it."
We would have laughed if you'd told us the fix would be an unrelated part in an empty port.
Act two: the guest Wi-Fi
With the internet back, we turned to the guest Wi-Fi. The client runs it on Cisco Meraki equipment (the Merakis), which you manage from Meraki's online dashboard. Meraki's documentation says each device must be able to reach Meraki's online service over a secure connection that the device opens itself.
Over the Rogers fibre, the Merakis could not get through, for reasons we still don't know, even though the rest of the network could by then.
So our technician improvised. They connected a laptop to an iPhone's hotspot, shared that connection through the laptop's network cable port, and plugged the Merakis into it. An iPhone stood in for a fibre line.
The Merakis reached Meraki's online service over that clean connection and downloaded their settings. We plugged them back into the Rogers connection, and they worked 100 per cent.
A week later
A week on, we're still waiting for Rogers to send the router a settings fix. Until it arrives, we can't say how the ASR 920 will handle the next power outage, so in the meantime we're hoping the power stays on.
What to take from this
- A short power outage can leave equipment in a bad state. That includes carrier equipment, the gear your internet provider installs and owns. Our client's outage lasted far longer than the power blip that caused it.
- Check the link lights first. A dark port tells you where the break is. In our case, one missing light on the router's LAN port explained the whole outage.
- Keep a way online that doesn't depend on your main internet line. A phone hotspot got the Merakis their settings when the main line couldn't. Know which laptop can share its connection through a network cable before you need it.
- Put the carrier router and your main network equipment on a battery backup (UPS). It gives your network a buffer against the next blip.
- When the real fix belongs to the carrier, open the ticket and keep it open. A workaround that holds today still needs the proper fix behind it.
- Choose IT support that keeps going after the first checks. Ask a provider what they do when those checks don't find a fix.
If your network has a story like this one, or you'd rather it never did, our managed IT team can help. Call us at 780-540-7606.
Frequently asked questions
Why is my internet not working after a power outage?
Equipment that loses power can come back in a bad state, and that includes the carrier's router. Start with the link lights (the small lights beside each port that show a live connection) on every connection between your internet provider's equipment and your own. In our case, the port on the Rogers router that connects to our client's network had no link light, and that one dark port took our client's whole network offline. If the dark port is on your provider's equipment, call your internet provider and open a support ticket.
What is a Cisco ASR 920?
The ASR 920 Series Aggregation Services Router is a line of Cisco routers. Cisco describes it as built for remote access and for smaller sites where network connections are gathered together. Cisco's own documents write it both ASR 920 and ASR920. On our client's fibre service, the ASR 920 comes from Rogers and Rogers manages it, which is why the permanent configuration fix has to come from Rogers.
What does it mean when a Meraki device has never connected to the Meraki cloud?
Meraki's documentation says the device has never reached Meraki's online management service, even though it is assigned to your account. Meraki suggests confirming the device's network cable connects to working equipment and checking the colour of its status light. Meraki devices have to open their own secure connection out to Meraki's service, so whatever sits between them and the internet has to let that through.
Why can't my Meraki access point reach the cloud?
Meraki devices must be able to communicate with Meraki's online service over a secure connection that the device opens itself, so anything blocking that path keeps them offline. In our case, we never learned why the Merakis could not get through over the Rogers fibre. We connected them to a clean connection, shared from an iPhone hotspot through a laptop's network cable port, and they downloaded their settings.