How Race Communications Leveraged Kentik to Stop Mirai Botnet Infection and Abuse

The Mirai botnet was first discovered back in 2016 but has continued to persist and abuse common vulnerabilities and exposures (CVEs) on IoT devices, including home routers and many other network-connected devices. In short, the network of bots was built by malicious actors who exploited remote access and control protocol ports over many different device types, producing damaging traffic levels and creating an advanced, powerful tool that can be used for large-scale DDoS attacks and many nefarious purposes.

When Kentik customer, Race Communications, a provider of reliable, high-speed internet and advanced communications to communities throughout California), learned of Mirai’s potential risk to its customers, the team knew it needed to act fast. In this blog post, we outline how Race Communications was able to leverage Kentik’s powerful network analytics to identify malicious traffic associated with Mirai, determine which of Race Communications’ customer IP addresses were being used by the botnet, and ultimately, save its online IP reputation.

The Race to Stop Mirai
Race Communications was alerted to the potential Mirai risk when the team received a letter from another network online noting that IP address(es) that Race Communications owned were acting maliciously over the internet. These addresses were apparently port scanning IP addresses that belonged to the company sending the letter. The letter took on the form of a formal complaint and asked that Race Communications cease this activity.

While the port scans were only coming from a few hosts, Race Communications noticed that the complaint was against an entire /24 IP block. This had the potential to lead to the entire /24 block getting blacklisted for malicious activity. This type of blacklisting could potentially cause other Race Communications customers to experience a loss of connectivity to services, due to poor IP reputation.

Race Communications knew that by utilizing network forensics capabilities from Kentik, the team would be able to quickly drill down into the incident and determine the root cause behind the formal complaint.

Network Forensics & Visibility from Kentik
The Race Communications network team turned to Kentik’s “Unique Destination Port” metric and was quickly able to see how many ports the address listed in the formal complaint might be hitting, and why it would be considered a scan. When this number revealed only 40 destination ports on average, the team again turned to the insights available with the Kentik platform.

Checking the pattern of all ports that were being utilized, Race Communications discovered that the vast majority of all destination ports were either port 23 (Telnet), or port 37215 (Huawei Remote Procedure Port). The Huawei port was of immediate interest to the network team as this port is part of a long-lived exploit cataloged as CVE-2017-17215. The exploit only requires a single authentication to work, and once it had been exploited, remote code execution is possible on the associated device. (Vuldb has additional information on the life of this exploit.)

The final port that Race Communications observed was port 2323, which was a tiny fraction of the total traffic, and always sourced from port 23 on protocol TCP. This is a potential sign of a Mirai variant botnet C&C traffic, as described in this article, and is additionally associated with CVE-2016-10401; an exploit associated with ZyXEL network devices and escalation of permissions.    The method of exploiting both the Huawei CVE and the ZyXEL are very similar, requiring one authentication first.

In just a three-hour period, Race Communications was able to see a very large Unique Destination IP count-per-port. As seen in the image below, this rate was highly consistent.

Race Communications was also able to leverage the Destination Port information from Kentik to pinpoint backtrace several additional Source IPs on its customers’ networks that were participating in similar traffic patterns.

During this portion of the investigation, while working through the traffic, Kentik and the Race Communications network team found additional interesting and actionable information within the Kentik platform:

  • Most destination 37215 ports were hitting IP addresses in the Asian market regions, such as China, Japan, and South Korea. To the team, this was a potential indicator that the attackers were targeting addresses within a region more likely to contain a Huawei product.
  • Destination port 23 traffic was hitting an evenly distributed traffic pattern between the Asian markets and the United States. To the Race Communications team, this had the potential to imply that the attacker(s) were searching for new devices using Telnet, aiming to find vulnerable Huawei devices in the process, and scripting to additionally hit these devices with a vendor-specific attack.
  • Each Source IP that was on its customers’ networks was reaching an hourly average of 3,500 Unique Destination IPs consistently, for weeks. Each Unique Destination is another potential report, and another hit against this Race Communications’s customer IP reputation.

An Interesting Forensic Finding
Perhaps the most interesting finding for Race Communications was that Kentik’s Spamhaus Botnet and Threat-List data feeds tagged all of the offending IPs that Race Communications had identified manually. Additionally, Kentik’s platform highlighted other suspicious activity among Race Communications’ customer subscriber IPs. This meant that there had been significantly more complaints against its customer IP blocks than Race Communications was initially aware of from the single, formal complaint the team received.

The Race Communications team knew it was not sufficient to wait for another entity online to formally complain. With insights from Kentik, the team could act proactively in order to detect malicious subscriber traffic heading to the larger internet.

Automated Detection
Kentik was able to offer Race Communications with automation in detection, by creating Kentik DDoS-type policies with modifications. Kentik will now alert the provider whenever a subscriber IP communicating outbound via ports 23 and 32715 reaches more than 500 unique destinations.

These insights give Race Communications the option to proactive notify internal operational groups of events that could impact IP reputation in the long term. Additionally, Race Communications can now contact their customers to alert them of a possible infection on one of their Internet-connected devices.

In addition to Kentik’s platform scrubbing partners Radware and A10 Networks and our BGP-based Remote Trigger Black Hole (RTBH), Kentik now offers BGP FlowSpec mitigations. FlowSpec can be utilized to mitigate traffic with low-to-no collateral effects versus RTBH. It excels in situations where non-volumetric attacks like the ones associated with the Mirai botnet are compromising commercial security and reputation.

Summary
With Kentik, Race Communications can protect its business and its online IP reputation by unlocking:

  • Information needed to remediate not just the reported problem, but the entire detectable problem
  • A path by which to contact subscribers and explain specifically why they have been the subject of a complaint (e.g. if they have infected machines on their network, response and action can quickly take place)
  • An automated way of being informed of new IP addresses that form the Mirai pattern or other botnet traffic patterns
  • Faster mean time to diagnose (MTTD)
  • Faster mean time to repair (MTTR)

Kentik is able to offer insights that can help identify almost any malicious or unwanted traffic on any network and provides automatic notification and mitigation capabilities. For more information on network visibility from Kentik, reach out to us or request a demo.

Race wins 2nd place at annual IEEP “Red Tape to Red Carpet” awards.

On Wednesday, November 7th the Inland Empire Economic Partnership hosted their sixth annual “Turning Red Tape to Red Carpet” awards ceremony. Race Communications was nominated for the “Smart City/Smart County” award alongside OntarioNet and Connect Anza. This award recognizes cities and counties that have taken innovative approaches to provide reliable high-speed internet to their residents and businesses.

ieep

Race has been in business for over 20 years and for the past 9 years we have been working in parallel with the California Public Utilities Commission (CPUC) to provide affordable gigabit fiber internet to rural unserved and underserved communities across California. We would not be able to do this without the California Advanced Services Fund (CASF).

“Gigafy Phelan” is our tenth grant project – and also one of our largest, covering 98 square miles. Once fully built, our network will provide fiber to the home gigabit internet to over 7,600 homes and businesses. To date, over 600 residents and business owners have been installed.

Since we began providing services in May, we have seen several positive social and economic improvements. Residents are now pursuing online higher education and working to complete degrees, local community groups are forming science and technology-based learning groups and advancing digital literacy, and homes and businesses are implementing the use of wireless security systems in crime-prone areas.

We want to thank the Inland Empire Economic Partnership for recognizing our efforts in the community as well as the staff at Race and the community of Phelan. This project would not be the success that it is without either.

Connect Anza took home first place for their efforts in Anza and the surrounding areas – The first strands of fiber were hung in July 2015 and bringing high-speed Internet has been a welcome change. The system is still being built, but now encompasses most of their cooperatives service territory in the Anza and Aguanga areas. Construction to serve the eastern areas of the cooperative will begin soon.

OntarioNet claimed 3rd place. The project, which is powered by Inyo Networks, will provide high-speed Internet, Ethernet, and Voice over IP services to the City of Ontario and its businesses.  All services are using a crystal clear, all-fiber network.

National Night Out 2018

National Night Out is an annual community-building campaign that promotes police-community partnerships and neighborhood camaraderie to make our neighborhoods safer, more caring places to live. Neighborhoods host block parties, festivals, parades, cookouts and various other community events with safety demonstrations, seminars, youth events, visits from emergency personnel, exhibits and much, much more.

This year, Race attended National Night Out in Stallion Springs and Bear Valley Springs while sponsoring National Night Out in Playa Vista. Both Stallion Springs and Bear Valley Springs had dunk tanks – BVS’ was hosted by Race and Race also had a water booth with games for the children in Stallion Springs.

The Race team enjoyed the beautiful weather and getting to know the members of the communities they serve. Residents who have Race services gave their feedback and those who didn’t have service through Race expressed their desire for better broadband in their neighborhoods.

The evening in Stallion Springs finished with a raffle where two lucky kids won bikes that were sponsored by Race. In Bear Valley Springs the line at the dunk tank never slowed down even the Assistant GM of the Bear Valley Springs Association, Cheramy Krueger got dunked!

The Race team looks forward to coming back next year!

MikroTik Security Vulnerability

Dear RACE Valued Customer,

Please be advised,

There is a MikroTik security issue:
MikroTik RouterBoard V-6.38.5 Denial Of Service | CPU Consumption

A vulnerability in the network stack of MikroTik Version 6.38.5 released 2017-03-09 could allow an unauthenticated remote attacker to exhaust all available CPU via a flood of TCP RST packets, preventing the affected router from accepting new TCP connections. 

***If you do not know what a MikroTik is, chances are you don’t have one, so please do not be alarmed***

Recommendation:
Check you Mikrotik  OS Version and update it to be newer than V-6.38.5.
We recommend you to keep it up to date.

If you have any further questions or concerns, please email us at support@race.com or click here and select support.

Thank you for your business and continued support,

RACE Communications Support Team