8: 40am Josh Ballard warns sirt about global increase in scanning for vulnerable Trend Micro ServerProtect (tmsp) instances



Yüklə 537 b.
tarix08.10.2017
ölçüsü537 b.



8:40am - Josh Ballard warns SIRT about global increase in scanning for vulnerable Trend Micro ServerProtect (TMSP) instances

  • 8:40am - Josh Ballard warns SIRT about global increase in scanning for vulnerable Trend Micro ServerProtect (TMSP) instances

    • Recommends making sure TMSP is patched.
    • Reports some inbound traffic on TCP port 5168
    • Refers to SANS Internet Storm Center report from that morning
    • SANS reports says “It does indeed look like machines are getting owned with this vulnerability” (actually in Aug. 22 diary)
  • 10:16am - Seth Galitzer reports odd behavior of 3 Windows servers that started the afternoon of Aug. 22, including the “blue screen of death” (BSOD) and continual rebooting



Seth tried backing out Aug. 14 Windows patches. Didn’t help

  • Seth tried backing out Aug. 14 Windows patches. Didn’t help

  • 11:39am – Seth associates reboots with a crash of spntsvc.exe – Trend Micro ServerProtect service; Disabled the service and systems stabilized. Other servers with ServerProtect not affected.

  • 1:42pm – Harvard speculates that Seth’s problem and Josh’s announcement are related; Points out that Trend Micro has a patch to fix the vulnerability in TMSP



8:44am – Marin Dowlin reports same problem with a server

  • 8:44am – Marin Dowlin reports same problem with a server

  • 8:53am – another one reported in Biochemistry

  • 9:41am – Seth reports TMSP patches won’t install

  • 10:18am – Shea McGrew recommends removing TMSP and upgrading to OfficeScan 8.0 if TMSP patches fail.

  • 3:28pm – Brandon Utech joins the chorus; reported two servers rebooted with the same error immediately after a ServerProtect-initiated scan started; provided detailed analysis that pointed to TMSP as the culprit

  • 4:55pm – sporadic network outages start, continued until nearly midnight



12:50pm – more network problems reported

  • 12:50pm – more network problems reported

  • 3:06pm – Redundant campus Internet connection disabled to stabilize the network

  • 7:21pm – A server is compromised a second time from a different source; uploads malware to the server



8:44am – Dereatha Cross reports no problems with Windows servers running Trend Micro OfficeScan, which does not have the same vulnerability

  • 8:44am – Dereatha Cross reports no problems with Windows servers running Trend Micro OfficeScan, which does not have the same vulnerability

  • Josh identifies four compromised servers that launched a Distributed Denial of Service Attack; network access for these servers is blocked

  • The DDoS packets failed to leave the campus since the source IP addresses were spoofed

  • Josh begins analyzing network flow data to look for other compromised servers



Distributed Denial of Service Attack (DDoS)

  • Distributed Denial of Service Attack (DDoS)

    • Multiple compromised systems flood the bandwidth or resources of a targeted system, usually one or more web servers (i.e., they deny the ability of the target to provide the intended service)
    • When the source is hundreds, maybe thousands of computers all over the world, is very difficult to stop


Source IP address identifies the computer where the network packet originated

  • Source IP address identifies the computer where the network packet originated

  • All K-State IP addresses start with 129.130 (for example, 129.130.12.18)

  • A “spoofed” source IP address re-writes the data packet header with a false address to hide its origin

  • Any packet with a source IP address NOT starting with 129.130 is discarded at the campus border

  • Thus the DDoS attack flooded our campus network with traffic, but not the intended target (trendmicro.com web server)



Cisco routers collect information about IP network traffic

  • Cisco routers collect information about IP network traffic

  • “Flow” = a unidirectional sequence of packets all sharing these 5 values:

    • Source + destination IP address
    • Source + destination port
    • IP protocol (IP, TCP, UDP, ICMP, etc.)


Timestamps for the flow start and finish time

  • Timestamps for the flow start and finish time

  • Number of bytes and packets observed in the flow

  • Layer 3 headers:

    • Source & destination IP addresses
    • Source and destination port numbers
    • IP protocol
    • Type of Service (ToS) value
  • Nothing about packet payload (content) recorded (i.e., we’re not snooping)





3:08pm – Harvard discusses strategy with SIRT; still some uncertainty and not much information in security community about this exploit; have not been able to confirm how our four servers were compromised

  • 3:08pm – Harvard discusses strategy with SIRT; still some uncertainty and not much information in security community about this exploit; have not been able to confirm how our four servers were compromised

  • 3:33pm – Bryan Boutz reports that one of the compromised servers had the ServerProtect patch applied on Thursday, Aug. 23, but he had to reinstall TMSP for patch to take

  • 4:06pm – Harvard indicates that activity on port 5168 started on Wednesday, Aug. 22, so his server was likely compromised the day before he patched it.



4:23pm – From flow data, Josh determines that the DDoS targeted trendmicro.com, further evidence that the servers were compromised by the TMSP vulnerability

  • 4:23pm – From flow data, Josh determines that the DDoS targeted trendmicro.com, further evidence that the servers were compromised by the TMSP vulnerability

  • 4:26pm – Harvard sends urgent warning to campus about four compromised servers, impending block of port TCP/5168 at the campus border, urges sysadmins to remove ServerProtect and upgrade to OfficeScan 8.0

  • 4:48pm – Bryan Boutz updated TM pattern file, scanned the compromised server, and the backdoor was identified and removed.



Peace and quiet returns to Oz

  • Peace and quiet returns to Oz

  • Wednesday, August 29

  • TCP port 5168 blocked at the border



June 2007 – Trend Micro privately informed of the vulnerability

  • June 2007 – Trend Micro privately informed of the vulnerability

  • July 27 – Trend Micro quietly releases patch, which is not part of automatic pattern file updates; has to be manually applied

  • Aug. 22 – Trend Micro announces the vulnerability and the patch since exploits are now occurring in the wild

  • Aug. 22 – SANS Internet Storm Center reports increased activity on port 5168 on the Internet

  • Aug. 22, 4:08pm – first server compromised

  • Aug. 22-23 – 10 servers compromised at K-State

  • Aug. 23 – first speculation that the rebooting servers related to TMSP vulnerability; sysadmins start patching ServerProtect, which prevents further spread



Aug. 23 – Trend Micro pattern file able to detect/clean malware from this exploit

  • Aug. 23 – Trend Micro pattern file able to detect/clean malware from this exploit

  • Aug. 25 – Dormant malware on four compromised server awakened by remote control via IRC to launch DDoS attack on trendmicro.com, causing network problems; DDoS unsuccessful because of K-State’s border protection

  • Aug. 25 – One server compromised a second time

  • Aug. 27 – Four compromised servers blocked; netflow analysis confirms source of compromise is exploited vulnerability in ServerProtect.

  • Aug. 27 – urgent call to patch/upgrade issued

  • Aug. 28 – Netflow analysis completed, revealing a total of 10 compromised servers that are blocked

  • Aug. 29 – blocked port TCP/5168 at the border



Defn: Attack launched either before or on the same day that a vulnerability is announced or discovered

  • Defn: Attack launched either before or on the same day that a vulnerability is announced or discovered

  • In our case, the vulnerability and patch were publicized on Aug. 22, the same day K-State servers were compromised

  • However, the patch was actually released on July 27



Josh is a netflow wizard and master analyst!

  • Josh is a netflow wizard and master analyst!

  • Critically important for the campus community to work together in response to threats (collective wisdom wins again!)

  • Communication critical to coordinated, appropriate response

  • Should have blocked 5168 at the border sooner, but it would not have prevented the infections

    • Except one case where a server was compromised twice via port 5168 from two different sources; the second
  • Zero-day exploits are very, very difficult to defend against

  • An aggressive “default deny” host-based firewall config would have prevented infection (i.e., only allow specific hosts to connect to port 5168)



Our normal IRC botnet detection missed this because the botnet controller was a new one (it’s now in the list of known controllers!)

  • Our normal IRC botnet detection missed this because the botnet controller was a new one (it’s now in the list of known controllers!)

  • Was a stealth botnet, so very difficult to detect until it launched the DDoS

    • No extra processes running
    • Injected its code in svchost.exe process so it wasn’t noticed
    • Log of IRC commands in memory, not on disk
  • It is sometimes difficult to correlate anomalous behavior with an exploit/attack

  • Even though the Trend Micro pattern file detected the malware on Aug. 23, it was installed on Aug. 22 so it might not get detected until the next scheduled scan on Wed. Aug. 29.

  • Is it time to recommend daily scheduled scans? Beware of the performance impact on a server



This could and has happened to other anti-virus software products, like Symantec. Trend Micro products are still an effective and important part of our security arsenal.

  • This could and has happened to other anti-virus software products, like Symantec. Trend Micro products are still an effective and important part of our security arsenal.

  • Ironic that security software (Trend Micro ServerProtect) was the source of the compromise, but not surprising since security software operates deep within the OS. It is therefore an attractive target for hackers.

  • Interesting connection between exploit of Trend Micro software that was then used to attack trendmicro.com

  • Hackers are now targeting applications as much as operating systems, compounding the challenge of keeping systems secure



The challenge of patching as a preventative measure

  • The challenge of patching as a preventative measure

    • Are hundreds of vulnerabilities announced each week and hundreds of patches in dozens of different applications and OSes
    • Zero-day exploits are relatively rare
    • Some patches break things
    • Best practice is to test patches thoroughly before deploying in production, but that’s not practical now
    • Yet, a vulnerable system is a substantial risk
    • So what’s a nerd to do?!?!


Aggressive firewalls

  • Aggressive firewalls

  • Regularly scan for open ports/services, and evaluate their necessity

  • Automated patching of as much as possible

  • Consider a daily TMOS scheduled scan (again, beware of performance impact)

  • More IT security staff to:



The threat is real

  • The threat is real

  • We are living dangerously







Dostları ilə paylaş:


Verilənlər bazası müəlliflik hüququ ilə müdafiə olunur ©genderi.org 2019
rəhbərliyinə müraciət

    Ana səhifə