Current topics in automatic incident handling for certs

Current topics in automatic incident handling for CERTs

• Current topics in automatic incident handling for CERTs

• IFAS

• HKCERT , IFAS and use-cases

• IHAP project

• ContactDB project

• Current R&D

Information Feed Analysis System

• Information Feed Analysis System

National CSIRTs need visibility on network in their economy

• National CSIRTs need visibility on network in their economy

• However, many national CSIRTs don’t operate networks themselves, and normally don’t have global (or any) direct visibility

• How does the CSIRT know what’s going on in their country?

Luckily, there are a lot of network operators, research teams, vendors, and other CSIRTs out there that collect information, and will share it with national CSIRTs.

• Luckily, there are a lot of network operators, research teams, vendors, and other CSIRTs out there that collect information, and will share it with national CSIRTs.

• And here comes the “but”...

There are many feeds, all with their own data formats and mediums:

• There are many feeds, all with their own data formats and mediums:

• Formats: CSV, JSON, XML, STIX, IODEF
• Mediums: HTML, RSS, email, HTTP APIs

• While there are efforts to standardise data formats, this will take a long time, and will likely never cover 100% of feeds

• We can’t change the format of remote feeds - we can only change what we do with the data.

Different feeds use many terms to mean the same thing:

• Different feeds use many terms to mean the same thing:

• ip, source_ip, src_ip, endpoint, attacker_ip, cnc_ip...

• If we receive events from many feeds, we need to normalise so we can compare them together.

As a national CSIRT, we’re concerned with the health of national networks: which means measurement.

• As a national CSIRT, we’re concerned with the health of national networks: which means measurement.

• We can only measure longterm if we store events, enabling us to analyse them.

• We also want to search through events, like:

• C&C servers in domestic networks in last week
• Bots infected with Trojan.abc on BigISP
• Defaced web sites targeting gov.zz

There’s way too much network event data out there to manually process

• There’s way too much network event data out there to manually process

• Options:

• a) use lots of analyst time doing tedious log processing
• b) write lots of small, independent scripts
• c) ignore inbound logs completely
• d) use an automated processing system

We need something which automatically:

• We need something which automatically:

• Gathers many different types of feeds
• Normalises the data in those feeds
• Stores that data somewhere
• Allows search and performs statistical analysis

IFAS = Information Feed Analysis System

• IFAS = Information Feed Analysis System

• Project sponsored by HKCERT and developed by HKCERT and CSIRT Foundry

• An integration of open source tools, released as open source for CSIRTs

Abusehelper: gather, process, and enrich feeds, generate events

• Abusehelper: gather, process, and enrich feeds, generate events

• Logstash: process and normalise feeds

• Elasticsearch: store events in schema-free index server

• Kibana: search through events

• IFAS Reporter: get overall statistics, build realtime dashboards

Visualize information

• Visualize information

Open source under Apache 2.0 License

• Open source under Apache 2.0 License

• Only possible with the hard work released under open source licenses from Abusehelper and Elasticsearch teams

• Contributions, bug reports, feature requests most welcome!

Production: 8-16GB memory machine

• Production: 8-16GB memory machine

• Dev: 4GB possible

• Multi-core machine (4+ ideal)

• Runs in a VM no problem

Currently under closed pilot to trusted CSIRTs

• Currently under closed pilot to trusted CSIRTs

• Eventually public release

• Please contact [e-mail] for details

IFAS = Information Feed Analysis System

• IFAS = Information Feed Analysis System

• Project sponsored by HKCERT and developed by HKCERT and CSIRT Foundry

• An integration of open source tools, released as open source for CSIRTs

Abusehelper: gather, process, and enrich feeds, generate events

• Abusehelper: gather, process, and enrich feeds, generate events

• Logstash: process and normalise feeds

• Elasticsearch: store events in schema-free index server

• Kibana: search through events

• IFAS Reporter: get overall statistics, build realtime dashboards; charts based on D3.js

Collaborate of global intelligences

• Collaborate of global intelligences

• No sensor installation required
• Assimilate different input formats / standards
• Normalize field names
• Collect and normalize information 24x7

• Situational Awareness

• Alerts
• Public Awareness
• Tracking Trends

Actionable

• Actionable

• Proactive Discovery
• Quality information for incident handling
• Automate Incident Response

• Engagement Tool

• Engage top ISPs to cooperate
• Cooperate on large scale clean up (bots)

• Standardization

• Standardize as measurement

Total governance on data

• Total governance on data

• Built on open source tools that has good community

• Abusehelper, Elasticsearch, D3.js

• Low hurdle to use and customize

• Open source under Apache 2.0 License
• Free of charge
• UTF Support
• Support JSON/CSV import & export
• Modular
• add plug-ins for new feeds
• add output chart plug-ins using D3.js

Machine with 8GB+ & Multi-core CPU

• Machine with 8GB+ & Multi-core CPU

• Ubuntu Server (can run in VM)

• Bitbucket Account (code repository)

• 30 minutes of installation and configuration

Powerful search on all the information collected

• Powerful search on all the information collected

Statistical analysis-Trends & Distributions

• Statistical analysis-Trends & Distributions

• Free form statistical reports

Set tracking criteria – get notify ASAP

• Set tracking criteria – get notify ASAP

• domain: *.gov.hk
• Alert lists : educational institutions (hkedu), NGOs (hkorg)

Correlate Cryptolocker 2013-Oct with Zeus

• Correlate Cryptolocker 2013-Oct with Zeus

Data do help HKCERT engaging ISPs (their sales team)

• Data do help HKCERT engaging ISPs (their sales team)

• Data do help a server hosting SP understand their customers’ security problems

Defacement

• Defacement

• Phishing

•  Export to CSV for batch processing, with some other scripts

• Malware hosting – a bit difficult

• Large volume of incidents – need prioritisation

All you can use

• All you can use

• All you can contribute

• Add input filters for new feeds
• Add new plug-in modules
• Add new chart and visualization
• Integrate with other systems, e.g. RTIR
• …

• Standard language: STIX, taxonomy of ENISA

An ongoing project that turn security events into Actionable Data

• An ongoing project that turn security events into Actionable Data

• Set Priority, Choose Monitors, Consolidate Results

Incident handling automation project

• Incident handling automation project

Very similar to IFAS, developed in parallel by CERT.pt, CERT.at

• Very similar to IFAS, developed in parallel by CERT.pt, CERT.at

• Also uses Logstash, Elastic Search and Abusehelper

• Less work on the Webinterface, more work on Ontology, „Data harmonisation document“

Discussions about CERT.AT developments/documents

• Discussions about CERT.AT developments/documents

• Discussions about cooperation between CERTs

• ENISA support

Open Source

• Open Source

• Maintainable

• Flexible and Modular - must be possible to integrate existing software and modules (Pastemon, AbuseHelper, etc..)

• Reusable

• Easily Extendable - should require little knowledge and basic programming skills

• Easily Deployable

• Easily Updatable – easy to share new developments with other CERTs and update the system with that new code

• Easily Configurable - config files that can be easily modified to fit CERT‘s needs

• Documented - must be well documented

• http://www.enisa.europa.eu/activities/cert/support/incident-handling-automation

https://bitbucket.org/clarifiednetworks/abusehelper/wiki/Data%20Harmonization%20Ontology

• https://bitbucket.org/clarifiednetworks/abusehelper/wiki/Data%20Harmonization%20Ontology

• A standard set of well defined field names within Abusehelper (AH)

• Allows CERTs to:

• Write bots which are interoperable within AH
• Measure in identical ways
• Easier to parse different feeds („generic santizer bot“) : you just have to define the mappings

abuse@ lookups suck (IRT object not in use, no standard; Just now RIPE DB is changing with abuse-c:)

• abuse@ lookups suck (IRT object not in use, no standard; Just now RIPE DB is changing with abuse-c:)

• Getting the right lookup is non-trivial, complex

• Many (national) CERTs create their own abuse contact lookup DBs.

• National CERT DB, TI directory, FIRST data can not be looked up automatically via scripts.

A caching contact database with more specific internal data

• A caching contact database with more specific internal data

• Some of this data (tel nos, etc) will never be in the public whois

• Unify with TI, FIRST etc data

• Make it query-able by scripts

What databases exist? What can we query?

• What databases exist? What can we query?

Public code repo ;-)

• Public code repo ;-)

• Whois server (thx Mauro)

• RESTful API (Mauro, Rafiot)

• Some scripts to import TI data (Aaron, David)

• Still some bugs ;-)

Document (WIP):

• Document (WIP):

• https://github.com/certtools/contactdb/blob/master/doc/contact-databases-for-abuse-handling.mkd

• Codebase: https://github.com/certtools/contactdb

• (thx Rafiot, David, Mauro!)

AH Architecture showed the way… but … it’s complex

• AH Architecture showed the way… but … it’s complex

• Current experiments underway with RabbitMQ, Redis, logstash + Elastic Search

• Promising results so far, easy to use, more testing needed

• Problem: re-implement all parsers??

The CERT community has limited ressources for development

• The CERT community has limited ressources for development

• We re-implement the same thing all the time

• Let‘s share code or at least exchange ideas on how to automate incident handling!

• Let‘s share on how to measure success

• Thanks HKCERT, ENISA, CERT.at, CERT.pt, CIRCL, etc..

• Mailinglist: https://tiss.trusted-introducer.org/mailman/listinfo/ihap