Initializing your environment¶

Installation as PyMISP user¶

The quick and dirty way:

sudo pip3 install pymisp

The clean approach as user:

pip3 install --user pymisp

Installation as PyMISP developer (recommended for this session)¶

git clone https://github.com/MISP/PyMISP.git

cd PyMISP

virtualenv -p python3 pymisp-env
source pymisp-env/bin/activate

pip install -e .

Setting up of jupyter¶

We assume you're in a virtual environment

If you want to follow along this workshop on your computer, this is the way to go:

pip3 install jupyter
cd docs/tutorial
jupyter-notebook

Using the PyMISP objects¶

This page aims to give recommendations about how to efficiently use the pymisp library.

It is strongly recommended (read "don't do anything else, please") to use the library this way and never, ever modify the python dictionary you get by loading the json blob you receive from the server.

This library is made in a way to hide as much as the complexity as possible and we're happy to improve it is there is something missing.

MISPEvent¶

MISPEvent is the main class to use when you want to create/update events on a MISP instance.

In [ ]:

from pymisp import MISPEvent

event = MISPEvent()

event.info = 'This is my new MISP event'  # Required
event.distribution = 0  # Optional, defaults to MISP.default_event_distribution in MISP config
event.threat_level_id = 2  # Optional, defaults to MISP.default_event_threat_level in MISP config
event.analysis = 1  # Optional, defaults to 0 (initial analysis)

print(event.to_json())

Tag Event¶

First example of helper aiming to make your life easier.

In [ ]:

event.add_tag('tlp:white')

print(event.to_json())

Set the Event date¶

The date can be in many different formats. This helper makes sure it normalizes it in a way that will be understood by your MISP instance.

In [ ]:

# As text
event.set_date('2018-04-13')
print('Simple', event.date)

# Some weird text format (anything supported by dateparse will work)
event.set_date('Sat Oct 11 00:13:46 2017')
print('Messy', event.date)

# datetime.date
from datetime import date
d = date.today()
print(type(d))
event.set_date(d)
print(event.date)

# datetime.datetime => MISP expects a day, so the hour will be dropped.
from datetime import datetime
d = datetime.now()
print(type(d))
event.set_date(d)
print(event.date)

Add Attribute to event¶

More useful things: adding attributes to an event.

Attributes have a bunch of parameters you can pass (if you feel like it). If you don't pass them, they'll be automatically set depending on their sane defaults.

The parameters are the following:

type (required)
value (required)
category: see default
to_ids: see default
distribution: defaults to inherit from parent (event or object)
disable_correlation: true for a normal attribute, fallback to the value defined in the template object if relevant
data: only for malware-sample or attachment, BytesIO object of the file. If it is a malware, the sample is decrypted in memory

In [ ]:

attribute = event.add_attribute('ip-dst', '8.8.8.8')  # Minimal parameters

print(type(attribute))

print(attribute.to_json())

Set parameters (inline)¶

This is the way to pass other parameters

In [ ]:

attribute_second = event.add_attribute('ip-dst', '8.8.8.9', disable_correlation=True)

print(attribute_second.to_json())

Modify existing attribute¶

Every parameter can be modified in a pythonic way.

In [ ]:

attribute.to_ids = False

print(attribute.to_json())

Tag Attribute¶

In [ ]:

# Using the list of attributes in the event
event.attributes[0].add_tag('tlp:green')

# ... or the variable we got from `add_attribute`
attribute_second.add_tag('tlp:amber')

print(attribute_second.to_json())

In [ ]:

print(event.to_json())

Soft delete attribute¶

Important note: the default approach to delete on MISP is to do a soft delete (meaning the attribute is not displayed on the default view on MISP). The reason we do it this way is that it allows to push delete updates to instances we synchronize with.

The delete method will set the default parameter of the attribute to True.

In [ ]:

attribute.delete()
print(attribute.to_json())

Mark event as published¶

Same idea: you can set the published flag from the api

In [ ]:

event.publish()
print(event.published)

MISPAttribute¶

In [ ]:

attr_type = 'ip-dst'
value = '1.1.1.1'

In [ ]:

from pymisp import MISPAttribute

# Attribute data already defined
attribute = MISPAttribute()
attribute.type = attr_type
attribute.value = value
print(attribute)

In [ ]:

# An attribute can also be loaded directly from a JSON
json = '''{
    "type": "ip-dst",
    "value": "127.0.0.1",
    "category": "Network activity",
    "to_ids": false
    }'''

attribute = MISPAttribute()
attribute.from_json(json)
print(attribute)

MISPObject¶

Objects in MISP are a way to group attributes together in a way that makes sense. The objects are based on templates that are bundled in the library itself.

Note: you can use your own templates, we will see how later

In [ ]:

from pymisp import MISPObject

circl_attr = event.add_attribute('ip-dst', '149.13.33.14')


misp_object = MISPObject('domain-ip', standalone=False, default_attributes_parameters=circl_attr)
# Notes: 
# * standalone: this object will be attached to a MISPEvent, so the references will be in the dump
# * default_attributes_parameters: keep parameters from a MISPAttribute (useful when expanding a existing one) 
misp_object.comment = 'My Fancy new object'

obj_attr = misp_object.add_attribute('domain', value='circl.lu')
obj_attr.add_tag('tlp:green')
misp_object.add_attribute('ip', value='149.13.33.14')
misp_object.add_attribute('first-seen', value='2018-04-11')
misp_object.add_attribute('last-seen', value='2018-06-11')

event.add_object(misp_object)
print(event.to_json())

Short version to add an object to a MISPEvent¶

You can also add the object directly in a misp event this way

In [ ]:

from pymisp import MISPObject

misp_object = event.add_object(name='domain-ip', comment='My Fancy new object, in one line')

obj_attr = misp_object.add_attribute('domain', value='circl.lu')
obj_attr.add_tag('tlp:green')
misp_object.add_attribute('ip', value='149.13.33.14')
misp_object.add_attribute('first-seen', value='2018-04-11')
misp_object.add_attribute('last-seen', value='2018-06-11')

misp_object.add_attributes('ip', {'value': '10.8.8.8', 'to_ids': False}, '10.9.8.8')


misp_object.add_reference(obj_attr.uuid, 'related-to', 'Expanded with passive DNS entry')

print(event.to_json())

New first/last seen¶

In [ ]:

from pymisp import MISPObject

misp_object = event.add_object(name='domain-ip', comment='My Fancy new object, in one line')

obj_attr = misp_object.add_attribute('domain', value='circl.lu')
obj_attr.add_tag('tlp:green')
misp_object.add_attribute('ip', value='149.13.33.14')

misp_object.first_seen = '2018-04-11'
misp_object.last_seen = '2018-06-11T23:27:40.23356+07:00'

print(misp_object.last_seen)

misp_object.add_attributes('ip', {'value': '10.8.8.8', 'to_ids': False}, '10.9.8.8')

misp_object.add_reference(obj_attr.uuid, 'related-to', 'Expanded with passive DNS entry')

print(event.to_json(indent=2))

Helpers for MISPObjects¶

For some objects, we have helpers in order to make your life easier. The most relevant example is the file object: when you have a file to push on MISP, there are plenty of indicators you can extract at once, and it is pretty simple to automate, so we made it a oneliner.

Note: This requires a few more dependencies to get the full power of the script:

lief to extract indicators out of PE/ELF/MachO files, and soon Android binaries.
python-magic to get the mime type
pydeep to compute the ssdeep of the binary whenever possible

pip install lief python-magic git+https://github.com/kbandla/pydeep.git

In [ ]:

from pymisp.tools import FileObject

file_obj = FileObject(filepath='../../tests/viper-test-files/test_files/EICAR.com', standalone=False)
print(file_obj.to_json())

In [ ]:

event.add_object(file_obj)
print(event.to_json())

Excel support¶

(okay, CSV, but that's the same thing, right?)

In [ ]:

%%bash 

cat ../../tests/csv_testfiles/valid_fieldnames.csv

In [ ]:

%%bash 

cat ../../tests/csv_testfiles/invalid_fieldnames.csv

In [ ]:

from pymisp.tools import CSVLoader
from pymisp import MISPEvent
from pathlib import Path

csv1 = CSVLoader(template_name='file', csv_path=Path('../../tests/csv_testfiles/valid_fieldnames.csv'))
event = MISPEvent()
event.info = 'Test event from CSV loader'
for o in csv1.load():
    event.add_object(**o)

print(event.to_json())

In [ ]:

event = MISPEvent()
event.info = 'Test event from CSV loader'
csv2 = CSVLoader(template_name='file', csv_path=Path('../../tests/csv_testfiles/invalid_fieldnames.csv'),
                 fieldnames=['SHA1', 'fileName', 'size-in-bytes'], has_fieldnames=True)

for o in csv2.load():
    event.add_object(**o)
    
print(event.to_json())

Generic helper¶

This helper is meant to be used when you already have a script that does the mapping between your own code, and the MISPObject template.

In [ ]:

from pymisp.tools import GenericObjectGenerator

attributeAsDict = [{'script': ':(){ :|:& };:', 'comment': 'Forkbomb'},
                   {'language': {'value': 'Bash', 'to_ids': False, 'disable_correlation': True}}, 
                   {'filename': {'value': 'forkbomb.sh', 'to_ids': True}},
                   {'state': 'Malicious'}]


misp_object = GenericObjectGenerator('script', strict=True)
misp_object.generate_attributes(attributeAsDict)

print(misp_object.to_json())

User defined objects¶

In [ ]:

from pymisp.tools import GenericObjectGenerator

attributeAsDict = [{'MyCoolAttribute': {'value': 'critical thing', 'type': 'text'}}, 
                   {'MyCoolerAttribute': {'value': 'even worse',  'type': 'text'}}]


misp_object = GenericObjectGenerator('my-cool-template', strict=True) # This is supposed to fail due to the strict parameter
misp_object.generate_attributes(attributeAsDict)

print(misp_object.to_json())

In [ ]:

from pymisp.tools import GenericObjectGenerator
from uuid import uuid4

attributeAsDict = [{'MyCoolAttribute': {'value': 'critical thing', 'type': 'text'}}, 
                   {'MyCoolerAttribute': {'value': 'even worse',  'type': 'text'}}]


misp_object = GenericObjectGenerator('my-cool-template')
misp_object.generate_attributes(attributeAsDict)
# The parameters below are required if no template is provided.
misp_object.template_uuid = uuid4()
misp_object.templade_id = 1
misp_object.description = "foo"
setattr(misp_object, 'meta-category', 'bar')

print(misp_object.to_json())

Use locally defined object templates¶

Important: The path you pass as parameter for misp_objects_path_custom needs to contain a directory equals to the value of the parameter name (same structure as the content of the misp-object repository)

In [ ]:

user_defined_obj = MISPObject(name='test_object_template', strict=True, misp_objects_path_custom='../../tests/mispevent_testfiles')

user_defined_obj.add_attribute('member3', value='foo')
user_defined_obj.add_attribute('member1', value='baz')

print(user_defined_obj.to_json())

Playing with a malware sample¶

The data you receive out of the JSON dump from a MISP instance is a base64 encoded zip with infected as a password. The zip file contains 2 files, one containing the original file name of the uploaded file, and the other one is the binary.

This is pretty much a pain to use as-is.

So there is an helper for that!

In [ ]:

sample = file_obj.get_attributes_by_relation('malware-sample')[0]
print(sample)
print('File name --->', sample.malware_filename)
print(sample.malware_binary)
print('Content of the malware (in bytes) ----->', sample.malware_binary.getvalue())

Use lief to extract indicators out of binaries¶

An other cool helper: one liner to whom you can pass the path to a binary, if it is supported by lief (PE/ELF/Mach-o), you get the file object, a PE, ELF, or Mach-o object, and the relevant sections.

If it is anything else, it will just generate the the file object.

In [ ]:

from pymisp.tools import make_binary_objects

file_obj, bin_obj, sections = make_binary_objects(filepath='../../tests/viper-test-files/test_files/whoami.exe', standalone=False)

event.add_object(file_obj)
if bin_obj: 
    event.add_object(bin_obj)
    for s in sections:
        event.add_object(s)

References¶

The references are also set by default by this method.

In [ ]:

print(bin_obj.uuid)
print(bin_obj.references[0].to_json())
print(event.to_json())

Dump valid MISP Event ready to push to MISP¶

We've been using to_json a lot. The thing you should know is that every python MISP objects have this method, and it always returns a valid json blob you can send to MISP.

In [ ]:

print(event.to_json())

Generate a feed¶

In [ ]:

from pymisp import MISPEvent, MISPOrganisation
from pymisp.tools import feed_meta_generator
from pathlib import Path
import json

out_dir = Path('feed_test')
out_dir.mkdir(exist_ok=True)

org = MISPOrganisation()
org.name = "Test Org"
org.uuid = "972360d2-2c96-4004-937c-ba010d03f925"

event = MISPEvent()

event.info = 'This is my new MISP event for a feed'
event.distribution = 1
event.Orgc = org
event.add_attribute('ip-dst', "8.8.8.8")

feed_event = event.to_feed()

with (out_dir / f'{event.uuid}.json').open('w') as f:
    json.dump(feed_event, f)


feed_meta_generator(out_dir)

In [ ]:

!ls feed_test

In [ ]:

!cat feed_test/manifest.json

!echo ''

!cat feed_test/hashes.csv

In [ ]:

!rm feed_test/*

Update an existing MISPEvent¶

We were creating new events, but you will also want to update an existing one.

In [ ]:

from pymisp import MISPEvent

existing_event = MISPEvent()
existing_event.load_file('../../tests/mispevent_testfiles/existing_event.json')

print(existing_event.attributes[0])
print(existing_event.attributes[0].tags)
print(existing_event.attributes[0].timestamp)
print(existing_event.attributes[0].to_json())

Edit, removes the timestamp when exporting¶

If you tried to edit an event manually, and never got the updates on the instance, it is probably because the timestamps weren't updated/removed. Or you removed them all, and adding a single tag was making every attributes as new.

PyMISP got you covered.

In [ ]:

existing_event.attributes[0].add_tag('tlp:white')
print(existing_event.attributes[0].to_json())

Specific use-cases¶

Change creator¶

In [ ]:

from pymisp import MISPOrganisation
orgc = MISPOrganisation()
orgc.name = 'bazbaz'
orgc.id = 15
orgc.uuid = '5888a98d-a7e8-4183-94bb-4d19950d210f'
# NOTE: Pushing this object will only work if the user has sync right (if not, the orgc key will be ignored)
event.Orgc = orgc

print(event.to_json())

Getting the API key (automatically generated on the training VM)¶

In [ ]:

from pathlib import Path

api_file = Path('apikey')
if api_file.exists():
    misp_url = 'http://127.0.0.1'
    misp_verifycert = False
    with open(api_file) as f:
        misp_key = f.read().strip()
    print(misp_key)
else:
    print("Unable to find the api key")

Initialize variables if you run the notebook locally¶

In [ ]:

# The URL of the MISP instance to connect to
misp_url = 'https://127.0.0.1:8443/'
# Can be found in the MISP web interface under 
# http://+MISP_URL+/users/view/me -> Authkey
misp_key = 'd6OmdDFvU3Seau3UjwvHS1y3tFQbaRNhJhDX0tjh'
# Should PyMISP verify the MISP certificate
misp_verifycert = False

In [ ]:

from pymisp import PyMISP

misp = PyMISP(misp_url, misp_key, misp_verifycert)

Full example¶

New API¶

Returns MISPEvent.

In [ ]:

from pymisp import MISPEvent, MISPObject

event = MISPEvent()
event.info = 'This is my new MISP event'  # Required
event.distribution = 0  # Optional, defaults to MISP.default_event_distribution in MISP config
event.threat_level_id = 2  # Optional, defaults to MISP.default_event_threat_level in MISP config
event.analysis = 1  # Optional, defaults to 0 (initial analysis)

mispObject = MISPObject('file')
mispObject.add_attribute('filename', type='filename',
                         value='filename.exe',
                         Tag=[{'name': 'tlp:amber'}])

event.add_object(mispObject)

print(misp)
existing_event = misp.add_event(event, pythonify=True)
print(existing_event)
mispObject = MISPObject('file')
mispObject.add_attribute('filename', type='filename',
                         value='filename2.exe',
                         Tag=[{'name': 'tlp:white'}])

existing_event.add_object(mispObject)
print(existing_event.to_json())

res = misp.update_event(existing_event)
existing_event = MISPEvent()
existing_event.load(res)
print(existing_event.to_json())

Interacting with a MISP instance¶

Creating An Event¶

Directly¶

In [ ]:

misp.toggle_global_pythonify()  # Returns PyMISP objects whenever possible, allows to skip pythonify

event = misp.add_event({'distribution': 1, "threat_level_id": 1, "analysis": 1, 'info':"Event from notebook"})
print("Event id: %s" % event.id)

Using the MISPEvent constructor¶

In [ ]:

from pymisp import MISPEvent

event_obj = MISPEvent()
event_obj.distribution = 1
event_obj.threat_level_id = 1
event_obj.analysis = 1
event_obj.info = "Event from notebook 2"
event = misp.add_event(event_obj, pythonify=True)
event_id = event.id
print("Event id: %s" % event_id)

Fetching an event¶

In [ ]:

event_id = 9

In [ ]:

# Fetch by ID
event = misp.get_event(event_id)
print(event)

In [ ]:

# Fetch by ID
event = misp_old.get_event(event_id)
print(event)

Add an attribute to an event¶

Directly¶

In [ ]:

attr_type = "ip-src"
value = "8.8.8.8"
category = "Network activity"
to_ids = False

Cleaner way¶

In [ ]:

value = "9.8.8.8"

In [ ]:

from pymisp import MISPAttribute

# Attribute data already defined
attribute = MISPAttribute()
attribute.type = attr_type
attribute.value = value
attribute.category = category
attribute.to_ids = to_ids

attribute_to_change = misp.add_attribute(event_id, attribute, pythonify=True)
print(attribute_to_change.id, attribute_to_change)

Propose new Attribute¶

In [ ]:

from pymisp import MISPAttribute

attr_type = "ip-src"
value = "10.8.8.8"
category = "Network activity"
to_ids = False

# Attribute data already defined
attribute = MISPAttribute()
attribute.type = attr_type
attribute.value = value
attribute.category = category
attribute.to_ids = to_ids

proposal = misp.add_attribute_proposal(event_id, attribute)
print(proposal.id, proposal)

Other things on proposals¶

In [ ]:

proposal = misp.get_attribute_proposal(1)
print(proposal.to_json())

In [ ]:

proposal = misp.accept_attribute_proposal(1)
print(proposal)

In [ ]:

proposal = misp.discard_attribute_proposal(2)
print(proposal)

Propose change to attribute¶

In [ ]:

from pymisp import MISPShadowAttribute

proposal = MISPShadowAttribute()
proposal.type = 'ip-dst'
proposal.category = 'External analysis'
proposal.to_ids = False

attribute = misp.update_attribute_proposal(attribute_to_change.id, proposal)
print(attribute.to_json())

In [ ]:

attribute = misp.update_attribute_proposal(attribute_to_change.id, {'to_ids': False, 'comment': "This is crap"})
print(attribute.to_json())

Update existing event¶

In [ ]:

from pymisp import MISPAttribute, MISPObject

attr_type = "ip-src"
value = "20.8.8.8"
category = "Network activity"
to_ids = False

# Attribute data already defined
attribute = MISPAttribute()
attribute.type = attr_type
attribute.value = value
attribute.category = category
attribute.to_ids = to_ids

# New Python 3.6 API
event = misp.get(event_id)

## Add the attribute to the event
event.add_attribute(**attribute)
event.add_attribute(type='domain', value='circl.lu', disable_correlation=True)

mispObject = MISPObject('file')
mispObject.add_attribute('filename', type='filename',
                         value='filename2.exe',
                         Tag=[{'name': 'tlp:white'}])

event.add_object(mispObject)

## Push the updated event to MISP
event_dict = misp.update_event(event)
print(event_dict)

Sightings¶

In [ ]:

misp.sighting(value=event.attributes[1].value)

In [ ]:

misp.sighting_list(event.attributes[1].id)

Direct call, no validation¶

In [ ]:

misp.direct_call('attributes/add/9', {'type': 'ip-dst', 'value': '8.11.8.8'})

In [ ]:

misp.direct_call('events')

Admin Stuff¶

In [ ]:

misp.sharing_groups()

User¶

In [ ]:

misp.users()

In [ ]:

misp.add_user({'email': 'bar@foo.de'})

Organisations¶

In [ ]:

misp.organisations()

Roles¶

In [ ]:

misp.roles()

Feeds¶

In [ ]:

misp.feeds(pythonify=True)

In [ ]:

misp.cache_feeds_all()

In [ ]: