Navigation
Analysis Client Shell¶
The Analysis Client Shell allows running the client from the command line. This provides an interactive shell for manually sending requests to the Lastline Analyst API, and it can be used to experiment with the API for analyzing files or URLs.
This client shell is available for download at analysis_apiclient_shell.py.
To start the shell, place the file in the same directory as the Analyst API client and invoke:
python analysis_apiclient_shell.py <API_KEY> <API_TOKEN>
replacing <API_KEY> and <API_TOKEN> with your API credentials. Once the shell is started, the
current context contains an analysis object. This is an instance of
analysis_apiclient.AnalysisClient, which can be used to access the
functionality of the Lastline Analyst API.
By default, the client connects to an API instance running in the hosted Lastline datacenters
at https://analysis.lastline.com . To connect to a different instance, for
example when using a Lastline On-Premises installation, please use the
--api-url parameter to point to the URL of the On-Premises API. For
example, to
connect to a Lastline Analyst On-Premises running at analyst.lastline.local,
use:
python analysis_apiclient_shell.py --api-url https://analyst.lastline.local/ <API_KEY> <API_TOKEN>
Analyst API Shell Example¶
This is an example of how the Analyst API shell can be used to analyze resources with the Lastline Analyst API:
$ python analysis_apiclient_shell.py XXXXXXXXXXXXXXXXXXXX yyyyyyyyyyyyyyyyyyyyyyyyy
-------------------------------------
Lastline Analyst API shell
-------------------------------------
[...]
In [1]:
Here, XXX and yyy need to be replaced with your API key and token respectively.
IPython features make the API shell easy to discover. For instance, you can use tab-autocompletion to list the methods of an object:
In [1]: analysis.
analysis.DATETIME_FMT analysis.get_completed analysis.submit_exe_file
analysis.DATE_FMT analysis.get_completed_with_metadata analysis.submit_exe_hash
analysis.ERRORS analysis.get_progress analysis.submit_file
analysis.FORMATS analysis.get_result analysis.submit_file_hash
analysis.SUB_APIS analysis.get_result_artifact analysis.submit_file_metadata
analysis.analyze_sandbox_result analysis.get_result_summary analysis.submit_url
analysis.completed analysis.rescore_task analysis.set_key
And you can use the question-mark character after an object or method to get documentation for it:
In [2]: analysis.submit_file?
...
File: analysis_apiclient.py
Definition: analysis.submit_file(self, file_stream, download_ip=None, download_port=None, download_url=None, download_host=None, download_path=None, download_agent=None, download_referer=None, download_request=None, full_report_score=None, bypass_cache=False, delete_after_analysis=False, backend=None, raw=False, verify=True)
Docstring:
Submit a file by uploading it.
For return values and error codes please
see :py:meth:`malscape.api.views.analysis.submit_file`.
If there is an error and `raw` is not set,
a :py:class:`AnalysisAPIError` exception will be raised.
:param file_stream: file-like object containing
the file to upload.
:param download_ip: ASCII dotted-quad representation of the IP address
from which the file has been downloaded
:param download_port: integer representation of the port number
from which the file has been downloaded
:param download_url: DEPRECATED! replaced by the download_host
and download_path parameters
:param download_host: host from which the submitted file
was originally downloaded, as a string of bytes (not unicode)
:param download_path: host path from which the submitted file
was originally downloaded, as a string of bytes (not unicode)
:param download_agent: HTTP user-agent header that was used
when the submitted file was originally downloaded,
as a string of bytes (not unicode)
:param download_referer: HTTP referer header that was used
when the submitted file was originally downloaded,
as a string of bytes (not unicode)
:param download_request: full HTTP request with
which the submitted file was originally downloaded,
as a string of bytes (not unicode)
:param full_report_score: if set, this value (between -1 and 101)
determines starting at which scores a full report is returned.
-1 and 101 indicate "never return full report";
0 indicates "return full report at all times"
:param bypass_cache: if True, the API will not serve a cached
result. NOTE: This requires special privileges. Further, this
only bypasses the MalScape caching and might still serve a
cached result from the analysis engine
:param delete_after_analysis: if True, the backend will delete the
file after analysis is done (and noone previously submitted
this file with this flag set)
:param backend: DEPRECATED! Don't use
:param verify: if False, disable SSL-certificate verification
:param raw: if True, return the raw JSON results of the API query
The same documentation is also available above.
Let’s see how we can submit a URL:
In [3]: r=analysis.submit_url("http://www.google.com")
In [4]: r
Out[4]: {u'data': {u'task_uuid': u'a553401f249f4c209541799c3ccede23'}, u'success': 1}
uuid=r["data"]["task_uuid"]
We now have a UUID for the submitted analysis task. We can use this to request the results:
In [5]: result=analysis.get_result(uuid)
In [6]: result["data"]["score"]
Out[6]: 0
In [7]: import pprint
In [8]: pprint.pprint(result["data"]["report"])
{u'analysis': {u'artifacts': None,
u'et_results': None,
u'exploits': None,
u'network': {u'redirects': None,
u'requests': [{u'content_md5': u'6294aa8e1853ee626776e7c8c9e6ff3b',
u'content_sha1': None,
u'content_type': u'text/html',
u'ip': None,
u'parent_url': u'USER_URL',
u'relation_type': u'6',
u'status': u'302',
u'url': u'http://www.google.com/'},
{u'content_md5': u'd0d9aabed132ac09e529453b7e26b864',
u'content_sha1': u'b680dab114a88277cd7364dbdc06bac358fa0eb8',
u'content_type': u'text/html',
u'ip': u'74.125.224.191',
u'parent_url': u'http://www.google.com/',
u'relation_type': u'4',
u'status': u'200',
u'url': u'http://www.google.co.uk/'},
{u'content_md5': u'cceaec1808de4b7c9e7a0df5c213db74',
u'content_sha1': u'd5c0854c026fd1ed9b15b32bae87dc73723cbd6e',
u'content_type': u'text/javascript',
u'ip': u'74.125.224.191',
u'parent_url': u'http://www.google.co.uk/',
u'relation_type': u'1',
u'status': u'200',
u'url': u'http://www.google.co.uk/xjs/_/js/k=xjs.hp.en_US.vpRA07Px3nw.O/m=sb_he,pcc/rt=j/d=1/sv=1/rs=AItRSTM2fR5rzErXQEgClenaRaSY3BLhMw'},
{u'content_md5': u'a0af21c60b0dddc27b96d9294b7d5d8f',
u'content_sha1': u'88ae1a2683797b31dc07d7128d7a24a628bdf52e',
u'content_type': u'text/javascript',
u'ip': u'74.125.239.15',
u'parent_url': u'http://www.google.co.uk/',
u'relation_type': u'1',
u'status': u'200',
u'url': u'http://ssl.gstatic.com/gb/js/sem_a0af21c60b0dddc27b96d9294b7d5d8f.js'}]},
u'plugins': None,
u'result': {u'analysis_ended': u'2013-07-09 04:44:51+0000',
u'classification': u'benign',
u'detector': u'2.6',
u'explanation': u'models:0.84:0.00:0.84:0.00:0.00'},
u'shellcodes': None,
u'signatures': None,
u'subject': {u'type': u'url',
u'url': u'http://www.google.com/'},
u'threats': None},
...
The score is 0 since this is a benign site. The report contains more information.
Submitting an executable for analysis is similar, except that we can first check to see if the executable is already known by submitting a hash:
In [21]: f=open('example.file')
In [22]: import hashlib
In [23]: md5=hashlib.md5(f.read()).hexdigest()
In [24]: md5
Out[24]: 'f19e59513a23e676495fa72bd97995f2'
In [25]: analysis.submit_file_hash(md5=md5)
---------------------------------------------------------------------------
AnalysisAPIError Traceback (most recent call last)
<ipython console> in <module>()
analysis_apiclient.py in submit_file_hash(self, md5, sha1, download_url, download_request, download_referer, full_report_score, bypass_cache, raw)
215 purge_none(files)
216 purge_none(params)
--> 217 return self._api_request(url, params, files=files, post=True, raw=raw)
218
219 def submit_file(self,
analysis_apiclient.py in _api_request(self, url, params, files, timeout, post, raw, requested_format)
423 response.raise_for_status()
424 page = response.text
--> 425 return self._process_response_page(page, raw, requested_format)
426
427 def init_shell(banner):
analysis_apiclient.py in _process_response_page(self, page, raw, requested_format)
362 else:
363 error_code = result.get('error_code', None)
--> 364 raise AnalysisAPIError(result['error'], error_code)
365
366 class AnalysisClient(AnalysisClientBase):
AnalysisAPIError: Analysis API error (101): No file found matching requested hash.
In this case, the hash was not previously known, so an exception is raised. We should instead submit the file:
In [26]: f.seek(0)
In [27]: analysis.submit_file(f)
Out[27]: {u'data': {u'task_uuid': u'1483924fa75c440ab5493b1557ab57c5'}, u'success': 1}
Analyst API Shell Helpers¶
The API client module provides helper classes for submitting artifacts and waiting for analysis results:
In [28]: import analysis_apiclient
In [29]: helper = analysis_apiclient.SubmissionHelper(analysis)
In [30]: ts = helper.get_api_utc_timestamp()
In [31]: result = helper.submit_url('https://www.lastline.com/')
Submitting URL https://www.lastline.com/
In [32]: result.is_complete()
Out[32]: False
In [33]: helper.wait_for_completion_of_submission(result, ts)
Waiting for completion of 1/1 submissions
Waiting for completion of 1 submissions
Got result for task 86a5fe607aaa4181867ed9a71bcab664
Got result for task 86a5fe607aaa4181867ed9a71bcab664: AnalysisTask 86a5fe607aaa4181867ed9a71bcab664(score: 1): URL=https://www.lastline.com/
Done waiting for completion of 1 submissions
In [34]: result.is_complete()
Out[34]: True
In [35]: result.task_uuid
Out[35]: u'86a5fe607aaa4181867ed9a71bcab664'
In [36]: result.score
Out[36]: 1
In [37]: result = helper.submit_filename('./myfile.exe')
In [38]: result.is_complete()
Out[38]: True
In [39]: result.score
Out[39]: 95
In [40]: helper.wait_for_completion_of_submission(result, ts)
No need to wait for completion for any of 1 submissions
These helper classes also support submitting multiple artifacts at once:
In [41]: results = helper.submit_urls_and_wait_for_completion(['http://www.google.com', 'https://www.lastline.com'])
Submitting 2 URLs
Submitting URL http://www.google.com
Submitting URL https://www.lastline.com
Waiting for completion of 0/2 submissions
Done waiting for completion of 2 submissions
In [42]: results['https://www.lastline.com'].task_uuid
Out[42]: u'0ea874bab6ac4c58b374087cc0047cfb'
In [43]: results['https://www.lastline.com'].score
Out[43]: 1
In [44]: results = helper.submit_filenames_and_wait_for_completion([ './sample.exe', './hello.exe' ], bypass_cache=True)
Submitting 2 files
Submitting file hello.exe (md5=ac8545103e85219d7ff98bcd5f5a12ff, sha1=969f7183dc81eb7e95e47f06d640623be4f5ed9f)
Submitting file by hash failed: Analysis API error (101): No file found matching requested hash.
Submitting file sample.exe (md5=e7742a9e48739ce3abb686c2c1299690, sha1=b58cd7d734f9b8deb93c0f52bec8437c3c1b99ca)
Submitting file by hash failed: Analysis API error (101): No file found matching requested hash.
Waiting for completion of 2/2 submissions
[...]
Got result for task dfde1ddce80b48488034bcaaf29bc6ab
Got result for task dfde1ddce80b48488034bcaaf29bc6ab: AnalysisTask dfde1ddce80b48488034bcaaf29bc6ab: MD5=ac8545103e85219d7ff98bcd5f5a12ff, SHA1=969f7183dc81eb7e95e47f06d640623be4f5ed9f, name=hello.exe
Waiting for completion of 1 submissions
Got result for task b735251ecd44475cad34fb6a20a1d4cf
Got result for task b735251ecd44475cad34fb6a20a1d4cf: AnalysisTask b735251ecd44475cad34fb6a20a1d4cf(score: 19): MD5=e7742a9e48739ce3abb686c2c1299690, SHA1=b58cd7d734f9b8deb93c0f52bec8437c3c1b99ca, name=sample.exe
Done waiting for completion of 2 submissions
In [45]: results['./sample.exe'].task_uuid
Out[45]: u'b735251ecd44475cad34fb6a20a1d4cf'
In [46]: results['./sample.exe'].score
Out[46]: 19
Furthermore, the API shell also provides helper classes for accessing analysis data, such as the primary analysis subject. To download the file using the task UUID (assuming it is available in the system), use:
In [1]: import analysis_apiclient
In [2]: helper = analysis_apiclient.QueryHelper(analysis)
In [3]: file_stream = helper.download_analysis_subject_file(task_uuid=<task-UUID>)
In [4]: if file_stream:
...: print 'Download successful, downloaded file has SHA1',
...: print analysis_apiclient.hash_stream(file_stream, 'sha1')
Download successful, downloaded file has SHA1 <analysis-subject-SHA1>
Similarly, if the file hash of the analysis subject is known, the file can be downloaded using:
In [1]: import analysis_apiclient
In [2]: helper = analysis_apiclient.QueryHelper(analysis)
In [3]: file_stream = helper.download_analysis_subject_by_file_hash(sha1=<sha1>)
In [4]: with open(<output-filename>, 'wb') as outf:
...: outf.write(file_stream.read())
Note that these helper classes are automatically instantiated and provided to the user when
using the API client shell analysis_apiclient_shell.py:
$ python analysis_apiclient_shell.py <key> <token>
--------------------------------------
Lastline Analyst API shell
--------------------------------------
[...]
In [1]: result = submission_helper.submit_url('https://www.lastline.com/')
In [2]: file_stream = query_helper.download_analysis_subject_by_file_hash(md5=<md5>)