In the tutorial we go over some simple examples using python to communicate with the NEWT API to access NERSC. More details on NEWT can be found at http://newt.nersc.gov
We use the requests
python module (See http://www.python-requests.org) for this tutorial since it really simplifies interactions over HTTP, and allows you to hold on to authenticated sessions.
from requests import Session
Let's initialize a session object
s = Session()
Now, we're going to do the simplest possible call to NEWT. This is the "Hello World Example"
r = s.get("https://newt.nersc.gov/newt")
Let's examine the content of the response
r.content
'{"status": "OK", "output": {"text": "Welcome to NEWT", "version": "0.3.2"}, "error": ""}'
Now we check our current authentication status
r = s.get("https://newt.nersc.gov/newt/auth")
r.content
'{"username": null, "session_lifetime": 0, "auth": false, "newt_sessionid": null}'
Now we authenticate. Just provide a dict with of the form {"username": "myusername, "password": "mypassword"}
. Here mypassword will be your NERSC password and your MFA string combined. POST
it to the https://newt.nersc.gov/newt/auth URL
password = 'XXXXXX' + 'YYYYYY' # XXXXXX is NERSC passwd, YYYYYY is NERSC MFA
r = s.post("https://newt.nersc.gov/newt/auth", {"username": "shreyas", "password": password})
Check our status code and the content to make sure it succeeded
r.status_code
200
r.content
'{"username": "shreyas", "session_lifetime": 43198, "auth": true, "newt_sessionid": "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"}'
Now we create a new batch job on cori. Post the path for the SLURM job submission file on the target system. This should be a post
request to https://newt.nersc.gov/newt/queue/cori/ using a dict of the form {"jobfile": "/path/to/file/on/cori"}
. You can also post a "jobscript"
(instead of a "jobfile"
) containing the raw contents of the script.
r = s.post("https://newt.nersc.gov/newt/queue/cori/", {"jobfile": "/project/projectdirs/osp/newt.sbatch"})
The JOB id is in the resulting "jobid"
field. Note that the requests object has a json()
method that will automatically convert json output into python objects
r.content
'{"status": "OK", "error": "", "jobid": "1250312"}'
jobid = r.json()['jobid']
Let's use this jobid to query the queue
r = s.get("https://newt.nersc.gov/newt/queue/cori/%s" % jobid)
r.status_code
200
r.content
'{"status": "PD", "repo": "mpccc", "rank_bf": "N/A", "qos": "normal", "name": "newt.sbatch", "timeuse": "0:00", "hostname": "cori", "jobid": "1250312", "queue": "regular", "submittime": "2016-03-02T10:50:04", "reason": "Priority", "user": "shreyas", "memory": "0", "nodes": "1", "timereq": "1:00", "procs": "1", "rank_p": "5607"}'
Now let's get everything for user shreyas
. Note you can query for any field in the job object with ?key=value
r = s.get("https://newt.nersc.gov/newt/queue/cori/?user=shreyas")
j = r.json()
len(j)
2
Sample shreyas job (the first job in the returned list)
j[0]
{u'hostname': u'cori', u'jobid': u'1250267', u'memory': u'0', u'name': u'newt.sbatch', u'nodes': u'1', u'procs': u'1', u'qos': u'normal', u'queue': u'regular', u'rank_bf': u'N/A', u'rank_p': u'5580', u'reason': u'Priority', u'repo': u'mpccc', u'status': u'PD', u'submittime': u'2016-03-02T10:38:01', u'timereq': u'1:00', u'timeuse': u'0:00', u'user': u'shreyas'}