python - MapReduce with paramiko how to print stdout as it streams -

-

i have created small python script using paramiko allows me run mapreduce jobs without using putty or cmd windows initiate jobs. works great, except don't see stdout until job completes. how can set can see each line of stdout generated, able via cmd window?

here script:

import paramiko  # define connection info host_ip = 'xx.xx.xx.xx' user = 'xxxxxxxxx' pw = 'xxxxxxxxx'  # commands list_dir = "ls /nfs_home/appers/cnielsen -l" mr = "hadoop jar /opt/cloudera/parcels/cdh/lib/hadoop-0.20-mapreduce/contrib/streaming/hadoop-streaming.jar -files /nfs_home/appers/cnielsen/product_lookups.xml -file /nfs_home/appers/cnielsen/mapper.py -file /nfs_home/appers/cnielsen/reducer.py -mapper '/usr/lib/python_2.7.3/bin/python mapper.py test1' -file /nfs_home/appers/cnielsen/process.py -reducer '/usr/lib/python_2.7.3/bin/python reducer.py' -input /nfs_home/appers/extracts/*/*.xml -output /user/loc/output/cnielsen/test51" getmerge = "hadoop fs -getmerge /user/loc/output/cnielsen/test51 /nfs_home/appers/cnielsen/test_010716_0.txt"  client = paramiko.sshclient() client.set_missing_host_key_policy(paramiko.autoaddpolicy()) client.connect(host_ip, username=user, password=pw) ##stdin, stdout, stderr = client.exec_command(list_dir) ##stdin, stdout, stderr = client.exec_command(getmerge) stdin, stdout, stderr = client.exec_command(mr)  print "executing command..."  line in stdout:     print '... ' + line.strip('\n') l in stderr:     print '... ' + l.strip('\n') client.close()

this code implicitly calling stdout.read() blocks until eof. you'll therefore have read stdout/stderr in chunks instantly output. this answer , modified version of this answer should resolving issue. i'd recommend adapt answer 2 use-case prevent common stalling scenarios.

here's adapted example taken answer 1

sin,sout,serr = ssh.exec_command("while true; uptime; done")  def line_buffered(f):     line_buf = ""     while not f.channel.exit_status_ready():         line_buf += f.read(1)         if line_buf.endswith('\n'):             yield line_buf             line_buf = ''  l in line_buffered(sout):   # or serr     print l

Comments

Post a Comment

Popular posts from this blog

ruby - Trying to change last to "x"s to 23 -

-
i have been playing around irb , have string looks this: irb(main):072:0> puts ["[\"4354 5432 5432 xxxxx\", \"6547 6547 8543 xxxxx\", \"2344 6543 6674 xxxxx\", \"2346 6236 7543 xxxxx\", \"1273 5585 5587 xxxxx\"]"] => nil irb(main):073:0> what want gsub last 2 "x" s of each 17 digit combination, 23 . what i've tried far (i've been using rubular): a.gsub(/\d[x]/,"23") <= grabs "x" s i'm close a.gsub(/\w[x{2}]/,"23") <= grabs "x" s grabs two digits each combination. is there easier way i'm not understanding? a.gsub(/\d{4} \d{4} \d{4} xxx\kxx/, '23') \d{4} = 4 digits what \k drop matched far. can use don't replace entire number, still able validate have in desired format.
Read more

jquery - Clone last and append item to closest class -

-
this bare-bones code on page: <div class="list-container"> <div class="single-list-item"> <button>remove</button> </div> <div class="single-list-item"> <button>remove</button> </div> <div class="single-list-item"> <button>remove</button> </div> </div> <p><button class="add-list-item">add</button></p> <div class="list-container"> <div class="single-list-item"> <button>remove</button> </div> <div class="single-list-item"> <button>remove</button> </div> <div class="single-list-item"> <button>remove</button> </div> </div> <p><button class="add-list-item">add</button></p> <di...
Read more

css - Can I use the :after pseudo-element on an input field? -

-
i trying use :after css pseudo-element on input field, not work. if use span , works ok. <style type="text/css"> .mystyle:after {content:url(smiley.gif);} .mystyle {color:red;} </style> this works (puts smiley after "buu!" , before "some more") <span class="mystyle">buuu!</span>a more this not work - colors somevalue in red, there no smiley. <input class="mystyle" type="text" value="somevalue"> what doing wrong? should use pseudo-selector? note: cannot add span around input , because being generated third-party control. :after , :before not supported in internet explorer 7 , under, on elements. it's not meant used on replaced elements such form elements (inputs) , image elements . in other words it's impossible pure css. however if using jquery can use $(".mystyle").after("add smiley here"); api docs on .after to ...
Read more