python - MapReduce with paramiko how to print stdout as it streams -

-

i have created small python script using paramiko allows me run mapreduce jobs without using putty or cmd windows initiate jobs. works great, except don't see stdout until job completes. how can set can see each line of stdout generated, able via cmd window?

here script:

import paramiko  # define connection info host_ip = 'xx.xx.xx.xx' user = 'xxxxxxxxx' pw = 'xxxxxxxxx'  # commands list_dir = "ls /nfs_home/appers/cnielsen -l" mr = "hadoop jar /opt/cloudera/parcels/cdh/lib/hadoop-0.20-mapreduce/contrib/streaming/hadoop-streaming.jar -files /nfs_home/appers/cnielsen/product_lookups.xml -file /nfs_home/appers/cnielsen/mapper.py -file /nfs_home/appers/cnielsen/reducer.py -mapper '/usr/lib/python_2.7.3/bin/python mapper.py test1' -file /nfs_home/appers/cnielsen/process.py -reducer '/usr/lib/python_2.7.3/bin/python reducer.py' -input /nfs_home/appers/extracts/*/*.xml -output /user/loc/output/cnielsen/test51" getmerge = "hadoop fs -getmerge /user/loc/output/cnielsen/test51 /nfs_home/appers/cnielsen/test_010716_0.txt"  client = paramiko.sshclient() client.set_missing_host_key_policy(paramiko.autoaddpolicy()) client.connect(host_ip, username=user, password=pw) ##stdin, stdout, stderr = client.exec_command(list_dir) ##stdin, stdout, stderr = client.exec_command(getmerge) stdin, stdout, stderr = client.exec_command(mr)  print "executing command..."  line in stdout:     print '... ' + line.strip('\n') l in stderr:     print '... ' + l.strip('\n') client.close()

this code implicitly calling stdout.read() blocks until eof. you'll therefore have read stdout/stderr in chunks instantly output. this answer , modified version of this answer should resolving issue. i'd recommend adapt answer 2 use-case prevent common stalling scenarios.

here's adapted example taken answer 1

sin,sout,serr = ssh.exec_command("while true; uptime; done")  def line_buffered(f):     line_buf = ""     while not f.channel.exit_status_ready():         line_buf += f.read(1)         if line_buf.endswith('\n'):             yield line_buf             line_buf = ''  l in line_buffered(sout):   # or serr     print l

Comments

Post a Comment

Popular posts from this blog

ruby - Trying to change last to "x"s to 23 -

-
i have been playing around irb , have string looks this: irb(main):072:0> puts ["[\"4354 5432 5432 xxxxx\", \"6547 6547 8543 xxxxx\", \"2344 6543 6674 xxxxx\", \"2346 6236 7543 xxxxx\", \"1273 5585 5587 xxxxx\"]"] => nil irb(main):073:0> what want gsub last 2 "x" s of each 17 digit combination, 23 . what i've tried far (i've been using rubular): a.gsub(/\d[x]/,"23") <= grabs "x" s i'm close a.gsub(/\w[x{2}]/,"23") <= grabs "x" s grabs two digits each combination. is there easier way i'm not understanding? a.gsub(/\d{4} \d{4} \d{4} xxx\kxx/, '23') \d{4} = 4 digits what \k drop matched far. can use don't replace entire number, still able validate have in desired format.
Read more

jquery - Clone last and append item to closest class -

-
this bare-bones code on page: <div class="list-container"> <div class="single-list-item"> <button>remove</button> </div> <div class="single-list-item"> <button>remove</button> </div> <div class="single-list-item"> <button>remove</button> </div> </div> <p><button class="add-list-item">add</button></p> <div class="list-container"> <div class="single-list-item"> <button>remove</button> </div> <div class="single-list-item"> <button>remove</button> </div> <div class="single-list-item"> <button>remove</button> </div> </div> <p><button class="add-list-item">add</button></p> <di
Read more

c - Unrecognised emulation mode: elf_i386 on MinGW32 -

-
i'm trying make kernel, , cannot link c output assembly. ld . i'm getting error: unrecognized emulation mode: elf_i386 i'm using windows 10 professional mingw32 , msys. code using: link.ld /* * link.ld */ output_format(elf32-i386) entry(start) sections { . = 0x100000; .text : { *(.text) } .data : { *(.data) } .bss : { *(.bss) } } kernel.c /* * kernel.c */ void kmain(void) { const char *str = "my first kernel"; char *vidptr = (char*)0xb8000; //video mem begins here. unsigned int = 0; unsigned int j = 0; /* loops clears screen * there 25 lines each of 80 columns; each element takes 2 bytes */ while(j < 80 * 25 * 2) { /* blank character */ vidptr[j] = ' '; /* attribute-byte - light grey on black screen */ vidptr[j+1] = 0x07; j = j + 2; } j = 0; /* loop writes string video memory */ while(str[j] != '\0') { /
Read more