Hadoop: Distinct count of a value (Java) -

-

example of (key,value) in mapper : (user,(logincount,commentcount))

public void map(longwritable key, text value, context context)                 throws ioexception, interruptedexception {              string tempstring = value.tostring();             string[] stringdata = tempstring.split(",");              string user = stringdata[2];             string activity = stringdata[1];              if (activity.matches("login")) {                 outcount.set(1,0);             }              if (activity.matches("comment")) {                 outcount.set(0,1);             }              outuserid.set(userid);              context.write(outuserid, outcount);          }

i count logins & comments of user. want change count: count every login & if user wrote comment. how can achieve mapper or reducer search 1 comment of user , "ignores" other comments (of user)?

edit:

log-file:

2013-01-01t16:50:56.056+0100,login,user14133,somedata,somedata 2013-01-01t16:55:56.056+0100,login,user14133,somedata,somedata 2013-01-01t05:20:44.044+0100,comment,user14133,somedata,somedata,{text: "something here"} 2013-01-01t05:24:44.044+0100,comment,user14133,somedata,somedata,{text: "something here"} 2013-01-01t20:50:13.013+0100,login,user76892,somedata,somedata

output @ moment:

user14133   logins: 2   comments: 2 user76892   logins: 1   comments: 0

input: 

mapper<longwritable, text, text, usercount> reducer<text, usercount, text, usercount>  public static class usercount implements writable {         public usercounttuple() {             set(new intwritable(0), new intwritable(0));         }

my mapreduce counts every login , every comment of user , sum them up. want achieve -> output:

user14133   logins: 2      comments: 0 or 1 (did user wrote 1 comment?)*   * in mapper or reducer (?)  every line in log{    if (user wrote comment){      return 1;      ignore other comments same user in log;    } else if (user didn't write anything) return 0;  }

if understand correctly, want total number of unique users logged in, along total number of comments?

i recommend using "aggregate" reducer in hadoop.

in mapper, output lines this:

uniqvaluecount:unique_users      user14133 longvaluesum:comments            1 uniqvaluecount:unique_users      user14133 longvaluesum:comments            1 uniqvaluecount:unique_users      user14133 longvaluesum:comments            1 uniqvaluecount:unique_users      user14133 longvaluesum:comments            1 uniqvaluecount:unique_users      user76892 longvaluesum:comments            1

and run "aggregate" reducer on this, should output looks like:

unique_users    2 comments        5

i'm assuming want?


Comments

Post a Comment

Popular posts from this blog

ruby - Trying to change last to "x"s to 23 -

-
i have been playing around irb , have string looks this: irb(main):072:0> puts ["[\"4354 5432 5432 xxxxx\", \"6547 6547 8543 xxxxx\", \"2344 6543 6674 xxxxx\", \"2346 6236 7543 xxxxx\", \"1273 5585 5587 xxxxx\"]"] => nil irb(main):073:0> what want gsub last 2 "x" s of each 17 digit combination, 23 . what i've tried far (i've been using rubular): a.gsub(/\d[x]/,"23") <= grabs "x" s i'm close a.gsub(/\w[x{2}]/,"23") <= grabs "x" s grabs two digits each combination. is there easier way i'm not understanding? a.gsub(/\d{4} \d{4} \d{4} xxx\kxx/, '23') \d{4} = 4 digits what \k drop matched far. can use don't replace entire number, still able validate have in desired format.
Read more

jquery - Clone last and append item to closest class -

-
this bare-bones code on page: <div class="list-container"> <div class="single-list-item"> <button>remove</button> </div> <div class="single-list-item"> <button>remove</button> </div> <div class="single-list-item"> <button>remove</button> </div> </div> <p><button class="add-list-item">add</button></p> <div class="list-container"> <div class="single-list-item"> <button>remove</button> </div> <div class="single-list-item"> <button>remove</button> </div> <div class="single-list-item"> <button>remove</button> </div> </div> <p><button class="add-list-item">add</button></p> <di...
Read more

css - Can I use the :after pseudo-element on an input field? -

-
i trying use :after css pseudo-element on input field, not work. if use span , works ok. <style type="text/css"> .mystyle:after {content:url(smiley.gif);} .mystyle {color:red;} </style> this works (puts smiley after "buu!" , before "some more") <span class="mystyle">buuu!</span>a more this not work - colors somevalue in red, there no smiley. <input class="mystyle" type="text" value="somevalue"> what doing wrong? should use pseudo-selector? note: cannot add span around input , because being generated third-party control. :after , :before not supported in internet explorer 7 , under, on elements. it's not meant used on replaced elements such form elements (inputs) , image elements . in other words it's impossible pure css. however if using jquery can use $(".mystyle").after("add smiley here"); api docs on .after to ...
Read more