hive - Hadoop Cluster Requirement Software /Hardware -

-

hi trying setup hadoop environment. in short problem trying solve involves billions of xml files of size few mb, extract relevant information them using hive , analytic work information. know trivial problem in hadoop world if hadoop solution works me size , number of files dealing increase in geometric progession form.

i did research referring various books "hadoop - definite guide", "hadoop in action". resources documents yahoo , hortonworks. not able figure out hardware /software specifications establishing hadoop environment. in resources had referred far had kind of found standard solutions

  1. namenode/jobtracker (2 x 1gb/s ethernet, 16 gb of ram, 4xcpu, 100 gb disk)
  2. datanode (2 x 1gb/s ethernet, 8 gb of ram, 4xcpu, multiple disks total amount
    of 500+ gb)

but if can give suggestions great.

first suggest consider: need more processing + storage or opposite, , view select hardware. case sounds more processing storage.
specify bit differently standard hardware hadoop
namenode: high quality disk in mirror, 16 gb hdd.
data nodes: 16-24 gb ram, dual quad or dual 6 cores cpu, 4 6 1-2-3 sata tb drives.

i consider 10 gbit option. think if not add more 15% of cluster price - makes sense. 15% came rough estimation data shipping mappers reducers takes 15% of job time.
in case more willing sacrifice disc sizes save money, not cpu/memory/number of drives.


Comments

Post a Comment

Popular posts from this blog

ruby - Trying to change last to "x"s to 23 -

-
i have been playing around irb , have string looks this: irb(main):072:0> puts ["[\"4354 5432 5432 xxxxx\", \"6547 6547 8543 xxxxx\", \"2344 6543 6674 xxxxx\", \"2346 6236 7543 xxxxx\", \"1273 5585 5587 xxxxx\"]"] => nil irb(main):073:0> what want gsub last 2 "x" s of each 17 digit combination, 23 . what i've tried far (i've been using rubular): a.gsub(/\d[x]/,"23") <= grabs "x" s i'm close a.gsub(/\w[x{2}]/,"23") <= grabs "x" s grabs two digits each combination. is there easier way i'm not understanding? a.gsub(/\d{4} \d{4} \d{4} xxx\kxx/, '23') \d{4} = 4 digits what \k drop matched far. can use don't replace entire number, still able validate have in desired format.
Read more

jquery - Clone last and append item to closest class -

-
this bare-bones code on page: <div class="list-container"> <div class="single-list-item"> <button>remove</button> </div> <div class="single-list-item"> <button>remove</button> </div> <div class="single-list-item"> <button>remove</button> </div> </div> <p><button class="add-list-item">add</button></p> <div class="list-container"> <div class="single-list-item"> <button>remove</button> </div> <div class="single-list-item"> <button>remove</button> </div> <div class="single-list-item"> <button>remove</button> </div> </div> <p><button class="add-list-item">add</button></p> <di
Read more

c - Unrecognised emulation mode: elf_i386 on MinGW32 -

-
i'm trying make kernel, , cannot link c output assembly. ld . i'm getting error: unrecognized emulation mode: elf_i386 i'm using windows 10 professional mingw32 , msys. code using: link.ld /* * link.ld */ output_format(elf32-i386) entry(start) sections { . = 0x100000; .text : { *(.text) } .data : { *(.data) } .bss : { *(.bss) } } kernel.c /* * kernel.c */ void kmain(void) { const char *str = "my first kernel"; char *vidptr = (char*)0xb8000; //video mem begins here. unsigned int = 0; unsigned int j = 0; /* loops clears screen * there 25 lines each of 80 columns; each element takes 2 bytes */ while(j < 80 * 25 * 2) { /* blank character */ vidptr[j] = ' '; /* attribute-byte - light grey on black screen */ vidptr[j+1] = 0x07; j = j + 2; } j = 0; /* loop writes string video memory */ while(str[j] != '\0') { /
Read more