hive - Avro,parquet and SequenceFileFormat position in Hadoop Ecosystem and their utility -

-

i have seen different file formats being used while importing , storing hdfs , data processing engines utilize these formats while performing own set of procedures.so kind of difference these file formats make , how choice made different use cases.being newbie creates confusion.kindly same.

the choice depends on use case facing according type of data have, compatibility processing tools, schema evolution, file size, type of query , read performances.

in general :

  • avro more suitable event data can change on time
  • sequence datasets sharded between mr jobs
  • parquet more suitable analytics due it's columnar format

here keys can you

writing performance ( more + have faster )

  • sequence : +++
  • avro : ++
  • parquet : +

reading performance ( more + have faster )

  • sequence : +
  • avro : + + +
  • parquet : + + + + +

file sizes ( more + have smaller file )

  • sequence : +
  • avro : ++
  • parquet : + + +

and here facts each file type

avro :

  • better in schema evolution
  • is row oriented binary format
  • has schema
  • the file contain schema in addition data.
  • supports schema evolution
  • can compressed
  • compact , fast binary format

parquet :

  • slow in writing fast in reading
  • is column oriented binary format
  • supports compression
  • optimized , efficient in terms of disk i/o when specific columns need queried

sequencefile :

  • is row oriented format
  • supports splitting if data compressed
  • can used pack small files in hadoop

i wish answer you


Comments

Post a Comment

Popular posts from this blog

ruby - Trying to change last to "x"s to 23 -

-
i have been playing around irb , have string looks this: irb(main):072:0> puts ["[\"4354 5432 5432 xxxxx\", \"6547 6547 8543 xxxxx\", \"2344 6543 6674 xxxxx\", \"2346 6236 7543 xxxxx\", \"1273 5585 5587 xxxxx\"]"] => nil irb(main):073:0> what want gsub last 2 "x" s of each 17 digit combination, 23 . what i've tried far (i've been using rubular): a.gsub(/\d[x]/,"23") <= grabs "x" s i'm close a.gsub(/\w[x{2}]/,"23") <= grabs "x" s grabs two digits each combination. is there easier way i'm not understanding? a.gsub(/\d{4} \d{4} \d{4} xxx\kxx/, '23') \d{4} = 4 digits what \k drop matched far. can use don't replace entire number, still able validate have in desired format.
Read more

jquery - Clone last and append item to closest class -

-
this bare-bones code on page: <div class="list-container"> <div class="single-list-item"> <button>remove</button> </div> <div class="single-list-item"> <button>remove</button> </div> <div class="single-list-item"> <button>remove</button> </div> </div> <p><button class="add-list-item">add</button></p> <div class="list-container"> <div class="single-list-item"> <button>remove</button> </div> <div class="single-list-item"> <button>remove</button> </div> <div class="single-list-item"> <button>remove</button> </div> </div> <p><button class="add-list-item">add</button></p> <di
Read more

c - Unrecognised emulation mode: elf_i386 on MinGW32 -

-
i'm trying make kernel, , cannot link c output assembly. ld . i'm getting error: unrecognized emulation mode: elf_i386 i'm using windows 10 professional mingw32 , msys. code using: link.ld /* * link.ld */ output_format(elf32-i386) entry(start) sections { . = 0x100000; .text : { *(.text) } .data : { *(.data) } .bss : { *(.bss) } } kernel.c /* * kernel.c */ void kmain(void) { const char *str = "my first kernel"; char *vidptr = (char*)0xb8000; //video mem begins here. unsigned int = 0; unsigned int j = 0; /* loops clears screen * there 25 lines each of 80 columns; each element takes 2 bytes */ while(j < 80 * 25 * 2) { /* blank character */ vidptr[j] = ' '; /* attribute-byte - light grey on black screen */ vidptr[j+1] = 0x07; j = j + 2; } j = 0; /* loop writes string video memory */ while(str[j] != '\0') { /
Read more