hadoop para syseng

Hadoop para SysEngs

Type to enter text

Leonardo Martins leonardo.martins@wal-mart.com

História

Type to enter text

Leonardo Martins leonardo.martins@wal-mart.com

História

Type to enter text

Yahoo!!

Facebook!

Cloudera !

Hortonworks!

Walmart Brasil ;)

Leonardo Martins leonardo.martins@wal-mart.com

Hadoop?

Type to enter text

Leonardo Martins leonardo.martins@wal-mart.com

HDFS "data storage"

Type to enter text

Leonardo Martins leonardo.martins@wal-mart.com

Map Reduce "data analysis"

Type to enter text

Leonardo Martins leonardo.martins@wal-mart.com

HDFS - Design

Type to enter text

Very large files Low-latency data access

Streaming data access! Lots of small files!

Commodity hardware Multiple writers, arbitrary file modifications!

Leonardo Martins leonardo.martins@wal-mart.com

HDFS - Conceitos

Type to enter text

Leonardo Martins leonardo.martins@wal-mart.com

HDFS - Conceitos

Type to enter text

Leonardo Martins leonardo.martins@wal-mart.com

HDFS - Conceitos

Type to enter text

Leonardo Martins leonardo.martins@wal-mart.com

HDFS - Arquitetura

Type to enter text

Leonardo Martins leonardo.martins@wal-mart.com

HDFS - Arquitetura

Type to enter text

Leonardo Martins leonardo.martins@wal-mart.com

HDFS - Arquitetura

Type to enter text

Leonardo Martins leonardo.martins@wal-mart.com

HDFS - Arquitetura

Type to enter texthttp://www.hadoopwizard.com/which-big-data-company-has-the-worlds-biggest-hadoop-cluster/

HDFS - Escrita

Type to enter text

Leonardo Martins leonardo.martins@wal-mart.com

HDFS - Leitura

Type to enter text

Leonardo Martins leonardo.martins@wal-mart.com

HDFS - Sysadmin

Type to enter text

Adicionar e remover Data Node ao cluster!

Check de integridade do filesystem!

Balanceamento do cluster!

Adicionar e remover Task Tracker!

Monitorar os recursos do cluster!

Leonardo Martins leonardo.martins@wal-mart.com

MapReduce

Type to enter text

Leonardo Martins leonardo.martins@wal-mart.com

MapReduce

Type to enter text

Desenvolvimento mais simples!

Escala em paralelo!

Distribuição automática de tarefas!

Tolerância a falhas!

Leonardo Martins leonardo.martins@wal-mart.com

MapReduce - Arquitetura

Type to enter text

Leonardo Martins leonardo.martins@wal-mart.com

MapReduce - Submit

Type to enter text

MapReduce - Submit

Type to enter text

Leonardo Martins leonardo.martins@wal-mart.com

HDFS + MapReduce

Type to enter text

Task Tracker !

Job Tracker

Data Node !

Name Node

Ecosistema

Type to enter text

Leonardo Martins leonardo.martins@wal-mart.com

Possibilidades de uso

Type to enter text

Text mining Index building

Graph creation and analysis Pattern recognition

Prediction models Sentiment analysis

Cálculo de frete Cálculo de rota de caminhões!

Mecanismos de Antifraude

Leonardo Martins leonardo.martins@wal-mart.com

Quem usa?

Type to enter text

Type to enter text

Obrigado! !

Perguntas?

Leonardo Martins leonardo.martins@wal-mart.com