Simple distributed wordcount with MapReduce¶

Check that file file.txt exists, view size.

In [ ]:

!ls -hal file.txt

Copy file to HDFS

In [ ]:

!hdfs dfs -put -f file.txt

Erase result folder.

In [ ]:

!hdfs dfs -rm -R result 2>/dev/null

Run the bash wordcount command wc in parallel on the distributed file.

In [ ]:

!mapred streaming \
  -input file.txt \
  -output result \
  -mapper /bin/cat \
  -reducer /usr/bin/wc 

Check result of MapReduce job

In [ ]:

!hdfs dfs -cat result/part*

Check that the word count is correct by comparing with wc on local host (warning: do not try with too large files).

In [ ]:

!wc file.txt