How to increase the Hadoop Heap Space

Re: What's the local heap size of Hadoop? How to increase it?

Jasmine (Xuanjing) Huang
Wed, 29 Apr 2009 20:56:05 -0700

Yeah, it works!
Bhupesh and Jason, thanks a lot!

Jasmine

----- Original Message ----- From: "jason hadoop"

To: 
Sent: Wednesday, April 29, 2009 11:19 PM
Subject: Re: What's the local heap size of Hadoop? How to increase it?

you can also specify it on the command line of your hadoop job
hadoop jar jarfile [main class] -D mapred.child.java.opts=-Xmx800M other
arguments
Note: there is a space between the -D and mapred, and the -D has to come
after the main class specification.

This parameter may also be specified  via
conf.set("mapred.child.java.opts","-Xmx800m"); before submitting the job.

On Wed, Apr 29, 2009 at 2:59 PM, Bhupesh Bansal wrote:

Hey ,

Try adding


 mapred.child.java.opts
 -Xmx800M -server


With the right JVM size in your hadoop-site.xml , you will have to copy
this
to all mapred nodes and restart the cluster.

Best
Bhupesh



On 4/29/09 2:03 PM, "Jasmine (Xuanjing) Huang" 
wrote:

> Hi, there,
>

> What's the local heap size of Hadoop? I have tried to load a local > cache > file which is composed of 500,000 short phrase, but the task failed. > The

> output of Hadoop looks like(com.aliasi.dict.ExactDictionaryChunker is a
> third-party jar package, and the records is organized as a trie
struction):
>
> java.lang.OutOfMemoryError: Java heap space
>         at java.util.HashMap.addEntry(HashMap.java:753)
>         at java.util.HashMap.put(HashMap.java:385)
>         at
> com.aliasi.dict.ExactDictionaryChunker$TrieNode.getOrCreateDaughter(Ex
> actDictionaryChunker.java:476)
>         at
> com.aliasi.dict.ExactDictionaryChunker$TrieNode.add(ExactDictionaryChu
> nker.java:484)
>

> When I reduce the total record number to 30,000. My mapreduce job > became

> succeed. So, I have a question, What's the local heap size of Hadoop's
Java
> Virtual Machine? How to increase it?
>
> Best,
> Jasmine
>

--
Alpha Chapters of my book on Hadoop are available
http://www.apress.com/book/view/9781430219422