Installing Hadoop

This is my note written while I was following the installation documentation from hadoop's webpage.
http://hadoop.apache.org/core/docs/current/quickstart.html
I installed as a root, but I'm not sure if it is necessary.

Step 0. You have to have ssh, rsync, and java VM on your machine. I used,
1)ssh OpenSSH_4.3p2, OpenSSL 0.9.8b 04
2)rsync version 2.6.8
3)java 1.5.0_12

Step 1. Download software from a Hadoop distribution site.
http://hadoop.apache.org/core/releases.html

Step 2. Untar file

Step 3. reset the JAVA_HOME under your_hadoop_dir/conf/hadoop-env.sh
*note: I had JAVA_HOME defined in my .bashrc file. But I had to specify it again in the hadoop-env.sh.

Step 4. now you can just run your standalone operation as it is.
$ mkdir input
$ cp conf/*.xml input
$ bin/hadoop jar hadoop-*-examples.jar grep input output 'dfs[a-z.]+'
$ cat output/*

Step 5. For the Pseudo-Distributed Operation, which runs multiple virtual machines in single node so that it imitates real distributed file systems you have to set up the configuration in conf/hadoop-site.xml
The 'name' element is defined by hadoop system. Therefore, you can just use the names in the example from the hadoop page. I change the value of fs.default.name to hdfs://localhost:54310, and the one of mapred.job.tracker to localhost:54311.

Step 6. Check ssh localhost
In my case, I could not connect to localhost, but I could access to my numerical ip address. I changed my /etc/hosts.allow to have ALL:127.0.0.1 and it started to recognize localhost.
If it requires your passphrase:

$ ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa
$ cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys

If you still prefer use the passphrase, it will cause problem during the starting the daemons.

Step 7. Formatting namenode and starting the daemon

$ bin/hadoop namenode -format

$ bin/start-all.sh

Now you can check your namenode at http://localhost:50070/

Also your job tracker is available at http://localhost:50030/

Step 8. Test functions

Copy the input files into the distributed filesystem:
$ bin/hadoop fs -put conf input

Run some of the examples provided:
$ bin/hadoop jar hadoop-*-examples.jar grep input output 'dfs[a-z.]+'

Copy the output files from the distributed filesystem to the local filesytem and examine them:
$ bin/hadoop fs -get output output
$ cat output/*

View the output files on the distributed filesystem:
$ bin/hadoop fs -cat output/*

Step 9. Stop the daemon

$ bin/stop-all.sh

Update OGCE:file-manager portlet with new tacc classes

This update adds sorted view to the directory. JSP files and two java classes are modified: FileManagerConstants.java and FileManagerPortlet.java
The list of file updated is following.
portlets/comp-file-management/src/main/webapp/jsp/fileBrowser.jsp
portlets/comp-file-management/src/main/webapp/jsp/view.jsp
portlets/comp-file-management/src/main/webapp/css/fileManagement.css
portlets/comp-file-management/src/main/java/edu/tacc/gridport/portlets/interactive/FileManagementConstants.java
portlets/comp-file-management/src/main/java/edu/tacc/gridport/portlets/interactive/FileManagementPortlet.java
portlets/gp-common/src/main/webapp/jsp/fileBrowser.jsp - identical tofileBrowser.jsp above
portlets/gp-common/src/main/webapp/javascript/fileBrowser.js

* editted part is noted as "lukas edit"

Teragrid access end-to-end test

Teragrid sistes end-to-end test
- condorG (grid universe)
- cap3 apps
- stage output file
(08/22/2008 current)
==============================================================
[bigred] gatekeeper.iu.teragrid.org:2119/jobmanager-loadleveler yes
[steele] tg-steele.purdue.teragrid.org:2119/jobmanager-pbs yes
[sdsc(ds)] dslogin.sdsc.teragrid.org:2119/jobmanager-loadleveler job state write error
[mercury] https://grid-hg.ncsa.teragrid.org:2119/jobmanager-pbs yes
[ornl] tg-login.ornl.teragrid.org:2119/jobmanager-pbs yes
[lonestar] gatekeeper.lonestar.tacc.teragrid.org:2119/jobmanager-lsf job state read error
[cobalt] grid-co.ncsa.teragrid.org:2119/jobmanager-pbs yes
[pople] gram.pople.psc.teragrid.org:2119/jobmanager-pbs cannot login
[sdsc(dtf)]tg-login1.sdsc.teragrid.org:2119/jobmanager-pbs disk quota error

Limit of job submissiont?

I'm gathering the information about the limit of job submissions.
http://kb.iu.edu/data/awyt.html
http://kb.iu.edu/data/axal.html
NCSA(cobalt)
http://www.ncsa.uiuc.edu/UserInfo/Resources/Hardware/SGIAltix/Doc/Jobs.html
qstat -Q
BigRed
llclass

Java memory setup

To run Job submitssion service or Swarm service, please note that your environment variable should be set up as,
export JAVA_OPTS="-server -Xms512m -Xmx1024m -XX:MaxPermSize=256m"

Otherwise, your java virtual machine will provide 8M of memory which might be too small for running the server.

Access to the Job Submission Service with PHP using NuSOAP

If you are new to the PHP and NuSOAP, my previous posting would be helpful too.
Here is the simple example for the getStatus operation,
// Pull in the NuSOAP code
require_once('../lib/nusoap.php');
$client = new soapclient("http://your.service.location:8080/axis2/services/JobSubmissionService?wsdl",true);
$TaskId = array('clusterID' => "403", 'jobID' =>"0");
$taskId = array('TaskId' => $TaskId);
$getStatus = array('taskId'=>$TaskId);
$result = $client->call('getStatus',array('parameters'=> $getStatus),
'http://jobsubmissionservice.ogce.org/xsd','',false,null,'rpc','encoded');
if ($client->fault){
echo 'Fault';
}
print_r($result);

Creating a PHP client of the Web Service(Axis2) using NuSOAP

You can download NuSOAP library from the NuSOAP project is hosted by SourceForge. It is a package of php files you can keep in your php document directory or in a separate directory. In my case, I copied /lib directory from the NuSOAP project under /usr/local/apache2/htdocs/ which is default location of documents for the apache2 server. Then, copied /sample directory of the NuSOAP project under the same directory, /usr/local/apache2/htdocs/. You can download document from the same site. It contains nusoap method summary. Now, it's time to build a WS client..

Step1. pull in the NuSOAP code into your php source code
require_once('../lib/nusoap.php');

Step2. Create now client and set the WSDL as "true".
soapclient("http://service.location.url:8080/axis2/services/YourTargetService?wsdl",true);

Step3. Send the request using call() method
mixed call (string $operation, [mixed $params = array()], [string $namespace = 'http://tempuri.org'], [string $soapAction = ''], [mixed $headers = false], [boolean $rpcParams = null], [string $style = 'rpc'], [string $use = 'encoded'])
Please note that if you don't specify the namespace, your outgoing SOAP message will include SOAP Body with namespace of 'http://tempuri.org'. This will cause fault from the server.
Here my test request,
$result = $client->call('yourMethod',array('parameters'=> $yourType),
'http://yourservice.namespace.org/xsd','',false,null,'rpc','encoded');
Also, the $params which is arguments passed to the service is the element defined in the wsdl:message . If your service has hierarchical data type structure, you need to track down your WSDL carefully.

Step4. see the result..
I just used print_r($result)...