Store and Register your EC2 machine image

To retrieve your machine image later, your machine image should be stored in S3. To do that, there are multiple steps required.

Step 1) Copy your X.509 certificate and private key to your instance.
scp -i id_rsa-gsg-keypair pk-HKZYKTAIG2ECMXYIBH3HXV4ZBZQ55CLO.pem cert-HKZYKTAIG2ECMXYIBH3HXV4ZBZQ55CLO.pem root@domU-12-34-31-00-00-05.compute-1.amazonaws.com:/mnt

This is done on your local desktop. Than, your security files will be stored in /mnt directory of your instance.

Step 2) Now you have to bundle your instance with the commmand:
ec2-bundle-vol -d /mnt -k /mnt/pk-HKZYKTAIG2ECMXYIBH3HXV4ZBZQ55CLO.pem -c /mnt/cert-HKZYKTAIG2ECMXYIBH3HXV4ZBZQ55CLO.pem -u 495219933132 -r i386 -p sampleimage

This takes some time. (several minutes) If it was successful, you can see the bundled files under your /mnt directory. They are bunch of files with names of smapleimage*.

Step 3) Upload bundled files to S3 from your instance
ec2-upload-bundle -b -m /mnt/sampleimage.manifest.xml -a -s

I had some problem with this process. Although I have S3 access, this command complained that I don't have S3 access. Interestingly, I created a bucket in S3 manually, and it worked.
I use the firefox extension that recommanded in a EC2 forum.
http://www.s3fox.net/
It is quite useful. If I want to sync part of my disk, it would be a great tool.

Step 4) Then, logout your instance, and on your desktop,
ec2-register /sampleimage.manifest.xml

Now you can see your machine image from,
ec2-describe-images -o self

Step 5) That's it! whenever you want to reload your machine image,
ec2-run-instances ami-5bae4b32

for the longer version of tutorial,
http://docs.amazonwebservices.com/AWSEC2/latest/GettingStartedGuide/

Getting started with EC2

To get start with EC2, you have to create an account in EC2 site.
After you successfully create your EC2 account from the amazon site, there are several steps to actually use EC2 instances.

Step1) Create X.509 certificate
Login to you Amazon account and AWS access identifiers page. In the X.509 certificate section, click Create New. Also Download your Certificate and store ot into your desktop safely.
mkdir .ec2
cd .ec2
mv ~/Desktop/*.pem .

Step2) Download EC2 command line tools. http://developer.amazonwebservices.com/connect/entry.jspa?externalID=351&categoryID=88
And unzip files under the ec2 directory.
mv ~/Desktop/ec2-api-tools.zip .
unzip ec2-api-tools.zip

Step3) Modify your shell script file. In my .bashrc file, I added,
export EC2_HOME=~/.ec2
export PATH=$PATH:$EC2_HOME/bin
export EC2_PRIVATE_KEY=pk-YOURKEYNAME.pem
export EC2_CERT=cert-YOURKEYNAME.pem

Step 4) Generate key pair to ssh to your instance
cd .ec2
ec2-add-keypair pstam-keypair > id_rsa-pstam-keypair

Step 5) Now select your new instance
ec2-describe-images -o amazon

Step 6) Create an instance
ec2-run-instances ami-6138dd08 -k pstam-keypair

Step 7) Check the description of the instance
Loading an instance takes some time. If you see the valid URL for the instance in the description of the instance, it's ready to access.
To check the description,
ec2-describe-instances

Step 8) Open the ports for ssh and http connection
You have to open some of the port you will allow to outside of the instances.
ec2-authorize default -p 22
ec2-authorize default -p 80

After this you can ssh to your instance.

Step 9) SSH to your instance
ssh -i id_rsa-pstam-keypair root@ec2-XXX-XXX-XXX-XXX.z-2.compute-1.amazonaws.com

Step 10) Getting a static IP address
For your application, sometimes you need a static IP address for your instance.
First you have to assign a static IP and tie the address to your instance.
ec2-allocate-address
ec2-associate-address -i i-yourinstance XXX.XXX.XXX.XXX

Step 11) Now you can SSH to your instance like to other remote machine.
ssh root@XXX.XXX.XXX.XXX

Step 12) Terminate Instance
ec2-terminate-instances i-yourinstance

Recent Change in TG

cobalt requires (queue=) in rsl string. It was optional previously. Now if you don't put either
standard or industrial, globus will through exception,
Globus error 37

Also condor will through exception,
Exec format error Code 6 Subcode 8

Submitting job to Ranger

Gatekeeper : gatekeeper.ranger.tacc.teragrid.org:2119/jobmanager-sge
Batch queue system: SGE
Check the queue status:showq -u


(Test 0) try locally installed executables with datafiles stored in scratch directory
./cap3 /scratch/00891/tg459282/2mil/cluster1807.fsa -p 95 -o 49 -t 100

(Test 1) try simple job submission through globus toolkit.
globusrun -o -r gatekeeper.ranger.tacc.teragrid.org:2119/jobmanager-sge '&(executable=/bin/ls)'

It works with swarm!

Submit and Manage Vanilla Condor Job to Swarm

To submit Vanilla condor job(s) to Swarm, first you have to install swarm and configure the service to process vanilla condor jobs. the source code is available from
svn co https://ogce.svn.sourceforge.net/svnroot/ogce/ogce-services-incubator

For the general information to install swarm, please refer,
http://decemberstorm.blogspot.com/2008/11/swarm-sever-installation-guide.html

Now, configure your swarm for the Vanilla condor pool.
Step 1. modify swarm/TGResource.properties, to process only vanilla condor pool. This is my properties:
matchmaking=FirstAvailableResource
taskQueueMaxSize = 100
taskQueueScanningInterval = 3000
condorCluster=true
teragridHPC=false
eucalyptus=false
condorCluster_numberOfToken=10
condorRefreshInterval = 2000

Step 2. start swarm server.
Upload swarm/build/jobsub.aar to your axis2 installation.

Step 3. Client example of vanilla condor job is stored at,
swarm/src/core/org/ogce/jobsub/clients/SubmitVanillaJob.java

There are three example methods:
submitJobWithStandardOutput(): job with standard output wihout input files
submitJobWithOutputTransfer(): job with standard output and output files
submitJobWithInputOutputTransfer(): job with standard output, output files, and local input files

Step 4. Compile example file
[swarm]ant clean
[swarm]ant

Step 5. Run example file
[swarm]./run.sh submit_vanilla_job

Step 6. Check the status
[swarm]cd ClientKit
[ClientKit]./swarm status http://serverIP:8080/axis2/services/Swarm your_ticket_ID

Step 7. Get the location of output file
[ClientKit]./swarm outputURL http://serverIP:8080/axis2/services/Swarm your_ticket_ID
This will return the URL of the output file.