nohup python start.py &Note: Please do not start a web server command from putty terminal with & at the end of the command. THE FOLLOWING MAY CAUSE unexpected website behave PROBLEM:<
command_to_start_a_web_server &
# untar tar -zvxf eclipse-standard-kepler-R-linux-gtk-x86_64.tar.gz # copy the uncompressed folder into /opt sudo cp -R eclipse /opt # change owner of all files to root sudo chown -R root:root /opt/eclipse2. link eclipse executable to
/usr/bin/
sudo ln -s /opt/eclipse/eclipse /usr/bin/eclipse3. Create the following .desktop file if you want to be able to pin eclipse to the luncher.
# create eclipse.desktop file at /usr/share/applications/ gedit /usr/share/applications/eclipse.desktop4. put the following contents in the file
[Desktop Entry] Name=Eclipse Type=Application Exec=/opt/eclipse/eclipse Terminal=false Icon=/opt/eclipse/icon.xpm Comment=Integrated Development Environment NoDisplay=false Categories=Development;IDE Name[en]=eclipse.desktopNow we are all set.
http://archive.apache.org/dist/hadoop/core/hadoop-1.1.2/hadoop-1.1.2.tar.gz1.2 untar the package to /opt 1.3 find the JAVA_HOME and HADOOP_HOME directory and add them to .bashrc file
echo export JAVA_HOME = /usr/lib/jvm/java-6-openjdk >> ~/.bashrc echo export HADOOP_PREFIX = /opt/Hadoop-1.1.2 >> ~/.bashrc1.4 add $HADOOP_PREFIX/bin to your $PATH variable
export PATH=$PATH:$HADOOP_PREFIX/bin export PATH=$PATH:$HADOOP_PREFIX/sbin2. Decide the running mode of your Hadoop:
local mode standard/local alone mode Pseudo-Disributed mode2.1 If you want to start Hadoop in Pseudo-Distributed mode, edit the following three configuration files at $HADOOP_PREFIX/conf, core-site.xml, hdfs-site.xml and mapred-site.xml
#core-site.xml <?xml version="1.0"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> <configuration> <property> <name>hadoop.tmp.dir</name> <value>/home/username/projects/hdfs_test</value> </property> <property> <name>fs.default.name</name> <value>hdfs://localhost:54310</value> </property> </configuration> #hdfs-site.xml <?xml version="1.0" encoding="UTF-8"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> <configuration> <property> <name>dfs.replication</name> <value>1</value> </property> </configuration> #mapred-site.xml <?xml version="1.0"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> <configuration> <property> <name>mapred.job.tracker</name> <value>localhost:54311</value> </property> <property> <name>mapred.system.dir</name> <value>/home/username/projects/mapred/system</value> <final>true</final> </property> </configuration>2.2 create file hadoop-env.sh in $HADOOP_PREFIX/etc/, and write the following to the file:
vim $HADOOP_PREFIX/conf/hadoop-env.sh export JAVA_HOME = /usr/lib/jvm/java-6-openjdk3. Format the HDFS system
hadoop namenode -format4. Start Hadoop
start-all.sh5. usefule comamnd to check the HDFS system
hadoop fs -ls /: list the root of HDFS hadoop fs -ls: (without '/'): # if you could not get anything other than the following error msg: #"ls: Cannot access .: No such file or directory.". #You need to do the following steps to make it work. hadoop fs -mkdir /user hadoop fs -mkdir /user/username # Now you should be able to run hadoop fs -ls because by default hadoop is looking for # "/user/username" structure within HDFS. The error msg is means there is no such structure in HDFS. #To avoide the error msg we need to create the structure within HDFS for Hadoop.6. copy files from local "linux" to HDFS I used two text files for testing: file1.txt and file2.txt, they are located at /home/username/projects/hadoop 6.1 copy three files from local to hadoop file system
hadoop dfs -copyFromLocal /home/username/projects/hadoop /user/username # check copy result hadoop dfs -ls /user/username/6.2 download the word-count example from http://repo1.maven.org/maven2/org/apache/hadoop/hadoop-examples/1.0.3/hadoop-examples-1.0.3.jar and put it /home/username/projects/hadoop 6.3 run the mapreduce job
hadoop jar hadoop-examples-1.0.3.jar wordcount /user/username /user/username-output # notes for this command: #(1) if you see io exception, you might use full path for the jar package #(2) path "/user/username-output" is the output path for mapreduce, it has not been there before running the job.6.4 Retrieve the job result from HDFS
# 1. merge mapreduce outputs and copy to local path: /home/username/projects/hadoop/output hadoop dfs -getmerge /user/username-output /home/username/projects/hadoop/output7. Hadoop Web Interfaces
NameNode daemon: http://localhost:50070/ JobTracker daemon: http://localhost:50030/ TaskTracker daemon: http://localhost:50060/7. Questions
sudo apt-get install apache2You may notice that http.conf file located at /etc/apach2/ is emplty, which is normal since we are using apache2.conf instead 2. Create symbolic links to enable the proxy modula in Apache2, then restart the server
sudo ln -s /etc/apache2/mods-available/proxy.load /etc/apache2/mods-enabled sudo ln -s /etc/apache2/mods-available/proxy_http.load /etc/apache2/mods-enabled sudo /etc/init.d/apache2 restart3. Create a virtual host file:
sudo gedit /etc/apache2/sites-enabled/proxiedhosts # and edit the file so that it resembles: NameVirtualHost *:80 # do not forget this line of code. <VirtualHost *:80> ServerName example.com ProxyRequests off ProxyPass / http://localhost:8080/ ProxyPassReverse / http://localhost:8080/ </VirtualHost>It is obviously we are forwarding any request on port 80 to port 8080 4.Activate the virtual host file by making a symbolic link to the Apache2 sites-enabled folder then restarting Apache2:
sudo ln -s /etc/apache2/sites-enabled/proxiedhosts /etc/apache2/sites-enabled sudo /etc/init.d/apache2 restart
# Print cat file_name | tr " " "\n"| sort | uniq # Count cat file_name | tr " " "\n" | sort | uniq -c # tr " " "\n" used here to replace space with newline, what it does is to put every word separated by space into a line
sed -n '/TITLE/p' history.txt | sed -n '/DVD/!p' | awk '{FS="/"; print $1 }' | awk '{FS=" "; print $2}'
By default sed will print anything from the original file. -n '/TITLE/p' will let sed print only lines contain 'TITLE'. -n '/DVD/!p' will further suppress lines contains 'DVD' awk '{FS="/"; print $1 } will remove the author information from titleawk '{FS=" "; print $2}' will print out only the title of the book.
echo 'export HISTTIMEFORMAT=" %F %T "' >> ~/.bashrc source ~/.bashrcYou will see time stamps on your commands when you execute
history next time.1013 2013-05-09 14:47:44 vim scripts_for_Distance_Matrix.txt 1014 2013-05-09 14:48:01 R < scripts_for_Distance_Matrix.txt --no-save
yum repolistIf you do not see EPEL in the output, then you have not enabled it yet. https://fedoraproject.org/wiki/EPEL 1. download the following package from fedoraproject.org
# The following link is for CentOS 5 wget http://dl.fedoraproject.org/pub/epel/5/x86_64/epel-release-5-4.noarch.rpm # check for and report potential conflicts rpm -ivh epel-release-5-4.noarch.rpm --testDuring test you might see the following NONKEY warning msg:
"warning: epel-release-5-4.noarch.rpm: Header V3 DSA signature: NOKEY, key ID 217521f6"wget http://fedoraproject.org/static/217521F6.txt mv 217521F6.txt /etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL sudo rpm --import /etc/pki/rpm-gpg/RPM-GPG-KEY-EPELVerify that the key got installed
rpm -qa gpg*Now run the test command again you should not see NOKEY msg:
sudo rpm -ivh epel-release-5-4.noarch.rpm --test2. Enable the repository:
sudo rpm -ivh epel-release-5-4.noarch.rpm
ssh-keygenBy default, your public key file is located at
~/.ssh/id_rsa.pub and it can be shared with anyone. Your private key file is located at ~/.ssh/id_rsa and it must be kept secret. ssh-copy-id -i ~/.ssh/id_rsa.pub username@host1.example.net
2. Manual Installation
# copy the key to the remote host scp ~/.ssh/id_rsa.pub user@remotehost:id_rsa.pub # ssh into the remote host using your password ssh user@remotehost # append the public key cat id_rsa.pub >> ~/.ssh/authorized_keys rm id_rsa.pubON WINDOWS You can download puttygen.exe to generate private and public key pairs. Copy public keys generated to your remote host
~/.ssh/authorized_keys file and make sure they are in ONE LINE.
#set 1 #displays all cuurentlly active network interfaces /sbin/ifconfig # IP address in the output of this command is you IP /sbin/ifconfig eth0 | grep 'inet addr:' #set 2 /sbin/ip addr list /sbin/ip addr list eth0
2. Copy file_name from local server to remote server: copy file_name from /srv/www on local server to remote server /srv/www/
scp -r user_name@remote_ip:/srv/www/pathogenbase/ .
scp /srv/www/file_name user_name@remote_ip:/srv/www/3. If you are copying files between windows and Linux you could use winSCP.
$ rsync -v -e ssh tommy@openbsd.nixcraft.in:~/webroot.txt /tmp $ rsync -r -a -v -e "ssh -l jerry" --delete openbsd.nixcraft.in:/webroot/ /local/webroot