Amazon Customer Service Phone Numbers

Customer Service: 1 (888) 280-3321
Camera Concierge Team: 1 (877) 344-0195

Direct Link to Contact US: including phone, email and chat online

Windows 8 Time Always Wrong When Dual Boot With Ubuntu

I installed Ubuntu on a new Dell computer which has windows 8 preinstalled. The time of Windows 8 computer always 5 hours ahead after returning from a Ubuntu session. After a little research I found that windows 8 is no longer supporting UTC time out of the box. To fix the problem,I configured Ubuntu to use localtime. To do so,

sudo vim /etc/default/rcS
# change UTC=yes to UTC=no in this file under the following sentence in rcS file
# assume that the BIOS clock is set to UTC time (recommended)
UTC=no

Create multidimensional table in ReportLab

I was debugging a ReportLab program which was used to display a multidimensional table (Tables within table). I used zip(*) function to prepare the data and found out it is not right, since Zip(*) creates an array of tuples not an array of arrays. 1. Review of data structure for table in ReportLab: ReportLab use array of arrays to prepare data for table, for example, data for a 2 column and 4 row table can be written in the following format in reportlab:

[ # table starts
['col1 on row1','col2 of row1'], # row 1 of table
['col1 of row2', 'col2 of row2'], # row 2 of table
['col1 of row3', 'col2 of row3'], # row 3 of table
['col1 of row4', 'col2 of row4'] # row 4 of table
] # table ends

Follow the same structure we could build multidimensional table, a 4-layer-array data structure:

[ #parent table start
[ # row 1 of parent table
[['dance', 'office.jpg'], ['office', 'dance.jpg'], ['panda', 'lock.jpg'], ['lock', 'shape.jpg'], ['shape', 'panda.jpg']], # child table 1 in row 1 of parent
[['herd', 'germ.jpg'], ['dock', 'house.jpg'], ['house', 'shrink.jpg'], ['shrink', 'herd.jpg'], ['germ', 'dock.jpg']] # child table 2 in row 1 of parent
], # end of row 1 of parent table
[ # row 2 of parent table
[['gift', 'letter.jpg'], ['desk', 'desk.jpg'], ['drum', 'world.jpg'], ['world', 'drum.jpg'], ['letter', 'gift.jpg']], # child table 1 in row 2 of parent
[['bee', 'study.jpg'], ['magnet', 'robot.jpg'], ['study', 'bee.jpg'], ['learn', 'magnet.jpg'], ['robot', 'learn.jpg']] # child table 2 in row 2 of parent
] # end of row 2 of parent table
]# parent table end

Here is the table created based on the upper data structure.

2. Issues of zip(*) I used this command and got a list of tuples which is not good to feed to ReportLab:

zip(*[word,image])
[('dance', 'office.jpg'), ('office', 'dance.jpg'), ('panda', 'lock.jpg'), ('lock', 'shape.jpg'), ('shape', 'panda.jpg'), ('herd', 'herd.jpg')]

Then I tried the following code. It worked.

map(list, zip(*[word,image]))
[['dance', 'office.jpg'], ['office', 'dance.jpg'], ['panda', 'lock.jpg'], ['lock', 'shape.jpg'], ['shape', 'panda.jpg'], ['herd', 'herd.jpg']]

Install Eclipse Kepler On Ubuntu 12.04.3

Here are the simple steps, I used to update eclipse to Kepler on my Ubuntu 12.04.3 box. 1. download the appropriate package of eclipse from http://www.eclipse.org/downloads/ untar the package and copy it to the location you prefer, /opt in my case

# untar
tar -zvxf eclipse-standard-kepler-R-linux-gtk-x86_64.tar.gz
# copy the uncompressed folder into /opt
sudo cp -R eclipse /opt
# change owner of all files to root
sudo chown -R root:root /opt/eclipse

2. link eclipse executable to /usr/bin/

sudo ln -s /opt/eclipse/eclipse /usr/bin/eclipse

3. Create the following .desktop file if you want to be able to pin eclipse to the luncher.

# create eclipse.desktop file at /usr/share/applications/
gedit /usr/share/applications/eclipse.desktop

4. put the following contents in the file

[Desktop Entry]
Name=Eclipse
Type=Application
Exec=/opt/eclipse/eclipse
Terminal=false
Icon=/opt/eclipse/icon.xpm
Comment=Integrated Development Environment
NoDisplay=false
Categories=Development;IDE
Name[en]=eclipse.desktop

Now we are all set.

Install Nvidia Driver on Ubuntu Box

I noticed that Xorg process is taking too much of the CPU in my UBuntu box. I suspect that the video driver for my Nvidia graphy card may be not good. So I started to try other versions of drivers in hope to find a better one.

# obtain graphy card information
lspci | grep -i vga
03:00.0 VGA compatible controller: NVIDIA Corporation G98 [Quadro NVS 295] (rev a1)
04:00.0 VGA compatible controller: NVIDIA Corporation G98 [Quadro NVS 295] (rev a1)

Before installing I add the X Ubuntu Updates team PPA repository

sudo add-apt-repository ppa:ubuntu-x-swat/x-updates
sudo apt-get update

1. Removing all the old Nvidia drivers

sudo apt-get --purge remove nvidia-*

2. Install the nvidia drivers.

# verify which nvidia package are available
apt-cache search nvidia | sort
sudo apt-get install nvidia-current-updates
# After installation run the following command to initiate the configuration:
sudo nvidia-xconfig
sudo reboot

Upgrade Ubuntu 12.04.3 Kernel and x-stack from 3.5 to 3.8

#In order to install the new kernel, I did
sudo apt-get install --install-recommends linux-image-generic-lts-raring
#After install the new kernel I issue the following comamnd to install the compatible x-stack
sudo apt-get install --install-recommends xserver-xorg-lts-raring

Running Hadoop on Clusters of Two Nodes using Ubuntu and CentOS

Here is an example of running Hadoop on a dummy cluster. For references of how to set up Hadoop on one computer, please visit Install Hadoop 1.1.2 on Ubuntu 12.04.
Two Linux computers one is running Ubuntu and the other one is running CentOS.To test the cluster configuration I used Ubuntu as master and slave and CentOS as slave only. Here is a summary of what I did. Section One: Install and configuration

Daemons for master nodes (Ubuntu):
NameNode
SecondaryNameNode
DataNode
TaskTracker
JobTracker
Daemons for slave node (CentOS):
DataNode
TaskTracker

1. Download and install Hadoop on both computers, make sure they are intalled into the same locations (highly recommend) by following my previous post.
2. Edit /etc/hosts to add host names for each computer and make sure that you can ping each other by host names.
2.1 for the master node make sure that your host names is listed behind your external ip not 127.0.0.1, to make sure that DataNodes could find NameNode.
use sudo netstat -tuplen to verify. host file for master node/Ubuntu

127.0.0.1       localhost
20.14.0.102 ncw01111.test.com ncw01111 # adding here for preventing internal binding
# The following lines are desirable for IPv6 capable hosts
::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters

host file for slave node/CentOS

127.0.0.1       ncw02222 localhost.localdomain localhost
::1             localhost6.localdomain6 localhost6
20.14.0.102     ncw01111.test.com ncw01111 # I added master node ip and host name here when I have problem pinging master node. # at that time my Ubuntu host name has not been put into dns server

2.2 Set passwordless SSH for each computer, so they can talk to each other without password. 3. make configuration for master, Ubuntu in this case: 3.1 Edit master and slave file located in conf folder of the master node (NOTE: For master node only)

# Edit master file
sudo vim $HADOOP_PREFIX/conf/masters
# then add ncw01111 into the file
# Edit salve file
sudo vim $HADOOP_PREFIX/conf/slaves
# then add ncw01111 and ncw02222 into the files
# one host name per line

4. Edit configuration files: the configuration files, core-site.xml, mapred-site.xml and hdfs-site.xml are all the same for all the computers within the cluster. 4.1 core-site.xml

hadoop.tmp.dir
/home/bee/projects/hdfs_test


fs.default.name
hdfs://ncw01111:54310
true

4.2 mapred-site.xml

mapred.job.tracker
ncw01111:54311


mapred.system.dir
/home/bee/projects/mapred/system
true

4.3 hdfs-site.xml

dfs.replication
2

5. Format the HDFS file system

hadoop namenode -format

6. Start the cluster

start-all.sh
# or, use the following start up sequence
start-dfs.sh
start-mapred.sh

then use jsp check the daemons

# master node, Ubuntu
9700 SecondaryNameNode
10093 TaskTracker
9169 NameNode
10367 Jps
9808 JobTracker
9432 DataNode
#slave node, CentOS
26392 Jps
26201 TaskTracker
26095 DataNode

Please note, results from jps could not guarantee that your cluster has no problem. I had an issue that DataNode could not connect to NameNode (Problem 1 in the following problems and solution section ), I still could get the correct output from jps. Therefore, always look at your log files to see if there is any problem. if you see all your nodes of your cluster from here http://ncw01111:50030/machines.jsp?type=active your cluster might be good. 7. Put data into cluster and run a MapReduce job I used 99 years' NOAA data as an example, total 263G. They were preprocessed into a single file for each year, for example 1901.all. 7.1 put the data into cluster

gzip -c $year | hadoop fs -put - noaa/cluster/`basename $year.gz`

7.2 run the python version of the mapreduce on noaa data

# do not use relative paths in the following command (see problem and solution #2)
hadoop jar /opt/hadoop-1.1.2/contrib/streaming/hadoop-streaming-1.1.2.jar -mapper /home/bee/projects/hadoop/max_temperature_map.py -reducer /home/bee/projects/hadoop/max_temperature_reduce.py -input /user/bee/noaa/cluster/ -output /user/bee/noaa/cluster_out

7.3 Running results comparison for 99 years' NOAA data. 7.1 Time respect Single node (Ubuntu only): Finished in: 59mins, 42sec Two Nodes Cluster (Ubuntu and CentOS): Finished in: 60mins, 12sec Two Nodes Cluster (Ubuntu and CentOS and using combiner): Finished in: 34mins, 12sec 7.2 Check results in HDFS

hadoop dfs -tail noaa/cluster_out/part-00000
1901    317
1902    244
...
1999    568

7.3 Retrieve the job result from HDFS

hadoop dfs -copyToLocal noaa/cluster_out/part-00000 /home/bee/projects/hadoop/output/cluster_out

Section 2: Problem and Solution
1. Error message: 2013-09-05 15:12:26,822 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: ncw01111/20.14.0.102:54310. Already tried 0 time(s); retry policy is RetryUpToMax imumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
Symptom:
when you check your port opening with netstat you found that your port 54310 is binding to a internal ip (127.0.0.1) tcp6 0 0 127.0.0.1:54310 :::* LISTEN 1001 624829 24900/java
Solution:
check you /etc/hosts file make sure that your hostname is not listed after 127.0.0.1. or 127.0.1.1
Put your external IP and hostname pair in hosts file if it doe not exist. Re-check the port now: tcp6 0 0 220.14.24.75:54310 :::* LISTEN 1001 624829 24900/java

2. Error message: # of failed Map Tasks exceed allowed limit error. # ERROR streaming.StreamJob: Job not successful. Error: # of failed Map Tasks exceeded allowed limit. FailedCount: 1. error.
Solution:
Go to master:50030/jobdetails.jsp?jobid=XXXXX(You can find it from your Hadoop output) to look for the real error. After debugging I found an IO error in my case.
Fix: Fix any relative paths in your job request with absolute paths when you using Hadoop streaming with python mapper and reducer.

Set Static IP for Ubuntu 12.04

1. Edit /etc/network/interfaces and put the IP address along with other information

auto lo eth0
iface lo inet loopback
iface eth0 inet static
address 192.14.0.102
netmask 255.255.224.0
gateway 192.14.0.1
dns-search example.com
dns-nameservers 192.14.0.194 192.14.0.180

2. use netstat -r to verify that you have default route.

If no, you need add one by using:

sudo route add default gw [ip_address_for_default_gateway]

3.check whether you have entries in /etc/resolve.conf file.For example,

nameserver 192.14.0.194
nameserver 192.14.0.180
search example.com EXAMPLE.COM

Install Hadoop 1.1.2 on Ubuntu 12.04

Here is the example of installation of Hadoop on a single machine, UBuntu box in this case. For a simple cluster installation, please reference to Running Hadoop on Clusters of Two Nodes using Ubuntu and CentOS.
1. install hadoop 1.1 download Hadoop-1.1.2 (STABLE VERSION OF THE HADOOP)

http://archive.apache.org/dist/hadoop/core/hadoop-1.1.2/hadoop-1.1.2.tar.gz

1.2 untar the package to /opt 1.3 find the JAVA_HOME and HADOOP_HOME directory and add them to .bashrc file

echo export JAVA_HOME = /usr/lib/jvm/java-6-openjdk >> ~/.bashrc
echo export HADOOP_PREFIX = /opt/Hadoop-1.1.2 >> ~/.bashrc

1.4 add $HADOOP_PREFIX/bin to your $PATH variable

export PATH=$PATH:$HADOOP_PREFIX/bin
export PATH=$PATH:$HADOOP_PREFIX/sbin

2. Decide the running mode of your Hadoop:

local mode
standard/local alone mode
Pseudo-Disributed mode

2.1 If you want to start Hadoop in Pseudo-Distributed mode, edit the following three configuration files at $HADOOP_PREFIX/conf, core-site.xml, hdfs-site.xml and mapred-site.xml

#core-site.xml
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
<name>hadoop.tmp.dir</name>
<value>/home/username/projects/hdfs_test</value>
</property>
<property>
<name>fs.default.name</name>
<value>hdfs://localhost:54310</value>
</property>
</configuration>
#hdfs-site.xml
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
</configuration>
#mapred-site.xml
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
<name>mapred.job.tracker</name>
<value>localhost:54311</value>
</property>
<property>
<name>mapred.system.dir</name>
<value>/home/username/projects/mapred/system</value>
<final>true</final>
</property>
</configuration>

2.2 create file hadoop-env.sh in $HADOOP_PREFIX/etc/, and write the following to the file:

vim $HADOOP_PREFIX/conf/hadoop-env.sh
export JAVA_HOME = /usr/lib/jvm/java-6-openjdk

3. Format the HDFS system

hadoop namenode -format

4. Start Hadoop

start-all.sh

5. usefule comamnd to check the HDFS system

hadoop fs -ls /: list the root of HDFS
hadoop fs -ls: (without '/'):
# if you could not get anything other than the following error msg:
#"ls: Cannot access .: No such file or directory.".
#You need to do the following steps to make it work.
hadoop fs -mkdir /user
hadoop fs -mkdir /user/username
# Now you should be able to run hadoop fs -ls because by default hadoop is looking for
# "/user/username" structure within HDFS. The error msg is means there is no such structure in HDFS.
#To avoide the error msg we need to create the structure within HDFS for Hadoop.

6. copy files from local "linux" to HDFS I used two text files for testing: file1.txt and file2.txt, they are located at /home/username/projects/hadoop 6.1 copy three files from local to hadoop file system

hadoop dfs -copyFromLocal /home/username/projects/hadoop /user/username
# check copy result
hadoop dfs -ls /user/username/

6.2 download the word-count example from http://repo1.maven.org/maven2/org/apache/hadoop/hadoop-examples/1.0.3/hadoop-examples-1.0.3.jar and put it /home/username/projects/hadoop 6.3 run the mapreduce job

hadoop jar  hadoop-examples-1.0.3.jar wordcount /user/username /user/username-output
# notes for this command:
#(1) if  you see io exception, you might use full path for the jar package
#(2) path "/user/username-output" is the output path for mapreduce, it has not been there before running the job.

6.4 Retrieve the job result from HDFS

# 1. merge mapreduce outputs and copy to local path: /home/username/projects/hadoop/output
hadoop dfs -getmerge /user/username-output /home/username/projects/hadoop/output

7. Hadoop Web Interfaces

NameNode daemon: http://localhost:50070/
JobTracker daemon: http://localhost:50030/
TaskTracker daemon: http://localhost:50060/

7. Questions
Q. $HADOOP_HOME is depreciated always showing up when you run hadoop commands.
A. Replace HADOOP_HOME in your ~/.bashrc file with HADOOP_PREFIX. Check whether $HADOOP_HOME is defined in other places using echo command when you open new terminal.

Use Apache Virtual Host/ Reverse Proxy on Ubuntu

1. install apache2 on Ubuntu

sudo apt-get install apache2

You may notice that http.conf file located at /etc/apach2/ is emplty, which is normal since we are using apache2.conf instead 2. Create symbolic links to enable the proxy modula in Apache2, then restart the server

sudo ln -s /etc/apache2/mods-available/proxy.load /etc/apache2/mods-enabled
sudo ln -s /etc/apache2/mods-available/proxy_http.load /etc/apache2/mods-enabled
sudo /etc/init.d/apache2 restart

3. Create a virtual host file:

sudo gedit /etc/apache2/sites-enabled/proxiedhosts
# and edit the file so that it resembles:
NameVirtualHost *:80  # do not forget this line of code.
<VirtualHost *:80>
ServerName example.com
ProxyRequests off
ProxyPass / http://localhost:8080/
ProxyPassReverse / http://localhost:8080/
</VirtualHost>

It is obviously we are forwarding any request on port 80 to port 8080 4.Activate the virtual host file by making a symbolic link to the Apache2 sites-enabled folder then restarting Apache2:

sudo ln -s /etc/apache2/sites-enabled/proxiedhosts /etc/apache2/sites-enabled
sudo /etc/init.d/apache2 restart

Slow Authentication when SSH Connecting To Your Ubuntu Box

I noticed a dramatic slower authentication process when doing ssh connection to my Ubuntu box than connecting to CentOS box.

time ssh 102.14.98.241

After logging in, type exit and it reveal the time consumed as the following:

real 0m30.397s
user 0m0.012s
sys  0m0.020s

After little research I found modify one line in /etc/ssh/sshd_config could fix the problem.

sudo vim  /etc/ssh/sshd_config
# add the following line
UseDNS no
#save file and restart sshd server
sudo /etc/init.d/ssh restart

Test the new time consuming, use time ssh 102.14.98.241 shows

real 0m5.309s
user 0m0.013s
sys  0m0.015s

Configure sqlldr for batch uploading data to Oracle

I was trying to batch load data to Oracle from client site of computer (Ubuntu 64bit) and found that sqlldr was not installed. Here is a simple instruction of installing/enabling sqlldr on Ubuntu 64Bit.
1. make sure that Oracle Client was installed. If not, following this link to install.
2. Download Oracle Database Express Edition (oracle-xe-11.2.0-1.0.x86_64.rpm.zip) from here:
3. extract rpm file from the downloaded file

unzip -d oracle-ex oracle-xe-11.2.0-1.0.x86_64.rpm.zip
cd oracle-ex/Disk1
# extract files from rpm package
rpm2cpio oracle-xe-11.2.0-1.0.x86_64.rpm | cpio -idmv
# copy file sqlldr and folder mesg to local $ORACLE_HOME
sudo cp u01/app/oracle/product/11.2.0/xe/bin/sqlldr /usr/lib/oracle/11.2/client64/bin
# create fold rdbms/mesg at $ORACLE_HOME
sudo mkdir -p /usr/lib/oracle/11.2/client64/rdbms/mesg
#then copy files from mesg folder
sudo cp u01/app/oracle/product/11.2.0/xe/rdbms/mesg/* /usr/lib/oracle/11.2/client64/rdbms/mesg/

4. now you could run sqlldr to batch load data

sqlldr user/pass@//oracle_server:1521/devl control="RXNCONSO.ctl";

Install cx_Oracle on Ubuntu from rpm package

Make sure that you have installed Oracle instant client. If not, please follow this post.
1. make sure that the path in your oracle.conf file contains libclntsh.so.11.1 file

cat /etc/ld.so.conf.d/oracle.conf
#it shows
/usr/lib/oracle/11.2/client64/lib
# verify that the path contains libclntsh.so.11.1
ll /usr/lib/oracle/11.2/client64/lib | grep libclntsh
lrwxrwxrwx 1 root root        17 Aug  5  2013 libclntsh.so -> libclntsh.so.11.1
-rw-r--r-- 1 root root  52761218 Sep 17  2011 libclntsh.so.11.1

2. update ld.so.cache

sudo ldconfig

3. download and install cx_Oracle library from rpm package

sudo alien -i cx_Oracle-5.1.2-11g-py27-1.x86_64.rpm

4. Configure cx_Oracle to work with Python2.7 within Ubuntu

cd /usr/lib/python2.7
sudo mv site-packages/cx_Oracle* dist-packages/
sudo rmdir site-packages/
sudo ln -s dist-packages site-packages

Install Oracle SQL Plus on Ubuntu

Recently, I switched my development environment from CentOS to Ubuntu. I have difficulties of running Oracle SQL Plus. The installation went fine by following my previous post. However, when I tried to run sqlplus,it complains that libsqlplus.so is missing "error while loading shared libraries: libsqlplus.so": Here is the solution I used to fix the problem: Create a file oracle.conf in /etc/ld.so.conf.d

# create file oracle.conf
sudo vim /etc/ld.so.conf.d/oracle.conf
# add the following line to the file
/usr/lib/oracle/11.2/client64/lib
# then run ldconfig to update the configuration
sudo ldconfig

But when you run sqlplus again you may find it shows another error: "missing libaio.so.1, to fix that you need install libaio1"

sudo apt-get install libaio1

Now you may be able to connect to Oracle:

sqlplus user/password@//oracle_server:port/SID

Install rpm Packages in Ubuntu

If rpm package is all you got for a software package, you could follow the following instructions to install it on Ubuntu:

# install alien and all the dependencies it needs
sudo apt-get install alien dpkg-dev debhelper build-essential
# Convert a package from rpm to debian format
sudo alien packagename.rpm
# Use dpkg to install the package
sudo dpkg -i packagename.deb

Enable alt for command line editing

When I first switched from CentOS to Ubuntu I noticed that Alt+f and Alt+b do not work in command line editing. Instead it opens terminal menu...
Here comes how to enable them

Open a terminal >> Go to Edit (on terminal menu) >> Keyboard Shortcuts >> Uncheck "Enable menu access keys (such as ....)"

Install Flash Player in Ubuntu

Can not watch youtube video in Ubuntu? Here is a solution:

sudo apt-get install flashplugin-installer

Enable print to pdf in Ubuntu

apt-get install cups-pdf

The printed pdf file will be located at ~/PDF/

Let Cron Send Email Only When Something Goes Wrong

Tired of getting too many emails from cron daemon? Here is a simple solution of letting cron send you an email when something goes wrong.

sudo crontab -e

Set "MAILTO=" on top of yout first command to be executed. And add ">/dev/null" to the end of each command. Here is my example:

#run emen2 local backup everyday at 1am and sync with production server everyday at 3am
MAILTO=your_email_address
0 1 * * * /home/user/cronjobs/emen2_backup.sh > /dev/null
0 3 * * * /home/user/cronjobs/syncserver.sh > /dev/null

Restart cront using command

sudo /etc/init.d/cron restart

Here is a very good example of explaing what /dev/null mean?

Print/Count Unique Words in a File Using Command Line

Here is a one-line-command used to print/count unique words in a file:

# Print
cat file_name | tr " " "\n"| sort | uniq
# Count
cat file_name | tr " " "\n" | sort | uniq -c
# tr " " "\n" used here to replace space with newline, what it does is to put every word separated by space into a line

sed and awk

Use sed and awk you could print certain lines that meet your query condition.
Here is a command I used to print book titles of my son's reading history from a text file. From the picture, we can tell that book titles are mixed with DVD titles and the author names are listed after titles if there is any.

My task is to pick up titles of books only. Here sed and awk comes handy.

sed -n '/TITLE/p' history.txt | sed -n '/DVD/!p' | awk '{FS="/"; print $1 }' | awk '{FS="        "; print $2}'

By default sed will print anything from the original file.
-n '/TITLE/p' will let sed print only lines contain 'TITLE'.
-n '/DVD/!p' will further suppress lines contains 'DVD'
awk '{FS="/"; print $1 } will remove the author information from title
Further investigate reveal that there are 8 spaces between TITLE and the title of the book,
awk '{FS=" "; print $2}' will print out only the title of the book.

Autostart application after CentOS reboots

If you only want to run a script after CentOS boots, you could put the command to rc.local file.

vim /etc/rc.d/rc.local
# type in your script at the bottom of this file

If you want to start and shutdown the program along with the system, you might need to create a service using chkconfig. To do so,

1.create a script file at /etc/init.d/
Here is an example of the content of the file

# create file( you need root privilege to do)
cd /etc/init.d/
vim emen2

and add the followings to the file

#!/bin/sh
#chkconfig: 235 80 05
#description: EMEN2 database server
#source function library
EMEN2_HOME=/srv/EMAN2
if [ ! -f $EMEN2_HOME/eman2.bashrc ]
then
  echo "EMEN2 startup: cannot start, missing eman2.bashrc"
  exit
fi
case "$1" in
  "start")
    echo -n "Start emen2 service"
    source $EMEN2_HOME/eman2.bashrc
    # To run it as root
    $EMEN2_HOME/Python/bin/emen2ctl start -h /srv/db_env -e default,site,em --port 8080
    # Or to run it as dedicated user
    /bin/su - username -c $EMEN2_HOME/Python/bin/emen2start
    echo "."
    ;;
  "stop")
    echo - n "Stop emen2 service"
    source $EMEN2_HOME/eman2.bashrc
    # To run it as root
    $EMEN2_HOME/Python/bin/emen2ctl stop -h /srv/db_env -e default,site,em --port 8080
    # Or to run it as dedicated user
    /bin/su - username -c $EMEN2_HOME/Python/bin/emen2stop
    echo "."
    ;;
  *)
    echo "Usage: /sbin/service emen2 {start|stop}"
    exit 1
esac
exit 0

**About the three values after #chkconfig
(1). The first value ’235′ states the default runlevels at which the service should be started. In this case, the service is configured to start at runlevel 2,3 & 5.
(2). The second value ’80′ states the start priority of the service. In this case the service shall be started after starting all those services having start priority less than 80.
(3). The third value ’05′ states the stop priority of the service. In this case the service shall be stopped after stopping all those services having stop priority less than 05.

**/bin/su -c can only accept simple command, which means there should be no "-" or "--" in the command.

2. make the file runnable:

chmod a+x emen2

3. add the newly created service to the start up list by

/sbin/chkconfig/emen2 on

Use wget to detect broken links in a website

while I was updating a website developed using Django, found out that some links were broken due to the change of Django version. To locate all broken links, I used the following commands:

wget --spider -r -l 1 --header='User-Agent: Mozilla/19' -o broken_links.txt http://example.com/updates/

Then I used grep to see whether 404 error was appearing in the output file.

grep -B 2 '404' broken_links.txt
# -B 2 :Print 2 lines of leading context before matching lines which contains the broken link

Install perl DBD::Oracle module without installing Oracle server onCentOS

I am in the situation of using ora2pg to migrate an Oracle database to postgreSQL. To install ora2pg, DBD::Oracle is required. To install DBD:Oracle, the environmental variable $ORACLE_HOME is required. However, it is not easy to set up an environmental variable for $ORACLE_HOME since I am using a remote dedicated Oracle server. To overcome the problem, I downloaded and installed the following three rpm packages: Please be advised that it should be no problem to set $ORACLE_HOME on a machine which happens to be the Oracle server.

# download link for Linux 64bit
http://www.oracle.com/technetwork/topics/linuxx86-64soft-092277.html
oracle-instantclient11.2-basic-11.2.0.3.0-1
oracle-instantclient11.2-devel-11.2.0.3.0-1
oracle-instantclient11.2-sqlplus-11.2.0.3.0-1

I then added the $ORACLE_HOME to ~/.bashrc file.

#By default, the installation path is /usr/lib/oracle/11.2/client64/
export ORACLE_HOME="/usr/lib/oracle/11.2/client64/"

If you want connect to the Oracle database from command line using sqlplus, you could add $ORACLE_HOME/bin to you PATH, though the syntax for connecting to a remote database using its SID is nasty.

PATH=$PATH:/usr/lib/oracle/11.2/client64/bin
sqlplus AERS/AERS@'(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=remote_server_address)(PORT=1521)))(CONNECT_DATA=(SID=sid_code)))'

Now, you have done all the steps for running sqlplus on a command line. Will you enjoy the using? You probably not at this moment. Since running sqlplus without further tweaking is a pain in Linux command line: you do not have your history(using upper arrow) or you can not use left/right arrow to edit your sql command. Here is a solution, using rlwrap

# install required repository
wget http://www.mirrorservice.org/sites/dl.fedoraproject.org/pub/epel/6/i386/epel-release-6-8.noarch.rpm
sudo rpm -Uvh epel-release-6-8.noarch.rpm
# install rlwrap if it does not exit in your system
sudo yum install rlwrap
#then run
rlwrap sqlplus ***

You could also add he following alias in your ~/.bashrc file

vim ~/.bashrc
#add the following into the file
alias sqlplus='rlwrap sqlplus'

Install SQL Developer on CentOS 5

1. Configure $JAVA_HOME if there is nothing outputs when you run

echo $JAVA_HOME

Mine is at /usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64/ so I put the following line into ~/.bashrc file:

export JAVA_HOME="/usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64/"

2. Download SQL Developer installation from Oracle SQL Developer RPM for Linux and install.

sudo rpm -ihv sqldeveloper-3.2.20.09.87-1.noarch.rpm

3. add /opt/sqldeveloper to your PATH

PATH=$PATH:/opt/sqldeveloper/

4. Run SQL Developer:

source ~/.bashrc
sqldeveloper

Show Time Stamp on Bash History

Ever wondered when did I execute the command? You could add a time stamp in front of each command you executed.

echo 'export HISTTIMEFORMAT=" %F %T "' >> ~/.bashrc
source ~/.bashrc

You will see time stamps on your commands when you execute history next time.
The result will look like:

1013 2013-05-09 14:47:44 vim scripts_for_Distance_Matrix.txt
1014 2013-05-09 14:48:01 R < scripts_for_Distance_Matrix.txt --no-save

Auto lock windows 7 by removing smart card

Have you ever wanted to lock your windows 7 work station automatically after you pull out your smart card/piv card from card reader? Here is an example of how to set it up.

1. Make sure that "Interactive logon: Smart card removal behavior" is configured with "Lock Workstation".
To set it up,
run gpedit.msc to open Local Group Policy Editor and locate: Computer Configuration\Windows Settings\Security Settings\Local Policies\Security Options. Locate "Interactive logon: Smart card removal behavior" on the right panel, double click it ans select "Lock Workstation".

2. Set "Smartcard Removal Policy" to "Automatic"
In order to make changes made in first step take effect, we have to set the startup type for "Smartcard Removal Policy" service to "Automatic".
Right click "My Computer" logo and choose "Manage" and go to "Services and Applications"", "Services" to change the startup type to automatic.

Keep program running after you close putty session to your CentOS server

If you want to keep your program running after you close your putty session to your CentOS server, you could use the following method: 1. install screen

sudo yum install screen

2. start screen then type whatever process you want to keep running

screen
# the command you want to keep running

3. on keyboard issue ctrl-d 4. exist putty session 5. when you logon next time, you could restore the previous session by doing

screen -r

Please refer here for an alternative solution.

Sourcecode syntaxhighlighter in wordpress

If you want to format source code in your wordpress blog, you could try the "sourcecode" tag.

[sourcecode language="python"]
your code here
[/sourcecode]

Supported languages are bash, python, java, JavaScript, perl etc.
The biggest benefits of doing so is to avoid change "--" to '.' automatically by wordpress when you copy commands from your post.

Install Berkeley DB-5.3.21 on Centos

Berkeley DB source file at Oracle. Install optional dependence: Tcl-8.6.0, OpenJDK-1.7.0.9, and Sharutils-4.13.3 (for the uudecode command)

sudo yum install tcl
sudo yum install java-1.7.0-openjdk java-1.7.0-openjdk-devel
sudo yum install sharutils

Install Berkeley DB

wget http://download.oracle.com/berkeley-db/db-5.3.21.tar.gz
tar zvxf db-5.3.21.tar.gz 
cd db-5.3.21/build_unix
# configure the program
../dist/configure --prefix=/opt --enable-compat185 --enable-dbm --disable-static --enable-cxx
make
# install
sudo make docdir=/usr/share/doc/db-5.3.21 install
sudo chown -v -R root:root /usr/bin/db_* /usr/include/db{,_185,_cxx}.h /usr/lib/libdb*.{so,la} /usr/share/doc/db-5.3.21

Questions:
when you run db_recover or other commands, you may encounter "libdb-5.3.so: cannot open shared object file: No such file or directory" Answer:

sudo /sbin/ldconfig

Enable EPEL Repository on CentOS

Extra Packages for Enterprise Linux (or EPEL) provide much more current versions of popular applications like PHP or MYSQL. Before starting you need to check check your current repositories and verify that you have not enabled EPEL

yum repolist

If you do not see EPEL in the output, then you have not enabled it yet. https://fedoraproject.org/wiki/EPEL 1. download the following package from fedoraproject.org

# The following link is for CentOS 5
wget http://dl.fedoraproject.org/pub/epel/5/x86_64/epel-release-5-4.noarch.rpm
# check for and report potential conflicts
rpm -ivh epel-release-5-4.noarch.rpm --test

During test you might see the following NONKEY warning msg: "warning: epel-release-5-4.noarch.rpm: Header V3 DSA signature: NOKEY, key ID 217521f6"
Download the specific GPG key for EPEL repository from fedoraproject.org/keys by matching the key ID.
In my case I need key with ID 217521f6.

wget http://fedoraproject.org/static/217521F6.txt
mv 217521F6.txt /etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL
sudo rpm --import /etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL

Verify that the key got installed

rpm -qa gpg*

Now run the test command again you should not see NOKEY msg:

sudo rpm -ivh epel-release-5-4.noarch.rpm --test

2. Enable the repository:

sudo rpm -ivh epel-release-5-4.noarch.rpm

Port Forwarding in CentOS Using Iptables

Port forwarding could be useful when you do not want to you client to put a port nuimber after you web address. In my case I forward port 8080 to 80. 1. check if IP forwarding is enabled:

/sbin/sysctl net.ipv4.ip_forward

if return net.ipv4.ip_forward = 1 then it is enabled. if not edit /etc/sysctl.conf and set net.ipv4.ip_forward = 1.
2. COnfigure and restart iptables

/sbin/sysctl -p /etc/sysctl.conf
/sbin/service iptables restart

3. Adding IP forwarding rules to IpTables

/sbin/iptables -I FORWARD 1 -m state --state ESTABLISHED,RELATED -j ACCEPT
/sbin/iptables -I FORWARD 1 -p tcp --dport 80 -j ACCEPT
/sbin/iptables -t nat -A PREROUTING -p tcp --dport 80 -j DNAT --to-destination 10.0.1.1:8080
/sbin/iptables -t nat -A POSTROUTING -j MASQUERADE
/sbin/service iptables save
/sbin/service iptables restart

The content of /etc/sysconfig/iptables files looks like:

# Generated by iptables-save v1.3.5 on Fri Mar 8 10:27:21 2013
*nat
:PREROUTING ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
-A PREROUTING -p tcp -m tcp --dport 80 -j DNAT --to-destination 10.0.1.1:8080
-A POSTROUTING -j MASQUERADE
COMMIT
# Completed on Fri Mar 8 10:27:21 2013
# Generated by iptables-save v1.3.5 on Fri Mar 8 10:27:21 2013
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [234:26336]
:RH-Firewall-1-INPUT - [0:0]
-A INPUT -j RH-Firewall-1-INPUT
-A FORWARD -p tcp -m tcp --dport 80 -j ACCEPT
-A FORWARD -m state --state RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -j RH-Firewall-1-INPUT
-A RH-Firewall-1-INPUT -i lo -j ACCEPT
-A RH-Firewall-1-INPUT -p icmp -m icmp --icmp-type any -j ACCEPT
-A RH-Firewall-1-INPUT -p esp -j ACCEPT
-A RH-Firewall-1-INPUT -p ah -j ACCEPT
-A RH-Firewall-1-INPUT -p udp -m udp --dport 631 -j ACCEPT
-A RH-Firewall-1-INPUT -p tcp -m tcp --dport 631 -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
....
....
....
-A RH-Firewall-1-INPUT -j REJECT --reject-with icmp-host-prohibited
COMMIT

SSH Public Key Authentication

Generating Keys 

To generate the public and private key files, you can use ssh-keygen command which is installed with OpenSSH.

ssh-keygen

By default, your public key file is located at ~/.ssh/id_rsa.pub and it can be shared with anyone. Your private key file is located at ~/.ssh/id_rsa and it must be kept secret.

Key Installation 

You now need to copy your public key file to the remote host(s) that you want to be able to use public key authentication.
1. Copy your public key to remote host1.example.net using one simple command:

ssh-copy-id -i ~/.ssh/id_rsa.pub username@host1.example.net

2. Manual Installation
To install the key manually you need to append the key to the ~/.ssh/authorized_keys file on each host you wish to use.

# copy the key to the remote host
scp ~/.ssh/id_rsa.pub user@remotehost:id_rsa.pub
# ssh into the remote host using your password
ssh user@remotehost
# append the public key
cat id_rsa.pub >> ~/.ssh/authorized_keys
rm id_rsa.pub

ON WINDOWS You can download puttygen.exe to generate private and public key pairs. Copy public keys generated to your remote host ~/.ssh/authorized_keys file and make sure they are in ONE LINE.

Cron Jobs

Usefult summaries for crontabs

#List the current running cron jobs for currently logged in user:
crontab -l
#remove the jobs from crontab 
#It is a good practice to do so before modifying your script
crontab -r
#add cron jobs
crontab /path/to/backup_script.cron

Modify other user's cron job 

You have to make sure that another user's user name was listed in /etc/cron.allow file.

#Add cron job for another user
crontab -u username -e
#list another user's cron job
xrontab -u username -l

Examples

#run command_1 every 5 minutes (do not miss / in */5)
*/5 * * * * /path/to/command_1
#run command_2 every 5 hours (do not miss / in */5)
0 */5 * * * /path/to/command_2
# run command_3 1:30am everyday
30 1 * * * /path_to_command_3
# run command_4 1pm everyday
0 13 * * * /path_to_command_4

Why Crontab Fails running my command?

Short answer is because of different running environment for command wish to be executed by cron. Cron passes a minimal set of environment variables to your jobs. To see the difference, add a dummy job like this:

* * * * * env > /tmp/env.output

Restart crontab and wait for /tmp/env.output to be created, then remove the job again. Now compare the contents of /tmp/env.output with the output of env running in your regular terminal. The big differences is PATH environment variable. To get around that, set your own PATH variable at the top of the script. E.g.

#!/bin/bash
PATH=/opt/someApp/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
# rest of scron script follows

Can not view man page on CentOS

I have been having issues with viewing man page when I logged in as a regular user. Everytime, I have to add sudo in front of man command. After troubleshooting the problem for a while, I found a solution:

Add execute permission for the following folder and its contents

sudo chmod o+x -R /usr/share/man/

Others also suggest that the $MANPATH is incorrect. However, this is not the case for me.

IPtables: open port for specific IP

The following are examples of allow a host or a range of hosts visit port 8080

sudo vim /etc/sysconfig/iptables

# the following rule specifies only Ip 101.2.3.44 can visit port 8080

-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp -s 101.2.3.44 --dport 8080 -j ACCEPT

# a range of IPs are allowed to visit port 8080

-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp -src-range 101.2.3.44-1.2.3.54 --dport 8080 -j ACCEPT

# restart iptables service to make the change take effect

sudo /etc/init.d/iptables restart