Windows 8 Time Always Wrong When Dual Boot With Ubuntu

I installed Ubuntu on a new Dell computer which has windows 8 preinstalled. The time of Windows 8 computer always 5 hours ahead after returning from a Ubuntu session. After a little research I found that windows 8 is no longer supporting UTC time out of the box. To fix the problem,I configured Ubuntu to use localtime. To do so,

sudo vim /etc/default/rcS
# change UTC=yes to UTC=no in this file under the following sentence in rcS file
# assume that the BIOS clock is set to UTC time (recommended)
UTC=no

Create multidimensional table in ReportLab

I was debugging a ReportLab program which was used to display a multidimensional table (Tables within table). I used zip(*) function to prepare the data and found out it is not right, since Zip(*) creates an array of tuples not an array of arrays. 1. Review of data structure for table in ReportLab: ReportLab use array of arrays to prepare data for table, for example, data for a 2 column and 4 row table can be written in the following format in reportlab:

[ # table starts
['col1 on row1','col2 of row1'], # row 1 of table
['col1 of row2', 'col2 of row2'], # row 2 of table
['col1 of row3', 'col2 of row3'], # row 3 of table
['col1 of row4', 'col2 of row4'] # row 4 of table
] # table ends

Follow the same structure we could build multidimensional table, a 4-layer-array data structure:

[ #parent table start
[ # row 1 of parent table
[['dance', 'office.jpg'], ['office', 'dance.jpg'], ['panda', 'lock.jpg'], ['lock', 'shape.jpg'], ['shape', 'panda.jpg']], # child table 1 in row 1 of parent
[['herd', 'germ.jpg'], ['dock', 'house.jpg'], ['house', 'shrink.jpg'], ['shrink', 'herd.jpg'], ['germ', 'dock.jpg']] # child table 2 in row 1 of parent
], # end of row 1 of parent table
[ # row 2 of parent table
[['gift', 'letter.jpg'], ['desk', 'desk.jpg'], ['drum', 'world.jpg'], ['world', 'drum.jpg'], ['letter', 'gift.jpg']], # child table 1 in row 2 of parent
[['bee', 'study.jpg'], ['magnet', 'robot.jpg'], ['study', 'bee.jpg'], ['learn', 'magnet.jpg'], ['robot', 'learn.jpg']] # child table 2 in row 2 of parent
] # end of row 2 of parent table
]# parent table end

Here is the table created based on the upper data structure.

2. Issues of zip(*) I used this command and got a list of tuples which is not good to feed to ReportLab:

zip(*[word,image])
[('dance', 'office.jpg'), ('office', 'dance.jpg'), ('panda', 'lock.jpg'), ('lock', 'shape.jpg'), ('shape', 'panda.jpg'), ('herd', 'herd.jpg')]

Then I tried the following code. It worked.

map(list, zip(*[word,image]))
[['dance', 'office.jpg'], ['office', 'dance.jpg'], ['panda', 'lock.jpg'], ['lock', 'shape.jpg'], ['shape', 'panda.jpg'], ['herd', 'herd.jpg']]

Install Eclipse Kepler On Ubuntu 12.04.3

Here are the simple steps, I used to update eclipse to Kepler on my Ubuntu 12.04.3 box. 1. download the appropriate package of eclipse from http://www.eclipse.org/downloads/ untar the package and copy it to the location you prefer, /opt in my case

# untar
tar -zvxf eclipse-standard-kepler-R-linux-gtk-x86_64.tar.gz
# copy the uncompressed folder into /opt
sudo cp -R eclipse /opt
# change owner of all files to root
sudo chown -R root:root /opt/eclipse

2. link eclipse executable to /usr/bin/

sudo ln -s /opt/eclipse/eclipse /usr/bin/eclipse

3. Create the following .desktop file if you want to be able to pin eclipse to the luncher.

# create eclipse.desktop file at /usr/share/applications/
gedit /usr/share/applications/eclipse.desktop

4. put the following contents in the file

[Desktop Entry]
Name=Eclipse
Type=Application
Exec=/opt/eclipse/eclipse
Terminal=false
Icon=/opt/eclipse/icon.xpm
Comment=Integrated Development Environment
NoDisplay=false
Categories=Development;IDE
Name[en]=eclipse.desktop

Now we are all set.

Install Nvidia Driver on Ubuntu Box

I noticed that Xorg process is taking too much of the CPU in my UBuntu box. I suspect that the video driver for my Nvidia graphy card may be not good. So I started to try other versions of drivers in hope to find a better one.

# obtain graphy card information
lspci | grep -i vga
03:00.0 VGA compatible controller: NVIDIA Corporation G98 [Quadro NVS 295] (rev a1)
04:00.0 VGA compatible controller: NVIDIA Corporation G98 [Quadro NVS 295] (rev a1)

Before installing I add the X Ubuntu Updates team PPA repository

sudo add-apt-repository ppa:ubuntu-x-swat/x-updates
sudo apt-get update

1. Removing all the old Nvidia drivers

sudo apt-get --purge remove nvidia-*

2. Install the nvidia drivers.

# verify which nvidia package are available
apt-cache search nvidia | sort
sudo apt-get install nvidia-current-updates
# After installation run the following command to initiate the configuration:
sudo nvidia-xconfig
sudo reboot

Upgrade Ubuntu 12.04.3 Kernel and x-stack from 3.5 to 3.8

#In order to install the new kernel, I did
sudo apt-get install --install-recommends linux-image-generic-lts-raring
#After install the new kernel I issue the following comamnd to install the compatible x-stack
sudo apt-get install --install-recommends xserver-xorg-lts-raring

Running Hadoop on Clusters of Two Nodes using Ubuntu and CentOS

Here is an example of running Hadoop on a dummy cluster. For references of how to set up Hadoop on one computer, please visit Install Hadoop 1.1.2 on Ubuntu 12.04.
Two Linux computers one is running Ubuntu and the other one is running CentOS.To test the cluster configuration I used Ubuntu as master and slave and CentOS as slave only. Here is a summary of what I did. Section One: Install and configuration

Daemons for master nodes (Ubuntu):
NameNode
SecondaryNameNode
DataNode
TaskTracker
JobTracker
Daemons for slave node (CentOS):
DataNode
TaskTracker

1. Download and install Hadoop on both computers, make sure they are intalled into the same locations (highly recommend) by following my previous post.
2. Edit /etc/hosts to add host names for each computer and make sure that you can ping each other by host names.
2.1 for the master node make sure that your host names is listed behind your external ip not 127.0.0.1, to make sure that DataNodes could find NameNode.
use sudo netstat -tuplen to verify. host file for master node/Ubuntu

127.0.0.1       localhost
20.14.0.102 ncw01111.test.com ncw01111 # adding here for preventing internal binding
# The following lines are desirable for IPv6 capable hosts
::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters

host file for slave node/CentOS

127.0.0.1       ncw02222 localhost.localdomain localhost
::1             localhost6.localdomain6 localhost6
20.14.0.102     ncw01111.test.com ncw01111 # I added master node ip and host name here when I have problem pinging master node. # at that time my Ubuntu host name has not been put into dns server

2.2 Set passwordless SSH for each computer, so they can talk to each other without password. 3. make configuration for master, Ubuntu in this case: 3.1 Edit master and slave file located in conf folder of the master node (NOTE: For master node only)

# Edit master file
sudo vim $HADOOP_PREFIX/conf/masters
# then add ncw01111 into the file
# Edit salve file
sudo vim $HADOOP_PREFIX/conf/slaves
# then add ncw01111 and ncw02222 into the files
# one host name per line

4. Edit configuration files: the configuration files, core-site.xml, mapred-site.xml and hdfs-site.xml are all the same for all the computers within the cluster. 4.1 core-site.xml

hadoop.tmp.dir
/home/bee/projects/hdfs_test


fs.default.name
hdfs://ncw01111:54310
true

4.2 mapred-site.xml

mapred.job.tracker
ncw01111:54311


mapred.system.dir
/home/bee/projects/mapred/system
true

4.3 hdfs-site.xml

dfs.replication
2

5. Format the HDFS file system

hadoop namenode -format

6. Start the cluster

start-all.sh
# or, use the following start up sequence
start-dfs.sh
start-mapred.sh

then use jsp check the daemons

# master node, Ubuntu
9700 SecondaryNameNode
10093 TaskTracker
9169 NameNode
10367 Jps
9808 JobTracker
9432 DataNode
#slave node, CentOS
26392 Jps
26201 TaskTracker
26095 DataNode

Please note, results from jps could not guarantee that your cluster has no problem. I had an issue that DataNode could not connect to NameNode (Problem 1 in the following problems and solution section ), I still could get the correct output from jps. Therefore, always look at your log files to see if there is any problem. if you see all your nodes of your cluster from here http://ncw01111:50030/machines.jsp?type=active your cluster might be good. 7. Put data into cluster and run a MapReduce job I used 99 years' NOAA data as an example, total 263G. They were preprocessed into a single file for each year, for example 1901.all. 7.1 put the data into cluster

gzip -c $year | hadoop fs -put - noaa/cluster/`basename $year.gz`

7.2 run the python version of the mapreduce on noaa data

# do not use relative paths in the following command (see problem and solution #2)
hadoop jar /opt/hadoop-1.1.2/contrib/streaming/hadoop-streaming-1.1.2.jar -mapper /home/bee/projects/hadoop/max_temperature_map.py -reducer /home/bee/projects/hadoop/max_temperature_reduce.py -input /user/bee/noaa/cluster/ -output /user/bee/noaa/cluster_out

7.3 Running results comparison for 99 years' NOAA data. 7.1 Time respect Single node (Ubuntu only): Finished in: 59mins, 42sec Two Nodes Cluster (Ubuntu and CentOS): Finished in: 60mins, 12sec Two Nodes Cluster (Ubuntu and CentOS and using combiner): Finished in: 34mins, 12sec 7.2 Check results in HDFS

hadoop dfs -tail noaa/cluster_out/part-00000
1901    317
1902    244
...
1999    568

7.3 Retrieve the job result from HDFS

hadoop dfs -copyToLocal noaa/cluster_out/part-00000 /home/bee/projects/hadoop/output/cluster_out

Section 2: Problem and Solution
1. Error message: 2013-09-05 15:12:26,822 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: ncw01111/20.14.0.102:54310. Already tried 0 time(s); retry policy is RetryUpToMax imumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
Symptom:
when you check your port opening with netstat you found that your port 54310 is binding to a internal ip (127.0.0.1) tcp6 0 0 127.0.0.1:54310 :::* LISTEN 1001 624829 24900/java
Solution:
check you /etc/hosts file make sure that your hostname is not listed after 127.0.0.1. or 127.0.1.1
Put your external IP and hostname pair in hosts file if it doe not exist. Re-check the port now: tcp6 0 0 220.14.24.75:54310 :::* LISTEN 1001 624829 24900/java

2. Error message: # of failed Map Tasks exceed allowed limit error. # ERROR streaming.StreamJob: Job not successful. Error: # of failed Map Tasks exceeded allowed limit. FailedCount: 1. error.
Solution:
Go to master:50030/jobdetails.jsp?jobid=XXXXX(You can find it from your Hadoop output) to look for the real error. After debugging I found an IO error in my case.
Fix: Fix any relative paths in your job request with absolute paths when you using Hadoop streaming with python mapper and reducer.

Set Static IP for Ubuntu 12.04

1. Edit /etc/network/interfaces and put the IP address along with other information

auto lo eth0
iface lo inet loopback
iface eth0 inet static
address 192.14.0.102
netmask 255.255.224.0
gateway 192.14.0.1
dns-search example.com
dns-nameservers 192.14.0.194 192.14.0.180

2. use netstat -r to verify that you have default route.

If no, you need add one by using:

sudo route add default gw [ip_address_for_default_gateway]

3.check whether you have entries in /etc/resolve.conf file.For example,

nameserver 192.14.0.194
nameserver 192.14.0.180
search example.com EXAMPLE.COM