sudo vim /etc/default/rcS # change UTC=yes to UTC=no in this file under the following sentence in rcS file # assume that the BIOS clock is set to UTC time (recommended) UTC=no
Windows 8 Time Always Wrong When Dual Boot With Ubuntu
I installed Ubuntu on a new Dell computer which has windows 8 preinstalled.
The time of Windows 8 computer always 5 hours ahead after returning from a Ubuntu session. After a little research I found that windows 8 is no longer supporting UTC time out of the box. To fix the problem,I configured Ubuntu to use localtime.
To do so,
Create multidimensional table in ReportLab
I was debugging a ReportLab program which was used to display a multidimensional table (Tables within table). I used zip(*) function to prepare the data and found out it is not right, since Zip(*) creates an array of tuples not an array of arrays.
1. Review of data structure for table in ReportLab:
ReportLab use array of arrays to prepare data for table, for example, data for a 2 column and 4 row table can be written in the following format in reportlab:
2. Issues of zip(*)
I used this command and got a list of tuples which is not good to feed to ReportLab:
[ # table starts ['col1 on row1','col2 of row1'], # row 1 of table ['col1 of row2', 'col2 of row2'], # row 2 of table ['col1 of row3', 'col2 of row3'], # row 3 of table ['col1 of row4', 'col2 of row4'] # row 4 of table ] # table endsFollow the same structure we could build multidimensional table, a 4-layer-array data structure:
[ #parent table start [ # row 1 of parent table [['dance', 'office.jpg'], ['office', 'dance.jpg'], ['panda', 'lock.jpg'], ['lock', 'shape.jpg'], ['shape', 'panda.jpg']], # child table 1 in row 1 of parent [['herd', 'germ.jpg'], ['dock', 'house.jpg'], ['house', 'shrink.jpg'], ['shrink', 'herd.jpg'], ['germ', 'dock.jpg']] # child table 2 in row 1 of parent ], # end of row 1 of parent table [ # row 2 of parent table [['gift', 'letter.jpg'], ['desk', 'desk.jpg'], ['drum', 'world.jpg'], ['world', 'drum.jpg'], ['letter', 'gift.jpg']], # child table 1 in row 2 of parent [['bee', 'study.jpg'], ['magnet', 'robot.jpg'], ['study', 'bee.jpg'], ['learn', 'magnet.jpg'], ['robot', 'learn.jpg']] # child table 2 in row 2 of parent ] # end of row 2 of parent table ]# parent table endHere is the table created based on the upper data structure.
zip(*[word,image])
[('dance', 'office.jpg'), ('office', 'dance.jpg'), ('panda', 'lock.jpg'), ('lock', 'shape.jpg'), ('shape', 'panda.jpg'), ('herd', 'herd.jpg')]
Then I tried the following code. It worked.
map(list, zip(*[word,image])) [['dance', 'office.jpg'], ['office', 'dance.jpg'], ['panda', 'lock.jpg'], ['lock', 'shape.jpg'], ['shape', 'panda.jpg'], ['herd', 'herd.jpg']]
Install Eclipse Kepler On Ubuntu 12.04.3
Here are the simple steps, I used to update eclipse to Kepler on my Ubuntu 12.04.3 box.
1. download the appropriate package of eclipse from http://www.eclipse.org/downloads/
untar the package and copy it to the location you prefer, /opt in my case
# untar tar -zvxf eclipse-standard-kepler-R-linux-gtk-x86_64.tar.gz # copy the uncompressed folder into /opt sudo cp -R eclipse /opt # change owner of all files to root sudo chown -R root:root /opt/eclipse2. link eclipse executable to
/usr/bin/
sudo ln -s /opt/eclipse/eclipse /usr/bin/eclipse3. Create the following .desktop file if you want to be able to pin eclipse to the luncher.
# create eclipse.desktop file at /usr/share/applications/ gedit /usr/share/applications/eclipse.desktop4. put the following contents in the file
[Desktop Entry] Name=Eclipse Type=Application Exec=/opt/eclipse/eclipse Terminal=false Icon=/opt/eclipse/icon.xpm Comment=Integrated Development Environment NoDisplay=false Categories=Development;IDE Name[en]=eclipse.desktopNow we are all set.
Install Nvidia Driver on Ubuntu Box
I noticed that Xorg process is taking too much of the CPU in my UBuntu box. I suspect that the video driver for my Nvidia graphy card may be not good. So I started to try other versions of drivers in hope to find a better one.
# obtain graphy card information lspci | grep -i vga 03:00.0 VGA compatible controller: NVIDIA Corporation G98 [Quadro NVS 295] (rev a1) 04:00.0 VGA compatible controller: NVIDIA Corporation G98 [Quadro NVS 295] (rev a1)Before installing I add the X Ubuntu Updates team PPA repository
sudo add-apt-repository ppa:ubuntu-x-swat/x-updates sudo apt-get update1. Removing all the old Nvidia drivers
sudo apt-get --purge remove nvidia-*2. Install the nvidia drivers.
# verify which nvidia package are available apt-cache search nvidia | sort sudo apt-get install nvidia-current-updates # After installation run the following command to initiate the configuration: sudo nvidia-xconfig sudo reboot
Upgrade Ubuntu 12.04.3 Kernel and x-stack from 3.5 to 3.8
#In order to install the new kernel, I did sudo apt-get install --install-recommends linux-image-generic-lts-raring #After install the new kernel I issue the following comamnd to install the compatible x-stack sudo apt-get install --install-recommends xserver-xorg-lts-raring
Running Hadoop on Clusters of Two Nodes using Ubuntu and CentOS
Here is an example of running Hadoop on a dummy cluster. For references of how to set up Hadoop on one computer, please visit Install Hadoop 1.1.2 on Ubuntu 12.04.
Two Linux computers one is running Ubuntu and the other one is running CentOS.To test the cluster configuration I used Ubuntu as master and slave and CentOS as slave only. Here is a summary of what I did. Section One: Install and configuration
2. Edit /etc/hosts to add host names for each computer and make sure that you can ping each other by host names.
2.1 for the master node make sure that your host names is listed behind your external ip not 127.0.0.1, to make sure that DataNodes could find NameNode.
use sudo netstat -tuplen to verify. host file for master node/Ubuntu
Section 2: Problem and Solution
1. Error message:
Symptom:
when you check your port opening with netstat you found that your port 54310 is binding to a internal ip (127.0.0.1)
Solution:
check you
Put your external IP and hostname pair in hosts file if it doe not exist. Re-check the port now:
2. Error message:
Solution:
Go to
Fix: Fix any relative paths in your job request with absolute paths when you using Hadoop streaming with python mapper and reducer.
Two Linux computers one is running Ubuntu and the other one is running CentOS.To test the cluster configuration I used Ubuntu as master and slave and CentOS as slave only. Here is a summary of what I did. Section One: Install and configuration
Daemons for master nodes (Ubuntu): NameNode SecondaryNameNode DataNode TaskTracker JobTracker Daemons for slave node (CentOS): DataNode TaskTracker1. Download and install Hadoop on both computers, make sure they are intalled into the same locations (highly recommend) by following my previous post.
2. Edit /etc/hosts to add host names for each computer and make sure that you can ping each other by host names.
2.1 for the master node make sure that your host names is listed behind your external ip not 127.0.0.1, to make sure that DataNodes could find NameNode.
use sudo netstat -tuplen to verify. host file for master node/Ubuntu
127.0.0.1 localhost 20.14.0.102 ncw01111.test.com ncw01111 # adding here for preventing internal binding # The following lines are desirable for IPv6 capable hosts ::1 ip6-localhost ip6-loopback fe00::0 ip6-localnet ff00::0 ip6-mcastprefix ff02::1 ip6-allnodes ff02::2 ip6-allroutershost file for slave node/CentOS
127.0.0.1 ncw02222 localhost.localdomain localhost ::1 localhost6.localdomain6 localhost6 20.14.0.102 ncw01111.test.com ncw01111 # I added master node ip and host name here when I have problem pinging master node. # at that time my Ubuntu host name has not been put into dns server2.2 Set passwordless SSH for each computer, so they can talk to each other without password. 3. make configuration for master, Ubuntu in this case: 3.1 Edit master and slave file located in conf folder of the master node (NOTE: For master node only)
# Edit master file sudo vim $HADOOP_PREFIX/conf/masters # then add ncw01111 into the file # Edit salve file sudo vim $HADOOP_PREFIX/conf/slaves # then add ncw01111 and ncw02222 into the files # one host name per line4. Edit configuration files: the configuration files, core-site.xml, mapred-site.xml and hdfs-site.xml are all the same for all the computers within the cluster. 4.1 core-site.xml
4.2 mapred-site.xmlhadoop.tmp.dir /home/bee/projects/hdfs_test fs.default.name hdfs://ncw01111:54310 true
4.3 hdfs-site.xmlmapred.job.tracker ncw01111:54311 mapred.system.dir /home/bee/projects/mapred/system true
5. Format the HDFS file systemdfs.replication 2
hadoop namenode -format6. Start the cluster
start-all.sh # or, use the following start up sequence start-dfs.sh start-mapred.shthen use jsp check the daemons
# master node, Ubuntu 9700 SecondaryNameNode 10093 TaskTracker 9169 NameNode 10367 Jps 9808 JobTracker 9432 DataNode #slave node, CentOS 26392 Jps 26201 TaskTracker 26095 DataNodePlease note, results from jps could not guarantee that your cluster has no problem. I had an issue that DataNode could not connect to NameNode (Problem 1 in the following problems and solution section ), I still could get the correct output from jps. Therefore, always look at your log files to see if there is any problem. if you see all your nodes of your cluster from here http://ncw01111:50030/machines.jsp?type=active your cluster might be good. 7. Put data into cluster and run a MapReduce job I used 99 years' NOAA data as an example, total 263G. They were preprocessed into a single file for each year, for example 1901.all. 7.1 put the data into cluster
gzip -c $year | hadoop fs -put - noaa/cluster/`basename $year.gz`7.2 run the python version of the mapreduce on noaa data
# do not use relative paths in the following command (see problem and solution #2) hadoop jar /opt/hadoop-1.1.2/contrib/streaming/hadoop-streaming-1.1.2.jar -mapper /home/bee/projects/hadoop/max_temperature_map.py -reducer /home/bee/projects/hadoop/max_temperature_reduce.py -input /user/bee/noaa/cluster/ -output /user/bee/noaa/cluster_out7.3 Running results comparison for 99 years' NOAA data. 7.1 Time respect Single node (Ubuntu only): Finished in: 59mins, 42sec Two Nodes Cluster (Ubuntu and CentOS): Finished in: 60mins, 12sec Two Nodes Cluster (Ubuntu and CentOS and using combiner): Finished in: 34mins, 12sec 7.2 Check results in HDFS
hadoop dfs -tail noaa/cluster_out/part-00000 1901 317 1902 244 ... 1999 5687.3 Retrieve the job result from HDFS
hadoop dfs -copyToLocal noaa/cluster_out/part-00000 /home/bee/projects/hadoop/output/cluster_out
Section 2: Problem and Solution
1. Error message:
2013-09-05 15:12:26,822 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: ncw01111/20.14.0.102:54310. Already tried 0 time(s); retry policy is RetryUpToMax
imumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)Symptom:
when you check your port opening with netstat you found that your port 54310 is binding to a internal ip (127.0.0.1)
tcp6 0 0 127.0.0.1:54310 :::* LISTEN 1001 624829 24900/javaSolution:
check you
/etc/hosts file make sure that your hostname is not listed after 127.0.0.1. or 127.0.1.1Put your external IP and hostname pair in hosts file if it doe not exist. Re-check the port now:
tcp6 0 0 220.14.24.75:54310 :::* LISTEN 1001 624829 24900/java2. Error message:
# of failed Map Tasks exceed allowed limit error.
# ERROR streaming.StreamJob: Job not successful. Error: # of failed Map Tasks exceeded allowed limit. FailedCount: 1. error. Solution:
Go to
master:50030/jobdetails.jsp?jobid=XXXXX(You can find it from your Hadoop output) to look for the real error.
After debugging I found an IO error in my case.Fix: Fix any relative paths in your job request with absolute paths when you using Hadoop streaming with python mapper and reducer.
Set Static IP for Ubuntu 12.04
1. Edit 
If no, you need add one by using:
/etc/network/interfaces and put the IP address along with other information
auto lo eth0 iface lo inet loopback iface eth0 inet static address 192.14.0.102 netmask 255.255.224.0 gateway 192.14.0.1 dns-search example.com dns-nameservers 192.14.0.194 192.14.0.1802. use
netstat -r to verify that you have default route.
sudo route add default gw [ip_address_for_default_gateway]3.check whether you have entries in
/etc/resolve.conf file.For example,
nameserver 192.14.0.194 nameserver 192.14.0.180 search example.com EXAMPLE.COM
Subscribe to:
Posts (Atom)