StonePark http://www.xulei.me A new journal begins Wed, 08 Feb 2012 20:41:55 +0000 en hourly 1 Another Google summer internship http://www.xulei.me/2012/02/08/another-google-summer-internship/ http://www.xulei.me/2012/02/08/another-google-summer-internship/#comments Wed, 08 Feb 2012 20:41:55 +0000 xulei http://www.xulei.me/?p=45 A little bit of lazy and busy in the beginning of 2012, I haven’t written too much. Yep, I just got another session of internship in Google this summer. What I’ve heard so far is only that I will work in the Ads backend team to optimize their map-reduce stuff…It sounds like a awesome job, considering Ads contributing 96% of Google’s revenue…$$$$…

Share/Bookmark

]]> http://www.xulei.me/2012/02/08/another-google-summer-internship/feed/ 0 Dynamically set CPU offline on Linux http://www.xulei.me/2011/11/01/dynamically-set-cpu-offline-on-linux/ http://www.xulei.me/2011/11/01/dynamically-set-cpu-offline-on-linux/#comments Wed, 02 Nov 2011 01:51:53 +0000 xulei http://www.xulei.me/?p=44 Linux provides a mechanism to hotplug a CPU (core) online or offline during the runtime, which is very convenient for me to dynamically adjust the CPU topology. As usual, I have written a python script (Link: gist:1331343) to make this manually task easy. This script accepts similar parameters as taskset(8):

# Set core 1,2,3,5,6,7,9,12 online, and other cores offline.
$ set_cpus.py 1-3,5-7,9,12
# Reset all cores to be online
$ set_cpus.py -r

Share/Bookmark

]]> http://www.xulei.me/2011/11/01/dynamically-set-cpu-offline-on-linux/feed/ 0 HomeNAS reborn with ZFS http://www.xulei.me/2011/09/16/homenas-reborn-with-zfs/ http://www.xulei.me/2011/09/16/homenas-reborn-with-zfs/#comments Sat, 17 Sep 2011 04:42:38 +0000 xulei http://www.xulei.me/?p=41 After came back from California, I bought a HP ProLiant MicroServer for its 4 hard drive bays. The machine was upgraded to 5(1+4) GB ECC RAMs, and added two more Seagate Barracuda Green 1.5TB 5900RPM HDDs. Without any confidence of storing important data (e.g family photos, research data and drafts) on the 3 years old LeCie 1TB USB disk, I decided to build a FreeBSD + ZFS solution this time for the sake of ZFS’s ultimately end-to-end integrations. The two seagate disks are configured as mirrored ZFS pool, and the LeCie 1T one is formatted as an individual ZFS pool without checksums (Because I do not really care about the data on it). Now I have 7+ TB hard drives at home! It is way to geeky!

The detailed NAS configurations are similar to my previous home NAS solution (e.g Netatalk, Avahi, Samba, Transmission and etc.), with the exception that there is no a decent Dropbox solution on FreeBSD, which actually adds the complexity for me and my wife to backup data around. I did look for building a Debian within VirtualBox, in which the home directory was mounted to the FreeBSD disks. Unfortunately, NFS clients lack the support of inotify that is required by Dropbox monitoring the local file changes. Though there is a potential solution: let the debian virtual machine using local file system (e.g Ext4) and rsyncing the Dropbox directory to FreeBSD every so often. But I just think that it is too much for a Dropbox application.

Share/Bookmark

]]> http://www.xulei.me/2011/09/16/homenas-reborn-with-zfs/feed/ 0 Google intern summary 1 http://www.xulei.me/2011/06/28/google-intern-summary-1/ http://www.xulei.me/2011/06/28/google-intern-summary-1/#comments Tue, 28 Jun 2011 21:31:51 +0000 xulei http://www.xulei.me/?p=39 Three busy weeks have passed by. For most of the time, I was attending tutorials, reading documents and at the end of this period, I was writing documents. Yes, no a single line of code I wrote.

Google has fantastic infrastructures for developments and a bunch of awesome people giving tutorials to us nooglers (a.k.a new googlers). Though there are plenty of stuffs that are confidential, I would like to write down some other public parts that I feels very useful here:

  1. Yeah, they use TDD as well! I personally transfer one of my most important projects at school from using cppuint to google test
  2. Unit tests should be as short/fast as possible. They should not rely on any other components, e.g file system, database, and so. Since once you have tens of thousands of test cases, those slow tests would kill the productively of the entire team.
  3. Do not implement the functions that not necessary for now. If you need this interface, then put an assert in this function in case someone mistakenly use it
  4. Automatic, Automatic, Automatic.
  5. Code is a piece of knowledge. Reuse it whenever is possible

Share/Bookmark

]]> http://www.xulei.me/2011/06/28/google-intern-summary-1/feed/ 0 Got google summer internship http://www.xulei.me/2011/04/02/got-google-summer-internship/ http://www.xulei.me/2011/04/02/got-google-summer-internship/#comments Sat, 02 Apr 2011 20:37:58 +0000 xulei http://www.xulei.me/?p=35 I am so excited to get the google summer internship this year for a great matched team, the storage team for Google Apps. Two back-to-back tech phone screens were taken at the last Monday (March 21). I felt good about the most of the questions except the last one, for which my brain was shortcut due to a very bad sleep last night ( less than 2 hours, maybe I was too nervous). However, I still got an email to tell me I passed the tech interview at that Wed. . At the next Thus., I got a Host Interview with the target team. The process was very smooth. It was basically exchange the informations of their projects and my personal experiences, and I got the offer for just 10 minutes. Everything works more smoothly than I expected.

Share/Bookmark

]]> http://www.xulei.me/2011/04/02/got-google-summer-internship/feed/ 4 Reading: Cheap and Large CAMS for High Performance Data-Intensive Network Systems (NSDI’10) http://www.xulei.me/2011/02/04/cheap-and-large-cams/ http://www.xulei.me/2011/02/04/cheap-and-large-cams/#comments Fri, 04 Feb 2011 07:02:59 +0000 xulei http://www.xulei.me/?p=34 This paper presents how to build cheap and large CAMs (what’s CAM?) using a combination of DRAM and flash memory(SSD) to balance the cost and query latency. The main approach of this work is building multiple small hash tables, which are stored on SSD, and holding a two-level bloom-filter in RAM to provide fast hash value lookup. As the author claimed, this work aims to improve the lookup latency but not throughput. By grouping hash values into small groups and flush them in batch mode, it reduces the latency of the costly in-place insert/delete on SSD devices.

Share/Bookmark

]]> http://www.xulei.me/2011/02/04/cheap-and-large-cams/feed/ 0 Developing environment for solo developer http://www.xulei.me/2011/02/01/developing-environment-for-solo-developer/ http://www.xulei.me/2011/02/01/developing-environment-for-solo-developer/#comments Tue, 01 Feb 2011 22:16:09 +0000 xulei http://www.xulei.me/?p=32 Yes, I am still a computer science Ph.D student right now. Therefore, for my long-term research projects, there is nobody taking charge of building automatically test environments, document management, several platform configurations and so on. I have to build my own to fit my specific workloads: typically a project starts from collecting ideas (head storms), looking for solutions, prototyping and do some experiments. I know, normally, you will create several directories on you laptop, and put files into it. But I want more than that. As worked in giant internet company, software company and start-ups, I have learnt the importances of developing processes. Therefore I built my working environment as following:

  1. Version control: git. Git is powerful and light-weight. It does not require any centralized server, which makes it is super easy to create to use in my daily development and writing processes. One killer feature of git is its lightweight branch operation, which is great for you to prototyping a new experimental features. However, the official documents suck for newbies, if you wanna learn git, go Pro Git Book first.
  2. Trac, trac is a feature-rich SCM system written in Python with wiki, issue tracking, source browsing features. I have used Trac since 2005 for several projects. It is not as fancy as Github. But it is still a good choice if you have no willings to spend extra money on your private projects. Moreover, Trac has git plugin to observe local git repository.
  3. Continuous Integration(CI). I set up a C.I server in our research lab, because it has a public URL to access. For C.I server I chose Buildbot, and set up two build slaves on Ubuntu Server and Mac OSX , respectively. Since I use my macbook pro as the primary development platform, but do not expect to “accidentally” figure out my projects are not working on Ubuntu for months. Under such configuration, the buildbot runs a fully integration tests on the fresh checked out source codes for each commits. What is missing here is that there is no continuous performance tests here, which I am still looking for an elegant solution.

Share/Bookmark

]]> http://www.xulei.me/2011/02/01/developing-environment-for-solo-developer/feed/ 0 Note: Move MySQL database to another location http://www.xulei.me/2011/01/04/note-move-mysql-database-to-another-location/ http://www.xulei.me/2011/01/04/note-move-mysql-database-to-another-location/#comments Wed, 05 Jan 2011 03:15:51 +0000 xulei http://www.xulei.me/?p=29 For the purpose to evaluate the performance of MySQL database on Ubuntu, I have to move it to an separated disk. I write the steps here for future use:

  1. mkdir /mnt/test/mysql

    and

    chmod -R mysql:mysql /mnt/test/mysql
  2. Stop mysql
  3. cp -rp /var/lib/mysql/mysql /var/lib/mysql/otherdb /mnt/test/mysql
  4. Modify /etc/mysql/my.cnf, find the line with
    datadir = /var/lib/mysql

    , then change it to

    datadir = /mnt/test/mysql
  5. Note: change
    /etc/apparmor.d/usr.sbin.mysqld

    to grant the execution privileges to

    /mnt/test/mysql
  6. Restart apparmor and mysql

Share/Bookmark

]]> http://www.xulei.me/2011/01/04/note-move-mysql-database-to-another-location/feed/ 0 A sed script to extract file name from trace file http://www.xulei.me/2010/12/25/a-sed-script-to-extract-file-name-from-trace-file/ http://www.xulei.me/2010/12/25/a-sed-script-to-extract-file-name-from-trace-file/#comments Sat, 25 Dec 2010 22:15:09 +0000 xulei http://www.xulei.me/?p=26 The following script is used to extract the full path from a file.

sed -En 's#^[^/]+(/([^ ])+).+#\1#gp' < trace.txt

Share/Bookmark

]]> http://www.xulei.me/2010/12/25/a-sed-script-to-extract-file-name-from-trace-file/feed/ 0 Reading: Debunking the 100X GPU vs. CPU Myth http://www.xulei.me/2010/12/02/100x-gpu-cpu-myth/ http://www.xulei.me/2010/12/02/100x-gpu-cpu-myth/#comments Fri, 03 Dec 2010 05:09:18 +0000 xulei http://www.xulei.me/?p=23 This paper comes from ISCA’10.

This paper re-evaluates the performance gap, especially for high throughput computing, between CPU and GPU. Contrast to the previous papers it mentioned claimed 100x performance differences between CPU and GPU, with appropriate optimizations, the performance gap between CPU (Intel Core i7 960) and Nvidia GTX280 narrows to only 2.5x on average. The main factors that impact performance differences between CPU and GPU are:

  1. Bandwidth
  2. Compute Flops
  3. Cache
  4. Gather/Scatter
  5. Reduction and Synchronization

CPUs have large cache but CPUs have higher bandwidth. Thus the average performance gaps between these two are not that extreme huge depended on the characteristics of workloads.

Share/Bookmark

]]> http://www.xulei.me/2010/12/02/100x-gpu-cpu-myth/feed/ 0