Reading: Eraser: A Dynamic Data Race Detector for Multithreaded Programs

By xulei | Published: November 18, 2010

This paper is chosen from an advanced operating system course.

As a common programming technology, multithreading programing leads to potential data race that two threads may access same piece of data without synchronization guarantee. Thus the authors of this paper presented a mutex exclusion monitoring mechanism to detect the misbehaviors of shared data. The basic concept of this research is that: for each access to shared variable v, testing whether there is at lease one lock consistently protects v by intersecting the locks being held and the locks which have accessed v:

  1. Let
    locks_held(t)

    be the set of locks held by thread t

  2. For each v, initialize C(v) to the set of all locks
  3. For each access to v, set C(v) := C(v) intersect locks_held(t)
  4. If C(v) = {}, means no lock consistently protecting v, then issue a warning
Posted in Reading | Tagged , | Leave a comment

Use sshfs to mount remote server on Mac

By xulei | Published: November 9, 2010

On an environment that you have high speed internet connections, a remote ssh server is a good choice for you to store and process data, especially most of these severals are more reliable and powerful than your Macbook Pro. MacFuse and sshfs are your friends. However, by default, the sshfs mounted diretory can not be shown explicitly in Finder. To make the sshfs mounted server be shown as your extern disks, you can use

sshfs USER:HOST MOUNT_POINT -o local,volname=VOLNAME

. You can even use

-o volicon=/path/to/icon

to customize the icon shown in Finder.

Posted in Geek | Tagged , , , | Leave a comment

Write file systems on top of ZFS components

By xulei | Published: November 2, 2010

Despite of ZPL(ZFS Posix Layer), the left ZFS stack has a bunch of features and functionalities that all other file system developers need and are going to implement more or less, such as DMU, ZIL and ZVOL, which provide object storage, logs, and transaction supports. The successful of zfs-fuse project and the promising native zfs implementation have proven that it is possible to implement a well-behavior file system by utilizing DMU/ZIL/ZVOL on a linux machine. I believe that is opening a door for file system developers to save years for prototyping and implementing a modern, reliable and scalable file system.

Posted in Storage | Tagged , , | Leave a comment

Reading: SPORC: Group Collaboration using Untrusted Cloud Resources(OSDI’10)

By xulei | Published: November 1, 2010

This paper is from Princeton University.

This paper presented a distributed group collaboration method based on the assumptions that the cloud server, the remote server actually, is not secure and reliable. In fact, this paper does not focus on cloud service, but only treat it as an untrusted remote server, in terms of privacy and security. However, this remote server is reliable and globally accessible. With holding such assumptions, SPORC uses Operational Transformation (OT) and fork* consistency protocol to ensure the changes between clients can be propagated to all other clients appropriately. And the fork* consistency embeds the change histories in each synchronize request to keep the integrity of all client changes.

Posted in Reading | Tagged , , | Leave a comment

A portable getppid for specific pid

By xulei | Published: September 20, 2010

I posted a gist that can obtain the id of the parent process of any given process(pid), which is expected to be useful to get data flow of program executions. Since on Mac, there is no /proc directory by default, thus we access the process info by invoke sysctl (reference)

/**
 See http://gist.github.com/589115
*/
pid_t getppid_for_process(pid_t pid) {
#if defined(__APPLE__)
    struct kinfo_proc info;
    size_t length = sizeof(struct kinfo_proc);
    int mib[4] = { CTL_KERN, KERN_PROC, KERN_PROC_PID, pid };
    if (sysctl(mib, 4, &info, &length, NULL, 0) < 0)
        return -1;
    if (length == 0)
        return -1;
    return info.kp_eproc.e_ppid;
#elif defined(__linux__)
    char proc_filepath[BUFSIZE];
    char buffer[BUFSIZE];
    pid_t parent = -1;
    memset(proc_filepath, 0, BUFSIZE);
    snprintf(proc_filepath, BUFSIZE, "/proc/%d/status", pid);
    FILE * fp = fopen(proc_filepath, "r");
    while (fgets(buffer, BUFSIZE, fp) != NULL) {
        if(strncmp(buffer, "PPid:", 5) == 0) {
            sscanf(buffer, "PPid: %d", &parent);
            break;
        }
    }
    fclose(fp);
    return parent;
#endif /* __linux__ */
}
Posted in Computer Science | Tagged , , | Leave a comment

wrapperfs — A file system wrapper on top of existing file systems

By xulei | Published: September 20, 2010

Today, I created a new git repo for wrapperfs on github. Wrapper-fs works as a thin wrapper on top of the existing file systems to provide an simple entry for developers and researchers to evaluate their ideas. I have done this wrapper thing twice, which motivates me to extract the main skeleton of source codes out to an individual open source project, in addition to several useful data structures that one file system developer will meet, such as journaling, LRU cache and etc. So here the fuseutils project is.

Posted in Storage | Tagged , , | Leave a comment

Set vim to jump between files located in include and src sub-directories

By xulei | Published: September 14, 2010

There is one extra configuration required for a.vim, you should put the following line into ~/.vimrc:

let g:alternateSearchPath = 'sfr:../source,sfr:../src,sfr:../include,sfr:../inc,reg:/include/src/g,reg:/src/include/g'
Posted in Geek | Leave a comment

Use awk to count the occurrences of lines

By xulei | Published: September 10, 2010

This is a one-line command to calculate the occurrences of particular lines in one file. I have used this to analyze the access patterns of I/O traces by count the file lookup operations:

grep OP trace.txt | awk '{++count[$1]} END { for (x in count) print count[x] " " x; }' | sort -nr

It works pretty well to replace my python script.

Posted in Computer Science | Leave a comment

Home NAS solution

By xulei | Published: September 8, 2010

It has been a tradition that I wrote a post for my home NAS solution after each re-installation of wordpress. Since the old macbook was stolen in San Francisco, I bought a Zotac Mag as home server. Besides that, I bought some other equipments to build a home entertainment network:

The main purposes of this home NAS are:

  1. Entertainment ( Movies, TV episodes )
  2. Backup & Storage Service (Time Machine, Photos, Source Control and etc)

Therefore, the follows are my configuration.

Centralized Server: I installed Ubuntu 10.04 LTS (i386) on Zotac Mag to let it be the home server. Since we do not have PC(w/ windows) in this environment, I configured this ubuntu followed the link. Simply put, Netatalk and Avahi servers are what you want. The LaCie 1TB HD and 60GB HD are connected to Zotac Mag through USB 2.0 ports. The 1TB LaCie HD is formatted as HFS+ (hfsplus in ubuntu) to store personal files and data in case it will be accessed by a Mac directly for restoration or something. The 60GB portable HD is formatted as Ext4 to used as downloading pool and storing movies. Since this HD is very old, no serious data should be put onto it. In order to support streaming movies to WD Live media player, a samba service was configured to export a read-only volume of movie directory. My Zotac Mag mount the hard drive shared by AirPort extreme as well. Furthermore several cron jobs are placed to do some regular maintenance jobs, e.g, backing up all source repos, syncing personal data between local disk(LaCie) and remote disk(WD 1.5T behind AirPort Extreme), checking new TV episodes and etc. Finally a transmission daemon and a mldonkey server are configured as BT client and eMule client respectively.

AirPort Extreme. The WD 1.5T HD mounted behind AirPort Extreme is split 400 MB (HFS+) as wireless Time Machine partition for my wife and myself. The left space is used for storing personal data (photos, documents, videos and etc). In addition, a directory is syncing with Zotac Mac for duplicating very important data. Several NAT redirections are configured to Zotac Mag as well ( e.g, a customized ssh port, several rpc ports for buildbot, a customized http port for restricted access of some http service)

Media Player, A WD Live TV media player is set to connected with TV through HDMI cable and connected to AirPort extreme through 1Gb Ethernet. It delivers the contents from Zotac Mag samba volume to TV. Actually a Boxee is installed in Zotac Mag as well. However due to lack of time to optimize and install drivers for ION, Boxee is not fully functional right now.

The things left to do:

  1. A cronjob that automatically downloads chinese subtitles of movies through shooter.cn
  2. A new WD drive to build a soft RAID-1(md) with the above WD HD.

I just realized how many works I have done for this home NAS solution by writing all of them out. Amazing…

Posted in Geek, Life | Tagged , , , | 1 Comment

Another fresh installation of wordpress

By xulei | Published: September 7, 2010

Last month I received an email from Dreamhost, the web host company for this site, for some announcements about the server side migrations. I haven’t read it seriously, at least at that time, since I thought it would be a fundamental requirement of the migration operations in a commercial company that would not hurt user’s data. However the old site was down without reasons. Therefore, the lazy me make a fresh installation of wordpress to this site. Apologizes to the bare readers, if any: No further data restore attempts will be issued for this site. Let it be.

Posted in Life | Leave a comment