For the impatient, scroll to the bottom of the page for an interactive example.

Date.fromString() is a method that allows you to easily parse user input into a date. There are currently many implementations of calendar pop-ups that aim to solve a similar problem. However, these calendars are cumbersome to use if the target date is more than a couple months away. The goal of this project is to parse free-form user input into a valid Date object.

Date.fromString() accepts two parameters:

• input
  string — User input string to parse
• options optional
  object — An object that can override the default behavior
  { order: ‘MDY’, strict: false }

Examples

console.log(Date.fromString('Sept 5th, 2006  4:00 pm'));
// Tue Sep 05 2006 16:00:00 GMT-0600 (MDT)
 
console.log(Date.fromString('Sept 5th, 2006  4:23pm'));
// Tue Sep 05 2006 16:23:00 GMT-0600 (MDT)
 
console.log(Date.fromString('Sept 5th, 2006  4pm'));
// Tue Sep 05 2006 16:00:00 GMT-0600 (MDT)
 
console.log(Date.fromString('Sept 5th, 2006  12:34:56 am'));
// Tue Sep 05 2006 00:34:56 GMT-0600 (MDT)
 
console.log(Date.fromString('Sept 5th, 2006  7:30 am'));
// Tue Sep 05 2006 07:30:00 GMT-0600 (MDT)
 
console.log(Date.fromString('2006/09/05  4:23pm'));
// Tue Sep 05 2006 16:23:00 GMT-0600 (MDT)
 
console.log(Date.fromString('2006/09/05'));
// Tue Sep 05 2006 00:00:00 GMT-0600 (MDT)
 
console.log(Date.fromString('9/5/06'));
//Tue Sep 05 2006 00:00:00 GMT-0600 (MDT)
 
console.log(Date.fromString('2/13/78'));
// Mon Feb 13 1978 00:00:00 GMT-0700 (MST)
 
console.log(Date.fromString('Feb 13 09'));
// Fri Feb 13 2009 00:00:00 GMT-0700 (MST)

When part of the date is left off, it fills in the missing parts with today’s details. When these examples were written, it was November 25, 2008.

console.log(Date.fromString('Feb 13th'));
// Wed Feb 13 2008 00:00:00 GMT-0700 (MST)
 
console.log(Date.fromString('2/13'));
// Wed Feb 13 2008 00:00:00 GMT-0700 (MST)
 
console.log(Date.fromString('4pm'))
// Tue Nov 25 2008 16:00:00 GMT-0700 (MST)

If you would rather have the date not fill in those parts, pass the option strict set to true.

console.log(Date.fromString('4pm'))
// Tue Nov 25 2008 16:00:00 GMT-0700 (MST)
 
console.log(Date.fromString('4pm', {strict:true}))
// Invalid Date

Sometimes the date cannot be parsed without ambiguity. In these cases, the default ordering of month/day/year (MDY) is used. To change this behavior, pass the option order set to your preferred ordering; YMD, DMY, etc.

console.log(Date.fromString('09/05/06'));
// Tue Sep 05 2006 00:00:00 GMT-0600 (MDT)
 
console.log(Date.fromString('09/05/06', {order: 'YMD'}));
// Wed May 06 2009 00:00:00 GMT-0600 (MDT)
 
console.log(Date.fromString('09/05/06', {order: 'DMY'}));
// Tue May 09 2006 00:00:00 GMT-0600 (MDT)

Finally, try it for yourself. I have not been able to test this much in browsers other than Firefox on Linux, so if you find any bugs, please let me know.

Downloaded a total of 84 times

Javascript Date.fromString() (5.27 KB)

(in all due respect, of course )

Let me explain. If you want to use a library, tool, or feature in a way that the original developer didn’t care about or think about, then odds are that the documentation will claim it will work, but it won’t. It will work for the one case the developer tested, but that is it. Of course there are also many projects and features out there that will prove me wrong, and kudos to those ones.

I am getting sidetracked. Today I wanted annotations in PHP. I wanted to provide meta data for the methods and classes I was writing.

It would be easy enough to write a code generator that parses the source, handles the annotations, and spits out the desired code. In fact, many things already do this, like documenters and style checkers. However, that is not what I want. I want it done at runtime and I want it to change the behavior of the code.

Sometimes I have to back up my complaints with some action, so today I implemented this feature. In an entirely proof-of-concept way, but a real solution for runtime PHP annotations nonetheless.

Since examples speak louder than words, here we go.

<?php
 
class Pi {
 
  /**
   * Use the monte carlo method of estimating PI
   * @return float
   */
  public static function estimate () {
    $rounds = 1000000;
    $hits = 0;
    $max = mt_getrandmax();
    for ($i=0; $i<$rounds; $i++) {
      $x = mt_rand() * 1.0 / $max;
      $y = mt_rand() * 1.0 / $max;
      $hits += (($x*$x + $y*$y) <= 1.0)? 1: 0;
    }
    return 4.0 * $hits / $rounds;
  }
 
}
 
// Consistent return value
mt_srand(0xDEAD);
echo Pi::estimate(), "\n";
mt_srand(0xDEAD);
echo Pi::estimate(), "\n";

Does anyone else enjoy naming a php variable ‘hit’ or ‘hits’? Anyway, for this example I wanted a function that would take some time to compute. Admittedly contrived, but I didn’t ask you to read this post. Using the monte carlo method, this static class method estimates the value of PI and is called twice.

time php Pi.php 
3.141196
3.141196
 
real    0m2.186s

Obviously, if many calls are made to this method, the time adds up quickly. No one in their right mind would actually release code like this. I said right mind. Something I find myself doing often is caching the result before returning it. I’m not going to show that example since it is trivial to implement. Clearly all subsequent calls to the time-consuming method would return immediately.

I don’t like it. I do it, but I don’t like it. The method starts out very clean. All it does is estimate the value of PI (or search a graph, or prune a tree, whatever). When you add caching into it, suddenly the method is doing two things. And two completely unrelated things at that. This should bother you at least a little bit. Off the top of my head, I can avoid this approach by:

• Relying on consumer to cache the results. I don’t think so.
• Extending the class with a new “Caching class”. Yuck. Not to mention the example above uses a static method.
• Moving the two routines into to separate methods: public getEstimate() and private calculateEstimate(). getEstimate() would do the caching and call calculateEstimate() internally. Ok, but pretty tedious and causes the code to grow quickly.
• Use annotations to modify the runtime behavior of the method.

What I really want to do is annotate the method, indicating that the operation is idempotent and that the results can be cached. I don’t want to design a whole framework right now, I’ll leave that for another day. To see if it can or can’t be done, I just want to implement this single feature for now.

<?php
 
class Pi {
 
  /**
   * Use the monte carlo method of estimating PI.
   * @cache
   * @return float
   */
  public static function estimate () {
    $rounds = 1000000;
    $hits = 0;
    $max = mt_getrandmax();
    for ($i=0; $i<$rounds; $i++) {
      $x = mt_rand() * 1.0 / $max;
      $y = mt_rand() * 1.0 / $max;
      $hits += (($x*$x + $y*$y) <= 1.0)? 1: 0;
    }
    return 4.0 * $hits / $rounds;
  }
 
}
 
require_once 'Annotations.php';
Annotations::annotate('Pi');
 
// Consistent return value
mt_srand(0xDEAD);
echo Pi::estimate(), "\n";
mt_srand(0xDEAD);
echo Pi::estimate(), "\n";

Ideally, I’d like to add just that one line to the documentation comment above the method. Of course I’d need some way to trigger the parsing, so I put that at the end for the sake of the example. The code of the method remains solely dedicated to estimating the value of PI. I’ve only hinted that at runtime, it’d be great if the results were only calculated once.

It took less than twenty lines of code to implement this annotation. The methods of the class are retrieved, the annotation is searched for, and if found, the code of the method is altered to cache the results. (Actually I created a new class method that does the same thing, and rewrote the original one to do the caching only, calling the new method for the calculated value.)

Then I ran it, and sure enough it runs in half the time.

time php Pi.3.php 
3.141196
3.141196
 
real    0m1.175s

By the way, there are many attempts at a solution out there, none of which do what I want from what I can tell in 30 seconds of reading their summaries. The PHP reflection API seems to be “look but don’t touch”. I want to grope. I want a [@deprecated] method to raise a warning when the method is called. Likewise for a [@testing] method. I want my [@cache]. I want annotations for parameter validation… maybe that is a little too far, but you get the idea. The solutions I have seen let you poke and prod, but they don’t let you add/change functionality to the running code. Implementing these types of annotations would be pretty straightforward to do with the method I used above.

Since I made it work, and claim it was easy, why am I complaining? The problem is that the features I had to use to get it working are marked as unstable, unmaintained, subject to change, and risky to rely on for critical production code. The runkit extension doesn’t even compile for newer versions of PHP and some features even segfault.

So close. Half-assed.

The following is a useful javascript module for getting and setting cookies. (It uses jQuery.) The usage is as straightforward as it gets. It provides three useful functions:

1. Cookies.get(name);
2. Cookies.set(name, value, [options]);
3. Cookies.remove(name);

In addition, it provides the function Cookies.all() for debugging purposes.

var Cookies = {
 
  options :
    { path   : '/',
      expiry : null, // Days
      secure : (document.location.protocol == 'https:')
    },
 
  set :
    function (name, value, options) {
      var opts = {};
      jQuery.extend(opts, Cookies.options, options || {});
      options = opts;
 
      if (options.expiry) {
        // convert number of days from now to string
        options.expiry = new Date(
            new Date().getTime() + (options.expiry * 86400000)
        ).toGMTString();
      }
 
      // Build the cookie string
      document.cookie = name + '=' + value +
          ((options.expiry)? '; expires=' + options.expiry: '')
          + '; path=' + options.path + 
          ((options.secure)? '; secure=true': '');
    },
 
  get :
    function (name) {
      return $.trim($.grep(document.cookie.split(';'),
        function (cookie) {
          return $.trim(cookie).substring(0, name.length) == name;
        })[0] || name + '=').substring(name.length + 1);
    },
 
  remove :
    function (name) {
      return Cookies.set(name, "", -1);
    },
 
  all :
    function () {
      return document.cookie.split(';').map($.trim);
    }
 
}; // end Cookies

So what’s the big deal? The first weakness of this code is that the default options for setting a cookie are modifiable by the world. I don’t want your code changing settings for my code, nor mine for yours. This wouldn’t be too hard to fix by placing them as a local variable inside the set function. But what we’d really like is some sort of read-only access to those default options.

The second issue is more interesting. If you notice, there are four functions. And each of the functions takes a unique set of parameters. If this were Java, we could (but ought not to) implement this all with polymorphism. Sounds quite dumb, but bear with me. If the object Cookies were also a function, we could create a dispatcher and remove the need for explicit calls to the individual methods. Instead of Cookies.get(’something’), we could just call Cookie(’something’). Likewise for all the methods. Of course, you’d still be able to call the function by its full name. Let’s see it.

// Create a dispatcher
var Cookies = function (name, value, options) {
  // Case 1: no parameters supplied, dump the cookies.
  if (!name) {
    return Cookies.all();
 
  // Case 2: null value, delete cookie
  } else if (value === null) {
    return Cookies.remove(name);
 
  // Case 3: name supplied only, so get value
  } else if (value == undefined) {
    return Cookies.get(name);
 
  // Case 4: set a cookie with a value
  } else {
    return Cookies.set(name, value, options || {});
  }
 
}; // end dispatcher
 
$.extend(Cookies, (function () {
 
  // Create the private variables
  var defaults = {
    path   : '/',
    expiry : null, // Days
    secure : (document.location.protocol == 'https:')
  };
 
  // Create the public methods
  return {
    defaults : function () {
      return defaults;
    },
 
    set : function (name, value, options) {
      var opts = {};
      jQuery.extend(opts, defaults, options || {});
      options = opts;
 
      if (options.expiry) {
        // convert number of days from now to string
        options.expiry = new Date(
            new Date().getTime() + (options.expiry * 86400000)
        ).toGMTString();
      }
 
      // Build the cookie string
      document.cookie = name + '=' + value +
          ((options.expiry)? '; expires=' + options.expiry: '')
          + '; path=' + options.path + 
          ((options.secure)? '; secure=true': '');
    },
 
    get : function (name) {
      return $.trim($.grep(document.cookie.split(';'), 
        function (cookie) {
          return $.trim(cookie).substring(0, name.length) == name;
        })[0] || name + '=').substring(name.length + 1);
    },
 
    remove : function (name) {
      return Cookies.set(name, "", { expiry: -1});
    },
 
    all : function () {
      return document.cookie.split(';').map($.trim);
    }
  };
 
})());

The first thing you’ll notice is the dispatcher function. The does the simple logic for mapping the parameter combinations to the intended function. Then we extend the Cookies objects (remember, functions are just objects) with a new object. If you look, you’ll see that it is extended with a function that is immediately applied and returns an object. The advantage to doing this is that we can create closures, and encompass some “private” variables for those methods. Look at the defaults for an example of this. I added another function for inspecting the defaults, but you cannot modify them.

Now this library can be used as:

• Cookies.get(name) or just Cookies(name)
• Cookies.set(name, value, [options]) or just Cookies(name, value, [options])
• Cookies.remove(name) or just Cookies(name, null)
• Cookies.all() or just Cookies()
• Cookies.defaults()

Hopefully this makes sense. If it gives you any good ideas, let me know.

Oh, and if you really were looking for a cookie library for javascript, you can download and use this (documented and commented) one if you have jQuery installed.

Downloaded a total of 12 times

Javascript Cookies (2.57 KB)

Today I had my follow up visit to the same doctor. I went to sign in and the nurse couldn’t find my name. Who knew “one day” would come so soon.

If you are like most people, you just gave me that - “Did you just fart?” - look. Yes, JavaScript. Most of the time, my questioner doesn’t know how to respond, and I imagine they are having flashbacks of rainbow-colored trailing cursors. I figure they think I am joking, so I offer a couple reasons why I like it. I try to explain that JavaScript just does what I want it to do. I think of the best way to explain 1st class functions and closures, but I give up before I start, “I’m just kidding. I like Ruby.”

“Yeah,” they nod, “my cousin has his certification for Ruby with Rails, and he says that in ten years anybody will be able to write web software by just talking to their Mac.”

“Have I introduced you to my two year old yet? He’s good with his hands…”

From my experience, anyone that understands any concepts of functional programming generally doesn’t give a rat’s ass which language I prefer. (Ruby is my other favorite language.)

I really wish I could more elegantly explain why I like JavaScript, but the examples I come up with are usually contrived and show no benefit to doing it “that way”. At work last week, I found myself using closures to solve a problem in a way I felt was a clear example of why I like JavaScript. So, I’ll share.

Without wandering too far from the topic, I’ll explain the background of the problem. Often when you are creating an interactive web page, your JavaScript needs to write out HTML. Whether you are updating some status, showing some dialog box or message, or inserting rows into a table, it needs to be done. There are more or less two ways of doing this. Concatenating HTML and variables is ugly, but fast. Creating DOM elements and manipulating the DOM is dreadfully tedious and slow, but clean. Both methods tend to drive me crazy. I personally solve this problem by using Prototype’s Template object. You don’t need to be familiar with it, just know that you can create an object, then call an evaluate method on that object to render a template.

I put my HTML snippets into a global collection, and use them from there. This lets me keep the code at least looking clean. This is where the JavaScript fun comes in. There are a million ways to accomplish this, so I am going to throw out some requirements, just to narrow down the number of ways to do this. I don’t think these are unreasonable.

1. It must be simple. By simple, I don’t mean easy to understand, just easy to use. I like succinct.
2. It must be efficient. I only want one Template object that I can reuse, and it must be instantiated when it is first used (lazily).
3. I must be able to insert debugging statements at will. I don’t want to have to refactor code just to put logging messages.
4. It must be elegant. This is, of course, purely opinion, but wasn’t that the point of this whole thing?
5. And one more, I want to be able to reference these snippets, too. You know, a way to pass them around as parameters or something.

In other words, I want to be able to do this:

header.innerHTML = Snippets.welcome({name:'Joey'});

That would replace the token “name” with “Joey” in a template that probably looks like:

new Template('Welcome, #{name}!')

I want welcome to be a function, I don’t want it to be an object that I have to call evaluate on. To me, that violates requirements 1, 3, and 4. If I were writing this in PHP, I would do something along the lines of the following in order to meet all the requirements. (I’ll just use two snippets for the sake of the example, “welcome” and “status”. Usually I’d have many.)

class Snippets {
 
  private static $templates = array();
 
  private static function load ($name, $template) {
    if (!array_key_exists($name, self::$templates)) {
      self::$templates[$name] = new Template($template);
    }
    return self::$templates[$name];
  }
 
  private static function evaluate ($template, $data) {
    // insert debugging statements here
    return $template->evaluate($data);
  }
 
  public static function welcome ($data) {
    $template = self::load('welcome', 'Welcome, #{name}!');
    return self::evaluate($template, $data);
  }
 
  public static function status ($data) {
    $template = self::load('status', 'System is #{status}.');
    return self::evaluate($template, $data);
  }
 
}

That is pretty clean, and can be pretty much translated directly into any other object oriented language. It meets requirements 1, 2, 3, and 4. The load method caches the template objects, and only instantiates them when they are first used. The evaluate method is abstracted out to allow me to put all debugging/logging statements in one location. For some languages, requirement 5 can still be met. For example, with PHP you can “reference” which snippet to use by doing something like:

/**
 * Greet a user by name
 * @param string $name Name of user
 * @param array $template Option snippet to use, defaults to "welcome"
 * @return void Displays output directly
 */
function greetUser ($name, $snippet = array('Snippets', 'welcome')) {
  echo call_user_func($template, array('name'=>$name));
}
greetUser('Joey');

Requirement 5 is met, but requirement 4 is now broken. It’s not pretty by any definition. Less dynamic languages, like Java, will undoubtedly be even uglier, resorting to reflection. The cleaner way in Java would be to make those functions be object references instead of functions, require them to implement some interface, and have the user call the evaluate method. To me, that is the opposite of requirement 1: it is easy to understand, but annoying to use. This explains it much better, and even if you disagree, you should read it.

The JavaScript code will basically be the same, except that I will be able to reference the functions directly.

var Snippet = {
  create : function (html) {
    var template = null;
    return function (data) {
      if (!template) {
        template = new Template(html);
      }
      // insert debugging/logging statements here
      return template.evaluate(data);
    };
  }
};
 
var Snippets = {
  welcome : Snippet.create('Welcome, #{name}!'),
  status : Snippet.create('System is #{status}.')
};

That code satisfies all the requirements, and it does so beautifully. Not only is Snippets.welcome a method that I can call and pass the data to, but I can pass it around as a parameter, or assign it to a variable. Imagine adding another 20 snippets to both the PHP and JavaScript code above, and I think the effect is even more dramatic.

As I said in the beginning, JavaScript just does what I want it to do. No, I didn’t fart.

So anyway, her site finally has a new look and feel to it. I think it looks much more reputable. That is something you can fake on the web with fancy graphics, didn’t you know? In celebration of the new website, why don’t you go buy something from her? If you don’t know where her website is, then she doesn’t want your business anyway.

I’ll kick it off with a couple portraits. Although both of them constantly beg for attention, only one of them enjoys having pictures taken of him.

JD has had a mohawk for over a month now. We gave him liberty spikes for the 4th of July as headed to the park for the fireworks. He wasn’t so tough when they started exploding. You can see Zephyr in his usual pose, making sure you still know how cute he is.

Now a couple action shots.

Ok, so the action is somewhat lacking in that last one. But you should have seen him turn around and cheer when the ball finally touches a pin.

First, I need to be able to login to all the various nodes on the cluster through SSH without being prompted for a password.

ssh-keygen -t rsa
#Leave the password empty.
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

Next, get Nutch built.

mkdir -p {~/src/nutch,~/opt/nutch/build}
cd ~/src/nutch
svn co http://svn.apache.org/repos/asf/lucene/nutch/trunk .
echo dist.dir=$HOME/opt/nutch/build > build.properties
ant package

I want mine working with one copy for the crawling, and another copy for searching or just poking around with, without changing the settings for the crawling one.

cd ~/opt/nutch/
cp -r build crawler
cp -r build sandbox

~/opt/nutch now contains three directories:

1. build – Ant builds Nutch into this directory.
2. crawler – The instance used for crawling.
3. sandbox – The instance used for searching the indices created by the crawls, or whatever I want to play with.

If I need to rebuild Nutch, I can copy the project from the build directory into the other two. Since my home directory is mounted via NFS, there is no need to log into the other nodes of the cluster and repeat this process. It is done.

Now hadoop needs configured. I am only going to go over the configuration of the crawler instance. The sandbox one will not use hadoop, and therefore is straightforward. I need a place for the logs, the pid files for managing processes, and the filesystem hadoop uses.

cd ~/opt/nutch/crawler
mkdir {logs,pids}
cd conf
vim hadoop-env.sh

I added/modified hadoop-env.sh in the following manner:

export HOSTNAME=`hostname`
export HADOOP_HOME=/home/jmazzare/opt/nutch/crawler
export JAVA_HOME=/usr/local/java
export HADOOP_LOG_DIR=${HADOOP_HOME}/logs/${HOSTNAME}
export HADOOP_PID_DIR=${HADOOP_HOME}/pids/${HOSTNAME}
export HADOOP_IDENT_STRING=${USER}_${HOSTNAME}

You will, of course, adjust the path to your home directory, and your java home as needed. Now the xml configuration files need set up. I suggest you read this articles on which files should contain which information. http://wiki.apache.org/lucene-hadoop/HowToConfigure. That said, I threw everything into hadoop-site.xml. To keep these examples short, I am only showing one master node (node00) and two slave nodes (node01, node02).

conf/masters

node00

conf/slaves

node01
node02

conf/hadoop-site.xml

 
<configuration>
<property>
    <name>fs.default.name</name>
    <value>node00:9000</value>
    <description>
      The name of the default file system. Either the literal string
      "local" or a host:port for NDFS.
    </description>
  </property>
<property>
    <name>mapred.job.tracker</name>
    <value>node00:9001</value>
    <description>
      The host and port that the MapReduce job
      tracker runs at. If "local", then jobs are
      run in-process as a single map and reduce task.
    </description>
  </property>
<property>
    <name>mapred.map.tasks</name>
    <value>2</value>
    <description>
      Define mapred.map.tasks to be the number
      of slave hosts.
    </description>
  </property>
<property>
    <name>mapred.reduce.tasks</name>
    <value>2</value>
    <description>
      Define mapred.reduce.tasks to be the
      number of slave hosts.
    </description>
  </property>
<property>
    <name>dfs.name.dir</name>
    <value>/tmp/hadoop-filesystem-jmazzare/name</value>
  </property>
<property>
    <name>dfs.data.dir</name>
    <value>/tmp/hadoop-filesystem-jmazzare/data</value>
  </property>
<property>
    <name>mapred.system.dir</name>
    <value>/tmp/hadoop-filesystem-jmazzare/mapreduce/system</value>
  </property>
<property>
    <name>mapred.local.dir</name>
    <value>/tmp/hadoop-filesystem-jmazzare/mapreduce/local</value>
  </property>
<property>
    <name>dfs.replication</name>
    <value>2</value>
    <description>
      Define dfs.replication to be the
      number of slave hosts.
    </description>
  </property>
</configuration>

The filesystem hadoop uses expects to have its own directories on each host to write to, and since I am using NFS, this isn’t good. So I put the filesystem in /tmp since that is mounted separately on each node. I linked to it for convenience.

cd ~/opt/nutch/crawler
mkdir /tmp/hadoop-filesystem-`whoami`
chmod 700 /tmp/hadoop-filesystem-`whoami`
ln -s /tmp/hadoop-filesystem-`whoami` ./filesystem
bin/hadoop namenode -format

It seems to create the directories for the log files just fine, but not so much for the pids. So…

cd ~/opt/nutch/crawler/conf
for node in $(cat slaves masters);
do
  mkdir ../pids/$node;
done;
cd ..
bin/start-all.sh

The easiest way I have found to verify that everything went correctly is to run the bin/stop-all.sh command. If it complains that there was nothing to stop, then something isn’t configured correctly. If claims to have stopped everything, then all is well. If things don’t seem right, make sure you don’t have any processes that have escaped your attention. Kill those.

When you run Nutch now, it will use the filesystem from Hadoop. So any files that Nutch needs to be aware of need put into Hadoop’s filesystem. I will show the classic example of crawling the apache website.

cd ~/opt/nutch/crawler/
mkdir urls
echo 'http://lucene.apache.org' > urls/urllist.txt

This file needs put into the Hadoop filesystem.

# Must be running... bin/start-all.sh
cd ~/opt/nutch/crawler
bin/hadoop dfs -put urls urls
# You can verify with:
bin/hadoop dfs -ls
bin/hadoop dfs -cat urls/urllist.txt

More references for configuring Nutch can be found here and here show.

In particular, make sure a http.agent.name is set in conf/ and add lucene.apache.org to the “whitelist” in conf/crawl-urlfilter.txt

Now, finally do the crawl.

# Again, make sure it is all running
cd ~/opt/nutch/crawler
bin/nutch crawl urls -dir crawled -depth 3

You can monitor the output directly, or open a browser and go to port 50030 of the master node. http://node00:50030. You will be able to see the output very easily from there. Check http://node00:50030/machines.jsp and check for failures. If everything is going fine, just wait for it to finish…

After it is done, you can export the data to the sandbox.

cd ~/opt/nutch/crawler
bin/hadoop dfs -copyToLocal crawled ../sandbox/crawl
bin/stop-all.sh

Now you can do whatever you want with that; point tomcat at it, or query it directly.

cd ~/opt/nutch/sandbox
bin/nutch org.apache.nutch.searcher.NutchBean apache

I don’t have a girlfriend, but I do know a woman who’d be mad at me for saying that.

And she’s my wife…