Monday, March 26, 2012

Word Tokenization

Word tokenization is an NLP task where one breaks a string of characters--letters and spaces--into words. In English and other words with spaces, this is pretty easy; it's virtually been done by the writers (with exceptions of sentence terminal periods, n't, and 's).

However, other languages, like Chinese, require a lot more work. Chinese doesn't put spaces between words.

Often, this is written about as a purely NLP task. This isn't quite correct. Spaces proper weren't invented until the middle ages. ASI'LLDEMONSTRATEHERE,YOUREALLYDON'TNEEDSPACES.ITMIGHTBEALITTLE HARDERTOREAD,BUTTHEANCIENTROMANSBUILTANEMPIREWRITINGTHISWAY. WORDTOKENIZATIONINTHEAUDITORYANDVISUALSENSEISEASILYPARTOFOUR LANGUAGEFACULTY.

Since we can do the task naturally, it's just as much a problem of linguistics as natural language processing.

Sunday, March 25, 2012

No Exceptions



I haven't kept up with politics lately. Watching a bunch of idiots run around and call one another even stupider names is entertaining, but I am preoccupied. To play catch-up, I was reading about that Santorum guy everyone is talking about, in particular his attention-seeking crusade against porn.

For a long time, I've tried to understand the conservative mindset, and while reading Santorum's opinions, I had a realization. Santorum spoke of pornography as if it angered God himself in direct opposition against the United States. I'd heard this perspective before, but for the first time, I thought of the story of Sodom and Gomorrah, where God sought to destroy the cities because they were full of sinners. It became clear that this is what hardcore conservatives actually think will happen to the United States if we don't fall in line with their image of what God wants.

This may not seem a novel concept, and it's really not. What I realized was that we--the further ends of the right and the left--aren't that different. Part of the reason I abandoned my faith was that I got tired of writing exceptions. The more I learned about the world, the more exceptions I had to creatively twist into place to include faith in my life; for example, "well, things were different in Bible times," "well, that line of scripture is archaic, they just didn't know any better." Fundamentally, I was lying to myself, as does anyone justifying something they know is wrong.

Instead, I put my faith in the capacity of rigorous observation to yield knowledge about the universe. The knowledge isn't purely faith per se, as it's constantly under scrutiny. What's really taken on faith is the method, though it does have results to show--the computer you're reading this on ought to suffice as a triumph of science.

In the end, I've put my faith in this group of people over the centuries. I've trusted them with the work of explaining the universe and their way of doing it. In the conservative mindset, they too have put their faith in a group of people, just a different one. Just as I could not allow exceptions to what I had learned, they do not allow exceptions either. They really think the return of God is at hand because that's what the Bible says. They really think that God tears down forsaken nations.

As much as I disagree with them, I respect their integrity. Perhaps I see a bit of myself in them; despite coming to different conclusions and even having different techniques for coming to those conclusions, we have some core value that is unwavering.
-->