My story began with SOLR DIH. It was way too slow. So, I ended up building another tool to replace DIH. Something friendly to CPU and memory. I did it. Not.

After indexing I realized that my text was full of ???????. WTF. Yeah, it’s encoding problem. So I’ve spent a day trying to solve this thing. What works for me was this advice from 2005:

PHP can input and output Unicode, but a little different from what Microsoft means: when Microsoft says “Unicode”, it unexplicitly means little-endian UTF-16 with BOM(FF FE = chr(255).chr(254)), whereas PHP’s “UTF-16″ means big-endian with BOM. For this reason, PHP does not seem to be able to output Unicode CSV file for Microsoft Excel. Solving this problem is quite simple: just put BOM infront of UTF-16LE string.

Example:

$unicode_str_for_Excel = chr(255).chr(254).mb_convert_encoding( $utf8_str, ‘UTF-16LE’, ‘UTF-8′);

I get no ??? char anymore. I don’t know if it is the proper way to do it. And I still get occasional htmlspecialchars invalid multibyte sequence. I think I’ll classify this solution as “miracle”.

When’s PHP 6 finally come?

Update:

CRAP. DOES NOT WORK!.

So, yes, I’ve been interested in SOLR, Apache Tika, and of course Mahout. The promise of classifying and clustering data are enough to persuade me digging up examples about Mahout. So far, what really helps is Seinfeld demo example. It gives me a proper example to try. We can replace the data with our own to get the gist on how Mahout would work.

However, I haven’t get the gist yet. So far, I’ve tried to cluster 2 datasource. One of them is blog post from navinot.com. Here’s an excerpt from cluster-dump:

C-18 [Ponsel, Mobile, Internet, Mobile internet, Iphone]
- /6 Hal Tentang Mobile Internet.txt
- /Mobile Application_ Masa Depan Yang Ditunggu?.txt
- /Netbook_ Bakal Lenyap Seperti PDA?.txt
- /Premium Mobile Internet?.txt
- /The Gaps in Indonesian Internet.txt
- /iPhone & Telkomsel_ Deal or No Deal?.txt

I’m imagining Mahout with cluster it into similarity groups. My guess is, it was clustered by keyword. I was using kmeans.

Anyway, obviousy we need to filter out stopwords. Mahout can read directly from SOLR/Lucene index. But I didn’t have much luck on it. Something to do with empty terms or whatever. Probably, feed my raw data to SOLR and then query it out to get text files will make a decent workaround.

That’s a wrap for today. Time for Pocket Legend!

Fear not. Just download mahout 0.5, both source and binaries. Extract them, it will reside on the same folder i.e: mahout-distribution-0.5. Now, open up that pom.xml. Find lucene and replace 3.1.0 with 3.4.0. I reckon there are only 4 of them. The do mvn install. You may want to skip tests with: mvn -DskipTests=true install.

Once done, do: export MAHOUT_CORE=1

Run mahout from mahout-distribution-0.5/bin folder.

I don’t get index incompatibility anymore. But, I keep getting not enough term vector on document. Even I’ve set the schema.xml dan reindex my docs.

Will write more once I pass it.

1. Hatsune Miku, either having her figure or ultimately watching her live concert.
2. Dolfie, see one, touch it. Owning it can wait until I have moved into my own apartment (and permission from my wife)
3. Akihabara. It’s a common thing in everyone’s list I believe. Probably visiting a maid cafe would be nice.
4. Comiket. Seeing a field of figures would be awesome. The crowd looks scary tho.
5. Tokyo Game Show. Same as above.
6. My own figures collection. I believe I have a different taste with@dannychoo. Not on the dolfie part tho . I have been wanting a Macross figure and other figures I saw in some dannychoo’s pics.
7. See cosplayer at Yoyogi Park.
8. Tanabata festival. Firework show. I want to wear yukata someday.
9. Taking photos of lots sailorfuku school girls.
10. Talking of sailorfuku, I love Scandal girlband. I want to have their merchandise.
11. Dir En Grey concert? I want to scream in Dozing Green song.

To grab more clues, run this line on bacula-fd machine:

sudo /usr/sbin/bacula-fd -f -d100 -c /etc/bacula/bacula-fd.conf

Then do bconsole dance on bacula-dir machine. Use “status” command to test connection to client. I you see cram-md5 authentication failed in bacula-fd output then you have the same problem as I did. Otherwise, check your connection between bacula-dir and nacula-fd

Here’s the solution:

in bacula-fd.conf:

Director {
  Name = bacula-director
  Password = "remote-fd-passwd"
}

“Name” should be your bacula-dir Name. You can found this in bacula-dir.conf. See below:

Director {                            # define myself
  Name = bacula-director
  DIRport = 9101                # where we listen for UA connections
  QueryFile = "/etc/bacula/scripts/query.sql"
  WorkingDirectory = "/var/lib/bacula"
  PidDirectory = "/var/run/bacula"
  Maximum Concurrent Jobs = 1
  Password = "blahblahblah"         # Console password
  Messages = Daemon
  DirAddress = 127.0.0.1
}

Then the password part on bacula-fd.conf should be the same with your client definition in bacula-dir.conf. eg:

Client {
  Name = remote-fd
  Address = remote.fd.ip
  FDPort = 9102
  Catalog = MyCatalog
  Password = "remote-fd-passwd"          # password for FileDaemon
  File Retention = 30 days            # 30 days
  Job Retention = 6 months            # six months
  AutoPrune = yes                     # Prune expired Jobs/Files
}

Don’t forget to restart bacula-dir and bacula-fd after modifying conf files. Good luck!

The definition of the catalog Mycatalog contains a line starting with ‘ dbname = “bacula;”‘. The semicolon inside the quotes should follow the quotes, so should start with ‘ dbname = “bacula” ;’

Another tip, Pool resources by default are not enabling auto-volume naming. This is pretty annoying for a newbie. And it is way better to have it enabled by default to make it work out-of-the-box. To this, add label format option into your Pool resource definition. Something like this:

Pool {
  Name = File
  Pool Type = Backup
  Volume Use Duration = 23h
  LabelFormat = "VolFile-${Year}-${Month:p/2/0/r}-${Day:p/2/0/r}"
}

It will automagically creating proper Pool Volume when job runs, eg: Vol-2011-12-02.

You can use bat GUI to list your jobs and volumes. To restore files, see my tips here.

PS:

When you changed bacula-sd.conf, aside from restarting bacula-sd service do restart bacula-director service as well.

Because it is logical. Youtube has many great content. I love watching CN Blue video on it all the times.

Why does it feel awkward?

Awkward? How? You mean the pop up window? Because making the Youtube sidebar reside in G+ page would take space. It’s logical.

Will other button from Google product catalog follow suit?

Obviously. Perhaps. Aside from Youtube? Blogger?

Are you sure?

What? Why looking at me like that? Do I look like Vic?

More resource on javascript and NPL, there are some on github. There are some Entity Extractors as well. And it concludes that some extractor simply cannot get away from using a dictionary. Maybe I will end up with one.

That made me think, I and many of us have been trapped at the end of the tides for too long. WE never see the small ripple. But we always ended up being washed away by the resulting tide. That is probably the “perks” of staying in developing country. To make it even worse, the internet that has been helping us cutting the gap has also keeping us away from the ripple. Small changes, controversial innovation that may be the next mainstream.

Needless to say, given those ripples put up upon our eyes, determining the next mainstream will need eagle hunch. Still, the years of gap feels unfair. Those years may have been spent on learning and honing what matters most.

All we can do is cutting more gap, catch up faster. And hopefully we will land on the same plateu. Hear what everyone hear, see what everyone see.

Sounds like american dream eh?

Today, I’ve just found another approach on understanding user query. I thought it will help alot if we can determine the Subject, Predicate and Object of a query. We do\n’t need to understand the whole sentence but we do need to extract the essence of the query. What should our clone do if user says: how is the weather? where is bandung? do I have any meeting today?

Fortunately there are free implementation of typed dependency. If you want to know more about typed dependency, just google it. I will only give you an example of it. Given the query “how is the weather in jakarta”, typed dependency analysis will give us:

advmod(is-2, how-1)
det(weather-4, the-3)
nsubj(is-2, weather-4)

From this output, we can use the availability of subject or object to determine the essence of a query. Example above show us, it probably, weather is the essence of the query. You can test more typed dependency here. Below are some more examples:

Do I have meeting today

aux(have-3, do-1)
nsubj(have-3, I-2)
dobj(have-3, meeting-4)
tmod(have-3, today-5)

call John

amod(John-2, call-1)

make appointment with John on 3

dobj(make-1, appointment-2)
prep_with(make-1, John-4)
prep_on(John-4, 3-6)

texts John, send me detail

prep_text(send-4, John-2)
nsubj(detail-6, me-5)
ccomp(send-4, detail-6)

From above example, it is possible for us to choose a pattern as a trigger for a datasource query. However, it will not always adequate. Some question may be hard to understand, still. As is it still too vague, such as: how do I get home. To understand this, we need to be aware that “home” is a destination/location. This should trigger some sort of map datasource.

I have been imagining the clone as a pluggable framework. The main function of the host program is to provide as many analysis as it can, via plugins. And then decide which datasource plugin to trigger. Typed dependency should be one plugin, feature extraction should be another plugin.

Hmm, interesting.

PS:

There are more dependency parsers I still need to check.