Subversion Do and Don’ts
When developing on a project with subversion (or any other centralized revision control system) do not:
- commit without useful log, unless you are not a human (eg: scripty)
- pass 1 commit/minute commit rate
When developing on a project with subversion (or any other centralized revision control system) do:
- Test your changes locally
- Use a staging (devel) server
History Meme
toni@toni:~$ uname -a
Linux toni 2.6.18-53.1.13.el5 #1 SMP Tue Feb 12 13:01:45 EST 2008 i686 i686 i386 GNU/Linux
toni@toni:~$ history|awk '{a[$2]++ } END{for(i in a){print a[i] ” ” i}}’|sort -rn|head
176 cd
101 ls
89 svn
84 cat
69 less
60 sudo
53 ps
33 ping
29 vim
26 find
Scribefire is not f***ing working in FF Beta 5. So I’m manually write my post in my WP dashboard. I’ve tried to do this post twice in Scribefire. Both never made it to this blog. So, I’m pretty pissed off and will left the interpretation of my bash history to you.
Database Scalability (and Drupal)
Disclaimer:
- I am not a DBA nor having any other database expert title attached.
- I am not Drupal expert either
Having spend some good days browsing the net and read like crazy, I managed to write up the following.
Database Scalability options:
- Scale up
- more powerful hardware: RAM, processors, storage
- Scale out
- Federation
- MySQL 5.x support federated table (remote table). However its still have some issues: heavy traffic between federated server
- Benefits: spreaded storage, relatively reducing main server load (delegated to federated server)
- Disadvantages: heavy network traffic between federated server
- Issues: network connection capacity between federated servers
- Sharding (Partitioning)
- Benefits: spreaded server load, spreaded storage
- Disadvantages: relatively complicated since it’s involving application layer changes
- Issues: Involves application layer
Federation is more transparent to developer as application can be totally unaware of the federation setup. However, its bottleneck for sure is giving a limit to scaling (out).
More to explore:
- Replicating federated database.
- Load balancing federated database;
Sharding on the other hand, while giving enourmous flexibility to scaling (out) options, is likely requiring ‘built-from-scratch’ application setup/environment. A special data hashing/sharding logic must be incorporated to the application layer in order to implement sharding, which sometimes against the intention of some framework.
What about Drupal then? Drupal depends so much on the node table. Though developer can new type of nodes which uses external table (for extra attributes), those nodes will still be related to the core node table (API restriction, eg: node_load, node_save, and other node_xxx) From these points, the only avaliable scaling option for Drupal would be scaling up or using federated option (which is somehow felt like scaling up, eventually)
There has been rumour about ASQL (automated sharding proxy) but I haven’t found any available code yet.
PS:
- For load balancing, there are many options includes using hardware (BIG-IP) or software (mysqlproxy, sql-relay, sequoia)
- Sharding approach is similar to BigTable approach which spreads storage to localize (spread) load via colum based database structure.
- The more you want to scale (your app/database) the more you will love data redundancy and giving up normalization
References:
[1] http://en.wikipedia.org/wiki/Federated_database_system
[2] http://dev.mysql.com/tech-resources/articles/mysql-federated-storage.html
[3] http://www.onlamp.com/pub/a/databases/2006/08/10/mysql-federated-tables.html
[4] http://buytaert.net/scaling-with-mysql-replication
[5] http://www.johnandcailin.com/blog/john/scaling-drupal-step-four-database-segmentation-using-mysql-proxy
[6] http://mysqldba.blogspot.com/2006/11/unorthodox-approach-to-database-design.html
[7] http://sequoia.continuent.org/HomePage
Dear lazy web, does Lua have regex support?
mysql-proxy, being developed by MySQL AB itself, is very powerful tool to have. It can intercept raw mysql packets from authentication to query and query result. Creating a read/write load balancer has never been easier. Instead of filtering query (as seen in sqlrelay), we can use mysql-proxy to defines which back-end should be used for reading (–proxy-read-only-backend-addresses) or writing. No need to mess with each query executed. However, I miss regex in query filter or router of sql-relay. I wonder if we can use regex in Lua so we don’t have to do substr thingy in read_query() hook.
Powered by ScribeFire.
sqlrelay and mysql-proxy
I am not yet going to have the vs tag. I just want to do a simple comparison here. Installation and a quick glimpse on both tool.
I have been astonished by the query filter/route feature in sqlrelay. I’ve setup a simple simulation in my box using two databases. And I have been happy about how I can direct my query based on some lines of regex. Working with mysql client directly and sqlrsh tool was totally smooth. However, making it works with LAMP(hp) was totally PITA. The mysql drop-in library can’t help here. Casting LD_PRELOAD spell before starting httpd was futile. Maybe it has something to do with mod_php or something else. I haven’t dig deeper. There’s only one option left — as working directly with sqlrelay-PHP API is no-option — that is using the available PEAR::DB driver for sqlrelay. That would be scwheet if only Drupal doesn’t start dropping PEAR::DB support in 4.6. Gah.
Okay, it’s now time for mysqlproxy to shine. There’s no ready library for CentOS/RHEL 5. Compiling from source. Requires Lua. Must compile Lua to get the recent 5.1.3. `make linux install` is failed on doc section. Commented out doc section. make linux install passed. but no lua.pc in any pkgconfig directory, ./configure on mysqlproxy failed. Copying lua.pc manually doesn’t work with configure. Manually use pkg-config –libs and –cflags to supply LUA_CFLAGS and LUA_LIBS for ./configure. make failed on dlopen stuff. Browsed Google. Google said: ./configure LDFLAG=’-ldl’. make && make install is OK.
Run mysqlproxy. Fired up mysql -u root -P 4041. show databases;. Errrrr. Nanni .. Nanni … Stuck. Bump. Will see you later tomorrow.
Update:
On one end:
toni@toni:~/public_html$ mysql -u root -P 4040 Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 0 to server version: 5.0.0 Type 'help;' or '\h' for help. Type '\c' to clear the buffer. mysql>show databases; ERROR: Failed to get a cursor id. A network error may have ocurred. Success mysql>
And on the other end:
toni@toni:~/Devs/mysql-proxy-0.6.1$ sudo /usr/local/sbin/mysql-proxy file network-mysqld-proto.c: line 210 (network_mysqld_proto_get_int_len): assertion failed: (*_off < packet->len) Aborted
Update #2:
mysql-proxy works swell today. Dunno, maybe I needed to restart mysql yesterday. mysql-proxy fives a more granular filter feature compared to sqlrelay simple regex. However, it must be paid by learning Lua.
Another tips, always use explicit mysql –host=localhost when you want to use mysql-proxy proxy feature. Otherwise your connection will likely goes via local socks, which is sucks. Eg: mysql -P 4040, this one will not connect to mysqld instance via port 4040, alas (apparently) it will use socket and ignores the port. Crazy param bastard took me forever to figure out.
Powered by ScribeFire.
Determining Indefinite Number
I just run out feeds to read (haven’t got private time to read dannychoo’s), so I start pushing my not-yet-addded reading materials into my GReader. Then, I stumble upon this search pagination pattern from ypattern. I don’t have any issue with regular pagination pattern, but the search pagination pattern tickles me a bit. As stated there, search result can be indefinite (and making it definite could take up some valuable resource) so how can we render the control effectively (without renering links to empty page). What upper bound we should display? I wonder there’s a special technique to work on this search related pattern.
have anything to say?
Powered by ScribeFire.
kernel panic, missing /data
Monday is here allrite. Things are cool except one thing. I got a frozen box after lunch. Restarting the box lead me to kernel panic cannot found /dev/root. Oh, crap. Rebooting the machine a few times didn’t solve the problem.
Okay, looking at the error, it seem I have a problem with my / directory. Okay, it seem that I have to manually fsck my / manually. How? This is a SATA, and I can’t found any adapter to attach to. A quick solution was suggested from a fellow from Customer Support Technician: a live cd.
Okay, I can boot with a Kubuntu 6.10. fsck on /dev/sda10. Stucked in the middle with freezing keyboard. Reset button to the rescue. And then retrying fsck. Oh, crap. Holy Batman, I have a darn long fsck log. And most lines gave funny feeling into my stomach. Things are going to be messy.
And it does. After a long fsck, I mounted the device and voila! ls command only return “lost+found”. Shimatta~. There’s nothing much left but some sql data files, some .so and some useless files. My ‘/data’ is now completely gone. ‘/data’ is my universal trunk. I have lots of files mindlessly thrown there. I can’t even remember the details :D.
Fortunately, /home was separated. Documents are also safe. I symlink-ed it to my LVM volume, which is phisically located on other disk. Yatta~! Hopefully nothing important are lost.
Well, I’ll consider this a bless of not having to think and filter my /data of which I have to clean up/back up anyway. Hahahaha.
Okay, I’ll need to move some source to my laptop to continue my week. See ya 
Fun map mashup ideas
I admit, this is my first time looking deeper into Maps API. There’s been a lot of overhaul since my last visit a long time ago. There’s this mapplet thing, promising to be able to make maps based app in within 5 minutes. I totally hooked on it.
I have these several data in my kitchen sink. I’ve created several Dapps and pipes. I thought why not using them for some fun activities with Maps API. There’s this geocoder object capable of translating human-readable address into machine-understandable number. OTOH, I have a city list from 21cineplex.com available via Dapper/Pipes. Voila, with a little _IG_FetchContent method and a few lines of javascript (in fact, it’s available on geocoder example) I can make a marker for every city having 21cineplex theater. And to go further, I’ve also made a dapper capable of fetching now playing list from a certain city_id. Okay, this would be great. I’ll fill the marker with now playing list, on demand when user clicked the marker. And the marker for the cities will be initialized with realtime data as well. Now, that’s what I call mashup. A pure mix and match on the web. I don’t even need to provide any backend or space nor bandwidth for hosting purposes.
Too good to be true. If only Dapper is not having error when fetching from 21cineplex.com site, I’d have been able to show something cool now. There’re two possibilities: the error is exclusively on dapper side, or 21cineplex.com is blocking dapper user agent from fetching their content. And if the later is what really happen, then it would be a total-foolness-moron-score-one-they-do-not-get-web-two-point-oh curse.
Powered by ScribeFire.
Why, IE?
Dear AlMighty, why are You keeping IE market alive? Is it some sort of a balance of goods and evils? I know it’s up to us to choose. But dear Lord, some of us are just plain dumb and not capable of choosing in between.
Dear AlMighty, shed your light upon us. Show us the the path You’ve been blessed. The standard compliant one.
And dear Lord, give me the strength to withstand this wreck created by “Why your job look awful in IE”.
Now I lay me down to sleep
Pray the lord my soul to keep
If I die before I wake
Pray the lord my soul to take
Cineplex21 Plasmoid
Yesterday was the data engine, and today is the plasmoid. I have to refine the data structure within Cineplex21 Data Engine. Mostly to comply with the API. There’s no ultimate rule on how you should structure your data. Though it would be nice to follow the example given on other plasmoids. At least until there’s an official standard for it.
I started this plasmoids from Clock code. Copy the folder and rename the bits. CMakeList.txt almost made me pulling out hairs, until I found the typo. I missed one line that still referring to ${clock_SRCS}. That’s what you’ll get for being lazy :p
So, that’s it. You can see it in the screenshot. A very minimal look with very minimal info. TODO: add schedule information, and maybe put some credits information in it. I need to tell that data is pulled from 21cineplex.com via Dapper.
For you Indonesian, put this plasmoid on your desktop! Huzzah for KDE4!! Banzai~
PS:
it’s now hardcoded to show schedule for Ambarukmo Plaza Studio. It’s a matter of config to make it possible to choose which theater to display.
