so for this, i’ve decided on DeviceAtlas.  just head on over to DeviceAtlas and open a developers account – you will get a one year developer’s license to play around to see how cool it is.

after registering – click on the downloads link – then go to the php example and download the source files.  after de-tar-and-un-zipping the file, just extract the contents.  at the moment all we are really interested in, is the Mobi/Mtld/DA directory and its contents – also create a json directory and drop the Sample.json file inside.

dump everything into a web directory somewhere.

create a deviceatlas.php file in the working directory root and put the following inside.

//includes the DeviceAtlas API

include 'Mobi/Mtld/DA/Api.php';//fetch the previously populated apc cache instance into a variable

$tree = apc_fetch('tree');

//Get the properties from a variable containing a JSON tree loaded from cache

$properties = Mobi_Mtld_DA_Api::getProperties($tree, $_SERVER['HTTP_USER_AGENT']);

//Output the phone properties for your viewing pleasure

print_r($properties);

‘hhmmm, that’s it?’, you might ask, and if you were clever you would have noticed something strange in the code above.

yip, that is pretty much it – but you also need caching.  the apc_fetch(‘tree’) command in the code above actually fetches the json file already dumped into a ‘tree’ cache instance in PHP APC (Alternative PHP Cache).

why PHP APC, you may also ask.  well, besides using the overstated memcached (you might want to read this), there are some methods to the madness.  with PHP APC, the cache is shared on a server.  it might not be accessible from other servers, but this is not really a con in my mind.

if you wish to scale your application later one and add another server, there is going to be some issues, you might say – but there will also be issues if you have millions of users hit one central instance of DeviceAtlas.  so, having one instance of DeviceAtlas per server might not be too much of an issue.  also when it comes to updating the json file, you might want to consider the single point of failure issue of just having one server.

so have a couple of servers, and just write some scripts to update the single instances of DeviceAtlas, rather than updating one, already busy server which still needs a cache update.

but, back to business.

first you have to install PHP APC.  i found this handy guide on the net, which helped me install it on my debian system and integrate it with apache and php5 – there is also a guide at the bottom for fedora based machines, so don’t fret my linux loving friends

you can create a php script to run the following – which should cache your DeviceAtlas tree into PHP APC.

//ads the sample json data from DeviceAtlas

apc_add('tree', Mobi_Mtld_DA_Api::getTreeFromFile("json/Sample.json"));

from the first code example, the output might look something like this (depending on your phone)

Array
(
[wmv] => 0
[vendor] => Nokia
[mobileDevice] => 1
[memoryLimitMarkup] => 357000
[memoryLimitDownload] => 61440
[midiMonophonic] => 1
[midiPolyphonic] => 1
[mpeg4] => 1
[3gpp] => 1
[drmOmaForwardLock] => 1
[drmOmaCombinedDelivery] => 1
[drmOmaSeparateDelivery] => 1
[markup.xhtmlMp10] => 1
[markup.xhtmlBasic10] => 1
[image.Jpg] => 1
[image.Png] => 1
[amr] => 1
[mp3] => 1
[aac] => 1
[h263Type0InVideo] => 1
[gprs] => 1
[edge] => 1
[image.Gif87] => 1
[uriSchemeTel] => 0
[qcelp] => 0
[hsdpa] => 0
[amrInVideo] => 1
[3gpp2] => 0
[displayColorDepth] => 18
[model] => N70
[https] => 1
[image.Gif89a] => 0
[umts] => 0
[displayWidth] => 176
[displayHeight] => 208
[markup.xhtmlMp11] => 0
[markup.xhtmlMp12] => 0
[csd] => 0
[hscsd] => 0
[id] => 204255
[_matched] => NokiaN70
[_unmatched] => -1/5.0609.2.0.1 Series60/2.8 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Link/6.3.1.13.0
)

you can view the attribute explanations here

now you can use the phone’s specific properties to your advantage.  you also might want to look at the user agent switcher add-on for firefox for testing purposes.

that’s it.  now every once in a while you can just update the json file, delete the current cache (http://www.php.net/manual/en/function.apc-delete.php) and bob’s your device detecting, apc caching uncle.

as a developer, we kind of try and convince ourselves that developing for mobile and developing for a desktop browser is kind of the same thing.  but we all know that this is a stalling technique for the inevitable, since invariably it becomes a whole different field of play.

every little feature you add, every little flow created and every branch of navigational hierarchy is a challenge on its own.

enter the game theory pattern

this is was where my idea of a game theory patterns came into play.  game theory is a branch of applied mathematics, most notably used in social sciences (economics, biology, engineering etc.).  it is also used a bit in computer science, although referred to in that field as artificial intelligence.

thanks wikipedia

game theory allows someone to apply a bit of mathematics to capture data from a situation and use that data to build a strategic base, which allows you to make decisions to determine your success.

take for example morning traffic – you have 2 routes to take to work – one a bit longer than the other, but the longer route is less likely to be busy.

what do you do?  most of us will take the pro’s and con’s of both routes and use that to base our decision upon.

game theory works exactly, well almost, the same.  except you will probably use a bit of math to finalize your decision (average speed, distance etc.)

so why not use game theory in development, especially in a field like mobile, which is very limiting and very frustating at first.  but the good kind of limitation, the kind of limitation that could possibly create innovation.  the guys behind a phenomenon like twitter decided to limit people to 140 characters.  and despite this limitation, people have found innovative ways to increase the effect of that 140 character limited textual communication.

game theory would allow us to mathematically decide on feature sets, the use of algorythms, objects and could even change the flow of an application.

say on an example mobile application we have 3 major features – chat, activity, content sharing.

which one do we choose as our killer feature, and which would probably make the application more bloated and more problematic to use.  why not use the main variable in the equation to help you decide – the user.

i often secretly chuckle to myself, when a system gets developed for one purpose and all the users start using it for a totally different end result, no one really every thought of, or intended for.  you could use this to your advantage with game theory.

a simple case could be made using application flow as an example.  we have a menu with chat, activity feed and shared content.  which one is more important, which one would you make easier to access for the user.  by simply adding a click count to each feature, you could sort a menu by popularity, rather than obvious choice as this will invariably change someday.

if users decide this month, chat is the next best thing, let chat be number one on your menu – and i know some user interface expert will maintain that consistency is important, but if users consistently choose chat as their most important feature, what would the issue be?

this would change the way you develop and architecture software as well.  since you know chat is gaining in popularity according to your available data, then you should be able to utilize servers better for your chat functionality.  write scaling scripts which would use databases sharding.  and on the other hand, allow your game theory pattern to simplify things – why use a 40 slave/master instances of a database if all you need is one.  eliminate unnecessary overheads in that way – more layers makes for more complexity, makes for more overhead, and will eventually kill your application.

i’ve still got to convince a couple of people to maybe try this game theory pattern approach to development and if, by chance, martin fowler reads this article:

‘please martin, consider this pattern as a topic for a new book! ’

so i’ve been looking into solr the last couple of days.  solr is relatively new in the arena and probably outshined a bit in popularity by other search engines such as lucene and nutch.  “but why solr?”, you may find yourself asking.

Well solr has a couple of tricks up the sleave – which is likely due to the fact that its a fresher version of the old, dare i call it legacy, search engines.

some of the features of solr includes:

• highly scalabe java search server (as i will try to show you through this article)
• works with the lucene search library (tried and tested)
• you can update your engine with xml using a kind of lightweight restful service.
• can parse html, openoffice, microsoft office suite documents, pdf’s etc using lucene parsers.
• custom tokenizer, filter and analyzer steps for control over indexing and query processing
• extremely rich indexing of fields and metadata, including numbers
• can combine fields for fulltext-type searching
• tuned for high performance even when updating
• spell checkers included, etc, only 2 name a couple of features.

click here to view a more complete list.

solr currently have api’s for ruby, php, python, json, forrest/cocoon.

the obvious elephant that is missing from the list above is the fact that solr is not compiled with a crawler of some sort.  luckily some friendly open source guys already released a guide for integrating nutch with solr for an all round experience.

you might also want to look into localsolr – which enables geographical and spatial searches with solr

now… where to begin:

first things first – since solr is java based, it has to be wrapped in a java container of some sorts.  i choose the friendly open source tomcat as a basis.  i installed tomcat 5.5 using this helpful guide as a reference.  read a bit more than what is required and make sure you understand the concept of using web applications and the configuration of tomcat.

after tomcat is installed, its time for solr.  and as luck would have it – a solr installation in tomcat is probably one of the easiets most straight forward ways of doing it – and i just followed this handy guide, provided by solr, as a reference.

now after solr is up and running, you should be able to access your admin screen via the following url http://127.0.0.1:8180/solr

that would be assuming your tomcat port is 8180 and solr is installed under the directory solr in your web applications.

now after installing solr – and accessing the admin screen you should – you should be able view your configuration and perform a basic search.  to update the schema and import data into solr is a whole other post on its own, and as soon as i figure it out, you will be the first to know – but for know we are going to focus on replication and scalability.

look at the following diagram (done by me using gliffy)

as you can see above – there is 3 main levels

1. load balancers (wrapped in a monitoring service – keepalived which is optional)
2. solr slave instances
3. solr master

you can see that it isn’t the most original idea in the book, but it works.  especially if you should run it within a cloud where your instances could almost be infinite.

searching with solr

lets start from the bottom up – at the base – you have the solr master – which would be the solr instance where all your data importing from xml happens.

above that is the slave instances, which is will replicate their indexes from the master.  the slaves will in turn handle all search request from the load balancers which sits at the top.  this reduces load on the master, which will be used as the primary indexer.

also the load balancers will be wrapped in a monitoring system, which will alert you in case something goes wrong, and also reduced the risk of a single point of entry failure.  this monitoring system is really optional, since the load balancer i will be using has built in failover checks.  but nevertheless it is good practice to look into a monitoring system like keepalived for your architecture.

now for the specifics – to update the index of your master is almost a trivial exercise – since solr has a kind of a restful approach of doing this -

#the following would be in a .sh file
FILES=$*
URL=http://localhost:8180/solr/update

for f in $FILES; do
echo Posting file $f to $URL
curl $URL --data-binary @$f -H 'Content-type:text/xml; charset=utf-8'

#send the commit command to make sure all the changes are flushed and visible
curl $URL --data-binary '<commit/>' -H 'Content-type:text/xml; charset=utf-8'

so – if you save the above script as for e.g. post.sh, make it executable (chmod a+x post.sh), then you could run it with a xml dataset (./post.sh dataset.xml), and it will update the index using its semi restful interface.  all you have to do is change your url in the above script to the solr instance you wish to update.

then comes the replication of data to the slaves – this is done by using scripts in the bin directory of your solr installation.  here is the descriptions of these scripts from the solr websit.  we would most likely be interested in the snapshooter – which takes a snapshot of the current master index – from there you would use the snappuller – which would pull the latest snapshot from the master index and update the index.

after you’ve mastered these scripts, you can store run them as cron jobs every so often and update the slave indexes.  i’ve also read on there site, that in the new release they would be compiling replication as a feature into solr, where you can actually just tweak a couple of configuration settings, and it should replicate without external tools, so just keep a lookout for the new solr version.

load balancing with haproxy

so as a load balancing solution i chose haproxy, which offers high availability and balancing for tcp and http based applications.  haproxy does not deliver content such as apache, neither does it do caching in a way squid does it, but it’s small, simple and works very well.

to install haproxy:

mkdir /opt/haproxy
cd /opt/haproxy
wget http://haproxy.1wt.eu/download/1.3/src/haproxy-1.3.15.7.tar.gz
gunzip haproxy-1.3.15.7.tar.gz
tar -xf haproxy-1.3.15.7.tar
cd haproxy-1.3.15.7
make
cp ./haproxy /usr/bin/haproxy

then create a file called haproxy.cfg that contains the following data:


global
log 127.0.0.1   local0               #logs all haproxy info to local0 log
log 127.0.0.1   local1 notice    #logs all notifications to local1 log

daemon                                     #specifies haproxy to run as a deamon instance
maxconn         4096                 # total max connections (dependent on ulimit)

defaults   #setup some default values
log            global
mode       http
option      httplog
option      dontlognull

clitimeout        60000       # maximum inactivity time on the client side
srvtimeout        30000       # maximum inactivity time on the server side
timeout connect   4000        # maximum time to wait for a connection attempt to a server to succeed

option            httpclose     # disable keepalive (HAProxy does not yet support the HTTP keep-alive mode)
option            abortonclose  # enable early dropping of aborted requests from pending queue
option            httpchk       # enable HTTP protocol to check on servers health
option            forwardfor    # enable insert of X-Forwarded-For headers

balance roundrobin            # each server is used in turns, according to assigned weight

stats enable                  # enable web-stats at /haproxy?stats
stats auth        admin:pass  # force HTTP Auth to view stats
stats refresh     5s        # refresh rate of stats page

listen myloadbalancer 127.0.0.1:8888 #where the loadbalancer should listen for requests
server slave1 192.168.1.6:8180 #a slave
server slave2 192.168.1.6:8180 #another slave

–

and that is it.  now all you do is run:

/usr/bin/haproxy -f haproxy.cfg

and bob’s your uncle – just go to http://127.0.0.1:8888/haproxy?stats to verify that it is running (the username is: admin and the password: pass – as specified in the config file above)

just note – if you get binding errors, make sure the port you specify in the config file is open, and not in use by something like apache, tomcat, yaws etc.

now if you have all your slave instances running a version of solr – you should be easily able to connect to them in a round robin fasion – for e.g. connecting to http://127.0.0.1:8888/solr – you can confirm this in the syslog or with the above stats link.

monitor that architecture, with keepalived

last, but not the least would be the enabling of the load balancing monitor – keepalived was suggested to me as a viable option, however, with haproxy  you can have do failover checks and run multiple loadbalancers under a shared ip so this isn’t really a necessary part, but as explained earlier, it is good practice to run a monitor like keepalived on your system.

conclusion

because it is late, and i still have to proofread this sucker, i haven’t really got around to keepalived, although – do yourself a favor and read up on it – i know i certainly will have a cup of java and research it a bit more.

so – there you go – you have a fully load balanced system – running a search engine, which should scale horizontally with your hardware buying budget.