2012/12/23

interesting times

Only a year ago I was sitting home on leave after telling some bigwigs at the company exactly what I thought of them and of the way they were running things. Bit like this, with about as much drama, just as true, less funny though. And I had been working in the company - a retailer based in Belgium - for 18+ years.

A year later I have published a book @O'Reilly, am running my own consultancy company, am working as consultant for two big companies and have taught a course of NetKernel at a third.

I have chinos and shirts in my cupboard now and some of these actually do have a colour/color other than black. I don't tuck the shirts in yet, but as the belly I got from the good life is receding, that will soon become an option. I now own a pair of decent looking shoes too.

Most of the time I work from a shared office located across from my son's school. Occasionally I have to go see my customers and mentors though, so I spent time in Minneapolis, Brussels, London and even Chipping Sodbury over the past year.

Professionally I've learned to do whole infrastructure setups with Amazon Web Services, am ever more proficient at Resource Oriented Computing and NetKernel and I am learning very quickly about linked data.


It wasn't all roses the past year.

Resistance to change is big, even when the change is the right thing to do, even when you can prove that the change is the right thing to do. That brings frustration and I don't have the I don't care but I will take your money attitude that I see many other employees/consultants/freelancers have. Nor will I ever have it.

Planes are a shitty (if unavoidable) way to travel. Actually jetlag doesn't really bother me, but when I get off my ears are only one step away from bleeding and it takes a lot of paracetamol to clear the killer headache. My legs take about two days to un-cramp. Very weird, the general population gets wider and taller, the allotted space on planes goes the other way. I was so wasted on one of these trips that I had myself talked into taking a BMW X5 as a rental car rather than a standard model. Didn't make a lot of money on that trip (I paid for that expense myself, like I said, I do care).

Also, I underwrite the following (do not click this link if you are easily offended) manifesto. Although with Resource Oriented Computing that should be developing rather than programming. Process (in whatever form) is a means, not an end in itself. If you think (or act) otherwise, you are on a collision course with me. I don't stick up companies (even if they don't care) for more time than I need to deliver the goods.


All in all I agree with Peter Rodgers when he says that it has not been a bad year. Not at all. Although I was raised in an environment where a 10+/10 was considered barely enough and might get some remarks if the teacher had too obviously been a bit lax in the marking job. There is always a lot of room for improvement. But not a bad year, not bad at all.

For the next year I have the good intention to start blogging about my ROC-adventures on a weekly basis again and continue changing the world (which seems to have survived the Mayan equivalent of the year 2000) one bit at a time.

2012/11/30

in memory of pds

These are busy times, the end of the world is coming closer quickly. Maybe I should go out on the streets with a big sign stating Repent sinners before it is too late ! Be saved, use NetKernel !

Actually the Mayans didn't say anything about the end of the world, but it would definitely be one of the more interesting publicity campaigns.

Right, no more philosophy this week. In his Tic-Tac-Toe series Peter Rodgers bumps into the problem that the available in memory implementation of the PDS accessor doesn't quite implement all the functionality that a PDS accessor should have. He then quickly skips to using the H2 backed implementation.

While that of course works (the H2 implementation) I noticed in my Connect Four implementation that things are not as snappy as they should be for a game. Especially if the board gets bigger, the game becomes database bound.

So, here is my implementation of the in memory PDS accessor. It uses the golden thread pattern for expiry. Enjoy !

// The usual suspects for an accessor
import org.netkernel.layer0.meta.impl.SourcedArgumentMetaImpl;
import org.netkernel.layer0.nkf.*;
import org.netkernel.module.standard.endpoint.StandardAccessorImpl;

// Processing
import org.netkernel.layer0.representation.IHDSNode;
import org.netkernel.layer0.representation.IHDSNodeList;
import org.netkernel.layer0.representation.IReadableBinaryStreamRepresentation;
import org.netkernel.layer0.representation.impl.HDSBuilder;
import org.netkernel.request.IRequestResponseFields;

import java.util.Map;
import java.util.concurrent.ConcurrentHashMap;

public class InMemoryPDSAccessor extends StandardAccessorImpl {
    private static ConcurrentHashMap<String, InMemoryResource> mResources;
   
    private static class InMemoryResource {
        private final Object mRepresentation;
        private final IRequestResponseFields mUserMetaData;

        @SuppressWarnings("rawtypes")
        public InMemoryResource(INKFResponseReadOnly aResponse) {      
            if (aResponse != null) {  
                mRepresentation = aResponse.getRepresentation();
                mUserMetaData = aResponse.getHeaders();
            }
            else {  
                mRepresentation = null;
                mUserMetaData = null;
            }          
        }
      
        public Object getRepresentation() {  
            return mRepresentation;
        }
      
        public IRequestResponseFields getUserMetaData(){
            return mUserMetaData;
        }
    }

    public static class PDSArguments {
        public String mInstance;
        public String mZone;
        public String mIdentifier;
      
        private PDSArguments(INKFRequestContext aContext) throws NKFException {
            // Verify instance
            if (aContext.getThisRequest().argumentExists("instance")) {
                mInstance = aContext.getThisRequest().getArgumentValue("instance");
                if (mInstance.equals("pbv:instance")) {
                    mInstance = aContext.source("arg:instance",String.class);
                }
            }
            else {
                throw new NKFException("request does not have the required - instance - argument");
            }

            // Verify zone          
            if (aContext.getThisRequest().argumentExists("zone")) {
                mZone = aContext.getThisRequest().getArgumentValue("zone");
                if (mZone.equals("pbv:zone")) {
                    mZone = aContext.source("arg:zone",String.class);
                }
            }
            else {
                throw new NKFException("request does not have the required - zone - argument");
            }
            if (mZone.equals("")) {
                throw new NKFException("request does not have a valid - zone - argument");
            }          
          
            // Verify identifier
            if (aContext.getThisRequest().argumentExists("pds")) {
                mIdentifier = aContext.getThisRequest().getArgumentValue("pds");
                if (mIdentifier.equals("pbv:pds")) {
                    mIdentifier = aContext.source("arg:pds",String.class);
                }
                if (! mIdentifier.startsWith("/")) {
                    mIdentifier = "/" + mIdentifier;
                }
            }
            else {
                throw new NKFException("request does not have the required - identifier - argument");
            }
            if (mIdentifier.equals("")) {
                throw new NKFException("request does not have a valid - identifier - argument");
            }
        }

        private PDSArguments(String aInstance, String aZone, String aIdentifier) {
            mInstance = aInstance;
            mZone = aZone;
            mIdentifier = aIdentifier;
        }
      
        public Boolean isSet() {
            return mIdentifier.endsWith("/");
        }
      
        public String getCombined() {
            return mInstance + ":" + mZone + ":" + mIdentifier;
        }
      
        public String getGoldenThread() {
            return "gt:pds:" + mInstance + ":" + mZone + ":" + mIdentifier;
        }
      
        public String getIdentifier() {
            return mIdentifier;
        }

        public String getZone() {
            return mZone;
        }      

        public String getInstance() {
            return mInstance;
        }

        public boolean equals(Object aObject) {
            boolean vResult=false;
          
            if (aObject instanceof PDSArguments) {
                PDSArguments vOther = (PDSArguments) aObject;
                vResult = (mIdentifier.equals(vOther.mIdentifier) && mZone.equals(vOther.mZone) && mInstance.equals(vOther.mInstance));
            }
            return vResult;          
        }
      
        public int hashCode() {
            return mInstance.hashCode() ^ mZone.hashCode() ^ mIdentifier.hashCode();
        }
    }

    public InMemoryPDSAccessor() {
        this.declareThreadSafe();
        this.declareSourceRepresentation(IReadableBinaryStreamRepresentation.class);
        this.declareSourceRepresentation(IHDSNode.class);
        this.declareInhibitCheckForBadExpirationOnMutableResource();
        this.declareArgument(new SourcedArgumentMetaImpl("instance",null,null,new Class[] {String.class}));
        this.declareArgument(new SourcedArgumentMetaImpl("zone",null,null,new Class[] {String.class}));
        this.declareArgument(new SourcedArgumentMetaImpl("pds",null,null,new Class[] {String.class}));
        mResources = new ConcurrentHashMap<String, InMemoryResource>();
    }

    public void onSource(INKFRequestContext aContext) throws Exception {
        // SOURCE requires three arguments, instance zone and pds
      
        PDSArguments aArguments = new PDSArguments(aContext);
      
        if (aArguments.isSet()) {
            HDSBuilder vSet = new HDSBuilder();
            vSet.pushNode("set");

            for(Map.Entry<String, InMemoryResource> vEntry: mResources.entrySet()) {
                if (vEntry.getKey().startsWith(aArguments.getCombined())) {
                    int i = vEntry.getKey().indexOf(aArguments.getIdentifier());
                    String vPDS = vEntry.getKey().substring(i);
                    vSet.addNode("identifier", "pds:" + vPDS);          
                    vSet.pushNode("pds");
                    vSet.addNode("instance", aArguments.getInstance());
                    vSet.addNode("zone", aArguments.getZone());
                    vSet.addNode("pds", vPDS);
                    vSet.popNode();
                }
            }

            vSet.popNode();
          
            INKFRequest subrequest = aContext.createRequest("active:attachGoldenThread");
            subrequest.addArgument("id", aArguments.getGoldenThread());      
            aContext.issueRequest(subrequest);
          
            aContext.createResponseFrom(vSet.getRoot());
        }
        else {
            InMemoryResource vResource = mResources.get(aArguments.getCombined());
          
            if (vResource != null) {
                INKFRequest subrequest = aContext.createRequest("active:attachGoldenThread");
                subrequest.addArgument("id", aArguments.getGoldenThread());      
                aContext.issueRequest(subrequest);

                INKFResponse vResponse = aContext.createResponseFrom(vResource.getRepresentation());
                if (vResource.getUserMetaData() != null) {
                    vResponse.setHeaders(vResource.getUserMetaData());
                }
            }
            else {
                // default response is null
            }          
        }
    }
   
    public void onExists(INKFRequestContext aContext) throws Exception {
        // SOURCE requires three arguments, instance zone and pds
      
        PDSArguments aArguments = new PDSArguments(aContext);

        INKFResponse vResponse;

        if (aArguments.isSet()) {
            Boolean vResult = false;
            for(Map.Entry<String, InMemoryResource> vEntry: mResources.entrySet()) {
                if (vEntry.getKey().startsWith(aArguments.getCombined())) {
                    vResult = true;
                }
            }
            vResponse = aContext.createResponseFrom(vResult);
            if (vResult) {              
                INKFRequest subrequest = aContext.createRequest("active:attachGoldenThread");
                subrequest.addArgument("id", aArguments.getGoldenThread());      
                aContext.issueRequest(subrequest);
            }
            else {
                vResponse.setExpiry(INKFResponse.EXPIRY_ALWAYS);                          
            }
        }
        else {
            if (mResources.containsKey(aArguments.getCombined())) {
              
                INKFRequest subrequest = aContext.createRequest("active:attachGoldenThread");
                subrequest.addArgument("id", aArguments.getGoldenThread());      
                aContext.issueRequest(subrequest);
              
                vResponse = aContext.createResponseFrom(true);          
            }
            else {
                vResponse = aContext.createResponseFrom(false);
                vResponse.setExpiry(INKFResponse.EXPIRY_ALWAYS);          
            }          
        }

    }
   
    public synchronized void onSink(INKFRequestContext aContext) throws Exception {
        // Important : the onSink is synchronized ... only one at a time
        // SINK requires three arguments, instance, zone and pds and will persist
        // the primary argument
      
        PDSArguments aArguments = new PDSArguments(aContext);

        if (aArguments.isSet()) {
            throw new NKFException("unable to SINK to a set");
        }

        INKFRequest subrequest = aContext.createRequest("active:cutGoldenThread");
        subrequest.addArgument("id", aArguments.getGoldenThread());      
        aContext.issueRequest(subrequest);

        @SuppressWarnings("rawtypes")
        INKFResponseReadOnly vPrimary = aContext.getThisRequest().getPrimaryAsResponse();
        InMemoryResource vResource = new InMemoryResource(vPrimary);
        mResources.put(aArguments.getCombined(), vResource);
    }
   
    public void onDelete(INKFRequestContext aContext) throws Exception {
        // SOURCE requires three arguments, instance zone and pds
      
        PDSArguments aArguments = new PDSArguments(aContext);

        if (aArguments.isSet()) {
            IHDSNode vSet = null;
          
            INKFRequest subrequest = aContext.createRequest("active:pds");
            subrequest.addArgument("instance", aArguments.getInstance());
            subrequest.addArgument("zone", aArguments.getZone());
            subrequest.addArgument("pds", aArguments.getIdentifier());
            subrequest.setVerb(INKFRequestReadOnly.VERB_SOURCE);
            subrequest.setRepresentationClass(IHDSNode.class);
          
            vSet = (IHDSNode)aContext.issueRequest(subrequest);
            IHDSNodeList vNodes = vSet.getNodes("/set/pds");
            for (IHDSNode vNode : vNodes) {
                subrequest = aContext.createRequest("active:pds");
                subrequest.addArgument("instance", aArguments.getInstance());
                subrequest.addArgument("zone", aArguments.getZone());
                subrequest.addArgumentByValue("pds", vNode.getFirstValue("pds"));
                subrequest.setVerb(INKFRequestReadOnly.VERB_DELETE);
                subrequest.setRepresentationClass(Boolean.class);
                aContext.issueRequest(subrequest);
            }
            subrequest = aContext.createRequest("active:cutGoldenThread");
            subrequest.addArgument("id", aArguments.getGoldenThread());      
            aContext.issueRequest(subrequest);
          
            INKFResponse vResponse = aContext.createResponseFrom((vNodes != null));
            vResponse.setExpiry(INKFResponse.EXPIRY_ALWAYS);                      
        }
        else {
            INKFRequest subrequest = aContext.createRequest("active:cutGoldenThread");
            subrequest.addArgument("id", aArguments.getGoldenThread());      
            aContext.issueRequest(subrequest);
          
            InMemoryResource vResource = mResources.remove(aArguments.getCombined());
          
            INKFResponse vResponse = aContext.createResponseFrom((vResource != null));
            vResponse.setExpiry(INKFResponse.EXPIRY_ALWAYS);          
        }
    }
}

 

2012/11/01

from tic-tac-toe to connect four

The only way to find out if something is merely noise or actual information is sit down, read it and try what it says. Yourself. They named me Tom for a reason.

So last weekend I sat down, started reading [part1] of the Tic-Tac-Toe (TTT) example in the 1060 Research newsletters and set myself the goal of using it to create a Connect Four game.


There are a couple of differences :

The board is bigger. Typically Connect Four is being played on a 6 (rows) x 7 (columns) board, but variations exist and I wanted to have as general a solution as I could get. This resulted in two resources :
res:/connectfour/rows
res:/connectfour/columns
These can be backed by anything (a literal, a fileset, a database implementation), but the point is that they return the dimensions of the board.

There are more diagonals in play. While having a general solution for rows and columns, Peter rather quickly glosses over the diagonals in the TTT-solution, just defining the two that count (one of which is actually an antidiagonal).

I decided to have a more general solution where a diagonal goes from top left downwards. The diagonal:0 starts in the top left corner. Diagonals above that one get a positive index (diagonal:1 and so on, counting to the right), diagonals underneath that one get a negative index (diagonal:-1 and so on, counting down).

Antidiagonals go from top right downwards. The antidiagonal:0 starts in the the top right corner.  Antidiagonals above that one get a positive index (antidiagonal:1 and so on, counting to the left), antidiagonals underneath that one get a negative index (antidiagonal:-1 and so on, counting down).  

A token drops down the column
. As you know, a token (I went with X and O again) is not put in a specific place, but drops down the column into the lowest available position.


The victory condition is four of the same token next to each other in a row, column, diagonal or antidiagonal
. That after all is the name of the game ...



All that resulted in a bit more code. Since the dimensions of the board are unknown, I couldn't just map a row, column, diagonal and antidiagonal to a cell{} resource. Here for example is my groovy code for the row :


import org.netkernel.layer0.nkf.*;

INKFRequestContext aContext = (INKFRequestContext)context;

int vNumberOfRows = aContext.source("res:/connectfour/rows", Integer.class);
int vNumberOfColumns = aContext.source("res:/connectfour/columns", Integer.class);

int vRow = Integer.parseInt(aContext.getThisRequest().getArgumentValue("x"));

String vIdentifier = "cells{";


if ( (vRow >=0) && (vRow < vNumberOfRows)) {

  for (int vColumn = 0; vColumn < vNumberOfColumns; vColumn++) {
    vIdentifier = vIdentifier + "c:" + vRow + ":" + vColumn + ",";
  }
}
else {

  throw new NKFException("argument - x - should be in the range [0 - " + (vNumberOfRows - 1) + "]");
}

vIdentifier = vIdentifier + "}";

INKFRequest subrequest = aContext.createRequest(vIdentifier);
subrequest.setVerb(INKFRequestReadOnly.VERB_SOURCE);

aContext.createResponseFrom(aContext.issueRequestForResponse(subrequest));


In case you are wondering why I take context and put it into aContext ... that works a lot easier in my editor.

Obviously, the SINK to a cell also requires a bit of code :

case INKFRequestReadOnly.VERB_SINK:
  int i;
  for(i = vNumberOfRows - 1; i >= 0; i--) {
    if (! (aContext.exists("pds:/connectfour/cell/" + i + "-" + vColumn) ) ) {
      aContext.sink("pds:/connectfour/cell/" + i + "-" + vColumn, aContext.sourcePrimary(String.class));

      aContext.sink("pds:/connectfour/lastmove", "c:" + i + ":" + vColumn + ":" + aContext.sourcePrimary(String.class));

      break;
    }
  }
  if (i < 0) {
    NKFException e = new NKFException("invalid move");
    aContext.createResponseFrom(e);
  }
  break;

This also shows that I keep track of the last move made, a very convenient resource, since in order to check if there's a win, I need to know where the token fell.


Now, it is very easy to stand on the shoulders of a giant and shout out how great you are ... while ignoring the giant (which then proceeds to punch you in the face). However, given the above variations ... it was plain sailing all the way. The TTT story was great information, it delivers the goods !

For those of you not on Facebook, the amount of very (!) popular games that are mere variations (!) on this theme (with a bit of graphical polish) is huge. Just looking [here], I note Bubble Safari, Bubble Witch Saga, Diamond Dash, Candy Crush Saga, Bejeweled Blitz, Bubble Island. Look at the numbers. Look again, you missed some zeroes the first time.

Now, I'm not a graphical wizard (know thy strenghts and weaknesses), but when showing my first cut of the game to a friend, she wanted graphical icons instead of the textual X's and O's. I also added a multigame layer so it becomes a hotseat game that you can play with your loved ones. You can reach the (early) Christmas edition online here : http://netkernelbook.org/connectfour/<youridentifier>/

Obviously you need to replace <youridentifier> with some number or word or whatever that only you know.


A couple of quid pro quos to end this blogentry :

* A real multiplayer version is coming up soon. I'm thinking about using websockets and you can see that lastmove-resource is going to be very handy.
* I noticed as well as you - when you try it - that it is dreamily slow, not blindingly fast. This is due to the persistance mechanism being in a database (remember Peter switched to database persistance) rather than in memory. For a TTT that's sufficient, for larger boards that works too slow. I'm working on a fix for that. [Here] you can have a look at the Visualizer trace for a first move (with cleared cache). It does add up.
* Note that I only have a small host out there, it is a showcase host, not a massive player host. If you are interested in your own copy, contact me and I'll send you the full source.



2012/10/05

the matrix revisited - neo4j

A long time ago in a galaxy far far away ... hold on, that's the cue-line from a different movie. Anyway, some time ago I was an IDMS database administrator. For those who don't know it, search for CODASYL in wikipedia. Those who do know will agree that it was superior to a relational database (and older), but it required specific knowledge of the problem you wanted to solve and often the database would then perform badly for a different problem.

Relational databases - once machines were fast enough to alleviate the speed difference - were (and still are) easier to work with. You don't need a specialist team of database administrators/architects.


But relational databases were apparantly not safe against all competition (enter tune of The Empire strikes back here). At the moment there's once more a war going on between SQL and NOSQL. By the way, did you know that NOSQL stands for Not Only SQL ? I think that's important, because it means that side of the war at least knows that both sides are there to stay.


In my book I had an example using MongoDB. So when I was recently asked to use something else than a relational database in a project ... the choice was clear, was it not ? Of course it was. I went out, searched the web and came up with graph databases of which Neo4j is the top dog.


And since my own knowledge about NetKernel is more advanced than at the time of writing the MongoDB chapter, I set out to write a complete Neo4j integration.


You can find the main module here : https://dl.dropbox.com/u/65770556/urn.org.netkernel.neo4j.embedded-1.7.2.zip (note : you have to put the neo4j 1.7.2 libs in the lib subdirectory)

You can find the unittest module here : https://dl.dropbox.com/u/65770556/urn.test.org.netkernel.neo4j.embedded-1.7.2.zip

I was extremely pleased when I saw what Neo4j could do. This is the way Computer Associates should have taken IDMS, rather than pay lip service to relation databases and produce a flesh nor fowl hybrid that nobody wants. For yes, a Graph database is very much like a CODASYL database (given 20+ years of fast evolving). My old skills are back in demand !


The issue IDMS had remains in Neo4j though (in my opinion). It is a specialist database. You will need good database administrators/architects (preferably the same team) to make it work for your problems.

Right, that's it for this week.

Disclaimer :
  • I'm voicing my own personal opinion on Neo4j based on working with it in a limited environment for three weeks. 
  • The code I deliver is not production ready. It allows you to play with Neo4j in NetKernel and enters a couple of constraints to make things easier which are not part of Neo4j as such. The code is delivered "as is" and is not guaranteed to do anything. It contains no documentation yet (that'll change in a couple of days), check the unittests for use examples.

2012/09/21

short and sweet

Sometimes you want to do some horrible things to another human being. It is true, it happens to the best of us. While I love the series on tic-tac-toe in the 1060 newsletter there was literally smoke escaping through my nostrils (and I don't smoke) when I read this was possible :

<rootspace> 
    <!-- provides res:/etc/ConfigRDBMS.xml -->
    <literal type="xml" uri="res:/etc/ConfigRDBMS.xml">
      <config>
        <rdbms>
          <jdbcDriver>org.h2.Driver</jdbcDriver>
          <jdbcConnection>jdbc:h2:/db;CACHE_SIZE=1024</jdbcConnection>
          <poolSize>8</poolSize>
        </rdbms>
      </config>
    </literal>
</rootspace>

It was probably somewhere in a newsletter and I overlooked it. A new feature in one or another module that got mentioned in passing. It is so powerful it deserves another mention right here.

Until now, when I wanted to scaffold, I used filesets. They do the job, allow for some flexibility, blah blah and some more blah ...

Literals ROCk ! Especially in unittests for modules that require configuration to work they offer huge value for little work.

That is it for today, I'm quite sure the god-of-thunder will be having some interesting stuff to talk about in the 1060 newsletter so I'll leave the stage to him.

Before I forget ... happy anniversary 1060 Research !

2012/09/15

back to the ivory tower

Disclaimer : This is an opinion piece, not a technical piece. If you want a technical piece, read prior posts or wait a couple of days more, I'm working on some interesting stuff. In this piece I'm voicing a personal opinion only. If you don't like it (the opinion), kudos to you.

I have done my time in the Ivory Towers. Did I like it ? Of course I did ! Developers and architects would come to me - and needed an appointment - and I'd design and create their database or systems the way I liked it, often causing an overhaul of what they had in mind. I could speed up projects or kill them. My word was law and my code reviews were legendary. The little Latin proverbs (in a time before online access was granted) I had in the Friday entry of my company agenda (accessible to all), were coffee-corner talk minutes after they went up.

It was a time of people who knew and understood and people who didn't.

Around the turn of the century this changed. The Internet changed it. Everything became available for everybody and everybody could tinker with it. In the database world SQL-databases ruled supreme. Everybody could understand the mathematics behind those and machines became powerful enough so they became sufficiently performant. Virtualization opened up the system administration world. While dual booting always remained a tricky thing only a geek would do, booting a virtual machine is a no-brainer and the Internet is full of how-to guides.

Everybody could now do everything and this opened the Information Technology gates for a lot of people and small companies with great ideas. The towers toppled.

Recently I notice a pushback, did you notice it too ? SQL databases are no longer hot, no, we've got big data, semantics and map reduce, tripple stores and key-value-stores, ... and most importantly, you will go out of business if you don't have them too, yes, really, look Google and Facebook have them ... you must have them too !

So, following the hype a lot of people buy into all of it. And only afterwards find out they are biting of a lot more than they can chew. For :
  • Did somebody explain to you what Big Data is ? Not to mention Cloud Computing, Tripple Stores, semantics, ... And did you find two people that gave you the same explanation on all of those ? Really ? Can I have their email-addresses please ?
  • Did somebody explain to you what map reduce is ? You are going to use it to query your data ! Did you look it up ? Do you understand the mathematics ? Really ? Oh, you're a PhD Mathematics ... figures.
It would seem too me that we're moving back to the Ivory Towers, because what all these have in common is that the added value for the companies that create and hype them lies in the fact that they require consultancy to get right and in the end it will be the consultant that says what goes and what does not ... regardless of what the business question was !

I also want to add that a lot of this is far from new. Everybody that ever used LDAP knows that a key-value-store has been around forever. And while a graph database (the thing Facebook is build on) is super flexible and a hell to query in a performant way - can you hear Kerching! too ? - it's roots lie in the CODASYL findings of 1969 (remember IDMS anyone)... yes, the year a man put foot on the moon. Server- and database clusters with quorums ... been there and done that with HP systems and Oracle as early as 2001.

Personally I play with it all (watch this space for upcoming technical posts) and I see the benefits and the drawbacks of many a hype. For me they are all tools to get the job done, the job defines the tool, not the other way around. Remember that before you buy into one ! And the day the new Edgar Codd stands up and starts bringing down the towers of complexity once more ... lets give her/him a Noble Prize !


2012/08/30

minimalistic OXO

When you're up against a master it is very noble to let him/her pick both the battleground and the battle. You're going to get beaten though. And while you might learn some important lessons along the way (watch Karate Kid for full list) ... you're still going to get beaten.

Of course I could not refuse to pick up the gauntlet on the Tic-Tac-Toe (or noughts and crosses or oxo as we call it) challenge. But it was with a bit of despair. Sure, I might point out that it looks more like a matrix with elements than a set with cells and that there are in fact 5 diagonals and 5 antidiagonals (only two of those counting for a "Winning Set", that is true), but I might then get slapped around the head with my own weapons. I'm not a mathematician and I'm challenging a PhD in Quantum Physics.


And then I remembered something. I used to play a text MUD (try them, I haven't seen anything graphical that beats them for depth of involvement and gameplay ... once you get used to the interface). Inside the game you could play chess against other players. Yes, really. It was minimalistic. I mean, you had to know the shorthand codes for moving a piece and only newbies would occasionally request a visual (ascii art) view of the board (the rest of us did it all in the head). The only thing the game itself knew was what the valid moves were at a given state for the board.


Did the chess game know about the victory conditions ? Of course not. Only a computer needs to be told what the victory conditions are !


A smile crossed my face when I remembered that. For I can not beat Peter at his own game and I'll be devouring his upcoming newsletter(s) that explain his solution (and no doubt learn lots from them). But a minimalistic OXO in under two hours ? Yes, I can do that.


You find my solution here
  • It has no concept of rows, columns or diagonals other than is needed for the interface (which is REST, using most of the verbs to get the job done).
  • There is a little bit of code, the biggest part going to showing the board (like the ascii art for the chess game).
  • The game has no concept of a victory condition, the players can decide what that is.
  • It comes with documentation, search for "oxo" after you deploy the module. 
  • There is no computer player or any other AI (you can add those as a layer on top if you want them, it's easy to find the rules for a perfect game on wikipedia).
Let the games begin !

2012/08/29

second oreilly webcast

Yesterday I gave my second O'Reilly webcast this month. I used a much quieter laptop and it went a lot smoother (for me at least), but you'll understand that I'm quite happy it is now a thing of the past.

You can find the
nCode applications I demo'd here. I also noticed that the previous talk is now up on Youtube, so you should be able to find that one there. This one will follow in about a week. The slideshow itself can be viewed on http://netkernelbook.org/wink/wiki/webcast_20120828.

This officially ends my holiday. I'll now be moving on to some serious stuff such as ... tic-tac-toe.

2012/08/17

fine grained control

September and the return of the normal flow of things are just around the corner. For this holiday post I'm going to share another snippet of code that I've recently been working on. It gives you fine grained control over NetKernel's kernel properties :

// Author: Tom Geudens
// Date  : 2012/08/15

// The usual suspects for an accessor.
import org.netkernel.layer0.nkf.*;
import org.netkernel.layer0.meta.impl.SourcedArgumentMetaImpl;
import org.netkernel.module.standard.endpoint.StandardAccessorImpl;

// Processing.
import java.util.Properties;
import org.netkernel.container.IKernel;
import org.netkernel.layer0.boot.*;
import java.io.ByteArrayOutputStream;

import org.netkernel.layer0.representation.IBinaryStreamRepresentation;

// The accessor itself.
public class KernelPropertyAccessor extends StandardAccessorImpl {

    public KernelPropertyAccessor() {
        this.declareThreadSafe();
        this.declareArgument(new SourcedArgumentMetaImpl("key",null,null,new Class[] {String.class}));
        this.declareArgument(new SourcedArgumentMetaImpl("value",null,null,new Class[] {String.class}));
        this.declareSourceRepresentation(String.class);
        this.declareInhibitCheckForBadExpirationOnMutableResource(); // golden thread takes care of it
        this.declareSupportedVerbs(
                INKFRequestReadOnly.VERB_NEW|
                INKFRequestReadOnly.VERB_DELETE|
                INKFRequestReadOnly.VERB_EXISTS|
                INKFRequestReadOnly.VERB_SOURCE|
                INKFRequestReadOnly.VERB_SINK);
    }
   
    public void onSource(INKFRequestContext aContext) throws Exception {
        // SOURCE requires one argument, key, which is mandatory for all
        // verbs and checked by the grammar.
        String aKey = aContext.getThisRequest().getArgumentValue("key");
        
        // Check validity of arguments.
        if ( aKey.equals("") ) {
            throw new NKFException("request does not contain a valid key argument");
        }

        // getting the properties data
        Properties vKernelProperties = new Properties();
        IKernel aKernel = aContext.getKernelContext().getKernel();
        vKernelProperties.load(BootPersistence.sourceBootResource("kernel.properties",aKernel).getInputStream());

        if (! vKernelProperties.containsKey(aKey)) {
            throw new NKFException("property - " + aKey + " - does not exist in kernel.properties");
        }
        
        INKFRequest vGTrequest = aContext.createRequest("active:attachGoldenThread");
        vGTrequest.addArgument("id", "gt:kp:" + aKey);
        aContext.issueRequest(vGTrequest);

        // response (string)
        aContext.createResponseFrom(vKernelProperties.getProperty(aKey));
    }

    public void onExists(INKFRequestContext aContext) throws Exception {
        // EXISTS requires one argument, key, which is mandatory for all
        // verbs and checked by the grammar.
        String aKey = aContext.getThisRequest().getArgumentValue("key");

        // Check validity of arguments.
        if ( aKey.equals("") ) {
            throw new NKFException("request does not contain a valid key argument");
        }
        
        // getting the properties data
        Properties vKernelProperties = new Properties();
        IKernel aKernel = aContext.getKernelContext().getKernel();
        vKernelProperties.load(BootPersistence.sourceBootResource("kernel.properties",aKernel).getInputStream());

        INKFRequest vGTrequest = aContext.createRequest("active:attachGoldenThread");
        vGTrequest.addArgument("id", "gt:kp:" + aKey);
        aContext.issueRequest(vGTrequest);

        // response (boolean)
        aContext.createResponseFrom(vKernelProperties.containsKey(aKey));
    }
   
    public void onNew(INKFRequestContext aContext) throws Exception {
        // NEW requires two arguments, key, which is mandatory for all
        // verbs and value, which is only mandatory for NEW and SINK.
        String aKey = aContext.getThisRequest().getArgumentValue("key");
        String aValue = null;
        if (aContext.getThisRequest().argumentExists("value")) {
            aValue = aContext.getThisRequest().getArgumentValue("value");
        }

        // Check validity of arguments.
        if ( aKey.equals("") ) {
            throw new NKFException("request does not contain a valid - key - argument");
        }
        if ( aValue == null) {
            throw new NKFException("request does not contain a valid - value - argument");
        }

        // getting the properties data
        Properties vKernelProperties = new Properties();
        IKernel aKernel = aContext.getKernelContext().getKernel();
        vKernelProperties.load(BootPersistence.sourceBootResource("kernel.properties",aKernel).getInputStream());

        if (vKernelProperties.containsKey(aKey)) {
            throw new NKFException("property - " + aKey + " - already exists in kernel.properties");
        }
        
        // adding new key/value
        ByteArrayOutputStream vBAOS = new ByteArrayOutputStream();
        vKernelProperties.setProperty(aKey,aValue);
        vKernelProperties.store(vBAOS, null);
        
        IBinaryStreamRepresentation vBSR =
                aContext.transrept(vBAOS.toString().toString(), IBinaryStreamRepresentation.class);
        
        BootPersistence.sinkBootResource("kernel.properties",vBSR,aKernel);
        
        // response, the newly created property
        aContext.createResponseFrom(aKey + "=" + aValue);
    }

    public void onSink(INKFRequestContext aContext) throws Exception {
        // SINK requires two arguments, key, which is mandatory for all
        // verbs and value, which is only mandatory for NEW and SINK.
        String aKey = aContext.getThisRequest().getArgumentValue("key");
        String aValue = null;
        if (aContext.getThisRequest().argumentExists("value")) {
            aValue = aContext.getThisRequest().getArgumentValue("value");
        }

        // Check validity of arguments.
        if ( aKey.equals("") ) {
            throw new NKFException("request does not contain a valid - key - argument");
        }
        if ( aValue == null) {
            throw new NKFException("request does not contain a valid - value - argument");
        }

        // getting the properties data
        Properties vKernelProperties = new Properties();
        IKernel aKernel = aContext.getKernelContext().getKernel();
        vKernelProperties.load(BootPersistence.sourceBootResource("kernel.properties",aKernel).getInputStream());

        if (! vKernelProperties.containsKey(aKey)) {
            throw new NKFException("property - " + aKey + " - does not exist in kernel.properties");
        }
        
        // replacing the value of a key
        ByteArrayOutputStream vBAOS = new ByteArrayOutputStream();
        vKernelProperties.setProperty(aKey,aValue);
        vKernelProperties.store(vBAOS, null);
        
        IBinaryStreamRepresentation vBSR =
                aContext.transrept(vBAOS.toString().toString(), IBinaryStreamRepresentation.class);
        
        BootPersistence.sinkBootResource("kernel.properties",vBSR,aKernel);

        // SINK has no response but we do need to cut the golden thread
        INKFRequest vGTrequest = aContext.createRequest("active:cutGoldenThread");
        vGTrequest.addArgument("id", "gt:kp:" + aKey);
        aContext.issueRequest(vGTrequest);
    }

    public void onDelete(INKFRequestContext aContext) throws Exception {
        // DELETE requires one argument, key, which is mandatory for all
        // verbs and checked by the grammar.
        String aKey = aContext.getThisRequest().getArgumentValue("key");

        // Check validity of arguments.
        if ( aKey.equals("") ) {
            throw new NKFException("request does not contain a valid - key - argument");
        }

        // getting the properties data
        Properties vKernelProperties = new Properties();
        IKernel aKernel = aContext.getKernelContext().getKernel();
        vKernelProperties.load(BootPersistence.sourceBootResource("kernel.properties",aKernel).getInputStream());

        if (! vKernelProperties.containsKey(aKey)) {
            throw new NKFException("property - " + aKey + " - does not exist in kernel.properties");
        }
        
        // removing the key
        ByteArrayOutputStream vBAOS = new ByteArrayOutputStream();
        vKernelProperties.remove(aKey);
        vKernelProperties.store(vBAOS, null);
        
        IBinaryStreamRepresentation vBSR =
                aContext.transrept(vBAOS.toString(), IBinaryStreamRepresentation.class);
        
        BootPersistence.sinkBootResource("kernel.properties",vBSR,aKernel);

        INKFRequest vGTrequest = aContext.createRequest("active:cutGoldenThread");
        vGTrequest.addArgument("id", "gt:kp:" + aKey);
        aContext.issueRequest(vGTrequest);

        // response (boolean)
        INKFRequest vKPrequest = aContext.createRequest("active:kernelproperty");
        vKPrequest.addArgument("key", aKey);
        vKPrequest.setVerb(INKFRequestReadOnly.VERB_EXISTS);
        aContext.createResponseFrom(!(Boolean)(aContext.issueRequest(vKPrequest)));
    }
}
 

My code doesn't often come with warnings, but this time I have to state it is delivered "as is" and that changing kernel properties can potentially mess up your system (for which I take no responsibility).

I can also see some possible improvements on the above code, so by all means try it out and change it as you like.

Enjoy !