Capturing and killing EiTest!!!

The goal of this post is just to document what I did to solve my issue, in case I need to do it again, and as a plus it might help others struggling with the same.

Recently one of my email servers started to get rejections from other servers, and as usual the place to go is mxtoolbox.com to check if this was blacklisted, and indeed it was by CBLAbuse, this organization checks, among others, for botnet activities including a trap to check servers sending massive connections to different things, which usually means a botnet is installed under the category of EiTest (this has been around for a couple of years and keeps spreading, there're some news this week that an organization found a way to stop it.

Ok, based on the documentation I found, most of the time the infected machine is inside the network and the instructions are mainly to detect the machine and then run and antivirus, as my server is a web server this point is completely useless, I knew the connections were coming from my server, but based on the documentation there should be a botnet running that fires up some things, and then hides again, so I'd to check for options to capture the network traffic and analize where this connections were coming from, fortunately CBL gives a sinkhole (a server used to trap this activity) so I can check if there are connections coming from my server that goes to this address, so first step check the network activity.

tcpdump -w archive.log

This command allows to log all the activity on my linux to a file, if I want to leave it running for a couple of hours it's just a matter of doing:

nohup tcpdump -w archive.log 2>/dev/null &

but dont forget to kill it after some minutes/hours otherwise it will eat up the disk!

To analyze the contents of the file you will need to download it to your machine and open it up with Wireshark.

This confirmed what I already knew, that my server was producing a lot of HTTP calls to the IP Address 192.42.119.41 (sinkhole), next step is to figure out which application is the one causing this, so after some digging I've to rely on the trusty netstat, you will find a lot of different solutions out there but after loosing too much time trying to check how to make them work I decided to use the tools I already have at hand, so I run the following command for a couple of hours:

watch "netstat -atpun|grep 192.4"

unfortunately the connections flashed too quick and most of the time they were as "TIME_WAIT" (already closed) so I've created the following script (created is a big word here, I've just copied and paste from stackoverflow as 99% of the things we do nowadays):

while true
do
        netstat -atpun|grep 192.4 | tee -a log.txt
        sleep 1
done

This small script allowed me to trace for a couple of hours, and finally this showed that the application executing this was apache server (no news here, a lot of suggestions were pointing to wordpress been infected), now the aweful part of the history, I'd to take down one by one each server to find which of the 4 domains I've was causing this, once I'd narrowed it down to 1, I'd deactivated allow the plugins and voila! the log.txt stopped showing the ip... at the end the plugin infected was "documentor", so I've removed this and reinstalled new from the repos, and this fixed the 4 days of suffering...

Hope this helps to others, if not I'm sure this will help me in the future to avoid wasting time reading a lot of generic information about the EiTest.

Bizagi logging: The Right Way!

In many projects one of the repetitive questions is how to produce effective logs in our processes, and how to use them in production, unfortunately the answer is: "you can use logs in development, but you need to switch them off in production or the performance will be hurt".

There are several logs in the market that we can use to include in our rules, so I implemented a wrapper of the Log4Net that you can found here: https://github.com/crossleyjuan/bizagilogger.

The good thing about this is that you can use logs in production environments without having to worry (too much) about the impact in performance, it's just a matter of configuring this right and get the best based on configuration. The bad, this is something you need to configure in the config files and not from the Bizagi Management Console, therefore turning off the traces from Bizagi console will not have any effect.

How to use the BizagiLogger

To use Log4net in your projects you will need 3 elements:

• The Component Library: BizagiLogger
• Configure the .config files
• Create a Library Rule to wrap all the logs (Recommended)

Component Library: Bizagi Logger

Feel free to download the required dlls from here: https://github.com/crossleyjuan/bizagilogger/releases/tag/1.0 or download the source code and compile it yourself. This dll depends on log4net.dll, a great Apache Project that makes this possible.

Register both dlls as Component Libraries in Bizagi

Using the logger in your rules

Syntax:

 var log = Logger.getLogger(emitter);  
 log.Info(ctx, text);  
 log.Error(ctx, text);  
 log.Warn(ctx, text);  
 log.Debug(ctx, text);  

• The emitter is a very useful filter, as you can collapse the same messages related to the same emitter, my suggestion is to use the Case Number
• ctx is a tag variable called NDC in Log4Net, this helps to search for an additional parameter. for example you can use the name of the process

Example

 var sEmitter = Me.Case.CaseNumber;  
 var log = Logger.getLogger(sEmitter);  
 log.Info("Process A", sLogText);  

Configuration

To configure the way the logger works you can use the appenders as explained in the log4net documentation, to make this easier here's an example that could be useful in a production environment:

• Add the configuration section:

  To be able to use the log4net options you need to let .net known that you will add some configuration elements for log4net, to do this you need to add this to your config files in the "configSections" section:

    <section name="log4net" type="log4net.Config.Log4NetConfigurationSectionHandler, log4net" />  
  

  full section:

    <configSections>  
     <section name="bizAgiComponentLibrary" type="BizAgi.ComponentLibrary.CComponentLibrarySectionHandler,BizAgi.ComponentLibrary"/>  
     <section name="bizAgiWFESTouchPoints" type="BizAgi.WFES.TP.CTouchPointsSectionHandler,BizAgi.WFES"/>  
         <section name="log4net" type="log4net.Config.Log4NetConfigurationSectionHandler, log4net" />  
     <!-- Uncomment for federate authentication -->  
     <!--<section name="microsoft.identityModel" type="Microsoft.IdentityModel.Configuration.MicrosoftIdentityModelSection, Microsoft.IdentityModel, Version=3.5.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35" />-->  
    </configSections>  
  

Now you can add the appenders in any part of the <configuration> tag, you can use the following as starting point

  <log4net>  
           <appender name="BufferingForwarder" type="log4net.Appender.BufferingForwardingAppender">  
                <bufferSize value="512" />  
                <lossy value="false" />  
                <Fix value="268" />  
                <appender-ref ref="RollingFile" />  
                <!-- or any additional appenders or other forwarders -->  
           </appender>  
           <appender name="RollingFile" type="log4net.Appender.RollingFileAppender">  
                <file type="log4net.Util.PatternString" value="c:\temp\Logs\WebApp-%date{yyyy-MM-dd}.log" />  
                <appendToFile value="false"/>  
                <rollingStyle value="Composite"/>  
                <maxSizeRollBackups value="-1"/>  
                <maximumFileSize value="2MB"/>  
                <staticLogFileName value="true"/>  
                <datePattern value="yyyy-MM-dd"/>  
                <preserveLogFileNameExtension value="true"/>  
                <countDirection value="1"/>  
                <layout type="log4net.Layout.PatternLayout">  
                     <conversionPattern value="%date [%thread] %-5level %logger [%property{NDC}] - %message%newline" />  
                </layout>  
           </appender>   

       <root>  
            <appender-ref ref="BufferingForwarder" />  
       </root>  
  </log4net>  

These appenders have a very good performance so you can use as many traces as you want in production with almost no impact in your performance, the reason is that the BufferingForwarder keeps chunks of information in memory and writes them in an asynchronous way to the RollingFile appender, this stores the file creating a new one every day or of the file reaches the maximum defined in the "maximumFileSize> tag.

More information of the available appenders here: https://logging.apache.org/log4net/log4net-1.2.13/release/sdk/log4net.Appender.html

do you have questions? comments? use the comment below if you need additional information

Open Source vs Closed Source

When I started djondb some years ago I'd a different view on what an open source project means, therefore I want to share my short experience to help others to understand the hidden trade offs and the work that requires these kind of decisions. I just want to share with all of you my disappointing experience with the Open Source, my limited view on this and why I'm rolling back the decision I made 4 years ago to open the source code of djondb to the world.

With my limited experience in the open source world I must say that I was impress on how the open source projects created communities from nowhere, people willing to help, either sharing ideas or code, everybody working on a common goal, getting a nice product that solves the problem at hand. This is exactly what I want, share my passion to solve problems, the idea was good, everyone that I spoke to shared the same view, that NoSQL looked like a great idea, but the lack of transactions was something that is hard to trade, doing this on my own could be near to impossible, with my limited time and resources, also after so many years of using open source tools it was reasonable to jump in a put the code open source, so everyone will benefit from this new idea and I will bring balance to the world.

When I started the code I decided to keep it close until it was more mature, then someone pointed that as a bad idea:

KuntharAugust 22, 2012 at 9:21 AM

No source code on Github? Only drivers?
Good move. Bye bye djondb.

So I thought, yes, Kunthar is right, I'm planning to open source the code anyway, so why not now? and started the preparation, prepare makefiles to make easier the compilation by others, prepare scripts, documentation, etc... that took several weeks, even more than some of the code components I've made for the project, but the idea worth it.

The summary of this journey as open source project was firstly pointed here: http://crosstantine.blogspot.com.es/2013/03/balance-after-5-months-of-releasing.html in 2013, exactly 2 years ago, even that I was disappoint with this already, several people pointed the failures of my endeavor, their arguments:

• The documentation needs improvement
• You need to promote your database more
• Reporting issues is not easy, this could be improved.

And therefore I decided to stick with the idea and improve this things and check what happen, in 2014 I decided to focus on the code again, as I have been working too hard in two sides, coding and promoting, having more success in the coding side than in the promotion and engagement I decided to focus on what I know more, code. Making a stronger database, fulfilling the promises it will attract other developers and users to support the open source project.

The cruel reality is that users are too busy to report issues, the only one that will be a good tester is yourself, developers are too busy to get engage unless they see a money reward on this, (I got several emails saying they wanted to be partners on this, but at the end none of them had the experience or the time to really spend on this).

After 4 years of having this project in github I've produced 911 commits, the project is ranked among the 30% projects in github with more commits, even that I'm just 1 developer, from that... only 2 issues were reported and no one ever sent a pull request or patches,

So what I got from this experience? not too much, just a new experience and a bunch of biased conclusions:

• If you don't have a stable product don't go to open source
• People is willing to get involved if the product has created a name on its own.
• Use your product yourself, don't wait for others to test it.
• Don't waste time creating cross-platform compilation tools, etc. if the product got traction in one platform others will be willing to help with these things to compile it in their own platform.
• Don't listen to others unless they have a proven history of success in the field you are trying to get into.

Some of these may sound very obvious to others, but they were not for me, and I'd to learn it in the hard way.

One of the few wise decisions I made was to hold the copyright of the code, otherwise I will be screwed, and after 4 years of work, with no help at all, I will need to forget this product, I'm glad that I read that from MySQL and others.

What now?

djondb removed the support from github, and will change it's license to closed source, as per GPL states the versions that are around still will be free as in GPL, although the copyright of the name it's still mine, the GPL allows others to use it as an open source product, so everything below the version 0.3.3 that's is out there will continue under open source, and newer versions will be closed source.

Is djondb still free to use?

Yes, djondb is free to use and you can put it in as many servers as you want, commercial license will be provided if you require it and the enterprise support will return. In exchange you will require to register to download, not too much to ask for a free database don't you think?

Bizagi - Load Balancing - Where to specify the URL used in notifications to point to the load balancer?

Bizagi sends notifications to users when they are assigned to a particular task, this is known as Automatic notifications, these notifications include the following default message:

Here is a brief information about the case 1251 (Id Case: 1251):<br>Creation Date: 10/13/2014<br>Category: Processes<br>Process Definition: Vacation Request<br><br>Click here <a href=3Dhttp://dev-mymachine/BizagiApp/default.aspx?widget=3Dactivityform&idCase=3D1251>1251</a> to view the case online.

As default Bizagi will use the machine name where it was generated as part of the link, if you want to point these links to the load balancer address you will need to configure the PROTOCOL, SERVER_NAME and APP_NAME based on your configuration, to do this you will need to add the following elements to your WebApplication/web.config file:

    <add key="SERVER_NAME" value="LoadBalancerHostName" />
    <add key="APP_NAME" value="BizagiApplication" />

This will produce the following output:

Here is a brief information about the case 1251 (Id Case: 1251):<br>Creation Date: 10/13/2014<br>Category: Processes<br>Process Definition: Vacation Request<br><br>Click here <a href=3Dhttp://LoadBalancerHostName/BizagiApplication/default.aspx?widget=3Dactivityform&idCase=3D1251>1251</a> to view the case online.

Which will point users to the load balancer instead of the cluster machine

Bizagi Sub Processes Parent / Child

Sub processes is one of the best reusable elements that you can use to support your business, they allow you to encapsulate logic and share it with other processes, but it's also one of the elements that creates coupling, the concept is fully explained here: "Loose Coupling". So how do you design sub processes that are truly re-usable and easy to maintain?

Sub-processes as a functions

The typical approach to use sub processes creates dependencies between the caller and the sub process, therefore any change to the parent may impact the child and same applies for the changes in the child, that's called highly coupled design. To work around this and create cleaner processes you should consider your sub processes as "functions" that will be called and return a result. Let me clarify this with a simple "hello world!" code. Let's imagine that you want to create a sub process to standarise the "greeting" messages, therefore you want to solve this problem:

function parent() {
    recover_name;

    child_create_greeting(); 

    print greeting message;
}

Current approach most of the times:

var name;
var greeting;
function parent() {
    name = recover name;

    child_create_greeting();

    print greeting message;
}

function child_create_greeting() {
    greeting = "Hello " + name;
}

I know this is a silly sample, but bear with me for a moment, and check the next approach:

function parent() {
    var name = recover name;

    var greeting = child_create_greeting(name);

    print greeting message;
}

function child_create_greeting(name) {
    greeting = "Hello " + name;
    return greeting;
}

The result is the same, but now your child and parent may evolve without impacting each other every time you do a change. Same concept could be applied in Bizagi processes, in the sample above I was using the variables name and greeting to share information with the parent, but what if I create a "message" envelop, that both processes share and act as communication between both? something like this:

message {
    var name;
    var greeting;
}
function parent() {
    message.name = recover name;

    child_create_greeting(message);

    print message.greeting;
}

function child_create_greeting(message) {
    message.greeting = "Hello " + message.name;
}

This creates an element where both processes will agreed as sharing mechanism and therefore they will be able to evolve without any impact, as long as the "message" element follows the original agreement. This translated to Bizagi concepts is just an entity that will hold the information that sub-process requires and where it will store the result of it's work. The responsibility of the caller process is to create the message instance, assign the parameters and then use the result stored there. Does it make sense? let's see it in action:

A real problem

The "Hello world!" sample is good way to explain basic concepts, but let's face it, it's worthless if you don't see something that it's applicable to real life problems, so here's a typical problem: "Our company requires an standard way to authorize different requests across our organization, therefore we need a reusable sub process that will be plugged in processes like, but not limited to: Travel Expenses and Vacations".

Model your sub processes as reusable process

The first step is understanding that your approval process will have different contexts and therefore it should be modeled with that independence in mind, the trick here is to be able to understand that your sub process is not approving Expenses or Vacations, it's approving a Requests (Value of the expense) or actions (Taking vacations), and therefore you will model your sub process to approve "requests" or "actions".

The Process and sub process definition

Vacation request:

Approval:

Data Models

Vacaction Request Model

Approval Sub Process Model

As you may noticed the M_Message entity is used in both processes, I added the suffix _I or _O for Input and Output to make it easier to understand. Now it's time to fill in the variables and passing them from the caller to the sub process, let's jump to code.

Passing "parameters" to the sub process

As explained above the M_Message entity will be used as communication element between the caller and the sub process, and therefore any element require by the Approval Sub process should be placed there, that includes simple data, like "Value to approve", "Date requested" and Entities (like Employee, Request Detail, etc), here I'm using "request type" to enable the sub process to handle different kinds of messages.

var reqType = CHelper.getEntityAttrib("P_RequestType","idP_RequestType","RT_Code = 'REQ'");
<VacationRequest.entApproveRequest.entRequestType_I> = reqType;
var requestMessage = String.Format("Requesting vacactions starting from {0} for {1} days.", <VacationRequest.dStartDate>, <VacationRequest.iDays>);
<VacationRequest.entApproveRequest.entRequest_I.RequestDescription> = requestMessage;

This step is just populating the "message" that we are using to pass the values, now we will set the entity to the attribute that will be referenced in the sub process, <Sub_Approval.entMessage>

Congratulations, you've just created a re-usable sub process that is truly re-usable and it wont break the parent caller every time you create a change on it, and the most important element, you can reuse it in any process without changing the sub process to adapt new callers. Love to hear your comments!

Parsing command line arguments in a batch program (Windows)

Everytime I'd to create a batch to do compilations, or any other kind of stuff I always bumped to the same problem, parsing command line arguments. Usually I ended up with something like:

if "%1"=="-x32" (
   Do something for x32
)
if "%2"=="-x32" (
   Do something for x32
)
if "%1"=="-x64" (
   Do something for x64
)
if "%2"=="-x64" (
   Do something for x64
)

This is really bad way to do it, so I decided to spend sometime today and solve the problem once and for all, as a result I finally have a decent way to parse arguments, and I will document it in here in case I need it again or if someone else is struggling to get this for himself. (I love getopts from Linux! but I guess Windows will keep forcing us to this kind of solutions)

 @echo off  
 setlocal enabledelayedexpansion  
 if [%1] ==[] goto usage  
 call:parseArguments %*  
 if "%x32%" == "true" (  
   echo Well done you set x32 to true  
 )  
 if "%x64%" == "true" (  
   echo Well done you set x64 to true  
 )  
 if NOT "%d" == "" (  
   echo you set the output dir to: %d%  
 )  
 GOTO Exit  


 @rem ================================================================================  
 @rem Functions  
 @rem ================================================================================  
 :usage  
 Echo Usage: %0 [-x32] [-x64] [-d output-dir]  
 goto exit  

 :getArg  
 set valname=%~1  
 echo arg: !%valname%!  
 goto:eof  

 :parseArguments  
 rem ----------------------------------------------------------------------------------  
 @echo off  
 :loop  
 IF "%~1"=="" GOTO cont  
 set argname=%~1  
 set argname=%argname:~1,100%  
 set value=%~2  
 @rem if the next value starts with - then it's a new parameter  
 if "%value:~0,1%" == "-" (  
   set !argname!=true  
   SHIFT & GOTO loop  
 )  
 if "%value%" == "" (  
   set !argname!=true  
   SHIFT & GOTO loop  
 )  
 set !argname!=%~2  
 @rem jumps first and second parameter  
 SHIFT & SHIFT & GOTO loop  

 :cont  
 goto:eof  

 rem ----------------------------------------------------------------------------------  
 :Exit  

The magic ocurrs in the "parseArguments" function, (Oh yes, batch files have functions, I didn't know that until now. This page was really helpful: http://www.dostips.com/DtTutoFunctions.php).

This function contains two things that I learned today. The first one: SHIFT, this command shift the arguments putting the second argument in front and the third in second place, and so on. This is really useful if you don't know the number of arguments, so you only need to do something like the following code to print all the arguments:

 :loop  
 IF "%~1"=="" GOTO cont  
 echo %~1
 SHIFT & GOTO loop  
 :cont  

The second think I learned is how to create dynamic variables, thanks to this guy: http://batcheero.blogspot.com/2007/07/dynamic-variable-name.html, this is useful to create the variables that you're going to use later in your batch program. This is possible because I used: setlocal enabledelayedexpansion at the very beginning of the batch program, otherwise the !thing! won't work.

So the parse arguments function just iterate over the arguments guessing if they were written as: -x32 or -d c:\temp, (in the first case %x32% will be set to true and the former %d% will contain c:\temp.

Here're the results:

> test.bat -d "c:\test dir\blah"
you set the output dir to: c:\test dir\blah
> test.bat -d "c:\test dir\blah" -x32 -x64
Well done you set x32 to true
Well done you set x64 to true
you set the output dir to: c:\test dir\blah

That's it, enjoy

Writing tests that works

Disclaimer

Although some of the ideas exposed in here are too obvious to some developers, I wanted to write down some of my thoughts about how the "unit" tests should be implemented in some projects, this is a discussion I'd with a fellow co-worker after reading this article: Unit Testing Myths and Practices and I think worth the time to post it in here, maybe some of you will agree with me, and some could call me dumb, anyway, I would love to read your thoughts about this and discuss about them on the comments section.

My opinion about "Unit" Tests

Several definitions of Unit tests are around Internet and in books, but here's a simple demo of what I understand by unit tests, and the way I saw unit tests implemented in several projects.

int myMethod(int arg1, int arg2) {
   return arg1 + arg2;
}

Unit test:

void testMyMethod() {

    // Testing positive numbers
    int a = 1;
    int b = 2;
    int expected = 3;
    int result = myMethod(a, b);
    ASSERT(expected, result);

    // Testing negative numbers
    a = -1;
    b = 2;
    expected = 1;
    result = myMethod(a, b);
    ASSERT(expected, result);

    // testing zero arguments
    ...

    // testing big integer values
    ...
}

As you may noticed I required 10+ lines of code to test 1 line of the business method. Now, what will happen to our test cases if we want to test the following method:

int myMethod2(Customer customer, int arg2) {
   int age = customer.age();
   return myMethod(age, arg2);
}

Now I will need to write tons of lines to check how my code will react to every single combination of customers' age or any value to be added. For example: test cases to check NULL customer, customers without age, summing up a negative age, etc, etc... Is this what I wanted in the first place? will this ensure that, if a reckless developer changes myMethod to get sum Integers instead of int, avoid null errors in the myMethod2? or do we need to rewrite all the tests that depends on the first method? do I even care about nulls been sent to myMethod?

Dependency injection will save the day!

Maybe some of you thought, "hey this guy does not know a sh**t about coding and he lacks of expertise, this is easily managed by dependency injection and method contracts". Maybe you are right, but lets check how our methods would change to ensure the usage of dependency injection to separate both concepts (bear in mind that the tests are now going to insert a dummy class that will return the proper values for each test).

[di]
IMySumClass sumClass;

int myMethod2(Customer customer, int arg2) {
   int age = customer.age();
   return sumClass.myMethod(age, arg2);
}

Done!, now we have a new problem, both tests are going to pass up the tests phase, and the error will popup in our production system. Yes, I know... the methods' contract... someone would say "the method says that it will receive X and Y and if the X and Y changes then it's a new method because it's a new contract. Ok, here we go:

int myMethod(int arg1, int arg2) {
   return arg1 + arg2;
}

Integer myMethodWithIntegers(Integer arg1, Integer arg2) {
   return arg1 + arg2;
}

Solved!, now we have 2 methods that executes the same logic, therefore we will need to mantain them as well. (But don't forget that our tests cases will need to be maintained as well, so we have now 30+ lines to keep updated). Too much work to ensure that a simple sum will work, don't you think so?

Integrity tests

Ok, let's go back to the simple code we have in the first place:

int myMethod(int arg1, int arg2) {
   return arg1 + arg2;
}

int myMethod2(Customer customer, int arg2) {
   int age = customer.age();
   return myMethod(age, arg2);
}

Unit test:

void testMyMethod() {

    // Testing positive numbers
    int a = 1;
    int b = 2;
    int expected = 3;
    int result = myMethod(a, b);
    ASSERT(expected, result);

    // Testing negative numbers
    a = -1;
    b = 2;
    expected = 1;
    result = myMethod(a, b);
    ASSERT(expected, result);

    // testing zero arguments
    ...

    // testing big integer values
    ...
}

What I found useful is to keep the tests as simple as possible and targeting the class we want to check, so instead of doing a double check of all the different options for a "a + b" operation in the method testMyMethod2, or in the testMyMethod, I would add the tests that are useful to each case, and I will remove the negative checks, zero arguments, etc that will cumbersome my code, also I won't add tests to testMyMethod2 that are not related with myMethod2 itself, therefore I will not add tests that were tested somewhere else. Let's see the sample:

void testMyMethod() {

    // Testing positive numbers
    int a = 1;
    int b = 2;
    int expected = 3;
    int result = myMethod(a, b);
    ASSERT(expected, result);
}

void testMyMethod2() {
    // Customer with age
    Customer c = new Customer("x", "y", 32);
    int expected = 33;
    int result = myMethod2(c, 1);
    ASSERT(expected, result);

    // Test a customer too old (should be rejected because of his age)
    c = new Customer("x", "y", 32);
    try {
       myMethod2(c, 150);
       FAIL("Customer age was not properly checked");
    } catch (AnyCheckedException) {
        // Accepted case
    }
}

Noticed how I removed the "negative", zero and big numbers tests, I really don't care about these cases at this level, and they will be tested by "customer" cases anyway. These kind of tests are going to be more useful and they will protect my code from unexpected "real" failures reducing the amount of work I need to write down all the test cases and covering the business cases instead of arguments misusage.

Off course, this a very simple case, and it would not cover all the "what if" questions, but it will cover what is supported by the system. The last sample test code will check not only cases related to myMethod2, but myMethod as well, this means that myMethod will be checked for every single call made from any of the classes that uses this method, and it's going to be tested in a "business" way, not in a "what if" applied to every single argument.

What usually happens in production environments is that errors, not originally tested, will popup and we will need to add them to the proper test code to ensure they will not rise again, if the error was in the "customer" level then we will add proper lines to the testMyMethod2 (or create a new one), if the problem was with our sum method not covering Big Integers, then we will need to cover it in the customer test as well, any change to the testMyMethod2 will also test myMethod in a business like scenario.

Conclusion

Every project could be different and every one will need a solution that fits to it, but I've been using this heavily and this method of "accumulative" testing have worked really well to check and correct bugs in production systems.

I would love to read your thought, please post a comment. The "you're a dumba**" comments are welcome but please try to support your ideas.

Switching from autotools to something else

Although I'm really happy with the results of autotools on Linux and OSX, everytime I want to generate the installers on Windows I needed to do something, at the beginning I just created the sln and vcproj for each of the modules in djondb, later I'd to implement an script to do the compilation process even simpler, then I'd some problems with the libraries, third party libraries, x64, etc and I'd to change some settings on the project files, so far everything worked as expected (not smoothy but worked), lately I've been having some problems that only happened to crash on Windows environments, and here's where the nightmare have begun, I've all the tests written with cpptest (which is a really good option and I love to write tests on it), but now that I need to execute the same tests on windows, and my approach started to flood.

I've been reading about some other build tools, that are really cross-platform, and I will give a try to CMake, I tried scons, but looks really slow, I don't know why but compiling with it takes double time than same thing on Linux/make counterpart.

I'll do some tests on CMake, and if everything works I will switch djondb to this build system, I'll post my results in here later.

Any thoughts? please let me know what you thing about CMake, SCons and make tools.

Why NoSQL matters?

Web applications have been evolving on a daily basis, mainly because web is something so visible that will be shamed if your website is not at the same level of your competitors.

For years the languages and frameworks to build websites changed to fill the demand of new agile development methodologies, servers like nodejs appeared to deal with changes, jquery appeared to allow the creation of richer content, etc. you'll notice the evolution started from front-end technologies and it's moving towards the backend, now it's the time for the ultimate frontier to adjust to this change.

Evolution of the database model

One of the things that remains constant over the years is the need to map your information to storage applications like MySQL, Oracle database, Postgres or SQL server. One thing they have in common it's all of them are based on a rigid model, the model of tables, rows and columns, and these lead to several problems, these problems you need to address during this always changing web environment.

• Managing the instructions at code level to work with a new set of data or new columns at the database level.
• Addressing deployment problems, if you create a new field in a table you'll need to take down your server and do a painful process of upgrading, (run scripts to change your tables, execute an script to migrate, or at least update, your new model, and finally updating the code that changed)

What if these changes means that your model changed in a more radical way? What if you created a new type of product that requires a full new model? That's something to thing about.

This kind of problems have been a constant over the years until now, some software developers, as myself, started to wonder if there will be a better way to store the information that is more aligned with the current challenges, that's how the NoSql movement started, as a way to solve this new data model paradigm.

The new data model

To give a better explanation about how NoSQL address the problem of these always changing data models, let's work using an example as a basis. Lets assume we're going to store customers from our newly created website, to do this job we decided to create a table in an RDBMS, and we included columns like name, last name and genre. After some days we put together an application and finally it's running on web, then tweet about giving some benefits to the people that register during the weekend, after a good weekend you end up with 3M records, you're surprised because you didn't expected that kind of traffic, then you check the db and realized that you didn't provide enough space for some long last names and some of them are truncated, darn you'll need to expand the column.... But, you can not change the size of a column in SQL server, you need to create a new table, move the records and that will mean your server will be out for 1 hour. Does this sound familiar? Even worst, you need to include some new fields in the process, meaning changing the HTML, your server side, doing some migration, etc.

Although JSON does not solve all the problems that come with these kind of changes it's really easy to update the underlining model just modifying a JSON structure than changing the tables, columns, rows and SQL statements, and that's how NoSQL helps in the development of web application, allowing to make this kind of changes easily, storing JSON instead of traditional data model.

A Customer could be model in a JSON like this:

    {
         name: "John",
         lastName: "Smith",
         genre: "Male"
    }

Same problem stated before, what if you need to store a longer last name? Because the model is not rigid, your users will never had the problem in the first place, and their last names were not truncated. What if you need to store new information?

    {
         name: "John",
         lastName: "Smith",
         genre: "Male"
    }

    {
         name: "Peter",
         lastName: "Korn",
         genre: "Male", 
         email: "test@mycompany.com"
    }

Done!, you don't need to change anything at the database level. If your REST services are treated as general REST services that get JSON and post them to the DB, you will not need to change your backend either, and the only change you will need is at the HTML and javascript level, which will allow you to do a cleaner deployment of the new version, just post the new HTML and javascripts at the web sever folder and that's it, you are ready to go.

Conclusions

Although a good requirements/design/test will always be better than having to get into a production environment to realize that you forgot a piece of data that is really important to your business, the NoSQL databases allows you to upgrade your applications easier, and relieves the pain of creating an application from ground up, it really empowers the developer to explore the data structure while he is creating his master piece.

Later I will keep exploring samples on how to use NoSQL to create applications in a faster/easier way, in the mean time please go to here: http://djondb.com/blog and check out some of the full examples that you will find in there.

Balance after 5 months of releasing djondb as Open Source

After 5 months of releasing a big product as an open source project I want to write about the progress, the goods, the bads, and share the future releases of djondb.

Briefing

I've been working on djondb for a year, it started as a proof of concept, a learning project and a challenge, rapidly it grew to a running version of a server and I started to wonder if there was space for a new NoSQL database, and here's the video of the speech I gave at BogoDev in 2012

Although it was a living thing it was far from been a production server. After this speech I realized that there was a lot of people with the same requirements, a NoSQL database that can store JSON, ensure everything goes to disk and not into "lose-able" memory, and been able to handle transactions over multiple documents. Last, but not least, been able to run it in Linux, OSX and Windows.

Several tasks were required to be completed once the MVP was a real application:

• Create an stable full minimal featured release (Insert, Updates, Deletes, Find with meaningful queries, etc)
• Implement drivers for most common languages.
• Document all the features
• Create samples
• Installers for each platform
• Drink couple of beers! (this was started even before the first line of code)

The first version of all the features was released a couple of months later, and djondb was open sourced in github since September 2012.

I almost remember the face of my wife once I told her that I will put the code as open source, "Is he crazy?, he is doing this great product and he will give it away?", but I was completely sure that was the best choice, here is the list of the reasons to go to this path:

• Give something to the world. I'm using open source tools so seems fair enough to give something back.
• An open source is easier to promote, mainly because Open source promotes itself, the advocates of open source push this kind of developments to keep them going.
• Some developers might join the development team and we will share the glory, and the money that could come with this solution.
• Been open source the community will help with feedback, samples, document translation, etc.

In November I did the first "official" speech about djondb in Colombia 3.0, one of the biggest Content Summit of Latin America, here's the video:

After this I've been working on drivers, documentation, new features, bug fixing, etc. And released the version 0.2, which had several performance improvements.

During the latest months I've been working on new features that are going to be released at the end of March.

Balance

djondb brough a lot of learning, not only about coding databases, or high performance systems, but on how the Open source ecosystem works, what means to be an Entrepreneur (I'm still learning, and I think I will never stop learning), what traction is, what an investor sees, etc.

djondb has been a very successful project, more than 350 downloads during the last three months and a lot of great reviews. Some good projects are already using it, and people really share the same though of NoSQL as a profesional tool to develop business applications.

Not everything is nice

Although I've been working every night to improve and release a better product, it's very disappointing to realize that Open source is not what I thought it will be, it's not a great community trying to help out, sharing ideas or helping with spreading the word, it's more like a bunch of guys (like me), working really hard to share a dream with bare hands and passion. Most of the histories I read everyday about lone riders that try to create open source is just like mine, working hard and getting few or none feedback, just a lot of downloads.

Lessons I learnt from the open source, maybe I'm wrong and it's only my perception, but this is what I think:

• Most people think open source means free, they don't even think that feedback or test is a way to share and help.
• Investors put weird face when you say "Open source", unless you already show that there's a lot of traction. And I mean huge traction, not the kind you would need if you "own" the code.
• Other developers look at the open source as a way to get their projects done without paying anything. (there're several ways to give back in the open source, like feedback, helping documentation, maybe fixing a bug or two, etc)

Summary

Don't go with the open source option if you don't have a company that supports you. Open source will mean that you put all your ideas on the web and almost anyone can copy you and create their own product without any responsibility, I know... you could argue that you can show you posted the code first, etc etc. (If you argue that you could patent your code let me remind you that procedure costs U$35.000 and takes 5 years to be approved). The truth is that you can not protect your idea as open source, you gave away your knowledge and you won't get anything back.

Preparación para taller de NoSQL con djondb

Este post lo realizo con el fin de explicar los pasos de instalación y preparación requeridos para el taller que se dictará en el Bogodev el próximo 31 de Enero /2013.

Herramientas necesarias

El ejemplo que se realizará usará 3 componentes:

• Ubuntu 11 o superior.
• djondb: Motor de base de datos NoSQL (http://djondb.com)
• KnockoutJS: Framework para patron Model-View-View Model MVVM (http://knockoutjs.com/)
• nodejs: Servidor multiproposito capaz de ejecutar javascript server-side (http://nodejs.org
• Cualquier editor de texto que facilite la edición de HTML y javascript (vim, Netbeans, emacs, etc)

Instalación

El primer paso será la instalación de la base de datos, para ello vamos al site de djondb y obtenemos la versión para linux correspondiente a nuestra arquitectura.

sudo dpkg -i djondb_Linux_i386.deb

Probablemente obtendremos unos errores indicando que hacen falta algunas dependencias, como antlr, etc. para instalar las dependencias faltantes se realiza:

sudo apt-get -f install

Procedemos a crear la carpeta donde se almacenaran los archivos de la base de datos y le daremos permisos al usuario actual, asi:

sudo mkdir /var/djondb
sudo chown `id -u` /var/djondb/

Ahora podremos verificar si el servidor quedó instalado correctamente ejecutando:

djondbd -n

El parametro -n permite que el servidor no ejecute en background permitiendo bajarlo de manera simple usando Ctrl+C.

Una vez instalado, realizamos la instalación del nodejs de la siguiente forma:

Entrar a la pagina http://nodejs.org y dar click en install, esto bajará un archivo: node-v0.8.18.tar.gz y seguir los siguientes pasos:

tar xvfz node-v0.8.18.tar.gz
cd node-v0.8.18
./configure
make
sudo make install

Luego instalamos algunas herramientas requeridas para instalar plugins de nodejs:

sudo apt-get install g++ make
sudo npm install -g node-gyp

La primera instrucción instalará las herramientas necesarias para ejecutar el servidor e instalar plugins. La segunda instalará el plugin node-gyp que permite compilar extensiones que usen este procedimiento de instalación.

Con estos pasos podemos proceder a crear la carpeta donde realizaremos el taller, e instalaremos el driver de djondb necesario para nodejs, asi:

mkdir proyecto_nosql
cd proyecto_nosql
npm install djondb

Estos son los pasos requeridos para la instalación de las herramientas necesarias para el taller, si tiene algun problema realizando la instalación coloquenos un comentario y con gusto les colaboro para corregir el paso que este generando inconvenientes.

Nos vemos el 31 de Enero en el bogodev.

Update 27 de Enero:

Ya que en un punto del taller usaremos una llamada al server por REST, conviene tener instalado el Chrome con la extensión "REST Console" o firefox con una extensión similar.

NoSQL or RDBMS, not again!

No matter what, I found the same question over and over again, "should I use a NoSQL solution to my <what ever problem>, help me!", but some days ago I crossed a question that I want to discuss. To avoid any problem with the author of the question I will try to rephrase it here:

"Our company XYZ created a solution ABC that is working really well on our production environment, at this moment we address thousands a records a month and 1 terabytes of information is created each month, our current architecture is <monster server here> and we're happy with our current performance, "...

If you're developer you will know at this point that the Universal developer rule should be applied here: "If it's not broken, please don't fix it".

He continues... "Our application follows the model of write once read many, because we do business intelligence to scrap out our data and get information from there, most of our queries uses 2 or 3 joins, and we don't have weird data types, just plain numbers, texts and dates, our system will get more data and more users and we're worried that our current platform will not support this new workload, that's why we want to do the move to a NoSQL system, which one do you recommend for our problem?"

Here's where the things are getting nasty, and I think I need to clarify that I'm a developer too, and I'm in love with NoSQL strategy, that's why I created djondb because I'm convinced that "One size does not fit all" applies to the RDBMS world, 4 years ago the only way to store information in an organized way was RDBMS and several people think that they should apply RDBMS to every single problem, and that's not true. But this same principle applies to the NoSQL world, take a look of Maslow's hammer law.

NoSQL is a really good solution to several data environments, but you need to ask yourself if you're aware that most of your code should change if you decide to move from one solution to other, it is like saying that you're going to recreate your application in Java because it has better performance according to <put your link here>, you need to know that this kind of changes will require a lot of work to achieve the same features you already have, so you need to evaluate if this is going to work as a benefit or you will have a lot of work to achieve the same thing.

To illustrate the SQL - NoSQL transition I will use a sample, the architecture of the sample could be very different according to the approach you decided on your application (EJBs, Spring, LINQ, etc) and that will change the way you coded your solution, but I will go with a very straightforward and well know solution.

Data Model comparison

Most of you are already aware of the differences between one model and the other, but here's a very simple sample to illustrate this.

In a ERP data model you will have the following tables:

Taken from http://www.databaseanswers.org

This is a very simple model, a customer order a product and you register the identification of the customer and product into the table "Orders", each order is related with one or multiple transactions, etc.

From this simple model it's very easy to assume some of the SQL statements you should use to store or retrieve elements from your DB, let's summarize some simple operations:

• Create products:

  Insert into Products (product_code, product_category_cod, description, vendor_code) VALUES ('00010101', 'E_01', 'iPad', 'Tablet blah blah', 'VEN_0010')
  

• List of products by category:

  Select product_code, description from Products where product_category_cod = 'E_01'
  

• Show the transactions ordered by a user:
  To retrieve this you will need to execute some SQLs or you will get a bunch or repeated information, first you will need to get the order "header":

  Select o.order_id, c.Customer_name, c.Customer_last_name, c.Customer_username, o.order_date, ... from Orders o inner join Customer c on c.customer_id = o.customer_id where c.username = 'JACKTHEREAPER'
  

  Now you will retrieve the details of the order using:

  Select p.description, o.price, o.units, ... from Orders o inner join Products p on p.product_code = o.product_code where o.order_id = 'XR1210120'
  

At this point I think you got the idea, no matter if you use some ORM, persistence framework, etc at the end your application will execute these SQLs to retrieve or store information into the database.

Let's see the same data model from NoSQL perspective, one of the key decision process about using NoSQL is to denormalize your data in a way that store it and retrieve it will be align with how the application uses it, instead of forcing a row/columns structure.

Questions to be solved by your NoSQL model

What will be benefit of storing the order detail as a separate entity? is the order detail an entity? or just an aggregation values that compounds an order? An order does not have any meaning without the details and they are useless if are not tight to an order. This will mean that the order should have a structure similar to this:

{
   { Order header info }
   { Order details }
}

Now lets figure out what are we going to store at the header info, 90% of the queries you will do to retrieve the order header will include the name of the customer, and his id, it's going to be pointless to show the order in the web application without this vital information, it's going to be easier to ignore information than adding new required information, so I will store the denormalized data from the customer inside the order, like this:

{
   { 
      order_id: "XR1210120",
      customer_id: "CI_29120",
      customer_name: "Jack",
      customer_lastname: "Reaper",
      date: "2012/07/26"
   }
   { Order details }
}

Note: "Hey!, what are you doing? what is going to happen if the customer changes his name?", how many times you face this kind of problem? anyway, if the customer changed his name and he comes back in 10 years, the invoice should be printed as it was at the time of the sale, because that's the way it's stored at the IRS, or any other state department, right?

What about the order details? same approach, we could save the product name along with the order and you will have everything you need at the time of printing/showing the order to the user. What about the product price? same thing, you will need to save the price at the time of the purchase, so it will be better to keep it within the order and not in an inventory table.

Here's an order with full details:

{
    "order_header": {
        "order_id": "XR1210120",
        "customer_id": "CI_29120",
        "customer_name": "Jack",
        "customer_lastname": "Reaper",
        "date": "2012/07/26"
    },
    "Orderdetails": [
        {
            "product_code": "00010101",
            "Category_code": "E_01",
            "description": "iPad",
            "price": 450,
            "units": 1
        },
        {
            "product_code": "00010203",
            "Category_code": "E_01",
            "description": "Xbox 360",
            "price": 320,
            "units": 1
        }
    ]
}

Let's do the same operations over djondb, using the djon-shell:

• Create products:

  insert('sampledb', 'products', { product_code: '00010101', category_code: 'E_01', name: 'iPad', description:'Tablet blah blah', vendor_code: 'VEN_0010'});
  

• List of products by category:

  find('sampledb', 'products', '$"category_code" == "E_01"');
  

• Show the transactions ordered by a user:
  

  Now, you'll only need one round trip to the database to retrieve all the information you will need to show the order:

  find('sampledb', 'orders', '$"user" == "JACKTHEREAPER"');
  

  done!, you will get something like:

  [{
      "order_header": {
          "order_id": "XR1210120",
          "customer_id": "CI_29120",
          "customer_name": "Jack",
          "customer_lastname": "Reaper",
          "date": "2012/07/26"
      },
      "Orderdetails": [
          {
              "product_code": "00010101",
              "Category_code": "E_01",
              "description": "iPad",
              "price": 450,
              "units": 1
          },
          {
              "product_code": "00010203",
              "Category_code": "E_01",
              "description": "Xbox 360",
              "price": 320,
              "units": 1
          }
      }
  },
  {
      "order_header": {
          "order_id": "NP112021",
          "customer_id": "CI_29120",
          "customer_name": "Jack",
          "customer_lastname": "Reaper",
          "date": "2012/07/28"
      },
  ....
  ]

  This result is a JSON, very simple to iterate and interact to, but the idea of this article is not to convince you to use NoSQL, but to help you understanding the difference between your RDBMS application approach and the NoSQL approach, and highlight how these approaches will lead you to change some key points in your architecture.

Now that you got the idea of how the storage is mapped from one side to the other, it's time to take a look of your code.

Code comparison

Until now, the things had been very straightforward, and it's very likely that you have some sentences executing SQLs similar to the ones I showed, but the code is something else, it's hard to know how every architect created their designs, but I will use a commonly known design pattern, known as "DAO" (Data Access Object) to illustrate my point.

The DAO pattern uses a class to retrieve the information from the database and create a Transfer Object that is going to be returned to the business layer, if you want to store information of an order it will work like this:

Note: I will do the assumption that you used jquery in your front-end and it will be able to create a full order in one step (header and dome details) and then bulk it to the server to be stored.

Note: My second assumption is that you have some json framework at java that will be able to handle json nicely.

Who?	What to do?
Web Controller	Catches the data from the "request" object and puts it in a Value Object (POJO, POCO, etc)
Model	Receives the Value Object and perform validations, etc
Model	Calls a DA class to store the VO
DA Class	Creates the required SQL to store the values from the VO

The code (using java) will look something like this:

Web Controller: Catches the data from the "request" object and puts it in a Value Object (POJO, POCO, etc)

Order o = new Order();
JSONObject json = JSONParser.parse(Request.getParameter("order"));
o.setCode(json.getString("order_code"));
o.setCustomerId(json.getString("customer_id"));
o. ... (all the order header information)
for (JSONObject detail : json.getJSONArray("Orderdetails")) {
     OrderDetail d = new OrderDetail();
     d.setProductCode(detail.getString("product_code"));
     d.setUnits(detail.getInt("units"));
     ...
     o.addDetail(d);
}

Now, you will get this and put it on a Model Class (following the MVC pattern)

Model: Receives the Value Object and perform validations, etc

    Model m = new Model();
    m.setOrder(o);
    m.saveOrder();

Model: Calls a DA class to store the VO

    MyDAClass da = new MyDAClass();
    da.insert(o);

DA Class: Creates the required SQL to store the values from the VO

    StringBuffer sql = new StringBuffer();
    sql.append("INSERT INTO Order (Order_code, customer_id....) ");
    sql.append(" VALUES ('" + o.getOrderCode() + "', ...");

    Statement stmOrder = connection.createStatement();
    stmOrder.execute(sql.toString());

    // at this point you will need to do the same for each order detail:
    for (OrderDetail detail : o.getOrderDetails()) {
       StringBuffer sqlDetails = new StringBuffer();
       sql.append("INSERT INTO OrderDetail (Order_detail_code, Order_code, product_code....) ");
       sql.append(" VALUES ('" + detail.getOrderDetailCode() + "', ...");
    }       

I know, you could use a preparedStatement here and use parameters, etc... or you could use EJB or ORM to store the VO, but here's not where you will see the difference between NoSQL and RDBMS approaches.

Now, let's take a look of the same code using djondb as NoSQL repository, I will keep the same MVC and DAO concept to keep the code side by side comparable.

Same code NoSQL approach

Web Controller: Catches the data from the "request" object and puts it in a Value Object (POJO, POCO, etc) at this point you will not need to map your data to a VO, it's pointless, you could use JSON at server side and it will work really well.

JSONObject order = JSONParser.parse(Request.getParameter("order"));

Now, you will get this and put it on a Model Class (following the MVC pattern)

Model: Receives the Value Object and perform validations, etc, this does not have any change:

    Model m = new Model();
    m.setOrder(order);
    m.saveOrder();

Model: Calls a DA class to store the VO, same thing the model does not suffer major changes.

    MyDAClass da = new MyDAClass();
    da.insert(o);

DA Class: Creates the required SQL to store the values from the VO

    connectionWrapper.insert("ERP", "Orders", order);

Just one line will do the trick, same with the controller, mainly you are avoiding the overhead of doing unnecessary transformations, you will get the json and store it directly to your database, no more additional processing will be needed here.

The same coding sample could be used to "listing orders" or "showing an order", you will see that you coded a lot less using JSON as transport and persistance mechanism

Conclusion

Although I tried to put it very simple, converting an application from RDBMS to NoSQL is not a trivial task, you will need to spend time creating a new design of your application, yes... you will get a lot from this, you will be able to create new features easily, you will be able to grasp the performance of NoSQL in your application, but you should not think that just putting a NoSQL behind will do the magic and boost your performance, you actually need to start thinking on "NoSQL" data model and new ways to code your application to get most of the NoSQL world.

djondb - Progress of my own NoSQL DB

The last two months has been really good, my own NoSQL Server is going like a skyrocket, and I'm close to finish the Milestone 2, here're the milestones I created 3 months ago:

Milestone 1 version 0.1

Basic features

• Allow inserts
• Allow updates
• Allow finds by key
• Shell
• PHP Driver
• C++ Driver

Milestone 2 version 0.1

Complete basic features

• Finds by any filter
• Arbitrary Index creation
• Backup / Restore
• Java Driver
• C Driver
• .NET Driver

Milestone 3 version 0.1

Sharding

• Transactions
• Sharding

Milestone 4 version 0.1

Nice to have

• Authentication
• Clustering

At this moment the db is fully functional and I'm doing a demo, which is a craiglist like site to demostrate why NoSQL is a nice technology for fast development.

What's next?

At January 26 I will be at @hubbog doing a speech showing an overview of what is NoSQL, what is it good for (and what is not), and some demos about how a NoSQL (primary djondb) improves the development cycle, and let finish your projects in no time

For the next months I will complete the Milestones 3 and 4, in the meantime I hope to get some help to develop drivers, sites around djondb, etc.

Going open source?

This is a great question, and I dont know the answer yet, I always wanted to share and give something to the world, but I'm not that big to avoid problems with someone else copying my code and using it for their own benefit, that's why I will wait until I finished all the milestones and upload the first version to the web to make this decision.

Building a protocol

Introduction

When I started to rewrite the save file algorithm for djon time tracker, I searched what will be the best way to do this, and I found a lot of people saying XML is the way to go, but everytime I tried this the files got very large and the time to read and write was huge, that's why I started to look at the binary formats.

The binary formats are simple files where you store bytes instead of chars, straighforward definition, but what is that? how do I know that a string started at some point, or I have an integer, when I read it the only think I see is HEX values? Here's where protocols are useful.

from wikipedia: "A communications protocol is a system of digital message formats and rules for exchanging those messages in or between computing systems and in telecommunications. A protocol may have a formal description."

When you write your own protocol you should define an unique way to write and read every single piece of data, then you have to follow the rules to read and write based on the protocol you defined.

Let's build our own protocol.

Let's say that you're going to save or transmit over a wire the data of a customer:

Customer

• Name
• Last Name
• Birth Date
• Salary

First, we need to define the type of each data:

Customer

• Name: chars
• Last Name: chars
• Birth Date: date
• Salary: integer

Now we define a unique set of rules to write a Customer:

Data Order: the data will follow the same order everytime (Name, Last Name, Birth Date, Salary)
Labels? if we have a fix set of data (like the example above) it's useless to name each piece of data, so we will avoid this and save space

Now we need to define how to save each type, let's start with the easy one.

Integer

In C an integer is a 2 bytes length data, that means you will have a 2 chars to store. Ex: 65000 as a salary will be FDE8 (2 chars, FD and E8 which could be translated to char:253 and 232).

Let's write some code here, and save an integer in the simplest way:

void saveInt(int a) {
    FILE* f = fopen("test.dat", "wb");
    fwrite(&a, 1, sizeof(a), f);
    fclose(f);
}

This code works well and it's very straightforward, but it has a big problem. It will write the 2 bytes (from the example above: FD and E8) in an unknown order, could be E8FD or FDE8 depending on the architecture of the machine it runs, this means that if the architecture of the machine where you're going to read the file changes you will get a very different result. That is called Little/Big Endian problem. To fix this we will ensure that the order will be the same all the time, this is done using the following code:

void writeInt(int a) {
    FILE* f = fopen("test.dat", "wb");
    unsigned char c = (a & 255);
    fwrite(&c, 1, 1, f);
    unsigned char c2= ((a >> 8) & 255);
    fwrite(&c2, 1, 1, f);
 
    fclose(f);
}

This code will ensure that the order will be same everytime, and it will not depend on the architecture of the machine it runs. Let's break down this instructions:

    unsigned char c = (a & 255);

If you have an integer of 65000 (FDF8) it will do an "and" operation with 00FF, this will "erase" the higher byte:

    FDF8
And 00FF
    ====
    00F8

The next instruction will do a similar operation, it will move the bytes from right to left and erase the higher part:

unsigned char c2= ((a >> 8) & 255)

FDF8 >> 8   = XXFD
XXFD & 00FF = 00FD

With this simple method (called Little Endian) we will ensure that the write will be always in the same order, now the read will be easy:

int readInt() {
    FILE* f = fopen("test.dat", "rb");
    unsigned char c;
    fread(&c, 1, 1, f);
    unsigned char c2;
    fread(&c2, 1, 1, f);
    
    int res = c & (c2 << 8);

    fclose(f);

    return res;
}

c2 will be FD and c will contain F8 doing the "<< 8" the FD will go up and adding will result in FDF8 (the original number)

Now that we solved the big issue, the other things are easier.

Strings

One of the main issues with strings is how to deal with the length of the string, one possible solution could be to put a fixed char at the end of the string and read until reach that character.

Strings solution 1

void writeString(char* c, int len) {
   FILE* f = fopen("test.dat" "wb");
   for (int x = 0; x < len; x++)
      fwrite(&c[x], 1, 1, f);

   char end = '*';
   fwrite(&end, 1, 1, f);
   fclose(f);
}

char* readString() {
   FILE *f = fopen("test.dat", "rb");
   char c;
   char buffer[256];
   int pos = 0;
   do {
      fread(&c, 1, 1, f);
      if (c != '*') {
          buffer[pos] = c;
          pos++;
      }
   } while (c != '*');
   buffer[pos] = '\0'; // terminated-string
   fclose(f);
   return buffer;
}

This solution works pretty well, and it can be improved using stringstreams or others, but it has a big problem, what if the original string contain the character '*' in between? change it for other char? what will be the odds that character is included too? This could be easily fixed if you write down the size of the string and then the content of the string, and you read it in the same way, first the length and then the contents.

Strings solution 2

void writeString(char* c, int len) {
   writeInt(len);
   FILE* f = fopen("test.dat" "wb");
   for (int x = 0; x < len; x++)
      fwrite(&c[x], 1, 1, f);

   char end = '*';
   fwrite(&end, 1, 1, f);
   fclose(f);
}

char* readString() {
   int len = readInt();
   FILE *f = fopen("test.dat", "rb");
   char c;
   char* result = (char*)malloc(len+1);
   for (int x = 0; x < len; x++) {
      fread(&c, 1, 1, f);
      if (c != '*') {
          result[x] = c;
      }
   };
   result[len] = '\0'; // terminated-string
   fclose(f);
   return result;
}

Solved! (off course you could change the methods to open the file, do all the operations and then close it, these were written this way to avoid complexity)

Now, the main code:

write(Customer c) {
    writeString(c.name());
    writeString(c.lastName());
    writeDate(c.birthDate()); // I will let this to the reader
    writeInt(c.salary());
}

Customer read() {
    Customer c;
    c.setName(readString());
    c.setLastName(readString());
    c.setBirthDate(readDate());
    c.setSalary(readInt());
}

This solution could be applied to network transmission, files, or anything you want. You could translate this solution to other languages.

Apache error 403: Forbidden

I'd run into an annoying problem that is very easy to solve, once you know the answer... as usual.

I was starting a simple project to demostrate how to create an application using djondb as a NoSQL db and wrote some simple pages emulating the famous craigslist page, but as soon as I added the "<Directory>" directive to the apache server started to get the message:

Forbidden

You don't have permission to access /demo2/temp.html on this server.

Run to google... do some searches and all of the results pointed to file permission problems, I just went to the console and run the chmod a+rwx (I know.. it's not secure, but it's a demo pc), restarted apache and... puff.. the error kept popping up, read more... did some changes... and nothing the error persisted, then I crossed to a post that was really helpful Fixing 403 Forbidden on alias directory with Apache one of the answers suggested to login using the apache user, and try to navigate to the file.

As soon as I did that I realized that my folder was: /home/cross/workspace/db/demo... etc... and I changed the permissions to the "demo" directory, but not to all the parent folders (workspace/db) and that was causing the problem. I added my private group (cross) to the user www-data (the user used to start apache) and now everything is working.

usermod -a -G cross www-data

Easy? yes... I know it is but I want to share this "enlightened knowledge" in case you crossed with the same problem. Took 1 hour of my time to solve this, wondering why on earth the test application worked fine with the PC at my office (ubuntu 10.10) and didn't at my home PC with ubuntu 11.10. (Actually I still wonder why I didnt run into this problem at my office... the path is the same)

d-jon is going to the open source world!

After 1 year of hard work I decided to open the code to the world, why? easy because I always wanted to give something to the community, I used several tools based on open source and I think it's good to give something back.

How d-jon started?

2 years ago I was working on the java version of Bizagi (http://www.bizagi.com) and the team started to use TSP (Team Software Process) in order to develop the new rendering module, one of the key features of TSP is the ability to measure the time spend on every single task you do on your job, that's how I started to use several linux tools to track my time and all of them were unsuccessful, most of the tools because they're unable to track if I'm working on my desk or leave the office soon, if I forgot to stop the timer when I decide to take my lunch then I had to get back and pinpoint at what time I leave the office, same happens when my boss (or a coworker) ask me a question about some other subject, when I come back to my desk I did not know how much time I spend assisting somebody else, one of the tools lost all my records without any warning, etc etc... I got tired of this and decided to create a simple tool to track my time.

Some months later I realized that my tool was getting stronger, and stronger, so I decided it worth the effort to put d-jon on the web, that's how I did the first refactor of all the code to create a less buggy application, at first I was the only user so I can deal with some bugs, but now that the application is going to be on web I have to fix them.

Vision Software, the company I have been working for 7 years, decided that this tool was good enough to replace the Allnetic solution that was buggy and has several issues. 1 year later d-jon has over 10.000 downloads and it is been used in several Countries, I'm pretty happy that it filled the gap between professional solutions and the free ones.

Why going open?

First of all, one of the most difficult tasks is to get some feedback from the users, not because they don't want, just because there is a lot of spammers out there willing to put porn and viagra ads in every site they can, I started a forum and tried to keep the spammers out of it but was a futile war, they won... crash and burn me, I do not have the time to deal with web issues, and be a policeman of every post on my site, that's why I closed the forum, I had to spend too much time deleting spam messages, then figuring out how to keep the spammers out, etc... I prefer to use that time to improve the application and not to keep these unwanted people out of my life,

So, what if the users do the extensions and fit the tool to their expectations? that's why I decided to open the source code, to let the users improve it and get the best from the tool.

What's comming?

I need to do several things before publishing the code, like license to be used, put all the documentation about the code, create a standard structrure (make and all of that stuff), etc etc... but I thing it worth the effort.

My own NoSQL DB, progress

The last few weeks have been good... hard... but good, I finally implemented some new layers in the DB, some of them were harder than others, and i realized that I dont know that much of some bare metal components like network layers as I thought I know, but anyway I managed to pass through all of this and the db is now running smoothly, this post is to write down some of the progress made and explain some of the problems I had implementing these new layers.

Command Layer 

This component is really one of the cores of the communications with the user, basically it's meant to process the commands made by an external player and figures out what is the request, how to handle it and what is expected to be returned to the client.

This component was not really a problem, basically because I already knew how to handle the network protocol, then I wrote a writer and a reader and that's it, the command is alive!.

Network Layer

This component was really a hard ball, I didn't know that synchronize the sends with the reads will be that hard, this component drove me crazy for some days, until I realized that I had the network layer does not sign a contract with you to send exactly what you put on the pipe, thus is not suppose to get the expected length at the end of it. Let me put it this simple:

Client A writes 1000 bytes on the pipe.

Server expects 1000 bytes...

But... that does not happened, the client actually sends some chunks of data, 100 bytes at first, 500 later, 200... etc. Thus the server should expect to read this mess in the same way, now that I read some literature about it (that's why it took me 1 week to implement this layer) I know that it's easy... but this layer was a real pain (if you have questions about this, just let me know and I will try to explain how to create a reliable server/client communication), I still to do some improvements in order to get better performance, but anyway it's working right now and I dont want further delays in this project.

Index implementation

Here's the tricky part of the databases, figure out how to create an efficient way to sort the data in order to retrieve ii later, in this part I tried to avoid the conventional methods, and try to figure it out from my own knowledge... big mistake!... hahahaha, anyway this tries outs were good to understand why the people came out with fancy solutions like B+Trees, BTrees, etc. here's the BUT... I read documentation on B+trees and it didnt fit well on what I think should be the separation between data and indexes, so I had to create my own solution to this problem, of course... based on B+trees knowledge and some of my tries outs.

Now, the tests runs very well... the db gets the following output:

20000 inserts per secs and 29978 finds by key operations per sec.

The thing with the network layer is not going that well, but I will fix this later, with the network layer on the output is: 8695 ops, close to the achieved output of MongoDB which makes me very happy for now. I know that when the time comes I will fix the network issues and that will be a nice boost for my server.

Later I will create some posts of my learning process and what problems I had to solve in order to keep on with this project.

The project keeps looking good, and I will keep going on.

My own NoSQL DB

15 days ago I read this article: http://blog.cubrid.org/dev-platform/nosql-benchmarking/ and got shock with the amazing performance, c'mmon! 21k operations per second? it's amazing...

After a long bath I started to wonder, how much performance will I achieve if I create my own NoSQL database, just an experiment for those rainy days, ok... here I am 1 week later after some hard work, I created my own NoSQL Database which is capable of:

1. Insert Records
2. Find records using the key (that's the benchmark)
3. Index the keys and whatever I want to index to find records later.
4. Split files to get indexes separated from data.
5. Created a layer to save the data using little endian (basically I copied the code from my other project d-jon)

Here's the test I did, with a modest machine and 8Gb of RAM:

1. 1M inserts of 1k (I know, the bench was over 50M, but I dont have free space on my machine right now)
2. A random search using random keys (300.662 searches)

And here are the stats:

the inserts took 23.48 secs, which means 42.584 operations per sec.
The searches took 20.3 secs, this means 14.763 operations per sec.

These results are preliminary, but I'm starting to think that I could beat MongoDB, Cassandra, and others,

I started my project because I was out of challenges, and it's turning to be a very attractive project, maybe this could be my next fulltime job.

A lot of work coming, but what the hell... djon started just like that and now have over a 3k of downloads and users all over the world.

C++11 C++x0 new C++ is here!

At last, after almost 3 years the new standard has become a reality, this new standard has a lot of cool features that I will try to cover within my next posts.

Follow the link for The C++ Standards Committee if you want to get a "light" read and have an understanding of what's going on with C++11. Or just keep connected because I will try to explain the major changes about the standard.

Almost completed

Ok, my game is almost complete, at least the main idea, interaction and elements required (graphics, sounds, gameplay, etc), during the last 3 hours I created the level system, an explosion animation (I didn't find an animation that suites my needs so I need to create it from scratch using gimp and imagination), I also added sounds.

In Addition, my tester (my 4 years old son) did a test and I think it worked as expected, everytime he see me on the computer ask to play the game again and again... the magic is done!

Now I need to work on the following things:

1. The health of the UFO
2. The health of the player
3. The fight back from the UFO (easy with the engine I created)
4. An score system.
5. More sounds.

At this rate I think I will finish my first game in 1 week (or less if I have time during my vacations).

I'm very happy with the current results and I will share some demos of the code (not the one used in the game, but I will manage to create some demos of how I used SDL in my first game).