tag:blogger.com,1999:blog-40535447581402961392024-03-05T20:22:39.716-08:00CrossCrosshttp://www.blogger.com/profile/17985383212473053331noreply@blogger.comBlogger32125tag:blogger.com,1999:blog-4053544758140296139.post-15005135676525357552018-04-21T02:25:00.001-07:002018-04-21T02:26:17.897-07:00Capturing and killing EiTest!!!The goal of this post is just to document what I did to solve my issue, in case I need to do it again, and as a plus it might help others struggling with the same.<br />
<br />
Recently one of my email servers started to get rejections from other servers, and as usual the place to go is mxtoolbox.com to check if this was blacklisted, and indeed it was by CBLAbuse, this organization checks, among others, for botnet activities including a trap to check servers sending massive connections to different things, which usually means a botnet is installed under the category of EiTest (this has been around for a couple of years and keeps spreading, there're some news this week that an organization found a way to stop it.<br />
<br />
Ok, based on the documentation I found, most of the time the infected machine is inside the network and the instructions are mainly to detect the machine and then run and antivirus, as my server is a web server this point is completely useless, I knew the connections were coming from my server, but based on the documentation there should be a botnet running that fires up some things, and then hides again, so I'd to check for options to capture the network traffic and analize where this connections were coming from, fortunately CBL gives a sinkhole (a server used to trap this activity) so I can check if there are connections coming from my server that goes to this address, so first step check the network activity.<br />
<br />
tcpdump -w archive.log<br />
<br />
This command allows to log all the activity on my linux to a file, if I want to leave it running for a couple of hours it's just a matter of doing:<br />
<br />
nohup tcpdump -w archive.log 2>/dev/null &<br />
<br />
but dont forget to kill it after some minutes/hours otherwise it will eat up the disk!<br />
<br />
To analyze the contents of the file you will need to download it to your machine and open it up with Wireshark.<br />
<br />
This confirmed what I already knew, that my server was producing a lot of HTTP calls to the IP Address 192.42.119.41 (sinkhole), next step is to figure out which application is the one causing this, so after some digging I've to rely on the trusty netstat, you will find a lot of different solutions out there but after loosing too much time trying to check how to make them work I decided to use the tools I already have at hand, so I run the following command for a couple of hours:<br />
<br />
watch "netstat -atpun|grep 192.4"<br />
<br />
unfortunately the connections flashed too quick and most of the time they were as "TIME_WAIT" (already closed) so I've created the following script (created is a big word here, I've just copied and paste from stackoverflow as 99% of the things we do nowadays):<br />
<br />
while true<br />
do<br />
netstat -atpun|grep 192.4 | tee -a log.txt<br />
sleep 1<br />
done<br />
<br />
This small script allowed me to trace for a couple of hours, and finally this showed that the application executing this was apache server (no news here, a lot of suggestions were pointing to wordpress been infected), now the aweful part of the history, I'd to take down one by one each server to find which of the 4 domains I've was causing this, once I'd narrowed it down to 1, I'd deactivated allow the plugins and voila! the log.txt stopped showing the ip... at the end the plugin infected was "documentor", so I've removed this and reinstalled new from the repos, and this fixed the 4 days of suffering...<br />
<br />
Hope this helps to others, if not I'm sure this will help me in the future to avoid wasting time reading a lot of generic information about the EiTest.Crosshttp://www.blogger.com/profile/17985383212473053331noreply@blogger.com0tag:blogger.com,1999:blog-4053544758140296139.post-85696436184120192682017-07-11T05:21:00.000-07:002017-07-11T05:40:44.634-07:00Bizagi logging: The Right Way!<p>In many projects one of the repetitive questions is how to produce effective logs in our processes, and how to use them in production, unfortunately the answer is: "you can use logs in development, but you need to switch them off in production or the performance will be hurt".</p>
<p>There are several logs in the market that we can use to include in our rules, so I implemented a wrapper of the Log4Net that you can found here: <a href="https://github.com/crossleyjuan/bizagilogger">https://github.com/crossleyjuan/bizagilogger</a>.</p>
<p>The good thing about this is that you can use logs in production environments without having to worry (too much) about the impact in performance, it's just a matter of configuring this right and get the best based on configuration. The bad, this is something you need to configure in the config files and not from the Bizagi Management Console, therefore turning off the traces from Bizagi console will not have any effect.</p>
<h2>
How to use the BizagiLogger</h2>
To use Log4net in your projects you will need 3 elements:<br />
<ul>
<li>The Component Library: BizagiLogger</li>
<li>Configure the .config files</li>
<li>Create a Library Rule to wrap all the logs (Recommended)</li>
</ul>
<h3>
Component Library: Bizagi Logger</h3>
<p>Feel free to download the required dlls from here: <a href="https://github.com/crossleyjuan/bizagilogger/releases/tag/1.0">https://github.com/crossleyjuan/bizagilogger/releases/tag/1.0</a> or download the source code and compile it yourself. This dll depends on log4net.dll, a great Apache Project that makes this possible.</p>
<p>Register both dlls as Component Libraries in Bizagi</p>
<h2>
Using the logger in your rules</h2>
<strong>Syntax:</strong>
<pre style="font-family:arial;font-size:12px;border:1px dashed #CCCCCC;width:99%;height:auto;overflow:auto;background:#f0f0f0;;background-image:URL(https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjAW4UyUDS4TcQQg1Dl61sieWswjoAPjx3-6rKBq3mSml1hASjFVhrxv8DyF7aGqfZ18xPTsK9x8TqoI4HYkCs2PWDoIDwJHvZCQMmRkHIkIV-7VGSSeDqHbvEzzN5vOAvjkcMHknvMvuIu/s320/codebg.gif);padding:0px;color:#000000;text-align:left;line-height:20px;"><code style="color:#000000;word-wrap:normal;"> var log = Logger.getLogger(emitter);
log.Info(ctx, text);
log.Error(ctx, text);
log.Warn(ctx, text);
log.Debug(ctx, text);
</code></pre>
<br />
<ul>
<li>The emitter is a very useful filter, as you can collapse the same messages related to the same emitter, my suggestion is to use the Case Number</li>
<li>ctx is a tag variable called NDC in Log4Net, this helps to search for an additional parameter. for example you can use the name of the process</li>
</ul>
<h3>
Example</h3>
<pre style="font-family:arial;font-size:12px;border:1px dashed #CCCCCC;width:99%;height:auto;overflow:auto;background:#f0f0f0;;background-image:URL(https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjAW4UyUDS4TcQQg1Dl61sieWswjoAPjx3-6rKBq3mSml1hASjFVhrxv8DyF7aGqfZ18xPTsK9x8TqoI4HYkCs2PWDoIDwJHvZCQMmRkHIkIV-7VGSSeDqHbvEzzN5vOAvjkcMHknvMvuIu/s320/codebg.gif);padding:0px;color:#000000;text-align:left;line-height:20px;"><code style="color:#000000;word-wrap:normal;"> var sEmitter = Me.Case.CaseNumber;
var log = Logger.getLogger(sEmitter);
log.Info("Process A", sLogText);
</code></pre>
<h2>
Configuration</h2>
To configure the way the logger works you can use the appenders as explained in the log4net documentation, to make this easier here's an example that could be useful in a production environment:<br />
<ul>
<li><h3>
Add the configuration section:</h3>
<p>To be able to use the log4net options you need to let .net known that you will add some configuration elements for log4net, to do this you need to add this to your config files in the "configSections" section:</p>
<pre style="font-family:arial;font-size:12px;border:1px dashed #CCCCCC;width:99%;height:auto;overflow:auto;background:#f0f0f0;;background-image:URL(https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjAW4UyUDS4TcQQg1Dl61sieWswjoAPjx3-6rKBq3mSml1hASjFVhrxv8DyF7aGqfZ18xPTsK9x8TqoI4HYkCs2PWDoIDwJHvZCQMmRkHIkIV-7VGSSeDqHbvEzzN5vOAvjkcMHknvMvuIu/s320/codebg.gif);padding:0px;color:#000000;text-align:left;line-height:20px;"><code style="color:#000000;word-wrap:normal;"> <section name="log4net" type="log4net.Config.Log4NetConfigurationSectionHandler, log4net" />
</code></pre>
<p>full section:</p>
<pre style="font-family:arial;font-size:12px;border:1px dashed #CCCCCC;width:99%;height:auto;overflow:auto;background:#f0f0f0;;background-image:URL(https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjAW4UyUDS4TcQQg1Dl61sieWswjoAPjx3-6rKBq3mSml1hASjFVhrxv8DyF7aGqfZ18xPTsK9x8TqoI4HYkCs2PWDoIDwJHvZCQMmRkHIkIV-7VGSSeDqHbvEzzN5vOAvjkcMHknvMvuIu/s320/codebg.gif);padding:0px;color:#000000;text-align:left;line-height:20px;"><code style="color:#000000;word-wrap:normal;"> <configSections>
<section name="bizAgiComponentLibrary" type="BizAgi.ComponentLibrary.CComponentLibrarySectionHandler,BizAgi.ComponentLibrary"/>
<section name="bizAgiWFESTouchPoints" type="BizAgi.WFES.TP.CTouchPointsSectionHandler,BizAgi.WFES"/>
<section name="log4net" type="log4net.Config.Log4NetConfigurationSectionHandler, log4net" />
<!-- Uncomment for federate authentication -->
<!--<section name="microsoft.identityModel" type="Microsoft.IdentityModel.Configuration.MicrosoftIdentityModelSection, Microsoft.IdentityModel, Version=3.5.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35" />-->
</configSections>
</code></pre>
</li>
</ul>
<p>Now you can add the appenders in any part of the <configuration> tag, you can use the following as starting point</p>
<pre style="font-family:arial;font-size:12px;border:1px dashed #CCCCCC;width:99%;height:auto;overflow:auto;background:#f0f0f0;;background-image:URL(https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjAW4UyUDS4TcQQg1Dl61sieWswjoAPjx3-6rKBq3mSml1hASjFVhrxv8DyF7aGqfZ18xPTsK9x8TqoI4HYkCs2PWDoIDwJHvZCQMmRkHIkIV-7VGSSeDqHbvEzzN5vOAvjkcMHknvMvuIu/s320/codebg.gif);padding:0px;color:#000000;text-align:left;line-height:20px;"><code style="color:#000000;word-wrap:normal;"> <log4net>
<appender name="BufferingForwarder" type="log4net.Appender.BufferingForwardingAppender">
<bufferSize value="512" />
<lossy value="false" />
<Fix value="268" />
<appender-ref ref="RollingFile" />
<!-- or any additional appenders or other forwarders -->
</appender>
<appender name="RollingFile" type="log4net.Appender.RollingFileAppender">
<file type="log4net.Util.PatternString" value="c:\temp\Logs\WebApp-%date{yyyy-MM-dd}.log" />
<appendToFile value="false"/>
<rollingStyle value="Composite"/>
<maxSizeRollBackups value="-1"/>
<maximumFileSize value="2MB"/>
<staticLogFileName value="true"/>
<datePattern value="yyyy-MM-dd"/>
<preserveLogFileNameExtension value="true"/>
<countDirection value="1"/>
<layout type="log4net.Layout.PatternLayout">
<conversionPattern value="%date [%thread] %-5level %logger [%property{NDC}] - %message%newline" />
</layout>
</appender>
<root>
<appender-ref ref="BufferingForwarder" />
</root>
</log4net>
</code></pre>
<p>These appenders have a very good performance so you can use as many traces as you want in production with almost no impact in your performance, the reason is that the BufferingForwarder keeps chunks of information in memory and writes them in an asynchronous way to the RollingFile appender, this stores the file creating a new one every day or of the file reaches the maximum defined in the "maximumFileSize> tag.</p>
<p>More information of the available appenders here: <a href="https://logging.apache.org/log4net/log4net-1.2.13/release/sdk/log4net.Appender.html">https://logging.apache.org/log4net/log4net-1.2.13/release/sdk/log4net.Appender.html</a></p>
<p>do you have questions? comments? use the comment below if you need additional information</p>Crosshttp://www.blogger.com/profile/17985383212473053331noreply@blogger.com0tag:blogger.com,1999:blog-4053544758140296139.post-23141230889422275592015-03-28T04:44:00.001-07:002015-03-28T04:44:54.022-07:00Open Source vs Closed SourceWhen I started djondb some years ago I'd a different view on what an open source project means, therefore I want to share my short experience to help others to understand the hidden trade offs and the work that requires these kind of decisions. I just want to share with all of you my disappointing experience with the Open Source, my limited view on this and why I'm rolling back the decision I made 4 years ago to open the source code of djondb to the world.<br />
<br />
With my limited experience in the open source world I must say that I was impress on how the open source projects created communities from nowhere, people willing to help, either sharing ideas or code, everybody working on a common goal, getting a nice product that solves the problem at hand. This is exactly what I want, share my passion to solve problems, the idea was good, everyone that I spoke to shared the same view, that NoSQL looked like a great idea, but the lack of transactions was something that is hard to trade, doing this on my own could be near to impossible, with my limited time and resources, also after so many years of using open source tools it was reasonable to jump in a put the code open source, so everyone will benefit from this new idea and I will bring balance to the world.<br />
<br />
When I started the code I decided to keep it close until it was more mature, then someone pointed that as a bad idea:<br />
<br />
<div class="comment-header" id="bc_0_0M" kind="m" style="background-color: white; color: #444444; font-family: Arial, Tahoma, Helvetica, FreeSans, sans-serif; font-size: 13px; line-height: 18.2000007629395px; margin: 0px 0px 8px;">
<cite class="user" style="font-weight: bold;"><a href="http://www.blogger.com/profile/11485603188334241137" rel="nofollow" style="color: #4d469c; text-decoration: none;">Kunthar</a></cite><span class="icon user" style="font-weight: bold;"></span><span class="datetime secondary-text" style="margin-left: 6px;"><a href="http://crosstantine.blogspot.com/2012/08/nosql-or-rdbms-again.html?showComment=1345652461775#c8955394077467176764" rel="nofollow" style="color: #4d469c; text-decoration: none;">August 22, 2012 at 9:21 AM</a></span></div>
<div class="comment-content" id="bc_0_0MC" style="background-color: white; color: #444444; font-family: Arial, Tahoma, Helvetica, FreeSans, sans-serif; font-size: 13px; line-height: 18.2000007629395px; margin-bottom: 8px; text-align: justify;">
<i>No source code on Github? Only drivers?<br />Good move. Bye bye djondb.</i></div>
<div class="comment-content" id="bc_0_0MC" style="background-color: white; color: #444444; font-family: Arial, Tahoma, Helvetica, FreeSans, sans-serif; font-size: 13px; line-height: 18.2000007629395px; margin-bottom: 8px; text-align: justify;">
<br /></div>
<div class="comment-content" id="bc_0_0MC" style="background-color: white; color: #444444; line-height: 18.2000007629395px; margin-bottom: 8px; text-align: justify;">
<span style="font-family: inherit;">So I thought, yes, Kunthar is right, I'm planning to open source the code anyway, so why not now? and started the preparation, prepare makefiles to make easier the compilation by others, prepare scripts, documentation, etc... that took several weeks, even more than some of the code components I've made for the project, but the idea worth it.</span></div>
<br />
The summary of this journey as open source project was firstly pointed here: <a href="http://crosstantine.blogspot.com.es/2013/03/balance-after-5-months-of-releasing.html">http://crosstantine.blogspot.com.es/2013/03/balance-after-5-months-of-releasing.html</a> in 2013, exactly 2 years ago, even that I was disappoint with this already, several people pointed the failures of my endeavor, their arguments:<br />
<br />
<br />
<ul>
<li>The documentation needs improvement</li>
<li>You need to promote your database more</li>
<li>Reporting issues is not easy, this could be improved.</li>
</ul>
And therefore I decided to stick with the idea and improve this things and check what happen, in 2014 I decided to focus on the code again, as I have been working too hard in two sides, coding and promoting, having more success in the coding side than in the promotion and engagement I decided to focus on what I know more, code. Making a stronger database, fulfilling the promises it will attract other developers and users to support the open source project. <br />
<br />
The cruel reality is that users are too busy to report issues, the only one that will be a good tester is yourself, developers are too busy to get engage unless they see a money reward on this, (I got several emails saying they wanted to be partners on this, but at the end none of them had the experience or the time to really spend on this).<br />
<br />
After 4 years of having this project in github I've produced 911 commits, the project is ranked among the 30% projects in github with more commits, even that I'm just 1 developer, from that... only 2 issues were reported and no one ever sent a pull request or patches,<br />
<br />
So what I got from this experience? not too much, just a new experience and a bunch of biased conclusions:<br />
<br />
<br />
<ul>
<li>If you don't have a stable product don't go to open source</li>
<li>People is willing to get involved if the product has created a name on its own.</li>
<li>Use your product yourself, don't wait for others to test it.</li>
<li>Don't waste time creating cross-platform compilation tools, etc. if the product got traction in one platform others will be willing to help with these things to compile it in their own platform.</li>
<li>Don't listen to others unless they have a proven history of success in the field you are trying to get into.</li>
</ul>
<br />
Some of these may sound very obvious to others, but they were not for me, and I'd to learn it in the hard way.<br />
<br />
One of the few wise decisions I made was to hold the copyright of the code, otherwise I will be screwed, and after 4 years of work, with no help at all, I will need to forget this product, I'm glad that I read that from MySQL and others.<br />
<br />
<h3>
What now?</h3>
djondb removed the support from github, and will change it's license to closed source, as per GPL states the versions that are around still will be free as in GPL, although the copyright of the name it's still mine, the GPL allows others to use it as an open source product, so everything below the version 0.3.3 that's is out there will continue under open source, and newer versions will be closed source.<br />
<br />
<h3>
Is djondb still free to use?</h3>
Yes, djondb is free to use and you can put it in as many servers as you want, commercial license will be provided if you require it and the enterprise support will return. In exchange you will require to register to download, not too much to ask for a free database don't you think?<br />
<br />
<br />Crosshttp://www.blogger.com/profile/17985383212473053331noreply@blogger.com0tag:blogger.com,1999:blog-4053544758140296139.post-83970744049287663672014-10-13T08:34:00.000-07:002014-10-13T08:39:50.787-07:00Bizagi - Load Balancing - Where to specify the URL used in notifications to point to the load balancer?<p>Bizagi sends notifications to users when they are assigned to a particular task, this is known as <a href="http://help.bizagi.com/bpmsuite/en/index.html?automatic_email_messages.htm">Automatic notifications</a>, these notifications include the following default message:</p>
<p><strong>Here is a brief information about the case 1251 (Id Case: 1251):<br>Creation Date: 10/13/2014<br>Category: Processes<br>Process Definition: Vacation Request<br><br>Click here <a href=3Dhttp://dev-mymachine/BizagiApp/default.aspx?widget=3Dactivityform&idCase=3D1251>1251</a> to view the case online.</strong></p>
<p>As default Bizagi will use the machine name where it was generated as part of the link, if you want to point these links to the load balancer address you will need to configure the PROTOCOL, SERVER_NAME and APP_NAME based on your configuration, to do this you will need to add the following elements to your WebApplication/web.config file:</p>
<pre><code>
<add key="SERVER_NAME" value="LoadBalancerHostName" />
<add key="APP_NAME" value="BizagiApplication" />
</code></pre>
<p>This will produce the following output:</p>
<p><strong>Here is a brief information about the case 1251 (Id Case: 1251):<br>Creation Date: 10/13/2014<br>Category: Processes<br>Process Definition: Vacation Request<br><br>Click here <a href=3Dhttp://LoadBalancerHostName/BizagiApplication/default.aspx?widget=3Dactivityform&idCase=3D1251>1251</a> to view the case online.</strong></p>
<p>Which will point users to the load balancer instead of the cluster machine</p>
Crosshttp://www.blogger.com/profile/17985383212473053331noreply@blogger.com0tag:blogger.com,1999:blog-4053544758140296139.post-57538929448954958322014-08-22T03:55:00.002-07:002015-05-15T04:27:43.941-07:00Bizagi Sub Processes Parent / ChildSub processes is one of the best reusable elements that you can use to support your business, they allow you to encapsulate logic and share it with other processes, but it's also one of the elements that creates coupling, the concept is fully explained here: <a href="http://en.wikipedia.org/wiki/Loose_coupling">"Loose Coupling"</a>. So how do you design sub processes that are truly re-usable and easy to maintain?<br />
<br />
<h2>
Sub-processes as a functions</h2>
<br />
The typical approach to use sub processes creates dependencies between the caller and the sub process, therefore any change to the parent may impact the child and same applies for the changes in the child, that's called highly coupled design. To work around this and create cleaner processes you should consider your sub processes as "functions" that will be called and return a result. Let me clarify this with a simple <a href="http://en.wikipedia.org/wiki/Hello_world_program">"hello world!"</a> code. Let's imagine that you want to create a sub process to standarise the "greeting" messages, therefore you want to solve this problem:<br />
<pre style="background-image: URL(https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjAW4UyUDS4TcQQg1Dl61sieWswjoAPjx3-6rKBq3mSml1hASjFVhrxv8DyF7aGqfZ18xPTsK9x8TqoI4HYkCs2PWDoIDwJHvZCQMmRkHIkIV-7VGSSeDqHbvEzzN5vOAvjkcMHknvMvuIu/s320/codebg.gif); background: #f0f0f0; border: 1px dashed #CCCCCC; color: black; font-family: arial; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;">
function parent() {
recover_name;
child_create_greeting();
print greeting message;
}
</code></pre>
Current approach most of the times:<br />
<pre style="background-image: URL(https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjAW4UyUDS4TcQQg1Dl61sieWswjoAPjx3-6rKBq3mSml1hASjFVhrxv8DyF7aGqfZ18xPTsK9x8TqoI4HYkCs2PWDoIDwJHvZCQMmRkHIkIV-7VGSSeDqHbvEzzN5vOAvjkcMHknvMvuIu/s320/codebg.gif); background: #f0f0f0; border: 1px dashed #CCCCCC; color: black; font-family: arial; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;">var name;
var greeting;
function parent() {
name = recover name;
child_create_greeting();
print greeting message;
}
function child_create_greeting() {
greeting = "Hello " + name;
}
</code></pre>
I know this is a silly sample, but bear with me for a moment, and check the next approach:<br />
<pre style="background-image: URL(https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjAW4UyUDS4TcQQg1Dl61sieWswjoAPjx3-6rKBq3mSml1hASjFVhrxv8DyF7aGqfZ18xPTsK9x8TqoI4HYkCs2PWDoIDwJHvZCQMmRkHIkIV-7VGSSeDqHbvEzzN5vOAvjkcMHknvMvuIu/s320/codebg.gif); background: #f0f0f0; border: 1px dashed #CCCCCC; color: black; font-family: arial; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;">function parent() {
var name = recover name;
var greeting = child_create_greeting(name);
print greeting message;
}
function child_create_greeting(name) {
greeting = "Hello " + name;
return greeting;
}
</code></pre>
The result is the same, but now your child and parent may evolve without impacting each other every time you do a change. Same concept could be applied in Bizagi processes, in the sample above I was using the variables name and greeting to share information with the parent, but what if I create a "message" envelop, that both processes share and act as communication between both? something like this:<br />
<pre style="background-image: URL(https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjAW4UyUDS4TcQQg1Dl61sieWswjoAPjx3-6rKBq3mSml1hASjFVhrxv8DyF7aGqfZ18xPTsK9x8TqoI4HYkCs2PWDoIDwJHvZCQMmRkHIkIV-7VGSSeDqHbvEzzN5vOAvjkcMHknvMvuIu/s320/codebg.gif); background: #f0f0f0; border: 1px dashed #CCCCCC; color: black; font-family: arial; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;">message {
var name;
var greeting;
}
function parent() {
message.name = recover name;
child_create_greeting(message);
print message.greeting;
}
function child_create_greeting(message) {
message.greeting = "Hello " + message.name;
}
</code></pre>
This creates an element where both processes will agreed as sharing mechanism and therefore they will be able to evolve without any impact, as long as the "message" element follows the original agreement. This translated to Bizagi concepts is just an entity that will hold the information that sub-process requires and where it will store the result of it's work. The responsibility of the caller process is to create the message instance, assign the parameters and then use the result stored there. Does it make sense? let's see it in action:<br />
<h2>
A real problem</h2>
The "Hello world!" sample is good way to explain basic concepts, but let's face it, it's worthless if you don't see something that it's applicable to real life problems, so here's a typical problem: "Our company requires an standard way to authorize different requests across our organization, therefore we need a reusable sub process that will be plugged in processes like, but not limited to: Travel Expenses and Vacations".<br />
<h3>
Model your sub processes as reusable process</h3>
The first step is understanding that your approval process will have different contexts and therefore it should be modeled with that independence in mind, the trick here is to be able to understand that your sub process is not approving Expenses or Vacations, it's approving a Requests (Value of the expense) or actions (Taking vacations), and therefore you will model your sub process to approve "requests" or "actions".<br />
<h2>
The Process and sub process definition</h2>
<h3>
Vacation request:</h3>
<br />
<div class="separator" style="clear: both; text-align: left;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgOoFVxymfTZud8V7z6XWRi6jtAqBVFJfyKHBg8rhIbi0O2ow7MXw1kvbZdw14XI3IUkxtdSLfSXhcwnClX3YuPeM4HnnRDlCE8svxcsWZJIUyJxDet4kt13qMK6_DdXy3iWMXLZtyN6SI/s1600/Vacation+Request.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgOoFVxymfTZud8V7z6XWRi6jtAqBVFJfyKHBg8rhIbi0O2ow7MXw1kvbZdw14XI3IUkxtdSLfSXhcwnClX3YuPeM4HnnRDlCE8svxcsWZJIUyJxDet4kt13qMK6_DdXy3iWMXLZtyN6SI/s550/Vacation+Request.png" /></a></div>
<br />
<h3>
Approval:</h3>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjOOGS_RyL36mrsNzcwYsd1wReVtgBnwEAu2oYbUrZHT9HkyGklOf9HPkFbS_HwBQZJyUrcEZMrrGO4WKQGwwdZ9wnOVElwaWmqe7c5aZ9eehNshsh5oeJMcHHItKSfZp1Qsxq0uTImUQU/s1600/Sub_Approval.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjOOGS_RyL36mrsNzcwYsd1wReVtgBnwEAu2oYbUrZHT9HkyGklOf9HPkFbS_HwBQZJyUrcEZMrrGO4WKQGwwdZ9wnOVElwaWmqe7c5aZ9eehNshsh5oeJMcHHItKSfZp1Qsxq0uTImUQU/s550/Sub_Approval.png" /></a></div>
<br />
<h2>
Data Models</h2>
<h3>
Vacaction Request Model</h3>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhohtjwgUBqEAnRXRaGl2vaYLu7Eu8G0JaPPshacSmyFbXWGtUbL0PcndSumjjf9xFRurnhZemKNOCKL66waL7Pq0ZGVUCMrfV2PfymnYlqHYATW8vsqhoBlRm6FqI-WfXk5aWjS34JQdk/s1600/Vac+DataModel.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhohtjwgUBqEAnRXRaGl2vaYLu7Eu8G0JaPPshacSmyFbXWGtUbL0PcndSumjjf9xFRurnhZemKNOCKL66waL7Pq0ZGVUCMrfV2PfymnYlqHYATW8vsqhoBlRm6FqI-WfXk5aWjS34JQdk/s550/Vac+DataModel.png" /></a></div>
<br />
<h3>
Approval Sub Process Model</h3>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiCOnKw7uskdiES0mgVro_lRxEU2Tu4GYYuijatd_gM9FpiArwddr-0wd9qrO38KCV8I8ommS7kTV18ELO0MRLLrWjiew_lmkQidNL25rSnqRNTnGN33hTV7tsQMRFWBGter97DPf9cSSM/s1600/Appr+DataModel.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiCOnKw7uskdiES0mgVro_lRxEU2Tu4GYYuijatd_gM9FpiArwddr-0wd9qrO38KCV8I8ommS7kTV18ELO0MRLLrWjiew_lmkQidNL25rSnqRNTnGN33hTV7tsQMRFWBGter97DPf9cSSM/s550/Appr+DataModel.png" /></a></div>
<br />
As you may noticed the M_Message entity is used in both processes, I added the suffix _I or _O for Input and Output to make it easier to understand. Now it's time to fill in the variables and passing them from the caller to the sub process, let's jump to code.<br />
<h2>
Passing "parameters" to the sub process</h2>
As explained above the M_Message entity will be used as communication element between the caller and the sub process, and therefore any element require by the Approval Sub process should be placed there, that includes simple data, like "Value to approve", "Date requested" and Entities (like Employee, Request Detail, etc), here I'm using "request type" to enable the sub process to handle different kinds of messages.<br />
<pre style="background-image: URL(https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjAW4UyUDS4TcQQg1Dl61sieWswjoAPjx3-6rKBq3mSml1hASjFVhrxv8DyF7aGqfZ18xPTsK9x8TqoI4HYkCs2PWDoIDwJHvZCQMmRkHIkIV-7VGSSeDqHbvEzzN5vOAvjkcMHknvMvuIu/s320/codebg.gif); background: #f0f0f0; border: 1px dashed #CCCCCC; color: black; font-family: arial; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;">var reqType = CHelper.getEntityAttrib("P_RequestType","idP_RequestType","RT_Code = 'REQ'");
<VacationRequest.entApproveRequest.entRequestType_I> = reqType;
var requestMessage = String.Format("Requesting vacactions starting from {0} for {1} days.", <VacationRequest.dStartDate>, <VacationRequest.iDays>);
<VacationRequest.entApproveRequest.entRequest_I.RequestDescription> = requestMessage;</code></pre>
This step is just populating the "message" that we are using to pass the values, now we will set the entity to the attribute that will be referenced in the sub process, <Sub_Approval.entMessage><br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiRlDjzoUfbIYr2Ghacv-ISKLsZIMCFx9zV8aTOas01nOreYADeaLTQ1r0bM1g6tjUQ_bXdg1g0R11xFsbF4BlVcvFr7PUjqGDa8RK0UjA1uK-DtpQmkYcneiN5MPGc_ME11hHIaqVRf-Y/s1600/Assign.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiRlDjzoUfbIYr2Ghacv-ISKLsZIMCFx9zV8aTOas01nOreYADeaLTQ1r0bM1g6tjUQ_bXdg1g0R11xFsbF4BlVcvFr7PUjqGDa8RK0UjA1uK-DtpQmkYcneiN5MPGc_ME11hHIaqVRf-Y/s550/Assign.png" /></a></div>
<br />
Congratulations, you've just created a re-usable sub process that is truly re-usable and it wont break the parent caller every time you create a change on it, and the most important element, you can reuse it in any process without changing the sub process to adapt new callers. Love to hear your comments!Crosshttp://www.blogger.com/profile/17985383212473053331noreply@blogger.com0tag:blogger.com,1999:blog-4053544758140296139.post-79841744902381230142013-06-05T12:32:00.000-07:002013-06-05T12:32:00.255-07:00Parsing command line arguments in a batch program (Windows)<p>Everytime I'd to create a batch to do compilations, or any other kind of stuff I always bumped to the same problem, parsing command line arguments. Usually I ended up with something like:</p>
<pre><code>
if "%1"=="-x32" (
Do something for x32
)
if "%2"=="-x32" (
Do something for x32
)
if "%1"=="-x64" (
Do something for x64
)
if "%2"=="-x64" (
Do something for x64
)
</code></pre>
<p>This is really bad way to do it, so I decided to spend sometime today and solve the problem once and for all, as a result I finally have a decent way to parse arguments, and I will document it in here in case I need it again or if someone else is struggling to get this for himself. (I love getopts from Linux! but I guess Windows will keep forcing us to this kind of solutions)</p>
<pre style="font-family:arial;font-size:12px;border:1px dashed #CCCCCC;width:99%;height:auto;overflow:auto;background:#f0f0f0;;background-image:URL(https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjAW4UyUDS4TcQQg1Dl61sieWswjoAPjx3-6rKBq3mSml1hASjFVhrxv8DyF7aGqfZ18xPTsK9x8TqoI4HYkCs2PWDoIDwJHvZCQMmRkHIkIV-7VGSSeDqHbvEzzN5vOAvjkcMHknvMvuIu/s320/codebg.gif);padding:0px;color:#000000;text-align:left;line-height:20px;"><code style="color:#000000;word-wrap:normal;"> @echo off
setlocal enabledelayedexpansion
if [%1] ==[] goto usage
call:parseArguments %*
if "%x32%" == "true" (
echo Well done you set x32 to true
)
if "%x64%" == "true" (
echo Well done you set x64 to true
)
if NOT "%d" == "" (
echo you set the output dir to: %d%
)
GOTO Exit
@rem ================================================================================
@rem Functions
@rem ================================================================================
:usage
Echo Usage: %0 [-x32] [-x64] [-d output-dir]
goto exit
:getArg
set valname=%~1
echo arg: !%valname%!
goto:eof
:parseArguments
rem ----------------------------------------------------------------------------------
@echo off
:loop
IF "%~1"=="" GOTO cont
set argname=%~1
set argname=%argname:~1,100%
set value=%~2
@rem if the next value starts with - then it's a new parameter
if "%value:~0,1%" == "-" (
set !argname!=true
SHIFT & GOTO loop
)
if "%value%" == "" (
set !argname!=true
SHIFT & GOTO loop
)
set !argname!=%~2
@rem jumps first and second parameter
SHIFT & SHIFT & GOTO loop
:cont
goto:eof
rem ----------------------------------------------------------------------------------
:Exit
</code></pre>
<p>The magic ocurrs in the "parseArguments" function, (Oh yes, batch files have functions, I didn't know that until now. This page was really helpful: <a href="http://www.dostips.com/DtTutoFunctions.php">http://www.dostips.com/DtTutoFunctions.php</a>).</p>
<p>This function contains two things that I learned today. The first one: SHIFT, this command shift the arguments putting the second argument in front and the third in second place, and so on.
This is really useful if you don't know the number of arguments, so you only need to do something like the following code to print all the arguments:</p>
<pre><code>
:loop
IF "%~1"=="" GOTO cont
echo %~1
SHIFT & GOTO loop
:cont
</code></pre>
<p>The second think I learned is how to create dynamic variables, thanks to this guy: <a href="http://batcheero.blogspot.com/2007/07/dynamic-variable-name.html">http://batcheero.blogspot.com/2007/07/dynamic-variable-name.html</a>, this is useful to create the variables that you're going to use later in your batch program. This is possible because I used: setlocal enabledelayedexpansion at the very beginning of the batch program, otherwise the !thing! won't work.</p>
<p>So the parse arguments function just iterate over the arguments guessing if they were written as: -x32 or -d c:\temp, (in the first case %x32% will be set to true and the former %d% will contain c:\temp.</p>
<p>Here're the results:</p>
<pre style="background-color:black;color:white"><code>
> test.bat -d "c:\test dir\blah"
you set the output dir to: c:\test dir\blah
> test.bat -d "c:\test dir\blah" -x32 -x64
Well done you set x32 to true
Well done you set x64 to true
you set the output dir to: c:\test dir\blah
</code></pre>
<p>That's it, enjoy</p>Crosshttp://www.blogger.com/profile/17985383212473053331noreply@blogger.com1tag:blogger.com,1999:blog-4053544758140296139.post-52511829154107878152013-04-16T15:41:00.002-07:002013-04-16T15:41:40.061-07:00Writing tests that works<h2>Disclaimer</h2>
<p>Although some of the ideas exposed in here are too obvious to some developers, I wanted to write down some of my thoughts about how the "unit" tests should be implemented in some projects, this is a discussion I'd with a fellow co-worker after reading this article: <a href="https://www.simple-talk.com/dotnet/.net-framework/unit-testing-myths-and-practices/">Unit Testing Myths and Practices</a> and I think worth the time to post it in here, maybe some of you will agree with me, and some could call me dumb, anyway, I would love to read your thoughts about this and discuss about them on the comments section.</p>
<h2>My opinion about "Unit" Tests</h2>
<p>Several definitions of Unit tests are around Internet and in books, but here's a simple demo of what I understand by unit tests, and the way I saw unit tests implemented in several projects.</p>
<pre><code>int myMethod(int arg1, int arg2) {
return arg1 + arg2;
}</code></pre>
Unit test:
<pre><code>void testMyMethod() {
// Testing positive numbers
int a = 1;
int b = 2;
int expected = 3;
int result = myMethod(a, b);
ASSERT(expected, result);
// Testing negative numbers
a = -1;
b = 2;
expected = 1;
result = myMethod(a, b);
ASSERT(expected, result);
// testing zero arguments
...
// testing big integer values
...
}</code></pre>
<p>As you may noticed I required 10+ lines of code to test 1 line of the business method. Now, what will happen to our test cases if we want to test the following method:</p>
<pre><code>int myMethod2(Customer customer, int arg2) {
int age = customer.age();
return myMethod(age, arg2);
}</code></pre>
<p>Now I will need to write tons of lines to check how my code will react to every single combination of customers' age or any value to be added. For example: test cases to check NULL customer, customers without age, summing up a negative age, etc, etc... Is this what I wanted in the first place? will this ensure that, if a reckless developer changes myMethod to get sum Integers instead of int, avoid null errors in the myMethod2? or do we need to rewrite all the tests that depends on the first method? do I even care about nulls been sent to myMethod?</p>
<h2>Dependency injection will save the day!</h2>
<p>Maybe some of you thought, "hey this guy does not know a sh**t about coding and he lacks of expertise, this is easily managed by dependency injection and method contracts". Maybe you are right, but lets check how our methods would change to ensure the usage of dependency injection to separate both concepts (bear in mind that the tests are now going to insert a dummy class that will return the proper values for each test).</p>
<pre><code>
[di]
IMySumClass sumClass;
int myMethod2(Customer customer, int arg2) {
int age = customer.age();
return sumClass.myMethod(age, arg2);
}</code></pre>
<p>Done!, now we have a new problem, both tests are going to pass up the tests phase, and the error will popup in our production system. Yes, I know... the methods' contract... someone would say "the method says that it will receive X and Y and if the X and Y changes then it's a new method because it's a new contract. Ok, here we go:</p>
<pre><code>int myMethod(int arg1, int arg2) {
return arg1 + arg2;
}
Integer myMethodWithIntegers(Integer arg1, Integer arg2) {
return arg1 + arg2;
}</code></pre>
<p>Solved!, now we have 2 methods that executes the same logic, therefore we will need to mantain them as well. (But don't forget that our tests cases will need to be maintained as well, so we have now 30+ lines to keep updated). Too much work to ensure that a simple sum will work, don't you think so?</p>
<h2>Integrity tests</h2>
<p>Ok, let's go back to the simple code we have in the first place:</p>
<pre><code>int myMethod(int arg1, int arg2) {
return arg1 + arg2;
}
int myMethod2(Customer customer, int arg2) {
int age = customer.age();
return myMethod(age, arg2);
}</code></pre>
Unit test:
<pre><code>void testMyMethod() {
// Testing positive numbers
int a = 1;
int b = 2;
int expected = 3;
int result = myMethod(a, b);
ASSERT(expected, result);
// Testing negative numbers
a = -1;
b = 2;
expected = 1;
result = myMethod(a, b);
ASSERT(expected, result);
// testing zero arguments
...
// testing big integer values
...
}</code></pre>
<p>What I found useful is to keep the tests as simple as possible and targeting the class we want to check, so instead of doing a double check of all the different options for a "a + b" operation in the method testMyMethod2, or in the testMyMethod, I would add the tests that are useful to each case, and I will remove the negative checks, zero arguments, etc that will cumbersome my code, also I won't add tests to testMyMethod2 that are not related with myMethod2 itself, therefore I will not add tests that were tested somewhere else. Let's see the sample:</p>
<pre><code>void testMyMethod() {
// Testing positive numbers
int a = 1;
int b = 2;
int expected = 3;
int result = myMethod(a, b);
ASSERT(expected, result);
}
void testMyMethod2() {
// Customer with age
Customer c = new Customer("x", "y", 32);
int expected = 33;
int result = myMethod2(c, 1);
ASSERT(expected, result);
// Test a customer too old (should be rejected because of his age)
c = new Customer("x", "y", 32);
try {
myMethod2(c, 150);
FAIL("Customer age was not properly checked");
} catch (AnyCheckedException) {
// Accepted case
}
}
</code></pre>
<p>Noticed how I removed the "negative", zero and big numbers tests, I really don't care about these cases at this level, and they will be tested by "customer" cases anyway. These kind of tests are going to be more useful and they will protect my code from unexpected "real" failures reducing the amount of work I need to write down all the test cases and covering the business cases instead of arguments misusage.</p>
<p>Off course, this a very simple case, and it would not cover all the "what if" questions, but it will cover what is supported by the system. The last sample test code will check not only cases related to myMethod2, but myMethod as well, this means that myMethod will be checked for every single call made from any of the classes that uses this method, and it's going to be tested in a "business" way, not in a "what if" applied to every single argument.</p>
<p>What usually happens in production environments is that errors, not originally tested, will popup and we will need to add them to the proper test code to ensure they will not rise again, if the error was in the "customer" level then we will add proper lines to the testMyMethod2 (or create a new one), if the problem was with our sum method not covering Big Integers, then we will need to cover it in the customer test as well, any change to the testMyMethod2 will also test myMethod in a business like scenario.</p>
<h2>Conclusion</h2>
<p>Every project could be different and every one will need a solution that fits to it, but I've been using this heavily and this method of "accumulative" testing have worked really well to check and correct bugs in production systems.</p>
<p>I would love to read your thought, please post a comment. The "you're a dumba**" comments are welcome but please try to support your ideas.</p>Crosshttp://www.blogger.com/profile/17985383212473053331noreply@blogger.com0tag:blogger.com,1999:blog-4053544758140296139.post-31879918008715091582013-04-10T13:30:00.000-07:002013-04-10T13:30:19.508-07:00Switching from autotools to something elseAlthough I'm really happy with the results of autotools on Linux and OSX, everytime I want to generate the installers on Windows I needed to do something, at the beginning I just created the sln and vcproj for each of the modules in djondb, later I'd to implement an script to do the compilation process even simpler, then I'd some problems with the libraries, third party libraries, x64, etc and I'd to change some settings on the project files, so far everything worked as expected (not smoothy but worked), lately I've been having some problems that only happened to crash on Windows environments, and here's where the nightmare have begun, I've all the tests written with cpptest (which is a really good option and I love to write tests on it), but now that I need to execute the same tests on windows, and my approach started to flood.<br />
<br />
I've been reading about some other build tools, that are really cross-platform, and I will give a try to CMake, I tried scons, but looks really slow, I don't know why but compiling with it takes double time than same thing on Linux/make counterpart.<br />
<br />
I'll do some tests on CMake, and if everything works I will switch djondb to this build system, I'll post my results in here later.<br />
<br />
Any thoughts? please let me know what you thing about CMake, SCons and make tools.Crosshttp://www.blogger.com/profile/17985383212473053331noreply@blogger.com2tag:blogger.com,1999:blog-4053544758140296139.post-51799534248012933982013-03-25T06:36:00.000-07:002013-06-16T19:34:10.686-07:00Why NoSQL matters?<p>Web applications have been evolving on a daily basis, mainly because web is something so visible that will be shamed if your website is not at the same level of your competitors.</p>
<p>For years the languages and frameworks to build websites changed to fill the demand of new agile development methodologies, servers like nodejs appeared to deal with changes, jquery appeared to allow the creation of richer content, etc. you'll notice the evolution started from front-end technologies and it's moving towards the backend, now it's the time for the ultimate frontier to adjust to this change.</p>
<h2>Evolution of the database model</h2>
<p>One of the things that remains constant over the years is the need to map your information to storage applications like MySQL, Oracle database, Postgres or SQL server. One thing they have in common it's all of them are based on a rigid model, the model of tables, rows and columns, and these lead to several problems, these problems you need to address during this always changing web environment.</p>
<ul>
<li>Managing the instructions at code level to work with a new set of data or new columns at the database level.</li>
<li>Addressing deployment problems, if you create a new field in a table you'll need to take down your server and do a painful process of upgrading, (run scripts to change your tables, execute an script to migrate, or at least update, your new model, and finally updating the code that changed)</li>
</ul>
<p>What if these changes means that your model changed in a more radical way? What if you created a new type of product that requires a full new model? That's something to thing about.</p>
<p>This kind of problems have been a constant over the years until now, some software developers, as myself, started to wonder if there will be a better way to store the information that is more aligned with the current challenges, that's how the NoSql movement started, as a way to solve this new data model paradigm.</p>
<h2>The new data model</h2>
<p>To give a better explanation about how NoSQL address the problem of these always changing data models, let's work using an example as a basis. Lets assume we're going to store customers from our newly created website, to do this job we decided to create a table in an RDBMS, and we included columns like name, last name and genre. After some days we put together an application and finally it's running on web, then tweet about giving some benefits to the people that register during the weekend, after a good weekend you end up with 3M records, you're surprised because you didn't expected that kind of traffic, then you check the db and realized that you didn't provide enough space for some long last names and some of them are truncated, darn you'll need to expand the column.... But, you can not change the size of a column in SQL server, you need to create a new table, move the records and that will mean your server will be out for 1 hour. Does this sound familiar? Even worst, you need to include some new fields in the process, meaning changing the HTML, your server side, doing some migration, etc.</p>
<p>Although JSON does not solve all the problems that come with these kind of changes it's really easy to update the underlining model just modifying a JSON structure than changing the tables, columns, rows and SQL statements, and that's how NoSQL helps in the development of web application, allowing to make this kind of changes easily, storing JSON instead of traditional data model.</p>
<p>A Customer could be model in a JSON like this:</p>
<pre><code>
{
name: "John",
lastName: "Smith",
genre: "Male"
}
</code></pre>
Same problem stated before, what if you need to store a longer last name? Because the model is not rigid, your users will never had the problem in the first place, and their last names were not truncated. What if you need to store new information?
<pre><code>
{
name: "John",
lastName: "Smith",
genre: "Male"
}
{
name: "Peter",
lastName: "Korn",
genre: "Male",
email: "test@mycompany.com"
}
</code></pre>
<p>Done!, you don't need to change anything at the database level. If your REST services are treated as general REST services that get JSON and post them to the DB, you will not need to change your backend either, and the only change you will need is at the HTML and javascript level, which will allow you to do a cleaner deployment of the new version, just post the new HTML and javascripts at the web sever folder and that's it, you are ready to go.</p>
<h2>Conclusions</h2>
<p>Although a good requirements/design/test will always be better than having to get into a production environment to realize that you forgot a piece of data that is really important to your business, the NoSQL databases allows you to upgrade your applications easier, and relieves the pain of creating an application from ground up, it really empowers the developer to explore the data structure while he is creating his master piece.</p>
<p>Later I will keep exploring samples on how to use NoSQL to create applications in a faster/easier way, in the mean time please go to here: <a href="http://djondb.com/blog">http://djondb.com/blog</a> and check out some of the full examples that you will find in there.</p>Crosshttp://www.blogger.com/profile/17985383212473053331noreply@blogger.com0tag:blogger.com,1999:blog-4053544758140296139.post-309017323746352682013-03-01T20:07:00.002-08:002013-03-01T20:07:45.558-08:00Balance after 5 months of releasing djondb as Open Source<p>After 5 months of releasing a big product as an open source project I want to write about the progress, the goods, the bads, and share the future releases of djondb.</p>
<h2>Briefing</h2>
<p>I've been working on djondb for a year, it started as a proof of concept, a learning project and a challenge, rapidly it grew to a running version of a server and I started
to wonder if there was space for a new NoSQL database, and here's the video of the speech I gave at BogoDev in 2012</p>
<iframe width="420" height="315" src="http://www.youtube.com/embed/lSGkLVRSh6Q" frameborder="0" allowfullscreen></iframe>
<p>Although it was a living thing it was far from been a production server. After this speech I realized that there was
a lot of people with the same requirements, a NoSQL database that can store JSON, ensure everything goes to disk and not into "lose-able" memory, and been able to handle transactions
over multiple documents. Last, but not least, been able to run it in Linux, OSX and Windows.</p>
<p>Several tasks were required to be completed once the MVP was a real application:</p>
<ul>
<li>Create an stable full minimal featured release (Insert, Updates, Deletes, Find with meaningful queries, etc)</li>
<li>Implement drivers for most common languages.</li>
<li>Document all the features</li>
<li>Create samples</li>
<li>Installers for each platform</li>
<li>Drink couple of beers! (this was started even before the first line of code)</li>
</ul>
<p>The first version of all the features was released a couple of months later, and djondb was open sourced in github since September 2012.</p>
<p>I almost remember the face of my wife once I told her that I will put the code as open source, "Is he crazy?, he is doing this great product and he will give it away?", but I was completely sure that was the best choice, here is the list of the reasons to go to this path:</p>
<ul>
<li>Give something to the world. I'm using open source tools so seems fair enough to give something back.</li>
<li>An open source is easier to promote, mainly because Open source promotes itself, the advocates of open source push this kind of developments to keep them going.</li>
<li>Some developers might join the development team and we will share the glory, and the money that could come with this solution.</li>
<li>Been open source the community will help with feedback, samples, document translation, etc.</li>
</ul>
<p>In November I did the first "official" speech about djondb in Colombia 3.0, one of the biggest Content Summit of Latin America, here's the video: </p>
<iframe width="560" height="315" src="http://www.youtube.com/embed/xNygESBRCxo?list=PLhTx9maDqE9LOYJHgGgMy4fMqCrUmuf6f" frameborder="0" allowfullscreen></iframe>
<p>After this I've been working on drivers, documentation, new features, bug fixing, etc. And released the version 0.2, which had several performance improvements.</p>
<p>During the latest months I've been working on new features that are going to be released at the end of March.</p>
<h2>Balance</h2>
<p>djondb brough a lot of learning, not only about coding databases, or high performance systems, but on how the Open source ecosystem works, what means to be an Entrepreneur (I'm still learning, and I think I will never stop learning), what traction is, what an investor sees, etc.</p>
<p>djondb has been a very successful project, more than 350 downloads during the last three months and a lot of great reviews. Some good projects are already using it, and people really share the same though of NoSQL as a profesional tool to develop business applications.</p>
<h2>Not everything is nice</h2>
<p>Although I've been working every night to improve and release a better product, it's very disappointing to realize that Open source is not what I thought it will be, it's not a great community trying to help out, sharing ideas or helping with spreading the word, it's more like a bunch of guys (like me), working really hard to share a dream with bare hands and passion. Most of the histories I read everyday about lone riders that try to create open source is just like mine, working hard and getting few or none feedback, just a lot of downloads.</p>
<p>Lessons I learnt from the open source, maybe I'm wrong and it's only my perception, but this is what I think:</p>
<ul>
<li>Most people think open source means free, they don't even think that feedback or test is a way to share and help.</li>
<li>Investors put weird face when you say "Open source", unless you already show that there's a lot of traction. And I mean huge traction, not the kind you would need if you "own" the code.</li>
<li>Other developers look at the open source as a way to get their projects done without paying anything. (there're several ways to give back in the open source, like feedback, helping documentation, maybe fixing a bug or two, etc)</li>
</ul>
<h2>Summary</h2>
<p>Don't go with the open source option if you don't have a company that supports you. Open source will mean that you put all your ideas on the web and almost anyone can copy you and create their own product without any responsibility, I know... you could argue that you can show you posted the code first, etc etc. (If you argue that you could patent your code let me remind you that procedure costs U$35.000 and takes 5 years to be approved). The truth is that you can not protect your idea as open source, you gave away your knowledge and you won't get anything back.</p>
Crosshttp://www.blogger.com/profile/17985383212473053331noreply@blogger.com0tag:blogger.com,1999:blog-4053544758140296139.post-76698257900091569032013-01-25T19:14:00.001-08:002013-01-30T20:55:24.614-08:00Preparación para taller de NoSQL con djondb<br />
Este post lo realizo con el fin de explicar los pasos de instalación y preparación requeridos para el taller que se dictará en el Bogodev el próximo 31 de Enero /2013.<br />
<br />
<h3>
Herramientas necesarias</h3>
<br />
El ejemplo que se realizará usará 3 componentes:<br />
<ul>
<li>Ubuntu 11 o superior.</li>
<li>djondb: Motor de base de datos NoSQL (http://djondb.com)</li>
<li>KnockoutJS: Framework para patron Model-View-View Model MVVM (<a href="http://knockoutjs.com/">http://knockoutjs.com/</a>)</li>
<li>nodejs: Servidor multiproposito capaz de ejecutar javascript server-side (http://nodejs.org</li>
<li>Cualquier editor de texto que facilite la edición de HTML y javascript (vim, Netbeans, emacs, etc)</li>
</ul>
<br />
<div>
<h3>
Instalación</h3>
</div>
<div>
El primer paso será la instalación de la base de datos, para ello vamos al site de djondb y obtenemos la versión para linux correspondiente a nuestra arquitectura.</div>
<div>
<br /></div>
<div>
<pre><code>sudo dpkg -i djondb_Linux_i386.deb</code></pre>
</div>
<div>
<br /></div>
<div>
Probablemente obtendremos unos errores indicando que hacen falta algunas dependencias, como antlr, etc. para instalar las dependencias faltantes se realiza:</div>
<div>
<br /></div>
<div>
<pre><code>sudo apt-get -f install</code></pre>
</div>
<div>
<br /></div>
<div>
Procedemos a crear la carpeta donde se almacenaran los archivos de la base de datos y le daremos permisos al usuario actual, asi:</div>
<div>
<br /></div>
<div>
<div>
<pre><code>sudo mkdir /var/djondb
sudo chown `id -u` /var/djondb/</code></pre>
</div>
</div>
<div>
<br /></div>
<div>
Ahora podremos verificar si el servidor quedó instalado correctamente ejecutando:</div>
<div>
<br /></div>
<div>
<div>
<pre><code>djondbd -n</code></pre>
</div>
</div>
<div>
<br /></div>
<div>
El parametro -n permite que el servidor no ejecute en background permitiendo bajarlo de manera simple usando Ctrl+C.</div>
<div>
<br /></div>
<div>
Una vez instalado, realizamos la instalación del nodejs de la siguiente forma:</div>
<div>
<br />
Entrar a la pagina http://nodejs.org y dar click en install, esto bajará un archivo: node-v0.8.18.tar.gz y seguir los siguientes pasos:<br />
<pre><code>
tar xvfz node-v0.8.18.tar.gz
cd node-v0.8.18
./configure
make
sudo make install
</code></pre>
<br /></div>
<div>
<div>Luego instalamos algunas herramientas requeridas para instalar plugins de nodejs:</div>
<pre><code>sudo apt-get install g++ make
sudo npm install -g node-gyp</code></pre>
</div>
<div>
<br /></div>
<div>
La primera instrucción instalará las herramientas necesarias para ejecutar el servidor e instalar plugins. La segunda instalará el plugin node-gyp que permite compilar extensiones que usen este procedimiento de instalación.</div>
<div>
<br /></div>
<div>
Con estos pasos podemos proceder a crear la carpeta donde realizaremos el taller, e instalaremos el driver de djondb necesario para nodejs, asi:</div>
<div>
<br /></div>
<div>
<div>
<pre><code>mkdir proyecto_nosql
cd proyecto_nosql
npm install djondb</code></pre>
</div>
</div>
<div>
<br /></div>
<div>
Estos son los pasos requeridos para la instalación de las herramientas necesarias para el taller, si tiene algun problema realizando la instalación coloquenos un comentario y con gusto les colaboro para corregir el paso que este generando inconvenientes.</div>
<div>
<br /></div>
<div>
Nos vemos el 31 de Enero en el bogodev.<br/><br/></div>
<div><strong>Update 27 de Enero:</strong><br/></div>
<div>Ya que en un punto del taller usaremos una llamada al server por REST, conviene tener instalado el Chrome con la extensión "REST Console" o firefox con una extensión similar.</div>Crosshttp://www.blogger.com/profile/17985383212473053331noreply@blogger.com0tag:blogger.com,1999:blog-4053544758140296139.post-47775568783387603552012-08-22T08:18:00.000-07:002012-09-03T15:59:45.747-07:00NoSQL or RDBMS, not again!<p>No matter what, I found the same question over and over again, "should I use a NoSQL solution to my <what ever problem>, help me!", but some days ago I crossed a question that I want to discuss. To avoid any problem with the author of the question I will try to rephrase it here:</p>
<p>"Our company XYZ created a solution ABC that is working really well on our production environment, at this moment we address thousands a records a month and 1 terabytes of information is created each month, our current architecture is <monster server here> and we're happy with our current performance, "...</p>
<p>If you're developer you will know at this point that the Universal developer rule should be applied here: "If it's not broken, please don't fix it".</p>
<p>He continues... "Our application follows the model of write once read many, because we do business intelligence to scrap out our data and get information from there, most of our queries uses 2 or 3 joins, and we don't have weird data types, just plain numbers, texts and dates, our system will get more data and more users and we're worried that our current platform will not support this new workload, that's why we want to do the move to a NoSQL system, which one do you recommend for our problem?"</p>
<p>Here's where the things are getting nasty, and I think I need to clarify that I'm a developer too, and I'm in love with NoSQL strategy, that's why I created <a href="http://djondb.com">djondb</a> because I'm convinced that "One size does not fit all" applies to the RDBMS world, 4 years ago the only way to store information in an organized way was RDBMS and several people think that they should apply RDBMS to every single problem, and that's not true. But this same principle applies to the NoSQL world, take a look of <a href="http://en.wikipedia.org/wiki/Law_of_the_instrument">Maslow's hammer law</a>.</p>
<p>NoSQL is a really good solution to several data environments, but you need to ask yourself if you're aware that most of your code should change if you decide to move from one solution to other, it is like saying that you're going to recreate your application in Java because it has better performance according to <put your link here>, you need to know that this kind of changes will require a lot of work to achieve the same features you already have, so you need to evaluate if this is going to work as a benefit or you will have a lot of work to achieve the same thing.</p>
<p>To illustrate the SQL - NoSQL transition I will use a sample, the architecture of the sample could be very different according to the approach you decided on your application (EJBs, Spring, LINQ, etc) and that will change the way you coded your solution, but I will go with a very straightforward and well know solution.</p>
<h2>Data Model comparison</h2>
<p>Most of you are already aware of the differences between one model and the other, but here's a very simple sample to illustrate this.</p>
<p>In a ERP data model you will have the following tables:</p>
<div style="text-align: center">
<img src="http://www.databaseanswers.org/data_models/erp/images/erp_top_level_dezign_v3.gif" width="100%" />Taken from http://www.databaseanswers.org</div>
<p>This is a very simple model, a customer order a product and you register the identification of the customer and product into the table "Orders", each order is related with one or multiple transactions, etc.</p>
<p>From this simple model it's very easy to assume some of the SQL statements you should use to store or retrieve elements from your DB, let's summarize some simple operations:</p>
<ul>
<li><b>Create products:</b>
<pre><code>Insert into Products (product_code, product_category_cod, description, vendor_code) VALUES ('00010101', 'E_01', 'iPad', 'Tablet blah blah', 'VEN_0010')
</code></pre>
</li>
<li><b>List of products by category:</b>
<pre><code>Select product_code, description from Products where product_category_cod = 'E_01'
</code></pre>
</li>
<li><b>Show the transactions ordered by a user:</b></br>
To retrieve this you will need to execute some SQLs or you will get a bunch or repeated information, first you will need to get the order "header":
<pre><code>Select o.order_id, c.Customer_name, c.Customer_last_name, c.Customer_username, o.order_date, ... from Orders o inner join Customer c on c.customer_id = o.customer_id where c.username = 'JACKTHEREAPER'
</code></pre>
Now you will retrieve the details of the order using:
<pre><code>Select p.description, o.price, o.units, ... from Orders o inner join Products p on p.product_code = o.product_code where o.order_id = 'XR1210120'
</code></pre>
</li>
</ul>
<p>At this point I think you got the idea, no matter if you use some ORM, persistence framework, etc at the end your application will execute these SQLs to retrieve or store information into the database.</p>
<p>Let's see the same data model from NoSQL perspective, one of the key decision process about using NoSQL is to denormalize your data in a way that store it and retrieve it will be align with how the application uses it, instead of forcing a row/columns structure.</p>
<p><b>Questions to be solved by your NoSQL model</b></p>
<p>What will be benefit of storing the order detail as a separate entity? is the order detail an entity? or just an aggregation values that compounds an order? An order does not have any meaning without the details and they are useless if are not tight to an order. This will mean that the order should have a structure similar to this:</p>
<pre><code>{
{ Order header info }
{ Order details }
}
</code></pre>
<p>Now lets figure out what are we going to store at the header info, 90% of the queries you will do to retrieve the order header will include the name of the customer, and his id, it's going to be pointless to show the order in the web application without this vital information, it's going to be easier to ignore information than adding new required information, so I will store the denormalized data from the customer inside the order, like this:</p>
<pre><code>{
{
order_id: "XR1210120",
customer_id: "CI_29120",
customer_name: "Jack",
customer_lastname: "Reaper",
date: "2012/07/26"
}
{ Order details }
}
</code></pre>
<p>Note: "Hey!, what are you doing? what is going to happen if the customer changes his name?", how many times you face this kind of problem? anyway, if the customer changed his name and he comes back in 10 years, the invoice should be printed as it was at the time of the sale, because that's the way it's stored at the IRS, or any other state department, right?</p>
<p>What about the order details? same approach, we could save the product name along with the order and you will have everything you need at the time of printing/showing the order to the user. What about the product price? same thing, you will need to save the price at the time of the purchase, so it will be better to keep it within the order and not in an inventory table.</p>
<p>Here's an order with full details:</p>
<pre><code>{
"order_header": {
"order_id": "XR1210120",
"customer_id": "CI_29120",
"customer_name": "Jack",
"customer_lastname": "Reaper",
"date": "2012/07/26"
},
"Orderdetails": [
{
"product_code": "00010101",
"Category_code": "E_01",
"description": "iPad",
"price": 450,
"units": 1
},
{
"product_code": "00010203",
"Category_code": "E_01",
"description": "Xbox 360",
"price": 320,
"units": 1
}
]
}
</code></pre>
<p>Let's do the same operations over <a href="http://djondb.com">djondb</a>, using the djon-shell:</p>
<ul>
<li><b>Create products:</b>
<pre><code>insert('sampledb', 'products', { product_code: '00010101', category_code: 'E_01', name: 'iPad', description:'Tablet blah blah', vendor_code: 'VEN_0010'});
</code></pre>
</li>
<li><b>List of products by category:</b>
<pre><code>find('sampledb', 'products', '$"category_code" == "E_01"');
</code></pre>
</li>
<li><b>Show the transactions ordered by a user:</b></br>
<p>Now, you'll only need one round trip to the database to retrieve all the information you will need to show the order:</p>
<pre><code>find('sampledb', 'orders', '$"user" == "JACKTHEREAPER"');
</code></pre>
done!, you will get something like:
<pre><code>[{
"order_header": {
"order_id": "XR1210120",
"customer_id": "CI_29120",
"customer_name": "Jack",
"customer_lastname": "Reaper",
"date": "2012/07/26"
},
"Orderdetails": [
{
"product_code": "00010101",
"Category_code": "E_01",
"description": "iPad",
"price": 450,
"units": 1
},
{
"product_code": "00010203",
"Category_code": "E_01",
"description": "Xbox 360",
"price": 320,
"units": 1
}
}
},
{
"order_header": {
"order_id": "NP112021",
"customer_id": "CI_29120",
"customer_name": "Jack",
"customer_lastname": "Reaper",
"date": "2012/07/28"
},
....
]</code></pre>
<p>This result is a JSON, very simple to iterate and interact to, but the idea of this article is not to convince you to use NoSQL, but to help you understanding the difference between your RDBMS application approach and the NoSQL approach, and highlight how these approaches will lead you to change some key points in your architecture.</p>
</li>
</ul>
</p>
<p>Now that you got the idea of how the storage is mapped from one side to the other, it's time to take a look of your code.</p>
<h2>Code comparison</h2>
<p>Until now, the things had been very straightforward, and it's very likely that you have some sentences executing SQLs similar to the ones I showed, but the code is something else, it's hard to know how every architect created their designs, but I will use a commonly known design pattern, known as <a href="http://java.sun.com/blueprints/corej2eepatterns/Patterns/DataAccessObject.html">"DAO" (Data Access Object)</a> to illustrate my point.</p>
<p>The DAO pattern uses a class to retrieve the information from the database and create a Transfer Object that is going to be returned to the business layer, if you want to store information of an order it will work like this:</p>
<p><b>Note:</b> I will do the assumption that you used jquery in your front-end and it will be able to create a full order in one step (header and dome details) and then bulk it to the server to be stored.</p>
<p><b>Note:</b> My second assumption is that you have some json framework at java that will be able to handle json nicely.</p>
<table>
<tr><td><b>Who?</b></td><td><b>What to do?</b></td></tr>
<tr><td>Web Controller</td><td>Catches the data from the "request" object and puts it in a Value Object (POJO, POCO, etc)</td></tr>
<tr><td>Model</td><td>Receives the Value Object and perform validations, etc</td></tr>
<tr><td>Model</td><td>Calls a DA class to store the VO</td></tr>
<tr><td>DA Class</td><td>Creates the required SQL to store the values from the VO</td></tr>
</table>
<p>The code (using java) will look something like this:</p>
<p><b>Web Controller: Catches the data from the "request" object and puts it in a Value Object (POJO, POCO, etc)</b></p>
<pre><code>Order o = new Order();
JSONObject json = JSONParser.parse(Request.getParameter("order"));
o.setCode(json.getString("order_code"));
o.setCustomerId(json.getString("customer_id"));
o. ... (all the order header information)
for (JSONObject detail : json.getJSONArray("Orderdetails")) {
OrderDetail d = new OrderDetail();
d.setProductCode(detail.getString("product_code"));
d.setUnits(detail.getInt("units"));
...
o.addDetail(d);
}
</pre></code>
<p>Now, you will get this and put it on a Model Class (following the MVC pattern)</p>
<p><b>Model: Receives the Value Object and perform validations, etc</b></p>
<pre><code> Model m = new Model();
m.setOrder(o);
m.saveOrder();
</pre></code>
<p><b>Model: Calls a DA class to store the VO</b></p>
<pre><code> MyDAClass da = new MyDAClass();
da.insert(o);
</pre></code>
<p><b>DA Class: Creates the required SQL to store the values from the VO</b></p>
<pre><code> StringBuffer sql = new StringBuffer();
sql.append("INSERT INTO Order (Order_code, customer_id....) ");
sql.append(" VALUES ('" + o.getOrderCode() + "', ...");
Statement stmOrder = connection.createStatement();
stmOrder.execute(sql.toString());
// at this point you will need to do the same for each order detail:
for (OrderDetail detail : o.getOrderDetails()) {
StringBuffer sqlDetails = new StringBuffer();
sql.append("INSERT INTO OrderDetail (Order_detail_code, Order_code, product_code....) ");
sql.append(" VALUES ('" + detail.getOrderDetailCode() + "', ...");
}
</pre></code>
<p>I know, you could use a preparedStatement here and use parameters, etc... or you could use EJB or ORM to store the VO, but here's not where you will see the difference between NoSQL and RDBMS approaches.</p>
<p>Now, let's take a look of the same code using <a href="http://djondb.com">djondb</a> as NoSQL repository, I will keep the same MVC and DAO concept to keep the code side by side comparable.</p>
<h2>Same code NoSQL approach</h2>
<p><b>Web Controller: Catches the data from the "request" object and puts it in a Value Object (POJO, POCO, etc)</b> at this point you will not need to map your data to a VO, it's pointless, you could use JSON at server side and it will work really well.</p>
<pre><code>
JSONObject order = JSONParser.parse(Request.getParameter("order"));
</pre></code>
<p>Now, you will get this and put it on a Model Class (following the MVC pattern)</p>
<p><b>Model: Receives the Value Object and perform validations, etc</b>, this does not have any change:</p>
<pre><code> Model m = new Model();
m.setOrder(order);
m.saveOrder();
</pre></code>
<p><b>Model: Calls a DA class to store the VO</b>, same thing the model does not suffer major changes.</p>
<pre><code> MyDAClass da = new MyDAClass();
da.insert(o);
</pre></code>
<p><b>DA Class: Creates the required SQL to store the values from the VO</b></p>
<pre><code> connectionWrapper.insert("ERP", "Orders", order);
</pre></code>
<p>Just one line will do the trick, same with the controller, mainly you are avoiding the overhead of doing unnecessary transformations, you will get the json and store it directly to your database, no more additional processing will be needed here.</p>
<p>The same coding sample could be used to "listing orders" or "showing an order", you will see that you coded a lot less using JSON as transport and persistance mechanism</p>
<h2>Conclusion</h2>
<p>Although I tried to put it very simple, converting an application from RDBMS to NoSQL is not a trivial task, you will need to spend time creating a new design of your application, yes... you will get a lot from this, you will be able to create new features easily, you will be able to grasp the performance of NoSQL in your application, but you should not think that just putting a NoSQL behind will do the magic and boost your performance, you actually need to start thinking on "NoSQL" data model and new ways to code your application to get most of the NoSQL world.</p>Crosshttp://www.blogger.com/profile/17985383212473053331noreply@blogger.com7tag:blogger.com,1999:blog-4053544758140296139.post-38358680718085936142012-01-11T11:40:00.000-08:002012-01-11T11:40:15.960-08:00djondb - Progress of my own NoSQL DBThe last two months has been really good, my own NoSQL Server is going like a skyrocket, and I'm close to finish the Milestone 2, here're the milestones I created 3 months ago:
<h2>Milestone 1 version 0.1</h2>
<h3>Basic features</h3>
<ul>
<li>Allow inserts</li>
<li>Allow updates</li>
<li>Allow finds by key</li>
<li>Shell</li>
<li>PHP Driver</li>
<li>C++ Driver</li>
</ul>
<h2>Milestone 2 version 0.1</h2>
<h3>Complete basic features</h3>
<ul>
<li>Finds by any filter</li>
<li>Arbitrary Index creation</li>
<li>Backup / Restore</li>
<li>Java Driver</li>
<li>C Driver</li>
<li>.NET Driver</li>
</ul>
<h2>Milestone 3 version 0.1</h2>
<h3>Sharding</h3>
<ul>
<li>Transactions</li>
<li>Sharding</li>
</ul>
<h2>Milestone 4 version 0.1</h2>
<h3>Nice to have</h3>
<ul>
<li>Authentication</li>
<li>Clustering</li>
</ul>
<P>At this moment the db is fully functional and I'm doing a demo, which is a craiglist like site to demostrate why NoSQL is a nice technology for fast development.</P>
<H2>What's next?</H2>
<P>At January 26 I will be at @hubbog doing a speech showing an overview of what is NoSQL, what is it good for (and what is not), and some demos about how a NoSQL (primary djondb) improves the development cycle, and let finish your projects in no time</P>
<P>For the next months I will complete the Milestones 3 and 4, in the meantime I hope to get some help to develop drivers, sites around djondb, etc.</P>
<H2>Going open source?</H2>
<P>This is a great question, and I dont know the answer yet, I always wanted to share and give something to the world, but I'm not that big to avoid problems with someone else copying my code and using it for their own benefit, that's why I will wait until I finished all the milestones and upload the first version to the web to make this decision.</P>Crosshttp://www.blogger.com/profile/17985383212473053331noreply@blogger.com3tag:blogger.com,1999:blog-4053544758140296139.post-41397957664201551872012-01-11T07:59:00.000-08:002012-01-11T07:59:22.500-08:00Building a protocol<h1>Introduction</h1>
<P>When I started to rewrite the save file algorithm for djon time tracker, I searched what will be the best way to do this, and I found a lot of people saying XML is the way to go, but everytime I tried this the files got very large and the time to read and write was huge, that's why I started to look at the binary formats.</P>
<P>The binary formats are simple files where you store bytes instead of chars, straighforward definition, but what is that? how do I know that a string started at some point, or I have an integer, when I read it the only think I see is HEX values? Here's where protocols are useful.</P>
<P>from wikipedia:
<i>"A communications protocol is a system of digital message formats and rules for exchanging those messages in or between computing systems and in telecommunications. A protocol may have a formal description."
</i></P>
When you write your own protocol you should define an unique way to write and read every single piece of data, then you have to follow the rules to read and write based on the protocol you defined.
<h2>Let's build our own protocol.</h2>
<P>Let's say that you're going to save or transmit over a wire the data of a customer:
<P>Customer
<ul>
<li>Name</li>
<li>Last Name</li>
<li>Birth Date</li>
<li>Salary</li>
</ul></P>
<P>First, we need to define the type of each data:</P>
<P>
Customer
<ul>
<li>Name: chars</li>
<li>Last Name: chars</li>
<li>Birth Date: date</li>
<li>Salary: integer</li>
</ul></P>
<P>Now we define a unique set of rules to write a Customer:</P>
<P>Data Order: the data will follow the same order everytime (Name, Last Name, Birth Date, Salary)<br/>
Labels? if we have a fix set of data (like the example above) it's useless to name each piece of data, so we will avoid this and save space</P>
<P>Now we need to define how to save each type, let's start with the easy one.</P>
<h2>Integer</h2>
<P>In C an integer is a 2 bytes length data, that means you will have a 2 chars to store. Ex: 65000 as a salary will be FDE8 (2 chars, FD and E8 which could be translated to char:253 and 232).</P>
<P>Let's write some code here, and save an integer in the simplest way:</P>
<pre><code>
void saveInt(int a) {
FILE* f = fopen("test.dat", "wb");
fwrite(&a, 1, sizeof(a), f);
fclose(f);
}
</code></pre>
<P>This code works well and it's very straightforward, but it has a big problem. It will write the 2 bytes (from the example above: FD and E8) in an unknown order, could be E8FD or FDE8 depending on the architecture of the machine it runs, this means that if the architecture of the machine where you're going to read the file changes you will get a very different result. That is called Little/Big Endian problem. To fix this we will ensure that the order will be the same all the time, this is done using the following code:</P>
<pre><code>
void writeInt(int a) {
FILE* f = fopen("test.dat", "wb");
unsigned char c = (a & 255);
fwrite(&c, 1, 1, f);
unsigned char c2= ((a >> 8) & 255);
fwrite(&c2, 1, 1, f);
fclose(f);
}
</code></pre>
<P>This code will ensure that the order will be same everytime, and it will not depend on the architecture of the machine it runs. Let's break down this instructions:</P>
<pre><code>
unsigned char c = (a & 255);
</code></pre>
<P>If you have an integer of 65000 (FDF8) it will do an "and" operation with 00FF, this will "erase" the higher byte:</P>
<pre><code>
FDF8
And 00FF
====
00F8
</code></pre>
<P>The next instruction will do a similar operation, it will move the bytes from right to left and erase the higher part: </P>
<pre><code>
unsigned char c2= ((a >> 8) & 255)
FDF8 >> 8 = XXFD
XXFD & 00FF = 00FD
</code></pre>
<P>With this simple method (called Little Endian) we will ensure that the write will be always in the same order, now the read will be easy:</P>
<pre><code>
int readInt() {
FILE* f = fopen("test.dat", "rb");
unsigned char c;
fread(&c, 1, 1, f);
unsigned char c2;
fread(&c2, 1, 1, f);
int res = c & (c2 << 8);
fclose(f);
return res;
}
</code></pre>
<P>c2 will be FD and c will contain F8 doing the "<< 8" the FD will go up and adding will result in FDF8 (the original number)</P>
<P>Now that we solved the big issue, the other things are easier.</P>
<h2>Strings</h2>
<P>One of the main issues with strings is how to deal with the length of the string, one possible solution could be to put a fixed char at the end of the string and read until reach that character.</P>
<h3>Strings solution 1</h3>
<pre><code>
void writeString(char* c, int len) {
FILE* f = fopen("test.dat" "wb");
for (int x = 0; x < len; x++)
fwrite(&c[x], 1, 1, f);
char end = '*';
fwrite(&end, 1, 1, f);
fclose(f);
}
char* readString() {
FILE *f = fopen("test.dat", "rb");
char c;
char buffer[256];
int pos = 0;
do {
fread(&c, 1, 1, f);
if (c != '*') {
buffer[pos] = c;
pos++;
}
} while (c != '*');
buffer[pos] = '\0'; // terminated-string
fclose(f);
return buffer;
}
</code></pre>
<P>This solution works pretty well, and it can be improved using stringstreams or others, but it has a big problem, what if the original string contain the character '*' in between? change it for other char? what will be the odds that character is included too? This could be easily fixed if you write down the size of the string and then the content of the string, and you read it in the same way, first the length and then the contents.</P>
<h3>Strings solution 2</h3>
<pre><code>
void writeString(char* c, int len) {
writeInt(len);
FILE* f = fopen("test.dat" "wb");
for (int x = 0; x < len; x++)
fwrite(&c[x], 1, 1, f);
char end = '*';
fwrite(&end, 1, 1, f);
fclose(f);
}
char* readString() {
int len = readInt();
FILE *f = fopen("test.dat", "rb");
char c;
char* result = (char*)malloc(len+1);
for (int x = 0; x < len; x++) {
fread(&c, 1, 1, f);
if (c != '*') {
result[x] = c;
}
};
result[len] = '\0'; // terminated-string
fclose(f);
return result;
}
</code></pre>
<P>Solved! (off course you could change the methods to open the file, do all the operations and then close it, these were written this way to avoid complexity)</P>
<P>Now, the main code:</P>
<pre><code>
write(Customer c) {
writeString(c.name());
writeString(c.lastName());
writeDate(c.birthDate()); // I will let this to the reader
writeInt(c.salary());
}
Customer read() {
Customer c;
c.setName(readString());
c.setLastName(readString());
c.setBirthDate(readDate());
c.setSalary(readInt());
}
</code></pre>
<P>This solution could be applied to network transmission, files, or anything you want. You could translate this solution to other languages.</P>Crosshttp://www.blogger.com/profile/17985383212473053331noreply@blogger.com0tag:blogger.com,1999:blog-4053544758140296139.post-52434088812657585462012-01-03T20:31:00.000-08:002012-01-03T20:31:01.516-08:00Apache error 403: Forbidden<p>I'd run into an annoying problem that is very easy to solve, once you know the answer... as usual.</p>
<p>I was starting a simple project to demostrate how to create an application using djondb as a NoSQL db and wrote some simple pages emulating the famous craigslist page, but as soon as I added the "<Directory>" directive to the apache server started to get the message:</p>
<pre><code>
Forbidden
You don't have permission to access /demo2/temp.html on this server.
</code></pre>
<p>Run to google... do some searches and all of the results pointed to file permission problems, I just went to the console and run the chmod a+rwx (I know.. it's not secure, but it's a demo pc), restarted apache and... puff.. the error kept popping up, read more... did some changes... and nothing the error persisted, then I crossed to a post that was really helpful <a href="http://stackoverflow.com/questions/6807317/fixing-403-forbidden-on-alias-directory-with-apache">Fixing 403 Forbidden on alias directory with Apache</a> one of the answers suggested to login using the apache user, and try to navigate to the file.</p>
<p>As soon as I did that I realized that my folder was: /home/cross/workspace/db/demo... etc... and I changed the permissions to the "demo" directory, but not to all the parent folders (workspace/db) and that was causing the problem. I added my private group (cross) to the user www-data (the user used to start apache) and now everything is working.</p>
<pre><code>usermod -a -G cross www-data</code></pre>
<p>Easy? yes... I know it is but I want to share this "enlightened knowledge" in case you crossed with the same problem. Took 1 hour of my time to solve this, wondering why on earth the test application worked fine with the PC at my office (ubuntu 10.10) and didn't at my home PC with ubuntu 11.10. (Actually I still wonder why I didnt run into this problem at my office... the path is the same) </p>Crosshttp://www.blogger.com/profile/17985383212473053331noreply@blogger.com0tag:blogger.com,1999:blog-4053544758140296139.post-12743248645055113122011-11-02T07:26:00.000-07:002011-11-02T13:13:17.845-07:00d-jon is going to the open source world!After 1 year of hard work I decided to open the code to the world, why? easy because I always wanted to give something to the community, I used several tools based on open source and I think it's good to give something back.<br />
<br />
<h2>How d-jon started?</h2>
<br />
2 years ago I was working on the java version of Bizagi (<a href="http://www.bizagi.com">http://www.bizagi.com</a>) and the team started to use TSP (Team Software Process) in order to develop the new rendering module, one of the key features of TSP is the ability to measure the time spend on every single task you do on your job, that's how I started to use several linux tools to track my time and all of them were unsuccessful, most of the tools because they're unable to track if I'm working on my desk or leave the office soon, if I forgot to stop the timer when I decide to take my lunch then I had to get back and pinpoint at what time I leave the office, same happens when my boss (or a coworker) ask me a question about some other subject, when I come back to my desk I did not know how much time I spend assisting somebody else, one of the tools lost all my records without any warning, etc etc... I got tired of this and decided to create a simple tool to track my time.<br />
<br />
Some months later I realized that my tool was getting stronger, and stronger, so I decided it worth the effort to put d-jon on the web, that's how I did the first refactor of all the code to create a less buggy application, at first I was the only user so I can deal with some bugs, but now that the application is going to be on web I have to fix them.<br />
<br />
Vision Software, the company I have been working for 7 years, decided that this tool was good enough to replace the Allnetic solution that was buggy and has several issues. 1 year later d-jon has over 10.000 downloads and it is been used in several Countries, I'm pretty happy that it filled the gap between professional solutions and the free ones.<br />
<br />
<h2>Why going open?</h2>
<br />
First of all, one of the most difficult tasks is to get some feedback from the users, not because they don't want, just because there is a lot of spammers out there willing to put porn and viagra ads in every site they can, I started a forum and tried to keep the spammers out of it but was a futile war, they won... crash and burn me, I do not have the time to deal with web issues, and be a policeman of every post on my site, that's why I closed the forum, I had to spend too much time deleting spam messages, then figuring out how to keep the spammers out, etc... I prefer to use that time to improve the application and not to keep these unwanted people out of my life,<br />
<br />
So, what if the users do the extensions and fit the tool to their expectations? that's why I decided to open the source code, to let the users improve it and get the best from the tool.<br />
<br />
<h2>What's comming?</h2>
<br />
I need to do several things before publishing the code, like license to be used, put all the documentation about the code, create a standard structrure (make and all of that stuff), etc etc... but I thing it worth the effort.<br />
<br />
<br />Crosshttp://www.blogger.com/profile/17985383212473053331noreply@blogger.com0tag:blogger.com,1999:blog-4053544758140296139.post-72854121688891522452011-10-23T11:20:00.000-07:002011-10-23T11:20:04.663-07:00My own NoSQL DB, progress<br />
The last few weeks have been good... hard... but good, I finally implemented some new layers in the DB, some of them were harder than others, and i realized that I dont know that much of some bare metal components like network layers as I thought I know, but anyway I managed to pass through all of this and the db is now running smoothly, this post is to write down some of the progress made and explain some of the problems I had implementing these new layers.<br />
<br />
<span class="Apple-style-span" style="font-size: large;">Command Layer </span><br />
<br />
This component is really one of the cores of the communications with the user, basically it's meant to process the commands made by an external player and figures out what is the request, how to handle it and what is expected to be returned to the client.<br />
<br />
This component was not really a problem, basically because I already knew how to handle the network protocol, then I wrote a writer and a reader and that's it, the command is alive!.<br />
<br />
<span class="Apple-style-span" style="font-size: large;">Network Layer</span><br />
<br />
This component was really a hard ball, I didn't know that synchronize the sends with the reads will be that hard, this component drove me crazy for some days, until I realized that I had the network layer does not sign a contract with you to send exactly what you put on the pipe, thus is not suppose to get the expected length at the end of it. Let me put it this simple:<br />
<br />
Client A writes 1000 bytes on the pipe.<br />
<br />
Server expects 1000 bytes...<br />
<br />
But... that does not happened, the client actually sends some chunks of data, 100 bytes at first, 500 later, 200... etc. Thus the server should expect to read this mess in the same way, now that I read some literature about it (that's why it took me 1 week to implement this layer) I know that it's easy... but this layer was a real pain (if you have questions about this, just let me know and I will try to explain how to create a reliable server/client communication), I still to do some improvements in order to get better performance, but anyway it's working right now and I dont want further delays in this project.<br />
<br />
<span class="Apple-style-span" style="font-size: large;">Index implementation</span><br />
<br />
Here's the tricky part of the databases, figure out how to create an efficient way to sort the data in order to retrieve ii later, in this part I tried to avoid the conventional methods, and try to figure it out from my own knowledge... big mistake!... hahahaha, anyway this tries outs were good to understand why the people came out with fancy solutions like B+Trees, BTrees, etc. here's the BUT... I read documentation on B+trees and it didnt fit well on what I think should be the separation between data and indexes, so I had to create my own solution to this problem, of course... based on B+trees knowledge and some of my tries outs.<br />
<br />
Now, the tests runs very well... the db gets the following output:<br />
<br />
20000 inserts per secs and 29978 finds by key operations per sec.<br />
<br />
The thing with the network layer is not going that well, but I will fix this later, with the network layer on the output is: 8695 ops, close to the achieved output of MongoDB which makes me very happy for now. I know that when the time comes I will fix the network issues and that will be a nice boost for my server.<br />
<br />
Later I will create some posts of my learning process and what problems I had to solve in order to keep on with this project.<br />
<br />
The project keeps looking good, and I will keep going on.<br />
<br />Crosshttp://www.blogger.com/profile/17985383212473053331noreply@blogger.com0tag:blogger.com,1999:blog-4053544758140296139.post-71878905597611361892011-09-22T13:10:00.000-07:002011-09-29T15:54:16.697-07:00My own NoSQL DB15 days ago I read this article: <a href="http://blog.cubrid.org/dev-platform/nosql-benchmarking">http://blog.cubrid.org/dev-platform/nosql-benchmarking/</a> and got shock with the amazing performance, c'mmon! 21k operations per second? it's amazing...<br />
<br />
After a long bath I started to wonder, how much performance will I achieve if I create my own NoSQL database, just an experiment for those rainy days, ok... here I am 1 week later after some hard work, I created my own NoSQL Database which is capable of:<br />
<br />
1. Insert Records<br />
2. Find records using the key (that's the benchmark)<br />
3. Index the keys and whatever I want to index to find records later.<br />
4. Split files to get indexes separated from data.<br />
5. Created a layer to save the data using little endian (basically I copied the code from my other project d-jon)<br />
<br />
Here's the test I did, with a modest machine and 8Gb of RAM:<br />
<br />
1. 1M inserts of 1k (I know, the bench was over 50M, but I dont have free space on my machine right now)<br />
2. A random search using random keys (300.662 searches)<br />
<br />
And here are the stats:<br />
<br />
the inserts took 23.48 secs, which means 42.584 operations per sec.<br />
The searches took 20.3 secs, this means 14.763 operations per sec.<br />
<br />
These results are preliminary, but I'm starting to think that I could beat MongoDB, Cassandra, and others, <br />
<br />
I started my project because I was out of challenges, and it's turning to be a very attractive project, maybe this could be my next fulltime job.<br />
<br />
A lot of work coming, but what the hell... djon started just like that and now have over a 3k of downloads and users all over the world.Crosshttp://www.blogger.com/profile/17985383212473053331noreply@blogger.com1tag:blogger.com,1999:blog-4053544758140296139.post-72497487696058441402011-08-17T15:31:00.000-07:002011-08-17T15:31:15.411-07:00C++11 C++x0 new C++ is here!At last, after almost 3 years the new standard has become a reality, this new standard has a lot of cool features that I will try to cover within my next posts.<br />
<br />
Follow the link for <a href="http://www.open-std.org/jtc1/sc22/wg21/">The C++ Standards Committee</a> if you want to get a "light" read and have an understanding of what's going on with C++11. Or just keep connected because I will try to explain the major changes about the standard.<br />
<br />
Crosshttp://www.blogger.com/profile/17985383212473053331noreply@blogger.com0tag:blogger.com,1999:blog-4053544758140296139.post-51152452346562194522011-04-21T21:09:00.000-07:002011-04-21T21:09:35.134-07:00Almost completedOk, my game is almost complete, at least the main idea, interaction and elements required (graphics, sounds, gameplay, etc), during the last 3 hours I created the level system, an explosion animation (I didn't find an animation that suites my needs so I need to create it from scratch using gimp and imagination), I also added sounds.<br />
<br />
In Addition, my tester (my 4 years old son) did a test and I think it worked as expected, everytime he see me on the computer ask to play the game again and again... the magic is done!<br />
<br />
Now I need to work on the following things:<br />
<br />
1. The health of the UFO<br />
2. The health of the player<br />
3. The fight back from the UFO (easy with the engine I created)<br />
4. An score system.<br />
5. More sounds.<br />
<br />
At this rate I think I will finish my first game in 1 week (or less if I have time during my vacations).<br />
<br />
I'm very happy with the current results and I will share some demos of the code (not the one used in the game, but I will manage to create some demos of how I used SDL in my first game).Crosshttp://www.blogger.com/profile/17985383212473053331noreply@blogger.com0tag:blogger.com,1999:blog-4053544758140296139.post-2438166297047034802011-04-19T22:58:00.000-07:002011-04-19T23:00:47.840-07:00Week 2 ResultsThen game is going pretty well and Im starting to love this new career, the results are very good (from my limited and biased point of view) and I want to share the results that I got till now,<br />
<br />
Game Engine:<br />
<br />
The main idea of a game engine is to create something that you could reuse easily on other projects and avoid the creation of the code from scratch everytime, my 2d game engine has the following features:<br />
<br />
<ul><li>Image loads (maps, sprites, etc)</li>
<li>Movement (gravity and simple movement from left to right)</li>
<li>Event managed in a proper way</li>
<li>main loop routine</li>
<li>Collision detection</li>
</ul><br />
So far this is a very good result for the second week of work. But I didn't start this from scratch, the www.sdltutorials.com was a very good help in order to understand where I needed to start from.<br />
<br />
Game Content:<br />
<br />
This was a little bit difficult, I changed my mind several times about how the game will be, cause everytime you see the screen you want something different, it's hard to keep the original idea because you see how the gameplay will be and then you will change the way you thought it's going to work at the first place, but anyway... that's the idea, improve, adapt and get the best from the game.<br />
<br />
Most of the spend time was trying to figure out how to create the images you need to put on screen, I tried to create them from scratch but was a big failure, that's why I decided to google a little bit and found some images that I could modify to create the background of my game, also the ufo was taken from a picture found at google, removed the background and get the ufo out of the picture, the cannon was harder but I found a nice place with tons of sprites that I could use (I will post his link in here later), the missil was created from scratch using inkscape (very good tool).<br />
<br />
The mathematics was a little bit tricky but finally I managed to create the cannon effect of following the ufo and also the missil will goes into the direction of the UFO's center.<br />
<br />
The results of the second week were saved in a video <br />
<br />
<iframe title="YouTube video player" width="480" height="390" src="http://www.youtube.com/embed/i9q1hYMV8Qo" frameborder="0" allowfullscreen></iframe><br />
<br />
Hope you liked the progress and if you have any questions I will be glad to answer them.Crosshttp://www.blogger.com/profile/17985383212473053331noreply@blogger.com0tag:blogger.com,1999:blog-4053544758140296139.post-57113725062196447152011-04-19T19:39:00.000-07:002011-04-19T19:51:22.805-07:00Shoot the missil: Maths in action!I came across a problem these days, how can I send a bullet right into the direction of another target, here's when you need to get back into the school and use your Pythagorean theorem knowlegde,<br />
<br />
After spending some hours trying to solve a simple equation I realized that I've to get back to the math books, I found a very good discussion about this subject in <a href="http://gamedev.stackexchange.com/questions/2029/what-math-should-all-game-programmers-know">yahoo answers</a> what you need to know about maths in order to create games.<br />
<br />
Ok, I wasted 2 hours into this equations and then remembered something (very blur btw) about vectors, how they should be used to move a point into the direction of a line (vector), that's how I crossed with this: <a href="http://www.gamedev.net/topic/600086-find-point-between-2-points/">http://www.gamedev.net/topic/600086-find-point-between-2-points/</a> actually the problem was very easy to solve:<br />
<br />
<pre><code>
p1x=1;
p1y=1;
p2x=4;
p2y=4;
dirx=p2x-p1x;
diry=p2y-p1y;
p3x = p1x+dirx*0.25;
p3y = p1y+diry*0.25;
</code></pre><br />
First you need to find the direction of the vector, you get that doing a substraction between the 2 points (x2, y2) - (x1, y1), now that you have the direction you should apply it to the new point and that's it. easy right?<br />
<br />
Now my game has a beatiful rocket moving into the direction of the UFO, one step further in the game creation.Crosshttp://www.blogger.com/profile/17985383212473053331noreply@blogger.com0tag:blogger.com,1999:blog-4053544758140296139.post-40143129425100998902011-04-14T13:57:00.000-07:002011-04-14T14:40:00.506-07:00Week 2: The engine is ready and I'm working on the graphicsThe last week was amazing, I thought was going to be harder to create a game, but thanks to sdltutorials.com the engine was created in 2 days, and I started to work on the character interactions.<br />
<br />
The first thing I noticed is that the game is not going to be what I started with, actually is very different from the sketch I created 1 week ago, that's because everytime you see the screen working you change your mind and want something new, at this moment the game is going to keep it's scent but several details has been changed from the original idea (things like how the cannon will shoot, when will be the user dead, etc etc).<br />
<br />
After the plot of the game was created I started to learn I little bit about sprites, and how they worked, this is very easy subject, but the animation... that was a very different history, I tried to create my own sprites and was a great failure, that's why I started to look for some free sprites in order to avoid this work.<br />
<br />
This is how the screen looks like at this moment:<br />
<br />
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg820DukpYk2HaNyyVm8V6LrzLfa_wzoggbASmZ7ss-JE23YD5aHyzcP2VgqsdUdjjq_ggZS8Eo1Ga_C6kNVFfTAmap3ZdHgoKOOG96qlH7w_4l83nQqPsnsOh5asILQxoqu2VShyphenhyphen_HUCI/s1600/screenshot-April14.png" imageanchor="1" style="margin-left:1em; margin-right:1em"><img border="0" height="250" width="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg820DukpYk2HaNyyVm8V6LrzLfa_wzoggbASmZ7ss-JE23YD5aHyzcP2VgqsdUdjjq_ggZS8Eo1Ga_C6kNVFfTAmap3ZdHgoKOOG96qlH7w_4l83nQqPsnsOh5asILQxoqu2VShyphenhyphen_HUCI/s320/screenshot-April14.png" /></a></div><br />
The cannon follows the ufo and the ufos moves as expected. I had to remember how the angles worked and what is all about cos, sin, tan etc that I saw at school and put under the bed a very long time ago, I'm realizing that I need to study very hard mathematics, my brain has some "rusty" at this moment, but anyway I will need to get my mathematics book and put in shape again.<br />
<br />
My accomplishments this week:<br />
<br />
1. Engine created<br />
2. Basic graphics created<br />
3. Basic movement of the UFO and cannon working<br />
<br />
Next week...<br />
<br />
1. Cannon shooting<br />
2. UFO crashing and burning<br />
<br />
The main idea of the graphics are settled, and I think I will post a video on youtube with the current animation that way you will give an idea of how the animations are, anyway as always I will keep the game idea hidden.Crosshttp://www.blogger.com/profile/17985383212473053331noreply@blogger.com0tag:blogger.com,1999:blog-4053544758140296139.post-74132915060274121602011-04-05T21:44:00.001-07:002011-04-05T21:54:39.474-07:00Learning SDLI'm really impressed about how easy is to use the SDL library in order to create your games, I followed the tutorials Basic, Coordinates and Blitting, events and the tic tac toe sample from <a href="http://www.sdltutorials.com/">http://www.sdltutorials.com</a> and it's really simple and straighforward, actually I though it will be harder, but the explanation was amazing an I understood everything in minutes, the examples ran very well and I'm really thinking that this is piece of cake, the game is on its way to sucess.<div><br /></div><div>Here's the result:</div><div><br /></div><div><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEht45yjbrC6KQ2DKRMrtFdAeNP-mmE0gh7MiCNZJeljypQbh5oy75rUaj8g2geTgjnm-6uH951w-htS2MQoBiXZQMV3XFSsTkbuKa3roLaqyZZVce4Y5oPvhPG6eDE5JYsOmX5CL1eQ070/s1600/Test.png" onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}"><img style="display:block; margin:0px auto 10px; text-align:center;cursor:pointer; cursor:hand;width: 320px; height: 180px;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEht45yjbrC6KQ2DKRMrtFdAeNP-mmE0gh7MiCNZJeljypQbh5oy75rUaj8g2geTgjnm-6uH951w-htS2MQoBiXZQMV3XFSsTkbuKa3roLaqyZZVce4Y5oPvhPG6eDE5JYsOmX5CL1eQ070/s320/Test.png" border="0" alt="" id="BLOGGER_PHOTO_ID_5592329321786687410" /></a><br /></div><div>I will finish all the tutorials of sdltutorials and then I will start the game I'm planning, but I think everything will be set when the animation tutorial finishes (and that will be tomorrow), I will spend sometime understanding how the collision system works but I will not need it in my first game.</div><div><br /></div><div>After this tutorials the next step will be the graphics, that will be easier because I've some experience using graphics tools, and I dont think will face any problems in this area.</div>Crosshttp://www.blogger.com/profile/17985383212473053331noreply@blogger.com0tag:blogger.com,1999:blog-4053544758140296139.post-71845922542450885582011-04-05T13:22:00.000-07:002011-04-05T13:32:15.684-07:00Step 2: API decision OpenGL or DirectXThis step is a little hard, I need to choose what will be the API I will use to create my first game, in the future I will learn both but I need to start with something, this time I will focus my decision on the following principles:<div><br /></div><div>1. I want to learn how to work with games (both will be handy for this)</div><div>2. The game is fairly easy and will not require tons of hardware/software to run, so I don't care which one is faster/better than the other.</div><div>3. I usually in love with the open source/freedom and dont like to be pressured to use something</div><div>4. The game will run mainly on desktop machines (Windows, Linux or Mac), some of the games will use the keyboard/mouse to run.</div><div><br /></div><div>After reading this article <a href="http://blog.wolfire.com/2010/01/Why-you-should-use-OpenGL-and-not-DirectX">http://blog.wolfire.com/2010/01/Why-you-should-use-OpenGL-and-not-DirectX</a> I decided that my first API will be OpenGL, and I will try to see how to use the MVC pattern in games in order to abstract the API usage, if I can achieve to run the game using directx and opengl will be great, but my first game will be as easy as it could, that's why I will not shoot myselft if I cannot manage to run it on directx too.</div><div><br /></div><div>Ok, decision made, now is time to read a lot of OpenGL, to my advantage I have some knowledge about Qt so I could use the Qt-OpenGl interface and speed up the process, let's see how it works.</div>Crosshttp://www.blogger.com/profile/17985383212473053331noreply@blogger.com0