Matrex 1.3.5 almost ready.

Last tests for the version 1.3.5 of Matrex, which will be probably released in the last half of May.
The changes are many, but surely the most important are:

• Flexible configuration files, which allow Matrex to be installed in packages or setup files.
  
• The project diagram

together with many bug fixes.

Project diagram

Matrex version 1.3.5 will have the possibility to show the project in form of a diagram.
Here is the diagram for the example project projection, that is included in the Matrex setup:


It shows all items of the projects (but timers) and their connections.
Each item is labeled with its type (matrix, function...), each connection is displayed as an arrow.

Compared to the item trees and to the information views, the project diagram has the advantage that It gives a global overview of the project. In one single window it is possible to see all items of the project.

The meaning of the arrows is the following:

• An arrow from a matrix to a function means that the matrix is input of the function
• An arrow from a function to a matrix means that the matrix is output (result) of the function
• An arrow from a matrix to a presentation or chart means that the matrix is used in the presentation or chart.

Since a project can contain many items and therefore many connections among them items, the project diagram can become complex.
To make it easier to understand it, it is possible to:

• Click on an item: the item becomes green and all the connected items and their arrows become blue (in) and red (out).
• Click on one of the items type on the tool bar: all the items of that type become green.
  

In the tool bar there is also a button to print the diagram.

Matrex easier to package

Until now once you installed Matrex in a directory, Matrex used the same directory to:

• write its log file
• write the changes to its configuration files
• write additional templates
• write scripts used as base of templates

This is natural for a java application that is installed with a java installer and does not pretend to integrate with the platform in which it is installed.

But what if we want to create installers for specific platforms (Windows setup, MacOSX application bundles, Linux .deb and .rpm files)?
In this case:

• it is the operating system that defines in which directory or directories Matrex is installed.
• Matrex cannot write the files it produces (templates, configurations, log...) in the directories in which it is installed, but must use a writable directory, which can be under Documents and Settings in Windows or under the home directory in Linux.
  

Version 1.3.5 will make this possible with two changes:

• the new configuration file main.properties, which contains the paths of all other configuration files (which can contain environment variables).
• the new concept of configuration files that contain the locations of other files.
  

The configuration files that contain the locations of other files are:

• templates.cld, which contains the paths of the top directories of directory trees containing functions templates.
• plugins.cld, which contains the directories and jar files containing additional java classes used as plugins and their dependencies.
• scrips.cld, which contains the top directories of directory trees containing scripts used by templates.

With the addition of main.properties and of the .cld files it is possible to have the configuration files that change after the installation and additional templates, java classes and scripts in writable directories.
In this way anyone can change these files, add them or delete them without touching the original installation.

It will be then easy to install Matrex:

• In Windows in a directory under the Program Files directory, with the additional files and the writable configuration files in a directory under Documents and Settings.
• In Linux in the standard directories (/usr/bin, usr/share...), with the additional files and the writable configuration files in a directory (.matrex ?) under the home directory.
• In MacOSX in a directory under the Applications directory, with the additional files and the writable configuration files in a directory under Users

I tested this new feature installing version 1.3.5 under a read-only directory and configuring it so that it keeps all the files that need to be changed under a writable directory.
It works fine.
As soon as possible I will release a beta of this version.

ToolBarWithMenu component

The SWT ToolBar component has a minor problem: when there is no space enough to show all the buttons contained in the ToolBar, it just shows the first ones.
It is true that the SWT.WRAP property partially fixes the problem wrapping the buttons in two or more lines, but this does not work in some platforms (e.g. Linux) in which the property is ignored.
This can be a problem, because the user of the application can be completely unaware of the buttons that are not displayed.
I have seen other graphical libraries solve this problem adding an additional button at the end of the toolbar. This button when clicked shows a menu with the missing buttons.
So I adopted the same logic in SWT: I made a class called ToolBarWithMenu
, which adds a button at the start of the toolbar. The button has a menu showing the buttons that are not displayed in the toolbar because there is no space. Here is an example in Matrex:


Clicking on one of the menu items has the same effect as clicking on the related toolbar button.
It works as expected also for a button with an attached menu, if the button's selection listener that shows the menu implements the IHasMenu interface.

Matrex on MacOSX

Matrex uses the SWT library for its GUI. SWT uses the native GUI of the platform. The current version of Matrex, 1.3, uses SWT 3.4.
On MacOSX SWT 3.4 is based on Carbon, which is only 32 bits.
So Matrex 1.3 on MacOSX does not work using Java 1.6 (Java 6), which is only 64 bits and therefore not compatible with Carbon.
If you want to use Matrex 1.3 in MacOSX please either:

• run it with Java 1.5 (Java 5) instead of Java 1.6
• try the beta version of SWT 3.5 for Cocoa 64 bits. Download the zip file and unzip it in the /swt directory, where is the directory where you installed Matrex.
  

The following is the description of the SWT 3.4 problem written by the SWT developers:

SWT cannot be used with OS X JRE version 1.6 (Mac OSX only)

OS X JRE version 1.6 assumes that pointers have a size of 64 bits, but SWT's Carbon port only uses 32-bit pointers, so SWT and Eclipse cannot be used with OS X JRE version 1.6. The workaround is to use an earlier supported version of the OS X JRE.

Updated Matrex adapter to Scilab

The Matrex adapter to Scilab has been updated for Matrex 1.3.5.
The adapter has been tested with Scilab 5.1.

The adapter contains a special function template that calls Scilab functions to calculate Matrex functions.
Input matrices and parameters of the Matrex function are passed to the related Scilab functions, which is calculated. The result of the Scilab function is passed back to Matrex as output matrices of the Matrex function.

More information can be found in the documentation included in the Scilab adapter setup.

New Matrex adapter to Groovy

I released in these days the new Matrex adapter to the Groovy language.
This follows the two adapters to Jython (integrated in Matrex) and to JRuby.
This adapter is good:

• for fast prototyping and test of function templates
  
• to write more compact code than in Java
  
• to use features, for example closures, that are not available in Java

Like the other language adapters it used in this way:

1. create a Groovy script with the code for the function template
2. create the function template definition
3. use the function template to create functions in Matrex projects

More information can be found in the documentation included in the Groovy adapter setup.

Subversion

For who is interested in the Matrex code, I'm moving Matrex and all its satellite projects to Subversion (SVN).
With CVS I was always adding to the repository a lot of files by mistake and was never able to really remove them.

For more information just go to the Matrex SVN page in Sourceforge.net.

As soon as possible I will move to Subversion also Internal SQL Library and SWT Web Installer.

Plans

These are the plans for the future of Matrex, in order of time:

1. Upgrade all adapters to libraries, script languages and scientific applications to work with the version 1.3.x of Matrex, which uses a new plugins architecture. I already upgraded the Jama adapter and I'm working on the JRuby adapter. Clearly if there are new versions of the libraries, languages and applications the adapters will work with them.
   
2. New adapter to the Groovy interpreter.
   
3. New intermediate version of Matrex, probably called 1.3.5. This version will not contain major changes from version 1.3. The idea is to make possible to have configuration files and additional templates in different directories. In this way it is possible to build .deb or .rpm setup packages for Matrex and make it easier to use under Vista and MacOSX.
   
4. New version 2.0: It will be possible to use Matrex as a standalone application, like now, or as a client/server system.

The upgrade of the adapters and the Groovy adapter will be done in the next weeks.
Versions 1.3.5 will be probably finished in ~ 2/3 months.
2.0 will be finished in ~ 6 months.
Versions 1.3.5 and version 2.0 will be developed in parallel.

In the same time Braxton and me are trying to build a setup file to install Matrex on MacOSX as an application bundle.

Compatibility with Java5 : Matrex 1.3.01

I published a new release of Matrex, 1.3.01, to cope with the fact that Matrex 1.3 was able to run with Java6, but not with Java5.
The problem was a call to the String.isEmpty() method, which is only available from Java6.
In 1.3.01 this call has been replaced and this version has been tested with Java5.

Thanks to Braxton who found the problem.

Matrex 1.3 relased

So we finally made it. Version 1.3 has been released!

A summary of the changes from version 1.2 is available in the Sourceforge's release notes.
The big changes have been:

• Main View usability improvement.
  The main view is now easier to use thanks to the buttons that hide and show specific trees (for example you can hide the function tree and show the presentation tree just clicking two buttons).
  
  It is also possible to search items or folders in a tree just typing a part of its name (in red).
  
  In each tree is not possible to move a set of items to a package.
  Just select some items of a tree (also in different packages), right-click with them mouse on the tree to display the popup menu and select Move... (for example Move Matrices). With a dialog box you choose the destination package and the items gets moved.
  The system takes care of the dependencies.
  
  
• Viewers usability improvement . In the presentation viewer and matrix viewer it is now possible to display the content of presentations and matrices in the best way clicking the fit columns button.
  
  
  
  
• Generic setup. The Matrex generic setup allows to install Matrex on every platform. The idea is that, once installed, when executed the first time, Matrex downloads the correct version of the SWT library and installs it.
  Executing it once again, the downloaded library is used and Matrex works regularly.
  All thanks to the SWT Web Installer library.
  
  
• New high low charts. Added candle and high low charts.
  
  
  
• Plug-in architecture. Matrex now uses its own class loader, which allows to specify libraries (e.g. function template libraries) that need to be loaded using the Plugin Dialog:
  
  
  No more class paths to change in the batch files!
  
  
• Code quality improvement. The code is now well covered with JUnit tests and has been checked with Findbugs. The GUI has also been unit tested using SWTBot.
  

Also the experimental new feature of the callbacks has been added. It is used in the directsearch example project, which demonstrates the Nelder Mead optimization method.

As usual, feedback is precious. Please let me know if something is wrong!

SWT Web Installer and ISQL Library updated

Since I published the pre-release of Matrex 1.3, I also updated the two libraries that Matrex uses, SWT Web Installer (now 1.0) and ISQL Library (now 1.01). 

SWT Web Installer downloads now SWT 3.4 instead of 3.3.1 and has a small change to the WebInstaller class constructor (new parameter systemProperties).

I also added a test that verifies that all sites in the swt_list.xml configuration file exist and are accessible.

ISQL Library had a small bug related to the conditions of the query's where clause. That has been fixed in version 1.01.

I have checked both libraries with Findbugs before I published them. 

Matrex 1.3 pre-release

I published a pre-release version of Matrex 1.3.

You can find it under the Matrex Unstable download section in SourceForge.

The changes from version 1.2 are listed under the Matrex File Release Notes and Changelog page in SourceForge.

The idea is to make sure that if there are bugs in Matrex 1.3 they get fixed before the final version is released.

So please give a try to this version and, if you find bugs, use the tracker page in Sourceforge to report them.

The pre-release version includes only the setup for the generic architecture, which means that when you start Matrex the first time it is supposed to download the SWT library for your PC.

After that Matrex stops, so you need to start it once again to use it.

The download of SWT is a new feature, so if you have problems with it please use the tracker page to report them.

SWTBot for dialogs

I started writing JUnit test cases to test the GUI of Matrex using SWTBot.

Since I did not have any experience with GUI testing, I wanted to start with something simple.
The simplest GUI items to test in Matrex are the dialogs, since they do something specific and limited.
So I started writing test cases to test dialogs.

It turned out that testing dialogs is not so easy as it seems (and this is a general problem, not limited to SWTBot).
With SWTBot you simulate the actions of your application's user: press buttons, write in text boxes, select from list and check boxes...
But you cannot simply open a dialog and simulate the user's actions: dialogs are modal, which means that the function that opens a dialog cannot continue until it is closed.
So, simulating the actions before the dialog is open does not work because there is no dialog; simulating the actions after the dialog is closed does not work because there is no dialog.

There are two practical solutions to this problem:

1. make the dialog not modal
2. open the dialog in the main thread and simulate the user's action in a secondary thread.
   

The first option is not possible in my case. The dialogs are opened by factory methods and I cannot change the code in these methods only to test them.
Moreover, if a second dialog gets opened by the first dialog we have the same problem again.

So I used threads.
I wrote a class for this purpose, SWTBotDialogTestThread. An example of use of this class is in
the test case InputDialogHandlerTest.

The right way to use a thread that acts on the SWT GUI is to call the function Display.asyncExec on the dialog's display, otherwise the GUI becomes unstable.
This is what I do in SWTBotDialogTestThread.workWithDialog.

Also if the actions on the dialog result in an exception, the exception must be caught by the test case in the main thread. I solved this problem running the actions on the dialog in a FutureTask instance (wrapped in the ITestAndWait interface). When the main thread calls the FutureTask.get method (ITestAndWait.waitForWorkDone) this throws the exception that was raised by the thread started by asyncexec, if something went wrong.

In synthesis, when I write a JUnit test case to test a dialog I:

• create an instance of SWTBotDialogTestThread, which opens the shell used as starting point for the dialog.
• call SWTBotDialogTestThread.workWithDialog, which starts the thread with the actions to do on the dialog. The thread waits until the dialog is available.
• open the dialog. At this point the actions are done on the dialog. One of the actions (click on the Ok button) closes the dialog.
  
• Call ITestAndWait.waitForWorkDone that re-throws the exception thrown by the other thread, if that thread thrown exceptions.
• check the result of the dialog.
• call SWTBotDialogTestThread.close to close the shell opened at the start.
  

I don't know if this is the best way to solve the problem, but it works, even if several test cases are run together.

Matrex 1.3. Development finished.

Matrex 1.3 coding has been finished.
I'm now testing it and updating the documentation.
Hopefully I'll release it at the end of september.

Callbacks

A Matrex project consists in a network of matrices and functions (let's forget about charts, presentations and timers).
They are connected together by the relations:

• The matrix Mi is input parameter of the function Fj
  
• The matrix Mi is output (result) of the function Fj

When a matrix changes its content, it fires the functions of which is input, which recalculate their output matrices, which in turn fires the functions of which are input and so on.

One peculiar property of this network is that it does not have loops. In mathematics it is representable as a simple graph.
This means that a matrix cannot be, directly or through other functions and matrices, input and output of the same function.

This is a good property, because it guarantees that, if the calculation of each single function terminates in a finite time, the calculation of the whole project terminates in a finite time.
In other words, the calculation of the project cannot contain an infinite loop.

In the upcoming version 1.3 I made an exception to this property, to solve the following problem:
I added a new solver function template to Matrex, that finds the minima of functions.
To use a solver, you need to pass to it the function you want to minimize.
In a programming language, you pass the function to mimimize to the solver function as a callback function (or a closure).

But how I did this in Matrex?
In Matrex, a programming language's function is equivalent to a network of functions and matrices, which, seen as a black box, has input and output matrices:

This block of functions and matrices can be equivalent to a callback function, with a trick:

• the output of the callback block and the input of the caller must have one matrix in common
• the output of the caller and the input of the callback block must have one matrix in common

The output matrix of the caller that is input matrix of the callback block is called callback matrix.

Here is a picture showing the concept:

In this way the caller, by changing the content of one of its input matrices recalculates the callback block.
When the callback block executes, it updates the content of its output matrices. But one of those is a input matrix of the caller, therefore the caller is recalculated.

Caller and callback block execute each other until the caller decides to stop and does not change the content of the output matrix that it has in common with the callback block.

But when does the caller decide to stop? This depends by the caller code.

The solver I added In Matrex is Nelder Mead (sys.mathstat.optimization.neldermead), an implementation of the direct search algorithm, a simplex optimization algorithm (it uses the Apache Commons Math library).
At each step of the solving process the solver needs to call a function to calculate the cost of the current solution. The cost calculation function is our callback function.
The following picture shows the solver, the cost calculation function and their input and output matrices:

x is the callback matrix of the solver. It is both output matrix of the solver and input matrix of the cost calculation function.
When the content of x is changed, the cost calculation function is fired and it calculates the cost matrix, which is input of the solver.
The solver is executed and produces a new content for the x matrix, therefore firing the cost calculation function, until the cost is good enough to stop. At this point it stops changing the content of x and writes the final solution in the result matrix.

What I learned writing the Nelder Mead solver template is that one has to write the caller code carefully making always sure that the mutual execution of caller and callback function stops at a certain point, in a way or the other. For example a solver must stop after a certain number of iterations, even if it does not converge.

Adding callbacks in Matrex required some changes in the code:

• The callback matrix is special: adding a function that has a callback output matrix requires that this matrix is already in the project (it is the input matrix for the callback block).
• Functions that have callback output matrices are not calculated when the project is loaded.
  Calculating them only once would leave them incomplete (for a solver this means to calculate only one step).
  Calculate them until they stop would be a risk. Better to leave to the user the responsability to calculate them.
  

Because of the complexity of this new feature, I introduce it in Matrex as experimental.

Test test test

Matrex is a project that gets tested manually by me before it is released.
For this purpose I use a check list.
I run Matrex and I verify that the operations listed in the check list, like adding a matrix, adding a function... are executed correctly and don't produce side effects.

Matrex is also tested by some of its user, but generally the users test it after a new version has been released. This is not very satisfactory, even if it allows me to release corrective versions.

To reduce the possibility that a version is released with bugs, I started, from version 1.2, to add some unit testing.

In the incoming release 1.3 unit testing will be a main concept:

• the tested classes are at this moment more than 160, covering almost all Matrex template functions and a good section of the application code.
• I'm checking the possibility to have also some GUI classes tested, using the SWTBot GUI unit testing tool.
  

In this way I hope I will be able to:

• Improve the quality of the code.
• Help developers understand the code of Matrex.
• Reduce the possibility of regression bugs.
  

If you have other ways to improve the reliability of Matrex please let me know.

Changes in the main window

Version 1.3 changes something in the main window:

1. The buttons that are highlighted with a green rectangle hide or show trees. For example the first button hides the Matrix tree. Clicking it once again the Matrix tree is displayed again.
   The second button toggles the Function tree visibility, the third the Presentation tree, the 4th the Chart tree and 5th the Timer tree.
2. On the top of each tree, together with the tree menu, there is a search bar, highlighted with a red rectangle.
   Writing a part of the name/package of the item to search (matrix, function...) and clicking on the green arrow button, the next item with name-package containing that text is selected and showed in the tree.
   For example the text ding.pri finds the item trading.price.
   

Exception stack trace for final users

In Matrex, like in other programs, it is not always possible to show a simple message box when an error occurred.
Sometimes what I have is an exception occurred deep down in the code, may be in a library.
Nevertheless something has to be shown to the user to let him understand that an error has occurred and give him all the available information to decide what to do.
A classic exception stack trace is not an option: together with the exception text it contains a lot of information to localize the code that generated the exception.
This information, that is precious for the developer, is probably unuseful and confusing for the final user.
For this reason I wrote a small function that transform the exception in a string that contains only the texts of the exception and of all contained (cause) exceptions. Here is the code:

public static String flatException(Throwable e)
{
StringBuilder b = new StringBuilder();
while(e != null)
{
b.append(e.getMessage()).append('\n');
e = e.getCause();
}
return b.toString();
}

Even if the number of contained exception is large, the message is generally simple and gives immediately an idea of what happened.
This is not intended as a substitute of the classic stack trace, which is written in the application log file so that if needed it can be examined by the developer.

Fit Columns

A new feature that will be released with version 1.3 is the possibility to fit the width of the columns of the table in a matrix viewer or in a presentation viewer.

This has been a problem from the moment I started to use virtual tables for the two viewer.
Virtual tables have the advantage that they can show tables with a huge amount of rows (> 10000) instantly, because they load only the rows of the table that are displayed in that moment, not the whole table.
As a side effect, there is no way to automatically set the width of the columns so that they can fit the text displayed in the columns' cells.
Because of this, before version 1.3 the use has to manually change the width of the columns.
In version 1.3 I added a menu in the two viewers, called Fit Columns:


This menu calls a function that takes a sample of the table's content: a fixed amount of rows. It measures the lengths of the texts contained in the cells of these rows, and with them it calculates an estimation of the columns' width. Then sets the columns' with to this value.

In conclusion there is no automatic columns' width calculation. The user decides when to calculate them. And that is a good thing, because if the matrix or the presentation change (because matrices get recalculated in Matrex), he can click the menu when he wants to update the width accordingly.