Plans

These are the plans for the future of Matrex, in order of time:

1. Upgrade all adapters to libraries, script languages and scientific applications to work with the version 1.3.x of Matrex, which uses a new plugins architecture. I already upgraded the Jama adapter and I'm working on the JRuby adapter. Clearly if there are new versions of the libraries, languages and applications the adapters will work with them.
   
2. New adapter to the Groovy interpreter.
   
3. New intermediate version of Matrex, probably called 1.3.5. This version will not contain major changes from version 1.3. The idea is to make possible to have configuration files and additional templates in different directories. In this way it is possible to build .deb or .rpm setup packages for Matrex and make it easier to use under Vista and MacOSX.
   
4. New version 2.0: It will be possible to use Matrex as a standalone application, like now, or as a client/server system.

The upgrade of the adapters and the Groovy adapter will be done in the next weeks.
Versions 1.3.5 will be probably finished in ~ 2/3 months.
2.0 will be finished in ~ 6 months.
Versions 1.3.5 and version 2.0 will be developed in parallel.

In the same time Braxton and me are trying to build a setup file to install Matrex on MacOSX as an application bundle.

Compatibility with Java5 : Matrex 1.3.01

I published a new release of Matrex, 1.3.01, to cope with the fact that Matrex 1.3 was able to run with Java6, but not with Java5.
The problem was a call to the String.isEmpty() method, which is only available from Java6.
In 1.3.01 this call has been replaced and this version has been tested with Java5.

Thanks to Braxton who found the problem.

Matrex 1.3 relased

So we finally made it. Version 1.3 has been released!

A summary of the changes from version 1.2 is available in the Sourceforge's release notes.
The big changes have been:

• Main View usability improvement.
  The main view is now easier to use thanks to the buttons that hide and show specific trees (for example you can hide the function tree and show the presentation tree just clicking two buttons).
  
  It is also possible to search items or folders in a tree just typing a part of its name (in red).
  
  In each tree is not possible to move a set of items to a package.
  Just select some items of a tree (also in different packages), right-click with them mouse on the tree to display the popup menu and select Move... (for example Move Matrices). With a dialog box you choose the destination package and the items gets moved.
  The system takes care of the dependencies.
  
  
• Viewers usability improvement . In the presentation viewer and matrix viewer it is now possible to display the content of presentations and matrices in the best way clicking the fit columns button.
  
  
  
  
• Generic setup. The Matrex generic setup allows to install Matrex on every platform. The idea is that, once installed, when executed the first time, Matrex downloads the correct version of the SWT library and installs it.
  Executing it once again, the downloaded library is used and Matrex works regularly.
  All thanks to the SWT Web Installer library.
  
  
• New high low charts. Added candle and high low charts.
  
  
  
• Plug-in architecture. Matrex now uses its own class loader, which allows to specify libraries (e.g. function template libraries) that need to be loaded using the Plugin Dialog:
  
  
  No more class paths to change in the batch files!
  
  
• Code quality improvement. The code is now well covered with JUnit tests and has been checked with Findbugs. The GUI has also been unit tested using SWTBot.
  

Also the experimental new feature of the callbacks has been added. It is used in the directsearch example project, which demonstrates the Nelder Mead optimization method.

As usual, feedback is precious. Please let me know if something is wrong!

SWT Web Installer and ISQL Library updated

Since I published the pre-release of Matrex 1.3, I also updated the two libraries that Matrex uses, SWT Web Installer (now 1.0) and ISQL Library (now 1.01). 

SWT Web Installer downloads now SWT 3.4 instead of 3.3.1 and has a small change to the WebInstaller class constructor (new parameter systemProperties).

I also added a test that verifies that all sites in the swt_list.xml configuration file exist and are accessible.

ISQL Library had a small bug related to the conditions of the query's where clause. That has been fixed in version 1.01.

I have checked both libraries with Findbugs before I published them. 

Matrex 1.3 pre-release

I published a pre-release version of Matrex 1.3.

You can find it under the Matrex Unstable download section in SourceForge.

The changes from version 1.2 are listed under the Matrex File Release Notes and Changelog page in SourceForge.

The idea is to make sure that if there are bugs in Matrex 1.3 they get fixed before the final version is released.

So please give a try to this version and, if you find bugs, use the tracker page in Sourceforge to report them.

The pre-release version includes only the setup for the generic architecture, which means that when you start Matrex the first time it is supposed to download the SWT library for your PC.

After that Matrex stops, so you need to start it once again to use it.

The download of SWT is a new feature, so if you have problems with it please use the tracker page to report them.

SWTBot for dialogs

I started writing JUnit test cases to test the GUI of Matrex using SWTBot.

Since I did not have any experience with GUI testing, I wanted to start with something simple.
The simplest GUI items to test in Matrex are the dialogs, since they do something specific and limited.
So I started writing test cases to test dialogs.

It turned out that testing dialogs is not so easy as it seems (and this is a general problem, not limited to SWTBot).
With SWTBot you simulate the actions of your application's user: press buttons, write in text boxes, select from list and check boxes...
But you cannot simply open a dialog and simulate the user's actions: dialogs are modal, which means that the function that opens a dialog cannot continue until it is closed.
So, simulating the actions before the dialog is open does not work because there is no dialog; simulating the actions after the dialog is closed does not work because there is no dialog.

There are two practical solutions to this problem:

1. make the dialog not modal
2. open the dialog in the main thread and simulate the user's action in a secondary thread.
   

The first option is not possible in my case. The dialogs are opened by factory methods and I cannot change the code in these methods only to test them.
Moreover, if a second dialog gets opened by the first dialog we have the same problem again.

So I used threads.
I wrote a class for this purpose, SWTBotDialogTestThread. An example of use of this class is in
the test case InputDialogHandlerTest.

The right way to use a thread that acts on the SWT GUI is to call the function Display.asyncExec on the dialog's display, otherwise the GUI becomes unstable.
This is what I do in SWTBotDialogTestThread.workWithDialog.

Also if the actions on the dialog result in an exception, the exception must be caught by the test case in the main thread. I solved this problem running the actions on the dialog in a FutureTask instance (wrapped in the ITestAndWait interface). When the main thread calls the FutureTask.get method (ITestAndWait.waitForWorkDone) this throws the exception that was raised by the thread started by asyncexec, if something went wrong.

In synthesis, when I write a JUnit test case to test a dialog I:

• create an instance of SWTBotDialogTestThread, which opens the shell used as starting point for the dialog.
• call SWTBotDialogTestThread.workWithDialog, which starts the thread with the actions to do on the dialog. The thread waits until the dialog is available.
• open the dialog. At this point the actions are done on the dialog. One of the actions (click on the Ok button) closes the dialog.
  
• Call ITestAndWait.waitForWorkDone that re-throws the exception thrown by the other thread, if that thread thrown exceptions.
• check the result of the dialog.
• call SWTBotDialogTestThread.close to close the shell opened at the start.
  

I don't know if this is the best way to solve the problem, but it works, even if several test cases are run together.

Matrex 1.3. Development finished.

Matrex 1.3 coding has been finished.
I'm now testing it and updating the documentation.
Hopefully I'll release it at the end of september.

Callbacks

A Matrex project consists in a network of matrices and functions (let's forget about charts, presentations and timers).
They are connected together by the relations:

• The matrix Mi is input parameter of the function Fj
  
• The matrix Mi is output (result) of the function Fj

When a matrix changes its content, it fires the functions of which is input, which recalculate their output matrices, which in turn fires the functions of which are input and so on.

One peculiar property of this network is that it does not have loops. In mathematics it is representable as a simple graph.
This means that a matrix cannot be, directly or through other functions and matrices, input and output of the same function.

This is a good property, because it guarantees that, if the calculation of each single function terminates in a finite time, the calculation of the whole project terminates in a finite time.
In other words, the calculation of the project cannot contain an infinite loop.

In the upcoming version 1.3 I made an exception to this property, to solve the following problem:
I added a new solver function template to Matrex, that finds the minima of functions.
To use a solver, you need to pass to it the function you want to minimize.
In a programming language, you pass the function to mimimize to the solver function as a callback function (or a closure).

But how I did this in Matrex?
In Matrex, a programming language's function is equivalent to a network of functions and matrices, which, seen as a black box, has input and output matrices:

This block of functions and matrices can be equivalent to a callback function, with a trick:

• the output of the callback block and the input of the caller must have one matrix in common
• the output of the caller and the input of the callback block must have one matrix in common

The output matrix of the caller that is input matrix of the callback block is called callback matrix.

Here is a picture showing the concept:

In this way the caller, by changing the content of one of its input matrices recalculates the callback block.
When the callback block executes, it updates the content of its output matrices. But one of those is a input matrix of the caller, therefore the caller is recalculated.

Caller and callback block execute each other until the caller decides to stop and does not change the content of the output matrix that it has in common with the callback block.

But when does the caller decide to stop? This depends by the caller code.

The solver I added In Matrex is Nelder Mead (sys.mathstat.optimization.neldermead), an implementation of the direct search algorithm, a simplex optimization algorithm (it uses the Apache Commons Math library).
At each step of the solving process the solver needs to call a function to calculate the cost of the current solution. The cost calculation function is our callback function.
The following picture shows the solver, the cost calculation function and their input and output matrices:

x is the callback matrix of the solver. It is both output matrix of the solver and input matrix of the cost calculation function.
When the content of x is changed, the cost calculation function is fired and it calculates the cost matrix, which is input of the solver.
The solver is executed and produces a new content for the x matrix, therefore firing the cost calculation function, until the cost is good enough to stop. At this point it stops changing the content of x and writes the final solution in the result matrix.

What I learned writing the Nelder Mead solver template is that one has to write the caller code carefully making always sure that the mutual execution of caller and callback function stops at a certain point, in a way or the other. For example a solver must stop after a certain number of iterations, even if it does not converge.

Adding callbacks in Matrex required some changes in the code:

• The callback matrix is special: adding a function that has a callback output matrix requires that this matrix is already in the project (it is the input matrix for the callback block).
• Functions that have callback output matrices are not calculated when the project is loaded.
  Calculating them only once would leave them incomplete (for a solver this means to calculate only one step).
  Calculate them until they stop would be a risk. Better to leave to the user the responsability to calculate them.
  

Because of the complexity of this new feature, I introduce it in Matrex as experimental.

Test test test

Matrex is a project that gets tested manually by me before it is released.
For this purpose I use a check list.
I run Matrex and I verify that the operations listed in the check list, like adding a matrix, adding a function... are executed correctly and don't produce side effects.

Matrex is also tested by some of its user, but generally the users test it after a new version has been released. This is not very satisfactory, even if it allows me to release corrective versions.

To reduce the possibility that a version is released with bugs, I started, from version 1.2, to add some unit testing.

In the incoming release 1.3 unit testing will be a main concept:

• the tested classes are at this moment more than 160, covering almost all Matrex template functions and a good section of the application code.
• I'm checking the possibility to have also some GUI classes tested, using the SWTBot GUI unit testing tool.
  

In this way I hope I will be able to:

• Improve the quality of the code.
• Help developers understand the code of Matrex.
• Reduce the possibility of regression bugs.
  

If you have other ways to improve the reliability of Matrex please let me know.

Changes in the main window

Version 1.3 changes something in the main window:

1. The buttons that are highlighted with a green rectangle hide or show trees. For example the first button hides the Matrix tree. Clicking it once again the Matrix tree is displayed again.
   The second button toggles the Function tree visibility, the third the Presentation tree, the 4th the Chart tree and 5th the Timer tree.
2. On the top of each tree, together with the tree menu, there is a search bar, highlighted with a red rectangle.
   Writing a part of the name/package of the item to search (matrix, function...) and clicking on the green arrow button, the next item with name-package containing that text is selected and showed in the tree.
   For example the text ding.pri finds the item trading.price.
   

Exception stack trace for final users

In Matrex, like in other programs, it is not always possible to show a simple message box when an error occurred.
Sometimes what I have is an exception occurred deep down in the code, may be in a library.
Nevertheless something has to be shown to the user to let him understand that an error has occurred and give him all the available information to decide what to do.
A classic exception stack trace is not an option: together with the exception text it contains a lot of information to localize the code that generated the exception.
This information, that is precious for the developer, is probably unuseful and confusing for the final user.
For this reason I wrote a small function that transform the exception in a string that contains only the texts of the exception and of all contained (cause) exceptions. Here is the code:

public static String flatException(Throwable e)
{
StringBuilder b = new StringBuilder();
while(e != null)
{
b.append(e.getMessage()).append('\n');
e = e.getCause();
}
return b.toString();
}

Even if the number of contained exception is large, the message is generally simple and gives immediately an idea of what happened.
This is not intended as a substitute of the classic stack trace, which is written in the application log file so that if needed it can be examined by the developer.

Fit Columns

A new feature that will be released with version 1.3 is the possibility to fit the width of the columns of the table in a matrix viewer or in a presentation viewer.

This has been a problem from the moment I started to use virtual tables for the two viewer.
Virtual tables have the advantage that they can show tables with a huge amount of rows (> 10000) instantly, because they load only the rows of the table that are displayed in that moment, not the whole table.
As a side effect, there is no way to automatically set the width of the columns so that they can fit the text displayed in the columns' cells.
Because of this, before version 1.3 the use has to manually change the width of the columns.
In version 1.3 I added a menu in the two viewers, called Fit Columns:


This menu calls a function that takes a sample of the table's content: a fixed amount of rows. It measures the lengths of the texts contained in the cells of these rows, and with them it calculates an estimation of the columns' width. Then sets the columns' with to this value.

In conclusion there is no automatic columns' width calculation. The user decides when to calculate them. And that is a good thing, because if the matrix or the presentation change (because matrices get recalculated in Matrex), he can click the menu when he wants to update the width accordingly.

High Low charts

One of the new features that will come with version 1.3 are the High Low charts.

It is a kind of financial charts that shows the evolutions of the quantities high, low, open, close and volume in a time period.
As all the other charts in Matrex, the charts are built using the JFreeChart library.

They can be shown in two variants: candlestick chart (top chart in the following picture) or high low chart (bottom chart). The picture shows the chart viewer for two high low charts that share the domain axis:

To build this kind of chart there is a special editor. In the following picture we show the editor for the two charts in the viewer.


There is one row in the table for each chart in the viewer. The values of high, low, open, close, volume are 1-column matrices in the same Matrex project.

The time period that defines the domain axis can be defined in two ways:

1. it can be a 1-column matrix containing date/times.
2. it can be an interval defined as [start,step]

The editor for each row of the table (which opens if you click the Edit button) is the following:

which shows the values for the first chart in the viewer: it is a candlestick chart, with the time period composed by an interval [start,step] and has all the values (high,low,open,close,volume) assigned to 1-column matrices of the project.

SWT Web installer RC1 released

The documentation is in the SWT Web Installer site, and you can download it from SourceForge.
Please report any bugs/problems/feature requests.


Moreover, there is a new version of the client/server architecture document in the Matrex site.
The main change is that the document considers also sequences of operations on the project that need to be executed atomically (transactions).

Status

SWT web installer

SWT web installer is almost ready to be released.
I tested it and wrote the documentation. It just needs the final touches.
I think it will be released during this week as version 1.0 beta.
License is LGPL 3.0.

ISQL

I made a library of ISQL, which I will release as soon as possible.
ISQL can be actually using separately from Matrex.
It is just to call a function with, as parameters, the SQL query and the input and output arrays.
I think it can be very interesting for people that wants to use SQL on arrays in memory.
Also in this case the license will be GPL 3.0-

Matrex

There will be a version 1.3 of Matrex.
The idea is to include in this version the last changes made to the source code of Matrex:

• new classloader and plugins structure
• SWT web installer to install SWT from internet
• separated ISQL library

Also, the 2D charts will be updated and new Matrex functions will be added, thanks to the new version of the commons math library .


I'm also working to plan version 2.0 of Matrex. I tested an example of RMI callbacks and I've found that it works very well. I will soon publish a new document about the future Matrex Client/Server architecture.

The SWT Web Installer

You wrote this nice SWT application and you want to release it.
And you realized it is not so easy, especially if the user has to install it himself.
Yes, because there is a different SWT version for each platform: a version for Windows, a version for Mac OSX, a version for Linux... not only, but also different versions for 32 and 64 bits.
What you do? You build a setup file for your application for each platform?

With these thoughts, I wrote a library that does the job for you.
It is called OS dependent installer (osdepinstaller).
You call it from your installer or from your application and it installs the right SWT version for the platform where your application is installed.

It works in this way:

• it loads from the internet a file containing a list of links to download the SWT library from, one for each platform.
  
• It checks the platform in which it is installed (two properties, os.type and os.arch).
• it gets the link for that platform
• it downloads the SWT library (it is a zip file)
  
• it unzips the SWT library in a specific directory

If something goes wrong (for example the platform was not found in the list of links), osdepinstaller offers other solutions to the user.
For example if the automatic choice of the platform does not work, the user is asked to do the choice manually.

A principle of osdepinstaller is to give all the information about what it is doing. In this way the user can continue manually if for some reason the library is not able to continue.

Since osdepinstaller cannot work without internet, and it can happen that the internet can be accessed only through a firewall or a proxy, the system checks with the user if there is a proxy and, in this case, if the proxy needs authorization.

To get data from the user and to show the status osdepinstaller uses by default the console (since it is normal that your application does not have a GUI until osdepinstaller has finished), but it can be easily adapted to other solutions.

Also, osdepinstaller is not only for SWT: it can be used for any library the is platform dependent.

Osdepinstaller is still in test, but you can download a test version and check the code.

I will use osdepinstaller to let the user install SWT with the generic installer of Matrex.

Matrex 1.2 for MacOSX alternative setup

I've got some one reporting me that the Matrex 1.2 version for Mac OSX does not install perfectly on all Macs.
The problem is that sometimes after the setup Matrex does not look like an application but like a simple java folder.
Surely you can open a terminal and run the shell script (the batch file) from there, but this is not really an intuitive way of running it!

I use IzPack to install my software and IzPack claims to be able to install also on Mac OSX, even if from some messages in mailing lists it seems that there could be some problems.

I tried to see what I can do.

First I will modify the shell script so that it can be called from everywhere (an initial statement to let it use the current directory as starting directory).
Then I will update Izpack to the last version (3.11).

But one thing that I would really like to do is to create a MacOSX application bundle containing the Matrex files.
The application bundle is the natural way applications are installed on MacOSX.
To do this you need:

• a .icns file, that contains the application icon.
  
• a way to generate a .dmg file, which is an autoinstalling disk image containing the application bundle.
  

There is no way, as far as I know, to generate .icns and .dmg files outside MacOSX.

So, if someone has a Mac and wants to help me to build this application bundle and to test it, can he please let me know?

List file class loader

Matrex is composed by several jars:

• The Matrex jars: api, gui, fun.
• The libraries used by Matrex, for example SWT, jfreechart.
• The plug-ins, for example the library to connect Matrex to Matlab.

To have java loading all these jars when starting Matrex, they was listed in the classpath parameter of the java call.
Here is the typical content of a batch file starting Matrex (it uses the two variables MATREX_CP and LIB_CP):

MATREX_CP=matrex_api.jar:matrex_fun.jar:matrex_gui.jar
LIB_CP=freehepj3d.jar:jython.jar:commons-math-1.1.jar:jfreechart-1.0.8a.jar:jcommon-1.0.12.jar:jxl.jar:lucene-core-2.0.0.jar:swt/swt.jar:antlr-runtime-3.0.1.jar:javacsv.jar:net.sf.paperclips_1.0.2.jar
java -Djava.library.path=swt -Djava.util.logging.config.file=logging.properties -classpath $MATREX_CP:$LIB_CP matrex.gui.Start $1

This approach has several problems:

• It is not readable, and it becomes less readable adding more libraries.
• If the user wants to add a plug-in, he has to edit the batch file. This is unconfortable, because the user needs to learn the syntax of the batch file, and dangerous: if the user does a small mistake changing the batch file the program does not start.
• you need to have one copy of this batch file for each platform in which Matrex is released (Windows, Linux, MacOSX...).
  

In these days I worked on a new approach. The idea is to have 3 files containing the list of jars needed to start Matrex:

• matrex.cld containing the main Matrex jars
• libraries.cld containing the libraries used by Matrex
• plugis.cld containing the plug-ins (it is empty when Matrex is installed)

Here is an content of the matrex.cld file. The other two files follow the same rules:

matrex_api.jar
matrex_fun.jar
matrex_gui.jar

To load these files and to load the jars listed in them, Matrex uses a custom class loader, TextFilesClassLoader.

The start class of Matrex is now called MatrexLoader.
MatrexLoader loads the main class of Matrex, Start, using TextFilesClassLoader as class loader.
Since Start all the other classes of Matrex (included the ones in the library) are loaded, directly or indirectly, from Start, they all use the same class loader used by Start, TextFilesClassLoader.

In this way it has been also possible to write a special Matrex GUI dialog, PluginsDialog, to update the content of plugins.cld:



This is much easier than to change manually the batch file.

Matrex functions unit testing

I added a small chapter at the end of the function coding document about the Matrex functions code unit testing.


Before version 1.2 some of the Matrex functions had JUnit unit testing, but it was very limited.
In version 1.2 I had to write the code for the ISQL function template, which is very complex compared to the other templates.
So I needed to test it well.
For this reason I wrote some unit testing help classes (for JUnit) that I grouped under the package matrex.item.test.
These classes can be used for the unit testing of any Matrex function template.

With these classes unit testing of matrex functions becomes very easy. For standard function templates it is a matter of minutes to write a test case.
Give a look to them!

Matrex 1.2 released

Finally Matrex 1.2 has been released!

The big step forward of this version is the new ISQL module.
The ISQL module is a function template to apply SQL queries to vectors in a Matrex project.
It is therefore possible:

• to apply filters on parts of the vectors
• to order the elements of the vectors
  
• to aggregate them

all with simple SQL queries. The internalsql document contained in the distributions explains how.
As far as I know, no other spreadsheet application allows something similar.

Other enhancements are:

• Print matrices and presentations.
• Export matrices and presentations to csv and xls.
  
  
  
  
• Import data from database, using an SQL query.
• Menus to remove multiple items, duplicate matrices and functions, add function from a matrix (the function will have the matrix as first argument).
  
  
  
• A summary step for the expression parser.
  
  
• Possibility to change the colors of the windows headers.
  
  
  

Also, several bugs have been removed (dates formatting, items renaming, boolean matrices editing).

Pre-release of Matrex 1.2 to test

I'm in the test phase of the version 1.2 of Matrex.
If someone wants to test this pre-release version himself, he can:

• download the pre-release generic setup
  
• install it
• install the SWT library for his platform in the swt directory of the installation
• use one of the matrix_generic.* files in doc/batch_files to launch Matrex

The changes from version 1.1 are in short:

• The new ISQL (internal SQL) module. There is a document about it in the setup.
• New import and export modules.

If you find some bugs, please report them in the Matrex bugs page.

Thanks

Version 1.2 status

Coding and documentation are finished.
I'm now working on an example project for the ISQL module.
After that I will terminate the test of the system and release it.

There will be a version for:

• Windows 32 bits
• Linux 32 bits
• Mac OSX

and also a generic version, which you can use if you want to install it in a different environment (64 bits?).
The generic version does not install SWT. You have to get it and unzip it yourself under the swt directory.