Changes in the main window

Version 1.3 changes something in the main window:

1. The buttons that are highlighted with a green rectangle hide or show trees. For example the first button hides the Matrix tree. Clicking it once again the Matrix tree is displayed again.
   The second button toggles the Function tree visibility, the third the Presentation tree, the 4th the Chart tree and 5th the Timer tree.
2. On the top of each tree, together with the tree menu, there is a search bar, highlighted with a red rectangle.
   Writing a part of the name/package of the item to search (matrix, function...) and clicking on the green arrow button, the next item with name-package containing that text is selected and showed in the tree.
   For example the text ding.pri finds the item trading.price.
   

Exception stack trace for final users

In Matrex, like in other programs, it is not always possible to show a simple message box when an error occurred.
Sometimes what I have is an exception occurred deep down in the code, may be in a library.
Nevertheless something has to be shown to the user to let him understand that an error has occurred and give him all the available information to decide what to do.
A classic exception stack trace is not an option: together with the exception text it contains a lot of information to localize the code that generated the exception.
This information, that is precious for the developer, is probably unuseful and confusing for the final user.
For this reason I wrote a small function that transform the exception in a string that contains only the texts of the exception and of all contained (cause) exceptions. Here is the code:

public static String flatException(Throwable e)
{
StringBuilder b = new StringBuilder();
while(e != null)
{
b.append(e.getMessage()).append('\n');
e = e.getCause();
}
return b.toString();
}

Even if the number of contained exception is large, the message is generally simple and gives immediately an idea of what happened.
This is not intended as a substitute of the classic stack trace, which is written in the application log file so that if needed it can be examined by the developer.

Fit Columns

A new feature that will be released with version 1.3 is the possibility to fit the width of the columns of the table in a matrix viewer or in a presentation viewer.

This has been a problem from the moment I started to use virtual tables for the two viewer.
Virtual tables have the advantage that they can show tables with a huge amount of rows (> 10000) instantly, because they load only the rows of the table that are displayed in that moment, not the whole table.
As a side effect, there is no way to automatically set the width of the columns so that they can fit the text displayed in the columns' cells.
Because of this, before version 1.3 the use has to manually change the width of the columns.
In version 1.3 I added a menu in the two viewers, called Fit Columns:


This menu calls a function that takes a sample of the table's content: a fixed amount of rows. It measures the lengths of the texts contained in the cells of these rows, and with them it calculates an estimation of the columns' width. Then sets the columns' with to this value.

In conclusion there is no automatic columns' width calculation. The user decides when to calculate them. And that is a good thing, because if the matrix or the presentation change (because matrices get recalculated in Matrex), he can click the menu when he wants to update the width accordingly.

High Low charts

One of the new features that will come with version 1.3 are the High Low charts.

It is a kind of financial charts that shows the evolutions of the quantities high, low, open, close and volume in a time period.
As all the other charts in Matrex, the charts are built using the JFreeChart library.

They can be shown in two variants: candlestick chart (top chart in the following picture) or high low chart (bottom chart). The picture shows the chart viewer for two high low charts that share the domain axis:

To build this kind of chart there is a special editor. In the following picture we show the editor for the two charts in the viewer.


There is one row in the table for each chart in the viewer. The values of high, low, open, close, volume are 1-column matrices in the same Matrex project.

The time period that defines the domain axis can be defined in two ways:

1. it can be a 1-column matrix containing date/times.
2. it can be an interval defined as [start,step]

The editor for each row of the table (which opens if you click the Edit button) is the following:

which shows the values for the first chart in the viewer: it is a candlestick chart, with the time period composed by an interval [start,step] and has all the values (high,low,open,close,volume) assigned to 1-column matrices of the project.

SWT Web installer RC1 released

The documentation is in the SWT Web Installer site, and you can download it from SourceForge.
Please report any bugs/problems/feature requests.


Moreover, there is a new version of the client/server architecture document in the Matrex site.
The main change is that the document considers also sequences of operations on the project that need to be executed atomically (transactions).

Status

SWT web installer

SWT web installer is almost ready to be released.
I tested it and wrote the documentation. It just needs the final touches.
I think it will be released during this week as version 1.0 beta.
License is LGPL 3.0.

ISQL

I made a library of ISQL, which I will release as soon as possible.
ISQL can be actually using separately from Matrex.
It is just to call a function with, as parameters, the SQL query and the input and output arrays.
I think it can be very interesting for people that wants to use SQL on arrays in memory.
Also in this case the license will be GPL 3.0-

Matrex

There will be a version 1.3 of Matrex.
The idea is to include in this version the last changes made to the source code of Matrex:

• new classloader and plugins structure
• SWT web installer to install SWT from internet
• separated ISQL library

Also, the 2D charts will be updated and new Matrex functions will be added, thanks to the new version of the commons math library .


I'm also working to plan version 2.0 of Matrex. I tested an example of RMI callbacks and I've found that it works very well. I will soon publish a new document about the future Matrex Client/Server architecture.

The SWT Web Installer

You wrote this nice SWT application and you want to release it.
And you realized it is not so easy, especially if the user has to install it himself.
Yes, because there is a different SWT version for each platform: a version for Windows, a version for Mac OSX, a version for Linux... not only, but also different versions for 32 and 64 bits.
What you do? You build a setup file for your application for each platform?

With these thoughts, I wrote a library that does the job for you.
It is called OS dependent installer (osdepinstaller).
You call it from your installer or from your application and it installs the right SWT version for the platform where your application is installed.

It works in this way:

• it loads from the internet a file containing a list of links to download the SWT library from, one for each platform.
  
• It checks the platform in which it is installed (two properties, os.type and os.arch).
• it gets the link for that platform
• it downloads the SWT library (it is a zip file)
  
• it unzips the SWT library in a specific directory

If something goes wrong (for example the platform was not found in the list of links), osdepinstaller offers other solutions to the user.
For example if the automatic choice of the platform does not work, the user is asked to do the choice manually.

A principle of osdepinstaller is to give all the information about what it is doing. In this way the user can continue manually if for some reason the library is not able to continue.

Since osdepinstaller cannot work without internet, and it can happen that the internet can be accessed only through a firewall or a proxy, the system checks with the user if there is a proxy and, in this case, if the proxy needs authorization.

To get data from the user and to show the status osdepinstaller uses by default the console (since it is normal that your application does not have a GUI until osdepinstaller has finished), but it can be easily adapted to other solutions.

Also, osdepinstaller is not only for SWT: it can be used for any library the is platform dependent.

Osdepinstaller is still in test, but you can download a test version and check the code.

I will use osdepinstaller to let the user install SWT with the generic installer of Matrex.

Matrex 1.2 for MacOSX alternative setup

I've got some one reporting me that the Matrex 1.2 version for Mac OSX does not install perfectly on all Macs.
The problem is that sometimes after the setup Matrex does not look like an application but like a simple java folder.
Surely you can open a terminal and run the shell script (the batch file) from there, but this is not really an intuitive way of running it!

I use IzPack to install my software and IzPack claims to be able to install also on Mac OSX, even if from some messages in mailing lists it seems that there could be some problems.

I tried to see what I can do.

First I will modify the shell script so that it can be called from everywhere (an initial statement to let it use the current directory as starting directory).
Then I will update Izpack to the last version (3.11).

But one thing that I would really like to do is to create a MacOSX application bundle containing the Matrex files.
The application bundle is the natural way applications are installed on MacOSX.
To do this you need:

• a .icns file, that contains the application icon.
  
• a way to generate a .dmg file, which is an autoinstalling disk image containing the application bundle.
  

There is no way, as far as I know, to generate .icns and .dmg files outside MacOSX.

So, if someone has a Mac and wants to help me to build this application bundle and to test it, can he please let me know?

List file class loader

Matrex is composed by several jars:

• The Matrex jars: api, gui, fun.
• The libraries used by Matrex, for example SWT, jfreechart.
• The plug-ins, for example the library to connect Matrex to Matlab.

To have java loading all these jars when starting Matrex, they was listed in the classpath parameter of the java call.
Here is the typical content of a batch file starting Matrex (it uses the two variables MATREX_CP and LIB_CP):

MATREX_CP=matrex_api.jar:matrex_fun.jar:matrex_gui.jar
LIB_CP=freehepj3d.jar:jython.jar:commons-math-1.1.jar:jfreechart-1.0.8a.jar:jcommon-1.0.12.jar:jxl.jar:lucene-core-2.0.0.jar:swt/swt.jar:antlr-runtime-3.0.1.jar:javacsv.jar:net.sf.paperclips_1.0.2.jar
java -Djava.library.path=swt -Djava.util.logging.config.file=logging.properties -classpath $MATREX_CP:$LIB_CP matrex.gui.Start $1

This approach has several problems:

• It is not readable, and it becomes less readable adding more libraries.
• If the user wants to add a plug-in, he has to edit the batch file. This is unconfortable, because the user needs to learn the syntax of the batch file, and dangerous: if the user does a small mistake changing the batch file the program does not start.
• you need to have one copy of this batch file for each platform in which Matrex is released (Windows, Linux, MacOSX...).
  

In these days I worked on a new approach. The idea is to have 3 files containing the list of jars needed to start Matrex:

• matrex.cld containing the main Matrex jars
• libraries.cld containing the libraries used by Matrex
• plugis.cld containing the plug-ins (it is empty when Matrex is installed)

Here is an content of the matrex.cld file. The other two files follow the same rules:

matrex_api.jar
matrex_fun.jar
matrex_gui.jar

To load these files and to load the jars listed in them, Matrex uses a custom class loader, TextFilesClassLoader.

The start class of Matrex is now called MatrexLoader.
MatrexLoader loads the main class of Matrex, Start, using TextFilesClassLoader as class loader.
Since Start all the other classes of Matrex (included the ones in the library) are loaded, directly or indirectly, from Start, they all use the same class loader used by Start, TextFilesClassLoader.

In this way it has been also possible to write a special Matrex GUI dialog, PluginsDialog, to update the content of plugins.cld:



This is much easier than to change manually the batch file.

Matrex functions unit testing

I added a small chapter at the end of the function coding document about the Matrex functions code unit testing.


Before version 1.2 some of the Matrex functions had JUnit unit testing, but it was very limited.
In version 1.2 I had to write the code for the ISQL function template, which is very complex compared to the other templates.
So I needed to test it well.
For this reason I wrote some unit testing help classes (for JUnit) that I grouped under the package matrex.item.test.
These classes can be used for the unit testing of any Matrex function template.

With these classes unit testing of matrex functions becomes very easy. For standard function templates it is a matter of minutes to write a test case.
Give a look to them!

Matrex 1.2 released

Finally Matrex 1.2 has been released!

The big step forward of this version is the new ISQL module.
The ISQL module is a function template to apply SQL queries to vectors in a Matrex project.
It is therefore possible:

• to apply filters on parts of the vectors
• to order the elements of the vectors
  
• to aggregate them

all with simple SQL queries. The internalsql document contained in the distributions explains how.
As far as I know, no other spreadsheet application allows something similar.

Other enhancements are:

• Print matrices and presentations.
• Export matrices and presentations to csv and xls.
  
  
  
  
• Import data from database, using an SQL query.
• Menus to remove multiple items, duplicate matrices and functions, add function from a matrix (the function will have the matrix as first argument).
  
  
  
• A summary step for the expression parser.
  
  
• Possibility to change the colors of the windows headers.
  
  
  

Also, several bugs have been removed (dates formatting, items renaming, boolean matrices editing).

Pre-release of Matrex 1.2 to test

I'm in the test phase of the version 1.2 of Matrex.
If someone wants to test this pre-release version himself, he can:

• download the pre-release generic setup
  
• install it
• install the SWT library for his platform in the swt directory of the installation
• use one of the matrix_generic.* files in doc/batch_files to launch Matrex

The changes from version 1.1 are in short:

• The new ISQL (internal SQL) module. There is a document about it in the setup.
• New import and export modules.

If you find some bugs, please report them in the Matrex bugs page.

Thanks

Version 1.2 status

Coding and documentation are finished.
I'm now working on an example project for the ISQL module.
After that I will terminate the test of the system and release it.

There will be a version for:

• Windows 32 bits
• Linux 32 bits
• Mac OSX

and also a generic version, which you can use if you want to install it in a different environment (64 bits?).
The generic version does not install SWT. You have to get it and unzip it yourself under the swt directory.

Status

Today I finally released matrexjruby 1.0.2 (you can find it in the Matrex download page). It took some time to do the final test because of the frenetical work on Matrex 1.2.
I'm now working on the last pieces of code of Matrex 1.2, since ISQL has now been tested and I consider it finished:

• the code that allows to add a function from the matrix tree using as first matrix parameter the selected matrix.
• the code that allows to create an item from the item selection dialog box.

Once these are finished, I need to do some more test and update the documentation. I hope 1.2 will be released in january.

MatrexJRuby

I updated the code of the adapter matrexjruby, which was still based on JRuby 0.9.
Now it works with JRuby 1.01.

Only a few fixes was needed (use the Ruby class instead of the IRuby interface, let the JRuby classes inherit from an abstract class instead of the IFunction interface).

Matrexjruby is used to write the scripts that are the "code" of the Matrex function templates.

I just need some final test and I will publish it as version 1.02.

See the light

Finally!

I 've got stock with the internal sql module for a long while, I'm now able to see some results.
The biggest problem derived from the fact that I made the case... when... then... clause originally too complex (the when clause was too general).

Now I'm able to run some "select ... from ... where..." queries (without group by or order by clauses).
The code is on CVS under the package matrex.fun.sys.sql , so you can give a look yourself if you like.

It looks like it will not take so long before I have an implementation of the group by clause (together with the aggregate functions).
Still some doubts about the order by clause (allow order by or only order by ?).

So, now we should be closer to version 1.2, if nothing comes in the way.

Antlr for the SQL module

To build the internal SQL module for Matrex I decided to work in the following way:

• Write a parser that converts the SQL expression in an internal object structure
• Write the code that applies the parsed SQL to the matrices/vectors arguments of the SQL function.

To do the parsing work, I chose the Antlr library.
Antlr has the following advantages compared to the other parsers:

• More people use it (at least it looks like)
• The parse result can be code in different languages (Java, C#, python...) that can be useful if I want to port the same grammar to other projects
• Together with the library you can download a graphical application called AntlrWorks to interactively test and debug your grammar. AntlrWorks is a very good tool, that let you find errors in your grammar before you start to use it.
  

Antlr is a wonderful product, but I suffered creating the SQL grammar I needed.
The reasons are probably:

• my inexperience in terms of parsers/lexers
• some confusion and some holes in the free documentation

My initial idea was to download Antlr, get a grammar describing the SQL SELECT statement, adapt the grammar to my needs, build the java sources from it and convert the produced AST trees to my internal structures.

Simple, right? Wrong. Here are the problems for this approach:

1. Antlr is in version 3 now (is normally called v3). The example grammars are most made for version 2 (2.7.x). Altough v2 and v3 grammars look very similar, to convert a v2 grammar to a v3 one is not easy.
2. It is possible to buy a book written by the Antlr's author. I did not want to buy the book because I'm not planning to use Antlr in the future. But then I discovered that the online documentation is partial and often referring to the old 2.7.2 version.
3. The produced java classes can have a method to get the AST tree, but as far as I understood the tree cannot be used for an interpreter, but only to check the result of the statement parsing.
4. To build an interpreter is not the only purpose for using a parser:
   Antlr is used for many other things, for example to compile, which
   means convert expressions from a grammar to another one. Consider this to avoid to get confused reading the documentation.
   
   

I struggled for a pair of weeks with these problems. At the end I was able to to understand the following concepts and to produce my grammar:

• You can add java code directly in the grammar. With this code you can build the interpreter structures directly in the generated java code.
• Be very careful about the case of the initial letter of the rule names. Upper case: lexer rule; Lower case : parser rule. It looks simple, but if only the initial letter of one rule name is wrong nothing works as it should.
  
• The lexer is used to parse single words (identifier, strings, numbers). The parser is used to parse phrases.
• Spaces are handled automatically by the parser.
  
• In the java code that you add to the grammar you can set the package of the generated classes.
• AntlrWorks generates two types of java classes: the ones to use in your application and the ones that it uses to debug. They are saved in the same place with the same names. The debug classes don't work in your application, so remember to generate the application classes after a debug session.
  

So, if you give a look to the grammar I have written, you'll see that it is confused (rules definitions together with java code), but it works. When I run the generated java classes against an SQL expression the structures declared in the @members block are populated with the correct values and from them I can interpret the expression.

Version 1.2: status

Not so much has been produced in the last period, for the following reasons:

• holiday
• got sick for one month
• some time used understanding the concepts needed to write a grammar to parse the new internal sql template.

I'm still writing the grammar for the sql template. Soon I will be able to produce some code.
The rest of the coding of version 1.2 is more or less in place.
It clearly needs to be tested.

JRuby released a new version, so I will soon update the Matrex JRuby adapter.

SQL on matrices in Matrex

I'm planning to write a new Matrex template to query matrices (only the ones with one column) using an SQL-like syntax
The idea is simple:

• consider the input matrices/vectors as columns of tables
• query them using SQL
• the SQL query result are the output matrices/vectors of the function.
  

The query is written in a language that is very similar to SQL. The differences are:

• the tables are defined in the query as sets of input matrices
• sub-queries are not allowed

Operators (+, - ...) and aggregate functions (sum, max...) are available.
For example, a query can look like:

select sum(a.price) as sumprice, sum(a.volume) as sumvolume, b.ticker as ticker, b.validity as validity
from [dealticker, price, volume] a, [ticker, validity] b
where a.dealticker = b.ticker
group by b.ticker

Here the matrices dealticker, price, volume are grouped together to form the table a, of which they are columns;
the matrices ticker, validity are grouped together to form the table b, of which they are columns.

Every result column needs to be assigned a name to; that is done using the as keyword.

All the table columns (e.g. a.price, b.ticker...) need to be expressed in the form [table].[column] (e.g. a.price).


This will be one of the most powerful templates in Matrex. With it it will be possible to express very easily and clearly problems that otherwise could require the combination of multiple functions.

Since it is not dependent by new features of the upcoming version 1.2, it is possible that the template will be available before 1.2 is released.

About nebula

Nebula is a sub-project of eclipse that produces custom SWT widgets.
I'm very interested in the grid, because it is supposed to have spreadsheet features like cells selection and row headers, which are very important in Matrex and I can just emulate in the SWT.Table widget.
Nebula Grid is still in alpha state but it should become final when Eclipse 3.3 is released.
I will check it (expecially with a big number of rows/columns) and possibly integrate it in Matrex in version 1.2 or 1.3.