Status

SWT web installer

SWT web installer is almost ready to be released.
I tested it and wrote the documentation. It just needs the final touches.
I think it will be released during this week as version 1.0 beta.
License is LGPL 3.0.

ISQL

I made a library of ISQL, which I will release as soon as possible.
ISQL can be actually using separately from Matrex.
It is just to call a function with, as parameters, the SQL query and the input and output arrays.
I think it can be very interesting for people that wants to use SQL on arrays in memory.
Also in this case the license will be GPL 3.0-

Matrex

There will be a version 1.3 of Matrex.
The idea is to include in this version the last changes made to the source code of Matrex:

• new classloader and plugins structure
• SWT web installer to install SWT from internet
• separated ISQL library

Also, the 2D charts will be updated and new Matrex functions will be added, thanks to the new version of the commons math library .


I'm also working to plan version 2.0 of Matrex. I tested an example of RMI callbacks and I've found that it works very well. I will soon publish a new document about the future Matrex Client/Server architecture.

The SWT Web Installer

You wrote this nice SWT application and you want to release it.
And you realized it is not so easy, especially if the user has to install it himself.
Yes, because there is a different SWT version for each platform: a version for Windows, a version for Mac OSX, a version for Linux... not only, but also different versions for 32 and 64 bits.
What you do? You build a setup file for your application for each platform?

With these thoughts, I wrote a library that does the job for you.
It is called OS dependent installer (osdepinstaller).
You call it from your installer or from your application and it installs the right SWT version for the platform where your application is installed.

It works in this way:

• it loads from the internet a file containing a list of links to download the SWT library from, one for each platform.
  
• It checks the platform in which it is installed (two properties, os.type and os.arch).
• it gets the link for that platform
• it downloads the SWT library (it is a zip file)
  
• it unzips the SWT library in a specific directory

If something goes wrong (for example the platform was not found in the list of links), osdepinstaller offers other solutions to the user.
For example if the automatic choice of the platform does not work, the user is asked to do the choice manually.

A principle of osdepinstaller is to give all the information about what it is doing. In this way the user can continue manually if for some reason the library is not able to continue.

Since osdepinstaller cannot work without internet, and it can happen that the internet can be accessed only through a firewall or a proxy, the system checks with the user if there is a proxy and, in this case, if the proxy needs authorization.

To get data from the user and to show the status osdepinstaller uses by default the console (since it is normal that your application does not have a GUI until osdepinstaller has finished), but it can be easily adapted to other solutions.

Also, osdepinstaller is not only for SWT: it can be used for any library the is platform dependent.

Osdepinstaller is still in test, but you can download a test version and check the code.

I will use osdepinstaller to let the user install SWT with the generic installer of Matrex.

Matrex 1.2 for MacOSX alternative setup

I've got some one reporting me that the Matrex 1.2 version for Mac OSX does not install perfectly on all Macs.
The problem is that sometimes after the setup Matrex does not look like an application but like a simple java folder.
Surely you can open a terminal and run the shell script (the batch file) from there, but this is not really an intuitive way of running it!

I use IzPack to install my software and IzPack claims to be able to install also on Mac OSX, even if from some messages in mailing lists it seems that there could be some problems.

I tried to see what I can do.

First I will modify the shell script so that it can be called from everywhere (an initial statement to let it use the current directory as starting directory).
Then I will update Izpack to the last version (3.11).

But one thing that I would really like to do is to create a MacOSX application bundle containing the Matrex files.
The application bundle is the natural way applications are installed on MacOSX.
To do this you need:

• a .icns file, that contains the application icon.
  
• a way to generate a .dmg file, which is an autoinstalling disk image containing the application bundle.
  

There is no way, as far as I know, to generate .icns and .dmg files outside MacOSX.

So, if someone has a Mac and wants to help me to build this application bundle and to test it, can he please let me know?

List file class loader

Matrex is composed by several jars:

• The Matrex jars: api, gui, fun.
• The libraries used by Matrex, for example SWT, jfreechart.
• The plug-ins, for example the library to connect Matrex to Matlab.

To have java loading all these jars when starting Matrex, they was listed in the classpath parameter of the java call.
Here is the typical content of a batch file starting Matrex (it uses the two variables MATREX_CP and LIB_CP):

MATREX_CP=matrex_api.jar:matrex_fun.jar:matrex_gui.jar
LIB_CP=freehepj3d.jar:jython.jar:commons-math-1.1.jar:jfreechart-1.0.8a.jar:jcommon-1.0.12.jar:jxl.jar:lucene-core-2.0.0.jar:swt/swt.jar:antlr-runtime-3.0.1.jar:javacsv.jar:net.sf.paperclips_1.0.2.jar
java -Djava.library.path=swt -Djava.util.logging.config.file=logging.properties -classpath $MATREX_CP:$LIB_CP matrex.gui.Start $1

This approach has several problems:

• It is not readable, and it becomes less readable adding more libraries.
• If the user wants to add a plug-in, he has to edit the batch file. This is unconfortable, because the user needs to learn the syntax of the batch file, and dangerous: if the user does a small mistake changing the batch file the program does not start.
• you need to have one copy of this batch file for each platform in which Matrex is released (Windows, Linux, MacOSX...).
  

In these days I worked on a new approach. The idea is to have 3 files containing the list of jars needed to start Matrex:

• matrex.cld containing the main Matrex jars
• libraries.cld containing the libraries used by Matrex
• plugis.cld containing the plug-ins (it is empty when Matrex is installed)

Here is an content of the matrex.cld file. The other two files follow the same rules:

matrex_api.jar
matrex_fun.jar
matrex_gui.jar

To load these files and to load the jars listed in them, Matrex uses a custom class loader, TextFilesClassLoader.

The start class of Matrex is now called MatrexLoader.
MatrexLoader loads the main class of Matrex, Start, using TextFilesClassLoader as class loader.
Since Start all the other classes of Matrex (included the ones in the library) are loaded, directly or indirectly, from Start, they all use the same class loader used by Start, TextFilesClassLoader.

In this way it has been also possible to write a special Matrex GUI dialog, PluginsDialog, to update the content of plugins.cld:



This is much easier than to change manually the batch file.

Matrex functions unit testing

I added a small chapter at the end of the function coding document about the Matrex functions code unit testing.


Before version 1.2 some of the Matrex functions had JUnit unit testing, but it was very limited.
In version 1.2 I had to write the code for the ISQL function template, which is very complex compared to the other templates.
So I needed to test it well.
For this reason I wrote some unit testing help classes (for JUnit) that I grouped under the package matrex.item.test.
These classes can be used for the unit testing of any Matrex function template.

With these classes unit testing of matrex functions becomes very easy. For standard function templates it is a matter of minutes to write a test case.
Give a look to them!

Matrex 1.2 released

Finally Matrex 1.2 has been released!

The big step forward of this version is the new ISQL module.
The ISQL module is a function template to apply SQL queries to vectors in a Matrex project.
It is therefore possible:

• to apply filters on parts of the vectors
• to order the elements of the vectors
  
• to aggregate them

all with simple SQL queries. The internalsql document contained in the distributions explains how.
As far as I know, no other spreadsheet application allows something similar.

Other enhancements are:

• Print matrices and presentations.
• Export matrices and presentations to csv and xls.
  
  
  
  
• Import data from database, using an SQL query.
• Menus to remove multiple items, duplicate matrices and functions, add function from a matrix (the function will have the matrix as first argument).
  
  
  
• A summary step for the expression parser.
  
  
• Possibility to change the colors of the windows headers.
  
  
  

Also, several bugs have been removed (dates formatting, items renaming, boolean matrices editing).

Pre-release of Matrex 1.2 to test

I'm in the test phase of the version 1.2 of Matrex.
If someone wants to test this pre-release version himself, he can:

• download the pre-release generic setup
  
• install it
• install the SWT library for his platform in the swt directory of the installation
• use one of the matrix_generic.* files in doc/batch_files to launch Matrex

The changes from version 1.1 are in short:

• The new ISQL (internal SQL) module. There is a document about it in the setup.
• New import and export modules.

If you find some bugs, please report them in the Matrex bugs page.

Thanks

Version 1.2 status

Coding and documentation are finished.
I'm now working on an example project for the ISQL module.
After that I will terminate the test of the system and release it.

There will be a version for:

• Windows 32 bits
• Linux 32 bits
• Mac OSX

and also a generic version, which you can use if you want to install it in a different environment (64 bits?).
The generic version does not install SWT. You have to get it and unzip it yourself under the swt directory.

Status

Today I finally released matrexjruby 1.0.2 (you can find it in the Matrex download page). It took some time to do the final test because of the frenetical work on Matrex 1.2.
I'm now working on the last pieces of code of Matrex 1.2, since ISQL has now been tested and I consider it finished:

• the code that allows to add a function from the matrix tree using as first matrix parameter the selected matrix.
• the code that allows to create an item from the item selection dialog box.

Once these are finished, I need to do some more test and update the documentation. I hope 1.2 will be released in january.

MatrexJRuby

I updated the code of the adapter matrexjruby, which was still based on JRuby 0.9.
Now it works with JRuby 1.01.

Only a few fixes was needed (use the Ruby class instead of the IRuby interface, let the JRuby classes inherit from an abstract class instead of the IFunction interface).

Matrexjruby is used to write the scripts that are the "code" of the Matrex function templates.

I just need some final test and I will publish it as version 1.02.

See the light

Finally!

I 've got stock with the internal sql module for a long while, I'm now able to see some results.
The biggest problem derived from the fact that I made the case... when... then... clause originally too complex (the when clause was too general).

Now I'm able to run some "select ... from ... where..." queries (without group by or order by clauses).
The code is on CVS under the package matrex.fun.sys.sql , so you can give a look yourself if you like.

It looks like it will not take so long before I have an implementation of the group by clause (together with the aggregate functions).
Still some doubts about the order by clause (allow order by or only order by ?).

So, now we should be closer to version 1.2, if nothing comes in the way.

Antlr for the SQL module

To build the internal SQL module for Matrex I decided to work in the following way:

• Write a parser that converts the SQL expression in an internal object structure
• Write the code that applies the parsed SQL to the matrices/vectors arguments of the SQL function.

To do the parsing work, I chose the Antlr library.
Antlr has the following advantages compared to the other parsers:

• More people use it (at least it looks like)
• The parse result can be code in different languages (Java, C#, python...) that can be useful if I want to port the same grammar to other projects
• Together with the library you can download a graphical application called AntlrWorks to interactively test and debug your grammar. AntlrWorks is a very good tool, that let you find errors in your grammar before you start to use it.
  

Antlr is a wonderful product, but I suffered creating the SQL grammar I needed.
The reasons are probably:

• my inexperience in terms of parsers/lexers
• some confusion and some holes in the free documentation

My initial idea was to download Antlr, get a grammar describing the SQL SELECT statement, adapt the grammar to my needs, build the java sources from it and convert the produced AST trees to my internal structures.

Simple, right? Wrong. Here are the problems for this approach:

1. Antlr is in version 3 now (is normally called v3). The example grammars are most made for version 2 (2.7.x). Altough v2 and v3 grammars look very similar, to convert a v2 grammar to a v3 one is not easy.
2. It is possible to buy a book written by the Antlr's author. I did not want to buy the book because I'm not planning to use Antlr in the future. But then I discovered that the online documentation is partial and often referring to the old 2.7.2 version.
3. The produced java classes can have a method to get the AST tree, but as far as I understood the tree cannot be used for an interpreter, but only to check the result of the statement parsing.
4. To build an interpreter is not the only purpose for using a parser:
   Antlr is used for many other things, for example to compile, which
   means convert expressions from a grammar to another one. Consider this to avoid to get confused reading the documentation.
   
   

I struggled for a pair of weeks with these problems. At the end I was able to to understand the following concepts and to produce my grammar:

• You can add java code directly in the grammar. With this code you can build the interpreter structures directly in the generated java code.
• Be very careful about the case of the initial letter of the rule names. Upper case: lexer rule; Lower case : parser rule. It looks simple, but if only the initial letter of one rule name is wrong nothing works as it should.
  
• The lexer is used to parse single words (identifier, strings, numbers). The parser is used to parse phrases.
• Spaces are handled automatically by the parser.
  
• In the java code that you add to the grammar you can set the package of the generated classes.
• AntlrWorks generates two types of java classes: the ones to use in your application and the ones that it uses to debug. They are saved in the same place with the same names. The debug classes don't work in your application, so remember to generate the application classes after a debug session.
  

So, if you give a look to the grammar I have written, you'll see that it is confused (rules definitions together with java code), but it works. When I run the generated java classes against an SQL expression the structures declared in the @members block are populated with the correct values and from them I can interpret the expression.

Version 1.2: status

Not so much has been produced in the last period, for the following reasons:

• holiday
• got sick for one month
• some time used understanding the concepts needed to write a grammar to parse the new internal sql template.

I'm still writing the grammar for the sql template. Soon I will be able to produce some code.
The rest of the coding of version 1.2 is more or less in place.
It clearly needs to be tested.

JRuby released a new version, so I will soon update the Matrex JRuby adapter.

SQL on matrices in Matrex

I'm planning to write a new Matrex template to query matrices (only the ones with one column) using an SQL-like syntax
The idea is simple:

• consider the input matrices/vectors as columns of tables
• query them using SQL
• the SQL query result are the output matrices/vectors of the function.
  

The query is written in a language that is very similar to SQL. The differences are:

• the tables are defined in the query as sets of input matrices
• sub-queries are not allowed

Operators (+, - ...) and aggregate functions (sum, max...) are available.
For example, a query can look like:

select sum(a.price) as sumprice, sum(a.volume) as sumvolume, b.ticker as ticker, b.validity as validity
from [dealticker, price, volume] a, [ticker, validity] b
where a.dealticker = b.ticker
group by b.ticker

Here the matrices dealticker, price, volume are grouped together to form the table a, of which they are columns;
the matrices ticker, validity are grouped together to form the table b, of which they are columns.

Every result column needs to be assigned a name to; that is done using the as keyword.

All the table columns (e.g. a.price, b.ticker...) need to be expressed in the form [table].[column] (e.g. a.price).


This will be one of the most powerful templates in Matrex. With it it will be possible to express very easily and clearly problems that otherwise could require the combination of multiple functions.

Since it is not dependent by new features of the upcoming version 1.2, it is possible that the template will be available before 1.2 is released.

About nebula

Nebula is a sub-project of eclipse that produces custom SWT widgets.
I'm very interested in the grid, because it is supposed to have spreadsheet features like cells selection and row headers, which are very important in Matrex and I can just emulate in the SWT.Table widget.
Nebula Grid is still in alpha state but it should become final when Eclipse 3.3 is released.
I will check it (expecially with a big number of rows/columns) and possibly integrate it in Matrex in version 1.2 or 1.3.

Jython debugger?

You can write Matrex functions as Jython or JRuby scripts.

Jython is included in the Matrex setup, so you can just use it.



To debug these scripts you can use the log functionalities of Matrex.



But today I've found out that there is a better possibility. With PyDev your can remotely debug the script from Eclipse.

So, if a script is not working as it should you can put breakpoints, watch its variables until you don't find the problem.



I will test it myself and write the test results in an article in this blog.

Work started for version 1.2

Work started for version 1.2.



Version 1.2 will indicatively contain:

• Export matrix view and presentation view to Excel file.
• Import matrices from database tables (like the import from Excel files).
• Import matrices from CSV files (like the import from Excel files).
  
• Remove multiple items (matrices, functions...) from a project in one shot.
• Print matrix view and presentation view.
• The matrix source viewer.
  
• More database functions templates (conditional query, update)
• Use more JFreeChart features: composite charts, annotations, pie charts with exploded section, step charts.

With version 1.2 a first pre-alpha version of the Matrex server will be delivered.



1.2 will probably be released in september/october. If some of the new features are not ready for that time, they will be moved to version 1.3.



If you want to see in version 1.2 some feature that I did not consider, just add a comment to this article or send me a mail.

Powered by ScribeFire.

The matrix source viewer

From a mathematical expression the expression parser generates matrices and functions in a Matrex project. This is a powerful functionality in Matrex.



Once the expression parser has generated matrices and functions, the original expression gets lost. This is usally not a problem, since in the project there are all the matrices and functions (final and intermediate) that calculate the expression.

But sometimes you want to see again the original expression, to:

• see in one shot how a matrix has been calculated
• know exactly which matrices and functions have been generated by the expression, for example because you want to replace the original expression with a new one and therefore you need to remove them
  
  

Fortunately the info window shows, among the rest:

• for a matrix, the function that has it as result
• for a function, the input matrices

So it is possible, from the expression's result matrix, to find out the original expression navigating to the source function, to the input matrices and so on.

But it is a lot of work to follow all the links among functions and matrices, expecially if the original expression is long.



To have a more valuable information, version 1.2 will contain a new functionality, called matrix source viewer.



The idea is to have a window that shows this information as a textual expression, with the same format used to enter an expression in the expression parser.



Suppose you entered the expression:



log(x + exp(y + z))



in the expression parser, and gave the name logres to the result matrix.

If you open the matrix source viewer on the logres matrix, you'll see a window containing a text like:



log ( logres.log )



Clicking on logres.log this part of the expression will be expanded and you'll have:



log (x + logres.exp )



Clicking on logres.exp this part of the expression will be expanded and you'll have:



log ( x + exp( y + z))



In this way with a limited number of clicks you'll have the original expression.

Also, a context menu on each item in the expression (excluded the parentheses) will open viewer or editor for that item.



Now the question is: why to see the original expression I need to click on links? Could it not appear as it was immediately?

The problem is that x, y and z could also be the result of other expressions and Matrex does not know which matrices are part of an expression and which are part of another.

The expression parser

While the Expression Parser is not technically fundamental in Matrex (one can use the Function Editor instead), it has become more and more central in the last versions.

The reason is simple: if you have a long expression (or formula as it is called in a classical spreadsheet) to write, opening the Function Editor for each function in the formula can be time consuming and potentially irritating.

Using the Expression Parser is much faster: you write the expression, as it is, in a text editor and in a few steps all the required functions and matrices are added to your project.

The first thing to do to use the Expression Parser is to select the Expression Parser menu in the function tree (or press Ctrl-X).



This opens the Expression Parser wizard:


In the text editor you write the expression to add to the project. The expression is written as a combination of function templates, operators (+,-,*,/,^) and matrices, like:

cos(sin(x) + sin(y))

where x and y are matrices names.

You just need to write the names of the expression parts, their packages are not required.

If you don't remember the name of a function template or a matrix, the Insert.. buttons allow you to add templates and matrices to the expression selecting them.

The expression is first parsed to a tree:

Then the templates repository and the project are queried to determine the complete name-package of each part of the expression:

At the end the name of the target matrix and function are requested. All the intermediate matrices and functions are written in special packages.

Summarizing, entering an expression in Matrex is different from entering a formula in a classic spreadsheet because:

• Spreadsheet formulas work on single cells, Matrex expressions on vectors/matrices.
• Therefore, spreadsheet formulas must be copied for each result cell, Matrex expressions must be written only once.
• Spreadsheet formulas are attached to cells, Matrex expressions are used to generate functions and matrices in the project.
• Therefore spreadsheet formulas can be updated as whole (each cell), whether in Matrex the single generated functions and matrices must be updated individually.
  
  

This last point is not always an advantage for Matrex. Once functions and matrices are generated, the original expression that generated them is lost.

To solve this potential problem, Matrex version 1.2 will have the following new features:

• It will be possible to see the origin of a matrix as an expression: the functions and matrices of which the matrix is directly or indirecly a result. This feature will be object of an article on this blog.
  
• It will be possible to remove multiple items from a project in one shot. In this way it will be possible to easily remove all functions and matrices generated by an expression.
  
• The last step of the expression parser will display all the functions and matrices that will be generated. This will reduce the risk of generating functions and matrices from a wrong expression.
  
  

Help wanted

This week I published a request in SourceForge for Matrex developers.

Matrex gives you fun when you program it and there are always new things to do, new features to add.

But it requires time and it could be nice to have some help.

So, if you are interested please let me know, answering to the announcement or sending me an e-mail.

Powered by ScribeFire.