Pre-release of Matrex 1.2 to test

I'm in the test phase of the version 1.2 of Matrex.
If someone wants to test this pre-release version himself, he can:

• download the pre-release generic setup
  
• install it
• install the SWT library for his platform in the swt directory of the installation
• use one of the matrix_generic.* files in doc/batch_files to launch Matrex

The changes from version 1.1 are in short:

• The new ISQL (internal SQL) module. There is a document about it in the setup.
• New import and export modules.

If you find some bugs, please report them in the Matrex bugs page.

Thanks

Version 1.2 status

Coding and documentation are finished.
I'm now working on an example project for the ISQL module.
After that I will terminate the test of the system and release it.

There will be a version for:

• Windows 32 bits
• Linux 32 bits
• Mac OSX

and also a generic version, which you can use if you want to install it in a different environment (64 bits?).
The generic version does not install SWT. You have to get it and unzip it yourself under the swt directory.

Status

Today I finally released matrexjruby 1.0.2 (you can find it in the Matrex download page). It took some time to do the final test because of the frenetical work on Matrex 1.2.
I'm now working on the last pieces of code of Matrex 1.2, since ISQL has now been tested and I consider it finished:

• the code that allows to add a function from the matrix tree using as first matrix parameter the selected matrix.
• the code that allows to create an item from the item selection dialog box.

Once these are finished, I need to do some more test and update the documentation. I hope 1.2 will be released in january.

MatrexJRuby

I updated the code of the adapter matrexjruby, which was still based on JRuby 0.9.
Now it works with JRuby 1.01.

Only a few fixes was needed (use the Ruby class instead of the IRuby interface, let the JRuby classes inherit from an abstract class instead of the IFunction interface).

Matrexjruby is used to write the scripts that are the "code" of the Matrex function templates.

I just need some final test and I will publish it as version 1.02.

See the light

Finally!

I 've got stock with the internal sql module for a long while, I'm now able to see some results.
The biggest problem derived from the fact that I made the case... when... then... clause originally too complex (the when clause was too general).

Now I'm able to run some "select ... from ... where..." queries (without group by or order by clauses).
The code is on CVS under the package matrex.fun.sys.sql , so you can give a look yourself if you like.

It looks like it will not take so long before I have an implementation of the group by clause (together with the aggregate functions).
Still some doubts about the order by clause (allow order by or only order by ?).

So, now we should be closer to version 1.2, if nothing comes in the way.

Antlr for the SQL module

To build the internal SQL module for Matrex I decided to work in the following way:

• Write a parser that converts the SQL expression in an internal object structure
• Write the code that applies the parsed SQL to the matrices/vectors arguments of the SQL function.

To do the parsing work, I chose the Antlr library.
Antlr has the following advantages compared to the other parsers:

• More people use it (at least it looks like)
• The parse result can be code in different languages (Java, C#, python...) that can be useful if I want to port the same grammar to other projects
• Together with the library you can download a graphical application called AntlrWorks to interactively test and debug your grammar. AntlrWorks is a very good tool, that let you find errors in your grammar before you start to use it.
  

Antlr is a wonderful product, but I suffered creating the SQL grammar I needed.
The reasons are probably:

• my inexperience in terms of parsers/lexers
• some confusion and some holes in the free documentation

My initial idea was to download Antlr, get a grammar describing the SQL SELECT statement, adapt the grammar to my needs, build the java sources from it and convert the produced AST trees to my internal structures.

Simple, right? Wrong. Here are the problems for this approach:

1. Antlr is in version 3 now (is normally called v3). The example grammars are most made for version 2 (2.7.x). Altough v2 and v3 grammars look very similar, to convert a v2 grammar to a v3 one is not easy.
2. It is possible to buy a book written by the Antlr's author. I did not want to buy the book because I'm not planning to use Antlr in the future. But then I discovered that the online documentation is partial and often referring to the old 2.7.2 version.
3. The produced java classes can have a method to get the AST tree, but as far as I understood the tree cannot be used for an interpreter, but only to check the result of the statement parsing.
4. To build an interpreter is not the only purpose for using a parser:
   Antlr is used for many other things, for example to compile, which
   means convert expressions from a grammar to another one. Consider this to avoid to get confused reading the documentation.
   
   

I struggled for a pair of weeks with these problems. At the end I was able to to understand the following concepts and to produce my grammar:

• You can add java code directly in the grammar. With this code you can build the interpreter structures directly in the generated java code.
• Be very careful about the case of the initial letter of the rule names. Upper case: lexer rule; Lower case : parser rule. It looks simple, but if only the initial letter of one rule name is wrong nothing works as it should.
  
• The lexer is used to parse single words (identifier, strings, numbers). The parser is used to parse phrases.
• Spaces are handled automatically by the parser.
  
• In the java code that you add to the grammar you can set the package of the generated classes.
• AntlrWorks generates two types of java classes: the ones to use in your application and the ones that it uses to debug. They are saved in the same place with the same names. The debug classes don't work in your application, so remember to generate the application classes after a debug session.
  

So, if you give a look to the grammar I have written, you'll see that it is confused (rules definitions together with java code), but it works. When I run the generated java classes against an SQL expression the structures declared in the @members block are populated with the correct values and from them I can interpret the expression.

Version 1.2: status

Not so much has been produced in the last period, for the following reasons:

• holiday
• got sick for one month
• some time used understanding the concepts needed to write a grammar to parse the new internal sql template.

I'm still writing the grammar for the sql template. Soon I will be able to produce some code.
The rest of the coding of version 1.2 is more or less in place.
It clearly needs to be tested.

JRuby released a new version, so I will soon update the Matrex JRuby adapter.

SQL on matrices in Matrex

I'm planning to write a new Matrex template to query matrices (only the ones with one column) using an SQL-like syntax
The idea is simple:

• consider the input matrices/vectors as columns of tables
• query them using SQL
• the SQL query result are the output matrices/vectors of the function.
  

The query is written in a language that is very similar to SQL. The differences are:

• the tables are defined in the query as sets of input matrices
• sub-queries are not allowed

Operators (+, - ...) and aggregate functions (sum, max...) are available.
For example, a query can look like:

select sum(a.price) as sumprice, sum(a.volume) as sumvolume, b.ticker as ticker, b.validity as validity
from [dealticker, price, volume] a, [ticker, validity] b
where a.dealticker = b.ticker
group by b.ticker

Here the matrices dealticker, price, volume are grouped together to form the table a, of which they are columns;
the matrices ticker, validity are grouped together to form the table b, of which they are columns.

Every result column needs to be assigned a name to; that is done using the as keyword.

All the table columns (e.g. a.price, b.ticker...) need to be expressed in the form [table].[column] (e.g. a.price).


This will be one of the most powerful templates in Matrex. With it it will be possible to express very easily and clearly problems that otherwise could require the combination of multiple functions.

Since it is not dependent by new features of the upcoming version 1.2, it is possible that the template will be available before 1.2 is released.

About nebula

Nebula is a sub-project of eclipse that produces custom SWT widgets.
I'm very interested in the grid, because it is supposed to have spreadsheet features like cells selection and row headers, which are very important in Matrex and I can just emulate in the SWT.Table widget.
Nebula Grid is still in alpha state but it should become final when Eclipse 3.3 is released.
I will check it (expecially with a big number of rows/columns) and possibly integrate it in Matrex in version 1.2 or 1.3.

Jython debugger?

You can write Matrex functions as Jython or JRuby scripts.

Jython is included in the Matrex setup, so you can just use it.



To debug these scripts you can use the log functionalities of Matrex.



But today I've found out that there is a better possibility. With PyDev your can remotely debug the script from Eclipse.

So, if a script is not working as it should you can put breakpoints, watch its variables until you don't find the problem.



I will test it myself and write the test results in an article in this blog.

Work started for version 1.2

Work started for version 1.2.



Version 1.2 will indicatively contain:

• Export matrix view and presentation view to Excel file.
• Import matrices from database tables (like the import from Excel files).
• Import matrices from CSV files (like the import from Excel files).
  
• Remove multiple items (matrices, functions...) from a project in one shot.
• Print matrix view and presentation view.
• The matrix source viewer.
  
• More database functions templates (conditional query, update)
• Use more JFreeChart features: composite charts, annotations, pie charts with exploded section, step charts.

With version 1.2 a first pre-alpha version of the Matrex server will be delivered.



1.2 will probably be released in september/october. If some of the new features are not ready for that time, they will be moved to version 1.3.



If you want to see in version 1.2 some feature that I did not consider, just add a comment to this article or send me a mail.

Powered by ScribeFire.

The matrix source viewer

From a mathematical expression the expression parser generates matrices and functions in a Matrex project. This is a powerful functionality in Matrex.



Once the expression parser has generated matrices and functions, the original expression gets lost. This is usally not a problem, since in the project there are all the matrices and functions (final and intermediate) that calculate the expression.

But sometimes you want to see again the original expression, to:

• see in one shot how a matrix has been calculated
• know exactly which matrices and functions have been generated by the expression, for example because you want to replace the original expression with a new one and therefore you need to remove them
  
  

Fortunately the info window shows, among the rest:

• for a matrix, the function that has it as result
• for a function, the input matrices

So it is possible, from the expression's result matrix, to find out the original expression navigating to the source function, to the input matrices and so on.

But it is a lot of work to follow all the links among functions and matrices, expecially if the original expression is long.



To have a more valuable information, version 1.2 will contain a new functionality, called matrix source viewer.



The idea is to have a window that shows this information as a textual expression, with the same format used to enter an expression in the expression parser.



Suppose you entered the expression:



log(x + exp(y + z))



in the expression parser, and gave the name logres to the result matrix.

If you open the matrix source viewer on the logres matrix, you'll see a window containing a text like:



log ( logres.log )



Clicking on logres.log this part of the expression will be expanded and you'll have:



log (x + logres.exp )



Clicking on logres.exp this part of the expression will be expanded and you'll have:



log ( x + exp( y + z))



In this way with a limited number of clicks you'll have the original expression.

Also, a context menu on each item in the expression (excluded the parentheses) will open viewer or editor for that item.



Now the question is: why to see the original expression I need to click on links? Could it not appear as it was immediately?

The problem is that x, y and z could also be the result of other expressions and Matrex does not know which matrices are part of an expression and which are part of another.

The expression parser

While the Expression Parser is not technically fundamental in Matrex (one can use the Function Editor instead), it has become more and more central in the last versions.

The reason is simple: if you have a long expression (or formula as it is called in a classical spreadsheet) to write, opening the Function Editor for each function in the formula can be time consuming and potentially irritating.

Using the Expression Parser is much faster: you write the expression, as it is, in a text editor and in a few steps all the required functions and matrices are added to your project.

The first thing to do to use the Expression Parser is to select the Expression Parser menu in the function tree (or press Ctrl-X).



This opens the Expression Parser wizard:


In the text editor you write the expression to add to the project. The expression is written as a combination of function templates, operators (+,-,*,/,^) and matrices, like:

cos(sin(x) + sin(y))

where x and y are matrices names.

You just need to write the names of the expression parts, their packages are not required.

If you don't remember the name of a function template or a matrix, the Insert.. buttons allow you to add templates and matrices to the expression selecting them.

The expression is first parsed to a tree:

Then the templates repository and the project are queried to determine the complete name-package of each part of the expression:

At the end the name of the target matrix and function are requested. All the intermediate matrices and functions are written in special packages.

Summarizing, entering an expression in Matrex is different from entering a formula in a classic spreadsheet because:

• Spreadsheet formulas work on single cells, Matrex expressions on vectors/matrices.
• Therefore, spreadsheet formulas must be copied for each result cell, Matrex expressions must be written only once.
• Spreadsheet formulas are attached to cells, Matrex expressions are used to generate functions and matrices in the project.
• Therefore spreadsheet formulas can be updated as whole (each cell), whether in Matrex the single generated functions and matrices must be updated individually.
  
  

This last point is not always an advantage for Matrex. Once functions and matrices are generated, the original expression that generated them is lost.

To solve this potential problem, Matrex version 1.2 will have the following new features:

• It will be possible to see the origin of a matrix as an expression: the functions and matrices of which the matrix is directly or indirecly a result. This feature will be object of an article on this blog.
  
• It will be possible to remove multiple items from a project in one shot. In this way it will be possible to easily remove all functions and matrices generated by an expression.
  
• The last step of the expression parser will display all the functions and matrices that will be generated. This will reduce the risk of generating functions and matrices from a wrong expression.
  
  

Help wanted

This week I published a request in SourceForge for Matrex developers.

Matrex gives you fun when you program it and there are always new things to do, new features to add.

But it requires time and it could be nice to have some help.

So, if you are interested please let me know, answering to the announcement or sending me an e-mail.

Powered by ScribeFire.

5000 downloads!

I hope for a much shorter time to come to 10000.

Powered by ScribeFire.

Article about Matrex in an internet japanese magazine

This is the link to the article, written in Japanese.

Matrex 1.1 released!

I'm glad to announce that Matrex 1.1 has been released.
The purpose of this version is to make the system more open, easier to use and complete.
The changes are the following:

• Matrex is now able to import xls (Excel) files: open a Matrex project, from the project menu open the import dialog and load the Excel file:
  
  
  Here you can select rectangles of cells that you import as matrices in the project.
  
  
• Matrex has now a context help. Pressing Ctrl-F1 (or F1) in each window, the browser shows the help page for that window.
  A copy of the help is available online.
  The help has also a summary page and can be full-text searched in the search dialog:
  
  
  
  
  
• The 2D charts have been simplified and improved. Now there are only 3 kinds of charts (category, xy, time), but you can display their series in up to 7 different ways (line, line3d, bar, bar3d, area, scattered, scattered line):
  
  
  
• The Expression Parser is now able to interpret expressions containing +,-,*,/,^ operators. The operators are translated to the related functions.
  Now it becomes really easy to build projects in Matrex.
  
  
  

Matrex 1.1: some delays...

The development is terminated, the documentation is upgraded.
Even the changes in the Matrex site are done (FAQ, new videos).

The problem is the final test.
It takes time, and I will not be able to release Matrex 1.1 before the middle of April.

SWT: cell selection in Table

The SWT Table allows only to select rows, not the cells in the rows.
This is a strong limitation for applications that use tables to do something more complex than showing the content of database tables.

In Matrex I needed to import xls (Excel) files and select from them rectangles of cells to add as matrices.
So, I built the TableWithCellSelection class, which allows the user to select rectangles of cells in the table.



It works listening the mouseMove, mouseDown and mouseUp events to keep track of the mouse movements and to change the background of the cells when they are selected.
The class works only for tables that are created with the SWT.FULL_SELECTION and SWT.VIRTUAL styles, which means that the table must be virtual.

Features: Clicking SHIFT together with the left mouse button the selection process restarts from a previously selected cells rectangle.

Limitations: It is not possible to scroll the table during the selection. To be able to select big rectangles you can use the SHIFT key.

Matrex 1.1 status

Matrex 1.1 coding should be finished.
What is missing is testing and documentation update.
The hope is to release it before easter.