Status

Today I finally released matrexjruby 1.0.2 (you can find it in the Matrex download page). It took some time to do the final test because of the frenetical work on Matrex 1.2.
I'm now working on the last pieces of code of Matrex 1.2, since ISQL has now been tested and I consider it finished:

• the code that allows to add a function from the matrix tree using as first matrix parameter the selected matrix.
• the code that allows to create an item from the item selection dialog box.

Once these are finished, I need to do some more test and update the documentation. I hope 1.2 will be released in january.

MatrexJRuby

I updated the code of the adapter matrexjruby, which was still based on JRuby 0.9.
Now it works with JRuby 1.01.

Only a few fixes was needed (use the Ruby class instead of the IRuby interface, let the JRuby classes inherit from an abstract class instead of the IFunction interface).

Matrexjruby is used to write the scripts that are the "code" of the Matrex function templates.

I just need some final test and I will publish it as version 1.02.

See the light

Finally!

I 've got stock with the internal sql module for a long while, I'm now able to see some results.
The biggest problem derived from the fact that I made the case... when... then... clause originally too complex (the when clause was too general).

Now I'm able to run some "select ... from ... where..." queries (without group by or order by clauses).
The code is on CVS under the package matrex.fun.sys.sql , so you can give a look yourself if you like.

It looks like it will not take so long before I have an implementation of the group by clause (together with the aggregate functions).
Still some doubts about the order by clause (allow order by or only order by ?).

So, now we should be closer to version 1.2, if nothing comes in the way.

Antlr for the SQL module

To build the internal SQL module for Matrex I decided to work in the following way:

• Write a parser that converts the SQL expression in an internal object structure
• Write the code that applies the parsed SQL to the matrices/vectors arguments of the SQL function.

To do the parsing work, I chose the Antlr library.
Antlr has the following advantages compared to the other parsers:

• More people use it (at least it looks like)
• The parse result can be code in different languages (Java, C#, python...) that can be useful if I want to port the same grammar to other projects
• Together with the library you can download a graphical application called AntlrWorks to interactively test and debug your grammar. AntlrWorks is a very good tool, that let you find errors in your grammar before you start to use it.
  

Antlr is a wonderful product, but I suffered creating the SQL grammar I needed.
The reasons are probably:

• my inexperience in terms of parsers/lexers
• some confusion and some holes in the free documentation

My initial idea was to download Antlr, get a grammar describing the SQL SELECT statement, adapt the grammar to my needs, build the java sources from it and convert the produced AST trees to my internal structures.

Simple, right? Wrong. Here are the problems for this approach:

1. Antlr is in version 3 now (is normally called v3). The example grammars are most made for version 2 (2.7.x). Altough v2 and v3 grammars look very similar, to convert a v2 grammar to a v3 one is not easy.
2. It is possible to buy a book written by the Antlr's author. I did not want to buy the book because I'm not planning to use Antlr in the future. But then I discovered that the online documentation is partial and often referring to the old 2.7.2 version.
3. The produced java classes can have a method to get the AST tree, but as far as I understood the tree cannot be used for an interpreter, but only to check the result of the statement parsing.
4. To build an interpreter is not the only purpose for using a parser:
   Antlr is used for many other things, for example to compile, which
   means convert expressions from a grammar to another one. Consider this to avoid to get confused reading the documentation.
   
   

I struggled for a pair of weeks with these problems. At the end I was able to to understand the following concepts and to produce my grammar:

• You can add java code directly in the grammar. With this code you can build the interpreter structures directly in the generated java code.
• Be very careful about the case of the initial letter of the rule names. Upper case: lexer rule; Lower case : parser rule. It looks simple, but if only the initial letter of one rule name is wrong nothing works as it should.
  
• The lexer is used to parse single words (identifier, strings, numbers). The parser is used to parse phrases.
• Spaces are handled automatically by the parser.
  
• In the java code that you add to the grammar you can set the package of the generated classes.
• AntlrWorks generates two types of java classes: the ones to use in your application and the ones that it uses to debug. They are saved in the same place with the same names. The debug classes don't work in your application, so remember to generate the application classes after a debug session.
  

So, if you give a look to the grammar I have written, you'll see that it is confused (rules definitions together with java code), but it works. When I run the generated java classes against an SQL expression the structures declared in the @members block are populated with the correct values and from them I can interpret the expression.

Version 1.2: status

Not so much has been produced in the last period, for the following reasons:

• holiday
• got sick for one month
• some time used understanding the concepts needed to write a grammar to parse the new internal sql template.

I'm still writing the grammar for the sql template. Soon I will be able to produce some code.
The rest of the coding of version 1.2 is more or less in place.
It clearly needs to be tested.

JRuby released a new version, so I will soon update the Matrex JRuby adapter.

SQL on matrices in Matrex

I'm planning to write a new Matrex template to query matrices (only the ones with one column) using an SQL-like syntax
The idea is simple:

• consider the input matrices/vectors as columns of tables
• query them using SQL
• the SQL query result are the output matrices/vectors of the function.
  

The query is written in a language that is very similar to SQL. The differences are:

• the tables are defined in the query as sets of input matrices
• sub-queries are not allowed

Operators (+, - ...) and aggregate functions (sum, max...) are available.
For example, a query can look like:

select sum(a.price) as sumprice, sum(a.volume) as sumvolume, b.ticker as ticker, b.validity as validity
from [dealticker, price, volume] a, [ticker, validity] b
where a.dealticker = b.ticker
group by b.ticker

Here the matrices dealticker, price, volume are grouped together to form the table a, of which they are columns;
the matrices ticker, validity are grouped together to form the table b, of which they are columns.

Every result column needs to be assigned a name to; that is done using the as keyword.

All the table columns (e.g. a.price, b.ticker...) need to be expressed in the form [table].[column] (e.g. a.price).


This will be one of the most powerful templates in Matrex. With it it will be possible to express very easily and clearly problems that otherwise could require the combination of multiple functions.

Since it is not dependent by new features of the upcoming version 1.2, it is possible that the template will be available before 1.2 is released.

About nebula

Nebula is a sub-project of eclipse that produces custom SWT widgets.
I'm very interested in the grid, because it is supposed to have spreadsheet features like cells selection and row headers, which are very important in Matrex and I can just emulate in the SWT.Table widget.
Nebula Grid is still in alpha state but it should become final when Eclipse 3.3 is released.
I will check it (expecially with a big number of rows/columns) and possibly integrate it in Matrex in version 1.2 or 1.3.

Jython debugger?

You can write Matrex functions as Jython or JRuby scripts.

Jython is included in the Matrex setup, so you can just use it.



To debug these scripts you can use the log functionalities of Matrex.



But today I've found out that there is a better possibility. With PyDev your can remotely debug the script from Eclipse.

So, if a script is not working as it should you can put breakpoints, watch its variables until you don't find the problem.



I will test it myself and write the test results in an article in this blog.

Work started for version 1.2

Work started for version 1.2.



Version 1.2 will indicatively contain:

• Export matrix view and presentation view to Excel file.
• Import matrices from database tables (like the import from Excel files).
• Import matrices from CSV files (like the import from Excel files).
  
• Remove multiple items (matrices, functions...) from a project in one shot.
• Print matrix view and presentation view.
• The matrix source viewer.
  
• More database functions templates (conditional query, update)
• Use more JFreeChart features: composite charts, annotations, pie charts with exploded section, step charts.

With version 1.2 a first pre-alpha version of the Matrex server will be delivered.



1.2 will probably be released in september/october. If some of the new features are not ready for that time, they will be moved to version 1.3.



If you want to see in version 1.2 some feature that I did not consider, just add a comment to this article or send me a mail.

Powered by ScribeFire.

The matrix source viewer

From a mathematical expression the expression parser generates matrices and functions in a Matrex project. This is a powerful functionality in Matrex.



Once the expression parser has generated matrices and functions, the original expression gets lost. This is usally not a problem, since in the project there are all the matrices and functions (final and intermediate) that calculate the expression.

But sometimes you want to see again the original expression, to:

• see in one shot how a matrix has been calculated
• know exactly which matrices and functions have been generated by the expression, for example because you want to replace the original expression with a new one and therefore you need to remove them
  
  

Fortunately the info window shows, among the rest:

• for a matrix, the function that has it as result
• for a function, the input matrices

So it is possible, from the expression's result matrix, to find out the original expression navigating to the source function, to the input matrices and so on.

But it is a lot of work to follow all the links among functions and matrices, expecially if the original expression is long.



To have a more valuable information, version 1.2 will contain a new functionality, called matrix source viewer.



The idea is to have a window that shows this information as a textual expression, with the same format used to enter an expression in the expression parser.



Suppose you entered the expression:



log(x + exp(y + z))



in the expression parser, and gave the name logres to the result matrix.

If you open the matrix source viewer on the logres matrix, you'll see a window containing a text like:



log ( logres.log )



Clicking on logres.log this part of the expression will be expanded and you'll have:



log (x + logres.exp )



Clicking on logres.exp this part of the expression will be expanded and you'll have:



log ( x + exp( y + z))



In this way with a limited number of clicks you'll have the original expression.

Also, a context menu on each item in the expression (excluded the parentheses) will open viewer or editor for that item.



Now the question is: why to see the original expression I need to click on links? Could it not appear as it was immediately?

The problem is that x, y and z could also be the result of other expressions and Matrex does not know which matrices are part of an expression and which are part of another.

The expression parser

While the Expression Parser is not technically fundamental in Matrex (one can use the Function Editor instead), it has become more and more central in the last versions.

The reason is simple: if you have a long expression (or formula as it is called in a classical spreadsheet) to write, opening the Function Editor for each function in the formula can be time consuming and potentially irritating.

Using the Expression Parser is much faster: you write the expression, as it is, in a text editor and in a few steps all the required functions and matrices are added to your project.

The first thing to do to use the Expression Parser is to select the Expression Parser menu in the function tree (or press Ctrl-X).



This opens the Expression Parser wizard:


In the text editor you write the expression to add to the project. The expression is written as a combination of function templates, operators (+,-,*,/,^) and matrices, like:

cos(sin(x) + sin(y))

where x and y are matrices names.

You just need to write the names of the expression parts, their packages are not required.

If you don't remember the name of a function template or a matrix, the Insert.. buttons allow you to add templates and matrices to the expression selecting them.

The expression is first parsed to a tree:

Then the templates repository and the project are queried to determine the complete name-package of each part of the expression:

At the end the name of the target matrix and function are requested. All the intermediate matrices and functions are written in special packages.

Summarizing, entering an expression in Matrex is different from entering a formula in a classic spreadsheet because:

• Spreadsheet formulas work on single cells, Matrex expressions on vectors/matrices.
• Therefore, spreadsheet formulas must be copied for each result cell, Matrex expressions must be written only once.
• Spreadsheet formulas are attached to cells, Matrex expressions are used to generate functions and matrices in the project.
• Therefore spreadsheet formulas can be updated as whole (each cell), whether in Matrex the single generated functions and matrices must be updated individually.
  
  

This last point is not always an advantage for Matrex. Once functions and matrices are generated, the original expression that generated them is lost.

To solve this potential problem, Matrex version 1.2 will have the following new features:

• It will be possible to see the origin of a matrix as an expression: the functions and matrices of which the matrix is directly or indirecly a result. This feature will be object of an article on this blog.
  
• It will be possible to remove multiple items from a project in one shot. In this way it will be possible to easily remove all functions and matrices generated by an expression.
  
• The last step of the expression parser will display all the functions and matrices that will be generated. This will reduce the risk of generating functions and matrices from a wrong expression.
  
  

Help wanted

This week I published a request in SourceForge for Matrex developers.

Matrex gives you fun when you program it and there are always new things to do, new features to add.

But it requires time and it could be nice to have some help.

So, if you are interested please let me know, answering to the announcement or sending me an e-mail.

Powered by ScribeFire.

5000 downloads!

I hope for a much shorter time to come to 10000.

Powered by ScribeFire.

Article about Matrex in an internet japanese magazine

This is the link to the article, written in Japanese.

Matrex 1.1 released!

I'm glad to announce that Matrex 1.1 has been released.
The purpose of this version is to make the system more open, easier to use and complete.
The changes are the following:

• Matrex is now able to import xls (Excel) files: open a Matrex project, from the project menu open the import dialog and load the Excel file:
  
  
  Here you can select rectangles of cells that you import as matrices in the project.
  
  
• Matrex has now a context help. Pressing Ctrl-F1 (or F1) in each window, the browser shows the help page for that window.
  A copy of the help is available online.
  The help has also a summary page and can be full-text searched in the search dialog:
  
  
  
  
  
• The 2D charts have been simplified and improved. Now there are only 3 kinds of charts (category, xy, time), but you can display their series in up to 7 different ways (line, line3d, bar, bar3d, area, scattered, scattered line):
  
  
  
• The Expression Parser is now able to interpret expressions containing +,-,*,/,^ operators. The operators are translated to the related functions.
  Now it becomes really easy to build projects in Matrex.
  
  
  

Matrex 1.1: some delays...

The development is terminated, the documentation is upgraded.
Even the changes in the Matrex site are done (FAQ, new videos).

The problem is the final test.
It takes time, and I will not be able to release Matrex 1.1 before the middle of April.

SWT: cell selection in Table

The SWT Table allows only to select rows, not the cells in the rows.
This is a strong limitation for applications that use tables to do something more complex than showing the content of database tables.

In Matrex I needed to import xls (Excel) files and select from them rectangles of cells to add as matrices.
So, I built the TableWithCellSelection class, which allows the user to select rectangles of cells in the table.



It works listening the mouseMove, mouseDown and mouseUp events to keep track of the mouse movements and to change the background of the cells when they are selected.
The class works only for tables that are created with the SWT.FULL_SELECTION and SWT.VIRTUAL styles, which means that the table must be virtual.

Features: Clicking SHIFT together with the left mouse button the selection process restarts from a previously selected cells rectangle.

Limitations: It is not possible to scroll the table during the selection. To be able to select big rectangles you can use the SHIFT key.

Matrex 1.1 status

Matrex 1.1 coding should be finished.
What is missing is testing and documentation update.
The hope is to release it before easter.

More features in Matrex 1.1

The next release of Matrex, 1.1, will also contain the following features:

• use of operators (+,-,*,/,^) in the expression parser. They will be converted respectively in the functions plus, minus, times, divide, pow.
  
  
• import and export of projects from/to zip files. In this way it will be possible for example to archive a project or to send it by mail.
  

New symbol

Matrex got a new symbol, used both for the application icon and for the graphics of its web site.

The symbol should suggest the idea of a matrix, with the word "Matrex" written over it.
It should look more professional than the old falcon symbol, that did not really have so much to do with the main concepts of Matrex.

I made the symbol using OpenOffice.org and GIMP.

Why should I use Matrex if Excel 2007 supports multithreading?

One of the peculiar features of Matrex has always been the fact that functions are executed in different threads.
Now Excel 2007 supports a similar feature, so one could think that there are less reasons to use Matrex instead of Excel.
But that's wrong: the multithreading capabilities offered by the two products are different.

Purposes

Excel

Multithreading has been added to Excel to use both the cores in the new double core processors (Intel Core Duo and AMD X2).

Matrex

Matrex has introduced multithreading before these processors became popular and the purposes are the following:

• Don't block the GUI during the function calculations
• Get immediately the results of some calculation without to wait for the whole project (worksheet) to be recalculated.
  
• Avoid that I/O dependent functions (formulas) block CPU intensive functions.

Sure, Matrex is also able to use both the cores of a double core processor, but that is only a side effect of its multithreading feature.

Implementation

Excel

I don't have details on the implementation, but that is what I think happens:

When you recalculate a sheet the first time, all the dependencies among cells of the sheet (formulas) are examined and based on this the sheet is divided in 2 parts (or 4 if you have 4 cores) .
Each part is calculated by a thread.

Matrex

Matrex uses a pool of threads. Each function is calculated by a thread of the pool.
In this way the whole Matrex internal engine is multithreaded.
Only the GUI runs as a single thread, with which the other threads need to synchronize when matrices, charts or presentations are displayed in the viewers.


Conclusions

Multithreading has been just added to Excel to use well the dual core processors, but it is not a main feature in Excel: a part of the fact that the calculation is divided in 2 or more threads, the system works as in the previous versions.

Matrex is instead built on threads. They are everywhere: functions, matrices, presentations, charts, timers and the GUI have to cope with them.

Printing support

To print, version 1.0 of Matrex converts the content of the table (matrix or presentation) to a web page and opens a browser with it. So you can print the web page from the browser.

This works fine, but is not very professional. So why I don't print directly from Matrex? The reason is that SWT, the user interface library I'm using, does still not allow to print in Linux.

But it seems that this will change in SWT version 3.3. So I will surely add the possibility to print directly from Matrex in version 1.2, or may be even 1.1 of Matrex.

Help on line

Today Matrex got an help on line!

The plan for version 1.1 of Matrex includes a context help.
The context help is made as a set of linked html pages, one for each window and dialog of Matrex, so that it is possible to open the right page when pressing F1 on one window.
This feature is more or less ready and I thought that it could be nice to have already all the help html pages as a help on the web.

So, if you have a problem understanding what is the purpose of the view, open the content page of the help, click on link for the window and read!

The content page is also reachable from the documents web page of the Matrex site.

Matrex 1.1 plan

It is time to start to work on the version 1.1 of Matrex.
The main new features of Matrex 1.1 will be:

• Context Help
• Excel files (.xls) import.
• Possibility to use bar series and line series in the same chart.

Context help

One of the features of the next version 1.1 will be the context help.
This means that pressing F1 in a window Matrex shows some help for that window.
This will be a winning feature for Matrex, because it reduces the time needed to learn how the system works.

The content of the help will come from the PDF document How to use Matrex.
The idea is to make a map of all the windows in Matrex and assign a piece of the document as an HTML page to each window.
Then refine the help text to make sense for each window and add links among the various help pages.

To maintain the help system and the PDF document updated in the next versions I will:

• update the HTML pages of the help (change them, make new pages)
• build the PDF document from the HTML pages using HTMLDOC. HTMLDOC is able to merge several HTML pages in one PDF document and is free.