Matrex 1.1: some delays...

The development is terminated, the documentation is upgraded.
Even the changes in the Matrex site are done (FAQ, new videos).

The problem is the final test.
It takes time, and I will not be able to release Matrex 1.1 before the middle of April.

SWT: cell selection in Table

The SWT Table allows only to select rows, not the cells in the rows.
This is a strong limitation for applications that use tables to do something more complex than showing the content of database tables.

In Matrex I needed to import xls (Excel) files and select from them rectangles of cells to add as matrices.
So, I built the TableWithCellSelection class, which allows the user to select rectangles of cells in the table.



It works listening the mouseMove, mouseDown and mouseUp events to keep track of the mouse movements and to change the background of the cells when they are selected.
The class works only for tables that are created with the SWT.FULL_SELECTION and SWT.VIRTUAL styles, which means that the table must be virtual.

Features: Clicking SHIFT together with the left mouse button the selection process restarts from a previously selected cells rectangle.

Limitations: It is not possible to scroll the table during the selection. To be able to select big rectangles you can use the SHIFT key.

Matrex 1.1 status

Matrex 1.1 coding should be finished.
What is missing is testing and documentation update.
The hope is to release it before easter.

More features in Matrex 1.1

The next release of Matrex, 1.1, will also contain the following features:

• use of operators (+,-,*,/,^) in the expression parser. They will be converted respectively in the functions plus, minus, times, divide, pow.
  
  
• import and export of projects from/to zip files. In this way it will be possible for example to archive a project or to send it by mail.
  

New symbol

Matrex got a new symbol, used both for the application icon and for the graphics of its web site.

The symbol should suggest the idea of a matrix, with the word "Matrex" written over it.
It should look more professional than the old falcon symbol, that did not really have so much to do with the main concepts of Matrex.

I made the symbol using OpenOffice.org and GIMP.

Why should I use Matrex if Excel 2007 supports multithreading?

One of the peculiar features of Matrex has always been the fact that functions are executed in different threads.
Now Excel 2007 supports a similar feature, so one could think that there are less reasons to use Matrex instead of Excel.
But that's wrong: the multithreading capabilities offered by the two products are different.

Purposes

Excel

Multithreading has been added to Excel to use both the cores in the new double core processors (Intel Core Duo and AMD X2).

Matrex

Matrex has introduced multithreading before these processors became popular and the purposes are the following:

• Don't block the GUI during the function calculations
• Get immediately the results of some calculation without to wait for the whole project (worksheet) to be recalculated.
  
• Avoid that I/O dependent functions (formulas) block CPU intensive functions.

Sure, Matrex is also able to use both the cores of a double core processor, but that is only a side effect of its multithreading feature.

Implementation

Excel

I don't have details on the implementation, but that is what I think happens:

When you recalculate a sheet the first time, all the dependencies among cells of the sheet (formulas) are examined and based on this the sheet is divided in 2 parts (or 4 if you have 4 cores) .
Each part is calculated by a thread.

Matrex

Matrex uses a pool of threads. Each function is calculated by a thread of the pool.
In this way the whole Matrex internal engine is multithreaded.
Only the GUI runs as a single thread, with which the other threads need to synchronize when matrices, charts or presentations are displayed in the viewers.


Conclusions

Multithreading has been just added to Excel to use well the dual core processors, but it is not a main feature in Excel: a part of the fact that the calculation is divided in 2 or more threads, the system works as in the previous versions.

Matrex is instead built on threads. They are everywhere: functions, matrices, presentations, charts, timers and the GUI have to cope with them.

Printing support

To print, version 1.0 of Matrex converts the content of the table (matrix or presentation) to a web page and opens a browser with it. So you can print the web page from the browser.

This works fine, but is not very professional. So why I don't print directly from Matrex? The reason is that SWT, the user interface library I'm using, does still not allow to print in Linux.

But it seems that this will change in SWT version 3.3. So I will surely add the possibility to print directly from Matrex in version 1.2, or may be even 1.1 of Matrex.

Help on line

Today Matrex got an help on line!

The plan for version 1.1 of Matrex includes a context help.
The context help is made as a set of linked html pages, one for each window and dialog of Matrex, so that it is possible to open the right page when pressing F1 on one window.
This feature is more or less ready and I thought that it could be nice to have already all the help html pages as a help on the web.

So, if you have a problem understanding what is the purpose of the view, open the content page of the help, click on link for the window and read!

The content page is also reachable from the documents web page of the Matrex site.

Matrex 1.1 plan

It is time to start to work on the version 1.1 of Matrex.
The main new features of Matrex 1.1 will be:

• Context Help
• Excel files (.xls) import.
• Possibility to use bar series and line series in the same chart.

Context help

One of the features of the next version 1.1 will be the context help.
This means that pressing F1 in a window Matrex shows some help for that window.
This will be a winning feature for Matrex, because it reduces the time needed to learn how the system works.

The content of the help will come from the PDF document How to use Matrex.
The idea is to make a map of all the windows in Matrex and assign a piece of the document as an HTML page to each window.
Then refine the help text to make sense for each window and add links among the various help pages.

To maintain the help system and the PDF document updated in the next versions I will:

• update the HTML pages of the help (change them, make new pages)
• build the PDF document from the HTML pages using HTMLDOC. HTMLDOC is able to merge several HTML pages in one PDF document and is free.
  

Mac OSX Matrex start

In the Matrex Sourceforge pages Kusashi Nameta (JPN) reports the bug that the Mac OSX version of Matrex does not start correctly.

The problem is that I can test the system only on Windows and Linux, so things like that can happen. There should be someone testing the Mac version before it is published.

The solution, as you can see in my comments, is to add the -XstartOnFirstThread option to the java command.
To do that, edit the matrex_macosx.sh file in the Matrex directory and to:

java

subtitute:

java -XstartOnFirstThread

This should make it work correctly.
I'll publish myself a patched version of the Mac OSX setup file this weekend.

What now?

Plans

After the release of Matrex version 1.0, it is time to make some plans for the future.

By now, I started to write the specification for the client/server architecture. This should start the process that takes to version 2.0 of Matrex.

After that:

• some adapters must be updated: I'm thinking about the adapter to JRuby, which must at least be tested for the last version of JRuby and the adapter to Scilab that will soon reach version 4.1.
• as usual I will try to test the adapter to Matlab, which has been ready for one year, but was never tested because I don't have Matlab. If someone wants to help, please contact me.
• I will try to produce an adapter for the R statistical language.
• start of the work to produce version 1.1.

Next versions

First of all, this is how I will number the next versions of Matrex:

• A change in the first number means a major change: 2.0 will introduce the client server architecture.

• A change in the second number means new important (visible) features: 1.1 will introduce the package to interact with Excel, the help framework and changes in the charts.

• A change in the second number means bug fixing and minor feature changes: if a version 1.0.1 will be released, it will fix bugs in version 1.0.

I think these are the features to add in the next versions:

• Package for the interaction with Excel: should contain import/export of Excel spreadsheet files, communication Excel->Matrex and Matrex->Excel via Java/COM.
• Package for the interaction with OpenOffice.org: communication OpenOffice.org->Matrex and Matrex->OpenOffice.org (import/export can be done through .xls files).
• Charts new features: scatter plot chart, possibility to have bars and lines in the same chart, chart attributes like line or bar type, vertical/horizontal bars.
  
• Context help. Convert the "How To Use" document to a context help in the Matrex desktop application.
• Add first phase in the expression parser where expressions of the type "a + b" (+,-,*,/,^) are converted to expressions of the type "plus(a,b)". This makes the expression parser simpler to understand.
  
• Zip/unzip of projects, so that they can be easily sent to other users.
  
• Custom dialogs to enter function parameters. They are needed for complex functions like query, csv and, in the future, link.

The idea is to publish these features in a set of 1.x versions (1.1, 1.2 ...) .
These versions should be released every 2/3 months to keep up the interest for the product.

The first major version will be Matrex 2.0, which will introduce the client/server Matrex.

Version 2.1 will (probably) introduce links between projects (also between projects running in different machines).

No more M1, M2 version will published; an RC1 version (or more if needed) will be released before the final version to have some feedback.

Matrex 1,0 final released!

Finally, after weeks of test, Matrex 1.0 final has been released!
The most important changes are the following:

• Expression parser: the previous version was complex to use, expecially in the last step, where you had to choose the names of all the matrices and functions generated.
  With this version the last step requires only the names of the primary (root) matrix and function. The intermediate matrices and functions get a default name and are stored in a special package.
  
  Also, in the first step, two buttons have been added to select templates and matrices and add them to the input expression; so if you don't remember a template name or its parameters or a matrix name you get help.
  
  
  
  
• Matrix editor: removed side effects when editing cells using the mouse; the editor remembers saves also the last edited cell content; better number and date parsing.
• Added several function templates: among them:
  • sort, to sort several matrices using as index the first matrix, called key (like the pair of a map).
  • sumby and similar, similar to the select sum/group by functionality in SQL.
  • tail to tail a matrix with the content of another one.
  • queue to use a matrix as a queue, to which you queue values of other matrices
    
    
• Removed any dependency of the Matrex API by the Matrex GUI, finally allowing to use Matrex as a library. In this way a Matrex project can be opened and used by any (java) application.
• Presentation editor: better and easier interface. The format/position editor has been put in a separate dialog and substituted by a format/position viewer:
  
  

• Presentation viewer: added vertical header:
  
  
  
  
  Also, it uses virtual tables. It is possible to view/edit large presentations without performance losses.
  
  
• Matrix viewer: added vertical header:
  
  
  
  
• Added a template full-text search dialog to find templates by their description:
  
  
  
  

Matrex 1.0 final

I will probably release Matrex 1.0 final this night.
After a very hard test (GUI, events, templates) I think Matrex is ready for production.
This version has very few but important changes from RC1: easier to use and more professional GUI, more function templates and the Matrex API.

RMI for Client/Server?

Matrex 1.0 final is almost ready, so I'm thinking about the future of the system.
As you know, there will be at least two main versions after 1.0:

• 1.1, with improvements in various areas of the desktop application.
• 2.0, which will introduce the client/server architecture

With version 2.0, it will be possible for a Matrex desktop (or client) to connect to a Matrex server.
The server will host Matrex projects shared among serveral clients.

Matrex has been tought from the start as configurable as a client/server architecture, so it should be not very difficult to introduce the new architecture.

Anyway, some problems remained unresolved for a while, for example which protocol to use to connect client and server.
To choose it, I considered these facts:

• The clients and servers can reside on a LAN, it does not make sense to have internet connections.
• At least in a first stage only Matrex java clients will connect to the servers. If other programs need to use Matrex to calculate they can directly use the matrex API.
• Performance is very important. The users expect the system to calculate fast even if it works in a network.
  

Here are the candidates possibilities:

• SOAP
• CORBA
• Jini
• RMI
  

The advantage of CORBA and SOAP is the compatibility with different platforms and languages; but this is not needed.
SOAP messages can pass through a firewall, but since Matrex just needs to work in a LAN, this is not needed.
On the other side, SOAP is the slowest protocol and CORBA is loosing popularity.

Jini gives high realiability and performance, but it is very complex. Using Jini for Matrex is like to hunt birds with tanks.

RMI is the right choice for Matrex. It is sure not the most recent (it was available already in Java 1.1) but it works fine, is fast and requires less coding, because is made exclusively for java.

SWT Wizard Component

I extracted from the code of Matrex 1 class and 2 interfaces to implement a generic wizard with the SWT GUI library.
It is used to implement the expression parser in Matrex:

You can find the code under the wizard directory in the Matrex CVS repository.
The license of Matrex is GPL, but may be for the wizard code it can be changed to LGPL.

Function Templates list

I added in the Matrex site a list of all function templates with:

• name
• description
• input matrices
• output matrices
• parameters

I hope this will help to:

• find fast which template you have to use to write your function
• find out templates that are needed but missing
  
• find out fails or imprecisions in the templates descriptions
  

SWT: table vertical header

Matrex RC1 introduced a vertical header column in the matrix editor table, to show the row index or, if needed, to show a vector as a vertical ruler.

The SWT table does not allow to have a vertical header column as needed for Matrex, i.e. such that:

• is a fixed column in the table (it remains always visible scrolling horizontally)
  
• has cells with raised border to distinguish it from the other columns.
  

In version RC1 I decided to not loose so much time verifying if there was a work around, so I simply used as vertical header the first column of the table and giving it a gray color to distinguish it from the others.

In version 1.0 final I want to have a vertical header in other two tables, the matrix viewer and the presentation viewer, just to show the row index.
Therefore I started to see if there is a better solution than the one implemented in version RC1.

So I tested the SWT snippet 2 which should solve the problem using as vertical header a different table with only one column, synchronized with the original one.
This is a good idea, but has these problems:

• the vertical header table appears with a vertical scrollbar, which lets it appear disconnected from the main table. The scrollbar can be hidden, but I'm not sure it can be done on any platform.
  
• the cells have no raised border.
• sometimes the rows in the two tables appear slightly unsynchronized. This happens because the main table shows its first row only partially.
  

These problems (expecially the last one) make this solution not very usable and definitely do not give a professional look to the application.
Therefore by now I keep the solution used in version RC1. I'll see in the future if there will be other possibilities.

Again the Matrex database project

It looks like in the currently published version of matrexdb (in SourceForge) there is a logical error: the result column of the position presentation contains wrong values.
I will publish a new version of matrexdb this week with (hopefully) correct values.
This version will also contain a readme file with more details about the structure of the testdb project.

Matrex 1.0

I'm working on the first production version of Matrex 1.0.
The changes from version 1.0 RC1 will probably be:

• Virtual tables on presentations, to be able to show presentations based on very big matrices/vectors.
• Row header for matrix viewer and presentation viewer.
• Simplification of the Expression Parser when saving the produced matrices and functions. It will ask for the names of the final matrix but not of the intermediate ones.
  
• New function templates (e.g. sumby, sort, asinh...). I'm trying to cover the most common needs.
• Possibility to create matrices from the clipboard content, so you build a matrix directly from the copied block of a spreadsheet.
  
• Better function debugging.
• Better terminology (less X,Y more columns,rows).

If you have other ideas or needs please let me know!