Matrex Blog

Matrex 2.2 out

2012-01-14T03:59:00.001-08:00

Finally relaeased Matrex 2.2.

Matrex 2.2 adds 42 new functions in the standard package:

sys.date.removetime

sys.date.subtract

sys.db.update

sys.standard.minmax.max

sys.standard.minmax.min

sys.standard.minmax.nthmax

sys.standard.minmax.nthmin

sys.standard.test.iseven

sys.standard.test.isodd

sys.standard.trigonometric.acosh

sys.standard.trigonometric.asinh

sys.standard.trigonometric.atanh

sys.standard.changebase

sys.standard.closesteven

sys.standard.closestodd

sys.standard.constant

sys.standard.constantas

sys.standard.factorial2

sys.standard.gcd

sys.standard.lcm

sys.standard.logbase

sys.string.chars.charstocodes

sys.string.chars.charstostring

sys.string.chars.codestochars

sys.string.chars.codestostring

sys.string.chars.stringtochars

sys.string.chars.stringtocodes

sys.string.regex.find

sys.string.regex.replace

sys.string.test.isboolean

sys.string.test.isdate

sys.string.test.isinteger

sys.string.test.isnumeric

sys.string.equalsignorecase

sys.string.find

sys.string.len

sys.string.repeat

sys.string.replace

sys.string.trimleft

sys.string.trimright

sys.string.uppercasefirst

sys.util.convertunit

of them, very important are:

sys.db.update

This is used to insert, update or delete data in a database.

It allows the classical database operation update row if it exists, otherwise insert a new one , very useful when you want to store data from a spreadsheet.

sys.util.convertunit

This used to convert data from a measurement unit to another.

For example convert from watt to horsepower, liter to pint...

You can find more information in the function templates list.

Internally, Matrex uses now JodaTime to handle dates. This allowed to write the sys.date.subtract function, which is used to do dates subtraction.

Matrex 2.2 needs Java 6 (jre 1.6) to run.

Here is the download link.

Thanks to Braxton for the MacOSX version.

New version of Matrex

2011-08-22T13:09:00.000-07:00

I'm working on a new version of Matrex (2.2).

The purpose of this version is to add functions (formulas).

Matrex has already all functions needed for most users, but with this version I will add functions for specific niches (e.g. financial), functions that are secondary but needed (e.g. fact, log10) and more functions to access the databases.

Now I am preparing a list of all the new functions, as soon as I'm finished I will start the coding. As usual, help is welcome.

Rowscope 1.1

2011-06-08T13:58:00.000-07:00

I updated Rowscope, that reaches version 1.1.

Rowscope fixes some problems of the previous version:

It adds the line numbers column, which shows the line numbers of the displayed rows in the file.
It has a more compact GUI (the top menu has been moved on the right, the log is now displayed in a combo box). In this way it looks much better in Windows.
The buttons that are related to a single row (expand up/down, open a new tab from that row) are displayed on the selected row; no more need to move the mouse to expand a row.

It also can use the fork-join framework (Scala 2.9) to search text in the rows, making use of multiple cores. This can improve the performances, depending by the number of cores and their use.

Rowscope got also a better icon (the previous one was too simple and flat), code has been simplified and, when possible, tested.

Linux Magazine speaks about Matrex

2011-04-09T02:18:00.000-07:00

The link

Matrex 2.1 released

2010-12-22T01:21:00.000-08:00

Matrex 2.1 has been released yesterday.

Version 2.0 was revolutionary because it introduced the concept of client/server; version 2.1 is more an evolution because its purpose is to help the user having full control on projects and on the system.

In detail, these are the big changes:

The grid/tables have now a real row header, making easier to work with them and giving them a more professional look.
The script editor, which was very simple, has become a small but complete IDE to write and test scripts. With it the user can write its own functions and can test them before he uses them in his projects.
For someone that uses Matrex for the first time it is very important to find out immediately how it works. To make it possible, Matrex has now a help button in every window. Here is an example:

The other changes are less visible, but still important:

Before 2.1 the user could change all the function templates, included the ones that come with the system. Now he can only change templates written by users.
Fixed permission problems in the installation under Windows, allowing to install under the Program Files directory.

Thanks to Braxton for building the Matrex setup

file for MacOSX. Here is a screenshot of Matrex under MacOSX:

In the next version I will try to change the grid row header buttons under MacOSX to make them rectangular.

Matrex 2.1 is downloadable from here.

The new script editor

2010-10-24T08:18:00.000-07:00

Matrex has always allowed to use a script language (Jython, Groovy, JRuby...) to write the code of the functions.
But until now the support was limited to the following functionalities:

A button in the function template editor to associate a script to a function template.
A simple editor to write the script.

From version 2.1, Matrex will have a small IDE to write scripts:

The IDE has:

An editor with a simple but effective syntax highlighting. For each script language defines its specific highlighting rules.
A graphical test environment that allows to set up input matrices and parameters, with their content, and output matrices of the script. Once the matrices are set, the script can be executed and the result of the script can be checked in the output matrices.

In this way it is possible to edit the script and test it before it gets use as a Matrex function.

Matrex 2.1: status

2010-09-04T09:13:00.000-07:00

I am working on version 2.1 of Matrex .

The work on version 2.0 has been completely dedicated to the conversion of Matrex from a pure standalone to a standalone + client/server architecture, so I had to postpone some improvements to the GUI and the implementation of new functions.

Now I can finally work to implement these changes.

The planned changes for version 2.1 are the following:

Conversion of all the grid/tables (editors/viewers) so that they use the new table row header, which has shown to work without glitches and to be extremely efficient, even with large matrices and virtual tables. Already implemented.
New Help button in every window/dialog. Already implemented.
Conversion of the script editor to a simple script IDE, which allows to test the function scripts before they are added to the system, and has simple syntax coloring. Working on it.
Add several new functions. I think Matrex has already all the functions needed by most of the people, but I think more specialized functions are need, so that every kind of user has what he needs. Will start soon.

This is the plan, but probably some more feature will be added to version 2.1. See the active tickets.

You can have a snapshot of version 2.1 from the subversion repository, which should always contain a working version.

If you want to help, you are welcome to join the project.

SWT Table: row header

2010-08-03T02:57:00.000-07:00

I always used the SWT Table as grid component for my project Matrex.

The problem of this grid component is that it does not have a row header; and this is a real problem for an application like Matrex, that is an alternative to spreadsheet applications.

There are alternatives to the SWT Table that have a row header, but even if some of them are very good I personally prefer to use the original SWT Table.

So I tried the fixed first column horizontal scroll remaining columns SWT snippet, where the row header consists in a second SWT Table with only one column, syncronized with the main table.

This solution works fine in Windows, but has the following disadvantages:

In Gnome/Linux scrolling the table up and down the first row can be partially hidden by the column header. This is not handled by the snippet, therefore from time to time the row header appears not synchronized with the table.
The cells of the row header table are normal cells, not header cells (3D gray-button look). This can be fixed, but it looks like for a table of 10000 rows you need to create 10000 buttons!

So I have found a different solution, which uses a separate control for the row header, like in the snippet.

The row header control is a panel containing a vertical set of buttons, one for each visible row of the table.

You can see the panel on the left in the following picture (which is based on the SWT snippet create a table (columns, headers, lines)) :

In the picture, each button shows the index of the related row.

When the user scrolls the table vertically, the buttons texts change accordingly.

If part of the row is partially hidden by the column header, like in the picture, the related button is also partially hidden.

If the horizontal scrollbar appears, the row header changes its size accordingly.

In synthesis, the row header is always synchronized with the table.

You can see a demo of the row header in this video.

The table row header is just a single class, that can be tested using this test class.

The only complicated part of the class is the function getOffset, which solves the problem of the "partially hidden first row", returning the part of the first row that is hidden.

It uses the SWT Table function table.getItem([point]), which returns the row at the given point.

With this function it can know where is the limit between the first and second row, and therefore when the first row finishes. Knowing the height of each row (table.getItemHeigh), it calculates which part of the row is hidden.

I tested it with Windows and Linux (Ubuntu) and it works fine. It could be nice if someone can test it on MacOSX. There should be no problem, since it uses only documented functions in a standard way.

I will use it on all tables of the Matrex project.

Rowscope: view large files

2010-07-30T07:47:00.000-07:00

I had a problem in the office: I had a server application that was generating a large log file every day (between 100 Megabytes and 1 Gigabytes) and, if something went wrong with the application, viewing this log file was very difficult.

I tried several file viewers, but none of them worked well for me. Some of them did not show the entire row, some of them did not show all rows, some of them was slow when searching.

So, I went back to use notepad, which worked generally very well, but used too many resources and when the file was over the 300/400 Megabytes it was using the whole CPU and memory of the server.

So I have decided to write a file viewer and to write it in Scala, also because I wanted to try this language. The new file viewer, called Rowscope, had to be lightweight and fast.

For this purpose, I used a trick. Instead of viewing the file and then search strings in it, the user must first enter the search string, then when he opens the file what he sees are the rows resulting from the search.

Then he can look what's around one or more of the found rows.

In this way, if the search string is good enough, the viewer shows only a few rows and therefore uses a limited amount of memory. Clearly it reads all rows, but if they don't match the search string they are discarded immediately and then garbage collected.

To avoid that the GUI blocks Rowscope uses multiple threads in a pipe (Scala actors) to do the real work, and the rows are passed to the GUI thread only when they need to be displayed.

With the correct settings for each actor (how many rows to read from the file before they get sent to the other actors, how many rows to send from one actor to the other in one shot), the application uses the correct amount of CPU and memory and remains reactive to the user input.

I was a little scared when I started to use it at work. Yes, theoretically it should work, but will I use it just because I wrote it or because it really works?

I can say that now I use it without even thinking I am doing it, like all the other tools I use in my daily job. The first search then view approach works.

If I don't know what to search I start it with an empty search string, just to have an idea of what the file contains. Rowscope shows only the first 1000 lines of the file. From this I find a good search string and I start to work.

I'm preparing a new version of Rowscope. If you miss some feature or have some good idea, please let me know.

Matrex 2.0 released

2010-04-20T11:52:00.000-07:00

Finally the version 2.0 of Matrex has been released.

The big change is the new optional client/server architecture that allows several users to work together on a project or for a user to delegate the calculations to a server (in the picture we see 3 Matrex Desktops sharing a project on a server run in the same PC).

We already told everything about this feature in the previous articles in this blog (read for example the article showing the steps to follow to connect to a server).

But we have other changes that are really useful.

First of all, Matrex does not need a batch file to start anymore. Just double-click on the matrex_start.jar file and Matrex starts (if java is configured correctly).

This also removes the shell/command line that was always starting together with the GUI, making the application look unprofessional.

Second, Matrex gets installed on MacOSX as an application bundle (packaged by Braxton), that's why its name is terminating with .app.dmg.

Unfortunately the application bundle can only be installed on MacOSX 64 bits.

For the 32 bits version of MacOSX (Java 5) you will need to install the generic version, then, using a shell, go to the top Matrex directory, run:

java -XstartOnFirstThread -jar Java/matrex_start.jar

and follow the instructions to install the GUI library (check also the F.A.Q. about this).

Screencast: connect to server

2010-04-14T10:32:00.000-07:00

I was not able to publish Matrex 2.0 until now, because the test took more time than I thought initially and because Braxton and me are trying to make it installable on MacOSX using an application bundle.

Hopefully in a few day the new version will be published.

Anyway, I made a screencast, published on ScreenToaster, on how to connect Matrex to a server on the same PC:

Screencasts and videos online

Matrex 2.0 Beta released

2010-03-03T12:40:00.000-08:00

A beta of the client/server version of Matrex, 2.0, has been released.

The plan is to release the final version in a pair of weeks, so please test it!

To test the client/server connection:

Install Matrex 2.0 beta.
Follow the instructions contained in the server document (which is included in the installed documents) to install the Matrex Server and start it.
Start Matrex Desktop and connect to the server following the Remote Projects section of the help content (press Ctrl-F1 in the Matrex Desktop).
Create a project on the server or copy a local project to the server, as described in the help content, and play with it.

If something is not clear, please contact me.

If you find bugs please report them in the trackers.

Matrex 2.0: project integrity in a multi-user system

2010-01-03T13:17:00.000-08:00

Now it is a long time the alpha version of Matrex 2.0 was published, but a beta is still not available. The reason is that the task to make the core objects (project, matrix, function...) thread safe is complex.

Until now Matrex was a single-user system. That does not guarantee against concurrency problems, but makes certainly things easier than on a multi-user system, like Matrex 2.0. In Matrex 2.0 a function or chart or presentation can potentially be changed by multiple users in the same time, which can lead to unpredictable results.

To avoid this, the work I'm doing now is to make the project object not only a repository of its items (matrices, functions...) but to give it the responsibility of its own integrity, and to this, the complete responsibility of adding, updating and deleting items.

In Matrex 2.0 the chart editor, for example, is not able to update a chart object directly. It tells the project how to change the chart object, and the project changes it, after it checked that the change is possible and does not break its own integrity: charts are dependent by matrices and it could be possible that the change involves adding a matrix to it that during the chart editing gets deleted by a different user; in this case the project rejects the update, the chart does not get change and the editor gets an error.

The project's methods are guarded, so that it is impossible to do incompatible changes in the same time. For example it is not possible to delete a matrix in the same time a chart is updated.

In other words the project in Matrex 2.0 looks a lot like an SQL server with transactions. The SQL server makes sure two SQL updates cannot take place on the same rows of a table in the same time and an SQL update (or a group of them) can be integrally rejected (rollback) if something goes wrong, leaving the database as it was.

A side effect of these changes is that, since everything is done through the project, the items become more or less read-only for the editors and the rest of the GUI, simplifying the system.

In terms of performance these changes should be in-influent. What is important for the system performance are not the items editing, but the matrices calculations that derive from it, which are much more complex and CPU consuming.

I'm now in the middle of the work of changing project, items, editors to make them work in this way. As soon as I am finished I will publish a beta.

CSV files again

2009-10-27T06:37:00.000-07:00

The alpha release of Matrex 2.0 solves the problem I reported about importing large CSV files.

Already in the previous versions Matrex used a virtual table to show the imported CSV files; this means the table (grid) loads from the memory only the rows that it needs to display.

The next step was simple: in version 2.0 the file is not even loaded in memory; only the rows that are displayed are actually loaded from the file.

In this way the memory used to import the CSV file decreased dramatically compared to the previous versions of Matrex.

To avoid performance losses with this new version, Matrex keeps a cache of 2000 rows from the file (the 2000 rows around the last row loaded from the file); in this way scrolling the table up and down is still fluid.

The following picture shows the 3 levels of the CSV file import: file, memory cache, table.

If the table is scrolled a lot up and down it can still be that many rows are loaded in memory, and released immediately after; to avoid memory losses because of this, Matrex calls directly the garbage collector every 50000 rows loaded.

In this way it was possible to import data from the 22 MBytes CSV file for which the memory problem was reported, running Matrex without any special memory option.

This fix will be part of version 2.0, but I also back-ported it to version 1.3.8, which will be published in a few days.

Matrex 2.0 alpha

2009-10-14T21:33:00.000-07:00

I published an alpha (unstable) version of the new Matrex 2.0, which adds the possibility to use Matrex as a client/server system.

You can download it from here.

To test it as a client/server system:

Install this version of Matrex. It is a generic version, so remember that when you start it the first time it will download the graphical library SWT. Only the second time you start it it will really start.
Install the Matrex Server. The setup file is in the Matrex directory, called matrex_server_2_0.jar.
Execute rmiregistry. It is the RMI registry server (Matrex Desktop and Server use RMI to communicate). It is part of the Java Runtime Environment (JRE).
In the Matrex Server directory, execute matrex_server.bat (Windows) or matrex_server.sh (Linux, MacOSX...) to start the server. Check that there are no errors.
In the Matrex directory start Matrex.
Follow this to let Matrex open a server project.

You can login as guest (password guest).

If you want to login as a different user, you need to change the config/accounts.xml file in the Matrex Server directory, adding an account element with the userid and password, always setting the encrypted attribute to false.

To become final release, Matrex 2.0 needs the following changes:

Fix potential issues when a single project in one server is opened concurrently by several users.
Some operations, like adding functions or functions expression (expression parser) cause the addition or update of several items in the project. Therefore these operations must be done atomically, possibly using some kind of transaction.
Check that the resources allocated to a client in the server are cleaned up correctly when the client disconnects.

If you find problems with this alpha version, please add a comment to this article.

Large CSV files

2009-10-05T12:05:00.000-07:00

A bug submitted the last month was showing that Matrex needs a lot of memory when importing large CSV files.

The CSV file mentioned in the bug has a size of 22 MBytes.

The file contains around 200000 lines. Each line has 14 fields and is around 100 characters long.

To import this file the memory used by the Matrex process increases of 300 MBytes.

The CSV file is large but it does not justify so much RAM to handle it.

I checked the code that imports CSV files in Matrex; nothing is wrong.

Matrex uses the Java CSV library to read CSV files, which works fine.

The file is loaeded in memory row after row.

I checked how much memory is used for each loaded row. This is not easy in Java, since nothing similar to a sizeof function exists in the standard libraries. But I have found the Javabi library, which is able to measure the amount of memory used in total by a java object.

Each row with its fields is handled as an array of strings, which uses around 800 bytes, 8 times the original row's size.

This is because:

Java strings use Unicode, which means that they use 2 bytes for each character
Strings use additional memory for their fields and their fields alignments

800 bytes * 200000 rows = ~160 MBytes.

As far as I understood the rest of the memory used to import the file is allocated to the intermediate strings that the CSV reader uses to read the file, and that remain allocated until the garbage collector frees them.

There are some solutions that could be applied to reduce the memory use:

Avoid loading all rows of the CSV file in memory: in other words, make the import editor extract the displayed lines directly from the file, and let it extract the lines only when they are actually displayed.
I'm not sure about the effectiveness of this solution, because in general the user wants to import to a matrix an entire column of the file. Therefore the file, sooner or later, has to be read entirely.
Another problem with this solution is that the Java CSV library, as far as I understood, does not allow to count the number of rows without reading them and does not allow to jump between rows without reading all the intermediate rows.
Read less fields: immediately at the start of the import process give the possibility to the user to discard some fields, so that they are not loaded in the import dialog.
This can work, but I am not sure that it can dramatically reduce the amount of memory used to load the file.
Optimize the reading process so that it uses less memory: this means to look for an alternative to the Java CSV library that uses less memory to read the files (for example using CharSequence objects that use less memory). There are alternative libraries, for example opencsv and the Ostermiller utilities. They need to be tested and see if they are better than the Java CSV library to reduce the used memory.

I will try to apply these solutions and explain, in one of the next articles, what has been done.

Matrex 2.0 is not just a specification anymore

2009-09-17T12:25:00.000-07:00

After more than two months of work I was able in these days to let Matrex open a project in a Matrex server.
This is how it works:

In Matrex click on the menu File->Connect. The following dialog appears:

In this case the server is on my PC, the same one in which I run Matrex, so I write localhost as server address.
Clicking on Names, the Matrex Server combo box is populated with the list of available servers in the PC with the given address.
In my case there is only one server, default.
I press OK. The login dialog appears:

Guest is the default user, with password guest. It is the user that is available by default in the server, if it has not been configured.
I write user id and password and press OK.
The remote machine tab for the localhost server appears, beside the local machine:

In the machine menu I click on Open Project. The Open Remote Project dialog appears:

Differently from the local projects there is only the possibility to choose in a list of projects. In facts on the server side the projects are all under the same directory, projects.
I select the example project popcolorado and the project opens:

By now I checked that it is possible to open matrix and functions viewers and editors.
In the next days I will check all the project's functionalities.
As usual, the sources for the last version of Matrex are under the Matrex subversion repository.

As soon as I have a version that is tested enough, I'll publish it as a pre-alpha.

Matrex 1.3.7

2009-08-24T21:30:00.000-07:00

I released version 1.3.7, which fixes the bug reported as a comment to the Matrex 1.3.6 blog entry, and that I entered in the Matrex bug tracker:

Matrex seems to forget the Project settings (threading, etc.) after a restart;

This was caused by Matrex not being able to overwrite the project file.
The files are the following (in alphabetic order):

matrex_1_3_7_generic.jar for any platform
matrex_1_3_7_linux_gtk.jar for Linux
matrex_1_3_7_macosx_32.jar for MacOSX with 32 bits Java (Java 5)
matrex_1_3_7_macosx_64.jar for MacOSX with 64 bits Java (Java 6)
matrex_1_3_7_win32.jar for Windows

The bug has also been fixed in the code of the next release, 2.0.

Client/Server: technical view

2009-08-08T04:40:00.001-07:00

As told in the previous article, I'm changing Matrex from a pure standalone desktop application to an application that allows to work standalone or in a client/server architecture.
To support the client/server architecture I use the RMI protocol.

This means that the calculation engine, the one that calculates the functions and therefore generates the content of matrices, presentations, charts, will be both in the desktop application and in the server.
For this reason, the GUI has to use in the same way the objects involved in the calculation (projects, matrices, functions...) , whether they are on the client side or on the server side.
To do this, the original calculation objects (projects, matrices functions,...) are wrapped in two different new categories of objects: Local (client) and Server:

Both the wrappers, share the same remote interface (which extends the RMI's Remote interface).

The reasons I use wrappers instead of the original objectts is because all the methods of a RMI business object must throw the RemoteException exception.
RemoteException is needed to understand when the server is down or there are problems of connection, so I would never do without it.
On the other side, it becomes annoying to catch it every time some code calls a method of a business object, so I want to do it only when it is strictly needed.
So I use the wrappers only in the GUI, where it is needed. Instead the calculation engine uses the original objects.

Now, why Local and Server wrappers? Why not use only Server wrappers, both on the server and on the client side?
There are several reasons:

Server machines and projects have slight different interfaces when they are on the client and on the server side, mainly because projects on the server side can only be saved in a specific directory, projects on the client side can be saved in any directory of the disk.
The server wrappers extends the UnicastRemoteObject, local wrappers don't. I don't really understand completely how the Java compiler and RMI compiler handle these objects, so I cannot be sure that they don't have some effects on the application's performance. If these performance effects are needed with the server business objects, I don't want them on the local objects.

And why I did not use the original classes instead of the Local wrappers? Because I needed a special wrapper for the Matrix class when it is used in the GUI, and only when used in the GUI, called SafeMatrix, which makes the Matrix methods thread safe.
But this means that all the other calculation classes need to have parameters of type SafeMatrix when called by the GUI, and instead use parameters of type Matrix when called by the calculation engine. And this means that I need special wrappers that use SafeMatrex parameters, the Local wrappers.

So, now I'm working on it. I will take some time, because in the GUI all the references to the original objects must be changed to the new remote interfaces.
Which means:

remote exceptions to handle.
utility functions to convert the original classes to the wrappers.
some code duplication.
many wrappers to write, expecially for the charts, for which there is one class for each chart type.

Also, I expect to reduce the number of methods in the calculation classes to reduce the number of remote calls.

When I have something that more or less works I'll publish it as an alpha version.

Working on client/server

2009-07-22T10:38:00.000-07:00

I started to work on the client/server version of Matrex, the 2.0.

This version, as explained in the specification, will give the possibility to use Matrex in two ways:

standalone, as today
connected to one or more Matrex servers.

When Matrex opens a project in a server, all calculations for that project are done in the server: Matrex acts only as a graphical interface.
One would open a project in a server:

to use the CPU of the PC running the server instead of the one of his own PC.
to share the project with other people. In fact two or more Matrex clients can work on the same project in the same server in the same time, without problems.

Matrex has been written from the start to become one day a client/server system, so the GUI will not change so much: not much more than a new menu to connect to a server (I will publish some pictures as soon as I have a stable version).

The protocol used is RMI, but I will keep the possibility to use different protocols in the future. It could be nice to have a version (based on REST?) that can work on the internet through the firewalls.

Matrex 1.3.6

2009-06-30T00:06:00.000-07:00

A few days ago the Eclipse Foundation published version 3.5 of SWT.
I was waiting for this because it means that Mac users can finally use Matrex with bot Java 5 and Java 6, and with better graphics, because the new version is based on Cocoa.
So, as soon as this new version has become available, I have published the new version of Matrex, 1.3.6.
Together with the new SWT, 1.3.6 comes the following changes:

There was still some incompatibilities with Java 5, so it was not possible to run Matrex 1.3.5 with Java 5. This problem was hitting mainly the Mac users, that could not use Java 6 because SWT was not working with it. Now I made sure this problem will not show up again.
From Matrex 1.3.5 maximizing the main window means maximizing only vertically, leaving the horizontal size of the window the same. In Windows it could happen that maximizing the main window it disappeared from the screen. In Linux from time to time the main window was becoming insensible to the mouse after the maximization. These problems got fixed.
Matrex is able now to read script templates from the script classpath defined in the Files Locations dialog. In this way also the script languages plugins (groovy, jruby...) can be installed outside the Matrex directory.
Minor bug fixing.

Now the versions are the follwing:

matrex_1_3_6_win32.jar for Windows 32 bits.
matrex_1_3_6_linux_gtk.jar for Linux 32 bits.
matrex_1_3_6_macosx_32.jar for MacOSX 32 bits (Java 5).
matrex_1_3_6_macosx_64.jar for MacOSX 64 bits (Java 6).
matrex_1_3_6_generic.jar for all the other architectures, included Windows 64 and Linux 64 (SWT is installed the first time Matrex is started).

The choice between 32 and 64 bits depends by the Java runtime interpreter used.

About MacOSX

2009-05-25T14:24:00.001-07:00

If you are not able to install Matrex 1.3.5 on MacOSX, please check the article about this topic.
You can download a more recent beta version of the SWT library for Cocoa here.
As soon as Eclipse 3.5 Galileo will be released, I will update the setup files to use the new SWT library, which will probably solve the problem.

Matrex 1.3.5 has been released

2009-05-23T10:33:00.000-07:00

Matrex 1.3.5 has finally been released.

It should have been called 1.4 for the new features it contains, but when the coding started it was called 1.3.5 and so it remained.
As the notes that come in the setup file report, these are the new features:

GUI:

Project diagram.
Plugins dialog has been evolved to a more general locations dialog:

In this way it is possible to add plugins to Matrex without installing anything in its directory.

If a toolbar becomes too small to contain all buttons, it displays a menu containing the missing buttons:
Progress bar that shows loading of the single items of a project:
The matrix editor has a toolbar, to make the GUI uniform in the whole application:

Internal:

Configuration files containing the location of the other Matrex files (see locations dialogs). With this Matrex becomes easy to package for specific operating systems.
Able to read/write tab separated files (together with the CSV files).

Test:

More unit testing (> 280 tests). The idea is to have a set of tests to fire to check the project before each release.
GUI unit testing of an editor (Function Editor) with SWTBot.
Used Findbugs annotations.

Several bugs have been fixed.

You can download it here.

Matrex 1.3.5 almost ready.

2009-05-07T11:27:00.000-07:00

Last tests for the version 1.3.5 of Matrex, which will be probably released in the last half of May.
The changes are many, but surely the most important are:

Flexible configuration files, which allow Matrex to be installed in packages or setup files.
The project diagram

together with many bug fixes.

Project diagram

2009-03-30T12:39:00.000-07:00

Matrex version 1.3.5 will have the possibility to show the project in form of a diagram.
Here is the diagram for the example project projection, that is included in the Matrex setup:

It shows all items of the projects (but timers) and their connections.
Each item is labeled with its type (matrix, function...), each connection is displayed as an arrow.

Compared to the item trees and to the information views, the project diagram has the advantage that It gives a global overview of the project. In one single window it is possible to see all items of the project.

The meaning of the arrows is the following:

An arrow from a matrix to a function means that the matrix is input of the function
An arrow from a function to a matrix means that the matrix is output (result) of the function
An arrow from a matrix to a presentation or chart means that the matrix is used in the presentation or chart.

Since a project can contain many items and therefore many connections among them items, the project diagram can become complex.
To make it easier to understand it, it is possible to:

Click on an item: the item becomes green and all the connected items and their arrows become blue (in) and red (out).
Click on one of the items type on the tool bar: all the items of that type become green.

In the tool bar there is also a button to print the diagram.

Matrex easier to package

2009-03-09T13:08:00.000-07:00

Until now once you installed Matrex in a directory, Matrex used the same directory to:

write its log file
write the changes to its configuration files
write additional templates
write scripts used as base of templates

This is natural for a java application that is installed with a java installer and does not pretend to integrate with the platform in which it is installed.

But what if we want to create installers for specific platforms (Windows setup, MacOSX application bundles, Linux .deb and .rpm files)?
In this case:

it is the operating system that defines in which directory or directories Matrex is installed.
Matrex cannot write the files it produces (templates, configurations, log...) in the directories in which it is installed, but must use a writable directory, which can be under Documents and Settings in Windows or under the home directory in Linux.

Version 1.3.5 will make this possible with two changes:

the new configuration file main.properties, which contains the paths of all other configuration files (which can contain environment variables).
the new concept of configuration files that contain the locations of other files.

The configuration files that contain the locations of other files are:

templates.cld, which contains the paths of the top directories of directory trees containing functions templates.
plugins.cld, which contains the directories and jar files containing additional java classes used as plugins and their dependencies.
scrips.cld, which contains the top directories of directory trees containing scripts used by templates.

With the addition of main.properties and of the .cld files it is possible to have the configuration files that change after the installation and additional templates, java classes and scripts in writable directories.
In this way anyone can change these files, add them or delete them without touching the original installation.

It will be then easy to install Matrex:

In Windows in a directory under the Program Files directory, with the additional files and the writable configuration files in a directory under Documents and Settings.
In Linux in the standard directories (/usr/bin, usr/share...), with the additional files and the writable configuration files in a directory (.matrex ?) under the home directory.
In MacOSX in a directory under the Applications directory, with the additional files and the writable configuration files in a directory under Users

I tested this new feature installing version 1.3.5 under a read-only directory and configuring it so that it keeps all the files that need to be changed under a writable directory.
It works fine.
As soon as possible I will release a beta of this version.

ToolBarWithMenu component

2009-02-26T12:18:00.000-08:00

The SWT ToolBar component has a minor problem: when there is no space enough to show all the buttons contained in the ToolBar, it just shows the first ones.
It is true that the SWT.WRAP property partially fixes the problem wrapping the buttons in two or more lines, but this does not work in some platforms (e.g. Linux) in which the property is ignored.
This can be a problem, because the user of the application can be completely unaware of the buttons that are not displayed.
I have seen other graphical libraries solve this problem adding an additional button at the end of the toolbar. This button when clicked shows a menu with the missing buttons.
So I adopted the same logic in SWT: I made a class called ToolBarWithMenu
, which adds a button at the start of the toolbar. The button has a menu showing the buttons that are not displayed in the toolbar because there is no space. Here is an example in Matrex:

Clicking on one of the menu items has the same effect as clicking on the related toolbar button.
It works as expected also for a button with an attached menu, if the button's selection listener that shows the menu implements the IHasMenu interface.

Matrex on MacOSX

2009-02-26T00:41:00.000-08:00

Matrex uses the SWT library for its GUI. SWT uses the native GUI of the platform. The current version of Matrex, 1.3, uses SWT 3.4.
On MacOSX SWT 3.4 is based on Carbon, which is only 32 bits.
So Matrex 1.3 on MacOSX does not work using Java 1.6 (Java 6), which is only 64 bits and therefore not compatible with Carbon.
If you want to use Matrex 1.3 in MacOSX please either:

run it with Java 1.5 (Java 5) instead of Java 1.6
try the beta version of SWT 3.5 for Cocoa 64 bits. Download the zip file and unzip it in the /swt directory, where is the directory where you installed Matrex.

The following is the description of the SWT 3.4 problem written by the SWT developers:

SWT cannot be used with OS X JRE version 1.6 (Mac OSX only)

OS X JRE version 1.6 assumes that pointers have a size of 64 bits, but SWT's Carbon port only uses 32-bit pointers, so SWT and Eclipse cannot be used with OS X JRE version 1.6. The workaround is to use an earlier supported version of the OS X JRE.

Updated Matrex adapter to Scilab

2009-02-22T12:55:00.000-08:00

The Matrex adapter to Scilab has been updated for Matrex 1.3.5.
The adapter has been tested with Scilab 5.1.

The adapter contains a special function template that calls Scilab functions to calculate Matrex functions.
Input matrices and parameters of the Matrex function are passed to the related Scilab functions, which is calculated. The result of the Scilab function is passed back to Matrex as output matrices of the Matrex function.

More information can be found in the documentation included in the Scilab adapter setup.

New Matrex adapter to Groovy

2009-02-22T12:18:00.000-08:00

I released in these days the new Matrex adapter to the Groovy language.
This follows the two adapters to Jython (integrated in Matrex) and to JRuby.
This adapter is good:

for fast prototyping and test of function templates
to write more compact code than in Java
to use features, for example closures, that are not available in Java

Like the other language adapters it used in this way:

create a Groovy script with the code for the function template
create the function template definition
use the function template to create functions in Matrex projects

More information can be found in the documentation included in the Groovy adapter setup.

Subversion

2009-01-25T23:23:00.000-08:00

For who is interested in the Matrex code, I'm moving Matrex and all its satellite projects to Subversion (SVN).
With CVS I was always adding to the repository a lot of files by mistake and was never able to really remove them.

For more information just go to the Matrex SVN page in Sourceforge.net.

As soon as possible I will move to Subversion also Internal SQL Library and SWT Web Installer.

Plans

2008-11-29T06:51:00.000-08:00

These are the plans for the future of Matrex, in order of time:

Upgrade all adapters to libraries, script languages and scientific applications to work with the version 1.3.x of Matrex, which uses a new plugins architecture. I already upgraded the Jama adapter and I'm working on the JRuby adapter. Clearly if there are new versions of the libraries, languages and applications the adapters will work with them.
New adapter to the Groovy interpreter.
New intermediate version of Matrex, probably called 1.3.5. This version will not contain major changes from version 1.3. The idea is to make possible to have configuration files and additional templates in different directories. In this way it is possible to build .deb or .rpm setup packages for Matrex and make it easier to use under Vista and MacOSX.
New version 2.0: It will be possible to use Matrex as a standalone application, like now, or as a client/server system.

The upgrade of the adapters and the Groovy adapter will be done in the next weeks.
Versions 1.3.5 will be probably finished in ~ 2/3 months.
2.0 will be finished in ~ 6 months.
Versions 1.3.5 and version 2.0 will be developed in parallel.

In the same time Braxton and me are trying to build a setup file to install Matrex on MacOSX as an application bundle.

Compatibility with Java5 : Matrex 1.3.01

2008-11-26T23:43:00.000-08:00

I published a new release of Matrex, 1.3.01, to cope with the fact that Matrex 1.3 was able to run with Java6, but not with Java5.
The problem was a call to the String.isEmpty() method, which is only available from Java6.
In 1.3.01 this call has been replaced and this version has been tested with Java5.

Thanks to Braxton who found the problem.

Matrex 1.3 relased

2008-11-12T11:13:00.000-08:00

So we finally made it. Version 1.3 has been released!

A summary of the changes from version 1.2 is available in the Sourceforge's release notes.
The big changes have been:

Main View usability improvement.
The main view is now easier to use thanks to the buttons that hide and show specific trees (for example you can hide the function tree and show the presentation tree just clicking two buttons).

It is also possible to search items or folders in a tree just typing a part of its name (in red).

In each tree is not possible to move a set of items to a package.
Just select some items of a tree (also in different packages), right-click with them mouse on the tree to display the popup menu and select Move... (for example Move Matrices). With a dialog box you choose the destination package and the items gets moved.
The system takes care of the dependencies.
Viewers usability improvement . In the presentation viewer and matrix viewer it is now possible to display the content of presentations and matrices in the best way clicking the fit columns button.
Generic setup. The Matrex generic setup allows to install Matrex on every platform. The idea is that, once installed, when executed the first time, Matrex downloads the correct version of the SWT library and installs it.
Executing it once again, the downloaded library is used and Matrex works regularly.
All thanks to the SWT Web Installer library.
New high low charts. Added candle and high low charts.
Plug-in architecture. Matrex now uses its own class loader, which allows to specify libraries (e.g. function template libraries) that need to be loaded using the Plugin Dialog:

No more class paths to change in the batch files!
Code quality improvement. The code is now well covered with JUnit tests and has been checked with Findbugs. The GUI has also been unit tested using SWTBot.

Also the experimental new feature of the callbacks has been added. It is used in the directsearch example project, which demonstrates the Nelder Mead optimization method.

As usual, feedback is precious. Please let me know if something is wrong!

SWT Web Installer and ISQL Library updated

2008-10-17T01:25:00.001-07:00

Since I published the pre-release of Matrex 1.3, I also updated the two libraries that Matrex uses, SWT Web Installer (now 1.0) and ISQL Library (now 1.01).

SWT Web Installer downloads now SWT 3.4 instead of 3.3.1 and has a small change to the WebInstaller class constructor (new parameter systemProperties).

I also added a test that verifies that all sites in the swt_list.xml configuration file exist and are accessible.

ISQL Library had a small bug related to the conditions of the query's where clause. That has been fixed in version 1.01.

I have checked both libraries with Findbugs before I published them.

Matrex 1.3 pre-release

2008-10-16T23:34:00.001-07:00

I published a pre-release version of Matrex 1.3.

You can find it under the Matrex Unstable download section in SourceForge.

The changes from version 1.2 are listed under the Matrex File Release Notes and Changelog page in SourceForge.

The idea is to make sure that if there are bugs in Matrex 1.3 they get fixed before the final version is released.

So please give a try to this version and, if you find bugs, use the tracker page in Sourceforge to report them.

The pre-release version includes only the setup for the generic architecture, which means that when you start Matrex the first time it is supposed to download the SWT library for your PC.

After that Matrex stops, so you need to start it once again to use it.

The download of SWT is a new feature, so if you have problems with it please use the tracker page to report them.

SWTBot for dialogs

2008-09-13T09:38:00.001-07:00

I started writing JUnit test cases to test the GUI of Matrex using SWTBot.

Since I did not have any experience with GUI testing, I wanted to start with something simple.
The simplest GUI items to test in Matrex are the dialogs, since they do something specific and limited.
So I started writing test cases to test dialogs.

It turned out that testing dialogs is not so easy as it seems (and this is a general problem, not limited to SWTBot).
With SWTBot you simulate the actions of your application's user: press buttons, write in text boxes, select from list and check boxes...
But you cannot simply open a dialog and simulate the user's actions: dialogs are modal, which means that the function that opens a dialog cannot continue until it is closed.
So, simulating the actions before the dialog is open does not work because there is no dialog; simulating the actions after the dialog is closed does not work because there is no dialog.

There are two practical solutions to this problem:

make the dialog not modal
open the dialog in the main thread and simulate the user's action in a secondary thread.

The first option is not possible in my case. The dialogs are opened by factory methods and I cannot change the code in these methods only to test them.
Moreover, if a second dialog gets opened by the first dialog we have the same problem again.

So I used threads.
I wrote a class for this purpose, SWTBotDialogTestThread. An example of use of this class is in
the test case InputDialogHandlerTest.

The right way to use a thread that acts on the SWT GUI is to call the function Display.asyncExec on the dialog's display, otherwise the GUI becomes unstable.
This is what I do in SWTBotDialogTestThread.workWithDialog.

Also if the actions on the dialog result in an exception, the exception must be caught by the test case in the main thread. I solved this problem running the actions on the dialog in a FutureTask instance (wrapped in the ITestAndWait interface). When the main thread calls the FutureTask.get method (ITestAndWait.waitForWorkDone) this throws the exception that was raised by the thread started by asyncexec, if something went wrong.

In synthesis, when I write a JUnit test case to test a dialog I:

create an instance of SWTBotDialogTestThread, which opens the shell used as starting point for the dialog.
call SWTBotDialogTestThread.workWithDialog, which starts the thread with the actions to do on the dialog. The thread waits until the dialog is available.
open the dialog. At this point the actions are done on the dialog. One of the actions (click on the Ok button) closes the dialog.
Call ITestAndWait.waitForWorkDone that re-throws the exception thrown by the other thread, if that thread thrown exceptions.
check the result of the dialog.
call SWTBotDialogTestThread.close to close the shell opened at the start.

I don't know if this is the best way to solve the problem, but it works, even if several test cases are run together.

Matrex 1.3. Development finished.

2008-08-31T02:30:00.000-07:00

Matrex 1.3 coding has been finished.
I'm now testing it and updating the documentation.
Hopefully I'll release it at the end of september.

Callbacks

2008-08-29T11:07:00.001-07:00

A Matrex project consists in a network of matrices and functions (let's forget about charts, presentations and timers).
They are connected together by the relations:

The matrix Mi is input parameter of the function Fj
The matrix Mi is output (result) of the function Fj

When a matrix changes its content, it fires the functions of which is input, which recalculate their output matrices, which in turn fires the functions of which are input and so on.

One peculiar property of this network is that it does not have loops. In mathematics it is representable as a simple graph.
This means that a matrix cannot be, directly or through other functions and matrices, input and output of the same function.

This is a good property, because it guarantees that, if the calculation of each single function terminates in a finite time, the calculation of the whole project terminates in a finite time.
In other words, the calculation of the project cannot contain an infinite loop.

In the upcoming version 1.3 I made an exception to this property, to solve the following problem:
I added a new solver function template to Matrex, that finds the minima of functions.
To use a solver, you need to pass to it the function you want to minimize.
In a programming language, you pass the function to mimimize to the solver function as a callback function (or a closure).

But how I did this in Matrex?
In Matrex, a programming language's function is equivalent to a network of functions and matrices, which, seen as a black box, has input and output matrices:

This block of functions and matrices can be equivalent to a callback function, with a trick:

the output of the callback block and the input of the caller must have one matrix in common
the output of the caller and the input of the callback block must have one matrix in common

The output matrix of the caller that is input matrix of the callback block is called callback matrix.

Here is a picture showing the concept:

In this way the caller, by changing the content of one of its input matrices recalculates the callback block.
When the callback block executes, it updates the content of its output matrices. But one of those is a input matrix of the caller, therefore the caller is recalculated.

Caller and callback block execute each other until the caller decides to stop and does not change the content of the output matrix that it has in common with the callback block.

But when does the caller decide to stop? This depends by the caller code.

The solver I added In Matrex is Nelder Mead (sys.mathstat.optimization.neldermead), an implementation of the direct search algorithm, a simplex optimization algorithm (it uses the Apache Commons Math library).
At each step of the solving process the solver needs to call a function to calculate the cost of the current solution. The cost calculation function is our callback function.
The following picture shows the solver, the cost calculation function and their input and output matrices:

x is the callback matrix of the solver. It is both output matrix of the solver and input matrix of the cost calculation function.
When the content of x is changed, the cost calculation function is fired and it calculates the cost matrix, which is input of the solver.
The solver is executed and produces a new content for the x matrix, therefore firing the cost calculation function, until the cost is good enough to stop. At this point it stops changing the content of x and writes the final solution in the result matrix.

What I learned writing the Nelder Mead solver template is that one has to write the caller code carefully making always sure that the mutual execution of caller and callback function stops at a certain point, in a way or the other. For example a solver must stop after a certain number of iterations, even if it does not converge.

Adding callbacks in Matrex required some changes in the code:

The callback matrix is special: adding a function that has a callback output matrix requires that this matrix is already in the project (it is the input matrix for the callback block).
Functions that have callback output matrices are not calculated when the project is loaded.
Calculating them only once would leave them incomplete (for a solver this means to calculate only one step).
Calculate them until they stop would be a risk. Better to leave to the user the responsability to calculate them.

Because of the complexity of this new feature, I introduce it in Matrex as experimental.

Test test test

2008-07-26T14:10:00.001-07:00

Matrex is a project that gets tested manually by me before it is released.
For this purpose I use a check list.
I run Matrex and I verify that the operations listed in the check list, like adding a matrix, adding a function... are executed correctly and don't produce side effects.

Matrex is also tested by some of its user, but generally the users test it after a new version has been released. This is not very satisfactory, even if it allows me to release corrective versions.

To reduce the possibility that a version is released with bugs, I started, from version 1.2, to add some unit testing.

In the incoming release 1.3 unit testing will be a main concept:

the tested classes are at this moment more than 160, covering almost all Matrex template functions and a good section of the application code.
I'm checking the possibility to have also some GUI classes tested, using the SWTBot GUI unit testing tool.

In this way I hope I will be able to:

Improve the quality of the code.
Help developers understand the code of Matrex.
Reduce the possibility of regression bugs.

If you have other ways to improve the reliability of Matrex please let me know.

Changes in the main window

2008-06-21T12:18:00.001-07:00

Version 1.3 changes something in the main window:

The buttons that are highlighted with a green rectangle hide or show trees. For example the first button hides the Matrix tree. Clicking it once again the Matrix tree is displayed again.
The second button toggles the Function tree visibility, the third the Presentation tree, the 4th the Chart tree and 5th the Timer tree.
On the top of each tree, together with the tree menu, there is a search bar, highlighted with a red rectangle.
Writing a part of the name/package of the item to search (matrix, function...) and clicking on the green arrow button, the next item with name-package containing that text is selected and showed in the tree.
For example the text ding.pri finds the item trading.price.

Exception stack trace for final users

2008-06-06T13:44:00.001-07:00

In Matrex, like in other programs, it is not always possible to show a simple message box when an error occurred.
Sometimes what I have is an exception occurred deep down in the code, may be in a library.
Nevertheless something has to be shown to the user to let him understand that an error has occurred and give him all the available information to decide what to do.
A classic exception stack trace is not an option: together with the exception text it contains a lot of information to localize the code that generated the exception.
This information, that is precious for the developer, is probably unuseful and confusing for the final user.
For this reason I wrote a small function that transform the exception in a string that contains only the texts of the exception and of all contained (cause) exceptions. Here is the code:

public static String flatException(Throwable e)
{
StringBuilder b = new StringBuilder();
while(e != null)
{
b.append(e.getMessage()).append('\n');
e = e.getCause();
}
return b.toString();
}

Even if the number of contained exception is large, the message is generally simple and gives immediately an idea of what happened.
This is not intended as a substitute of the classic stack trace, which is written in the application log file so that if needed it can be examined by the developer.

Fit Columns

2008-06-01T12:50:00.001-07:00

A new feature that will be released with version 1.3 is the possibility to fit the width of the columns of the table in a matrix viewer or in a presentation viewer.

This has been a problem from the moment I started to use virtual tables for the two viewer.
Virtual tables have the advantage that they can show tables with a huge amount of rows (> 10000) instantly, because they load only the rows of the table that are displayed in that moment, not the whole table.
As a side effect, there is no way to automatically set the width of the columns so that they can fit the text displayed in the columns' cells.
Because of this, before version 1.3 the use has to manually change the width of the columns.
In version 1.3 I added a menu in the two viewers, called Fit Columns:

This menu calls a function that takes a sample of the table's content: a fixed amount of rows. It measures the lengths of the texts contained in the cells of these rows, and with them it calculates an estimation of the columns' width. Then sets the columns' with to this value.

In conclusion there is no automatic columns' width calculation. The user decides when to calculate them. And that is a good thing, because if the matrix or the presentation change (because matrices get recalculated in Matrex), he can click the menu when he wants to update the width accordingly.

High Low charts

2008-05-29T13:38:00.001-07:00

One of the new features that will come with version 1.3 are the High Low charts.

It is a kind of financial charts that shows the evolutions of the quantities high, low, open, close and volume in a time period.
As all the other charts in Matrex, the charts are built using the JFreeChart library.

They can be shown in two variants: candlestick chart (top chart in the following picture) or high low chart (bottom chart). The picture shows the chart viewer for two high low charts that share the domain axis:

To build this kind of chart there is a special editor. In the following picture we show the editor for the two charts in the viewer.

There is one row in the table for each chart in the viewer. The values of high, low, open, close, volume are 1-column matrices in the same Matrex project.

The time period that defines the domain axis can be defined in two ways:

it can be a 1-column matrix containing date/times.
it can be an interval defined as [start,step]

The editor for each row of the table (which opens if you click the Edit button) is the following:

which shows the values for the first chart in the viewer: it is a candlestick chart, with the time period composed by an interval [start,step] and has all the values (high,low,open,close,volume) assigned to 1-column matrices of the project.

SWT Web installer RC1 released

2008-04-04T03:31:00.001-07:00

The documentation is in the SWT Web Installer site, and you can download it from SourceForge.
Please report any bugs/problems/feature requests.

Moreover, there is a new version of the client/server architecture document in the Matrex site.
The main change is that the document considers also sequences of operations on the project that need to be executed atomically (transactions).

Status

2008-03-31T13:25:00.001-07:00

SWT web installer

SWT web installer is almost ready to be released.
I tested it and wrote the documentation. It just needs the final touches.
I think it will be released during this week as version 1.0 beta.
License is LGPL 3.0.

ISQL

I made a library of ISQL, which I will release as soon as possible.
ISQL can be actually using separately from Matrex.
It is just to call a function with, as parameters, the SQL query and the input and output arrays.
I think it can be very interesting for people that wants to use SQL on arrays in memory.
Also in this case the license will be GPL 3.0-

Matrex

There will be a version 1.3 of Matrex.
The idea is to include in this version the last changes made to the source code of Matrex:

new classloader and plugins structure
SWT web installer to install SWT from internet
separated ISQL library

Also, the 2D charts will be updated and new Matrex functions will be added, thanks to the new version of the commons math library .

I'm also working to plan version 2.0 of Matrex. I tested an example of RMI callbacks and I've found that it works very well. I will soon publish a new document about the future Matrex Client/Server architecture.

The SWT Web Installer

2008-03-13T14:08:00.001-07:00

You wrote this nice SWT application and you want to release it.
And you realized it is not so easy, especially if the user has to install it himself.
Yes, because there is a different SWT version for each platform: a version for Windows, a version for Mac OSX, a version for Linux... not only, but also different versions for 32 and 64 bits.
What you do? You build a setup file for your application for each platform?

With these thoughts, I wrote a library that does the job for you.
It is called OS dependent installer (osdepinstaller).
You call it from your installer or from your application and it installs the right SWT version for the platform where your application is installed.

It works in this way:

it loads from the internet a file containing a list of links to download the SWT library from, one for each platform.
It checks the platform in which it is installed (two properties, os.type and os.arch).
it gets the link for that platform
it downloads the SWT library (it is a zip file)
it unzips the SWT library in a specific directory

If something goes wrong (for example the platform was not found in the list of links), osdepinstaller offers other solutions to the user.
For example if the automatic choice of the platform does not work, the user is asked to do the choice manually.

A principle of osdepinstaller is to give all the information about what it is doing. In this way the user can continue manually if for some reason the library is not able to continue.

Since osdepinstaller cannot work without internet, and it can happen that the internet can be accessed only through a firewall or a proxy, the system checks with the user if there is a proxy and, in this case, if the proxy needs authorization.

To get data from the user and to show the status osdepinstaller uses by default the console (since it is normal that your application does not have a GUI until osdepinstaller has finished), but it can be easily adapted to other solutions.

Also, osdepinstaller is not only for SWT: it can be used for any library the is platform dependent.

Osdepinstaller is still in test, but you can download a test version and check the code.

I will use osdepinstaller to let the user install SWT with the generic installer of Matrex.

Matrex 1.2 for MacOSX alternative setup

2008-03-10T04:52:00.001-07:00

I've got some one reporting me that the Matrex 1.2 version for Mac OSX does not install perfectly on all Macs.
The problem is that sometimes after the setup Matrex does not look like an application but like a simple java folder.
Surely you can open a terminal and run the shell script (the batch file) from there, but this is not really an intuitive way of running it!

I use IzPack to install my software and IzPack claims to be able to install also on Mac OSX, even if from some messages in mailing lists it seems that there could be some problems.

I tried to see what I can do.

First I will modify the shell script so that it can be called from everywhere (an initial statement to let it use the current directory as starting directory).
Then I will update Izpack to the last version (3.11).

But one thing that I would really like to do is to create a MacOSX application bundle containing the Matrex files.
The application bundle is the natural way applications are installed on MacOSX.
To do this you need:

a .icns file, that contains the application icon.
a way to generate a .dmg file, which is an autoinstalling disk image containing the application bundle.

There is no way, as far as I know, to generate .icns and .dmg files outside MacOSX.

So, if someone has a Mac and wants to help me to build this application bundle and to test it, can he please let me know?

List file class loader

2008-02-27T14:40:00.001-08:00

Matrex is composed by several jars:

The Matrex jars: api, gui, fun.
The libraries used by Matrex, for example SWT, jfreechart.
The plug-ins, for example the library to connect Matrex to Matlab.

To have java loading all these jars when starting Matrex, they was listed in the classpath parameter of the java call.
Here is the typical content of a batch file starting Matrex (it uses the two variables MATREX_CP and LIB_CP):
MATREX_CP=matrex_api.jar:matrex_fun.jar:matrex_gui.jar
LIB_CP=freehepj3d.jar:jython.jar:commons-math-1.1.jar:jfreechart-1.0.8a.jar:jcommon-1.0.12.jar:jxl.jar:lucene-core-2.0.0.jar:swt/swt.jar:antlr-runtime-3.0.1.jar:javacsv.jar:net.sf.paperclips_1.0.2.jar
java -Djava.library.path=swt -Djava.util.logging.config.file=logging.properties -classpath $MATREX_CP:$LIB_CP matrex.gui.Start $1

This approach has several problems:

It is not readable, and it becomes less readable adding more libraries.
If the user wants to add a plug-in, he has to edit the batch file. This is unconfortable, because the user needs to learn the syntax of the batch file, and dangerous: if the user does a small mistake changing the batch file the program does not start.
you need to have one copy of this batch file for each platform in which Matrex is released (Windows, Linux, MacOSX...).

In these days I worked on a new approach. The idea is to have 3 files containing the list of jars needed to start Matrex:

matrex.cld containing the main Matrex jars
libraries.cld containing the libraries used by Matrex
plugis.cld containing the plug-ins (it is empty when Matrex is installed)

Here is an content of the matrex.cld file. The other two files follow the same rules:

matrex_api.jar
matrex_fun.jar
matrex_gui.jar

To load these files and to load the jars listed in them, Matrex uses a custom class loader, TextFilesClassLoader.

The start class of Matrex is now called MatrexLoader.
MatrexLoader loads the main class of Matrex, Start, using TextFilesClassLoader as class loader.
Since Start all the other classes of Matrex (included the ones in the library) are loaded, directly or indirectly, from Start, they all use the same class loader used by Start, TextFilesClassLoader.

In this way it has been also possible to write a special Matrex GUI dialog, PluginsDialog, to update the content of plugins.cld:

This is much easier than to change manually the batch file.

Matrex functions unit testing

2008-02-06T12:41:00.001-08:00

I added a small chapter at the end of the function coding document about the Matrex functions code unit testing.

Before version 1.2 some of the Matrex functions had JUnit unit testing, but it was very limited.
In version 1.2 I had to write the code for the ISQL function template, which is very complex compared to the other templates.
So I needed to test it well.
For this reason I wrote some unit testing help classes (for JUnit) that I grouped under the package matrex.item.test.
These classes can be used for the unit testing of any Matrex function template.

With these classes unit testing of matrex functions becomes very easy. For standard function templates it is a matter of minutes to write a test case.
Give a look to them!

Matrex 1.2 released

2008-02-04T13:30:00.001-08:00

Finally Matrex 1.2 has been released!

The big step forward of this version is the new ISQL module.
The ISQL module is a function template to apply SQL queries to vectors in a Matrex project.
It is therefore possible:

to apply filters on parts of the vectors
to order the elements of the vectors
to aggregate them

all with simple SQL queries. The internalsql document contained in the distributions explains how.
As far as I know, no other spreadsheet application allows something similar.

Other enhancements are:

Print matrices and presentations.
Export matrices and presentations to csv and xls.
Import data from database, using an SQL query.
Menus to remove multiple items, duplicate matrices and functions, add function from a matrix (the function will have the matrix as first argument).
A summary step for the expression parser.
Possibility to change the colors of the windows headers.

Also, several bugs have been removed (dates formatting, items renaming, boolean matrices editing).

Pre-release of Matrex 1.2 to test

2008-01-23T04:21:00.001-08:00

I'm in the test phase of the version 1.2 of Matrex.
If someone wants to test this pre-release version himself, he can:

download the pre-release generic setup
install it
install the SWT library for his platform in the swt directory of the installation
use one of the matrix_generic.* files in doc/batch_files to launch Matrex

The changes from version 1.1 are in short:

The new ISQL (internal SQL) module. There is a document about it in the setup.
New import and export modules.

If you find some bugs, please report them in the Matrex bugs page.

Thanks

Version 1.2 status

2008-01-13T12:49:00.001-08:00

Coding and documentation are finished.
I'm now working on an example project for the ISQL module.
After that I will terminate the test of the system and release it.

There will be a version for:

Windows 32 bits
Linux 32 bits
Mac OSX

and also a generic version, which you can use if you want to install it in a different environment (64 bits?).
The generic version does not install SWT. You have to get it and unzip it yourself under the swt directory.

Status

2007-12-19T12:08:00.000-08:00

Today I finally released matrexjruby 1.0.2 (you can find it in the Matrex download page). It took some time to do the final test because of the frenetical work on Matrex 1.2.
I'm now working on the last pieces of code of Matrex 1.2, since ISQL has now been tested and I consider it finished:

the code that allows to add a function from the matrix tree using as first matrix parameter the selected matrix.
the code that allows to create an item from the item selection dialog box.

Once these are finished, I need to do some more test and update the documentation. I hope 1.2 will be released in january.

MatrexJRuby

2007-11-01T03:39:00.001-07:00

I updated the code of the adapter matrexjruby, which was still based on JRuby 0.9.
Now it works with JRuby 1.01.

Only a few fixes was needed (use the Ruby class instead of the IRuby interface, let the JRuby classes inherit from an abstract class instead of the IFunction interface).

Matrexjruby is used to write the scripts that are the "code" of the Matrex function templates.

I just need some final test and I will publish it as version 1.02.

See the light

2007-10-30T00:12:00.001-07:00

Finally!

I 've got stock with the internal sql module for a long while, I'm now able to see some results.
The biggest problem derived from the fact that I made the case... when... then... clause originally too complex (the when clause was too general).

Now I'm able to run some "select ... from ... where..." queries (without group by or order by clauses).
The code is on CVS under the package matrex.fun.sys.sql , so you can give a look yourself if you like.

It looks like it will not take so long before I have an implementation of the group by clause (together with the aggregate functions).
Still some doubts about the order by clause (allow order by or only order by ?).

So, now we should be closer to version 1.2, if nothing comes in the way.

Antlr for the SQL module

2007-10-04T23:24:00.001-07:00

To build the internal SQL module for Matrex I decided to work in the following way:

Write a parser that converts the SQL expression in an internal object structure
Write the code that applies the parsed SQL to the matrices/vectors arguments of the SQL function.

To do the parsing work, I chose the Antlr library.
Antlr has the following advantages compared to the other parsers:

More people use it (at least it looks like)
The parse result can be code in different languages (Java, C#, python...) that can be useful if I want to port the same grammar to other projects
Together with the library you can download a graphical application called AntlrWorks to interactively test and debug your grammar. AntlrWorks is a very good tool, that let you find errors in your grammar before you start to use it.

Antlr is a wonderful product, but I suffered creating the SQL grammar I needed.
The reasons are probably:

my inexperience in terms of parsers/lexers
some confusion and some holes in the free documentation

My initial idea was to download Antlr, get a grammar describing the SQL SELECT statement, adapt the grammar to my needs, build the java sources from it and convert the produced AST trees to my internal structures.

Simple, right? Wrong. Here are the problems for this approach:

Antlr is in version 3 now (is normally called v3). The example grammars are most made for version 2 (2.7.x). Altough v2 and v3 grammars look very similar, to convert a v2 grammar to a v3 one is not easy.
It is possible to buy a book written by the Antlr's author. I did not want to buy the book because I'm not planning to use Antlr in the future. But then I discovered that the online documentation is partial and often referring to the old 2.7.2 version.
The produced java classes can have a method to get the AST tree, but as far as I understood the tree cannot be used for an interpreter, but only to check the result of the statement parsing.
To build an interpreter is not the only purpose for using a parser:
Antlr is used for many other things, for example to compile, which
means convert expressions from a grammar to another one. Consider this to avoid to get confused reading the documentation.

I struggled for a pair of weeks with these problems. At the end I was able to to understand the following concepts and to produce my grammar:

You can add java code directly in the grammar. With this code you can build the interpreter structures directly in the generated java code.
Be very careful about the case of the initial letter of the rule names. Upper case: lexer rule; Lower case : parser rule. It looks simple, but if only the initial letter of one rule name is wrong nothing works as it should.
The lexer is used to parse single words (identifier, strings, numbers). The parser is used to parse phrases.
Spaces are handled automatically by the parser.
In the java code that you add to the grammar you can set the package of the generated classes.
AntlrWorks generates two types of java classes: the ones to use in your application and the ones that it uses to debug. They are saved in the same place with the same names. The debug classes don't work in your application, so remember to generate the application classes after a debug session.

So, if you give a look to the grammar I have written, you'll see that it is confused (rules definitions together with java code), but it works. When I run the generated java classes against an SQL expression the structures declared in the @members block are populated with the correct values and from them I can interpret the expression.

Version 1.2: status

2007-09-24T00:20:00.001-07:00

Not so much has been produced in the last period, for the following reasons:

holiday
got sick for one month
some time used understanding the concepts needed to write a grammar to parse the new internal sql template.

I'm still writing the grammar for the sql template. Soon I will be able to produce some code.
The rest of the coding of version 1.2 is more or less in place.
It clearly needs to be tested.

JRuby released a new version, so I will soon update the Matrex JRuby adapter.

SQL on matrices in Matrex

2007-07-31T12:32:00.001-07:00

I'm planning to write a new Matrex template to query matrices (only the ones with one column) using an SQL-like syntax
The idea is simple:

consider the input matrices/vectors as columns of tables
query them using SQL
the SQL query result are the output matrices/vectors of the function.

The query is written in a language that is very similar to SQL. The differences are:

the tables are defined in the query as sets of input matrices
sub-queries are not allowed

Operators (+, - ...) and aggregate functions (sum, max...) are available.
For example, a query can look like:

select sum(a.price) as sumprice, sum(a.volume) as sumvolume, b.ticker as ticker, b.validity as validity
from [dealticker, price, volume] a, [ticker, validity] b
where a.dealticker = b.ticker
group by b.ticker

Here the matrices dealticker, price, volume are grouped together to form the table a, of which they are columns;
the matrices ticker, validity are grouped together to form the table b, of which they are columns.

Every result column needs to be assigned a name to; that is done using the as keyword.

All the table columns (e.g. a.price, b.ticker...) need to be expressed in the form [table].[column] (e.g. a.price).

This will be one of the most powerful templates in Matrex. With it it will be possible to express very easily and clearly problems that otherwise could require the combination of multiple functions.

Since it is not dependent by new features of the upcoming version 1.2, it is possible that the template will be available before 1.2 is released.

About nebula

2007-06-26T23:13:00.001-07:00

Nebula is a sub-project of eclipse that produces custom SWT widgets.
I'm very interested in the grid, because it is supposed to have spreadsheet features like cells selection and row headers, which are very important in Matrex and I can just emulate in the SWT.Table widget.
Nebula Grid is still in alpha state but it should become final when Eclipse 3.3 is released.
I will check it (expecially with a big number of rows/columns) and possibly integrate it in Matrex in version 1.2 or 1.3.

Jython debugger?

2007-06-20T12:39:00.001-07:00

You can write Matrex functions as Jython or JRuby scripts.

Jython is included in the Matrex setup, so you can just use it.

To debug these scripts you can use the log functionalities of Matrex.

But today I've found out that there is a better possibility. With PyDev your can remotely debug the script from Eclipse.

So, if a script is not working as it should you can put breakpoints, watch its variables until you don't find the problem.

I will test it myself and write the test results in an article in this blog.

Work started for version 1.2

2007-06-20T00:52:00.001-07:00

Work started for version 1.2.

Version 1.2 will indicatively contain:

Export matrix view and presentation view to Excel file.
Import matrices from database tables (like the import from Excel files).
Import matrices from CSV files (like the import from Excel files).
Remove multiple items (matrices, functions...) from a project in one shot.
Print matrix view and presentation view.
The matrix source viewer.
More database functions templates (conditional query, update)
Use more JFreeChart features: composite charts, annotations, pie charts with exploded section, step charts.

With version 1.2 a first pre-alpha version of the Matrex server will be delivered.

1.2 will probably be released in september/october. If some of the new features are not ready for that time, they will be moved to version 1.3.

If you want to see in version 1.2 some feature that I did not consider, just add a comment to this article or send me a mail.

The matrix source viewer

2007-06-09T14:01:00.001-07:00

From a mathematical expression the expression parser generates matrices and functions in a Matrex project. This is a powerful functionality in Matrex.

Once the expression parser has generated matrices and functions, the original expression gets lost. This is usally not a problem, since in the project there are all the matrices and functions (final and intermediate) that calculate the expression.

But sometimes you want to see again the original expression, to:

see in one shot how a matrix has been calculated
know exactly which matrices and functions have been generated by the expression, for example because you want to replace the original expression with a new one and therefore you need to remove them

Fortunately the info window shows, among the rest:

for a matrix, the function that has it as result
for a function, the input matrices

So it is possible, from the expression's result matrix, to find out the original expression navigating to the source function, to the input matrices and so on.

But it is a lot of work to follow all the links among functions and matrices, expecially if the original expression is long.

To have a more valuable information, version 1.2 will contain a new functionality, called matrix source viewer.

The idea is to have a window that shows this information as a textual expression, with the same format used to enter an expression in the expression parser.

Suppose you entered the expression:

log(x + exp(y + z))

in the expression parser, and gave the name logres to the result matrix.

If you open the matrix source viewer on the logres matrix, you'll see a window containing a text like:

log ( logres.log )

Clicking on logres.log this part of the expression will be expanded and you'll have:

log (x + logres.exp )

Clicking on logres.exp this part of the expression will be expanded and you'll have:

log ( x + exp( y + z))

In this way with a limited number of clicks you'll have the original expression.

Also, a context menu on each item in the expression (excluded the parentheses) will open viewer or editor for that item.

Now the question is: why to see the original expression I need to click on links? Could it not appear as it was immediately?

The problem is that x, y and z could also be the result of other expressions and Matrex does not know which matrices are part of an expression and which are part of another.

The expression parser

2007-05-20T05:58:00.001-07:00

While the Expression Parser is not technically fundamental in Matrex (one can use the Function Editor instead), it has become more and more central in the last versions.

The reason is simple: if you have a long expression (or formula as it is called in a classical spreadsheet) to write, opening the Function Editor for each function in the formula can be time consuming and potentially irritating.

Using the Expression Parser is much faster: you write the expression, as it is, in a text editor and in a few steps all the required functions and matrices are added to your project.

The first thing to do to use the Expression Parser is to select the Expression Parser menu in the function tree (or press Ctrl-X).

This opens the Expression Parser wizard:

In the text editor you write the expression to add to the project. The expression is written as a combination of function templates, operators (+,-,*,/,^) and matrices, like:

cos(sin(x) + sin(y))

where x and y are matrices names.

You just need to write the names of the expression parts, their packages are not required.

If you don't remember the name of a function template or a matrix, the Insert.. buttons allow you to add templates and matrices to the expression selecting them.

The expression is first parsed to a tree:

Then the templates repository and the project are queried to determine the complete name-package of each part of the expression:

At the end the name of the target matrix and function are requested. All the intermediate matrices and functions are written in special packages.

Summarizing, entering an expression in Matrex is different from entering a formula in a classic spreadsheet because:

Spreadsheet formulas work on single cells, Matrex expressions on vectors/matrices.
Therefore, spreadsheet formulas must be copied for each result cell, Matrex expressions must be written only once.
Spreadsheet formulas are attached to cells, Matrex expressions are used to generate functions and matrices in the project.
Therefore spreadsheet formulas can be updated as whole (each cell), whether in Matrex the single generated functions and matrices must be updated individually.

This last point is not always an advantage for Matrex. Once functions and matrices are generated, the original expression that generated them is lost.

To solve this potential problem, Matrex version 1.2 will have the following new features:

It will be possible to see the origin of a matrix as an expression: the functions and matrices of which the matrix is directly or indirecly a result. This feature will be object of an article on this blog.
It will be possible to remove multiple items from a project in one shot. In this way it will be possible to easily remove all functions and matrices generated by an expression.
The last step of the expression parser will display all the functions and matrices that will be generated. This will reduce the risk of generating functions and matrices from a wrong expression.

Help wanted

2007-04-27T06:53:00.001-07:00

This week I published a request in SourceForge for Matrex developers.

Matrex gives you fun when you program it and there are always new things to do, new features to add.

But it requires time and it could be nice to have some help.

So, if you are interested please let me know, answering to the announcement or sending me an e-mail.

5000 downloads!

2007-04-26T02:06:00.001-07:00

I hope for a much shorter time to come to 10000.

Article about Matrex in an internet japanese magazine

2007-04-23T00:47:00.000-07:00

This is the link to the article, written in Japanese.

Matrex 1.1 released!

2007-04-15T03:43:00.000-07:00

I'm glad to announce that Matrex 1.1 has been released.
The purpose of this version is to make the system more open, easier to use and complete.
The changes are the following:

Matrex is now able to import xls (Excel) files: open a Matrex project, from the project menu open the import dialog and load the Excel file:

Here you can select rectangles of cells that you import as matrices in the project.
Matrex has now a context help. Pressing Ctrl-F1 (or F1) in each window, the browser shows the help page for that window.
A copy of the help is available online.
The help has also a summary page and can be full-text searched in the search dialog:
The 2D charts have been simplified and improved. Now there are only 3 kinds of charts (category, xy, time), but you can display their series in up to 7 different ways (line, line3d, bar, bar3d, area, scattered, scattered line):
The Expression Parser is now able to interpret expressions containing +,-,*,/,^ operators. The operators are translated to the related functions.
Now it becomes really easy to build projects in Matrex.

Matrex 1.1: some delays...

2007-03-29T01:03:00.000-07:00

The development is terminated, the documentation is upgraded.
Even the changes in the Matrex site are done (FAQ, new videos).

The problem is the final test.
It takes time, and I will not be able to release Matrex 1.1 before the middle of April.

SWT: cell selection in Table

2007-03-21T02:18:00.000-07:00

The SWT Table allows only to select rows, not the cells in the rows.
This is a strong limitation for applications that use tables to do something more complex than showing the content of database tables.

In Matrex I needed to import xls (Excel) files and select from them rectangles of cells to add as matrices.
So, I built the TableWithCellSelection class, which allows the user to select rectangles of cells in the table.

It works listening the mouseMove, mouseDown and mouseUp events to keep track of the mouse movements and to change the background of the cells when they are selected.
The class works only for tables that are created with the SWT.FULL_SELECTION and SWT.VIRTUAL styles, which means that the table must be virtual.

Features: Clicking SHIFT together with the left mouse button the selection process restarts from a previously selected cells rectangle.

Limitations: It is not possible to scroll the table during the selection. To be able to select big rectangles you can use the SHIFT key.

Matrex 1.1 status

2007-03-20T02:12:00.000-07:00

Matrex 1.1 coding should be finished.
What is missing is testing and documentation update.
The hope is to release it before easter.

More features in Matrex 1.1

2007-03-07T04:33:00.000-08:00

The next release of Matrex, 1.1, will also contain the following features:

use of operators (+,-,*,/,^) in the expression parser. They will be converted respectively in the functions plus, minus, times, divide, pow.
import and export of projects from/to zip files. In this way it will be possible for example to archive a project or to send it by mail.

New symbol

2007-03-05T03:57:00.000-08:00

Matrex got a new symbol, used both for the application icon and for the graphics of its web site.

The symbol should suggest the idea of a matrix, with the word "Matrex" written over it.
It should look more professional than the old falcon symbol, that did not really have so much to do with the main concepts of Matrex.

I made the symbol using OpenOffice.org and GIMP.

Why should I use Matrex if Excel 2007 supports multithreading?

2007-02-21T04:18:00.000-08:00

One of the peculiar features of Matrex has always been the fact that functions are executed in different threads.
Now Excel 2007 supports a similar feature, so one could think that there are less reasons to use Matrex instead of Excel.
But that's wrong: the multithreading capabilities offered by the two products are different.

Purposes

Excel

Multithreading has been added to Excel to use both the cores in the new double core processors (Intel Core Duo and AMD X2).

Matrex

Matrex has introduced multithreading before these processors became popular and the purposes are the following:

Don't block the GUI during the function calculations
Get immediately the results of some calculation without to wait for the whole project (worksheet) to be recalculated.
Avoid that I/O dependent functions (formulas) block CPU intensive functions.

Sure, Matrex is also able to use both the cores of a double core processor, but that is only a side effect of its multithreading feature.

Implementation

Excel

I don't have details on the implementation, but that is what I think happens:

When you recalculate a sheet the first time, all the dependencies among cells of the sheet (formulas) are examined and based on this the sheet is divided in 2 parts (or 4 if you have 4 cores) .
Each part is calculated by a thread.

Matrex

Matrex uses a pool of threads. Each function is calculated by a thread of the pool.
In this way the whole Matrex internal engine is multithreaded.
Only the GUI runs as a single thread, with which the other threads need to synchronize when matrices, charts or presentations are displayed in the viewers.

Conclusions

Multithreading has been just added to Excel to use well the dual core processors, but it is not a main feature in Excel: a part of the fact that the calculation is divided in 2 or more threads, the system works as in the previous versions.

Matrex is instead built on threads. They are everywhere: functions, matrices, presentations, charts, timers and the GUI have to cope with them.

Printing support

2007-02-08T23:58:00.000-08:00

To print, version 1.0 of Matrex converts the content of the table (matrix or presentation) to a web page and opens a browser with it. So you can print the web page from the browser.

This works fine, but is not very professional. So why I don't print directly from Matrex? The reason is that SWT, the user interface library I'm using, does still not allow to print in Linux.

But it seems that this will change in SWT version 3.3. So I will surely add the possibility to print directly from Matrex in version 1.2, or may be even 1.1 of Matrex.

Help on line

2007-02-04T04:14:00.000-08:00

Today Matrex got an help on line!

The plan for version 1.1 of Matrex includes a context help.
The context help is made as a set of linked html pages, one for each window and dialog of Matrex, so that it is possible to open the right page when pressing F1 on one window.
This feature is more or less ready and I thought that it could be nice to have already all the help html pages as a help on the web.

So, if you have a problem understanding what is the purpose of the view, open the content page of the help, click on link for the window and read!

The content page is also reachable from the documents web page of the Matrex site.

Matrex 1.1 plan

2007-01-08T00:45:00.000-08:00

It is time to start to work on the version 1.1 of Matrex.
The main new features of Matrex 1.1 will be:

Context Help
Excel files (.xls) import.
Possibility to use bar series and line series in the same chart.

Context help

2007-01-03T04:30:00.000-08:00

One of the features of the next version 1.1 will be the context help.
This means that pressing F1 in a window Matrex shows some help for that window.
This will be a winning feature for Matrex, because it reduces the time needed to learn how the system works.

The content of the help will come from the PDF document How to use Matrex.
The idea is to make a map of all the windows in Matrex and assign a piece of the document as an HTML page to each window.
Then refine the help text to make sense for each window and add links among the various help pages.

To maintain the help system and the PDF document updated in the next versions I will:

update the HTML pages of the help (change them, make new pages)
build the PDF document from the HTML pages using HTMLDOC. HTMLDOC is able to merge several HTML pages in one PDF document and is free.

Mac OSX Matrex start

2006-12-08T02:51:00.000-08:00

In the Matrex Sourceforge pages Kusashi Nameta (JPN) reports the bug that the Mac OSX version of Matrex does not start correctly.

The problem is that I can test the system only on Windows and Linux, so things like that can happen. There should be someone testing the Mac version before it is published.

The solution, as you can see in my comments, is to add the -XstartOnFirstThread option to the java command.
To do that, edit the matrex_macosx.sh file in the Matrex directory and to:

java

subtitute:

java -XstartOnFirstThread

This should make it work correctly.
I'll publish myself a patched version of the Mac OSX setup file this weekend.

What now?

2006-11-26T22:51:00.000-08:00

Plans

After the release of Matrex version 1.0, it is time to make some plans for the future.

By now, I started to write the specification for the client/server architecture. This should start the process that takes to version 2.0 of Matrex.

After that:

some adapters must be updated: I'm thinking about the adapter to JRuby, which must at least be tested for the last version of JRuby and the adapter to Scilab that will soon reach version 4.1.
as usual I will try to test the adapter to Matlab, which has been ready for one year, but was never tested because I don't have Matlab. If someone wants to help, please contact me.
I will try to produce an adapter for the R statistical language.
start of the work to produce version 1.1.

Next versions

First of all, this is how I will number the next versions of Matrex:

A change in the first number means a major change: 2.0 will introduce the client server architecture.

A change in the second number means new important (visible) features: 1.1 will introduce the package to interact with Excel, the help framework and changes in the charts.

A change in the second number means bug fixing and minor feature changes: if a version 1.0.1 will be released, it will fix bugs in version 1.0.

I think these are the features to add in the next versions:

Package for the interaction with Excel: should contain import/export of Excel spreadsheet files, communication Excel->Matrex and Matrex->Excel via Java/COM.
Package for the interaction with OpenOffice.org: communication OpenOffice.org->Matrex and Matrex->OpenOffice.org (import/export can be done through .xls files).
Charts new features: scatter plot chart, possibility to have bars and lines in the same chart, chart attributes like line or bar type, vertical/horizontal bars.
Context help. Convert the "How To Use" document to a context help in the Matrex desktop application.
Add first phase in the expression parser where expressions of the type "a + b" (+,-,*,/,^) are converted to expressions of the type "plus(a,b)". This makes the expression parser simpler to understand.
Zip/unzip of projects, so that they can be easily sent to other users.
Custom dialogs to enter function parameters. They are needed for complex functions like query, csv and, in the future, link.

The idea is to publish these features in a set of 1.x versions (1.1, 1.2 ...) .
These versions should be released every 2/3 months to keep up the interest for the product.

The first major version will be Matrex 2.0, which will introduce the client/server Matrex.

Version 2.1 will (probably) introduce links between projects (also between projects running in different machines).

No more M1, M2 version will published; an RC1 version (or more if needed) will be released before the final version to have some feedback.

Matrex 1,0 final released!

2006-11-17T12:57:00.001-08:00

Finally, after weeks of test, Matrex 1.0 final has been released!
The most important changes are the following:

Expression parser: the previous version was complex to use, expecially in the last step, where you had to choose the names of all the matrices and functions generated.
With this version the last step requires only the names of the primary (root) matrix and function. The intermediate matrices and functions get a default name and are stored in a special package.

Also, in the first step, two buttons have been added to select templates and matrices and add them to the input expression; so if you don't remember a template name or its parameters or a matrix name you get help.
Matrix editor: removed side effects when editing cells using the mouse; the editor remembers saves also the last edited cell content; better number and date parsing.
Added several function templates: among them:
- sort, to sort several matrices using as index the first matrix, called key (like the pair of a map).
- sumby and similar, similar to the select sum/group by functionality in SQL.
- tail to tail a matrix with the content of another one.
- queue to use a matrix as a queue, to which you queue values of other matrices
Removed any dependency of the Matrex API by the Matrex GUI, finally allowing to use Matrex as a library. In this way a Matrex project can be opened and used by any (java) application.
Presentation editor: better and easier interface. The format/position editor has been put in a separate dialog and substituted by a format/position viewer:

Presentation viewer: added vertical header:

Also, it uses virtual tables. It is possible to view/edit large presentations without performance losses.
Matrix viewer: added vertical header:
Added a template full-text search dialog to find templates by their description:

Matrex 1.0 final

2006-11-16T22:49:00.000-08:00

I will probably release Matrex 1.0 final this night.
After a very hard test (GUI, events, templates) I think Matrex is ready for production.
This version has very few but important changes from RC1: easier to use and more professional GUI, more function templates and the Matrex API.

RMI for Client/Server?

2006-10-18T05:25:00.000-07:00

Matrex 1.0 final is almost ready, so I'm thinking about the future of the system.
As you know, there will be at least two main versions after 1.0:

1.1, with improvements in various areas of the desktop application.
2.0, which will introduce the client/server architecture

With version 2.0, it will be possible for a Matrex desktop (or client) to connect to a Matrex server.
The server will host Matrex projects shared among serveral clients.

Matrex has been tought from the start as configurable as a client/server architecture, so it should be not very difficult to introduce the new architecture.

Anyway, some problems remained unresolved for a while, for example which protocol to use to connect client and server.
To choose it, I considered these facts:

The clients and servers can reside on a LAN, it does not make sense to have internet connections.
At least in a first stage only Matrex java clients will connect to the servers. If other programs need to use Matrex to calculate they can directly use the matrex API.
Performance is very important. The users expect the system to calculate fast even if it works in a network.

Here are the candidates possibilities:

SOAP
CORBA
Jini
RMI

The advantage of CORBA and SOAP is the compatibility with different platforms and languages; but this is not needed.
SOAP messages can pass through a firewall, but since Matrex just needs to work in a LAN, this is not needed.
On the other side, SOAP is the slowest protocol and CORBA is loosing popularity.

Jini gives high realiability and performance, but it is very complex. Using Jini for Matrex is like to hunt birds with tanks.

RMI is the right choice for Matrex. It is sure not the most recent (it was available already in Java 1.1) but it works fine, is fast and requires less coding, because is made exclusively for java.

SWT Wizard Component

2006-10-02T10:17:00.000-07:00

I extracted from the code of Matrex 1 class and 2 interfaces to implement a generic wizard with the SWT GUI library.
It is used to implement the expression parser in Matrex:

You can find the code under the wizard directory in the Matrex CVS repository.
The license of Matrex is GPL, but may be for the wizard code it can be changed to LGPL.

Function Templates list

2006-09-26T23:28:00.002-07:00

I added in the Matrex site a list of all function templates with:

name
description
input matrices
output matrices
parameters

I hope this will help to:

find fast which template you have to use to write your function
find out templates that are needed but missing
find out fails or imprecisions in the templates descriptions

SWT: table vertical header

2006-09-26T04:47:00.000-07:00

Matrex RC1 introduced a vertical header column in the matrix editor table, to show the row index or, if needed, to show a vector as a vertical ruler.

The SWT table does not allow to have a vertical header column as needed for Matrex, i.e. such that:

is a fixed column in the table (it remains always visible scrolling horizontally)
has cells with raised border to distinguish it from the other columns.

In version RC1 I decided to not loose so much time verifying if there was a work around, so I simply used as vertical header the first column of the table and giving it a gray color to distinguish it from the others.

In version 1.0 final I want to have a vertical header in other two tables, the matrix viewer and the presentation viewer, just to show the row index.
Therefore I started to see if there is a better solution than the one implemented in version RC1.

So I tested the SWT snippet 2 which should solve the problem using as vertical header a different table with only one column, synchronized with the original one.
This is a good idea, but has these problems:

the vertical header table appears with a vertical scrollbar, which lets it appear disconnected from the main table. The scrollbar can be hidden, but I'm not sure it can be done on any platform.
the cells have no raised border.
sometimes the rows in the two tables appear slightly unsynchronized. This happens because the main table shows its first row only partially.

These problems (expecially the last one) make this solution not very usable and definitely do not give a professional look to the application.
Therefore by now I keep the solution used in version RC1. I'll see in the future if there will be other possibilities.

Again the Matrex database project

2006-09-18T04:30:00.000-07:00

It looks like in the currently published version of matrexdb (in SourceForge) there is a logical error: the result column of the position presentation contains wrong values.
I will publish a new version of matrexdb this week with (hopefully) correct values.
This version will also contain a readme file with more details about the structure of the testdb project.

Matrex 1.0

2006-09-10T04:39:00.000-07:00

I'm working on the first production version of Matrex 1.0.
The changes from version 1.0 RC1 will probably be:

Virtual tables on presentations, to be able to show presentations based on very big matrices/vectors.
Row header for matrix viewer and presentation viewer.
Simplification of the Expression Parser when saving the produced matrices and functions. It will ask for the names of the final matrix but not of the intermediate ones.
New function templates (e.g. sumby, sort, asinh...). I'm trying to cover the most common needs.
Possibility to create matrices from the clipboard content, so you build a matrix directly from the copied block of a spreadsheet.
Better function debugging.
Better terminology (less X,Y more columns,rows).

If you have other ideas or needs please let me know!

The Matrex database project

2006-09-07T14:13:00.000-07:00

I published yesterday a Matrex sample project, matrexdb. You can find it under the Matrex download page in Sourceforge.
I published the project zip file alone because it is big (around 2.5 Mb), since it contains an entire database and the embedded McKoi library to access it from Matrex.

The database contains the data of a fake chocolate market (oh yeah), with products, spot and closing prices and more than 4000 contracts.
The database has been generated automatically by a Groovy script (also in the zip file), since it was impossible to find free data to build it. Even if fake, the data looks realistic.

The project queries the database using JDBC and contains a presentation (something like a read-only spreadsheet) showing the positions built with the contracts in the database (average prices, results).
The project contains also time charts showing the spot prices and the closing prices of two products.

The projects uses the function templates:

query to query the database
sumby to build the positions by product
lookup and at to connect the contracts with their products hours.

I think it is good demonstration of the features of Matrex.

FindBugs

2006-09-06T05:43:00.000-07:00

Since I have the need to find the most possible amount of bugs in Matrex to come to version 1.0, which is supposed to be stable, I tried FindBugs (http://findbugs.sourceforge.net/) with success.

FindBugs finds possible bugs in your code.
Or better, as th FindBugs site says: "FindBugs looks for bugs in Java programs. It is based on the concept of bug patterns. A bug pattern is a code idiom that is often an error."

FindBugs can be used as a standalone application or as an Eclipse plugin. Since Matrex is an Eclipse project I tried both the installations. The Eclipse plugin is easier for me to use, because I don't need to configure anything, just install it and use it on my project.
That's how it looks:

You can see that the potential bugs are listed in the Problems view on the bottom; Clicking on a bug line, you get a good explanation of the problem in the Bug Details view on the left.

FindBugs does not pretend to be infallible. The idea is to find areas in your code that can potentially produce problems. So it can happen that some of the potential bugs FindBugs found in your project are false positives, i.e. not bugs. When you can see that one of the bugs of FindBugs is, in all cases, a false positive, you can remove it from the list of bugs FindBugs checks, changing the configuration:

As I told, my experience of FindBugs in my project Matrex is positive.
I have to tell that the current version of Matrex is beta, but it was tested several times, so I would not expect to find too many errors.
Findbugs found around 200 possible bugs, which is a considerable number, but not too high to check them one by one.
These are the bugs it found:

EI2: public method exposed internal rapresentation: FindBugs found this bug several times in the code of Matrix.
One of the examples is in the Matrix class.
The Matrix class is a light wrapper around a double array, that gives to it some features, among them thread synchronization.
The array can be directly accessed from outside a Matrix object using the getValues method.
Another method, setValues, has to be used to change the array.
But some code could potentially access the array using getValues and change its content, causing inconsistencies.
FindBugs proposes that getValues returns a copy of the array instead the array itself. That's generally a good solution, but cannot be applied to Matrex, which needs high performance, expecially when accessing matrices.
So in this case all I could do is to add comment to the getValues method to tell how to use it.
Since all the warning of this kind are not bugs in Matrex and the code cannot be changed to remove them, I remove EI2 from the list of bugs pattern in the configuration.

IS2: inconsistent synchronization:
In some Matrex classes some of the methods was synchronized, some not.
That is actually very dangerous and I fixed it.

MS: is not final but it should be:
I forgot to declare some static constants as final. I don't find that dangerous because I know what is supposed to be constant in the Matrex code, but if more developers will work on the same code this bug could result in some problems.
I fixed it.

RCN: null check of previously referenced value:
I was calling a method of an object, then I was checking if the object was null (which is totally meaningless). In this case, I looked at the code and I could see that the object could not possibly be null. So, I removed the check. Nothing dangerous but confusing for someone reading the code.

SIC: should be a static inner class (was not static):
There was some cases in Matrex in which a class had an inner class that was not declared static, even if it could. No danger, but better to have clean code.
I fixed it.
URF: unread field:
I use the java.util.logging API to trace Matrex. This requires to declare a static log variable in the class that needs to be traced. It happens that sometimes the variable is declared but never used. Again no danger, but better to have clean code.
I removed the redundant declaration.

These potential bugs, expecially IS2 and RCN, are dangerous. So, I think FindBugs worked well, and I will re-examine the code of Matrex every time I will release a new version of it.

The only thing I can ask for is the possibility to exclude the checking of a potential bug category (like EI2) only for a certain set of classes: it can be that I know that these classes are correct even if FindBugs finds the bug pattern in them, but I still want to check the rest of the code for it.

Virtual tables

2006-08-27T22:54:00.000-07:00

In version 1.0 RC1 I used a feature of the SWT library, the virtual tables (SWT.VIRTUAL), in the matrix viewer.
This feature allows to show in the matrix viewer a matrix of thousand of rows and columns with a very good performance.
The idea is simple: since a table shows only a few rows at a time, you don't need to load the whole matrix in the table in one shot; you need to load only the rows that are shown in that moment.

My plan is to use the same feature also in the presentation viewer in a future version of the system (1.0 RC2?).

In this entry, I wanted to explain how I used the virtual tables in the matrix viewer.
For the implementation you can look in the CVS tree for the classes:

matrex.gui.viewer.matrix.MatrixViewer, the viewer

matrex.gui.viewer.matrix.MatrixViewerCache the cache for the viewer, containing the current version of the matrix (in other words, the viewer's model)

I based my code on the article SWT - Virtual Tables Tutorial.
As in the tutorial, I use the SWT.SetData event to fill the rows of the table. When the table shows a new row, it triggers a SWT.SetData event; in this event the code fills the row with data.
Differently from the tutorial, the matrix showed in the table can change its content at any time, and when this happens the table content must be updated.

Each time the matrix is updated, the code checks how big is the change, examining a small sample of items of the matrix:

if only a small amount of items of the matrix is changed, only those items in the viewer need to be changed.

Doing some test I discovered that extracting a row that the table did not show, using the statement table.getItem, the row is not null but each of its item is empty and the table does not try to populate it triggering a SWT.SetData event.
Therefore it does not make sense to update a row that was not shown before.

So, the viewer cache keeps an array that tells for each row if the row was shown before (for which an SWT.SetData event has been triggered).
When an item of a row needs to be updated:
- If the row was shown before it is updated immediately using the table.getItem.setText statement.
- Otherwise nothing is done: if the table later shows the row, it triggers the SWT.SetData event for that row, which reads its current, updated content from the cache.

Otherwise the table is cleared and filled up again with the new matrix; as we saw in this case the performances are good because the only rows that are loaded are the ones that are shown.
To clear the table I used the following statements:

table.clearAll();
table.redraw();
table.setItemCount([number of rows of the matrix]);

The last statement triggers the SWT.SetData events for the shown rows.

I tested this solution with small and big matrices, small and big changes and it seems to work correctly and with good performances.

The Matrex API

2006-08-27T22:53:00.001-07:00

From version RC1, the Matrex executable code is divided in 3 jar files:

matrex_api.jar containing the Matrex underlying structure, engine and file system interaction

matrex_fun.jar containing the java executable code for the function templates

matrex_gui.jar containing the code for the GUI

matrex_fun.jar was already there in version M2 (it was called function.jar), matrex_api.jar and matrex_gui.jar was merged in a single jar file, matrex.jar.

So what's the big deal?

Now you can use matrex_api.jar (probably together with matrex_fun.jar) as library in your own application.

For example if you have an application that needs to calculate a special function you can:

start Matrex and build a project that calculates the function (with several matrix and function items)

Let the application use that project (through matrex_api.jar and matrex_fun.jar) to calculate the function

In the future I will produce:

Javadoc documentation of matrex_api.jar

a guide for building an application that uses it

Matrix editor: custom headers

2006-08-27T22:53:00.000-07:00

Often in a spreadsheet you need to work in the same time with two (or more) vectors that are connected by a key-value relationship.
For example:

days in a period and the quantity of rain in these days

categories of people with the average books they read in a year

cities with their population amount

bank accounts and the amount of money in them

In a spreadsheet you normally write them as two consecutive columns or rows.
In this way you can use the static vector (days, categories, cities, bank accounts) as ruler for the value vector (rain, books, population, money).

In Matrex that was not possible before version RC1. Matrex uses a "spreadsheet" concept only in the data presentation; to edit the vectors/matrices, it uses a matrix editor:

The editor can only edit one single vector/matrix, not two in the same time.

In version RC1 I solved the problem using customized headers.

In the editor popup menu, I added the Set header menu. It has the sub-menus:

Set vertical header

Set horizontal header

With the first sub-menu (vertical) you select a matrix int the project. The first, static, gray column will show the content of the first column of the selected matrix:

The second sub-menu (horizontal) you change the content of the first, static, gray row so that it shows the content of the first row of another matrix selected in the project.

In this way you can use the header data as ruler to write the content of the edited matrix.

Navigation

2006-08-27T22:52:00.000-07:00

Matrex RC1 has a new feature, named navigation.

Navigation is based on the info window, which appear clicking on the info menu (Ctrl-I) on a tree item (matrix, function, chart ...).

The info windows are not new: they show some textual information about the item and the lists of items to which it is connected.
For example, the info window related to a matrix shows the function that calculates the matrix, the functions calculated with that matrix, the chart showing that matrix and the presentations containing that matrix.

What is new is that the each connected item displayed in the info window has a popup menu (view, edit, info).
Using the info menu you can navigate the connections of the project items: for example you can open the info window of a function, from it open the info window of one of the matrices that it calculates, from it open the info window of a chart that shows that matrix.

In this way you can, for example, know which items in the project change if you change the content of a matrix.

It is a powerful feature to understand the structure of a project.

The Matrex idea

2006-08-27T22:51:00.000-07:00

I'm working with Excel in my job.
Generally we use Excel to calculate formulas on data extracted from a database and present everything as sheets and charts.
This is how we do it:
- write the SQL query
- get the result set in a sheet
- write the formulas for the cells of the result set's first row
- copy the formulas for all the other rows of the result set

It is easy to copy formulas from one cell to the other in Excel, but why we need to do it at all?
It would not be easier to get the result set as one vector for each column, and calculate the formulas on this vectors, not on sheet cells?

So I wrote Matrex.

Matrex is equivalent to a spreadsheet, but works with vectors, not cells.

So, to do the same in Matrex, we need to:
- write a function that produces vector/columns from an SQL query
- apply the functions/formulas on the resulting vectors (only once).

Working with Matrex you have also these advantages:
- you can name vectors/matrices and functions (formulas)
- multithreading
- in the future, client/server and distributed calculation