Matrices and Advanced Data Handling

AFL is built around one-dimensional arrays: one value per bar, for one symbol. A matrix is the escape hatch for the problems where that shape does not fit — every symbol against every other symbol, a system of linear equations, a table you need to sort by a column. By the end of this lesson you will know exactly which matrix operations exist, which operator is the real matrix product and which merely look like one, and how to decide whether a problem is genuinely two-dimensional or whether you are about to make an array problem harder.

Creating and reading a matrix

Fragment — not a complete formula

m = Matrix( 5, 6, 9 );          // 5 rows, 6 columns, every element = 9
value = m[ 2 ][ 3 ];            // row 2, column 3 - both indices are zero based
m[ 0 ][ 0 ] = 1.5;              // elements are writable

The full signature is Matrix( rows, cols, initvalue, increment = 0 ). The optional increment creates a matrix whose elements increase monotonically instead of all holding the same value. Row indices run 0 .. rows-1 and column indices 0 .. cols-1, exactly as with array subscripts, and going outside that range fails the same way.

Matrix support arrived in AmiBroker 6.00. Matrix values became storable in static and dynamic variables in 6.10, and matrix identifiers became usable in if/else conditions in 6.90. Everything in this lesson is verified against 7.00.1.

✕Common mistake

The official Matrix page’s own example contains a typo: it creates x and y and then writes z = y - z;, where the surrounding text says the answer should be the difference between y and x. The language reference page has it right. If you copy that example and get an error about an undefined identifier, the example is wrong, not you.

Every operator is element-wise, except one

This is the sentence to remember. All the standard arithmetic and logical operators on matrices work element by element, and two matrices must have identical dimensions to be combined that way. A * B multiplies corresponding cells. It is not matrix multiplication.

@ is. A @ B is the true linear-algebra product: if A is n×k and B is k×m, then A @ B is n×m, and A’s column count must equal B’s row count or the operation is undefined.

Two operators that look similar and are not

Element-wise needs matching dimensions. The matrix product needs A's columns to match B's rows, and the result is a different shape from both inputs.

Bar	shape of A	shape of B	shape of result	cell (i,j) is
A * B (element-wise)	3 x 4	3 x 4	3 x 4	A(i,j) x B(i,j)
A @ B (matrix product)	3 x 4	4 x 2	3 x 2	sum over k of A(i,k) x B(k,j)

Element-wise needs matching dimensions. The matrix product needs A's columns to match B's rows, and the result is a different shape from both inputs.

One more semantic worth knowing: assignment copies. The official page notes that after z = m; and a subsequent write into z, “m will remain unaffected”. There are no matrix references to worry about — except in MxCopy, which is the one function that takes its destination by reference.

The complete function list

There are fifteen. Knowing that the list is closed is as useful as knowing what is on it, because it tells you which operations you will have to write yourself.

Function	Since	Returns	What it does
Matrix( rows, cols, initvalue, increment = 0 )	6.00	Matrix	Creates a matrix filled with initvalue
MxGetSize( matrix, dim )	6.00	Number	Size in one dimension: 0 = rows, 1 = columns
MxIdentity( size )	6.00	Matrix	Square identity matrix
MxTranspose( matrix )	6.00	Matrix	Rows become columns
MxInverse( mx )	6.10	Matrix	Inverse; undefined for a singular matrix
MxSolve( A, B )	6.10	Matrix	Solves A @ X = B for X
MxDet( mx, method = 0 )	6.10	Number	Determinant
MxSum( matrix )	6.20	Number	Grand sum of every element
MxSort( mx, dim = -1, ascening = True )	6.10	Matrix	Sorts values within rows or within columns
MxSortRows( mx, ascending = True, col1 = 0, col2 = -1, col3 = -1 )	6.10	Matrix	Reorders whole rows by up to three key columns
MxGetBlock( matrix, startrow, endrow, startcol, endcol, asArray = False )	6.10	Matrix or Array	Reads a rectangular block out
MxSetBlock( matrix, startrow, endrow, startcol, endcol, values = 0 )	6.10	Matrix	Writes a rectangular block in
MxFromString( "string" )	6.10	Matrix	Parses Wolfram, Matlab or Octave notation
MxToString( mx )	6.10	String	Wolfram-style text, for printing
MxCopy( &dst, src, ... )	6.40	Nothing	In-place block copy between matrices

Note the spelling of MxSort’s third argument in the official syntax line: ascening. That is how the page prints it. Pass it positionally and the question never arises.

A few of these have behaviour that is easy to get wrong.

MxSort( mx, dim = -1, ... ) has three modes. The default -1 sorts a single row if the matrix has one row, a single column if it has one column, and otherwise each column separately. dim = 0 sorts within each row; dim = 1 sorts within each column. It moves values, so it destroys the correspondence between columns.

MxSortRows is what you want when the correspondence matters — it reorders entire rows by the contents of col1, breaking ties with col2 and then col3. The documented hint for sorting columns instead is to transpose, sort rows, and transpose back.

MxSolve( A, B ) is preferred over MxInverse for solving equations, and the page explains why in a way worth repeating: MxSolve does all its arithmetic in 64-bit double precision and converts only the final result back, whereas X = MxInverse( A ) @ B converts the inverse back to single precision first and then multiplies. The documented consequence is that polynomial-fit code works better with MxSolve.

MxDet( mx, method = 0 ) has three methods: 0 chooses automatically, 1 is the slow Laplace expansion which is more accurate, 2 is fast LU decomposition which is less accurate. Because Laplace is O(N!), even method = 1 falls back to LU above 10×10. The page gives a concrete cautionary example: a singular matrix whose determinant LU-based tools report as −1.4495e−12 rather than zero.

!Important

@ is the only true matrix product, MxSolve/MxInverse/MxDet are the only linear algebra, and there is no matrix function for eigenvalues, singular value decomposition, QR or Cholesky. If a technique you have read about needs one of those, AFL does not have it and you will have to implement it or move that step outside AmiBroker.

Getting data in and out

The awkward part of matrices in AFL is the boundary with ordinary arrays, and it has one rule that catches everybody.

MxSetBlock( matrix, startrow, endrow, startcol, endcol, values ) fills the block “from left to right and from top to bottom with consecutive values taken from that array” — and it starts from element 0 of the array. Your bars of interest are almost always at the end of an array, so they have to be moved to the front of a scratch array first. If there are more cells in the block than values in the array, the counter wraps round to zero and starts again, which turns a mistake into a plausible-looking matrix rather than an error.

MxGetBlock( ..., asArray = False ) is the way back. With asArray = True it hands you a normal AFL data series instead of a smaller matrix, padding unused elements with Null.

MxFromString accepts three notations, which is convenient when you are transcribing from a paper or another tool:

Fragment — not a complete formula

a = MxFromString( "{ { 1, 2, 3 }, { 4, 5, 6 } }" );   // Wolfram / Mathematica
b = MxFromString( "[ [ 1, 2, 3 ], [ 4, 5, 6 ] ]" );   // Matlab / Maple
c = MxFromString( "[ 1, 2, 3; 4, 5, 6 ]" );           // GNU Octave

MxToString always emits the Wolfram form, and is how you look at a matrix at all — there is no matrix viewer.

MxCopy( &dstmatrix, src_matrix, ... ) is the exception to copy-on-assignment: it works in place, so the destination must be passed with &. Source and destination rectangles need not have the same shape — you can copy a column into a row — but the element counts must match exactly.

When a matrix earns its place

Ask what the second dimension is. If you cannot name it, you do not have a matrix problem.

It earns its place when the answer is genuinely two-dimensional and the second dimension is not bars: symbol against symbol, parameter against parameter, an equation system’s coefficients. It also earns it when the operation you need is linear algebra — a least-squares fit, a solved system — because writing Gaussian elimination in AFL loops is both slower and more error-prone than one MxSolve call.

It does not earn its place when the second dimension is just “a handful of named things”. Five moving averages are five arrays, and five arrays work with every AFL function; the same five packed into a matrix work with fifteen. It does not earn its place for per-symbol results you only iterate over — that is the dynamic-variables or plain-array decision from the previous lesson. And it is the wrong tool whenever the natural expression is bar-wise, because none of AFL’s several hundred array functions accept a matrix.

Worked example: a correlation matrix

Goal

Measure how similarly a small set of instruments moved over a chosen window — every pair, in one pass. The second dimension here is unmistakable: symbol against symbol.

The complete formula

Complete runnable AFL

correlation-matrix.afl

// correlation-matrix.afl
// Part 36 - Matrices and Advanced Data Handling
//
// Builds the correlation matrix of daily returns for a list of symbols using
// AFL's matrix type and the @ matrix-product operator.
//
// WHY A MATRIX
//   Correlation between every pair in a list is a two-dimensional answer.
//   With ordinary arrays you would need one named variable per pair, or a
//   nest of loops. Standardise each symbol's returns, stack them as the
//   columns of one Lookback x K matrix R, and the entire answer is a single
//   product: MxTranspose( R ) @ R.
//
// ASSUMPTIONS
//   - Daily end-of-day bars. No real-time feed, no Professional edition.
//   - Bars are matched by position in the window, so all symbols must share
//     the database's trading calendar. Mixing exchanges with different
//     holidays would silently pair up different dates.
//   - Correlation is measured over the last Lookback bars only. It describes
//     that window. It is not a stable property of the pair and says nothing
//     about what the next window will look like.
//   - A symbol with any Null return inside the window is dropped and named
//     in the report rather than being quietly treated as a zero return.
//   - Matrix support needs AmiBroker 6.00; MxSetBlock and MxTranspose need
//     6.10. Verified against AmiBroker 7.00.1.

_SECTION_BEGIN("Correlation Matrix");

SymbolList = "AAPL,MSFT,KO,XOM,JNJ";
Lookback   = Param( "Lookback bars", 120, 20, 500, 10 );

LastBar  = BarCount - 1;
FirstBar = BarCount - Lookback;   // first bar of the measurement window

Usable   = "";                    // comma-separated list of symbols that qualify
Rejected = "";
Cols     = 0;

if( BarCount > Lookback + 1 )
{
    // -----------------------------------------------------------------------
    // Pass 1 - standardise each symbol's returns over the window and keep the
    //          result under a name built from the ticker.
    // -----------------------------------------------------------------------
    for( i = 0; ( Sym = StrExtract( SymbolList, i ) ) != ""; i++ )
    {
        SetForeign( Sym );

        Ret   = ROC( Close, 1 );
        Mean  = MA( Ret, Lookback );
        Sigma = StDev( Ret, Lookback, False );   // sample SD, divides by n-1

        // Count Nulls inside the window with array processing rather than a
        // loop: Sum() of a Boolean array is a rolling count.
        NullsInWindow = Sum( IsNull( Ret ), Lookback );

        // Pre-dividing by sqrt(n-1) means the matrix product below IS the
        // correlation matrix, with no scalar division of a matrix afterwards.
        Standardised = ( Ret - Mean ) / ( Sigma * sqrt( Lookback - 1 ) );

        BadData = NullsInWindow[ LastBar ] > 0 OR Sigma[ LastBar ] <= 0;

        RestorePriceArrays();

        if( BadData )
        {
            Rejected = Rejected + Sym + " ";
        }
        else
        {
            VarSet( "z_" + Sym, Standardised );
            Usable = Usable + Sym + ",";
            Cols   = Cols + 1;
        }
    }
}

if( Cols >= 2 )
{
    // -----------------------------------------------------------------------
    // Pass 2 - load the standardised windows into the columns of a matrix.
    // -----------------------------------------------------------------------
    R = Matrix( Lookback, Cols, 0 );

    for( j = 0; ( Sym = StrExtract( Usable, j ) ) != ""; j++ )
    {
        Standardised = VarGet( "z_" + Sym );

        // MxSetBlock takes its values starting at element 0 of the array, so
        // the window has to be moved to the front of a scratch array first.
        // Copying Close is simply a way to allocate an array of the right
        // length; every element used is overwritten below.
        Block = Close;

        for( k = 0; k < Lookback; k++ )
            Block[ k ] = Standardised[ FirstBar + k ];

        R = MxSetBlock( R, 0, Lookback - 1, j, j, Block );
    }

    // The only true matrix product in AFL. Every other operator is
    // element-wise. R is Lookback x Cols, so the transpose is Cols x Lookback
    // and the product is Cols x Cols.
    Corr = MxTranspose( R ) @ R;

    Report = "Correlation of daily returns, last "
             + NumToStr( Lookback, 1.0 ) + " bars\n"
             + "Order: " + Usable + "\n"
             + "Matrix: " + NumToStr( MxGetSize( Corr, 0 ), 1.0 ) + " x "
             + NumToStr( MxGetSize( Corr, 1 ), 1.0 ) + "\n";

    if( Rejected != "" )
        Report = Report + "Dropped: " + Rejected + "\n";

    printf( "%s\n", Report );
    printf( "%s\n", MxToString( Corr ) );
}
else
{
    Report = "Not enough usable symbols. BarCount = "
             + NumToStr( BarCount, 1.0 )
             + ", window = " + NumToStr( Lookback, 1.0 ) + " bars.\n";

    if( Rejected != "" )
        Report = Report + "Dropped: " + Rejected + "\n";

    printf( "%s\n", Report );
}

Plot( Close, "Close", colorDefault, styleCandle );

Title = Report;

_SECTION_END();

Download correlation-matrix.afl134 lines

How it works

The mathematics is arranged so that the matrix does the work. Correlation between two standardised return series is the sum of their products divided by n−1. If each column of a matrix R holds one symbol’s returns, already centred on its mean and already divided by its sample standard deviation and by the square root of n−1, then MxTranspose( R ) @ R is the correlation matrix directly — cell (i,j) is exactly the sum of products of column i and column j.

Pass one does the per-symbol arithmetic in ordinary AFL: MA for the window mean, StDev with Population = False for the sample standard deviation, and Sum( IsNull( Ret ), Lookback ) as a rolling count of missing bars. Symbols that fail the checks are named in the report rather than dropped silently. The surviving standardised series are parked under dynamic variable names keyed by ticker — the case the previous lesson called legitimate, because the names come from the symbol list.

Pass two allocates Matrix( Lookback, Cols, 0 ) and fills it one column at a time. The small loop that copies the window into Block is there entirely because of the element-0 rule above.

Dividing by sqrt( Lookback - 1 ) before the values enter the matrix is deliberate. It keeps the scaling in ordinary array arithmetic, where the behaviour is fully documented, instead of relying on a matrix-by-scalar division.

Key functions

StDev( ARRAY, periods, Population = True ) — the third argument is the one that matters here. The default computes the population standard deviation; correlation as defined above needs the sample version, so it is passed False.

MxSetBlock and MxTranspose are described above. MxGetSize( matrix, dim ) is used only to report the shape, but it is the function you will reach for constantly once matrices get built dynamically.

Expected result

The Interpretation window shows the symbol order, the matrix shape, any dropped symbols, and then the matrix in Wolfram notation. Two properties are checkable by eye: every diagonal element should be 1 to within rounding, and the matrix should be symmetric.

Test it

1. Put the same ticker in the list twice. Its off-diagonal correlation with itself must come out as 1.
2. Set Lookback to 20 and then to 200 on the same symbols. The numbers should move, often a lot. That instability is the finding, not a defect — a correlation is a description of the window you measured, and Part 30’s warnings about reading too much into a short sample apply here with full force.
3. Add a ticker with a gap in its history. It should appear on the Dropped: line rather than contributing a column of zeros.

Common errors

✕Common mistake

Filling a column from the wrong end of the array. MxSetBlock reads from element 0, so handing it Standardised directly would load the oldest bars in the database, not the last Lookback of them. The formula still runs and still prints a symmetric matrix with ones on the diagonal — it is simply describing a window from years ago. This is the failure mode to watch for, because nothing about the output looks wrong.

Mismatched calendars are the other silent one. Correlation here pairs values by position in the window, not by date. Two symbols from exchanges with different holidays will be compared bar 1 against bar 1 with the dates drifting apart. Everything in the alignment lesson in Part 15 applies.

Forgetting RestorePriceArrays() inside the loop leaves the price arrays on the last foreign symbol, so the pane plots the wrong instrument.

Extension

Turn the matrix into something you can read at a glance. MxGetBlock( Corr, i, i, 0, Cols-1, True ) pulls row i back as a normal AFL array; from there AddColumn in an exploration gives you one row per symbol and one column per peer, with conditional formatting. That is the version you would actually use.

Performance considerations

Two things dominate.

First, matrices are not part of AFL’s array engine. Everything you do with them at element level is a loop, and the performance guidance from the next lesson applies unchanged: loops can be ten to fifty times slower than the array expressions they replace. Build a matrix in as few MxSetBlock calls as possible rather than assigning cell by cell.

Second, size grows quadratically with the thing you are cross-tabulating. A correlation matrix over 500 symbols is 250,000 cells, and the n×k input matrix behind it is 500 columns wide by however many bars deep — before the @ product, which itself costs on the order of n²k multiplications. A matrix over a watch list of twenty is instantaneous; the same code over an entire market is a different kind of computation, and worth timing with GetPerformanceCounter before you leave it running.

What changed

You can now tell a matrix problem from an array problem: the test is whether you can name a second dimension that is not bars. You know that AFL gives you fifteen matrix functions and no more, that @ is the only true product while every other operator is element-wise, that MxSolve beats MxInverse on precision, and that the boundary between matrices and ordinary arrays runs through MxSetBlock’s element-0 rule — the place where a formula can go quietly wrong while producing perfectly plausible output.

Check your understanding

Question 1. A is 3 rows by 4 columns, B is 4 rows by 2 columns. Which expression is valid, and what shape is the result?

• A * B, giving 3 x 2
• A @ B, giving 3 x 2
• A @ B, giving 4 x 4
• A * B, giving 3 x 4

Show the answer and why

Answer: A @ B, giving 3 x 2

The element-wise operators require identical dimensions, so A * B is undefined here. @ is the matrix product: A’s 4 columns match B’s 4 rows, and the result takes A’s row count and B’s column count.

Question 2. You want the last 120 bars of an array to become one column of a matrix. What must you do first?

R = MxSetBlock( R, 0, 119, j, j, Series );

• Nothing — MxSetBlock reads from the end of the array
• Reverse the array with MxTranspose
• Move those 120 values to the front of a scratch array, because MxSetBlock starts at element 0
• Call MxGetBlock first to reserve the space

Show the answer and why

Answer: Move those 120 values to the front of a scratch array, because MxSetBlock starts at element 0

MxSetBlock takes consecutive values starting at element 0. Passed the array as-is it would load the oldest 120 bars, and the resulting matrix would look entirely reasonable while describing the wrong period.

Question 3. Which of these does AFL provide as a documented matrix function? Select all that apply.

• MxSolve
• MxDet
• MxEigenvalues
• MxSortRows

Show the answer and why

Answer: MxSolve, MxDet, MxSortRows

The matrix family has fifteen members and eigen-decomposition is not among them. Knowing the list is closed is what stops you calling a function that does not exist.

Question 4. Why does the worked example divide by sqrt(Lookback - 1) in ordinary array arithmetic rather than dividing the finished matrix?

• Because matrix division is slower
• Because it keeps the scaling in operations whose behaviour is fully documented, so the product itself is the correlation matrix
• Because MxSetBlock rejects values above 1
• Because sqrt does not accept a matrix

Show the answer and why

Answer: Because it keeps the scaling in operations whose behaviour is fully documented, so the product itself is the correlation matrix

Pre-scaling the columns means MxTranspose(R) @ R is already the correlation matrix. It also keeps every scaling step inside plain array arithmetic, which is the part of the language the reference describes exhaustively.

Check my answersClear