Module lua-ctables

Use numerical C-arrays as Lua tables/matrices (C-API)

Map or serialize C-arrays (void * buffers) of any type with/and multi-dimensional Lua tables/matrices.

Matrices are normal Lua-tables but with additional required property that each members in a dimension has the same length (and type). In other words: Vectors in a mathematical sense.

  • Lua-tables of any dimension
  • C-arrays of any type (preferably numerical)
  • Serialized or mapped. Mapped means a Lua table/matrix is used as an index-"map" superimposed onto a C-array making unpacking/packing (serializing) unnecessary, yet providing linear algebra operations on the object. This vastly boosts performance but user must be mindful of race-conditions. If mapped C-buffer for example is a shb which is updating during operation, designer must provide additional synchronisation to prevent basing operations on incoherent data.

Info:

  • Copyright: © Michael Ambrus 2024
  • License: MIT
  • Author: Michael Ambrus michael@ambrus.se

Functions

version () Return version strings
ctable (Sizes, Address, Size, userdata.) Serialize or map a C-array to N-dimensional Lua-table

C-array must be a valid address to any kind of buffer.
larray (Table, Address, Size) Serialize/pack a Lua-table of N-dimensional to a C-array (void * pointer)

Does the opposite of ctable: From a table or matrix (Vector), dump serialized values to C-array.
tlen (Table) Length of a table.
dim (Table) Lengths of each dimension of a matrix (multi-dimensional table)

Iterate each dimension of a table and count the number of elements.
tdim (Table) Lengths of each dimension of a matrix (multi-dimensional table)

Returning a table with lengths.
tsum (Table) Table sum of elements

Iterate table and summarise it's elements that are assumed to be numerical (i.e.
tmul (Table) Table product of elements

Iterate table and multiply each element assumed numerical (i.e.


Functions

version ()
Return version strings

Returns:

  1. String Module version
  2. String Git SHA1 for this module when built
  3. String For which Lua-version

Usage:

    ctables = require("lua-ctables")
ctables.version()                                   --> 1.0.0-1 5bef04a   5.3
ctable (Sizes, Address, Size, userdata.)

Serialize or map a C-array to N-dimensional Lua-table

C-array must be a valid address to any kind of buffer. OOB exceptions (i.e. SIGSEGV) can occur if the buffer isn't valid in it's entire length. I.e. OOB for reads

Mapping means all leafs are references to the C-buffer and no value is stored in the object, only their references are. I.e. one can think of a mapped ctableas having a "map" of how to find values in the C-buffer.

Mapped C-arrays:

There are two "modus-operandi" for mapped C-buffers:

  • Raw LUA_TLIGHTUSERDATA mode. Each leaf is a void * C-pointer
  • Managed mode where C-pointers are accessed using a helper-class lua-refarr

The difference between them are:

  • When using using object::refarr as leafs, linear algebra operations can be performed directly on the "map" using the Vector class, who's CTOR recognises lua-refarr maps.
  • refarr uses less memory, as not every position is stored as a reference, only each leafs base-pointer. The actual address is calculated using the objects index operator.
  • lua-refarr is slightly harder to set-up properly
  • Non-lua-refarr needs helper functions n2u & u2n to de-refer a reference (see lua-ctables-extra.n2u & lua-ctables-extra.u2n)

Common for both:

  • Very fast due to direct memory access.
  • Modify only what's needed. (If only one element, that's only one access...)
  • A serializer/packer (i.e. larray() is not needed. A value is modified instantly and does not need any copy-back of the values of a whole table.
  • Sensitive to race-conditions by nature. No critical-section guards or concurrency synchronisation. That has to be added by the application, which can be a little tricky if the C-buffer is a SHB (Shared-memory Buffer) - IPC side-channels for synchronisation (notify) could be considered. There exists process bridging semaphores (see POSIX 1001.3b) but those will not work for kernel/user-land synchronisation where other more traditional (and comparably crude) methods are needed.

Reference mapped C-buffers AND high-level linear algebra using class:Vector:

  • Only object::refarr based maps are supported
  • No operation altering the "map" may be used. I.e. automatic growing of dimensions or adding/removing any dimensions size ("length")
  • The above is usually not a problem until assigning, except when transposing (~V) which is implicit in Vector:vmul

Parameters:

  • Sizes table List of dimension sizes (D) in memory layout order, i.e. innermost first: {D[x], D[y], D[z], ...}
  • Address userdata of the source or mapped C-array (void *)
  • Size integer of each element in the source. Default: 4
  • userdata. boolean or refarr Boolean value or a refarr object. Create all leaf-elements as either raw LUA_TLIGHTUSERDATA references or refarr managed. Default: false

Returns:

    table Table of the requested dimensions

See also:

Usage:

  • -- ----------------------------------------------------------------------------
-- Matrix of Values (de-serializing/unpack mode) example
-- ----------------------------------------------------------------------------
ez = 8
v=Vector(1,2,3,4); p=Vector(v,10+v,20+v); m=Vector(p,p+100);    -- Matrix
blob, sz = ctables.balloc(m:dim(), ez);                         -- Alloc
ctables.larray(m, blob, ez)                                     -- Conv
ctables.braw(blob, sz, ez)                                      -- Dump
-- 0000: 0000000000000001 0000000000000002 0000000000000003 0000000000000004
-- 0018: 000000000000000b 000000000000000c 000000000000000d 000000000000000e
-- 0038: 0000000000000015 0000000000000016 0000000000000017 0000000000000018
-- 0058: 0000000000000065 0000000000000066 0000000000000067 0000000000000068
-- 0078: 000000000000006f 0000000000000070 0000000000000071 0000000000000072
-- 0098: 0000000000000079 000000000000007a 000000000000007b 000000000000007c
M = Vector(ctables.ctable(m:dim(), blob, ez))               --> M from blob
-- Compare original matrix with the generated from blob
print(m:dim()); print(M:dim())
--> {4, 3, 2}
--> {4, 3, 2}
print(m); print(M)
--> {{{1, 2, 3, 4}, {11, 12, 13, 14}, {21, 22, 23, 24}}, {{101, 102, 103, 104}, {111, 112, 113, 114}, {121, 122, 123, 124}}}
--> {{{1, 2, 3, 4}, {11, 12, 13, 14}, {21, 22, 23, 24}}, {{101, 102, 103, 104}, {111, 112, 113, 114}, {121, 122, 123, 124}}}
ctables.bfree(blob)                                             -- Free
  • -- ----------------------------------------------------------------------------
-- Raw references R/W example. Note that Vector does not care what it's leafs
-- are. I.e. it's happy happy with anything, including LUA_TLIGHTUSERDATA
-- ----------------------------------------------------------------------------
ez = 4
v=Vector(0,0,0,0); p=Vector(v, v, v); m=Vector(p, p);   -- Matrix
blob, sz = ctables.balloc(m:dim(), ez, {0x01, 0x02, 0x03, 0x04});
ctables.braw(blob, sz, ez)                              -- Dump
-- 0000: 00000001 00000002 00000003 00000004 00000001 00000002 00000003 00000004
-- 001c: 00000001 00000002 00000003 00000004 00000001 00000002 00000003 00000004
-- 003c: 00000001 00000002 00000003 00000004 00000001 00000002 00000003 00000004

M = Vector(ctables.ctable(m:dim(), blob, ez))           -- Values from blob
print(M)
-- {{{1, 2, 3, 4}, {1, 2, 3, 4}, {1, 2, 3, 4}}, {{1, 2, 3, 4}, {1, 2, 3, 4}, {1, 2, 3, 4}}}

-- Start mapping

r = ctables.ctable(m:dim(), blob, ez, true)             -- Refs from blob
R = Vector(r)                                           -- Wrap it in Vector
-- Confirm dimensions
R:dim()                                                 --> {4, 3, 2}
m:dim()                                                 --> {4, 3, 2}
r:dim()                                                 --> {4, 3, 2} (ref-mod)

-- Read/set a few indexes using helpers

-- Sanity read:
print( ctables.u2n( R[1][1][1]) )                       --> 1.0
print( ctables.u2n( R[2][3][3]) )                       --> 3.0
-- Write then read-back:
ctables.n2u( R[2][3][3], 0x10)                  -- set value 0x10 at [2 ,3, 3]
print( ctables.u2n( R[2][3][3]) )               --> 16.0 (confirmed)

ctables.braw(blob, sz, ez)                              -- Dump
-- 0000: 00000001 00000002 00000003 00000004 00000001 00000002 00000003 00000004
-- 001c: 00000001 00000002 00000003 00000004 00000001 00000002 00000003 00000004
-- 003c: 00000001 00000002 00000003 00000004 00000001 00000002 00000010 00000004
--                                  Confirms correct memory offset --^

R = nil                                                 -- GC
r = nil                                                 -- GC
ctables.bfree(blob)                                     -- Free
  • -- ----------------------------------------------------------------------------
-- A lua-refarr R/W example. Note that Vector is aware it's leafs are a
-- lua-refarr's
-- ----------------------------------------------------------------------------
-- Re-assign variables used to construct the "map" (previous set values are GC'd)
ez = 4
v=Vector(0, 0, 0, 0); p=Vector(v, v, v); m=Vector(p, p); -- Matrix {2x3x4}
blob, sz = ctables.balloc(m:dim(), ez, {0x01, 0x02, 0x03, 0x04});
ctables.braw(blob, sz, ez)                      -- Dump to see the pre-set pattern
-- 0000: 00000001 00000002 00000003 00000004 00000001 00000002 00000003 00000004
-- 001c: 00000001 00000002 00000003 00000004 00000001 00000002 00000003 00000004
-- 003c: 00000001 00000002 00000003 00000004 00000001 00000002 00000003 00000004

-- Set-up the lua-refarr based map
tt = ctables.tmul(Vector(m:dim()))                      -- Total elements (24)
ro = refarr.new(blob, tt, ez, true, false, true)        -- A reference object
print(ro[1])                                            --> 1
print(ro[4])                                            --> 4
print(ro[23])                                           --> 3
for i=1, tt do                              -- Ditto on a loop
    print("info u["..i.."]: " .. ro[i])     --> {1..4} x 3 x 2 (24)
end

-- Apply reference "map" over it. I.e. Makes it dimensions aware beyond a
-- single dimension.

rm = ctables.ctable(m:dim(), blob, ez, ro)  -- A Lua {2x3x4} matrix of references
M = Vector(rm)                              -- Ditto Vector {2x3x4}

for z=1, #M do
    for y=1, #M[z] do
        for x=1, #M[z][y] do
            print("Value M["..z..","..y..","..x.."]: " .. M[z][y][x])
        end
    end
end
-- Value M[1,1,1]: 1
-- Value M[1,1,2]: 2
--       :                -- <skipped>
-- Value M[2,1,2]: 2
--       :                -- <skipped>
-- Value M[2,3,3]: 3
-- Value M[2,3,4]: 4

-- Read-set-read test at index [1][2][3]
print(M[1][2][3])                           --> 3
M[1][2][3]=27
print(M[1][2][3])                           --> 27
ctables.braw(blob, sz, ez)                  -- Dump mapped C-buffer to confirm

--                                              M[1][2][3] is here --.
-- 0000: 00000001 00000002 00000003 00000004 00000001 00000002 0000001b 00000004
-- 001c: 00000001 00000002 00000003 00000004 00000001 00000002 00000003 00000004
-- 003c: 00000001 00000002 00000003 00000004 00000001 00000002 00000003 00000004

print(M)
-- {{{1, 2, 3, 4}, {1, 2, 27, 4}, {1, 2, 3, 4}}, {{1, 2, 3, 4}, {1, 2, 3, 4}, {1, 2, 3, 4}}}
--   M[1][2][3] is here --^

rm = nil                                    -- GC reference keeper
ro = nil                                    -- GC reference keeper
ctables.bfree(blob)                         -- Free innermost last (principle)
  • -- ----------------------------------------------------------------------------
-- Linear algebra using Vector directly on a mapped C-buffer
-- Note: The helper Vectors v, p, m only used to get a dimension via
-- m:dim() replaced with a single variable dim
-- ----------------------------------------------------------------------------
ez = 4
dim = {4, 2}
blob, sz = ctables.balloc(dim, ez, {0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, 0x08});
tt = ctables.tmul(dim)
ro = refarr.new(blob, tt, ez, true, false, true)
rm = ctables.ctable(dim, blob, ez, ro)
M = Vector(rm)                      --> Our mapped {4x2} Matrix

print(M)                            --> {{ 1, 2, 3, 4 }, { 5, 6, 7, 8 }}
print(M - 1)                        --> {{0, 1, 2, 3}, {4, 5, 6, 7}}
print(10 + M)                       --> {{11, 12, 13, 14}, {15, 16, 17, 18}}

-- Note: M+1 above is assigned a temporary value-based variable.
-- M itself is untouched:
print(M)                            --> {{1, 2, 3, 4}, {5, 6, 7, 8}}

-- Lua grammar supports only a simple assignment operator. The syntax to
-- also actuate results:
M = M + 1
print(M)                            --> {{2, 3, 4, 5}, {6, 7, 8, 9}}

-- Non scalar operation examples - be mindful about accidentally altering
-- the _"map"_ of the C-buffer mapped matrix
t = Vector(1, 9, 6, 5)
P = Vector( t+11, t+25 )            -- Same dimension for P as M is safe

print(P)                            --> {{12, 20, 17, 16}, {26, 34, 31, 30}}
print(P - M)                        --> {{10, 17, 13, 11}, {20, 27, 23, 21}}
print(M + P)                        --> {{14, 23, 21, 21}, {32, 41, 39, 39}}

rm = nil
ro = nil
ctables.bfree(blob)
larray (Table, Address, Size)
Serialize/pack a Lua-table of N-dimensional to a C-array (void * pointer)

Does the opposite of ctable: From a table or matrix (Vector), dump serialized values to C-array. Serialisation is only needed if a C-buffer is not reference mapped (see ctable regarding "mapped C-buffers").

Note: C-array must be a valid address but can be any kind of native buffer with enough reserved space to hold the dump. I.e. shared-/file-mapped memory. In other words, larray/ctable are usable for IPC (a common/normal use-case)

Parameters:

  • Table table Source table
  • Address userdata of the destination C-array (void *)
  • Size integer of each element to destination. Default: 4

Returns:

    integer Length, ... Lengths of dimension(s) (multiple returns)

See also:

Usage:

    -- Work with a 3D Matrix of dimensions: 2x3x4
--                           dim    ele       pattern
--                            |      |           |
--                            v      v           v
blob, sz = ctables.balloc({4, 3, 2}, 8, {0x0, 0x1, 0x2, 0x3})   -- Allocate
v=Vector(1,2,3,4); p=Vector(v,10+v,20+v); m=Vector(p,p+100);    -- Matrix
ctables.larray(m, blob, 8)                                      -- Convert
--> 4       3       2
ctables.braw(blob, sz, 8)                                       -- Dump
-- 0000: 0000000000000001 0000000000000002 0000000000000003 0000000000000004
-- 0018: 000000000000000b 000000000000000c 000000000000000d 000000000000000e
-- 0038: 0000000000000015 0000000000000016 0000000000000017 0000000000000018
-- 0058: 0000000000000065 0000000000000066 0000000000000067 0000000000000068
-- 0078: 000000000000006f 0000000000000070 0000000000000071 0000000000000072
-- 0098: 0000000000000079 000000000000007a 000000000000007b 000000000000007c
ctables.bfree(blob)                                             -- Free

-- Play with various element sizes (variable ez below):
--
ez = 1
blob, sz = ctables.balloc({4, 3, 2}, ez, {0x0, 0x1, 0x2, 0x3}); -- Allocate
v=Vector(1,2,3,4); p=Vector(v,10+v,20+v); m=Vector(p,p+100);    -- Matrix
ctables.larray(m, blob, ez)                                     -- Convert
--> 4       3       2
ctables.braw(blob, sz, ez)                                      -- Dump
-- 0000: 01 02 03 04 0b 0c 0d 0e 15 16 17 18 65 66 67 68 6f 70 71 72 79 7a 7b 7c
ctables.bfree(blob)                                             -- Free

-- Note:
--   * The values are the same but with different layout in memory(braw)
--   * Pattern for balloc is 1D, of arbitrary length and can even be omitted
--     (used only to spot errors in layout)
--   * The Matrix step is constructed using the Vector class, but can be any
--     Lua table (just more to type). "Vector" also helps keeping the Matrix
--     equally filled per dimension, equal type per element and with printouts.
--   * Only scalars (integers 8, 16, 32 & 64 bit, and double long i.e. 80
--     bit floats on most architectures) & booleans are currently supported.
tlen (Table)
Length of a table. If a matrix, it's total dimension (1D, 2D ... N'thD)

Parameters:

Returns:

    integer Length (1D) or number of dimensions (matrix)

Usage:

    ctables.tlen({3,2,1,0})                             --> 4

-- Dimensions of a 2x4 matrix example
d=ctables.tlen(ctables:tdim({{1,2,3,4},{2,3,4,5}})) --> 2
print("This is a " ..d.."D matrix")                 --> "This is a 2D matrix"
dim (Table)
Lengths of each dimension of a matrix (multi-dimensional table)

Iterate each dimension of a table and count the number of elements. If input is a 1D-table, this function is identical to tlen.

Parameters:

  • Table table Source table

Returns:

    integer Length, ... Lengths of dimension(s) (multiple returns)

See also:

Usage:

  • ctables.dim({{5,4}, {3,2}, {1,0}})                  --> 2 3
  • -- This is a variant of tdim with identical results if result is scoped.
-- I.e.:

t = { ctables.dim({{5,4}, {3,2}, {1,0}}) }
Vector(t)                                           --> {2, 3}
  • -- Function is in certain circumstances more convenient to use (small
-- dimensions 1-3D) and if table isn't needed it's also faster. Drawback
-- however is that number of return-values are limited in Lua. Example of
-- convenience usage:

cols, rows = ctables.dim({{5,4}, {3,2}, {1,0}})
print(cols, rows)
--> 2       3
tdim (Table)
Lengths of each dimension of a matrix (multi-dimensional table)

Returning a table with lengths. Variant of dim

Parameters:

  • Table table to inspect. Any type inherited from Lua a table

Returns:

    table with each dimensions length in corresponding index

See also:

Usage:

    ctables.tdim({{5,4}, {3,2}, {1,0}})                 --> table: 0x5637a7bcd9d0
Vector(ctables.tdim({{5,4}, {3,2}, {1,0}}))         --> {2, 3}
tsum (Table)
Table sum of elements

Iterate table and summarise it's elements that are assumed to be numerical (i.e. Lua-number or Lua-integer)

Parameters:

  • Table table Source table

Returns:

    number Sum

Usage:

    ctables.tsum({1,2,3})                               --> 6.0
tmul (Table)
Table product of elements

Iterate table and multiply each element assumed numerical (i.e. Lua-number or Lua-integer)

Verbatim copy of tsum except for the operator and start-value.

Parameters:

  • Table table Source table

Returns:

    number product

See also:

Usage:

    ctables.tmul({4,34,60})                             --> 8160.0
generated by LDoc 1.5.0 Last updated 2024-09-16 20:16:45