AMI Script

AMI Script Introduction

Overview

AMI has an embedded, versatile language which is a combination of the well-known C/Java style, SQL and String Templating languages.  Here is a very quick example of the incredible versatility of the language which does a cross-database join:

 1{
 2  //Inspect the ACCOUNTS table of two pre-defined databases (qadb & proddb).
 3  //Two temporary in-memory tables are created (qatemp and prtemp)
 4  String table = "ACCOUNTS";
 5  CREATE TABLE qatemp AS USE datasource=qadb   EXECUTE SELECT * FROM ${table};
 6  CREATE TABLE prtemp AS USE datasource=proddb EXECUTE SELECT * FROM ${table};
 7
 8  //Gather some statistics, Note that the results of the queries on the in-memory tables are
 9  //stuffed into the local variables
10  int qaCnt = select count(*) from qatemp;
11  int prCnt = select count(*) from prtemp;
12  int bothCnt = select count(*) from qatemp, prtemp where qatemp.id == prtemp.id;
13
14  //Alert the user with some statistics, including the number of users in both qa and prod
15  session.alert("Found ${qaCnt + prCnt} ${table} rows: ${qaCnt} in qa and ${prCnt} in prod. \
16                 There are ${bothCnt} ids existing in both");
17
18  //Create a Temporary ProdOnly table that contains all users in prod but not qa.
19  //This table can be used in the visualization layer
20  Create table ProdOnly as select * from prtemp where !(id in (select id from qatemp));
21}

In this introductory language you can see the interoperability of SQL and a procedural style language. The ${...} syntax is used for easily creating dynamic strings and SQL statements.

Key Features

AMI Script is a versatile language designed for optimized "on the fly" compilation, meaning expressions can be re-compiled and evaluated in an instant.  Some key features of the language:

• Type Safe - All variables have well defined types, ex: String vs. Integer
• Compile Time Binding - Code is compiled before execution to minimize runtime issues
• Procedural -  Custom methods (with overloading), along with a set of predefined procedures
• Object Oriented - Objects in AMI are represented as such in Ami Script
• Event Driven - Executes based on events, such as running a datamodel or clicking a button
• Implicit Casting - Casting from one type to another can be explicitly or implicitly defined
• Embedded SQL - For accessing external sources and tables directly in the language. See below
• String Templating - Ability to template code for reuse. See below
• Task Swapping - Tasks can be exported runtime for out-of-band execution. (separate thread)

Embedded SQL is an SQL-style language extension to AMI Script with near-full SQL support along with several enhancements geared towards data preparation for the visualization of data.  All of these functions act on an in-memory database. It also supports "calling out" to external datasources. Some key features are:

• Query Clauses - For filtering, joining, and grouping data on the in-memory database
• Data Modification - For adding, updating, and deleting data on the in-memory database
• Schema Modification - For adding, editing, and deleting tables/columns on the in-memory database
• Use … Execute - For seamlessly running code on external datasources and storing results to the in-memory database
• Advanced Data Preparation and Analysis - For complex data calculations, not typically available on a standard SQL database
• Lambda Support -The ability to run AMI script code inside SQL. For example, to run a code snippet per row returned in a table
• Nested Queries - The in(…) clause supports full nested queries, and also supports multiple column in-clauses
• Intelligent Indexing - As joins, in-clauses, etc. are evaluated, indexes will automatically be spun up depending on data size and cardinality

AMI Script Template syntax is an extension of AMI Script with the aim of simplifying dynamic text generation, useful for creating dynamic HTML, dynamic SQL, etc.

• Embed Ami Script inside Text - Embed AMI script inside text allows for easy dynamic text generation
• Embed Ami Script inside SQL - Embed AMI script inside SQL allows for easy dynamic SQL generation
• Injection Protection - Properly escaping control characters, quotes, etc.
• Conditional Templating - Ability to do conditional & loop based template construction

AMI Script

Variable Naming

Case-sensitive combination of a-z, A-Z, _, 0-9.  The first character must not be a number. Alternatively, variables can be wrapped in back ticks (`), in which case all characters are supported, allowing for accessibility with external language references.  Note: `test` is equivalent to test. Variables are declared with the following syntax (the value will default to null if an expression is not supplied):

type variable_name [ = expression][, variable_name [ = expression] ... ];

Literals (Constants)

digits	Int constant
0xdigits	Hex Int constant
0digits	Octal Int constant
digitsL	Long constant
0xdigitsL	Hex long constant
0digitsL	Octal long constant
digits.digits	Float constant.  Note, scientific (aka 'E') notation is supported
digits.digitsD	Double constant.  Note, scientific (aka 'E') notation is supported
"chars"	String constant.  Use backslash (\) to escape quotes, backslash (\\)   and other control chars, ex: \n. A string can be defined over multiple lines using a trailing backslash on all but the last line, note that preceding white space is trimmed out for subsequent lines. Please see String Templating section for  details on ${…} syntax
\n\r\t<space>	White space (token delimiter)

Reserved Type

All reserved types' values are immutable. Null is a valid value

Long	64 bit signed whole number
Double	64 bit signed float
String	Variable length string of UTF 8 characters
Integer	32 bit signed whole number
Float	32 bit signed float
Byte	8 bit signed whole number
Boolean	boolean (true or false)
Number	Base class for Long,Double,Integer,Float,Byte
UTC	Timestamp with millisecond precision (since unix epoch)

Examples

Integer i = 32;
Double d = 4.3, f = 132.d; //declaring multiple variables
Integer j = 0xaabcc;
String test = "hello world";
String Test = "Hello \
               world";     //Multi line 
Integer lng = 100000000000L;
Boolean flag = true;
Byte some_thing = null;

Casting

Variables are strongly typed, but casting can be explicit or implicit.  If a cast fails, the result evaluates to null. Also, equality operations (==, <=, >=, >, <, !=) will also auto cast when possible as well.

Examples

String test1 = 123;    //implicit cast from integer to string 
String test2 = "0x123";//implicit cast from hexidecimal integer to string 
Double val = "123.32"; //implicit case for string to double 
Boolean b = "true";    //implicit cast from string to boolean
String val = (Integer) 123.d; //explicit cast from double to integer, then implicit cast to String
Boolean c = "test";    //Evaluates to null because the implicit cast from string to boolean is invalid
Byte d;                //Evaluates to null by default

Variable Scoping

Scopes are defined using statement blocks, which are denoted by curly brackets: {…}. Within a statement block, variables are only visible after they are declared. When statement blocks are nested inside other statement blocks, all variables declared in the outer block are visible within the inner block as long as the variable is declared before the inner block. A variable cannot be re-declared when another variable with the same name is visible.

Examples

{
  Byte a;
  //a is visible;
  Byte b;
  //a & b are visible; 
  {
    Byte c;
    //a, b & c are visible;
    Byte d;
    //a, b, c & d are visible;
    {
      Byte e;
      //a, b, c, d & e are visible;
    }
    //a, b, c & d are visible;   
  }
  //a & b are visible;
  Byte a; //Compiler error: a already exists and can not be redefined
}

Standard Operators (in order of significance)

math_expressions

(evaluates to num_var type)

num_expr	*	num_expr	multiplication
num_expr	/	num_expr	division
num_expr	%	num_expr	modulus (remainder)
num_expr	+	num_expr	addition for numbers
num_expr	-	num_expr	subtraction

binary_expressions

(evaluates to num_int_expr type)

int_expr	~	int_expr	bitwise NOT
int_expr	<<	int_expr	signed bit shift left
int_expr	>>	int_expr	signed bit shift right
int_expr	>>>	int_expr	unsigned bit shift right
int_expr	&	int_expr	bitwise AND
int_expr	^	int_expr	bitwise XOR
int_expr	!	int_expr	bitwise OR

boolean_expressions

(evaluates to boolean)

	!	bool_expr	boolean (logical) NOT
expr	<	expr	less than (strings are case sensitive)
expr	>	expr	greater than (strings are case sensitive)
expr	<=	expr	less than or equal to (strings are case sensitive)
expr	>=	expr	greater than or equal to (strings are case sensitive)
expr	==	expr	equal to (strings are case sensitive)
expr	!=	expr	is not  equal to (strings are case sensitive)
expr	=~	regex	matches regular expression (see Java Pattern for details)
expr	!~	regex	does not match regular expression (see Java Pattern for details)
expr	~~	simple_pattern	simplified text matching. See Simplified Text Matching section.
expr	instance of	type	does expr evaluate to supplied type (eg: would casting succeed)
expr	&&	expr	logical and. Short circuits if left expr returns false. Short circuits if right expr is const and false
expr		expr	logical or. Short circuits if left expr returns true. Short circuits if right expr is const and true

string_expressions

(evaluates to boolean)

string_expr+ expr

String concatenation.  Null values equate to empty string ("")

special_expressions

(expression)	Force Order of operations
array_of_type[num_expr]	Array index referencing
(type)statement	Explicit cast to type. Note that a failed cast returns null. Also, casting is automatic, so it's not necessary
bool_expr ? tr_expr : fl_expr	If bool_expr returns true, then evaluates to tr_expr, otherwise evaluates to fl_expr.

Examples

Long n = (10 + 5) * 2;  //n is 30 
n += 5;                 //n is 35 
n = n/2;                //n is 17 
Boolean b = n>10;       //b is true (n is still 17)
b =! b;                 //b is false 
b = (b || true) && true;//b is true         
String test = "hello" + "world"; //test is helloworld

Assignment & Declaration

special (evaluates to type)

new type(param[, param ...])	Object Creation
type var_name	Variable declaration (Value is defaulted to null)
type var_name = expression	Variable declaration and assignment
Variable declaration and assignment	Variable declaration and assignment

assignment (evaluates to num_var type)

num_var	++		combined post-increment assignment
	++	num_var	combined pre-increment assignment
num_var	--		combined post-decrement assignment
	--	num_var	combined pre-decrement assignment
num_var	*=	num_expr	combined multiplication assignment
num_var	/=	num_expr	combined division assignment
num_var	%=	num_expr	combined modulus assignment (remainder)
num_var	+=	num_expr	combined addition assignment
num_var	-=	num_expr	combined subtraction assignment

Flow Control

Complex AMI scripts can be achieved by combining multiple statements into statement blocks. A statement block is wrapped in curly brackets {...} and each statement is separated with a semi-colon;. Then, the execution of statement blocks can be determined using a variety of flow control statements including if, if/else, for, while, and for-each-loops. Note that for and while loops can use the continue and break directives as well. Examples

{
  Integer count = 10;
  Integer value = 1;
  for(Integer i = 0; i < count; i++){
    value *= 2;
  }
  if(value < 2000) {
    session.alert(value); //alert shows 1024
  } else   
    session.alert("High value"); // this is never hit. Also note: single statements don't need to be in {...} statement blocks
  while(true){
    value -= 10;
    if(value < 1000)
      break;
  }
  //value is now 994 
}

statement block

{ statement [;statement ...] [;]}

** Evaluates to return value if a return_statment exists, otherwise it evaluates to the last statement's evaluation. If no statements, evaluates to Void

for loop

for(initialization;termination_expression;increment) statement

initialization: statement that is called first, used to declare variables with local scope

termination_expression: statement that must evaluate to boolean, false terminates

increment: statement called after each completion of the loop's statements

statement: called once per loop. Note:

• continue_statement will exit the statement and proceed to increment statement

• break_statement will exit the for loop entirely

**Evaluates to Void

for each loop

for(type_expr  varname:iterable_or_array) statement

iterable_or_array - values to iterate over in order

statement: called once per loop. Note:

• continue_statement will exit the statement and proceed to increment statement

• break_statement will exit the for loop entirely

**Evaluates to Void

while loop

while(bool_expression) statement

same as for(;bool_expression;) statement

**Evaluates to Void

if statement

if(bool_expression) statement

if bool_expression evaluates to true then execute statement

**Evaluates to Void

if else statement

if(bool_expression) statement else statement2

if bool_expression evaluates to true then execute statement otherwise executestatement2

**Evaluates to Void

return statement

return statement

** Evaluates to statement's value. Will cause call stack to be popped out to caller function.

break statement

break

** See for_loop, for_each_loop and while_loop

continue statement

continue

** See for_loop, for_each_loop and while_loop

function call

function(param[, param ...])

**Evaluates to return_type value

member method call

target_object.function(param[, param ...])

**Evaluates to return_type value

Objects

There is a predefined set of object types (classes). These are used to facilitate the interaction between AMI Script and various backend components, user inputs, dashboards, etc. Additionally, data structures are represented as objects and can be instantiated using the new operator.  Objects can only be accessed via methods (not member variables). 

The Session Object

session - each user that is logged in has a session object (which is of type Session). This is the "starting point" for access to the user's information, along with various dashboard components.

Common Data Structures

List: A zero-indexed Array List implementation

Set: A set of unique values

Map: A Linked Hash Map implementation

Table: A 2-dimensional table with the same characteristics as a SQL style table. Tables have a title.  Columns are named and strongly typed.

Tableset: A map of tables. Can be loosely thought of as a mini database.

Examnples

session.alert("this is a test");
session.log("something worth logging);
List l = new List("apple", "orange", "berry", "banana");
l.splice(1, 2, "strawberry");
String s = l;                                //s is now apple, strawberry, banana
Map m = new Map(1, "one", 2, "two", 3, "three");  //map contains three key/value pairs, ex: 2 = "two"

Commenting

Commenting is identical to the commenting syntax in Java & C++.  There are two types of comments: single-line and multi-line. 

comments (single-line)

//comments<linefeed>

comments (multi-line)

/*comments*/

Examples

/*multi-line
comment using the
slash stars*/
//single-line comment

AMI Script Constants

AMI Script can reference constants just like other variables. Note the constant's values can not be changed using AMI Script.  The standard naming convention for global variables is 2 under bars (__) followed by an all uppercase name. Constants can be defined in three ways:

• At the server level (via a config file loaded at startup, typically local.properties)

Typically the syntax is: amiscript.variable.varname=value

(note that if value is a string, is must be surrounded in double quotes, see LITERALS for details)

For example, this could reside in your local.properties:

#My custom variables

amiscript.variable.__COMPANY_NAME="Acme Technologies"

amiscript.variable.__COPYRIGHT=2017

• At the user level  (using the authentication plugin, or the access.txt file)

Typically the syntax is: amiscript.variable.varname=value

(note that if value is a string, is must be surrounded in double quotes, see LITERALS for details)

For example, this could reside in your access.txt file for user demo (with password demo123):

demo|demo123|amiscript.variable.__GROUP="infosec"|amiscript.variable.__SECURITY_LEVEL=12

• There are also pre-determined variables defined by the AMI Web system:
  • __USERNAME: (String)  the username of the logged in user
  •  __SESSIONID: (String) a unique id for the web session
  • __TIMEZONE: (String) the timezone of the user's session
  • __ADDRESS: (String) the ip address of the user's browser
  • __LOADTIME: (Long) the time in milliseconds that the user started the session
  • session: (Session) the session object representing the user's session

Hint: Through the front end in editor mode you can view all declared constants via: Menu Bar -> Dashboard -> Variable Table

SimplifiedText Matching

Overview

The simplified text matching finds text entries using a simple pattern.  When included in AMI script, it takes the form: var ~~ "my pattern"

Rule 1, the general case: the pattern supplied will match any text that contains the pattern, case insensitive.  For example, assuming word is a variable let's consider the expression: word ~~ "AB". This will return true if word is cab or ABC or tabs but will return false if word is apple.

Rule 2, special symbols: If you need to look for symbols besides letters, numbers or white space, prefix with a back slash.  For example, to look for the word P&L, the expression is word ~~ "P\&L". Note this includes the following list of symbols: *^$.[]()~|&!?:

Rule 3, Additional syntax: There are many special symbols that can be used for more advanced searches, for example the star (*) means anything, so  "f*g" will match any thing that has an f followed by g, such as  3FORGE.  See the following sections for more advanced syntax.

Syntax The following syntaxes are used to do "basic" matching against expressions with some text.

1. Simple Matching Expression

Matches any text that contains the supplied pattern regardless of case. The following characters are reserved and may be embedded in the expression:

• * - Anything: Can be expanded to include any text.
• ^ - Starts With: Symbolizes the "start" of the text. In other words, if an expression starts with ^, then the text must start with the following expression. Please note that carets (^) not at the start of the expression will be treated as literals. (When used in conjunction with the tilde (~), the hat should follow the tilde.)
• $ - Ends With: Symbolizes the "end" of the text. In other words, if an expression ends with a dollar sign($) then the text must end with the preceding expression. Please note that dollar signs not at the end of the expression will be treated as literals.
• . - Any single char.: Will match to one character of any value.
• [charlist] Any of Chars.: Match any one of the characters enclosed within the brackets.
• ~: Case Sensitive. If the first character, indicates that matching characters should be case sensitive.
• Escaped chars: When referencing the following characters literally, they must be escaped with a back slash (\): *^$.[]()~|&!?:
• Literals: These are simply letter-for-letter matches to the expression

2. 'Exact Expression'

Matches the exact expression supplied in quotes (other than an escaped quote using \' ). This would be similar to using an equals ( == )

3. /regex/<options>/

This will use the java.util.regex package to match against regular expressions (see the java.util.regex.Pattern for details on building regular expressions). The <options> allow you to control behavior of how the regular expression operates. Simply supply the options you would like to apply in a continuous string between the 2nd and last forward slash (/). The following options are supported:

• q - Canonical Equivalence: When this flag is specified, two characters will be considered to match if and only if their full canonical decompositions match. The expression "a\u030A", for example, will match the string "?" when this flag is specified. By default, matching does not take canonical equivalence into account.
• f - Full Line: When this flag is specified, the text must fully match the regular expression. If this option is not specified, any portion of the text matching the expression will be considered a match.
• i - Case Insensitive: By default, case-insensitive matching assumes that only characters in the US-ASCII charset are being matched. Unicode-aware case-insensitive matching can be enabled by specifying the u (Unicode case) flag in conjunction with this flag.
• c - Comments: White space is ignored, and embedded comments starting with # are ignored until the end of a line.
• l - Literal: White space is ignored, and embedded comments starting with # are ignored until the end of a line.
• d - Disable Dot All: If the disable dot all flag is set then the dot expression (.) will match any character except a line terminator. By default, the dot expression only matches line terminators.
• m - Multiline: In multiline mode the expressions ^ and $ match just after or just before, respectively, a line terminator or the end of the input sequence. By default, these expressions only match at the beginning and the end of the entire input sequence.
• u - unicode Case: When this flag is specified, case-insensitive matching, enabled by the i (case insensitive) flag, is done in a manner consistent with the Unicode Standard. By default, case-insensitive matching assumes that only characters in the US-ASCII charset are being matched.
• x - Unix Lines: In this mode, only the '\n' line terminator is recognized in the behavior of ., ^, and $.
• v - Not Matching: Reverses the matching result. Expressions that normally would match will not match and expressions that would not normally match will match.
• n - No Error: If the regular expression is invalid do not throw an exception, instead just always return false, unless the v flag is also set, in which case always return true.

4. <null>

Null String - providing null will result in a matcher which matches nothing.

5. <Empty String>

Providing an empty (zero length) string to the Matcher method will result in a matcher which matches only the empty string.

Conjugated Syntaxes

The following syntaxes allow for more complicated pattern matching. Nested syntaxes may also be conjugated syntaxes. For nested syntaxes which are basic syntaxes open parenthesis [ ( ] and close parenthesis [ ) ] must be escaped with a backslash [ \ ].

6. (Expression): The parenthesis can be used to force order of operations, important when mixing or [ | ]  and and [ & ] clauses.

7. Expression1|Expression2: If either of the nested expressions match this expression will return true.

8. Expression1&Expression2: If both of the nested expressions match this expression will return true.

9. !Expression: Inverts the result of the expression.

10. (Expression) ? (ExpressionTrue) : (ExpressionFalse): If the Expression matches, return result of ExpressionTrue  Otherwise return result of ExpressionFalse.

Sample Expressions

Expression	Examples Match	Examples Mismatch	Comments
FORG	3Forge	Frg	Partial text, Case insensitive match
~FORG	FORGE	forge	Partial text, Case sensitive match
^FORG$	FORG	FORGE	Exact text, Case insensitive match
~^Forge$	Forge	forge	Exact text, Case sensitive match
!FORG	3FOR	3FORGE	Escape partial text, case insensitive
!^FORG$	FORGE	FORG	Escape exact text, case insensitive
[RB]ob	Bobby	lob	Using char list
[A-Z]ob	mob	7ob	Char list range
3F*ge	3Forge llc	Forge llc	Wild chars
3f.ge	3Frge	3Forge llc	Match single char
Acme llc\.	Acme llc.	Acme llcc	Escaping special chars
	robert	ted	Or clause partial match
	rob	robert	Or clause, exact match
bob&rob	robob	bobob	And clause partial
'Exact'	Exact	exact	Exact case sensitive text match
		a	Empty string, only matches empty string
a?bb:s	Abba test	acca tent	If contains 'a', must contain 'bb' If doesn't contain 'a', must contain 's'
	af	ab	Must contain a or b. Also must contain e or f
	def	DEF	Must be exactly 'abc' or 'def'
/reg/	regular	Regular	By default, regex must be case sensitive
/reg/i/	Regular	whatever	I option makes case insensitive
	3F reg	3Forge Regular	Mixing regex and non regex

Sample Usage The following snippet of code will print out all of the strings in the textList list that contain the phrase "my expression". See the Syntax section below for the various types of expressions that could be applied the someExpression variable.

 List<String> textList = someTextList;
 String someExpression = "my expression";
 TextMatcher matcher SH.m(someExpression);
 for(String text:textList){
   if(matcher.matches(text))
   System.out.println(text);
 }