Statscraft 0.7.0, release v0.7.0

The command line

statscraft describe   dictionary and counts of a data file or a session
statscraft run        run syntax
statscraft check      validate syntax without saving or writing anything
statscraft help       syntax of a command, or the list of commands
statscraft plan       lower syntax to an OpenStatSpec transformation plan
statscraft db         store datasets in an OpenStatSpec SQLite database and read them back
statscraft update     check for a newer release, or update this installation to it

statscraft --version prints the version, and statscraft <subcommand> --help lists the flags of a subcommand. The examples run from the root of the repository and use the synthetic files in tests/fixtures/; job.sps is the file from Getting started.

describe

statscraft describe FILE [--json]
statscraft describe -s SESSION [--json]

Give either a file or a session. The text form prints cases, variables and then, per variable, name, format, measurement level, valid count and label, separated by tabs. For a session it goes on with what the session keeps besides the data, one line each when there is something: settings:, PRESERVE levels:, directory:, active dataset:, other datasets:, macros:. In JSON these are the object session (see Sessions).

run

statscraft run [FILE | - | -e TEXT] [-s SESSION] [--lines FIRST-LAST] [--json] [--encoding NAME] [--xlsx FILE [--xlsx-one-sheet]]

Without --json, every table is printed as a Markdown table under a ### title, warnings as warning: line N: COMMAND: message, then a summary: the dataset loaded, variables added, changed and removed, the case count when it changed, and ran N command(s), with , session saved when the session was written.

statscraft run job.sps
statscraft run -s demo -e "FREQUENCIES VARIABLES=gender."
echo "DESCRIPTIVES VARIABLES=age monthly_spend." | statscraft run -s demo -

A run is all or nothing. When any command fails, nothing is saved: not the session, not the workbook. Files that SAVE wrote before the failing command stay on disk.

check

statscraft check [FILE | - | -e TEXT] [-s SESSION] [--lines FIRST-LAST] [--json] [--encoding NAME]

The same flags as run, without --xlsx. check runs the commands on a copy of the session’s dataset and skips every file write, so it finds both syntax errors and errors that depend on the data (an unknown variable, a string where a number is needed). The session and the disk stay unchanged. The summary says checked N command(s), and in JSON saved is always false.

statscraft check -s demo -e "COMPUTE age10 = age / 10."

variables added: age10
checked 1 command(s)

Every syntax error of the source is reported at once, one per bad command. Errors found while running stop at the first failing command.

help

statscraft help                     the supported commands with one line each, then the general rules
statscraft help FREQUENCIES         syntax, notes and an example of one command
statscraft help freq                names may be lowercase and shortened, as in syntax

A command of the language that Statscraft does not run yet (for example statscraft help LIST) gets “is a command of the language, but it is not supported yet”. An unknown name is an error. The same text, one page per command, is in commands.

Sessions and line ranges

A session is what one window would keep from one run to the next, stored on disk in .statscraft/sessions/<name>/ under the current folder. A name may hold letters, digits, - and _.

• The first run of a new session starts without data; its syntax usually begins with GET FILE. Settings, macros and a CD may come before any data.
• With --lines, only the macros defined on those lines are kept; a macro defined elsewhere in the file still serves the run, as before.
• A run saves the session when it changed anything in the table above, or when the session file did not exist yet. A failed run keeps none of its changes. A dataset keeps the encoding of the file it came from; when a label or value no longer fits that code page (a letter of another language), the session keeps it in UTF-8 instead.
• describe -s NAME shows the active dataset and, under session in the JSON, the SET values given, the number of PRESERVE levels, the CD folder, the active dataset’s name, the other named datasets and the macros.
• check and describe never write it.
• Delete the session’s folder to start over.

statscraft run -s demo --lines 1-1 job.sps
statscraft run -s demo --lines 4-6 job.sps

--lines selects the commands that lie inside the range. Blank lines and comments inside it are fine, and a syntax error on a line outside the range does not stop it, but the range must cover whole commands. A range that cuts through one fails:

error: line 1, column 1: line range 2-3 cuts through command FREQUENCIES, which occupies lines 1-2; hint: use lines 1-3 to include the whole command

Line numbers in the output always refer to the whole file, not to the range.

JSON output

run, check and describe print pretty JSON with --json. plan and db always print one line of JSON.

Run result

statscraft run -s demo --json -e "COMPUTE spend_year = monthly_spend * 12. followed by a second line FREQUENCIES VARIABLES=gender." prints (the table rows are shortened here):

{
  "ok": true,
  "session": "demo",
  "commands": [
    {
      "name": "COMPUTE",
      "first_line": 1,
      "last_line": 1,
      "tables": [],
      "warnings": []
    },
    {
      "name": "FREQUENCIES",
      "first_line": 2,
      "last_line": 2,
      "tables": [
        {
          "title": "gender: Gender",
          "row_headers": 3,
          "columns": ["Category", "Value", "Label", "Frequency", "Percent", "Valid Percent", "Cumulative Percent"],
          "column_groups": [[], [], [], [], [], [], []],
          "rows": [
            ["valid", 1.0, "Male", 106.0, 53.0, 53.0, 53.0],
            ["total", null, null, 200.0, 100.0, null, null]
          ]
        }
      ],
      "warnings": []
    }
  ],
  "changes": {
    "cases_before": 200,
    "cases_after": 200,
    "variables_added": ["spend_year"],
    "variables_removed": [],
    "variables_changed": []
  },
  "dataset": [200, 21],
  "saved": true
}

A table:

A custom table with two header levels, from CTABLES /TABLE region [COUNT, COLPCT.COUNT PCT5.1] BY gender. (first row only, shortened):

{
  "title": "Custom Tables",
  "row_headers": 2,
  "columns": ["", "", "Count", "Column %", "Count", "Column %", "Count", "Column %"],
  "column_groups": [["", ""], ["", ""], ["Gender", "Male"], ["Gender", "Male"], ["Gender", "Female"], ...],
  "rows": [["Region of residence", "North", 23.0, 21.69811320754717, 11.0, 12.5, 0.0, 0.0], ...],
  "formats": [["", "", "F40.0", "PCT5.1", "F40.0", "PCT5.1", "F40.0", "PCT5.1"], ...]
}

Error

On any failure --json prints this on standard output, and the exit code is 1:

{"ok":false,"errors":[{"kind":"syntax","message":"unexpected end of command","line":1,"column":18,"expected":["a value","a variable","a function","'('"]}],"saved":false}

More examples:

{"ok":false,"errors":[{"kind":"run","message":"unknown variable agee","command":"FREQUENCIES","line":2}],"saved":false}
{"ok":false,"errors":[{"kind":"syntax","message":"unknown command FREQUENCIS","line":1,"column":1,"hint":"did you mean FREQUENCIES?"}],"saved":false}

Without --json the same errors go to standard error as error: line 1, column 18: unexpected end of command; expected: a value, a variable, a function, '('.

Warnings

Warnings do not fail a run. They are in commands[].warnings of the run result, and in text output as:

warning: line 1: DESCRIPTIVES: no statistics for string variables: plan

describe

statscraft describe tests/fixtures/survey_basic_phpwriter.sav --json (shortened):

{
  "encoding": "UTF-8",
  "compression": "bytecode",
  "case_count": 200,
  "variable_count": 19,
  "file_label": "Statscraft synthetic customer survey",
  "weight_variable": "weight",
  "multiple_response_sets": [
    {"name": "$q5", "type": "dichotomy", "label": "Q5. Contact channels used", "counted_value": 1,
     "variables": ["q5_1", "q5_2", "q5_3", "q5_4", "q5_5"]},
    {"name": "$q6", "type": "category", "label": "Q6. Other providers mentioned", "counted_value": null,
     "variables": ["q6_mention1", "q6_mention2", "q6_mention3"]}
  ],
  "variables": [
    {
      "name": "q2_recommend",
      "type": "numeric",
      "label": "Q2. How likely are you to recommend us to a friend or colleague?",
      "format": "F2.0",
      "measure": "scale",
      "value_labels": {"0": "Not at all likely", "10": "Extremely likely", "97": "Not applicable", "98": "Don't know", "99": "Refused"},
      "missing_values": [],
      "missing_range": [97.0, 99.0],
      "n_valid": 176,
      "n_system_missing": 0,
      "n_user_missing": 24
    },
    {
      "name": "plan",
      "type": "string",
      "width": 10,
      "label": "Current subscription plan",
      "format": "A10",
      "measure": "nominal",
      "value_labels": {},
      "missing_values": [],
      "missing_range": null,
      "n_valid": 200,
      "n_system_missing": 0,
      "n_user_missing": 0
    }
  ]
}

Exit codes

update

statscraft update --check          current and latest version; exit code 10 when an update exists
statscraft update --check --json   the same as JSON
statscraft update                  update this installation to the latest release

The latest release is the newest published release on the GitHub repository; drafts and prereleases do not count. update --check prints the running and the latest version and exits with 0 when this copy is up to date (also when no release is published yet) and with 10 when a newer one exists, so a script can ask statscraft update --check || .... With --json it prints one line:

{"current":"0.1.0","latest":"0.2.0","update":true,"notes":"https://github.com/statscrafthq/statscraft/releases/tag/v0.2.0"}

latest is null when no release is published yet, and notes is the release page when there is an update.

update does what the window’s Update now does, for the way this copy was installed:

Every download is checked against the SHA256SUMS file of the release before anything is run or replaced; on a mismatch nothing changes and the file is deleted. The installer, the tarball and the .zip go to a temporary folder that is removed after use. Errors print as for the other subcommands, exit code 1.

The only requests go to GitHub over HTTPS: one to the API for the latest release and, on update, the download of SHA256SUMS and of the one package. They carry the User-Agent statscraft/VERSION and nothing else about you; there is no account, token or telemetry.

Excel export

statscraft run job.sps --xlsx tables.xlsx
statscraft run job.sps --xlsx all.xlsx --xlsx-one-sheet
statscraft run -s gui -e "FREQUENCIES VARIABLES=gender." --xlsx gender.xlsx

Every table of the run goes into the workbook, with or without a session, and also from the window session. With one sheet per table the sheet is named after the table title (gender Gender, region Region of residence, …); with --xlsx-one-sheet there is one sheet Tables with the tables one below the other.

Each table has its title in bold, the column headers with merged cells for every level, the row labels, thin borders and its caption below. Numbers stay numbers, with a number format that shows them as the output does: a percentage cell holds 45.2 and shows 45.2%. Text such as significance letters and hidden small counts (<5) stays text. The text output on the terminal is the same as without --xlsx.

A workbook that cannot be written fails the run, and the session is then not saved.

The live window session

While statscraft-gui is open, the session name gui means the window:

statscraft run -s gui -e "GET FILE='tests/fixtures/survey_basic.sav'."
statscraft run -s gui --json job.sps
statscraft check -s gui -e "COMPUTE x = age / 10."
statscraft describe -s gui --json

• The window runs the syntax as its own Run button does: the tables appear in Output, the commands in the log, the grid shows the new data.
• The command line prints what it prints for a file session, with "session": "gui". saved is true when the run changed the window’s session (the table under Sessions); nothing is written to disk unless the syntax says SAVE.
• The window keeps the same state between runs as a file session, in memory until it closes; describe -s gui shows it the same way.
• Relative paths in the syntax are read from the folder where statscraft was started, and the window stays in that folder.
• One run goes at a time. A request waits while the window is busy.
• When no window is open, -s gui fails with “no Statscraft window is running”. With several windows open, the one started last answers.

How it connects, and why it is safe: see AI agents.

plan

statscraft plan [FILE | - | -e TEXT] (--input-schema FILE | -s SESSION | --sqlite DB --name DATASET [--apply [--actor NAME]])
                [--profile 0.1|0.2|0.3] [--alias NAME]

plan turns a bounded part of the language into an OpenStatSpec Transformation Plan: one line of canonical JSON with the plan, its SHA-256 plan_hash and the source_hash of the syntax, or an error with a stable code, line and column.

Profile 0.1 accepts RECODE, VARIABLE LABELS and VALUE LABELS; 0.2 adds COMPUTE and IF with single values and simple comparisons, FORMATS, VARIABLE LEVEL and EXECUTE; 0.3 adds comments, TO lists, NE and NOT, LOWEST and HIGHEST ranges and ADD VALUE LABELS. Everything else is rejected, even where run accepts it. With schema.json holding {"variables": [{"name": "q1", "storage_kind": "numeric"}, {"name": "q2", "storage_kind": "numeric"}]}:

statscraft plan -e "RECODE q1 (1 = 0)." --input-schema schema.json --profile 0.1
statscraft plan -s demo -e "RECODE q1_satisfaction (98, 99 = SYSMIS)."
statscraft plan -e "FREQUENCIES q1." --input-schema schema.json

The last one prints {"error":{...,"message":"FREQUENCIES is not supported by frontend profile 0.3",...},"ok":false,...}, whose code is the profile’s code for an unsupported command, and exits with 1.

db

statscraft db export (FILE | -s SESSION) --sqlite DB [--name NAME]
statscraft db import --sqlite DB --name NAME (--out FILE | -s SESSION) [--allow-loss]

db stores a dataset in an SQLite database laid out as OpenStatSpec specifies: one table data_<name> with __case_ordinal and one column per variable, and the dictionary in catalog tables. The database file belongs to OpenStatSpec: export sets up the catalog in a new or empty file and refuses a file that holds anything else.

statscraft db export tests/fixtures/survey_basic.sav --sqlite study.db
statscraft db export -s demo --sqlite study.db --name work
statscraft plan -e "RECODE q1_satisfaction (98, 99 = SYSMIS)." --sqlite study.db --name survey_basic --apply --actor demo
statscraft db import --sqlite study.db --name survey_basic --out back.sav
statscraft db import --sqlite study.db --name work -s fromdb

Each prints one line of JSON, such as {"case_count":200,"dataset":"survey_basic","dataset_id":"...","ok":true,"table":"data_survey_basic","variable_count":19}, or {"ok":false,"error":{"code":...,"message":...}} with exit code 1.

Before writing, export checks the column limit (2000 including __case_ordinal), identifiers and value sizes, and refuses NaN and negative zero, which SQLite cannot keep. A failure leaves no table and no dataset row behind.

Databases other than SQLite

statscraft db export survey.sav --sqlite study.db [--name survey]   store a dataset in SQLite
statscraft db export survey.sav --postgres URL                      or in PostgreSQL
statscraft db export survey.sav --mysql URL                         or in MySQL or MariaDB
statscraft db export survey.sav --dolt URL                          or in Dolt
statscraft db capabilities                                          versions and limits per database
statscraft db export -s work --sqlite study.db --name work          store a session's dataset
statscraft db import --sqlite study.db --name survey --out back.sav read it back into a file
statscraft db import --sqlite study.db --name survey -s work        or into a session
statscraft plan job.sps --sqlite study.db --name survey --apply     apply syntax to it in place

db stores a dataset the way OpenStatSpec’s SAV/ZSAV profile 1.0 and its SQL dialect profiles specify: one table data_<name> with __case_ordinal (1, 2, 3, … in case order) and one column per variable in variable order, numbers as binary64 with system-missing as NULL, strings as text that is never NULL, user-missing codes kept as values, dates as numbers. The dictionary (labels, value labels, missing values, formats, measurement level, role, display settings, attributes, documents, weight variable, multiple response sets) goes into the catalog tables. The database belongs to OpenStatSpec: export sets up the catalog in a new or empty one and refuses one that holds anything else. Before writing, a preflight checks the column limit, identifiers, value sizes and the row size against the profile and the server’s settings. A failure leaves no table and no dataset row, only the failed operation and its diagnostic. The rows go in bulk inside the same transaction: PostgreSQL takes them with COPY in the binary format, MySQL and MariaDB in prepared multi-row INSERTs of at most 4 MiB, Dolt in multi-row INSERTs of literal values of at most 256 KiB, each within max_allowed_packet, SQLite through one prepared INSERT per case. Reading streams the rows from the server into the dataset instead of holding the whole result first. Variable names that are not plain ASCII get generated column names (v<position>); the catalog maps them back. Reading a dataset whose catalog holds items Statscraft cannot keep (dataset attributes, variable sets, write formats other than the print format) fails unless --allow-loss is given; the losses are then listed in the output.

MySQL, MariaDB and Dolt commit DDL implicitly, so export creates the table first, writes the rows and the catalog in one transaction, and drops the table again if that fails. A plan that would create a variable is refused before anything changes; create the column and its catalog variable in a separate step first.

The product is decided by what the server says about itself, not by the protocol: --mysql refuses a Dolt server and --dolt refuses MySQL and MariaDB. On Dolt, Statscraft never commits, branches, merges or resets. db export leaves its changes in the working set for you to commit; db import and the input schema of plan write nothing at all (OpenStatSpec’s Database I/O Execution Policy v1, which db capabilities declares), so allowed losses are reported in the output instead of in the catalog. plan --apply on Dolt needs --expected-branch and --expected-head: the session must be on that branch at that commit with a clean working set, and the branch must still point at that commit when the apply completes; the audit row records both, and the changes stay uncommitted. The session’s branch is the database’s default branch, or the one named as db/branch in the URL.

A URL without a password takes it from the STATSCRAFT_DB_PASSWORD environment variable. Passwords are never printed.

Connections use plain TCP unless the URL asks for TLS. TLS always verifies the server’s certificate chain, against the CA file when one is given and otherwise against the bundled Mozilla root certificates, and also the host name unless the mode says otherwise. Statscraft never makes an unverified TLS connection, and refuses the modes that fall back to plain TCP (prefer, allow, PREFERRED).

statscraft db export survey.sav --postgres "postgres://analyst@db.example.org/study?sslmode=verify-full"
statscraft db export survey.sav --postgres "postgres://analyst@10.0.0.5/study?sslmode=verify-ca&sslrootcert=/etc/ssl/study-ca.pem"
statscraft db export survey.sav --mysql "mysql://analyst@db.example.org/study?ssl-mode=VERIFY_IDENTITY&ssl-ca=/etc/ssl/study-ca.pem"

sslrootcert=FILE (PostgreSQL) or ssl-ca=FILE (MySQL, MariaDB, Dolt) names a PEM file of the CA certificates to trust, for a server whose certificate comes from a private CA; a CA file without a mode means verify-full. With a CA file, PostgreSQL trusts only that file; the MySQL client trusts the bundled roots as well. Note that require and REQUIRED check more than in the PostgreSQL and MySQL command line clients, where they encrypt without checking the certificate.

plan --name DATASET with a database option takes the input schema from that dataset; with --apply it then applies the plan to the same table in one transaction: direct UPDATEs, ALTER TABLE ... ADD COLUMN for new variables, catalog updates and one audit row in transformation_apply (plan and source hashes, actor, operation count). Nothing is copied, and on any error nothing changes. --actor names who applies the plan (default $USER).