May 30, 2016

A Lua LPeg lexer for TaskPaper documents

vis aims to be a modern, legacy free, simple yet efficient vim-like editor. It uses the Scintillua Lua LPeg Parsing Expression Grammars for syntax highlighting. (vis is also extensible in Lua.) Weighing in at a miserly 200K, vis is the “little vim” that I’ve been looking for. It is more suitable for coding than prose, but you get unlimited undo / redo (not, alas, persistent across sessions), true multiple cursor support, multiple buffers and a very interesting implementation of the regular expression language from Rob Pike’s sam editor for Plan 9, along with the syntax highlighting.

One thing that vis doesn’t come with ‘out of the box’ is syntax highlighting for TaskPaper documents, so I banged my head off that wall for a while and came up with the following:

-- TaskPaper LPeg lexer
-- Copyright (c) 2016 Larry Hynes
-- License: The MIT License
-- Important Disclaimer:
-- I have no idea what I'm doing

local l = require('lexer')
local token = l.token
local P, R, S = lpeg.P, lpeg.R, lpeg.S

local M = {_NAME = 'taskpaper'}

local delimiter = P('    ') + P('\t')

-- Whitespace
local ws = token(l.WHITESPACE,^1)

-- Tags
local day_tag = token('day_tag', (P('@today') + P('@tomorrow')))

local overdue_tag = token('overdue_tag', P('@overdue'))

local plain_tag = token('plain_tag', P('@') * l.word)

local extended_tag = token('extended_tag',
        P('@') * l.word * P('(') * (l.word + R('09') + P('-'))^1 * P(')'))

-- Projects
local project = token('project',
        l.nested_pair(l.starts_line(l.alnum), ':') * l.newline)

-- Notes
local note = token('note', delimiter^1 * l.alnum * l.nonnewline^0)

-- Tasks
local task = token('task', delimiter^1 * P('-') + l.newline)

M._rules = {
    {'note', note},
    {'task', task},
    {'project', project},
    {'extended_tag', extended_tag},
    {'day_tag', day_tag},
    {'overdue_tag', overdue_tag},
    {'plain_tag', plain_tag},
    {'whitespace', ws},

M._tokenstyles = {
    note = l.STYLE_CONSTANT,
    task = l.STYLE_FUNCTION,
    project = l.STYLE_TAG,
    extended_tag = l.STYLE_COMMENT,
    day_tag = l.STYLE_CLASS,
    overdue_tag = l.STYLE_PREPROCESSOR,
    plain_tag = l.STYLE_COMMENT,


return M

If I get to a point where I’m happy that this covers all (or most. or some!) use cases I hope to send it along upstream to Mitchell, who maintains the Scintillua project primarily, I think, for his Textadept editor. I did have to bother him at one stage to get past a mental block that I had around some syntax so I would like to repay his kindness.