Building syntax tree using antlr4/grun from separare lexer and parser grammars

Question

I'm trying to build an AST for a go file using antlr/grun. I've taken the GoLexer.g4 and GoParser.g4 from here.

When compiling, javac complains about now knowing GoParserBase. That class is not defined in the source code generated by antlr.

It appears the workflow is slightly different from that of a single grammar file like in the official doc.

Which steps are missing?

These are the commands to reproduce the error from the grammars mentioned above:

$ antlr4 -no-visitor GoLexer.g4 GoParser.g4
$ javac -cp ".:/usr/share/java/antlr-complete.jar" -g *.java
GoParser.java:12: error: cannot find symbol
public class GoParser extends GoParserBase {
                              ^
  symbol: class GoParserBase
# and lots more errors

EDIT: Thanks for the solution, @bart-kiers.

For completeness, here is the grun call:

grun Go sourceFile -gui $GOPATH/src/encoding/json/encode.go

Grammars-v4 is supposed to work for any target. The golang/*.g4 grammar calls code that is target dependent, but it's done in such a way as to be "target agnostic," meaning that the grammar can be used with other targets. After all, people write parsers in other languages than just Java. Using @header in the .g4 file makes the grammar target specific. Using `options { superClass = ...}` is the "agnostic approach" but it requires the grammar to be split into lexer/parser grammars and a base class to be introduced. These are the breaks of Antlr4. — kaby76, Jul 08 '21 at 16:11

Bart Kiers · Accepted Answer · 2021-07-08T13:09:59.180

You should also include GoParserBase.java.

The fact that the parser extends a base class is because inside the parser rules there are a couple of predicates (look for {noTerminatorAfterParams(int)}?, and other {...}? occurences). This is target specific code and the author(s) decided not to include this code inside the grammar (besides the invocation of the method, of course), but separate it from the grammar.

It also makes it easier to integrate within the grammars-v4 repository, but this will only make sense to you if you know how the grammars-v4 repository is setup and how unit tests run.

If you don't want to extend the base parser, you could also do this:

parser grammar GoParser;

options {
    tokenVocab = GoLexer;
    // superClass = GoParserBase; <-- can be removed
}

@header {
  // Add the methods present inside GoParserBase.java
}

// the rest of the grammar

More about ANTLR predicates:

Thanks @Bart Kiers! Can you elaborate what the role of the file is and why it doesn't seem to be necessary for the 'single grammar' case? — Banyoghurt, Jul 08 '21 at 12:52

Building syntax tree using antlr4/grun from separare lexer and parser grammars

1 Answers1