How can I use ANTLR generated lexer/parser source code?

Hi. I was convinced to use ANTLR to generate code for lexer and parser for my JSON Java library, however I can google anything and still can't seem to find info on how to use the ANTLR generated source codes. I found tutorials on how to set up ANTLR with JetBrains IDE (which I am using), and how to generate the source codes I need (and how to add them to the sources in project). What I didn't find, is how the actual generated Java source codes should be used. The generated source code files do not contain any JavaDoc, only informs about deprecated methods, nothing else. This isn't much helping. I can find tutorials on how to create own lexer/parser rules (own grammar), but nothing about the actual usage of the generated source code. Not even the ANTLR Mega Tutorial (https://tomassetti.me/antlr-mega-tutorial/) which displays as first in Google search and which is generally pointed to when someone asks about ANTLR. So I have JSON.g4 already downloaded, already used, and I already created the source code files for the lexer and parser, including visitor. But I'm stuck, I don't know how to use it :( Does anybody here know how to use ANTLR? How to use the generated code? I can't find well explained tutorial. I need something more like an idiotic way explained, because technically I'm still new to programming as whole (not just Java), still learning. I know the language, the types, but I'm still learning on how to work with Java. I can make simple program like "Hello World" or some kind of calculator. But I thought that this isn't actually something to pull of and decided to go with something bigger, like a library. These tutorials treat people like someone who know the language well and know how to work with it as well as they know it. That isn't my case. Like I said, I'd like to find more like an idiotic way of explanation. I found a book, but the thing is that I can't actually buy any book, not even virtual. I need money for my study, which isn't actually programming, it's Electrical Engineering. I am learning programming for a hobby, not a job...

As for the project, all you need can be found on GitHub: https://github.com/Polda18/JavaJSON

I found this ANTLR reference for free on the net,

http://lms.ui.ac.ir/public/group/90/...15738_ce57.pdf

I mean, I didn't download anything and I don't thank anybody ;) And I'm not gonna use it ;)

And I still can't find what I am looking for :( I told ya, a more idiotic way. That means an actual guide how to use the generated code, how to edit the code correctly so it does what it's intended to do. Did you check out the GitHub link I posted? Then you might see what the generated code looks like. The lexer and parser Java source code files contain no usable JavaDoc, the interface classes JSONVisitor and JSONListener do contain quite more helpful JavaDoc, but no information on what should actually go inside. And the implementations JSONBaseListener and JSONBaseVisitor contain nothing. Listener is empty and Visitor contains default visitChildren() method, which I'm not quite certain what it does and if this is actually what it should do... Well, frankly to say, I cannot tell at all what that default method actually does. The listener contains no code in implemented methods at all, so this is truly wrong. But I don't know how to fix it. The Definitive ANTLR4 takes it more broadly, it's not actually a guide for the use of ANTLR, only general explanation, so I am quite glad I didn't have to give my money for that. There was a section about parsing JSON, but it didn't contain the explanation of the generated methods (like the general broad explanation didn't quite explain generated classes), it moved straight to the testing of generated and fully implemented parser, which I cannot see :( Can't I really find anything that would tell me "put this there, remove that, change this"? This is an idiotic way of explanation that I am looking for. This is what we in Czech call "shovel method" of explanation...

Quote:

---

Originally Posted by Polda18

And I still can't find what I am looking for :( I told ya, a more idiotic way.

---

There's a small fee of admission. You will need to check out the Visitor design pattern (also called Walker). It's because ANTLR4 generates a parse tree that's supposed to be traversed using a visitor which job it is to lay down the actual code.

To learn how to put the ANTLR4 generated parts together I suggest you locate a small ANTLR4 example program with a main() method. Here's one such example that may fit the bill,

https://gist.github.com/mattmcd/5425206

First get your small working example of choice up and running. Then familiarize yourself with how it works, especially how the different parts fit together (by putting in traces and making changes etcetera). Then it's time to generate your own application. Start small and expand incrementally rather than doing all in one go.

So it's not "idiotic" explanations you should be looking for but the Java entry point main() in the ANTLR4 examples you find. Good luck!

The only problem is, there isn't any main program. This is a library I am building, no program. Shall I use Java console? If that works actually?

EDIT: Okay, I created a main class that generates the library. I used your link as an example and reworked it to use the JSON class methods. What does belong to the ANTLRFileStream? A file name? Or a stream of characters? Is it a JSON string or something else?

EDIT2: So, I guess that correct input is the file that contains the JSON string to walk. However, it does nothing. It only walks through and tests it. I need to make a parser that could be used to parse it in my library class JSON. As I said, I need an "idiotic" way (or how we in Czech call it the "shovel" way) of the ANTLR explanation. I am aware of the grammar syntax, I am aware what it makes and I am now thanks to your link aware of how the lexer is tested. But this isn't what I am looking for. You said that the visitor is intended to build the actual code of the parser. How?

EDIT3: Oh yes, can you, please, provide me like an oversimplifed explanation? Because I still can't make it out...

So I managed to somehow edit it, but I still can't make out how it works. I tried really hard to guess what exactly it does, but I can't quite see how it actually works. The visitor now outputs to the console the token id and text, but I don't see much logic here. The only thing I can see that the visitor first calls to enter every rule method and check what it is and call to enter the apropriate rule. Then it gets in recursion mode. Terminal visitor splits tokens as I can see. But that's all. How can I work out the tokenizer and what it does? Also, there is an issue. I need to somehow make the visitor to build up the object. But it has void type and I need it to determine the object to include. I need it to fit inside the JSON class I wrote by hand. Did you actually check out the repository before you started posting links? That's where all the issues lie. That's what I need to address. I need to use the generated classes in the JSON class, but I need it to actually work. What you instead gave me was some general walkthrough without any explanation and without any expected result example. Hello world is pretty good example to reproduce, but I guess that doesn't quite help me much in aid to build a Java class for JSON implementation.

This is the console output as related to current code state (found on linked GitHub repository):

Code:

---

"C:\Program Files\JetBrains\IntelliJ IDEA Community Edition 2019.1.2\jre64\bin\java.exe" "-javaagent:C:\Program Files\JetBrains\IntelliJ IDEA Community Edition 2019.1.2\lib\idea_rt.jar=56976:C:\Program Files\JetBrains\IntelliJ IDEA Community Edition 2019.1.2\bin" -Dfile.encoding=UTF-8 -classpath "C:\Users\Acer\Documents\GitHub\JavaJSON\out\production\main;C:\Program Files\JetBrains\IntelliJ IDEA Community Edition 2019.1.2\lib\validation-api-2.0.0.Final.jar;C:\Program Files\JetBrains\IntelliJ IDEA Community Edition 2019.1.2\lib\antlr-4.7.2-complete.jar" Main example.json
Entering every rule (token [], id 0): {"test":"string","array":[1,false],"object":{"zero":0,"exponent":1e89,"boolean":true},"null":null}
Entering 'json' (id 0): {"test":"string","array":[1,false],"object":{"zero":0,"exponent":1e89,"boolean":true},"null":null}
Entering every rule (token [10], id 4): {"test":"string","array":[1,false],"object":{"zero":0,"exponent":1e89,"boolean":true},"null":null}
Entering 'value' (id 4): {"test":"string","array":[1,false],"object":{"zero":0,"exponent":1e89,"boolean":true},"null":null}
Entering every rule (token [48 10], id 1): {"test":"string","array":[1,false],"object":{"zero":0,"exponent":1e89,"boolean":true},"null":null}
Entering 'obj' (id 1): {"test":"string","array":[1,false],"object":{"zero":0,"exponent":1e89,"boolean":true},"null":null}
Terminal node visit (id 0): {
Entering every rule (token [13 48 10], id 2): "test":"string"
Entering 'pair' (id 2): "test":"string"
Terminal node visit (id 1): "test"
Terminal node visit (id 2): :
Entering every rule (token [29 13 48 10], id 4): "string"
Entering 'value' (id 4): "string"
Terminal node visit (id 3): "string"
Exiting 'value' (id 4): "string"
Exiting every rule (token [29 13 48 10], id 4): "string"
Exiting 'pair' (id 2): "test":"string"
Exiting every rule (token [13 48 10], id 2): "test":"string"
Terminal node visit (id 4): ,
Entering every rule (token [15 48 10], id 2): "array":[1,false]
Entering 'pair' (id 2): "array":[1,false]
Terminal node visit (id 5): "array"
Terminal node visit (id 6): :
Entering every rule (token [29 15 48 10], id 4): [1,false]
Entering 'value' (id 4): [1,false]
Entering every rule (token [49 29 15 48 10], id 3): [1,false]
Entering 'array' (id 3): [1,false]
Terminal node visit (id 7): [
Entering every rule (token [32 49 29 15 48 10], id 4): 1
Entering 'value' (id 4): 1
Terminal node visit (id 8): 1
Exiting 'value' (id 4): 1
Exiting every rule (token [32 49 29 15 48 10], id 4): 1
Terminal node visit (id 9): ,
Entering every rule (token [34 49 29 15 48 10], id 4): false
Entering 'value' (id 4): false
Terminal node visit (id 10): false
Exiting 'value' (id 4): false
Exiting every rule (token [34 49 29 15 48 10], id 4): false
Terminal node visit (id 11): ]
Exiting 'array' (id 3): [1,false]
Exiting every rule (token [49 29 15 48 10], id 3): [1,false]
Exiting 'value' (id 4): [1,false]
Exiting every rule (token [29 15 48 10], id 4): [1,false]
Exiting 'pair' (id 2): "array":[1,false]
Exiting every rule (token [15 48 10], id 2): "array":[1,false]
Terminal node visit (id 12): ,
Entering every rule (token [15 48 10], id 2): "object":{"zero":0,"exponent":1e89,"boolean":true}
Entering 'pair' (id 2): "object":{"zero":0,"exponent":1e89,"boolean":true}
Terminal node visit (id 13): "object"
Terminal node visit (id 14): :
Entering every rule (token [29 15 48 10], id 4): {"zero":0,"exponent":1e89,"boolean":true}
Entering 'value' (id 4): {"zero":0,"exponent":1e89,"boolean":true}
Entering every rule (token [48 29 15 48 10], id 1): {"zero":0,"exponent":1e89,"boolean":true}
Entering 'obj' (id 1): {"zero":0,"exponent":1e89,"boolean":true}
Terminal node visit (id 15): {
Entering every rule (token [13 48 29 15 48 10], id 2): "zero":0
Entering 'pair' (id 2): "zero":0
Terminal node visit (id 16): "zero"
Terminal node visit (id 17): :
Entering every rule (token [29 13 48 29 15 48 10], id 4): 0
Entering 'value' (id 4): 0
Terminal node visit (id 18): 0
Exiting 'value' (id 4): 0
Exiting every rule (token [29 13 48 29 15 48 10], id 4): 0
Exiting 'pair' (id 2): "zero":0
Exiting every rule (token [13 48 29 15 48 10], id 2): "zero":0
Terminal node visit (id 19): ,
Entering every rule (token [15 48 29 15 48 10], id 2): "exponent":1e89
Entering 'pair' (id 2): "exponent":1e89
Terminal node visit (id 20): "exponent"
Terminal node visit (id 21): :
Entering every rule (token [29 15 48 29 15 48 10], id 4): 1e89
Entering 'value' (id 4): 1e89
Terminal node visit (id 22): 1e89
Exiting 'value' (id 4): 1e89
Exiting every rule (token [29 15 48 29 15 48 10], id 4): 1e89
Exiting 'pair' (id 2): "exponent":1e89
Exiting every rule (token [15 48 29 15 48 10], id 2): "exponent":1e89
Terminal node visit (id 23): ,
Entering every rule (token [15 48 29 15 48 10], id 2): "boolean":true
Entering 'pair' (id 2): "boolean":true
Terminal node visit (id 24): "boolean"
Terminal node visit (id 25): :
Entering every rule (token [29 15 48 29 15 48 10], id 4): true
Entering 'value' (id 4): true
Terminal node visit (id 26): true
Exiting 'value' (id 4): true
Exiting every rule (token [29 15 48 29 15 48 10], id 4): true
Exiting 'pair' (id 2): "boolean":true
Exiting every rule (token [15 48 29 15 48 10], id 2): "boolean":true
Terminal node visit (id 27): }
Exiting 'obj' (id 1): {"zero":0,"exponent":1e89,"boolean":true}
Exiting every rule (token [48 29 15 48 10], id 1): {"zero":0,"exponent":1e89,"boolean":true}
Exiting 'value' (id 4): {"zero":0,"exponent":1e89,"boolean":true}
Exiting every rule (token [29 15 48 10], id 4): {"zero":0,"exponent":1e89,"boolean":true}
Exiting 'pair' (id 2): "object":{"zero":0,"exponent":1e89,"boolean":true}
Exiting every rule (token [15 48 10], id 2): "object":{"zero":0,"exponent":1e89,"boolean":true}
Terminal node visit (id 28): ,
Entering every rule (token [15 48 10], id 2): "null":null
Entering 'pair' (id 2): "null":null
Terminal node visit (id 29): "null"
Terminal node visit (id 30): :
Entering every rule (token [29 15 48 10], id 4): null
Entering 'value' (id 4): null
Terminal node visit (id 31): null
Exiting 'value' (id 4): null
Exiting every rule (token [29 15 48 10], id 4): null
Exiting 'pair' (id 2): "null":null
Exiting every rule (token [15 48 10], id 2): "null":null
Terminal node visit (id 32): }
Exiting 'obj' (id 1): {"test":"string","array":[1,false],"object":{"zero":0,"exponent":1e89,"boolean":true},"null":null}
Exiting every rule (token [48 10], id 1): {"test":"string","array":[1,false],"object":{"zero":0,"exponent":1e89,"boolean":true},"null":null}
Exiting 'value' (id 4): {"test":"string","array":[1,false],"object":{"zero":0,"exponent":1e89,"boolean":true},"null":null}
Exiting every rule (token [10], id 4): {"test":"string","array":[1,false],"object":{"zero":0,"exponent":1e89,"boolean":true},"null":null}
Exiting 'json' (id 0): {"test":"string","array":[1,false],"object":{"zero":0,"exponent":1e89,"boolean":true},"null":null}
Exiting every rule (token [], id 0): {"test":"string","array":[1,false],"object":{"zero":0,"exponent":1e89,"boolean":true},"null":null}

Process finished with exit code 0

---

Quote:

---

Originally Posted by Polda18

But this isn't what I am looking for. You said that the visitor is intended to build the actual code of the parser. How?

---

This is where the Visitor part of the example kicks in,

Code:

---

    ParseTree tree = parser.r();
    ParseTreeWalker walker = new ParseTreeWalker();
    walker.walk( new HelloWalker(), tree );

---

The first line supplies the parse tree.

The second line supplies the Visitor (Walker).

The third line kicks off the traversal of the parse tree. At each node in the parse tree the methods of the HelloWalker object will be called with "context" information describing the node. This is where you can generate code. As you can see HelloWalker extends HelloBaseListener. You're supposed to override those methods of HelloBaseListener you're interested in and place them in HelloWalker and then do something useful when they get called based on the supplied context information.

This is my take on this with no knowledge of ANTLR4 at all but it should be quite close.

What? Okay. Simple question: Did you actually visit my GitHub repo? Did you understand what I am actually looking for? You just repeated me what you said before, this is going nowhere. Why override? Shall I actually NOT touch the JSONBaseListener at all and instead make the overriding class? Sorry, but I still don't quite understand. As I said many times before, I'm looking for a simplified walkthrough, in simplified way described what it does and how to benefit from it. That's all. This is neighter simplified nor quite helpful...

[wolle, please don't delete a post once it has been replied to. It breaks the thread and can make subsequent posts seem disjointed.]

Well, I don't mind if they're put back again if at all possible.

I reacted in anger because I find the OP rude and I've reported that.

I apologise if I sound rude or stupid. Part is, that I am little bit stupid, because clearly I just oversee something. Sorry, a guide is something that would help me. You can see in the linked repository (I link it again, for clarity: https://github.com/Polda18/JavaJSON) in the dir tree, that I have JSON class with some methods written out. They still need to be further implemented (they are implemented only partially), but the biggest implementation fault is in the parse() method, because it needs to address the parser, and I don't know how. It needs to build up the structure out of the JSON string given to it. The only thing that is missing from Java is the JSON object representation as it is in JavaScript (the content inside {}). That's what my class wants to bring in Java. Since Java does not natively support JSON, I have to do obstructions and implement the operations with JSON objects through Java class methods. I thought that the parser from ANTLR could be capable of providing the tokens to be implemented. Maybe I am wrong?

Quote:

---

Originally Posted by wolle

Well, I don't mind if they're put back again if at all possible.

I reacted in anger because I find the OP rude and I've reported that.

---

[Posts restored]