dcsimg
CodeGuru Home VC++ / MFC / C++ .NET / C# Visual Basic VB Forums Developer.com
Results 1 to 13 of 13

Thread: How can I use ANTLR generated lexer/parser source code?

  1. #1
    Join Date
    Jul 2019
    Location
    Czech Republic
    Posts
    11

    Question How can I use ANTLR generated lexer/parser source code?

    Hi. I was convinced to use ANTLR to generate code for lexer and parser for my JSON Java library, however I can google anything and still can't seem to find info on how to use the ANTLR generated source codes. I found tutorials on how to set up ANTLR with JetBrains IDE (which I am using), and how to generate the source codes I need (and how to add them to the sources in project). What I didn't find, is how the actual generated Java source codes should be used. The generated source code files do not contain any JavaDoc, only informs about deprecated methods, nothing else. This isn't much helping. I can find tutorials on how to create own lexer/parser rules (own grammar), but nothing about the actual usage of the generated source code. Not even the ANTLR Mega Tutorial (https://tomassetti.me/antlr-mega-tutorial/) which displays as first in Google search and which is generally pointed to when someone asks about ANTLR. So I have JSON.g4 already downloaded, already used, and I already created the source code files for the lexer and parser, including visitor. But I'm stuck, I don't know how to use it Does anybody here know how to use ANTLR? How to use the generated code? I can't find well explained tutorial. I need something more like an idiotic way explained, because technically I'm still new to programming as whole (not just Java), still learning. I know the language, the types, but I'm still learning on how to work with Java. I can make simple program like "Hello World" or some kind of calculator. But I thought that this isn't actually something to pull of and decided to go with something bigger, like a library. These tutorials treat people like someone who know the language well and know how to work with it as well as they know it. That isn't my case. Like I said, I'd like to find more like an idiotic way of explanation. I found a book, but the thing is that I can't actually buy any book, not even virtual. I need money for my study, which isn't actually programming, it's Electrical Engineering. I am learning programming for a hobby, not a job...

    As for the project, all you need can be found on GitHub: https://github.com/Polda18/JavaJSON

  2. #2
    Join Date
    Feb 2017
    Posts
    483

    Re: How can I use ANTLR generated lexer/parser source code?

    I found this ANTLR reference for free on the net,

    http://lms.ui.ac.ir/public/group/90/...15738_ce57.pdf

  3. #3
    Join Date
    Jul 2019
    Location
    Czech Republic
    Posts
    11

    Re: How can I use ANTLR generated lexer/parser source code?

    I mean, I didn't download anything and I don't thank anybody And I'm not gonna use it

  4. #4
    Join Date
    Jul 2019
    Location
    Czech Republic
    Posts
    11

    Re: How can I use ANTLR generated lexer/parser source code?

    And I still can't find what I am looking for I told ya, a more idiotic way. That means an actual guide how to use the generated code, how to edit the code correctly so it does what it's intended to do. Did you check out the GitHub link I posted? Then you might see what the generated code looks like. The lexer and parser Java source code files contain no usable JavaDoc, the interface classes JSONVisitor and JSONListener do contain quite more helpful JavaDoc, but no information on what should actually go inside. And the implementations JSONBaseListener and JSONBaseVisitor contain nothing. Listener is empty and Visitor contains default visitChildren() method, which I'm not quite certain what it does and if this is actually what it should do... Well, frankly to say, I cannot tell at all what that default method actually does. The listener contains no code in implemented methods at all, so this is truly wrong. But I don't know how to fix it. The Definitive ANTLR4 takes it more broadly, it's not actually a guide for the use of ANTLR, only general explanation, so I am quite glad I didn't have to give my money for that. There was a section about parsing JSON, but it didn't contain the explanation of the generated methods (like the general broad explanation didn't quite explain generated classes), it moved straight to the testing of generated and fully implemented parser, which I cannot see Can't I really find anything that would tell me "put this there, remove that, change this"? This is an idiotic way of explanation that I am looking for. This is what we in Czech call "shovel method" of explanation...

  5. #5
    Join Date
    Feb 2017
    Posts
    483

    Re: How can I use ANTLR generated lexer/parser source code?

    Quote Originally Posted by Polda18 View Post
    And I still can't find what I am looking for I told ya, a more idiotic way.
    There's a small fee of admission. You will need to check out the Visitor design pattern (also called Walker). It's because ANTLR4 generates a parse tree that's supposed to be traversed using a visitor which job it is to lay down the actual code.

    To learn how to put the ANTLR4 generated parts together I suggest you locate a small ANTLR4 example program with a main() method. Here's one such example that may fit the bill,

    https://gist.github.com/mattmcd/5425206

    First get your small working example of choice up and running. Then familiarize yourself with how it works, especially how the different parts fit together (by putting in traces and making changes etcetera). Then it's time to generate your own application. Start small and expand incrementally rather than doing all in one go.

    So it's not "idiotic" explanations you should be looking for but the Java entry point main() in the ANTLR4 examples you find. Good luck!
    Last edited by wolle; July 30th, 2019 at 12:35 AM.

  6. #6
    Join Date
    Jul 2019
    Location
    Czech Republic
    Posts
    11

    Re: How can I use ANTLR generated lexer/parser source code?

    The only problem is, there isn't any main program. This is a library I am building, no program. Shall I use Java console? If that works actually?

    EDIT: Okay, I created a main class that generates the library. I used your link as an example and reworked it to use the JSON class methods. What does belong to the ANTLRFileStream? A file name? Or a stream of characters? Is it a JSON string or something else?

    EDIT2: So, I guess that correct input is the file that contains the JSON string to walk. However, it does nothing. It only walks through and tests it. I need to make a parser that could be used to parse it in my library class JSON. As I said, I need an "idiotic" way (or how we in Czech call it the "shovel" way) of the ANTLR explanation. I am aware of the grammar syntax, I am aware what it makes and I am now thanks to your link aware of how the lexer is tested. But this isn't what I am looking for. You said that the visitor is intended to build the actual code of the parser. How?

    EDIT3: Oh yes, can you, please, provide me like an oversimplifed explanation? Because I still can't make it out...
    Last edited by Polda18; July 30th, 2019 at 02:53 AM.

  7. #7
    Join Date
    Jul 2019
    Location
    Czech Republic
    Posts
    11

    Re: How can I use ANTLR generated lexer/parser source code?

    So I managed to somehow edit it, but I still can't make out how it works. I tried really hard to guess what exactly it does, but I can't quite see how it actually works. The visitor now outputs to the console the token id and text, but I don't see much logic here. The only thing I can see that the visitor first calls to enter every rule method and check what it is and call to enter the apropriate rule. Then it gets in recursion mode. Terminal visitor splits tokens as I can see. But that's all. How can I work out the tokenizer and what it does? Also, there is an issue. I need to somehow make the visitor to build up the object. But it has void type and I need it to determine the object to include. I need it to fit inside the JSON class I wrote by hand. Did you actually check out the repository before you started posting links? That's where all the issues lie. That's what I need to address. I need to use the generated classes in the JSON class, but I need it to actually work. What you instead gave me was some general walkthrough without any explanation and without any expected result example. Hello world is pretty good example to reproduce, but I guess that doesn't quite help me much in aid to build a Java class for JSON implementation.

    This is the console output as related to current code state (found on linked GitHub repository):
    Code:
    "C:\Program Files\JetBrains\IntelliJ IDEA Community Edition 2019.1.2\jre64\bin\java.exe" "-javaagent:C:\Program Files\JetBrains\IntelliJ IDEA Community Edition 2019.1.2\lib\idea_rt.jar=56976:C:\Program Files\JetBrains\IntelliJ IDEA Community Edition 2019.1.2\bin" -Dfile.encoding=UTF-8 -classpath "C:\Users\Acer\Documents\GitHub\JavaJSON\out\production\main;C:\Program Files\JetBrains\IntelliJ IDEA Community Edition 2019.1.2\lib\validation-api-2.0.0.Final.jar;C:\Program Files\JetBrains\IntelliJ IDEA Community Edition 2019.1.2\lib\antlr-4.7.2-complete.jar" Main example.json
    Entering every rule (token [], id 0): {"test":"string","array":[1,false],"object":{"zero":0,"exponent":1e89,"boolean":true},"null":null}
    Entering 'json' (id 0): {"test":"string","array":[1,false],"object":{"zero":0,"exponent":1e89,"boolean":true},"null":null}
    Entering every rule (token [10], id 4): {"test":"string","array":[1,false],"object":{"zero":0,"exponent":1e89,"boolean":true},"null":null}
    Entering 'value' (id 4): {"test":"string","array":[1,false],"object":{"zero":0,"exponent":1e89,"boolean":true},"null":null}
    Entering every rule (token [48 10], id 1): {"test":"string","array":[1,false],"object":{"zero":0,"exponent":1e89,"boolean":true},"null":null}
    Entering 'obj' (id 1): {"test":"string","array":[1,false],"object":{"zero":0,"exponent":1e89,"boolean":true},"null":null}
    Terminal node visit (id 0): {
    Entering every rule (token [13 48 10], id 2): "test":"string"
    Entering 'pair' (id 2): "test":"string"
    Terminal node visit (id 1): "test"
    Terminal node visit (id 2): :
    Entering every rule (token [29 13 48 10], id 4): "string"
    Entering 'value' (id 4): "string"
    Terminal node visit (id 3): "string"
    Exiting 'value' (id 4): "string"
    Exiting every rule (token [29 13 48 10], id 4): "string"
    Exiting 'pair' (id 2): "test":"string"
    Exiting every rule (token [13 48 10], id 2): "test":"string"
    Terminal node visit (id 4): ,
    Entering every rule (token [15 48 10], id 2): "array":[1,false]
    Entering 'pair' (id 2): "array":[1,false]
    Terminal node visit (id 5): "array"
    Terminal node visit (id 6): :
    Entering every rule (token [29 15 48 10], id 4): [1,false]
    Entering 'value' (id 4): [1,false]
    Entering every rule (token [49 29 15 48 10], id 3): [1,false]
    Entering 'array' (id 3): [1,false]
    Terminal node visit (id 7): [
    Entering every rule (token [32 49 29 15 48 10], id 4): 1
    Entering 'value' (id 4): 1
    Terminal node visit (id 8): 1
    Exiting 'value' (id 4): 1
    Exiting every rule (token [32 49 29 15 48 10], id 4): 1
    Terminal node visit (id 9): ,
    Entering every rule (token [34 49 29 15 48 10], id 4): false
    Entering 'value' (id 4): false
    Terminal node visit (id 10): false
    Exiting 'value' (id 4): false
    Exiting every rule (token [34 49 29 15 48 10], id 4): false
    Terminal node visit (id 11): ]
    Exiting 'array' (id 3): [1,false]
    Exiting every rule (token [49 29 15 48 10], id 3): [1,false]
    Exiting 'value' (id 4): [1,false]
    Exiting every rule (token [29 15 48 10], id 4): [1,false]
    Exiting 'pair' (id 2): "array":[1,false]
    Exiting every rule (token [15 48 10], id 2): "array":[1,false]
    Terminal node visit (id 12): ,
    Entering every rule (token [15 48 10], id 2): "object":{"zero":0,"exponent":1e89,"boolean":true}
    Entering 'pair' (id 2): "object":{"zero":0,"exponent":1e89,"boolean":true}
    Terminal node visit (id 13): "object"
    Terminal node visit (id 14): :
    Entering every rule (token [29 15 48 10], id 4): {"zero":0,"exponent":1e89,"boolean":true}
    Entering 'value' (id 4): {"zero":0,"exponent":1e89,"boolean":true}
    Entering every rule (token [48 29 15 48 10], id 1): {"zero":0,"exponent":1e89,"boolean":true}
    Entering 'obj' (id 1): {"zero":0,"exponent":1e89,"boolean":true}
    Terminal node visit (id 15): {
    Entering every rule (token [13 48 29 15 48 10], id 2): "zero":0
    Entering 'pair' (id 2): "zero":0
    Terminal node visit (id 16): "zero"
    Terminal node visit (id 17): :
    Entering every rule (token [29 13 48 29 15 48 10], id 4): 0
    Entering 'value' (id 4): 0
    Terminal node visit (id 18): 0
    Exiting 'value' (id 4): 0
    Exiting every rule (token [29 13 48 29 15 48 10], id 4): 0
    Exiting 'pair' (id 2): "zero":0
    Exiting every rule (token [13 48 29 15 48 10], id 2): "zero":0
    Terminal node visit (id 19): ,
    Entering every rule (token [15 48 29 15 48 10], id 2): "exponent":1e89
    Entering 'pair' (id 2): "exponent":1e89
    Terminal node visit (id 20): "exponent"
    Terminal node visit (id 21): :
    Entering every rule (token [29 15 48 29 15 48 10], id 4): 1e89
    Entering 'value' (id 4): 1e89
    Terminal node visit (id 22): 1e89
    Exiting 'value' (id 4): 1e89
    Exiting every rule (token [29 15 48 29 15 48 10], id 4): 1e89
    Exiting 'pair' (id 2): "exponent":1e89
    Exiting every rule (token [15 48 29 15 48 10], id 2): "exponent":1e89
    Terminal node visit (id 23): ,
    Entering every rule (token [15 48 29 15 48 10], id 2): "boolean":true
    Entering 'pair' (id 2): "boolean":true
    Terminal node visit (id 24): "boolean"
    Terminal node visit (id 25): :
    Entering every rule (token [29 15 48 29 15 48 10], id 4): true
    Entering 'value' (id 4): true
    Terminal node visit (id 26): true
    Exiting 'value' (id 4): true
    Exiting every rule (token [29 15 48 29 15 48 10], id 4): true
    Exiting 'pair' (id 2): "boolean":true
    Exiting every rule (token [15 48 29 15 48 10], id 2): "boolean":true
    Terminal node visit (id 27): }
    Exiting 'obj' (id 1): {"zero":0,"exponent":1e89,"boolean":true}
    Exiting every rule (token [48 29 15 48 10], id 1): {"zero":0,"exponent":1e89,"boolean":true}
    Exiting 'value' (id 4): {"zero":0,"exponent":1e89,"boolean":true}
    Exiting every rule (token [29 15 48 10], id 4): {"zero":0,"exponent":1e89,"boolean":true}
    Exiting 'pair' (id 2): "object":{"zero":0,"exponent":1e89,"boolean":true}
    Exiting every rule (token [15 48 10], id 2): "object":{"zero":0,"exponent":1e89,"boolean":true}
    Terminal node visit (id 28): ,
    Entering every rule (token [15 48 10], id 2): "null":null
    Entering 'pair' (id 2): "null":null
    Terminal node visit (id 29): "null"
    Terminal node visit (id 30): :
    Entering every rule (token [29 15 48 10], id 4): null
    Entering 'value' (id 4): null
    Terminal node visit (id 31): null
    Exiting 'value' (id 4): null
    Exiting every rule (token [29 15 48 10], id 4): null
    Exiting 'pair' (id 2): "null":null
    Exiting every rule (token [15 48 10], id 2): "null":null
    Terminal node visit (id 32): }
    Exiting 'obj' (id 1): {"test":"string","array":[1,false],"object":{"zero":0,"exponent":1e89,"boolean":true},"null":null}
    Exiting every rule (token [48 10], id 1): {"test":"string","array":[1,false],"object":{"zero":0,"exponent":1e89,"boolean":true},"null":null}
    Exiting 'value' (id 4): {"test":"string","array":[1,false],"object":{"zero":0,"exponent":1e89,"boolean":true},"null":null}
    Exiting every rule (token [10], id 4): {"test":"string","array":[1,false],"object":{"zero":0,"exponent":1e89,"boolean":true},"null":null}
    Exiting 'json' (id 0): {"test":"string","array":[1,false],"object":{"zero":0,"exponent":1e89,"boolean":true},"null":null}
    Exiting every rule (token [], id 0): {"test":"string","array":[1,false],"object":{"zero":0,"exponent":1e89,"boolean":true},"null":null}
    
    Process finished with exit code 0

  8. #8
    Join Date
    Feb 2017
    Posts
    483

    Re: How can I use ANTLR generated lexer/parser source code?

    Quote Originally Posted by Polda18 View Post
    But this isn't what I am looking for. You said that the visitor is intended to build the actual code of the parser. How?
    This is where the Visitor part of the example kicks in,
    Code:
        ParseTree tree = parser.r();
        ParseTreeWalker walker = new ParseTreeWalker();
        walker.walk( new HelloWalker(), tree );
    The first line supplies the parse tree.

    The second line supplies the Visitor (Walker).

    The third line kicks off the traversal of the parse tree. At each node in the parse tree the methods of the HelloWalker object will be called with "context" information describing the node. This is where you can generate code. As you can see HelloWalker extends HelloBaseListener. You're supposed to override those methods of HelloBaseListener you're interested in and place them in HelloWalker and then do something useful when they get called based on the supplied context information.

    This is my take on this with no knowledge of ANTLR4 at all but it should be quite close.
    Last edited by wolle; July 30th, 2019 at 05:01 AM.

  9. #9
    Join Date
    Jul 2019
    Location
    Czech Republic
    Posts
    11

    Re: How can I use ANTLR generated lexer/parser source code?

    What? Okay. Simple question: Did you actually visit my GitHub repo? Did you understand what I am actually looking for? You just repeated me what you said before, this is going nowhere. Why override? Shall I actually NOT touch the JSONBaseListener at all and instead make the overriding class? Sorry, but I still don't quite understand. As I said many times before, I'm looking for a simplified walkthrough, in simplified way described what it does and how to benefit from it. That's all. This is neighter simplified nor quite helpful...

  10. #10
    2kaud's Avatar
    2kaud is offline Super Moderator Power Poster
    Join Date
    Dec 2012
    Location
    England
    Posts
    6,921

    Re: How can I use ANTLR generated lexer/parser source code?

    [wolle, please don't delete a post once it has been replied to. It breaks the thread and can make subsequent posts seem disjointed.]
    All advice is offered in good faith only. All my code is tested (unless stated explicitly otherwise) with the latest version of Microsoft Visual Studio (using the supported features of the latest standard) and is offered as examples only - not as production quality. I cannot offer advice regarding any other c/c++ compiler/IDE or incompatibilities with VS. You are ultimately responsible for the effects of your programs and the integrity of the machines they run on. Anything I post, code snippets, advice, etc is licensed as Public Domain https://creativecommons.org/publicdomain/zero/1.0/ and can be used without reference or acknowledgement. Also note that I only provide advice and guidance via the forums - and not via private messages!

    C++17 Compiler: Microsoft VS2019 (16.4.0)

  11. #11
    Join Date
    Feb 2017
    Posts
    483

    Re: How can I use ANTLR generated lexer/parser source code?

    Well, I don't mind if they're put back again if at all possible.

    I reacted in anger because I find the OP rude and I've reported that.

  12. #12
    Join Date
    Jul 2019
    Location
    Czech Republic
    Posts
    11

    Re: How can I use ANTLR generated lexer/parser source code?

    I apologise if I sound rude or stupid. Part is, that I am little bit stupid, because clearly I just oversee something. Sorry, a guide is something that would help me. You can see in the linked repository (I link it again, for clarity: https://github.com/Polda18/JavaJSON) in the dir tree, that I have JSON class with some methods written out. They still need to be further implemented (they are implemented only partially), but the biggest implementation fault is in the parse() method, because it needs to address the parser, and I don't know how. It needs to build up the structure out of the JSON string given to it. The only thing that is missing from Java is the JSON object representation as it is in JavaScript (the content inside {}). That's what my class wants to bring in Java. Since Java does not natively support JSON, I have to do obstructions and implement the operations with JSON objects through Java class methods. I thought that the parser from ANTLR could be capable of providing the tokens to be implemented. Maybe I am wrong?

  13. #13
    2kaud's Avatar
    2kaud is offline Super Moderator Power Poster
    Join Date
    Dec 2012
    Location
    England
    Posts
    6,921

    Re: How can I use ANTLR generated lexer/parser source code?

    Quote Originally Posted by wolle View Post
    Well, I don't mind if they're put back again if at all possible.

    I reacted in anger because I find the OP rude and I've reported that.
    [Posts restored]
    All advice is offered in good faith only. All my code is tested (unless stated explicitly otherwise) with the latest version of Microsoft Visual Studio (using the supported features of the latest standard) and is offered as examples only - not as production quality. I cannot offer advice regarding any other c/c++ compiler/IDE or incompatibilities with VS. You are ultimately responsible for the effects of your programs and the integrity of the machines they run on. Anything I post, code snippets, advice, etc is licensed as Public Domain https://creativecommons.org/publicdomain/zero/1.0/ and can be used without reference or acknowledgement. Also note that I only provide advice and guidance via the forums - and not via private messages!

    C++17 Compiler: Microsoft VS2019 (16.4.0)

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  


Windows Mobile Development Center


Click Here to Expand Forum to Full Width




On-Demand Webinars (sponsored)