• Vigneshwaran

    Hi everyone,

    Thanks for taking your time to read the poster. Problems I am facing with evaluation: Correctness of output in this approach is the order in which parts (component schemas) were assembled into whole (composite schemas) and whether each assembly was semantically valid. As there is no gold standard, it poses problem in evaluating the parser performance. Two solutions I could imagine (a) generate sentences by template-filling the schemas that make up a parse, let humans rate them manually and find average rating (b) assume that all the schemas that make up a parse should correspond to valid phrases in the constituency tree.

    1. Are there any problems in my assumptions?
    2. How can I evaluate the system better?

  • Emily Powell

    Hi Vignesh, I can’t help to answer your questions, but I would like to know more about the ‘why’ behind your research. What would your system be used for and why is it needed?

  • Kateryna

    Hi, Vignesh

    I think your practical approach is very interesting and has the potential to be developed into a fully-fledged theory. There is a major challenge, however, in pursuing this pledge. Cognitive schemas are notably general concepts, which are scantly described and studies in present-day linguistic literature, but the number of various patterns and rules in language is immense, in particular, when you consider that the language system is organized non-linearly (i.e. it consists of various layers and dimensions which constantly interact). Hence, the challenge is how to make rules and patterns of a language compatible and consistent with these cognitive schemas such that an accurate parsing becomes possible. Further, although I am not a computational linguist, my perception is that, in order to achieve a precision in the linguistic analysis, computational systems should me trained on large sets of manually annotated data (as, for example, in the case with the visual recognition task). Therefore, my suggestion is that it might be helpful to support the morpho-syntactic parser which you are developing with the considerable amount of the formal data annotated by humans.

  • Vigneshwaran M

    Hi Emily,

    Thanks for the comment. The reason I worked on this idea is to develop Natural Language Processing tools for less resourced language like Welsh where large, annotated corpora are not available so far. As a part of CorCenCC project we have developed various tools like corpus query tools, pedagogic toolkit, stemmer. This parser is also a tool that will be released on GitHub that everyone can use. Does this help?

  • Vigneshwaran M

    It is a useful suggestion Kateryna. I think this will have to be taken up as a separate work in future. Thanks for your feedback.

  • Andy Buerki

    Dear Vignesh,

    Fascinating topic! Your poster is at quite an abstract level, might you be able to provide examples of parses you would like to verify? Are the schemas at POS-level or higher, or can/must they have lemma or token components? It seems to me that if they are at POS-level or above, template filling with any filler of matching POS is not necessarily going to be a good indication of whether you had a good parse as often restrictions are at a lower level, so even though an abstract schema might be the correct parse for a particular phrase, filling that same schema with POS-matching lexical material is not necessarily going to be rated a good sentence. But I am rather making a lot of guesses as to what your schemas and filled templates might consist of.

  • Gerard O'Grady

    Thanks Vignesh, that was a fascinating poster. Upon reading it I wondered about the gain in accuracy by using wordnet -is it a significant gain? What is the target for accuracy? Also, if you know, are there differences in accuracy rates across different contexts and if so what are they?

  • Vigneshwaran M

    Dear Andy,

    Thanks a lot for the valuable feedback. You are right. If schemas are too abstract, lexical and lower level syntactic constraints are lost. If too specific, the point of schematising is lost and we cannot generalise specific usages into patterns of construction. So I have defined schemas at three levels : basic level, process and event level. Basic level schemas are defined in terms of POS tags. They are simple but too abstract and general. Process level schemas are composites of POS tags and basic schemas. They are relatively more specific. Event level schemas are composites of POS, basic and process level schemas. They are highly specific of all the three levels but compositionally complex. Since they are defined as composed of lower levels they capture usages such as ‘Unless EVENT_CLOSED-AUTONOMOUS ~ ‘.

    By way of these definitions, our intention is to capture self similar structures at different levels of syntax. For example, an auxiliary verb construction such as ‘has come’, whose POS sequence is (VBZ ~ VBN), is recognised as patterning with following sequence (PROCESS_CLOSED-AUTONOMOUS ~ STATUS_CLOSED-DEPENDENT) which in turn is recognised as PROCESS_CLOSED-AUTONOMOUS. However the same (PROCESS_CLOSED-AUTONOMOUS ~ STATUS_CLOSED-DEPENDENT) pattern also recognises usages such as ‘felt broken’ (VBD VBN), ‘seems interesting’ (VBZ ADJ).

    Words / tokens are not directly used in defining the schemas but they are used to provide context to recognise the pattern. For example, ‘Although they came…..’ (IN ~ PRP ~ VBD…) should ideally be recognised as (EVENT_CONTINUATIVE-DEPENDENT ~ EVENT_CLOSED-AUTONOMOUS) instead of choosing many other equally valid alternatives. This is facilitated by the token ‘although’; the POS tag ‘IN’ alone is too ambiguous to arrive at this interpretation. In summary: Words, POS tag sequences (optional: worndet synset) are given to neural network to provide context for recognising the schemas. The schemas themselves are defined at POS level, process level and event level manually by recognising self-similar structures at different levels of usage.

  • Vigneshwaran Muralidaran

    Hi Gerard,

    The evaluation result reported in the poster for English is not accurate. This is because I counted a parse to be correct if the words making up a schema are found as constituents in the phrase structure tree. This method of evaluation will report 100% accuracy if all that was learned by the system was word level schemas (i.e. equivalent to POS tagging). This is not correct.

    I have recently run experiments on new data for English and Welsh (WSJ sample corpus for English and CEG corpus for Welsh). Now the evaluation was done by comparing all intermediate schemas against manually annotated phrase structure trees for 50 sentences. Updated accuracies are: 83.46% (without sunsets) and 87.19% (with synsets) for English; 76.51% (without synsets) and 78.85% (with synsets) for Welsh. The improvement by adding synset is significant in both the languages. Parsers developed for English using supervised methods go higher than 90%. The target will be to reach similar levels of accuracy.

Leave a Reply

Your email address will not be published. Required fields are marked *