Add GEM/mlsum prompts for ES,DE #738

Shashi456 · 2022-04-26T08:52:14Z

No description provided.

Shashi456 · 2022-04-27T14:31:32Z

@jzf2101 Waiting for a non-english reviewer would probably take too much time, The only prompt that is in the foreign language was taken from Sebastian (verified and written by the GEM eval group).

Could I ask for someone in summarization to review this ?

promptsource/templates/GEM/mlsum/de/templates.yaml

jzf2101 · 2022-04-28T04:23:20Z

@sebastianGehrmann I will follow your formatting and try to include in spanish for consistency

Eval hackathon

Shashi456 · 2022-05-12T19:11:29Z

@jzf2101 could you take a look at this?

Shashi456 · 2022-05-19T18:50:43Z

@stephenbach could you take a look at this PR once?

jzf2101

Overall, I think these are mostly fine. Minor comments on Tweaks. The main concern I have is that we don't have 5 prompts in English for each task and 5 prompts in Spanish/German for the task. We should try to ensure that we're not mixing languages in evaluation.

jzf2101 · 2022-04-26T22:14:08Z

promptsource/templates/GEM/mlsum/de/templates.yaml

+  a4f6a6c1-ce10-463b-932e-41a9336c3ecf: !Template
+    answer_choices: null
+    id: a4f6a6c1-ce10-463b-932e-41a9336c3ecf
+    jinja: "{{document}}\n      ===\nWrite a summary of the text above : ||| {{summary}}"


{{summary}} should be {{target}}

I dont think target will produce prompts (having run the scripts to generate outputs)

jzf2101 · 2022-06-27T03:37:35Z

promptsource/templates/GEM/mlsum/de/templates.yaml

+  e9c85265-fabb-412d-9082-99750212df27: !Template
+    answer_choices: null
+    id: e9c85265-fabb-412d-9082-99750212df27
+    jinja: 'I will first show a news article and then provide a summary of it in German:


This wording IMHO seems ambiguous in that it could be read as "I will first show a news article and then I will then provide a summary of it in German" as opposed to "I will first show a news article and then you will provide a summary of it in German" From what @awebson has suggested, this should be understandable to a person. Thus, I think this ambiguity could be resolved with perhaps "I will show a news article. Please provide a summary of it in German"

cc: @sebastianGehrmann, This was just to give a different flavor of prompt, not sure if the meaning/prompt output quality has any meaningful change because of how we address the model

I am not sure I understand the comment. We are not treating this as a dialog system. Since BLOOM is a language model, it simulates generating both the article AND the summary. The implicit turn-taking of our evaluation is not of any consequence, and therefore, "I will show an article and I will show its summary" is correct.

jzf2101 · 2022-06-27T03:42:30Z

promptsource/templates/GEM/mlsum/de/templates.yaml

+    answer_choices: null
+    id: 104b2645-9b2a-4d08-8885-3de18af246fc
+    jinja: "My college roommate asked me what this German article meant:\n {{text}}\n\
+      So I recapped it in layman''s terms in German: ||| {{target}}"


There are double apostrophes here. Should be one apostrophe. Furthermore I'm not sure of the use of I in the prompt. @awebson ?

I did not add the \ explicitly, I think it was added by the formatter while creating the prompts

Is there a way to fix it using promptsource?

jzf2101 · 2022-06-27T03:43:04Z

promptsource/templates/GEM/mlsum/de/templates.yaml

+  104b2645-9b2a-4d08-8885-3de18af246fc: !Template
+    answer_choices: null
+    id: 104b2645-9b2a-4d08-8885-3de18af246fc
+    jinja: "My college roommate asked me what this German article meant:\n {{text}}\n\


@awebson use of the college roommate seem's a bit odd TBH?

promptsource/promptsource/templates/GEM/xsum/templates.yaml

Line 60 in c7486b6

jinja: 'My college roommate asked me what this article means:

This prompt was taken from xsum prompts that already existed in the repository.

maybe we should change both, I agree that this is a bit strange.

I asked @awebson for his opinion and he says this is fine.

jzf2101 · 2022-06-27T03:45:37Z

promptsource/templates/GEM/mlsum/de/templates.yaml

+  e071fb9c-b000-417d-a405-4f6032532f87: !Template
+    answer_choices: null
+    id: e071fb9c-b000-417d-a405-4f6032532f87
+    jinja: "{{text}}\n      ===\nGiven the above document, write few sentences in\


=== is rendered strangely. Moreover, I think German should be capitalized.

jzf2101 · 2022-06-27T03:56:49Z

promptsource/templates/GEM/mlsum/es/templates.yaml

+    answer_choices: null
+    id: 6fc70031-95ab-40fa-9cc7-e6eda42a4833
+    jinja: "My college roommate asked me what this Spanish article meant:\n {{text}}\n\
+      So I recapped it in layman''s terms in Spanish: ||| {{target}}"


See above comments on this prompt in German

This prompt was added from the XSUM task which already existed in the repository and has been a template for summarization tasks.

Let's change it.

jzf2101 · 2022-06-27T03:57:27Z

promptsource/templates/GEM/mlsum/de/templates.yaml

+    id: e9c85265-fabb-412d-9082-99750212df27
+    jinja: 'I will first show a news article and then provide a summary of it in German:
+
+      Fasse den folgenden Artikel zusammen: {{text}}


This seems like it should be a separate prompt as opposed to restating the prompt in German

This prompt was developed by sebastian and the GEM group for this task so has been included as is.

For GEM these are separate prompts. Let's do the same here and separate them.

jzf2101 · 2022-06-27T03:58:04Z

promptsource/templates/GEM/mlsum/es/templates.yaml

+    answer_choices: null
+    id: e3c60771-5e99-49b1-b477-c2b69f645d59
+    jinja: "I will first show a news article and then provide a summary of it in Spanish:\n\
+      Resume el siguiente art\xEDculo: {{text}}\n ===\nResumen: ||| {{target}}"


This should be a separate prompt in Spanish as opposed to part of a prompt in English. Mixing languages doesn't seem to make sense or be in current scope of multilingual prompting.

Added by sebastian and GEM group for this task.

@sebastianGehrmann could you explain this prompt?

Same comment as above - They were separate prompts at some point so we should separate them here (can just replace the roommate prompt with it or so)

jzf2101 · 2022-06-27T03:58:42Z

promptsource/templates/GEM/mlsum/es/templates.yaml

+  e3c60771-5e99-49b1-b477-c2b69f645d59: !Template
+    answer_choices: null
+    id: e3c60771-5e99-49b1-b477-c2b69f645d59
+    jinja: "I will first show a news article and then provide a summary of it in Spanish:\n\


See also comments about this prompt being ambiguous.

Same response as above - besides, this is a prompt that worked for PaLM :)

jzf2101 · 2022-06-27T03:59:02Z

promptsource/templates/GEM/mlsum/es/templates.yaml

+    id: 5e644239-d989-4531-b2ff-44b0e4310df6
+    jinja: '{{text}}
+
+      ===


This is rendered strangely, IMHO.

sebastianGehrmann · 2022-06-30T18:56:10Z

promptsource/templates/GEM/mlsum/de/templates.yaml

+  6ffadb8a-c670-4d6c-97fd-9eea35945452: !Template
+    answer_choices: null
+    id: 6ffadb8a-c670-4d6c-97fd-9eea35945452
+    jinja: "{{text}}\n TL;DR in Deutsch: ||| {{target}}"


Agreed, let's keep TL;DR. It's not super common in German but the model should have learned the association.

sebastianGehrmann · 2022-06-30T18:56:39Z

promptsource/templates/GEM/mlsum/de/templates.yaml

+  e071fb9c-b000-417d-a405-4f6032532f87: !Template
+    answer_choices: null
+    id: e071fb9c-b000-417d-a405-4f6032532f87
+    jinja: "{{text}}\n      ===\nGiven the above document, write few sentences in\


sebastianGehrmann · 2022-06-30T18:57:37Z

promptsource/templates/GEM/mlsum/de/templates.yaml

+    id: e9c85265-fabb-412d-9082-99750212df27
+    jinja: 'I will first show a news article and then provide a summary of it in German:
+
+      Fasse den folgenden Artikel zusammen: {{text}}


For GEM these are separate prompts. Let's do the same here and separate them.

sebastianGehrmann · 2022-06-30T18:57:54Z

promptsource/templates/GEM/mlsum/es/templates.yaml

+
+      ===
+
+      Write a summary of the text above : ||| {{target}}'


sebastianGehrmann · 2022-06-30T18:58:15Z

promptsource/templates/GEM/mlsum/es/templates.yaml

+    answer_choices: null
+    id: 6fc70031-95ab-40fa-9cc7-e6eda42a4833
+    jinja: "My college roommate asked me what this Spanish article meant:\n {{text}}\n\
+      So I recapped it in layman''s terms in Spanish: ||| {{target}}"


Let's change it.

sebastianGehrmann · 2022-06-30T18:58:45Z

promptsource/templates/GEM/mlsum/es/templates.yaml

+  e3c60771-5e99-49b1-b477-c2b69f645d59: !Template
+    answer_choices: null
+    id: e3c60771-5e99-49b1-b477-c2b69f645d59
+    jinja: "I will first show a news article and then provide a summary of it in Spanish:\n\


Same response as above - besides, this is a prompt that worked for PaLM :)

sebastianGehrmann · 2022-06-30T18:59:21Z

promptsource/templates/GEM/mlsum/es/templates.yaml

+    answer_choices: null
+    id: e3c60771-5e99-49b1-b477-c2b69f645d59
+    jinja: "I will first show a news article and then provide a summary of it in Spanish:\n\
+      Resume el siguiente art\xEDculo: {{text}}\n ===\nResumen: ||| {{target}}"


Same comment as above - They were separate prompts at some point so we should separate them here (can just replace the roommate prompt with it or so)

Add GEM/mlsum prompts for ES,DE

c5d0364

jzf2101 self-requested a review April 26, 2022 22:17

jzf2101 self-assigned this Apr 26, 2022

jzf2101 changed the base branch from main to eval-hackathon April 26, 2022 23:35

jzf2101 added Non-English Dataset help wanted Extra attention is needed labels Apr 27, 2022

jzf2101 mentioned this pull request Apr 27, 2022

Add GEM MLSum to Full Benchmark bigscience-workshop/evaluation#11

Open

Fix errors in the prompts

6d79e66

jzf2101 requested a review from sebastianGehrmann April 27, 2022 20:31

sebastianGehrmann suggested changes Apr 27, 2022

View reviewed changes

Review fixes

3990d8e

Shashi456 requested a review from sebastianGehrmann April 28, 2022 17:38

Merge pull request #1 from bigscience-workshop/eval-hackathon

9d4de11

Eval hackathon

jzf2101 removed the request for review from sebastianGehrmann June 26, 2022 23:11

jzf2101 requested changes Jun 27, 2022

View reviewed changes

sebastianGehrmann suggested changes Jun 30, 2022

View reviewed changes

jon-tow mentioned this pull request Aug 17, 2022

Revert option for minimum generation length bigscience-workshop/lm-evaluation-harness#118

Merged

Add GEM/mlsum prompts for ES,DE #738

Are you sure you want to change the base?

Add GEM/mlsum prompts for ES,DE #738

Uh oh!

Conversation

Shashi456 commented Apr 26, 2022

Uh oh!

Shashi456 commented Apr 27, 2022

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jzf2101 commented Apr 28, 2022

Uh oh!

Shashi456 commented May 12, 2022

Uh oh!

Shashi456 commented May 19, 2022

Uh oh!

jzf2101 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment