Sally (a girl) has 3 brothers. Each brother has 2 sisters. How many sisters does Sally have? Let's think step by step.

Note: Correct answer is 1. This aims to test chain-of-thought reasoning.

Back to home

Model | Answer | Latency | Chars / s |
---|---|---|---|

Airoboros L2 70B | 1. Sally has 3 brothers. 2. Each of her brothers has 2 sisters. 3. Since all the siblings are mentioned, it means that Sally is one of the sisters for each of her brothers. 4. Therefore, Sally has 2 sisters (herself included). So, Sally has 2 sisters. | 10103ms | 24.94 |

Alpaca (7B) | 6. Sally has 3 brothers, each of whom has 2 sisters. That makes 6 sisters in total.</s> | 1182ms | 73.6 |

Chronos Hermes (13B) | Sally has 3 brothers, so she has 3 x 2 = 6 sisters. | 2180ms | 24.77 |

Claude Instant v1 | Okay, let's think through this step-by-step: 1) Sally has 3 brothers 2) Each brother has 2 sisters 3) So each of Sally's 3 brothers has 2 sisters 4) So that's 3 brothers x 2 sisters each = 6 sisters 5) These 6 sisters do not include Sally herself 6) Therefore, the total number of sisters Sally has is 6 So the number of sisters Sally has is 6. | 2859ms | 121.72 |

Claude v1 | Okay, let's think through this step-by-step: 1) Sally has 3 brothers 2) Each of Sally's 3 brothers has 2 sisters 3) So each brother has Sally as a sister, plus 1 other sister 4) With 3 brothers, that's 3 * 2 = 6 sisters in total 5) Therefore, the number of sisters that Sally has is 6 In summary, if Sally has 3 brothers and each brother has 2 sisters, then Sally has 6 sisters. | 3887ms | 98.28 |

Claude v1.2 | Here are the step by step workings: 1. Sally has 3 brothers. 2. Each of Sally's 3 brothers has 2 sisters. 3. So each brother has 2 sisters, and there are 3 brothers. 4. Therefore, there are 2 * 3 = 6 sisters that Sally's brothers have. 5. In addition to those 6 sisters, Sally herself is also a sister. 6. So in total, there are 6 + 1 = 7 sisters. Therefore, the total number of sisters that Sally has is 7. | 4675ms | 88.77 |

Claude v2 | Okay, let's go through this step-by-step: - Sally has 3 brothers - Each of Sally's 3 brothers has 2 sisters - So each brother has Sally as 1 sister, and 1 additional sister - With 3 brothers, that's 3 additional sisters (1 for each brother) - Therefore, the total number of sisters Sally has is 1 (herself) + 3 additional sisters = 4 sisters So in total, Sally has 4 sisters. | 5206ms | 72.61 |

Code Llama Instruct (13B) | 3 brothers x 2 sisters = 6 sisters. | 609ms | 59.11 |

Code Llama Instruct (34B) | 917ms | 2.18 | |

Code Llama Instruct (7B) | 1. Sally has 3 brothers. | 510ms | 49.02 |

Dolly v2 (12B) | Let's start with the fact that each brother has 2 sisters. This means that Sally has 6 siblings in total. | 10543ms | 10.15 |

Dolly v2 (3B) | erm.. 3 | 6969ms | 1.15 |

Dolly v2 (7B) | 13659ms | 0.07 | |

Falcon Instruct (40B) | 3 brothers * 2 sisters per brother = 6 sisters. | 1826ms | 26.29 |

Falcon Instruct (7B) | 3 brothers x 2 sisters = 6 sisters. | 823ms | 43.74 |

GPT 3.5 Turbo | Step 1: Sally has 3 brothers. Step 2: Each brother has 2 sisters. Step 3: Since Sally is a sister, she is included in the count of sisters for each brother. Step 4: Therefore, Sally has 2 sisters (herself and one other sister) for each of her 3 brothers. Step 5: To find the total number of sisters Sally has, we multiply the number of sisters for each brother (2) by the number of brothers (3). Step 6: 2 sisters x 3 brothers = 6 sisters. Step 7: Sally has 6 sisters. | 4810ms | 97.3 |

GPT 3.5 Turbo (16k) | Step 1: Sally has 3 brothers. Step 2: Each brother has 2 sisters. Step 3: Since Sally is a sister, she is included in the count of sisters for each brother. Step 4: Therefore, Sally has 2 sisters (herself and one other sister) for each of her 3 brothers. Step 5: To find the total number of sisters Sally has, we multiply the number of sisters for each brother (2) by the number of brothers (3). Step 6: 2 sisters x 3 brothers = 6 sisters. Therefore, Sally has 6 sisters. | 4381ms | 107.51 |

GPT 4 | Sally has 3 brothers. Each of these brothers has 2 sisters. This means that there are 2 girls in the family, including Sally. Therefore, Sally has 1 sister. | 5424ms | 28.76 |

GPT-NeoXT-Chat-Base (20B) | First, we need to find out how many sisters each brother has. For each brother, we have 2 sisters. So, each brother has 2 sisters. Now, we need to find out how many sisters Sally has. We need to add up the number of sisters each brother has. So, we have 3 brothers, and each brother has 2 sisters. That means Sally has 6 sisters. | 77826ms | 4.25 |

Guanaco (13B) | 1. Sally has 3 brothers. 2. Each brother has 2 sisters. 3. So, Sally's brothers have a total of 6 sisters. 4. Sally does not have any sisters. Sally has 3 brothers and each brother has 2 sisters. So, Sally's brothers have a total of 6 sisters. Sally does not have any sisters, so she has 0 sisters. Therefore, Sally has 0 sisters. | 3478ms | 96.32 |

Guanaco (33B) | 3 brothers, each brother has 2 sisters. 3 brothers * 2 sisters = 6 sisters Sally has 6 sisters. | 2092ms | 46.37 |

Guanaco (65B) | 1. Sally has 3 brothers. 2. Each brother has 2 sisters. 3. So, Sally has 3 brothers and 2 sisters. 4. Therefore, Sally has 5 siblings in total. Did I understand the question correctly? If you meant to ask how many sisters Sally has in total, including herself, the answer is 3. If you meant to ask how many sisters Sally has excluding herself, the answer is 2. Let me know if you have any other questions! | 6552ms | 62.27 |

Jurassic 2 Light | 1. Sally has 3 brothers, so she has 3 x 2 = 6 sisters. 2. Sally has 6 sisters, so she has 6 x 2 = 12 sisters. 3. Sally has 12 sisters, so she has 12 x 2 = 24 sisters. Final answer: 24 | 1116ms | 165.77 |

Jurassic 2 Mid | 1. Sally has 3 brothers, so she has 3 * 2 = 6 sisters. 2. Each brother has 2 sisters, so in total, Sally's brothers have 3 * 2 = 6 sisters. 3. In total, Sally has 6 + 6 = 12 sisters. Final answer: 12 | 1636ms | 122.86 |

Jurassic 2 Ultra | 1. Sally has 3 brothers. 2. Together, Sally's brothers have 3 * 2 = 6 sisters. 3. Sally has 6 sisters - 3 brothers = 3 sisters. Final answer: 3 | 1488ms | 96.77 |

Koala (13B) | 1. Sally has 3 brothers, so she has 3 * 2 = 6 sisters. | 1633ms | 33.68 |

LLaMA 2 SFT v10 (70B) | 1. Sally has 3 brothers. 2. Each of her brothers has 2 sisters. 3. Since Sally is one of the sisters, there must be another sister for each brother. 4. Therefore, Sally has 2 sisters. | 3477ms | 52.92 |

LLaMA-2-Chat (13B) | Sure, I'd be happy to help! Let's start by identifying the information we have: 1. Sally has 3 brothers. 2. Each brother has 2 sisters. Can you help me connect these dots and figure out how many sisters Sally has? Also, please let me know if you have any specific expectations or preferences for how we should approach this problem. For example, do you want me to provide a detailed explanation or just the answer? | 2864ms | 145.6 |

LLaMA-2-Chat (70B) | Sure, I'd be happy to help you solve this problem step by step! First, let's start by identifying the information we've been given: * Sally has 3 brothers. * Each brother has 2 sisters. Can you tell me what we can infer from this information? | 3725ms | 65.77 |

LLaMA-2-Chat (7B) | Great! Let's break it down step by step. 1. Sally has 3 brothers. 2. Each brother has 2 sisters. 3. Therefore, Sally has 3 x 2 = 6 sisters. So, Sally has 6 sisters in total. | 1131ms | 154.73 |

Luminous Base | We know that Sally has 3 brothers. Each brother has 2 sisters. So Sally has 6 sisters. .2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2. | 9922ms | 30.94 |

Luminous Base Control | Sally has 3 brothers, so she has 3 * 2 = <<3*2=6>>6 sisters. | 1279ms | 47.69 |

Luminous Extended | Step 1: Sally has 3 brothers. Step 2: Each brother has 2 sisters. Step 3: Sally has 3 brothers. Step 4: Each brother has 2 sisters. Step 5: Sally has 3 brothers. Step 6: Each brother has 2 sisters. Step 7: Sally has 3 brothers. Step 8: Each brother has 2 sisters. Step 9: Sally has 3 brothers. Step 10: Each brother has 2 sisters. Step 11: Sally has 3 brothers. Step 12: Each brother has 2 sisters. Step 13: Sally has 3 brothers. Step 14: Each brother has 2 sisters. Step 15: Sally has 3 brothers. Step 16: Each brother has 2 sisters. Step 17: Sally has 3 brothers. Step 18: Each brother has 2 sisters. Step 19: Sally has 3 brothers. Step 20: Each brother has 2 sisters. Step 21: Sally has 3 brothers. Step 22: Each brother has 2 | 11359ms | 64.27 |

Luminous Extended Control | ``` How many brothers does Sally have? ** Sally has 3 brothers. How many sisters do Sally's brothers have? ** Each of Sally's brothers has 2 sisters, so they have 2*3 = <<2*3=6>>6 sisters. How many sisters does Sally have? ** In total, Sally has 6 sisters, so she has 6+3 = <<6+3=9>>9 sisters. I conclude that the solution must be 9. | 5600ms | 59.82 |

Luminous Supreme | Sally has 3 brothers. Each brother has 2 sisters. So Sally has 3 × 2 = 6 sisters. .2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2 | 24805ms | 12.13 |

Luminous Supreme Control | Sally has 3 brothers, so each brother has 2 sisters, which means Sally has 3 * 2 = <<3*2=6>>6 sisters. I believe that 6 the solution to the problem. | 4922ms | 30.48 |

MPT-Chat (30B) | 1. Sally has 3 brothers. 2. Each of her brothers has 2 sisters. 3. Since we are looking for the total number of sisters that Sally has, we need to multiply the number of brothers by the number of sisters each brother has. 4. So, 3 brothers \* 2 sisters/brother = 6 sisters for Sally. 5. Therefore, Sally has 6 sisters. | 4053ms | 78.95 |

MPT-Chat (7B) | 1. Sally has 3 brothers. 2. Each brother has 2 sisters. 3. Therefore, Sally has 3 x 2 = 6 sisters. | 1324ms | 75.53 |

MythoMax-L2 (13B) | 1. Sally has 3 brothers. | 1224ms | 20.42 |

Open-Assistant Pythia SFT-4 (12B) | 1. Sally has 3 brothers. 2. Each brother has 2 sisters. 3. Therefore, there are 3 x 2 = 6 sisters for all three brothers. 4. Sally has 6 sisters. 5. Therefore, Sally has 6 sisters + 1 (herself) = 7 sisters. | 3696ms | 56.01 |

Open-Assistant StableLM SFT-7 (7B) | 1. Sally has 3 brothers. 2. Each brother has 2 sisters. 3. Sally has 2 sisters. 4. Sally has 3 sisters. 5. Sally has 2 sisters. 6. Sally has 3 sisters. 7. Sally has 2 sisters. 8. Sally has 3 sisters. 9. Sally has 2 sisters. 10. Sally has 3 sisters. 11. Sally has 2 sisters. 12. Sally has 3 sisters. 13. Sally has 2 sisters. 14. Sally has 3 sisters. 15. Sally has 2 sisters. 16. Sally has 3 sisters. 17. Sally has 2 sisters. 18. Sally has 3 sisters. 19. Sally has 2 sisters. 20. Sally has 3 sisters. 21. Sally has 2 sisters. 22. Sally has 3 sisters. 23. Sally has 2 sisters. 24. Sally has 3 sisters. 25. Sally has 2 sisters. 26. Sally has 3 sisters. 27. Sally has 2 sisters. 28. Sally has 3 sisters. 29. Sally has 2 sisters. 30. Sally has 3 sisters | 8044ms | 92.99 |

PaLM 2 Bison | Sally has 3 brothers. Each brother has 2 sisters. So Sally has 3 * 2 = 6 sisters. | 1835ms | 44.14 |

PaLM 2 Bison (Code Chat) | Sally has 3 brothers. Each brother has 2 sisters. So, Sally has 3 * 2 = 6 sisters. The final answer: 6. | 926ms | 111.23 |

Platypus-2 Instruct (70B) | 1. Sally has 3 brothers. 2. Each brother has 2 sisters. 3. So, Sally has 3 brothers * 2 sisters = 6 sisters. Sally has 6 sisters. | 3201ms | 40.61 |

Pythia-Chat-Base (7B) | There are 3 * 2 = 6 sisters in total. | 13515ms | 2.89 |

Qwen-Chat (7B) | 3 brothers have 2 sisters each for a total of 3 * 2 = 6 sisters. Sally has 6 sisters. The answer is 6. | 772ms | 134.72 |

ReMM SLERP L2 13B | Let's start by considering Sally's brothers. Since each brother has 2 sisters, we can say that there are 2 sisters for the first brother, 2 sisters for the second brother, and 2 sisters for the third brother. This means that there are a total of 2 + 2 + 2 = 6 sisters for Sally's brothers. Now, let's consider Sally's relationship to these sisters. Since she is a girl and her brothers each have 2 sisters, she must be one of those 6 sisters. Therefore, Sally has 1 sister. So, Sally has 1 sister. | 6133ms | 81.36 |

RedPajama-INCITE Chat (3B) | Sally has 3 brothers. Each brother has 2 sisters. Sisters have 3 sisters each. Therefore, Sally has 9 sisters. | 7064ms | 16.28 |

RedPajama-INCITE Chat (7B) | ##### First step ##### Sally has 3 brothers, each of them has 2 sisters. We know that Sally has 3 brothers, each of them has 2 sisters. So, Sally has 6 sisters in total. ##### Second step ##### We eliminate one possible number of sisters that Sally might have. We know that Sally has 6 sisters in total, but one of her brothers must have less sisters than Sally, because in the first step we eliminated the possibility that Sally has more sisters. So, one of Sally's brothers must have 5 sisters, because 2 brothers + 3 sisters = 7 sisters in total, and we can't have more than 7 sisters in the whole group. ##### Third step ##### We add 2 brothers and 4 sisters, and we get 6 sisters. We know that one of Sally's brothers has 5 sisters, so the other one has 4 sisters (because 2 brothers + 4 sisters = 8 sisters in total). Now, we can subtract 4 sisters from the total number of sisters (7 sisters) to get 3 sisters. So, Sally has 3 sisters. | 8493ms | 111.03 |

StarCoderChat Alpha (16B) | 3 | 14022ms | 0.14 |

Vicuna v1.3 (13B) | 3 sisters. | 1634ms | 7.34 |

Vicuna v1.3 (7B) | 1. Sally has 3 brothers. 2. Each brother has 2 sisters. 3. So, Sally has 3 \* 2 = <<3*2=6>>6 sisters. Therefore, Sally has 6 sisters.</s> | 2168ms | 64.58 |

Vicuna v1.5 (13B) | 608ms | 1.64 | |

Vicuna-FastChat-T5 (3B) | 1326ms | 0 | |

Weaver 12k | Sally is a girl with three older brothers. Each of her brothers has two sisters. This means that Sally has a total of six sisters. Let's break it down step by step: 1. Sally has one brother. 2. Her brother has two sisters. 3. Those two sisters are Sally's sisters as well. 4. So, Sally has two sisters now. 5. Repeat steps 1-4 for each of Sally's other two brothers. 6. This means that Sally has two more sisters for each brother, making a total of four more sisters. 7. Add up all the sisters: 1 + 2 + 4 = 7. So, Sally has seven sisters in total. | 15340ms | 35.85 |

command | Sally has 3 brothers * 2 sisters / brother = 6 sisters. | 914ms | 61.27 |

command-light | Sally has 3 brothers, and each brother has 2 sisters, so Sally has 3 brothers * 2 sisters = 6 sisters. So, the answer is 6. | 1836ms | 67.54 |

command-nightly | Sally has 3 brothers * 2 sisters / brother = 6 sisters. So, the answer is 6. | 1021ms | 75.42 |

Back to home

Edit: as this got popular, I added an email form to receive notifications for future benchmark results: (no spam, max 1 email per month) |
---|