much: a program to generate random multiple-choice questionnaires using TeX =========================================================================== Version 0.03 Mihalis N. Kolountzakis, Department of Mathematics, Univ. of Crete, Greece E-mail: kolount@gmail.com 29 April 2006 Introduction ------------ "much" generates a number of multiple choice examination papers (tests). These are randomly generated according to some specifications (the "test description file") from a database of questions. Each such test receives a serial number and information is kept about which answers are the correct ones for each serial number. "much" can grade the answers to these tests if it knows for each answered test the serial numbers and the corresponding given answers. "much" creates a file in TeX which can be read in by an appropriate wrapper TeX or LaTeX file. This file specifies the general look of each exam paper. The wrapper TeX file is then compiled by the user as usual and its resulting dvi file printed. Command line ------------ "much" is controlled from a test-description file, an example of which is listed below in Appendix A. When "much" is used to create the tests the usage is much create whereas when it is used to grade the answers one invokes it using much grade If is ommitted then the program reads from the standard input. If the command line is just "much" the user is asked to choose creation or grading mode and the test-description file name. After the creation phase the user compiles a wrapper TeX file which reads in "much"'s output in order to produce the tests. (See the section "The master LaTeX file" below.) The database of questions ------------------------- Each question is written in a separate file, with similar questions having similar filenames. (Look in directory "example" of the program distribution.) The questions are described in TeX. Each question file consists of several parts, each ending with a line starting with a dot (.), even the last part. The first part is the question itself, the second is the (only) correct answer and the remaining (one or more) parts are wrong answers. Up to six (6) answers may be given to each question. See, for instance, the file "comb-binomial" in directory "example". This file is also listed in Appendix C of this document. Test description language ------------------------- The following commands may be given in the test description file (look at file "test-description" in directory "example", given also in Appendix A of this document): * directory ; This sets a prefix for each required or created file, with the exeption of "mc-logfile". * seed ; Initializes the random number generator. * serial ; Defines the first serial number. The serial numbers used will increase by one for each new paper created. * use from "" ; This asks "much" to use, for each paper it generates, a random susbset of size , of all questions whose names (there is one file per question) match the regular expression . (No quote (") character can be used in .) For example, if the questions existing have names a1, a2, a3, b1, b2, b3, b4 then the command use 2 from "a*"; instructs "much" to generate a random subset of the questions a1, a2, a3 of size 2. * create ; Set how many papers to create. Besides the random selection of questions matching each pattern specified in a "use" command, the set of questions that make up a test paper are also randomly permuted within the test and the answers to each question are randomly permuted among themselves. This command must come after the commands "seed", "serial" and "use". * scale ; This specifies that the resulting grades are to be rescaled to the interval 0 to . The default scale is 10. * grade from "" to "" ; Grade the answers in and put the grades in . The format of is described below in "Answers file". Both names are given in quotes (" ... "). A correct answer is always worth 1 unit, no answer is worth 0 and a wrong answer carries a negative weight, calculated so as to make random selection of answers have expected value 0. For example, if a question has 4 possible answers then a wrong answer costs -1/3 units. If only 2 answers are offered then a wrong answer costs -1 unit. The "grade" command must come after the "scale" command. In the test description file a comment is signified by the # character, until the end of the line. Answers file ------------ The file containing the answers given to the produced tests should contain one line per paper. Each such line should start with the paper serial number, be followed by a string containing the answers given and a description of the person who took the test in free format. The following is the file "papers" in directory "example": 506 B-ABBDCAC Knuth Donald CS 1535 525 ABAAADAAD Ullman Jeffrey CS 1541 500 ABBADACDB Cohen Paul MATH 1576 520 DAACCBDDC Katznelson Yitzhak MATH 1511 520 DBACCBDDC Papanicolaou George AMATH 1512 511 CDBBCCBAB Keller Joseph AMATH 1496 552 ABDABBBDA Karlin Samuel AMATH 287 A dash (-) in the answer string signifies that no answer was given. The resulting grades are the following (this is file "grades" from directory example): serial answers correct grade description 506 B-ABBDCAC BCAABDABA 2.96 Knuth Donald CS 1535 525 ABAAADAAD DDBABDDCC 0.00 Ullman Jeffrey CS 1541 500 ABBADACDB ADABCADDB 1.85 Cohen Paul MATH 1576 520 DAACCBDDC BBBCADDDB 0.00 Katznelson Yitzhak MATH 1511 520 DBACCBDDC BBBCADDDB 1.85 Papanicolaou George AMATH 1512 511 CDBBCCBAB DABBCDCBB 1.85 Keller Joseph AMATH 1496 552 ABDABBBDA ADCCBDAAA 0.37 Karlin Samuel AMATH 287 Generated files --------------- During the test creation phase "much" generates several files, all of whose names start with "mc-", except for the file "correct-answers.txt". * mc-logfile Always examine the contents of this file after both test creation and test grading, in case something went wrong. * mc-all-questions.tex All questions that may appear in one of the tests are listed here. This is a fragment of a TeX file and needs a wrapper file in order to be printed. This printout might be useful for a teacher to have with him during the administration of the test. * correct-answers.txt This file contains the correct answers for each serial number. Needed at the grading phase. DO NOT LOSE THIS FILE. If you lose it or destroy it you will not be able to grade the tests. Save a copy somewhere else. It may also be used in order to post the correct answers for each serial number (after the test, of course). This is the only generated file without the "mc-" prefix. This is intentional so that it does not get accidentally erased by a cleanup command like "rm mc-*". * mc-output.tex This is the TeX file (to be read in by the master TeX file) which contains all the examination papers. * mc-serials.txt This contains which questions, and with what order, where used for each serial number. Useful if something goes really wrong, but normally not used. The master TeX file ------------------- One should look in the directory "example", file "exam.tex" to see a very descriptive example/template for a master TeX file (written in LaTeX). In that the user must define certain macros which affect the looks of the papers to be generated: * \mcglobalheader : This macro will be called once only, before producing any of the tests. * \mcpaperheader : This macro will be called once before each test. * \mcpaperfooter : This macro will be called once after each test. * \mcquestionheader : This macro will be called once before each question. * \mcquestionfooter : This macro will be called once after each question. In doing this the user may count on the following macros to be defined by "much": * \mcserialnumber : This is equal to the serial number of the current paper being produced. It is typically used in the definition of \mcpaperheader to print serial number on the top of the page. * \mcquestionnumber : This is equal to the number of the current question in the paper. Typically used in the definition of \mcquestionheader. At the end the master TeX file must read in (using, say, \input mc-output.tex) the file with the generated tests. An example of a complete master TeX file is given in Appendix B below. Elementary precautions ---------------------- When using "much" to create many random tests drawn from a database of questions, according to the user's specifications, it is worth keeping in mind the following, so as to avoid finding yourself in a difficult position. (a) You do not want to have to grade manually the answered tests that you created. This will be much worse than grading as many tests which are all the same. Keep in mind that in order for "much" to be able to grade the tests for you all it needs is the file correct-answers.txt. If you lose this file you're on your own, so copy it somewhere else as a precaution. If you do lose it but you still have the test-description file that created the tests then all you have to do is recreate them and the file will be created anew, and will be the same as before, as the "randomness" is controlled by the random seed only. (b) If you are administering a test to a few hundred students with tests generated with "much", it is important to realize behorehand that you cannot answer a participant's question of the type "What do you mean by this in Question 3?". The reason is that you do not know which is Question 3 for this student. So it is a good idea to have (1) a printout with you with all possible questions as well as (2) a printout of the file "mc-serials.txt". This latter file contains for each serial number the names of the questions used so you can look them up in the first printout (1), after asking the student, of course, what is his serial number. To create the printout (1) you should create a wrapper file for file "mc-all-questions.tex". This could be as simple as \documentclass{article} \begin{document} \input mc-all-questions.tex \end{document} or a bit more complicated if you need to define macros or use packages. (c) It is even harder to correct an error during the test if you spot one. The reason is that you have no way of refering to a specific question. So prepare your database very carefully and make absolutely certain that for each question there is precisely one correct answer (which is listed first after the question). (d) (Not very likely to be a problem.) Try not to use the program on two different platforms for the same test. All precautions have been taken that the random number generator produces the same results starting from the same seed, no matter on what machine the program is run, but it is still possible that I have missed something. In particular I have not relied on the random number generator of the standard C library but have included a random number generator with the program distribution to be compiled with the program. Known limitations ----------------- The following are some limitations of the program. They should be corrected in later versions. (a) The program does not allow much flexibility in grading yet. In particular it is completely inflexible in defining the penalties for wrong answers. Also the user should be able to define non-linear grading schemes, such as those used to create "letter grades". (b) The user cannot specify that some randomness in the creation of the papers be skipped. For instance, a user might want to create tests where the ordering of the questions in the test is not random, but this is impossible right now. (c) The program does not do enough checking of the user given data. For example, it does not check that the letters used to describe the answers given to a test make sense. The program should not allow the user to specify 'C' as an aswer to a question with 2 options (only 'A' and 'B' are valid for this) yet all it does is to consider this 'C' as a wrong answer. Appendix A: An example of a test-description file ------------------------------------------------- The following are the contents of the file "test-description" in the directory "example" of the program distribution: ################################################## ################################################## ################################################## # This is a test description file for "much" # # To create the tests give the command # # much create file-name # # where file-name is the name of this file. # # Then compile the file wrapper latex file, # probably called "exam.tex" in this directory, # giving the command "latex exam". The ouput in # "exam.dvi" contains all generated tests. # # To grade the tests give the command # # much grade file-name # # Look at mc-logfile after each run for possible # errors. # ################################################## # All filenames will be relative to the directory # specified below, except for "mc-logfile" which # is always created in the current directory. directory "."; ################################################## # 1st part: describe the tests to be created # # # This part is ignored in grading mode # ################################################## # Set random seed # If you change this number the tests created # will be different. There is nothing wrong # with this, but then the answers in the file # "papers" will not make much sense and, after # grading, you will see too many zeros as grades. seed 1101; # Set first serial number # The serial number used will start with the number below # and will increase by 1. serial 500; # Define question types and how many # questions are required per type. # The second argument is a regular expression # which should not contain any quotes ("). # The first number denotes the number of questions # to be chosen from all files in the current directory # matching the corresponding regular expression. use 2 from "mult-*"; use 2 from "comb-*"; use 2 from "perm-*"; use 3 from "graph-*"; # Create a total of 100 papers create 100; ################################################## # 2nd part: how to grade the papers # # # # This part is ignored in creating mode # ################################################## # Set grading scale scale 10.0; # Grade the answers in file "papers" and put resulting # grades in file "grades.new" grade from "papers" to "grades"; ################################################## ### End of test-description file ################################################## Appendix B: An example of a TeX wrapper file -------------------------------------------- The following are the contents of the file "exam.tex" in the directory "example" of the program distribution. This example is written in LaTeX. Plain TeX or any other flavor of TeX would certainly be possible. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %%% %%% %%% Master latex wrapper file for generated file mc-output.tex %%% %%% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \documentclass[12pt,a4paper]{article} \usepackage{amsfonts,latexsym} %%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \oddsidemargin -0.5in \evensidemargin -0.5in \textwidth 7.5in \headheight -0.7in \topmargin -0.5cm \textheight=10.5in \footskip 0 in \parskip 0.10in %%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \begin{document} %%% Print no page numbers \pagestyle{empty} %%% No global header. This macro is called once at the beginning of the %%% batch of tests. \newcommand{\mcglobalheader}{ } %%% Print this on each test, before the questions \newcommand{\mcpaperheader}{ \ \\ Serial Number: {\bf\Large \mcserialnumber}, Answers: 1: 2: 3: 4: 5: 6: 7: 8: 9: \\ Name: \begin{center} {\sc University of Crete -- Department of Mathematics -- Discrete Mathematics I}\\ \ \\ Final examination \end{center} } %%% Print this on each test, after the questions \newcommand{\mcpaperfooter}{ \hrulefill \noindent {\small The examination lasts 2 hours and all books are closed. Return only this paper with your answers. Record the serial number of your paper and your answers on a piece of paper and keep it. Wrong answers reduce your score. Not answering a question counts as 0. There is precisely one correct answer per question. \noindent Instructor: Mihalis Kolountzakis\hfill Iraklio, 7 February 2004 \centerline{\bf RETURN THIS PAPER!} } %%% use two \newpage commands if you're printing a single page document on a duplex printer \newpage } %%% Print this before each question \newcommand{\mcquestionheader}{\noindent{\bf Question \mcquestionnumber}: } %%% Print this after each question \newcommand{\mcquestionfooter}{} %%% Now read in the actual tests \input mc-output.tex \end{document} %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %%% %%% %%% End of wrapper file %%% %%% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% Appendix C: An example of a question file ----------------------------------------- The following are the contents of the file "comb-binomial" in the directory "example" of the program distribution: The binomial coefficient ${n \choose k}$ equals . ${n \choose n-k}$. . $0$ if $k=0$. .