Newsletter – Spring 2014
Submissions open for the 15th Annual Conference, 6 – 8 November, 2014, Tallinn
Tallinn is a beautiful and interesting historic city, with a long tradition of academic achievement. Currently, Estonia sees the electronic future as a strategic priority for the economy and this makes for a very vibrant climate. The conference hotel, with its many restaurants, cafés and bars, is an excellent meeting place for the networking that goes on around the conference. Additionally, the hotel is a short stroll from Tallinn’s old town, with its famous cobbled streets and red roofs.
The call opened for submissions to this year’s annual conference in mid-February. The closing date for paper presentations, discussion group and poster presentation proposals is 30th April. We had an unprecedented number of submissions in 2013 and attendance at the conference was at record levels, with over 260 participants. Feedback indicated that the quality of the programme is increasingly strong.
We are trying out an electronic submission system called Easychair. For instructions on how to submit, see Tallinn conference submissions page. Several people have submitted already and it is pretty straightforward to use. As Chair of the Scientific Programme Committee this year, I look forward to reading the submissions for 2014. My colleagues on the Scientific Programme Committee are
- Gabriella Agrusti (University of Roma Tre, Council Member of AEA-Europe – Italy)
- Louise Hayward (University of Glasgow – UK)
- Aimi Püüa (Foundation Innove – Tallinn)
- Henk Moelands (Cito and Treasurer of AEA-Europe; Netherlands)
- Chris Wheadon (Consultant; UK)
Kathleen Tattersall New Assessment Researcher Award
The call has also been opened for the 8th New Assessment Researcher Award competition. This award, sponsored by AQA, has attracted excellent submissions and the winners have presented some outstanding research at our conferences that has provoked the more experienced researchers to think differently about our field. We look forward to receiving similarly high-standard applications this year. The closing date is 30th April. For details of how to apply, see New Assessment Researcher award.
Forthcoming Council Vacancies
In accordance with our constitution, a Nominations Committee has been appointed to canvass members for nominations to our two Council vacancies arising in 2014. Members of the Nominations Committee are Frans Kleintjes (Cito, Netherlands), Gordon Stobart (Institute of Education, England) and Christina Wikström (University of Umea, Sweden). Please do contact them if you would like to stand for election as a member of the Council or for the Vice Presidency position. Alternatively, if you would like to put forward a name for their consideration, please do get in touch with them directly. The Nominations Committee is due to report to the Council Meeting on 8 May.
At the General Assembly in Paris in November, it was agreed that the current Executive Secretary, Sarah Maughan, and Treasurer, Henk Moelands, should be asked to continue in post for another term.
The future of the Newsletter
This is the 12th edition of our Newsletter and the Publications Committee has taken the strategic decision that they would like to build upon the success so far and publish special issues of the newsletter. As such, there will be a call later in the year for proposals for a thematic edition of the AEA-Europe Newsletter. Thematic proposals relating to this year’s conference theme would be particularly welcomed, but the committee would be open to other suggestions. Look out for future announcements!
Work in Progress
Supporting Lifelong learning
In the 21st century there is a plethora of information available in the knowledge society, much of it on a variety of devices and in multimedia formats. This can have a positive impact on learning but learners need to be digitally competent in order to benefit fully – they need to be able to retrieve information and assess its suitability if it is to have a positive impact on learning outcomes.
The development of digital competence can aid educational progression: those who have these skills are more likely to make the transition from school to university and maintain a pattern of lifelong learning through independent study. They are also less likely to drop out of education altogether.
What is happening
This project aims to develop a learner-focused approach to develop, consolidate and secure learners’ competencies in retrieving and selecting text to read or data to analyse.
Three countries will trial an innovative e-learning management system intended to:
- Develop key information processing skills for ICT (literacy, numeracy and problem-solving) in young adults aged 16-24; the development of these skills will use an inquiry-based approach to learning and focus on young adults who are low achievers educationally;
- Produce a high level of personalization in learning, based on:
- automated computer-based assessment (CBA) and computer adaptive testing (CAT)
- an innovative way of delivering learning materials – modulating texts automatically to reduce reading comprehension difficulties.
The learning content management system will be delivered in four languages (English, Italian, Portuguese and Norwegian) and will provide information-centred courses to young people in upper secondary school, on undergraduate programmes and those who are unemployed.
Why inform members?
AEA-Europe wishes to promote projects that link researchers in different countries; this project is an example of such collaboration, linking universities in Italy, Norway, Portugal and the UK. The photograph shows the research group – with many AEA-Europe familiar faces!
The development of 21st century skills in the population is vital to the future of economies in Europe – and, indeed, worldwide. This project aims to contribute to this by attracting social groups that do not traditionally engage in formal training; if this is achieved, the project will help to meet the Five EU benchmarks for 2020 by reducing the percentage of young low achievers in Europe and increasing the percentage of young adults taking part in lifelong learning initiatives. In addition, the project aims to develop tools that will help the most disadvantaged tools attain the skills needed for employability, personal development and civic participation.
Project Ref. No: 543058-LLP-1-IT-KA3-KA3MP
Funded with support from the European Commission
Building a spoken learner corpus
Left to right: Tony McEnery,Faculty Dean, Arts and Social Sciences, Lancaster University; Elaine Boyd, Senior Academic, English Language Testing and Assessment, Trinity College, London and Vaclav Brezina, Senior Research Associate
Following successful feasibility studies conducted in 2011 and 2012, Trinity College London is working on a 5-year-project to create a corpus of spoken learner English. This project is in collaboration with the Centre for Corpus Approaches to Social Science (CASS) directed by Professor Tony McEnery. CASS is a new centre of excellence funded by the Economic and Social Research Council (ESCR) based at Lancaster University. The centre aims to bring the latest techniques in corpus analysis to bear on a range of questions in the social sciences. Lancaster University has been pioneering research in corpus linguistics for over 40 years. The new centre will involve different faculties at Lancaster, promoting a multi-disciplinary approach to language issues in society, industry and government. The centre views the creation of a Trinity learner corpus of spoken English as a development that would benefit not only Trinity but also the greater academic community.
Spoken learner corpora are few and far between; corpora of examiner language are, as yet, non-existent. The development of a corpus of spoken learner – and examiner – language is seen as an opportunity for Trinity to mine its rich data sources. Trinity has a wealth of data to offer as a result of its policy of audio recording all its speaking exams.
It is intended that the Trinity Lancaster Corpus will be a resource for:
- researching into:
- the communicative competence in English at different proficiency levels
- the characteristics of different non-native varieties of English
- features of intercultural communication, etc.
- refining Trinity tests
- devising support material
- investigating areas that will benefit the wider teaching, testing and publishing community
What is happening?
The project started in the Autumn of 2013 and we have spent some months ensuring robust quality processes and review of data entry. Currently, the project is in the corpus building stage, entering data at B1 level and upwards. This data covers a variety of first languages and learner ages as well as the examiner data from within the tests. To date, the corpus has approximately 1,000,000 words and work is on the next stage of building by a further 500,000.
Although the data is all responses from within the tests, it is seen as particularly valuable because Trinity’s spoken exams allow tasks where candidates choose what they wish to talk about which allows them to use language in a way that is natural to them. They speak for extended turns especially for presentations as well as engaging in spontaneous conversation. The pilot indicated that the data would be very useful for researching learners’ facility and progress in discourse features – an area that has so far received only limited attention in the research literature.
Why inform members?
Eventually it is intended that the corpus will be freely available to researchers. Trinity and CASS will keep the academic community posted on what is happening and when the data might be available for research. This is initially likely to be via specific requests to work with the ‘in progress’ version.
For further information please contact firstname.lastname@example.org
Developing a toolkit for assessing children’s spoken language skills
Left to right: Neil Mercer, Paul Warwick and Ayesha Ahmed
What are the first language speaking and listening skills that schools should be helping children to develop? How can teachers monitor and assess these oracy skills in a classroom setting?
The Cambridge Oracy Assessment Project is funded by the Educational Endowment Foundation and is being carried out in partnership with School 21 in London. Our aim is to research and develop a toolkit for teachers to use in Year 7 classrooms (age 11-12) to assess children’s oracy skills. One designated application is to provide a measure of the effectiveness of the oracy curriculum in School 21. We hope, though, that it will have much wider use when fully developed as a formative assessment toolkit.
What is happening
The toolkit will enable teachers to assess students’ skills in using spoken English, across a range of contexts, as they arrive in secondary school at the start of year 7. Teachers will then be able to plan any support or teaching needed to help children to develop those skills. The tasks can be used at any point during the school year, to monitor how well students are progressing on these skills and to plan for teaching and learning.
The toolkit will consist of several linked items:
- a set of initial oracy tasks – one individual, one paired and one group talk task;
- a set of curriculum embedded task outlines which teachers can adapt as needed;
- a set of end of year tasks – again, one individual, one paired and one group talk task;
- a system for assessing performance on the tasks and giving feedback to the children;
- a system for self and peer assessment.
The assessment scheme is skills-based, rather than context-based, and is grounded in our skills framework for oracy which we have developed using existing resources and research, and in consultation with a range of academic and professional experts.
Students are rated on a three-band scheme for each of the skills relevant for each task, and this information can be used by the teacher to get a picture of the skills profiles of individual students and of a class of students. Future teaching can then be planned accordingly.
We are currently trialling the tasks and assessment schemes with a small sample of children in three schools, including School 21, and collecting video exemplars of performance at different levels. These videos, along with commentaries, will form our performance level descriptors. We are also using paired-comparisons of videos in order to create rank orders of performances to help validate our scale. As part of the task trialling we are interviewing both teachers and children about their experiences using the tasks.
Why inform members
Assessing oracy is challenging and for this reason it is often side-lined. We are not so concerned with the set of skills that are assessed in second language learning, in which the emphasis is often on grammatical structures and essential vocabulary. Instead we are more concerned with situated language skills, adapted and used in a variety of authentic contexts over an extended period of time. Situated talk is hard to capture, yet it is critical that we have the means of monitoring children’s progress in the oracy skills that are so important for learning and for life.
We hope that by informing members about this project we will encourage others to consider the importance of spoken language and its assessment.
For further information please contact Ayesha Ahmed at:
The Tech Bacc: Providing excellence in vocational education
Daniel Acquah and Debra Malpass from The AQA Centre for Education Research and Practice
England is witnessing renewed interest in vocational educational and training (VET). Part of this interest is due to the ‘Raising of the Participation Age’ – from 2015, all young people in England must stay in some form of education or training until eighteen, though not necessarily full-time. The high profile Wolf Review of vocational education identified that many post-16 vocational qualifications are poorly designed with little labour market value. Thus there is real impetus for the creation of a strong VET route to ensure progression routes for all learners, as well as to contribute towards wider social goals of equity and inclusion.
Political discourse around VET has focused on a ‘Technical Baccalaureate’ (Tech Bacc). However, beyond very general principles, specific details of the curriculum and assessment are yet to be worked out in detail and so the goal of our work is to explore these issues, by drawing on a range of research evidence.
What is happening
It is likely that the Tech Bacc will be a ‘qualification wrapper’ rather than a true baccalaureate. A qualification wrapper involves shaping the curriculum by specifying combinations of existing single subject qualifications that students must follow. In the current English context, they are more appealing than ‘true’ baccalaureates, as they do not require wholesale reform of the curriculum or exams system.
The major component of the Tech Bacc would be a technical qualification. We focus especially on the features of the technical qualification that would ensure that they stand up to the best VET programmes internationally. This includes ensuring that the qualification provides the learner with an understanding of the theoretical underpinnings of the industry in question. We also review evidence on the benefits of including regular work experience as part of the Tech Bacc.
Our reading of the international literature on VET leads us to conclude that pursuing a vocational pathway should not be at the expense of continuing with general academic education. Indeed we review evidence from England that suggests that students who have pursued a mix of vocational and academic qualifications have more opportunities to progress to higher education than students who have pursued a vocational only pathway.
In exploring the assessment issues, we draw heavily upon our organisation’s own research. This includes modelling work carried out in preparation for the Diploma (a vocationally related qualification, now discontinued). We look at issues such as whether the results of individual qualifications making up the Tech Bacc should be reported separately, or whether it is meaningful to aggregate the results together to produce an overall grade. We also explore different ways of reporting achievement specifically in the technical qualification: using grades or via pass/fail competency. An equally important issue is how to set and maintain standards for the technical qualification within the Tech Bacc.
Why inform members
Most discussions of the Tech Bacc in England, and of VET in general, tend to focus mainly on subject content. Yet an equally important issue is how to assess the young person’s knowledge and skills and how to report the results, so our paper makes an important contribution to the debate.
It would be very beneficial to discuss this work with researchers from Europe who have experience in implementing VET programmes, especially around issues to do with assessment methods for VET.
For further information please contact:
Masters Research in Progress
“Investigating Washback: Teaching and Learning Realities of an English Oral Addition to the French National Secondary School Exiting Exam”
Gemma Bellhouse, Oxford University. Reading for an MSc in Applied Linguistics & Second Language Acquisition
Background and rationale:
Assessment is a crucial research field of Applied Linguistics in that every classroom, teacher, and learner is affected by language tests. This research project will explore possible positive and negative effects of the French Ministry of Education oral test to the secondary school exiting exam in 2013. While I believe this study is important as a reflection of the effect of testing on the implication process of communicative teaching in the classroom, it also analyses the driving learning strategies at an individual student level. The design is mixed methods using teacher/student (and other) interviews to further strengthen the information gathered from student surveys. This study is pertinent as the test is new and the teachers are presently adjusting their methodologies. The results of the dissertation should prove the advantages and disadvantages to national tests as per their effect on the microlevel, going as far as learner behaviour outside the classroom.
The participants are 7 teachers and their 9 classes of approximately 220 students. The students are 18 years old and in their final year of secondary school. The teachers have a range of 10-33 years teaching English experience. The 4 schools are located in Southern France in the cities of Rodez, Coliers, and Toulouse. I am going to collect data and conduct interviews mid-April.
The first stage of the study includes teacher questionnaires to investigate their teaching adjustments this year and their attitudes towards the test and also questionnaires given to the students to investigate changes in their attitudes/behaviours. This stage also includes the administering of an adapted SILL student strategy battery attached to each student questionnaire. The results of the student attitude questionnaires and the SILL battery (coded and grouped into types of strategies such as ‘cognitive’ and ‘motivational’) will be analysed for possible correlations as will be the student questionnaires with the teacher questionnaires, forming the quantitative section of the data analysis. Other variables that may provide significant correlation with the student questionnaires and strategies include the teachers, the schools, the students’ language proficiency and the type of English class.
For the second stage, I will analyse any available classroom data (such as oral recordings from language laboratories, mock exams prepared by the teacher, and relevant activities). These oral recordings and paper exams/homework will be analysed qualitatively for similarities with the actual exam in question and possible testing preparation strategy behaviour to provide supplementary support to the quantitative data.
The third stage includes possible further semi-structured interviews with all of the teachers and any student volunteers; questions will be based solely on the data gathered by the questionnaires.
1. What are the effects of the oral test on teacher and student attitudes and behavior (inside and outside the classroom)?
2. Do the attitudes (including awareness of the intentions of the test) and behaviors of the teachers correlate with that of the students?
3. What is the relationship between student strategy use and attitude/behaviour toward the test?
4. Are teachers aware of the intentions/criteria of the test, and if so, how are they trained/prepared?
Tests have always been a major motivator for teaching and learning and a powerful guide of content and form in the classroom. Washback studies are a prime research device to investigate these benefits as well as to provide a counter-test to the test itself—did it indeed accomplish what the test designers and policy-makers intend? I hope to answer this question and provide AEA with insights into the immediate effects of a high-stakes test.
Invitation to join an IEA Study
IEA has recently launched several new studies on a range of topics, including civic and citizenship education, early childhood education, and online reading.
- The IEA International Civic and Citizenship Education Study (ICCS) 2016 is the second cycle in IEA’s comparative research program investigating how young people are prepared to undertake their roles as citizens. ICCS will report on students’ knowledge and understanding of concepts and issues related to civics and citizenship, as well as their beliefs, attitudes, and behaviors.
- The IEA Early Childhood Education Study (ECES) is a cross-national study that aims to explore, describe, and analyze the provision of early childhood education across countries and how it contributes to children’s outcomes. ECES will provide a framework for countries to benchmark their early childhood education systems in an international context.
- As a new extension to the IEA Progress in International Reading Literacy Study (PIRLS) 2016, ePIRLS is an innovative assessment of online reading, making it possible for countries to assess how successful they are in preparing fourth grade students to read, comprehend, and interpret online information. ePIRLS will use an engaging, simulated Internet environment with authentic school-like assignments about science and social studies topics to measure achievement in reading for informational purposes.
Both IEA member and non-member countries are welcome to join IEA studies. For country enrollment information, please contact Dr. Paulína Koršňáková (email@example.com) at the IEA Secretariat as soon as possible.
International Association of Computerized Adaptive Testing (IACAT) young researcher award
Application deadline: 1 May 2014
The IACAT young researcher award is a prestigious award intended for young scholars (under 35 years) with an interest in computerized adaptive testing (CAT) and having obtained at least a Master’s degree or is in an advanced stage towards completing it. The main aim of the award is to encourage research in CAT. The recipient will receive a medal from the sponsor, EPEC Pty Ltd, (www.epecat.com) as well as a grant to help enable the recipient to attend the IACAT conference and he recipient will be invited to present a paper during a session at the IACAT conference.
To nominate a candidate, submit the following to Prof John Barnard (John@EPECat.com) before 1 May 2014:
- A recommendation letter from the nominator.
- A detailed curriculum vitae of the nominee with special reference to CAT.
- An abstract of up to 500 words on a topic in the field of CAT intended to be submitted for the invited address at the IACAT conference.
The recipient will be notified by 31 May 2014 and needs to confirm acceptance by 15 June 2014. The recipient will have to make their own travel arrangements and confirm the invited plenary address at the conference.
TIMSS and PIRLS 2011: Relationships Among Reading, Mathematics, and Science Achievement at the Fourth Grade—Implications for Early Learning
This publication presents research based on data from 34 countries and 3 benchmarking entities that participated in the 2011 cycles of the IEA Trends in International Mathematics and Science Study (TIMSS) and IEA Progress in International Reading Literacy Study (PIRLS) with the same fourth grade students. The coincidence of TIMSS and PIRLS in 2011 offered a unique opportunity to study relationships in educational achievement across three core curricular areas—mathematics, science, and reading—and to investigate the home and school characteristics that influence early learning.
This report includes four analyses:
- profiles of fourth grade achievement in reading, mathematics, and science for each of the participating countries and systems;
- investigation of the relationship between reading ability and the reading demands of the TIMSS 2011 fourth grade mathematics and science achievement items;
- school effectiveness study to analyze the relationships between school factors and achievement, after controlling for the influence of students’ home backgrounds;
- investigation of the influence of parental education and gender on achievement, and the mechanisms that mediate these relationships.
The profiles shown in this report of the percentages of fourth grade students reaching high and basic levels of achievement help to situate countries’ relative performance in reading, mathematics, and science. Most countries were relatively more successful in one or two of these subjects, especially at the higher levels. The report’s findings also support the idea that reading ability is associated with mathematics and science achievement: Greater reading demands made the fourth grade TIMSS items more challenging for weaker readers, although this varied across countries. Schools that were safe and orderly, supported academic success, and provided engaging instruction were considered to have several important characteristics for effectiveness. A home environment supportive of educational attainment was also shown to be important and interestingly, a stronger emphasis on early literacy than on numeracy activities seemed to be associated with both the level of children’s literacy and numeracy skills when entering school and their fourth grade achievement.
April 3 – 7, 2014, Philadelphia, Pennsylvania
April 10 – 14, 2014, Chicago
May 8 – 9, 2014, Singapore
May 25 – 30, 2014, Singapore
June 16 – 19, 2014, Nova Scotia, Canada
August 18 – 19, 2014, Bangkok, Thailand
Sept 24 – 26, 2014, Budapest, Hungary
8 – 10 October 2014 ETS, Princeton, New Jersey, USA
Nov 6 – 8, 2014, Tallinn, Estonia
June 22 – 26, 2015 Cape Town, South Africa
May 5 – 9, 2014
This course is aimed at doctoral students and postdoctoral fellows. The one‐week spring school is the fourth in a series of spring schools on advanced methods in educational research. The spring school is initiated by EERA and organized by the Universities of Cyprus, Dortmund, Gothenburg, Oslo, and Oxford. The fourth spring school’s topic is Propensity Score Matching & Instrumental Variables. Elisabeth A. Stuart (Johns Hopkins University) will deliver the main workshops. From May 5–9, 2014, the spring school will be hosted by the Centre for Educational Measurement at University of Oslo (CEMO) in Norway. The registration fee of € 200 includes full tuition and accommodation.
Contributions & Deadlines
Publications committee members: Gabriella Agrusti, Sandra Johnson, Newman Burdett, Anastassia Voronina, Julie Sewell (newsletter editor)
The AEA-Europe Publications and Communications Committee would like to make a call for contributions for the Association’s newsletter. The Newsletter is published twice a year, in the Spring and Autumn, with deadline dates for the next Autumn Newsletter being 20th August 2014. We would like help from members to make the information as up to date and relevant as possible. In particular we would like the following:
- Articles for the Work in progress section. These should be a maximum of 500 words long and be formatted under the headings “Context”, “What is happening”, “Why inform members” and “More information” (brief details of a website or references). Please see previous newsletters for more information.
- Articles for the Doctoral students’ Work in progress section. These should also be 500-600 words long and follow a similar format, covering the research and the context, with some details of methodology, potential impacts or directions.
- What’s new section: Information on conferences and courses at master and PhD-level. For conferences we need to know: Title, date, place, website address. For courses at master and PhD-level we would like information on courses relating to assessment open to international students, bearing in mind any language constraints. We will need the name of the institution, the title for the course in your own language as well as in English, plus dates, town, country, website or a contact address.
- Members’ news: any information on new appointments, promotions, etc.
We would also like some additional AEA-Europe national representatives who would be willing to send information about recent developments in assessment in their countries and approach relevant people to contribute to the Work in progress section. Please let Julie know if you would be interested in this role. We already have some volunteers but need more!
Julie Sewell (on behalf of the Publications and Communications Committee)
Please send all information to Julie Sewell: firstname.lastname@example.org