Evaluating Instructional Apps Using the App Checklist for Educators (ACE)

The use of iPads and apps has become common in K-12 inclusive classrooms. Special education teachers frequently use this tool to support instruction. Data from electronic surveys were used to determine criteria that teachers identified as important for choosing apps for classroom use. Using this information, the authors developed an App Checklist for Evaluators (ACE) to assist teachers in reviewing apps. Areas included on the ACE are: Student Interest, Design Features, Connection to the Curriculum, and Instructional Features. Suggestions for choosing an effective app include determining user needs, assessing student interest, evaluating design features of the app, documenting the connection to the curriculum, and identifying instructional features.


Background
Since the iPad was introduced to the public in 2010, the use of this device in the classroom setting has continued to grow (Perry, Thrasher, & Lee, 2016) and has become a preferred tool for special education teachers (Mautone, 2013).Overall, teachers have reported a positive instructional experience using iPads (Johnson, 2013).The iPad and its accompanying applications (apps) have become integral tools used to address many special education program objectives and Individualized Education Program (IEP) goals.Additionally, survey data have indicated that teachers' perceptions are that all students would respond positively to using iPads in the classroom (Johnson, Davies, & Thomas, 2013).
Researchers (Douglas, Wojcik, & Thompson, 2012) have demonstrated that apps can serve as supports for students with disabilities by providing efficient access as both an instructional tool and an assistive technology device.Further research has focused on the use of specific apps for students with autism and other developmental disorders (Cumming & Rodríguez, 2013).Apple iPad technology has had an impact on fostering new learning opportunities for students with disabilities by increasing engagement in learning, time on task, independent educational opportunities, and skill development (Baig, 2013;Flower, 2014;Rodríguez, Strnadová, & Cumming, 2014).Increased student engagement, motivation, and independence are apparent benefits of using iPad apps integrated into academic lessons (Baig, 2013;Flewitt, Kucirkova, & Messer, 2014;Johnson, D., 2013;Miller, Krockover, & Doughy, 2013).Maich and Hall (2016) provided suggestions for teachers using iPads in inclusive classroom settings, starting with planning for use and including using the data collection feature of some apps to support acquisition of IEP goals.However, these benefits can only be achieved if apps are chosen appropriately (Perry et al., 2016).Thus, overall, the use of apps in special education is emerging as an effective practice, and implementation in the classroom must be planned, intentional, and informed.
Since this type of technology (i.e., app use) is still relatively new, the rigor with which it is screened before being implemented as an instructional support is generally lacking.Authors (Newton & Dell, 2011;Powell, 2014) agree that apps should be chosen with a purpose in mind.The process of locating and evaluating apps can be difficult and time consuming (Perry et al., 2016).Teachers need a tool that allows them to evaluate iPad apps with relative ease, but will also guide them in effective use of the app with their students.Planning for the use of iPads and apps is a critical component of effective use in the classroom (Mautone, 2013).A rationale for the use of a specific app to individualize learning for a student with a disability should include consideration of how the app will meet the student's IEP goals.The choice of apps used for instruction must be based on more than a suggestion from another person, and more than an appealing advertisement.When used for instruction, they should not be chosen for entertainment value.Apps must meet specific predetermined academic criteria.It is becoming increasingly apparent that an evaluation tool or rubric should be used when choosing apps (Rodriguez et al., 2014).With new apps being released every day, lists of apps can become outdated quickly.This research was designed to gather data on the current use of apps in the classroom, to determine what teachers need in an evaluation tool, and then to create an effective iPad app evaluation scale based on those needs.

Development of the iPad App Evaluation Checklist for Educators
The first step in creating the scale involved collecting information on overall use of the iPad and apps by K-12 general and special education teachers.Principals in local school districts were contacted for permission to administer a survey to the classroom teachers in their schools.This online survey of 24 questions queried the way that devices and apps were currently being used by teachers in one east coast state -for individual students, whole class, whole school, or district-wide.Teachers were asked to define their main purpose of use of the iPad and apps -for organization, assistive technology, or content instruction.They listed the characteristics that they valued when choosing apps for content instruction, and identified favorite apps used in each content area.
After the initial pilot survey was completed, the questionnaire was revised for clarification purposes and emailed to the principals of all school districts in each county in one east coast state.Although some emails were returned as undeliverable, the majority were received and disseminated to the teachers.See Table 1 for a listing of responses to survey questions.Of the 151 completed surveys, about 88% (n=133) of teachers said that they were not familiar with any published tool that would help them evaluate an app prior to use in the classroom.The majority (n=127) of the teachers indicated that they worked with students eligible for special education services.The respondents were equally distributed between elementary and middle/secondary teaching assignments (n=67 elementary students; n=69 middle/high school teachers).Nearly 64% (n=70) reported that devices are used throughout their entire school district, not just in their individual classrooms.Content instruction (n = 52) was the primary stated reason for using the devices in the classroom.The predominant method for choosing apps was an online search (n=35) or a recommendation from another professional (n=35).About 63% (n=64) of teachers using apps would try the app themselves prior to use with students.Although most teachers (n=88) were not familiar with any published rating tools, nearly 80% responded that they felt it is important to have such a tool (n= 80).The respondents listed criteria for evaluation that are important for them in choosing an app for classroom or student implementation.Using data from this survey which was sent to all schools and districts in one east coast state, an initial evaluation tool, the App Checklist for Educators (ACE) was created and piloted with several groups of teachers.Revisions were made to the tool based on feedback from these groups.The responses from the teachers regarding the usefulness of ACE were overwhelmingly positive (see Table 2).Overall, they saw the benefit in having a tool to help them quickly rate the usefulness of the app and to provide guidance in choosing effective apps for classroom instruction.
Some comments referred to the ways apps are used with students.For example, several respondents noted that apps have entertainment features, but may not be truly educational.For instance, one respondent noted that teachers should "make sure that the app is accomplishing what you want the student to learn, that it is not just a game."Another added that teachers may mistakenly use an app because it is visually appealing or fun without determining its educational effectiveness.
Some comments denoted positive reactions to using a tool in general.One respondent stated that "while we are all encouraged as educators to utilize technology, it is important that we are evaluating how effective these technologies are for student development."Another respondent noted that these types of tools were a "useful resource for educational professionals."Similarly, another comment was that evaluation tools "help teachers make good choices," adding that they could be "useful for collaborative planning between families, IEP teams, and educators." The third type of comment was related to the specific tool (i.e., ACE) as compared to other evaluation tools.One respondent stated that the evaluation checklist was "really interesting because it opens up a variety of content, target skills, design features."She added, "The questions make you think, ok, does this app provide visual features that enhance a student learner?It's important to know because what may work for one student, may not work with another."Another respondent said that this tool covered more depth than other tools she had used, but was easier because the answers were straight forward.The categories of questions were also seen as a positive feature by one respondent who stated that "it really gets the teacher thinking about the overall quality of the app before he/she brings it into their classroom." Table 2. Sample Quotes submitted by teachers after using the tool.

Comment Type Quotes by Respondents
Using Apps in the Classroom Teachers may mistakenly use them because they are very appealing or students enjoy them.

Using an Evaluation Tool
These tools will tremendously help educators sift through the MANY apps that are out and it will shorten the time it takes for others to implement them.

Using ACE
I think that this form really allows the teacher to think about the app and how it would be beneficial to their students, although not all of the questions on the checklist apply to every app.On the other hand, the questions on this form are very specific which allows the teacher to target apps based on the skills that their students need.I also like that the form is universal across both grade level and subject.This makes it easier for teachers to complete the form, especially if they teach multiple grade levels.

Using ACE
I think that this could be a useful planning tool when trying to plan lessons using technology.I believe this evaluation tool helps teachers to make good choices about the apps they are using.This tool could also be useful for collaborative planning between families, IEP teams, and educators

Using ACE
I have never used one as in depth as the one provided.I also feel like this tool is easier to use than others because they have straight forward answers.What I like about this tool is that it can be adapted for any grade level or skills set.
Using ACE I like how the form has a section for content area and then the target skills section.I think this is very helpful because one app may be good for one target skill or one content are but not as good for another.
Overall, the comments strengthened the notion that apps are being used frequently, and respondents saw the benefit and need for an effective evaluation tool.They considered the App Checklist for Educators to be a useful device for accomplishing this goal.

Selecting an Effective App
As the use of technology increases, it helps if educators embrace the movement and find high quality technology to support students' learning in their classrooms.When finding an app for classroom use, it is important to first go to the "App Store" on the device.Once there, teachers can search in the toolbar by content area to find apps that are available.It is important as teachers plan to implement an app within the classroom that there is an evaluation of the app prior to use.Many apps are created but not necessarily evaluated by professionals in the field of education for purpose, age range, content area, targeted skills, student interest, design features, connection to the curriculum, or instructional features.
The following five basic steps, based on the App Checklist for Educators, can be used for consideration in choosing effective apps for classroom use: Step 1: The first step is to evaluate the age range, cost, content area, and targeted skills.Typically this is found when teachers click on the app within the "App Store," it provides them with the "details," "reviews," and other "related" apps.Scrolling down in the "details" section will allow the teacher to find a description of the tool that provides some of this information.Teachers should be aware that most of the time the user is required to purchase the app to find out more information.
Step 2: The second step is to assess the student's interest.
Depending on the least restrictive environment, this may be targeted for a whole class, a smaller group of students, or on an individual basis.When evaluating the interest level, it is important to note the ease of use.It is also important to consider if this app is engaging for the student's developmental level.The third area to assess is if it increases student interest in a topic.The teacher should reflect on the following question, "Are students asking questions about the content area after using the app?"The final step in the evaluation of student interest is documenting if students want or ask to use the app again.This is a critical component to help make a determination of the interest level for the app.
Step 3: The third step for consideration of an app is to evaluate the design features.It is important when looking at an app to assess the design features for students with disabilities because the app needs to have a clear and consistent layout.If the app includes graphics, it is important to note if these correspond with the activity to enhance student learning.Another critical factor is identifying the different types of devices that the app is compatible with because the student may have a different device at home or in another classroom and it will help with generalization of the skill if it is compatible with many devices.Some apps have support available but this can sometimes be difficult to find.Teachers should be sure when looking for the next design feature of technology support that they search the tool bar or icons within the app.Typically, they will find contact for the support through an e-mail address or the app.Education has become more data-driven, so it is crucial to evaluate if it provides students with feedback.The feedback can be immediate or delivered at the end of each level or activity.While providing feedback to the student, it is also important to note if educators can have access to the students' performance or data.Sometime this can be shared if the teacher has an account, or by having the students e-mail the data upon completion.There are many apps that still do not have access to the data, so teachers need to be aware that this may not be an option with the app being evaluated.Another question about data collection that is a critical feature is to make sure to note if the data is able to be collected over time.In almost all cases, students may need to set up an account to collect the data if it is available within the app.A final design feature that is critical to note is the affordability of the app.
Funding for apps can be requested through multiple sources (e.g., teacher, family, principal, district) so it is important to compare it to other apps to determine if it is reasonably priced.
Step 4: The fourth step of app evaluation is to document the connection to the curriculum.Common Core (CC) has been adopted by the U.S. Department of Education (State of New Jersey Department of Education, 2010), so the app should be connected in some way to the content area and grade level of the appropriate CC standards.The next important factor to note is if the content can be matched to the student's skill level.Students within a class may be at many different skill levels and it is important that the app is able to differentiate among the various needs of the students.For students with disabilities, it is critical to note if it can align with their Individualized Education Plan (IEP) goals.This way teachers can continue to have the students practice their skills while using the app that is connected to their specific goals.Another factor to consider is if it connects to the real world so that students can reinforce and practice skills that they will use in the future.Finally, when thinking about the connection to the curriculum, identifying whether or not the app improves students' academic skills or critical thinking skills is the last key element.
Step 5: The final components to assess when evaluating an app for educational purposes are the instructional features.First, a teacher should identify whether it requires students to memorize facts.This is important to note because depending on the purpose for the app this may support their learning, or it may not be appropriate.To deepen student learning, a teacher can evaluate whether or not the app has the students explain their ideas or concepts, apply their information to various situations, make connections among the concepts, or create original work.Those are all higher level skills that are important to take note of because this could potentially help support the students make connections on a more meaningful level.
Upon completion of the checklist, it is important to note how many of the questions received a response of "yes" compared to how many were answered "no."When counting responses, a total of 23-25 "yes" answers would receive a rating of 5 stars.If the app had 20-22 "yes" responses, then the teacher would rate it 4 stars.If the rating was 18-19 "yes" answers, then it would be rated with 3 stars.Two stars are earned for between 15-17 "yes" responses.Finally, if it received fewer than 14 "yes" answers, then this is a 1 star.To determine if a teacher should recommend this to another teacher, family, or district, a rating of a 4 or 5 stars would be an app that should be recommended.The teacher would check the "yes" box next to "Would you recommend this app to other professionals?"If it has been rated 3 stars or below then this app is not a recommended app to use for educational purposes and the teacher would check "no" (See Figure 1: App Checklist for Educators).Other areas of academic interest or individual need can be included in the section for "additional comments."Teachers responding to the survey indicated that they tend to use apps that are recommended by other teachers or other professionals.It is important when these tools are suggested that they are still evaluated to ensure that the app is appropriate for the specified academic purposes and student's needs.Based on teacher reported data on most frequently used apps from teachers, three apps were evaluated using the App Evaluation Checklist for Educators developed as part of this project.The three apps were chosen based on frequent recommendations in the survey.Fifteen teachers in a graduate special education course completed the checklist for each of the three apps.
The final section of the checklist allows teachers to provide additional comments.Since multiple teachers assessed the app using ACE, averages are listed in Table 3, along with comments submitted.Teachers evaluated the apps for appropriateness for their specific grade level, for an individual student, and/or for their classroom.Thus, a score may have an average based on the level to which it is deemed effective for an individual student or a specific group of students.This illustrates the need for this type of evaluation checklist for each classroom, or for individual students, where a teacher can determine the effectiveness for their own class and setting.The App Checklist for Educators enables teachers to determine the effectiveness of an app for their own educational needs.The apps listed in Table 3 had overall high ratings by all teachers using ACE.
The following is a summary of three apps rated highly by the 15 teachers surveyed.In the initial survey, teachers were asked to list apps frequently used in their classrooms.From these suggestions, lists were generated of those listed by at least three participants.In the development of the checklist, these apps were evaluated by teachers.
According to the published information, all are aligned with academic standards, which are searchable.This app was listed by many of the respondents in the survey, and ranked with high stars when using ACE.
Using Mathmateer (2010Mathmateer ( -2013)), students can build their own rockets by earning money by doing math problems.
The rocket is launched into space when complete.While in the space mode, students complete different math missions based on various math skills, including number sense, fractions, decimals, counting, time, money, shapes, computation, etc.This app was familiar to many of the respondents, and received high stars by those using ACE.

Final Thoughts
Since the iPad was introduced in the classroom, there has been an increase in the usage of educational apps.There are a variety of reasons why teachers choose to use educational apps.It is critical that teachers evaluate apps prior to use so that they choose the most effective instructional support for their students.ACE is a useful, easy to use, research-based checklist that supports teachers in this process and helps them to evaluate apps for today's classrooms.

Figure 1 .
Figure 1.App Checklist for Educators (ACE) Evaluation Tool with Specific Apps that Were Recommended by Teachers

Table 1 .
Responses to Survey Questions 50 Are you familiar with any published evaluation tools for educational APPs for students with special needs?Yes = 18 (16.98%)No = 88 (83.02%) 45 How important is it for you to have an evaluation tool to rate APPs that is easy to use? Very Important/Important = 31 (30.39%)Somewhat Important = 49 (48.04%)Not Important = 22 (21.57%)

Table 3 .
Example of Overall Scores from the App Evaluation for Educators Checklist Makes reading fun through songs and characters.Good supplement for instruction.Well-organized.BrainPop Jr. (and ESL) 4-5 Stars ESL students loved the ESL version.