Modified SNOW method

Results

The most in-depth review used a procedure developed at the University of Toronto Special Needs Opportunity Windows (SNOW; Gay, Harrison, Richards, and Treviranus, 1999). SNOW is a provider of information, resources, and reports for educators and parents of special needs children. They developed a procedure in 1999 to determine the accessibility of LMSs. At the time, SNOW evaluated seven LMS packages, including Blackboard 3.0, WebCT, and Web Course in a Box (WCB). WCB has since been purchased by Blackboard and phased out. We did not have access to the other packages, which are not supported by the UW System.

The SNOW procedure has three parts consisting of a series of accessibility-related checkpoints. Inherent Courseware Features (ICF) scores features that automatically appear in the LMS. It has 61 items each for student view (ICF-Student) and for instructor view (ICF-Instructor). Designer Controlled Utilities and Functions (DCUF) scores information or elements entered or chosen by the instructor; it has 32 items. Courseware Components (CC) scores specific areas, such as home page, content area, bulletin board, etc.; it has 35 total items.

Inherent Courseware Features (ICF) includes navigation areas and other items predetermined by the software developers. The instructor cannot change these, except in very minor ways. The Designer Controlled Utilities and Functions (DCUF) area focuses on those aspects of the program that the instructor (designer) can add to or change. This would include such things as uploading content or other materials and setting course colors or images.

Each of the 61 items in ICF and the 31 items in DCUF relates to a specific W3C/WAI guideline. Each item is assigned a priority factor, 3 for Priority 1, 2 for Priority 2, and 1 for Priority 3. For example, an item is "Provide alternative text for all images." This relates to a Priority 1 guideline, so its priority factor is 3.

During the software review, each item in ICF and DCUF is given a base score for Access Support on a scale of 0-4. For the ICF, 0 indicates that the feature is not present in the program. A score of 4 indicates that all instances of the item are accessible, 3 and 2 indicate that some proportion of the instances are accessible, and 1 indicates that no instances are accessible.

For the DCUF, an Access Support score of 0 indicates that the feature is not supported in the software. 1 indicates that the feature is supported but the designer (instructor) cannot make it accessible. 2 indicates that the program provides accessible authoring practices for this feature. 3 indicates that the accessibility practice is also conspicuous. 4 indicates that the practice is not only conspicuous but also required or prompted by the program.

For the "Provide alternative text for all images" item, 1 would be given if images can be uploaded, but the instructor cannot add alternate ("alt") text. 2 indicates images can be provided with alt text, but it is not obvious how to include such information. A 3 might be given if the place for adding alt text is directly below the place where the image is uploaded. To receive a 4, the instructor cannot include an image without a reminder to include the alternate text.

For each item in ICF and DCUF, a factored score is computed by multiplying the Access Support score by the priority factor, to give more weight to higher priority items.

In addition to accessibility-related scores, a Functionality score of 0 or 4 is given to each item in ICF and DCUF. This score is 4 if the feature associated with the accessibility practice exists in the software; 0 if the item does not. For instance, if the program allows images to be placed, the functionality score for "Provide alternative text for all images" is 4 whether or not the program allows alternate text to be written. If the instructor cannot include applets, the functionality score for items relating to applets is 0. This score is important so that if one program allows for more features, it does not receive a larger score simply by having more features available. Thus, it evens the playing field so that the actual accessibility of what is in the program is compared. On the other hand, both are important because high accessibility with low functionality is not worth much to an instructor.

These first two sections of the SNOW procedure, ICF and DCUF, also include a similar scoring system for accessibility information provided in the LMS or its help pages. An Information score is given to each of the items on the ICF and DCUF. An Information score of 0 indicates no information is provided about making that type of item accessible. 1 indicates that it is mentioned in the help system. 2 indicates that an explanation of this type of accessibility is provided. 3 indicates that examples of making the item accessible are given. 4 indicates context-sensitive prompts are provided within the instructor entry areas.

Using the alt text example again, a 0 is given if alt text is not mentioned in the program or its help pages. If alt text is mentioned in the section on how to upload images, it would receive a 1. A further explanation of how and why to use alt text would garner a 2. 3 would be given for showing an example of alt text for a specific image. 4 would be given if information was in the program on the same page as the alt text entry box.

We slightly modified this scoring sequence from the original procedure. The original procedure has prompts as the additional item for a score of 2. Although prompts are important, we thought this aspect was more similar to the score 4 than to the score 2 used in the rest of the SNOW procedure.

The third section of the SNOW procedure, Courseware Components (CC), scores specific course areas: home page, tools, content, navigation, bulletin board, internal email, and chat. Each area has five or six specific items related to checkpoints. Separate scores are given for each item for the parts that are inherent and the parts that can be controlled by the designer. Each item is rated on a scale of 1-4, with 0 indicating the aspect does not appear in the program. A score of 4 indicates that all instances of the aspect are accessible, 2 and 3 indicate proportions, while 1 indicates that the aspect is never accessible. These numbers are based on results from ICF and DCUF. However, this section essentially asks, "To what extent is the content accessible by with regards to alt test? To what extent is the navigation accessible with keyboard equivalents?" The ICF and DCUF asked "How much of the entire program is accessible with regards to alt text? How much is accessible with keyboard equivalents?"

The Total Accessibility score of a program is the sum of all scores for Access Support, Information, and Courseware Components. You can also add the DCUF and the designer controlled portion of the CC score to get the accessibility of just the designer controlled utilities. Similarly, the overall inherent accessibility could be found by adding the ICF and the inherent portion of the CC score.

Not all programs have the same features, so the Access Support Functionality (ASF) Ratio is computer to relate the factored score to program complexity. In ICF and in DCUF, the factored scores and functionality scores are summed across all items. The ASF Ratio is computed by dividing the total functionality score by the total factored score (ratio = factored/functionality). The maximum ratio possible is 2.183 [1]; no packages have come anywhere near this ratio yet. The ASF Ratio is computed for ICF-Student, ICF-Instructor, DCUF, and for all areas combined.

The combined ASF Ratio provides a quick assessment of a program's accessibility, however all computed scores are important. The Access Support, Functionality, and Total Accessibility scores indicate both the product's complexity and its degree of support in creating an accessible coursesite. Individual scores can be compared across packages to determine which products have the best capabilities.

We made three changes to the SNOW procedure. First, we did not score one item in ICF about deprecated HTML because we could not find a way to check it. Deprecated HTML is coding that does not conform to the most recent HTML standards. As one of 61 items, we did not believe this would change the scoring significantly.

Second, we added one piece. Originally, ICF was just one piece, Inherent Student Features (ICF-Student). We added ICF-Instructor, a copy of the student section. The accessibility of software programs to instructors is also important. For the student piece, we looked only at those parts seen by students. For the instructor piece, we looked at how instructors entered content and made changes to the course, such as Blackboard's Control Panel. The scores from both sections were added to get the ICF score.

Neither of these two modifications significantly changes the score comparison from one year to another. However, with the additional section, two programs reversed in the ranked order of the programs. Their score differential is only .001, unlikely to be significantly different. We provide the score with and without the additional section for comparison with the original SNOW report.

The third change is that we modified the scoring system for Information, as mentioned above. Since only one program had any Information scores above 0, this did not change the final score.

We reviewed the four LMSs and came to consensus on the score. We went through each program during a two-week period, returning a month later to review the scoring. We re-evaluated some scores as we came to a better understanding of the meaning of those items. We verified that scores matched across programs for same or similar occurrences. We used an already-created sample course in Blackboard 4 and created sample courses in Blackboard 4, WebCT, and Prometheus.