ASCENT Blog

A Look at Assessments

By Paul Burden

Companies have turned, and continue to turn, to technology products to gain a competitive edge. With widespread adoption, however, the competitive edge must come from how these tools are applied. A key to the application of complex technology tools, such as CAD software, is maintaining awareness of skill levels within the user community and providing the appropriate solutions when skill levels fall below an acceptable level. An absence or degradation in skill level with respect to complex technology significantly affects overall productivity.

That skill gaps occur as technology changes and evolves is not unexpected. With each new release of CAD software, hundreds of changes and enhancements are introduced. Suddenly, users trained on earlier releases are faced with a skill gap.

Even human nature leads to skill gaps. People of similar ability who receive the same training often show different levels of productivity afterwards. Why? Reasons include the following:

• Different learning styles
• Different levels of retention of training content
• Different amounts of time between training and use

There are many assessment options available in the market that are designed to provide companies with an inventory of the skill levels within their user communities. This is done by focusing on specific areas of CAD/CAM use with skill-assessing questions and tasks that collectively identify any skill gaps present for any of the reasons described above. Results may be accompanied by recommendations to target the skill gaps with focused training solutions that will ensure the technology tools are being applied to their maximum benefit.

Any assessment must have the attributes of reliability, validity, and fairness with respect to the purpose of the assessment and interpretation of the results.

Reliability refers to the consistency or reproducibility of assessment results. A user who takes the same assessment at different times should produce similar results. Likewise, a user who takes multiple versions of an assessment, which are intended to be equivalent in content and scope, should produce similar results each time.

Factors that can contribute to unreliable results include:

• Questions are ambiguous.
• Questions assess interpretation rather than application.
• Questions require selection of more than one answer.
• Responses from previous sessions are memorized.
• Response selection is the result of guessing.

Validity refers to the value of the meaning that can be drawn from assessment results. Customers are typically seeking baseline of user proficiency with a CAD/CAM tool; therefore, the validity of any assessment in determining proficiency is important. Factors that would be used to determine the validity of any assessment include:

• The extent to which the assessment addresses an appropriate sample of the skills and knowledge required for proficiency.
• The extent to which the assessment finding is representative of a user’s actual proficiency with all of the required domains of the CAD/CAM tool, not just the specific sample topics presented on the assessment.
• The extent to which the assessment is verified on representative sample groups of known proficiency within the target population.

Fairness refers to the extent that assessment findings are comparable between individuals and groups.

Three common assessment formats available are hands-on, multiple-choice quiz, and hybrid.

Hands-on assessments consist of specific tasks that the user is required to perform on actual data that is then analyzed. An example of a hands-on assessment task is shown below. For this type of question, the user accesses the provided CAD file from, performs the prescribed task, and submits the modified file for evaluation.

This assessment format may employ an application expert to examine each submitted model to verify the accuracy and completeness of each task and to make a determination of the user’s ability and proficiency. A score is determined for each task using evaluation criteria, such as the following:

This format of assessment has a high level of reliability because it eliminates guessing by users and requires them to demonstrate performance and ability. Because end results are examined, this format also lends itself to assessment of the quality and end-result proficiency measures. Models can be screened for organization-specific methodologies and best practices, resulting in a thorough examination of a user’s skills and knowledge. 

Assessing an appropriate sample of skills and knowledge with this format of assessment, ensuring validity, will require the highest amount of time of the three formats analyzed in this discussion. This format will also provide the truest results from which conclusions of skill and ability can be drawn.

Multiple-choice assessments consist of a series of questions that require the user to select the correct response from at least four possible responses. An example of a multiple-choice question is shown below.  For this type of question, the user does not use any provided data; the user simply reads the question, refers to the images, and selects the correct response.

This type of assessment can be analyzed very quickly with results available almost immediately upon completion. Many online assessments of this format auto-evaluate the responses and return a report to the user. 

This format of assessment does have a lower level of reliability than the hands-on format since there is no interaction by the user with the software required to complete the assessment. With one correct response out of four per question, users have a 25% chance of guessing the correct response for each question and are not required to demonstrate performance and ability. Lower reliability of this assessment format also lowers validity since conclusions about a user’s actual proficiency based on assessment results would be limited.

This format does permit a greater number of questions than the hands-on assessment format, and therefore a wider sample of skills and knowledge can be assessed. This format of assessment meets the requirements of many customers, as an indicator of proficiency for the purposes of assessing training needs.

Hybrid assessments are a compromise of the hands-on and multiple-choice assessment formats. It requires the user to perform specific tasks on actual data and then select the one correct multiple-choice response to verify that the tasks were completed as required. A sample of a hybrid question is shown below.

To balance the attributes of reliability, validity, and fairness, the assessments you use ideally consist of combinations of the three formats described here. This enables you to maximize the number of topics covered within a reasonable amount of time acceptable to your organization for users to spend completing the assessment. The results will provide valuable indicators of skill gaps within the user community that can be addressed appropriately with targeted training solutions, and enable you to leverage your resources, which include your people and your technology tools, to the fullest extent. 

Leading the Way

By Elvis R. Sverko

A couple of weeks ago, I had the honor of attending and presenting at the Project Lead The Way Ohio 2011 Fall Conference at the University of Akron. This conference is designed to inform Ohio educators about the implementation of pre-engineering strands for schools utilizing the PLTW curriculum. These educators are teaching the future engineers of our world.

One major tool that is used in the PLTW curriculum is Autodesk Inventor. Through project based instruction, the educators have their students use Inventor to model various real-world objects to learn how to design them better and solve problems. So as you can imagine, these educators need to understand engineering principles, teaching principles, and how to use Inventor! As they may not be experts in all the functionalities of Inventor, they are sure to have some questions.

And that’s where my presentation came in. However, it was more of a discussion/demonstration of typical issues they see in the classroom, as well as some tips and tricks to help streamline their curriculum. Thanks to Bill Small, PLTW Master Teacher, Timken Regional Campus, for presenting with me. He was instrumental in understanding what types of issues the educators do see, and what types of tips can help them, as he knows the curriculum being taught, and then we teamed up to solve the issue.

Many of the topics we discussed can benefit anyone using Inventor: instructors, designers, and engineers alike. I’d like to share a few of these topics: the ones that seemed to be simple, yet very popular and powerful.

Being that these are educators who constantly teach new students who constantly work on new models, a major topic related to settings. In the Application Options dialog box, there are some settings that can be very helpful. On the Part tab, you can set which origin plane the initial sketch is placed upon new part creation, if any at all. By default, the “x-y plane” is selected to help streamline the modeling process, as most new parts would typically use the same starting plane, but for new students, choosing “No new sketch” is a good idea so the student can understand the process thoroughly, and decide for themselves which origin plane would be best to use for a particular model.

The Sketch tab of the Application Options dialog box also contains a few great setting options. By default, the “Snap to grid” and “Edit dimension when created” settings are not selected. Having these two options selected can be helpful as well. The “Snap to grid” option can assist Inventor novices while drawing a sketch, making it easier for them to be sure that the endpoints of line or arc segments are coincident. While the “Edit dimension when created” option is useful for anyone, as with a typical workflow, initially drawing the sketch to approximate size, and then making the values precise by adding dimensions to them, and having them available to edit immediately upon placement can save a few mouse clicks with each dimension; a huge savings over the course of a single model.

Another fantastic Inventor Application Option tip for educators is the exporting and importing of options that are set for Inventor. You can setup a single computer with the settings that most benefit the students, and then save them via the Export button at the bottom of the Application Options dialog box, where all the settings are saved in an XML file, that you can then Import on all the other students computers, to match.

Well, there are definitely many more topics that we can discuss, but maybe I’ll save them for another post.

Again, thanks to Project Lead The Way for allowing me to present and share these helpful Inventor tips, thanks to Bill Small for presenting with me, thanks to Cindi Meier from IMAGINiT for presenting an overview of how the curriculum can be used by the educators, and also thanks to all those educators who attended my session. It’s great being able to teach those who are teaching others.

“To learn, read. To know, write. To master, teach.” – Hindu proverb

 

 

Tips for Working with Autodesk® Inventor® Publisher

By Jennifer MacMillan

On Nov 29^th my AU virtual class (MA4475) “From Concept to Published Document: Tips for Working Effectively with Autodesk® Inventor® Publisher” goes live. Over the last few weeks I have been polishing my scripts, demos, Power Point presentations, and Camtasia skills and it has all been uploaded and will be ready for you AU virtual attendees.

I thought with this posting I would provide some favorite tips that I will be discussing in the 45 min virtual class.

 Inventor Publisher Tips (in no particular order)

1. Specify the Import BOM View Type option when importing an Inventor assembly file. This is set through the Import options. This specifies the level to which the Bill of Materials can be displayed in the Autodesk Inventor Publisher file. The Parts Only option displays all of the parts but no subassembly structure and the Structured option enables you to display the subassembly structure. If this option is not set properly on model import it cannot be changed. The file must be recreated. 
2. When working with 3D data you can specify a Faceting level on import. It controls the model’s edge tessellation. Smaller faceting values result in smoother edges but larger file sizes. Large faceting values result in coarse edges but smaller file sizes. Once a file has been imported with a selected type you can further control the edge tessellation using the View Fidelity slider in the View tab. This is ideal if you experience reduced computer performance and need to lower the fidelity value to improve system performance. Controlling View Fidelity with the View tab is application specific. Once changed, the new setting is active until the slider is explicitly changed again. Consider reducing the setting while working in the file and then increasing it again before publishing.
3. Whenever possible, leverage the use of your Inventor data. Because of the associative nature of Inventor data within the Autodesk Inventor Publisher software, changes are passed through to your publisher file. Additionally, you have the added benefit of having access to all of the iProperty information from the Inventor file.
4. If iProperties are not listed in the Associative Property drop-down list, ensure that your callout’s leader references a component in the file. If the leader is placed so that it is not referencing a component, the drop-down list is empty. To verify the reference, select the callout. If the grip on the leader is red it is not referencing a component. Select the grip and drag it onto the necessary component. Once moved, the grip should display in green indicating that it is referencing the component properly and the iProperty drop-down list should now be visible.
5. All snapshot settings, such as its name, description, appearances, and duration can be saved so that any new snapshots use these settings. Set these values for the first snapshot, right-click on the snapshot and select Set as Default Setting. This does not modify the settings on snapshots that have already been created; the settings are only used for new snapshots.
6. When using the Move command, you can use a triad to move selected components. By default, the triad displays at the first selected component’s origin. To explicitly change the triad’s location, hover your cursor over the icon that displays with the triad, select Align, and then select another reference point in the model. To move the triad to a point in space, press and hold down the Space Bar (to temporarily disable movement) and select and drag the central gray dot to the required location.
7. Scope Editing is a great command to use as you return to existing snapshots and begin to refine a project by changing component placements, appearances, explode positions, etc. There are a total of eight options that can be used to determine how the change will propagate to other snapshots or storyboards. The key thing to remember with this tool is that the scope setting must be set before you make the change for it to propagate. The Scope Editing options are located in the right-click menu of a snapshot by selecting Set Affected Snapshots.
8. The Extract Camera option enables you take the camera view from a source Snapshot and copy it to the active Snapshot. This is perfect once you have a storyboard created and need to make viewpoint changes in the snapshots. Using Extract Camera you can ensure that every Snapshot’s viewpoint is identical by ensuring that the zoom, pan, and orientation settings of the source Snapshot are automatically updated in the active Snapshot. Combining the use of Extract Camera and the Scope editing tools enables you to apply the same camera position to multiple Snapshots. Just be sure that you set the scope editing option before you use the Extract Camera command.

For more information on AU virtual go to the following link:
http://au.autodesk.com/?nd=auv2011_event

More than the Model

By Martha Hollowell

In my last post I started talking about Teaching BIM and how important it is to work side by side in 2D and 3D views to help students get a feel for the “building modeling” concept.  Another important aspect of the whole BIM arena is the “information” part of BIM. To help students understand some of what BIM can do for them you can show them a simple schedule update as shown in the graphics below. On the left is a 3D view where a family made up of a dining table and chairs is being placed on the patio of a building. Under it is a furniture schedule before the family is added. This includes not only the number of the various furniture families in the model but also the cost.  Then, on the right, is the plan view that shows the family already placed and below it the schedule updated to include the new dining table and chairs.  

The information is actually stored in the family and then applied in the schedule. In this case, the Cost parameter of the dining table and chair family is stored in its Type Properties and also included in the fields of the schedule as shown below.  

This information from the schedule can also be exported to a text file (Application Menu: Export>Reports>Schedule) that can then be imported into another program to give you a quantity takeoff for estimating or purchasing.  The information goes beyond the building model and construction documents into the hands of construction bidders, building managers, purchasing agents, and more.

How do you like to show the power of BIM information in your classes? Share it in the comment section below.

If you are coming to AU 2011 please be sure to come by and see me. My class "The Brainy Bunch: Teacher Training Tips for Autodesk® Revit® Architecture, MEP, and Structure" is scheduled for November 29, 2011 at 10:15AM.  I will also be at the Ascent booth 601 off and on during the convention.

Martha Hollowell