Atomizing Assessments

Crowd-sourcing training and learning assessments
by Katin Imes

We have an interesting swirl of needs and resources appearing in the domains of learning and professional development, and I'm curious how we can add layers of efficiency and curation for learners via crowd-sourced or distributed methods. Learners helping learners, you could say.

The needs I see appearing:

  • in-the moment learning and skills development;
  • guidance for learning paths for new skills and knowledge;
  • measurement of accomplished learning or skill development.

These needs have always existed, and you might recognize these as exactly the services provided by teachers over the last 2000 years. The problem is that these needs are quickly gaining urgency and scale far beyond what our current teachers can meet or even address. Adult learning and professional development are now constant, life-long requirements and will be for the foreseeable future, and that one factor alone means the scale of need now exceeds the established system by orders of magnitude. And we're seeing the effects of that happening all over: the explosion of online learning, the mega-expansion of the corporate-owned universities, the rapid changes in employment requirements and the job market.

The resources I see appearing include:

  • an explosion of high-information sources about nearly any topic you want to learn about - both free and commercialized;
  • active online communities for even very small niches, and dozens or hundreds of online communities for large topics;
  • hundreds of new tools released each year aimed at organizing, supporting, or selling personal learning.

So many resources exist and are being created, in fact, that it is overwhelming. You need computer tools to support a personal learning agenda simply because there are so many resources out there, and information and training tidbits can be found and collected so quickly. Overwhelm is one of the most significant problems to solve here.

A Happy Coincidence of Overwhelming Proportions

How wonderful that these resources are appearing just at the time we need them. If there's one huge benefit digital materials bring us, it is that they can be developed and distributed quickly. That's one factor in our favor.

Anyone that needs to learn something specific can learn it, very likely via resources on the Internet, and in many cases for free. Of course, learning about something and developing a new skill are two different things. Just because you've read about Javascript all day doesn't mean you have the skill of Javascript programming. So, of course, there are hours of practice and application required to develop demonstrable competency. These hours can also often be completed using free tools and interaction available via the Internet. It is truly a new age of individual power in learning.

Quick is Easy; Depth and Detail is Hard

The issue is managing large-scale learning without a teacher. I'm not worried about things that can be learned in an hour or even a day. Most people can manage the focus and tenacity to accomplish a couple of hours or even a full day of learning something that they need to know and that will serve them immediately.

No, I'm worried about the other side of the scale: the large-scale learning projects. Learning to speak a new language, for example. Learning to specify and design for the new solar electric systems. Learning technology skills like programming, platforms administration, server management, network security, router configuration, troubleshooting, and learning new complex software like animation tools or spreadsheets and databases. How long does it take to learn how to publish an eBook from start to finish? How about setting up and managing a business web site and e-commerce? There are thousands of other examples.

These are no longer optional kinds of skills for today's businesses and professionals. Each professional domain is transforming rapidly with new tools, new systems, new data, new technology, and new markets and international partners.

Where to Create Efficiency

If it takes, for example and on average, say 200 hours of learning to become a competent Javascript programmer, then that is a significant investment of time and effort. It is important to be efficient with this learning time and effort.

Yet as someone who knows nothing about programming and are attempting to be self-taught, that 200 hours could easily turn into 400 or 600 hours. And how do you know that you aren't learning some arcane corner of the language or methods that will end up mostly unused in your application of this skill? How do you know which things to learn next, or learn first? How hard is it supposed to be, and how do you know if learning one or two other things would make this task much easier?

On the flip side, is there a way we can organize the materials and order the learning so that someone could learn all the need to know in only 100 hours?

Because you don't know what you don't know, developing a complete learning plan is difficult to impossible. Winging-it and learning-by-rapid-exploration are two strategies and talents that some people can access and use, but even these are less efficient than a competent learning plan created by people who know the whole. Plus, most people won't have the time or skills to effectively apply these strategies.

Applying Assessments

So here you are, looking up "Javascript tutorials" on Google and finding a dozen different free ones and dozens more books and paid courses. Which ones will teach you what you want to know? Which ones are intermediate and advanced? Which ones are you ready for now, and what order should you try to take them? Are any of them worth buying - will the purchased materials save you time greater than their cost?

What if you find different approaches and philosophies on teaching and learning the subject? Can you mix-and-match lessons ad hoc from various teachers and sources easily, or will this just set you back?

Unfortunately, these are all decisions that you will have to make and re-make every step of the way on your self-guided learning journey. Sometimes you will decide correctly, and sometimes you won't. Yes, you will learn a lot - maybe even a lot about the process of learning and how you learn best - but you can also burn a lot of time doing it.

And the question will always remain: when will you be done? Do you know enough yet? Can you begin professionally working (charging people, essentially) for your skill in, say, Javascript programming yet? How will you know? How do you compare to others offering similar services?

Even if you can develop your own learning objectives - which isn't easy - you'll be guessing at all of these other answers. That's where assessments come in.

Assessments are generally considered to be part of testing learning at the end of a course or lesson. Tests, quizzes, and exams are traditional methods of assessing learning. Now we are also looking for ways to demonstrate competency as an assessment of skill or application of knowledge. Demonstrations usually involve actually doing a project and then a teacher or advanced skilled person assessing your performance.

Post-learning assessment addresses half of those questions, like how do you know when you are done (competent) and how you compare to others.

Pre-Learning Assessments

If we also apply assessments to pre-learning phases, and use the pre-assessment as a selector for lessons and modules, then it can also answer questions like, "what should I be learning next?" "What is the focus of this next learning session?" "What is it I already know, and what don't I know?" "What is most useful to learn for my application and need?"

I believe we can design pre-assessments to specifically match learners with learning materials. A few moments taking an assessment reveals how much of the material in the lesson, module or book will be new to you, and if you already have the foundational knowledge needed to understand the lesson, module or book. This can save people an enormous amount of time and effort, improving the efficiency of bringing our work force up to speed for the 21st century, and making self-guided professional development and continuing education a rewarding endeavor for individuals.

Crowd-Sourcing Assessments

The next issue to address is how to create these assessments and make them available to learners. My ideas for characteristics and requirements for assessments that boost self-guided learning include:

  • stand-alone: anyone should be able to create a pre- or post-assessment for any chunk of content available.
  • universal functionality: assessments should work in a similar fashion, with understandable results and recommendations.
  • connection with people: while the assessment itself should be computer-operated and computer scored with automatic results that learners can use on their own, assessments ideally would be supported by a community of practice or at least a teacher or author. This isn't just for "tech support" or assessment feedback and improvement; it also connects new learners to skilled professionals in specific domains, which has a host of benefits for both sides.

We want time spent on taking assessments to be time focused on the material and learning match, not on how to operate the assessment tool.


Creating a system for developing consistent learning objectives and assessment questions is a bigger challenge than it looks.

There will always be a need for people to be involved in the development of learning objectives and assessments. Benefits abound when we further connect the people, since self-guided learning will always be a self-adjusting process, and domains of knowledge and skills will always be advancing and changing.

So how do we map in a social-connect system with an automated assessments system that allows advanced learners to help beginning learners select materials and learning objectives that best serve their immediate and long-term needs?

That is the question I'll be working on over the next few years, and it is an exciting continuation of the computer-based assessment tools and online courses that I've developed over the last decade.