#1ONREL Transforming ENERGY Photo by Dennis Schroeder NREL 56461 <-EXGUST Workforce Program Design and Evaluation lization and control of ion distribution infrastructure Allison Moe, Cory Chovanec, and Robin Tuttle Better Buildings Workforce Accelerator Workshop May 16, 2023 ng Cascinated Rand dale Des NREL/PR-5500-86302#2Agenda 1 Welcome and Updates 2 Overview: Program Design and Evaluation 3 Writing Goals and Objectives 4 Break/Activity #1 5 Evaluating Workforce Programs 6 Activity #2 (Breakout) 7 Closing NREL 2#3Overview: Program Design & Evaluation Allison Moe, NREL JL Allison Moe, NREL#4Steps to Develop Workforce Development Programs 1. Regional Market Analysis ⚫Challenge •Demand and opportunity •Job market •Potential partners 2. Program Development •High-level goals/ objectives •General approach •Implementation details 3. Partner Engagement •Partner responsibilities • Building trust •Formalizing the partnership 4. Program Implementation •Developing materials •Recruitment •Trainee engagement •Tracking 5. Program Evaluation •Tracking • Reviewing goals and metrics •Making changes NREL | 4#5Steps to Develop Workforce Development Programs 1. Regional Market Analysis 5. Program Evaluation 2. Program Development 4. Program Implementation 3. Partner Engagement NREL 5#6Today's Focus 5. Program Evaluation 1. Regional Market Analysis 2. Program Development 4. Program Implementation 3. Partner Engagement NREL | 6#7Writing Effective Goals & Objectives Cory Chovanec, NREL#8Session Objectives G Recognize the importance of SMART goals and objectives. ☑ Give examples of how SMART goals and the ADDIE model can be used to plan a project, program, or training. Review the principles of the ADDIE instructional design model. ? Discussion with Q&A.#9Background & Context#10Background ● SMART Goals and Objectives - Specific, measurable, achievable, realistic, and time- bound ADDIE Model - Analyze, design, develop, implement, and evaluate. evaluation implementation analysis development design This photo by Unknown Author is licensed under CC BY-SA. NREL 10#11Why Do Goals and Objectives Matter? • Define the desired outcome • Inform stakeholders of what to expect • Create a clear roadmap for you and your target audience • Ensure unnecessary content is not included, which can: - - Save time and resources Make things clearer for participants • Support ongoing continuous improvement. ROAD TO SUCCESS This photo by Unknown Author is licensed under CC BY-SA. NREL 11#12Programs vs. Courses GOALS OBJECTIVES • Program Design Course/Training Module Design Describes the broad or long-term change that will be seen as a result of this effort, and why it is important to the target audience. Describe how you will achieve and measure success toward program goals Describe what participants will be able to do or know as a result of the course Must be SMART* • Must be SMART* • Should align directly with • Should align directly with evaluation criteria. evaluation criteria. *SMART = Specific, measurable, achievable, realistic, time-bound NOTE: Programs can have tiered sets of goals and objectives if there are multiple sub- programs, focus areas, or courses offered.#13Writing Goals#14What is a Goal? "The object of a person's ambition or effort; an aim or desired result" Oxford dictionary . • Life . . Professional • Training ● Programmatic.#15What Should Be Included? State the purpose of the training or program. A well-written goal should include:#16. Workforce Program Goal Examples Build interest in and awareness of home energy retrofit careers in my county in order to build a pipeline of future workers. Simplify and streamline pathways from existing education and training programs into home energy retrofit careers in my county, in order to increase worker investment in their careers. Improve knowledge and skill competencies of current and future home energy retrofit workers in my county in order to enhance the quality and timeliness of work. NREL 16#17Training Module Goal Examples Smart Labs Toolkit To implement a Smart Labs program that maximizes the energy efficiency and safety of laboratory operations while minimizing carbon dioxide emissions. Targeted Dense Pack Insulation As a result of this session, participants will have the ability to pneumatically install cellulose insulation into enclosed wall cavities. NREL 17#18How to Get There . . . Goal: To have a safe, comfortable, and energy-efficient home To accomplish the goal, you need: • A fully insulated thermal envelope A well-sealed air barrier • The thermal and air boundaries to be continuous and in contact with one another • • Efficient, properly sized equipment to condition the living space and heat water A well-designed and balanced distribution system Adequate ventilation to promote healthy indoor air quality. NREL 18#19Writing Objectives#20What is an Objective? Statement illustrating how you will demonstrate achievement of the goal - Objectives support the goal but are not themselves a goal - Objectives must be specific and measurable. Example Goal: Lose five pounds in two weeks Example Objectives: - - Recognize the benefits of losing five pounds. - Identify your daily nutritional needs. Track all meals and snacks for one week. - Eat lean protein and a vegetable with each meal.#21Things to Consider . Know your audience. . • - Prerequisites, existing knowledge, or skills Determine what you want learner to do. Consider what conditions need to be present to achieve the objective. To what level does the learner need to demonstrate proficiency? - Beginner, mid-level, advanced, etc. NREL 21#22. . Workforce Program Objective Examples Program Goal: Improve knowledge and skill competencies of current and future home energy retrofit workers in my county in order to enhance the quality and timeliness of work. Program Objectives: • • • • Meet with a minimum of 3 area employers in the first quarter of the year to identify/update key skills needed. Offer 5 hands-on training courses each year (3 for training new workers and 2 for upskilling existing workers). Assess the skills of all participants who complete a course for a minimum of 3 skills badges. Within 1 year, increase the number of program participants sitting for industry certification exams by 10%. Resource for writing SMART program objectives: https://www.health.state.mn.us/communities/practice/resources/phqitoolbox/objectives.html#smartobjectives NREL 22#23Course or Training Learning Objectives Generally accepted domains of learning: - Developed by Dr. Benjamin Bloom 1. Affective (feelings, attitudes) 2. Cognitive (thinking) 3. Psychomotor (physical or doing). NREL 23#24Course or Training Learning Objectives Example Goal: Lose five pounds in two weeks Example Objectives: - Recognize the benefits of losing five pounds. (affective; why this is important) - Identify your daily nutritional needs. (cognitive; knowing this is important) - Track all meals and snacks for one week. (psychomotor; do this) - Eat lean protein and a vegetable with each meal. (psychomotor; apply this change) NREL 24#25Course or Training Learning Objectives Start each objective with a verb. - Use verbs that align with the appropriate level/type of learning. U.S. DEPARTMENT OF ENERGY Office of ENERGY EFFICIENCY & RENEWABLE ENERGY Sample Verbs for Learning Objectives Cognitive Remembering • Define Duplicate • Draw Affective Receiving (listening and being attentive) Ask Choose Psychomotor Imitation (learner imitates an action after a visual demonstration) Describe Erect . List • Label . Follow • Memorize . Give • Name • Hold Recall Identify • Recite Repeat Reproduce State Understanding Locate Name Point to Select Sit Avoid words that can not be measured. - Understand, know, learn, familiarize. • Classify • Describe Discuss Explain • Identify Locate • Reply Use Responding (active participation) Answer Assist Comply Conform Paraphrase Recognize Discuss • Report Greet . Select Help • Translate Label + Visualize Perform Practice Present Applying • Read Recite Report Select Tell Write + Apply • Change Choose + Construct • Dramatize Employ Illustrate Interpret . Complete Demonstrate Valuing (value attached to a subject) Align Balance . Follow Grasp Hold + Place Repeat Rest Step Manipulation (performance of an action with written/verbal instructions) Align Balance Follow Grasp * Hold Place ⚫ Repeat Precision Accurately Without error Independently Proficiently With balance . With control Articulation (display of coordination of a series of related acts) Confidence Coordination Harmony • Integration 25#26Training Module Objective Examples Course Goal: Targeted Dense Pack Insulation As a result of this session, participants will have the ability to pneumatically install cellulose insulation into enclosed wall cavities. Module Learning Objectives: To properly install the insulation, participants will • Confirm the structural integrity of the wall assembly. . Determine the target density of the insulation for their project. . Prepare the insulation blowing machine and hose assembly. . Install cellulose insulation into each wall cavity with an installed density of 3.5-4.5 pounds per cubic foot. NREL 26#27Q&A#28Activity #1 (5 minutes) Developing Goals and Objectives: -Workforce Development Program Design -Training Course/Module Design#29Evaluating Workforce Programs Robin Tuttle, NREL#30Session Objectives . • Discuss evaluation basics. . Recognize the importance of evaluation metrics in program design. Review different workforce program evaluation metrics and how to select the best metrics based on your goals. NREL 30#31Background#32Background • NREL is leading an effort to create a workforce evaluation framework for the State Energy Program, to help measure the success and inform the design of effective clean energy workforce development programs. • NREL completed the literature review in January 2023 and will share some initial findings in this presentation. • NREL is now working with ACEEE to draft and publish an evaluation framework. GEOTHERMAL WEATHERIZATION IMPROVED HOME ENERGY EFFICIENCY BUYING EFFICIENT PRODUCTS ENERGY RESILIENCY DISTRIBUTED GENERATION IMPROVED APPLIANCES POLICY IMPROVEMENTS COMMERCIAL FINANCING LED LIGHTING RETROFITS CARPOOLS AND VANPOOLS AUDITS AND PROGRAMS SOLAR FACILITE RETROFITS STATE ENERGY PLANS INDUSTRIAL REVITALIZATION SMALL HYDRO ENERGY ENERGY LEGISLATION TO PROMOTE EMERGENCY ALTERNATIVE FUELS BETTER MOTORS PLANNING RETROFITS ENERGY EDUCATION SMART WIND POWER HYDROGEN FUEL CELLS TELECOMMUTING BENCHMARKING WATER SAVINGS COMBINED HEAT AND POWER Image from https://www.energy.gov/sites/default/files/2022-02/sep-operations-manual-2022.pdf NREL 32#33Workforce evaluation: Key Terms Broadly used to include the study of workforce programs, systems, strategies, services, activities, or interventions (U.S. Department of Labor Employment and Training Administration 2020). Workforce evaluation framework: An approach for evaluating programs; it should include metrics to measure: (1) overall impact. compared to programmatic goals, and (2) impact considering target populations. NREL 33