<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Experimentation Works on Medium]]></title>
        <description><![CDATA[Stories by Experimentation Works on Medium]]></description>
        <link>https://medium.com/@exp-works?source=rss-c1fc7bcd7280------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*QP8z4TQi_4IlNPVQIdG9EQ.png</url>
            <title>Stories by Experimentation Works on Medium</title>
            <link>https://medium.com/@exp-works?source=rss-c1fc7bcd7280------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 07:18:57 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@exp-works/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com" rel="hub"/>
        <item>
            <title><![CDATA[Learning what works for hybrid teams]]></title>
            <link>https://exp-works.medium.com/learning-what-works-for-hybrid-teams-5e81180bb828?source=rss-c1fc7bcd7280------2</link>
            <guid isPermaLink="false">https://medium.com/p/5e81180bb828</guid>
            <category><![CDATA[future-of-work]]></category>
            <category><![CDATA[hybrid-work]]></category>
            <category><![CDATA[experimentation]]></category>
            <category><![CDATA[distributed-work]]></category>
            <category><![CDATA[distributed-teams]]></category>
            <dc:creator><![CDATA[Experimentation Works]]></dc:creator>
            <pubDate>Tue, 19 Jul 2022 19:07:46 GMT</pubDate>
            <atom:updated>2022-07-22T19:05:29.309Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SIKfRVAgzfFZx8eN8Bip5g.jpeg" /><figcaption>Image: iStock <a href="https://www.istockphoto.com/portfolio/Aleutie?mediatype=illustration">Aleutie</a></figcaption></figure><p>Our GC Experimentation team is back in the office two days a week.</p><p>Seeing colleagues in person, remembering how to pack a lunch, and discovering which work clothes still fit have all been part of the return to office experience.</p><p>We know many of you are living these same changes and testing out new ways of working in a hybrid world. As we all try to figure out “how to hybrid”, we thought it would be a good moment to share some learnings from a recent small-scale experiment that we ran at TBS (Treasury Board Secretariat) in partnership with the <a href="https://hrexperimentation.medium.com/">Research and Experimentation</a> team at OCHRO (Office of the Chief Human Resources Officer).</p><p>We are not experts in hybrid work and won’t pretend to have all (or any) of the answers. But what we do know is that testing, learning, and adapting — or what we would call the experimental mindset — can be a powerful tool to navigate uncertain environments. We hope that the results from our Distributed Work Trials might spark some ideas for your team as you shift into this new rhythm of work.</p><p><strong>The challenge</strong></p><p>As soon as you have one team member working in a different location — whether at home or a regional office — you are on a distributed team. Working in distributed teams can pose a range of new challenges. Three of the most common problems center on engagement, work-life balance, and focus time. It can be hard to feel engaged or connected to a team in a virtual environment. Sometimes the lines between work and life become blurred, with added pressure to always be “on” and responding to emails. And the tendency to multitask while attending back-to-back virtual or hybrid meetings can leave little time for deep thinking or heads down work.</p><p><strong>What we tested</strong></p><p>We decided to focus our trials on two promising practices: team charters and time blocking. Team charters are a list of principles and team norms that everyone agrees to follow. When it comes to distributed work, they usually cover things like when to be online, what communication platforms to use for different tasks, and how to deal with after-hour emails or urgent requests. By comparison, time blocking usually involves establishing meeting-free days or time blocks where all team members have uninterrupted time for deep focus work.</p><p>We tested both practices in the spring and summer of 2021. In total, 11 teams at TBS tested team charters (156 participants) and 9 teams tested time blocking (66 participants). The teams covered a wide range of functional communities including policy, human resources, finance, and technology. Teams were able to self-select and choose which practice they wanted to try out.</p><p>To get things going, we provided each team with a team charter kit or time blocking instructions, including workshop materials for a collaborative charter design session. Teams were then asked to test out their chosen practice for three to four months and report back through a before and after survey. In our experimental language, we would call this a “small e” experiment that uses a simple pre-post methodology.</p><p><strong>What we found</strong></p><p>Overall, the results were positive for both practices. Team charters resulted in a drop in stress related to work-life balance from 35% to 19%, a drop in working after hours from 44% to 29%, and an increase in the ability to follow a schedule from 49% to 57%. Similarly, time blocking helped increase the amount of focus time for 50% of participants and raised self-reported productivity for 57% of participants. While these are promising findings, the trials also pointed to some nuance and mixed results in certain teams, as well as critical success factors.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2K-lQjhLmFVjJJK2iwSGGQ.png" /></figure><p><strong>Team charter results</strong></p><p>Our results suggest that team charters can be an effective tool to promote well-being. Participants reported the charter was an important team building exercise and that the collaborative process of developing the charter was as important as the document itself. By having an open discussion and documenting the team’s values and expectations, participants felt less pressure to check emails, voicemail and MS Teams chat after their regular work hours. One element that worked well was having an external facilitator run the team charter workshops, to help overcome team dynamics and ensure everyone could participate on an equal footing.</p><p>This being said, team charters did not solve everything or work for every team. Over a fifth of participants (22%) reported that the team charter helped to clarify work expectations only to a small extent. Several participants flagged that the document itself is not enough to change culture and can quickly be ignored when work pressures rise. Other participants felt their charter had not been in place long enough to feel its effects. Team charters also appear to be a weaker tool for workload and productivity. None of the participants reported a change in their workload, and nearly two-thirds (61%) reported the same productivity before and after the charter.</p><p><em>“The charter was a great way for the team to formally document how we worked and what was expected of each individual and of the team as a collective.”</em></p><p><em>“Developing a team charter was a fantastic team building exercise. Clarifying our preferences for working both on our own as well as together will make us a more efficient and productive team.”</em></p><p><em>“My concern is that the guidelines we set for ourselves will not be maintained. Directors and managers need an incentive to be accountable to respecting their teams operating needs.”</em></p><p><strong>Time blocking results</strong></p><p>Time blocking showed more mixed results. Some teams found meeting-free days or time blocks improved their focus time and productivity, while others did not. Overall, half of the participants (50%) reported having more focus time and less work interruptions during blocked time. As a result, they were able to better plan and protect their time to complete work tasks as intended. For over half of participants (57%), this led to an improvement in self-reported productivity over the testing period.</p><p>Other teams found that distractions were still ever present. Changing priorities or urgent requests meant some participants could not stick with the meeting-free time. A lack of coordination was also challenging. The fact that other teams or managers were not blocking the same time meant it was often not respected or pushed aside to coordinate schedules. There is also little evidence that time blocking had an impact on workload. Four in ten participants (40%) reported no change in their workload, while others reported a small increase.</p><p><em>“It’s great to know that I have this block of time devoted to focus time. When I’m going about my day and get a task that needs focus, I can be confident that I can schedule the time needed during my time block and that I will be able to focus on it.”</em></p><p><em>“Time blocking is effective, but executives and managers have limited time so tend to call when they are available, regardless of time being blocked.”</em></p><p><em>“Easy to put in calendar, more difficult to actually focus on one task during that time.”</em></p><p><strong>The bottom line</strong></p><p>Our Distributed Work Trials offer some promising tools, but none are a perfect solution. So, if you are looking for new practices to try, your team may benefit from the collaboration of building a team charter. You may also find time blocking and meeting-free days helpful tools for focus time and productivity, especially if these can be coordinated with partner teams and management.</p><p>It is important to note our trials used a small TBS sample and a simple before and after comparison, so the results may not be generalizable. The practices we tested will work better in some contexts than others. If any of the ideas interest you, we encourage you to test things out yourself! With your team, you can co-create and iterate on new practices and taking an experimental approach can be as simple as comparing the before and after results.</p><p><strong>What’s next for hybrid?</strong></p><p>There is a growing experimental community around hybrid work. We have heard from several teams and departments running their own experiments and pilots on promising hybrid practices.</p><p>If you are interested in learning more, our colleagues from the OCHRO <a href="https://hrexperimentation.medium.com/">Research and Experimentation</a> team have recently developed a “hybrid-in-a-box” experimentation toolkit and <a href="https://gcxgce.sharepoint.com/teams/1000478">GCxchange page</a> to bring this community together. The site is available to Canadian government employees, and you can <a href="https://www.gcx-gce.gc.ca/home">create an account</a> using your government email.</p><p>As always, thank you for being part of our experimentation community!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5e81180bb828" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building an Experimental Government]]></title>
            <link>https://exp-works.medium.com/building-an-experimental-government-a1f0c0b60cf7?source=rss-c1fc7bcd7280------2</link>
            <guid isPermaLink="false">https://medium.com/p/a1f0c0b60cf7</guid>
            <category><![CDATA[experimentation]]></category>
            <category><![CDATA[public-policy]]></category>
            <dc:creator><![CDATA[Experimentation Works]]></dc:creator>
            <pubDate>Mon, 25 Apr 2022 19:13:49 GMT</pubDate>
            <atom:updated>2022-04-25T19:13:49.265Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="Lego brick wall with multiple coloured bricks" src="https://cdn-images-1.medium.com/max/1024/1*8jsZpXV20P8Vej1Gim-p9A.jpeg" /><figcaption>Photo credit: <a href="https://unsplash.com/photos/lQT_bOWtysE">Omar Flores</a></figcaption></figure><p>It has been a couple of months since our relaunch event of Experimentation Works (EW) on February 24th, 2022. We called this event “Building an Experimental Government” as we spent time reflecting on how experimentation has grown in the federal public service and where it’s heading next.</p><p>As our team moves into a steady pace of identifying and supporting new experimental projects, we thought it would be a good moment to provide an update to our community and unpack some of the ideas that have stayed with us since the launch event.</p><p><strong>EW service model</strong></p><p>For those following this space, most of you know by now that we have evolved the EW model from an annual cohort of projects to a more flexible and <a href="https://exp-works.medium.com/experimentation-works-is-back-with-a-twist-648235520739">on-demand service</a>. Since our relaunch, we have received EW requests from seven teams across the federal family. The experiments proposed touch on a wide range of policy areas, including housing, youth employment, lived experiences in healthcare, hybrid work, and employee health and well-being.</p><p>So far, we have had four projects move on to matching with an expert advisor. Most of the other projects were in early stages and have been directed to other resources to refine their experimentation plan before bringing in expert advice. Over the next few months, our team will be working to promote the EW service to other areas of government and to streamline our guidance materials for teams who are just starting out on their experimental journey.</p><p><strong>5 years of progress</strong></p><p>Of course, the type of experimentation we are talking about is not new to the Government of Canada. It has been just over five years since the 2016 <a href="https://www.canada.ca/en/innovation-hub/services/reports-resources/experimentation-direction-deputy-heads.html">Experimentation Direction to Deputy Heads</a> was released by the Treasury Board Secretariat and the Privy Council Office. We were pleased to see the continued enthusiasm for policy experimentation at our event and to reconnect with our community to celebrate how far we have come.</p><p>The 2016 Direction was a clear call to action, emphasizing that “ethical and rigorous experimentation is central to the Government’s focus on evidence-based policy-making, results and delivery.” Teams across government have responded to this Direction with spirit. It is remarkable to see the progress reflected in our Management Accountability Framework data. In 2020, 91% of federal departments and agencies committed resources to experimentation, 88% were running experiments and 65% had a framework in place for experimentation.</p><p>With more targeted resources dedicated to experimentation, we’re seeing an increase in the number of experimental projects across government. We’ve also seen the creation of governance bodies and specialized units for innovation and experimentation at the department level. What was once side-of-desk, or limited to scientists, has grown into a deliberate practice of innovation and ethical experimentation in many parts of government. <br> <br><strong>So, given the progress made, is experimentation still necessary?</strong> <br> <br>We think yes. There are still so many areas of Canadian policy and service delivery where we don’t know what works best. To take just one timely example, we’re all still finding what works best for managing and working in hybrid teams. And it’s not just about running the experiments. It is also critical to ensure that senior leaders are using experimental evidence in the decisions they make.</p><p>The need to keep experimenting was also repeatedly stated in our panel discussion at the launch event. We were joined by Nick Chesterley, Director of Strategic Policy and Experimentation at the Treasury Board Secretariat, and by Sasha Tregebov, Director of the <a href="https://www.bi.team/people/sasha-tregebov/">Behavioural Insights Team Canada</a>. It was a rich discussion full of reflection, hope, and a reminder that while we have made great progress, we are still only scratching the surface of what experimentation has to offer. Below are a few of the moments that have stayed with us since the event.</p><p><strong>Next time your team brings you three great ideas, try them all.</strong></p><p>When asked what managers and executives can do to support more experimentation, Nick’s answer was simple. Next time your team brings you three great ideas, see if you can find a way to try them all. Senior leaders can often feel pressure to make decisions and provide clarity to their teams. But sometimes, the best thing we can do is create space for experimentation and learn from doing.</p><p>This takes courage. It requires us to admit, as individuals and as teams, that we don’t always have the answers or know what will work in advance. But what better way to find out than to test our best ideas, in a smaller way, before rolling them out or making more permanent decisions? This reflex is what we often refer to as an experimental mindset. It is the impulse to keep asking questions, testing solutions, and finding smart ways to measure and compare our impact.</p><p><strong>The misconception of perfect data… and experiments.</strong></p><p>Maybe the above suggestion doesn’t sound like a perfectly designed experiment and of course it isn’t. But as Sasha raised in our panel, we need to overcome the myth of needing perfect experimental designs or data to run experiments. The reality is, we make decisions based on imperfect information every day. Even imperfect data or simple experiments can generate insights we didn’t have yesterday. What matters is to start somewhere and to be upfront about any limitations. As long as we continuously strive to improve our level of evidence, we will always end up knowing more than we did before.</p><p>So, with that, we continue to encourage you to keep trying, testing, and asking new questions. And if you need a helping hand along the way, please reach out to our GC Experimentation team or <a href="https://gcxgce.sharepoint.com/teams/1000172/SitePages/About-our-EW-service.aspx">submit a request</a> to the EW service. We are here to help, and eager to see what the next five years of experimentation will bring to the Government of Canada.</p><p><strong>Author</strong>: GC EXP Team</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a1f0c0b60cf7" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Experimentation Works is back — with a twist!]]></title>
            <link>https://exp-works.medium.com/experimentation-works-is-back-with-a-twist-648235520739?source=rss-c1fc7bcd7280------2</link>
            <guid isPermaLink="false">https://medium.com/p/648235520739</guid>
            <dc:creator><![CDATA[Experimentation Works]]></dc:creator>
            <pubDate>Mon, 21 Feb 2022 20:00:47 GMT</pubDate>
            <atom:updated>2022-02-21T20:00:47.057Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>Experimentation Works is back — with a twist!</strong></h3><p><em>Yes, EW is back! But before we reintroduce you, let’s pick up where we left off</em>…</p><p><strong>Creative, adaptable, eager, determined.</strong></p><p>We can think of many adjectives to describe the second Experimentation Works (EW) cohort, who started their projects in March 2020 and reached the finish line in April 2021. At our closing event, we celebrated the cohort’s progress on nine outstanding experimental projects and offered our thanks to the 32 experts who made EW possible. We also celebrated our community, which despite the pandemic, continued to find ways to connect and learn. Throughout the year, we gathered (virtually) for over 20 events and members regularly shared their project updates and experiences on this blog.</p><p>Since April, our team at TBS has had some more time to reflect on EW. We took stock of the past two cohorts: what worked, what didn’t. And in the true spirit of EW, we embraced the process of learning-by-doing, testing, and iterating. One lesson is that there is no predictable timeframe for an experiment. Each project has its own particularities, and we want to be more flexible to accommodate that.</p><p><strong>Expert advice on-demand</strong></p><p>So, with the lessons of the pandemic close at hand and a desire to improve on past iterations of EW, we have decided to pivot away from a cohort-based model towards a more flexible and user-centred design. With this comes a change from yearly intake of projects to on-demand requests for expert advice. We envision a new, ongoing <strong>EW service</strong>, one which can better accommodate our users and provide support at any stage of their experimentation process.</p><p>In other words, if you are planning or have an ongoing experimental project anywhere in the Government of Canada, we want to help. Support is also on offer for those who are earlier in the process and are looking for advice on pre-experimental or exploratory work.<br> <br>To access EW support for your project, just fill out our <a href="https://forms.office.com/Pages/ResponsePage.aspx?id=EN-XY5VFR0CcTwMxEoIVKyWnTxdK0fVCn7G50e38ocZUMkIzSVNHN1dSRDRWVjU0R0JLSEpRVlBOQSQlQCN0PWcu">service request form</a>. Our team assesses these requests on a rolling basis, making sure that all projects are (pre-)experimental in nature and have undergone a minimum level of development. For eligible requests, we’ll play matchmaker and facilitate the first meeting between the project team and the best suited expert.</p><p>Of course, the return of EW and the changes in its delivery model wouldn’t be possible without the support of a highly talented group of experts. Belonging to departments across government, as well as academia, we have recruited this year’s experts to what we’ve dubbed “the first Government of Canada Experimentation Advisory Panel.” The experts have in-depth skills in research and experimentation, with varied backgrounds in the natural and social sciences. We’ll be sure to profile the expert panel in our next posts, so stay tuned!<br> <br> To learn more about EW, visit us on <a href="https://gcxgce.sharepoint.com/teams/1000172/">gcxchange</a> (you may need to <a href="https://www.gcx-gce.gc.ca/home">register</a> first — currently Government of Canada employees only, unfortunately), contact us by <a href="mailto:zzexper@tbs-sct.gc.ca">email</a>, or attend our upcoming launch event (details below)!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3TKJU2Qu8KTLYrO8KWd6Nw.png" /></figure><h3>Experimentation Works Launch Event: Building an Experimental Government</h3><p><strong>Description:</strong><br>Experimentation in the public service is tightly linked to innovation and problem solving. By running experiments, we can rigorously test a suite of options to find the best solution. Across the Government of Canada (GoC), departments and teams have embraced the call to experiment, but sometimes, individual projects can benefit from some additional expertise. That’s where we come in…<br> <br>This event is for relaunching the TBS-led initiative, Experimentation Works (EW), an expert advisory program. EW started in 2018 as a cohort-based learning program, where teams across the GoC worked through the stages of running an experiment, together. After leading two successful project cohorts, we’re now taking things in a new direction. We’ve remodeled EW as an on-demand service where project teams can get expert advice at any stage of the experimentation process. So, teams can apply today, tomorrow, or whenever they need that extra boost of support.<br> <br>As we relaunch this new service, we will be joined by Kaili Levesque (Assistant Secretary, Privy Council Office) and Sasha Tregebov (Director, <a href="https://can01.safelinks.protection.outlook.com/?url=https://gcpedia.us16.list-manage.com/track/click?u%3D1be51ad6596702f823a797a96%26id%3D049bbe8b1e%26e%3Dab8b8f30ac&amp;data=04%7c01%7cCharlotte.DeKeyzer%40tbs-sct.gc.ca%7c404ee81a7ce441b8f8bf08d9ea75c302%7c6397df10459540479c4f03311282152b%7c0%7c0%7c637798613769200143%7cUnknown%7cTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7c3000&amp;sdata=jHRsUyQbyV0AHYBRoiuUoSaXhBWk2CF2FVosZvCvSuw%3D&amp;reserved=0">Behavioural Insights Team Canada</a>) for a panel discussion on the future of experimentation in the federal public service. We hope you will join us too!</p><p><strong>Date and time:</strong> February 24, 2022 | 1:00 pm to 2:30 pm (EST)<br><strong>Language: </strong>Bilingual<br><strong>Location: </strong>This event will be delivered virtually. A link to the MS Teams Webinar will be provided to registered participants.<br><strong>Registration: </strong>Please email the GC EXP team to register<strong> (</strong><a href="mailto:zzexper@tbs-sct.gc.ca">zzexper@tbs-sct.gc.ca</a><strong>)</strong></p><p><strong>Author</strong>: GC EXP Team</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4t6TKlBd7f2YDsV55cUIJA.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=648235520739" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Capturing Free Services in Canada]]></title>
            <link>https://exp-works.medium.com/capturing-free-services-in-canada-15ec2a61071?source=rss-c1fc7bcd7280------2</link>
            <guid isPermaLink="false">https://medium.com/p/15ec2a61071</guid>
            <category><![CDATA[productivity]]></category>
            <category><![CDATA[free-services]]></category>
            <category><![CDATA[digital-economy]]></category>
            <category><![CDATA[experimentation]]></category>
            <dc:creator><![CDATA[Experimentation Works]]></dc:creator>
            <pubDate>Wed, 16 Jun 2021 17:23:58 GMT</pubDate>
            <atom:updated>2021-06-16T18:06:31.281Z</atom:updated>
            <content:encoded><![CDATA[<p>Over the past two decades, the internet has revolutionized almost every aspect of life in Canada, from how we communicate to how we learn and entertain ourselves. As peoples’ daily lives move increasingly online, there is a growing body of literature studying the value that internet users place on the services they access digitally.</p><p>One angle of study pertains to the use of free online services, many of which serve as substitutes for paid goods and services (free streaming content, for instance, is a substitute for conventional television broadcasting; online mapping applications substitute for paper maps; etc.). Digital services can capture larger market shares when they offer their services free of charge, and instead cover their costs using advertising revenue rather than charging a price to users.</p><p>This has led to what some economists call the “vanishing productivity problem”. When a good or service has no market price, accounting for it in conventional economic measurements of GDP becomes a problem, as it requires a positive price to express economic activity in dollars. Therefore, when more consumption of services is free rather than paid, that economic activity is not recorded at all by conventional means. As a result, economic productivity and growth in affected sectors appears to “vanish”.</p><p>Economists in many countries have been studying the vanishing productivity problem, and have developed a variety of methods to describe and measure the scale of free services in the economy. One such method involves directly asking people how much they value the free online services that they use.</p><p>To reduce the amount of bias and subjectivity in conducting such a survey, our team seeks to replicate the work of MIT economist <a href="https://www.pnas.org/content/pnas/116/15/7250.full.pdf">Erik Brynjolfsson</a>, and ask the question “Would you give up (a service) for one month in exchange for $X dollars?”, where X is a random value for each respondent. Asking questions about the value of free services in this way reduces individual bias in responses, and considering responses across the population gives us a proxy for the median value that the average Canadian would place on a given service.</p><p>Although the value derived from this method is not exactly the same as the value the service would sell for on the market, we can use these values to estimate the overall value of free services in the economy, as well as track the value of free services over time, giving us an indication of the relative importance of services such as social media, information gathering services, and others.</p><p>Our project hopes to fill this important data gap regarding Canada’s digital economy, and to expand the scope of ongoing research on the digital economy at Statistics Canada. In pursuing this project as part of the Treasury Board of Canada Secretariat’s Experimentation Works program, we hope that we can make headway on this topic to better provide informative and relevant data to Canadians.</p><p>***</p><p><em>Post by David Wavrock and Roobina Keshishbanoosy, both from Statistics Canada.</em></p><p>Article également disponible en français ici : <a href="https://exp-oeuvre.medium.com/recensement-des-services-gratuits-au-canada-1327c8d1c74a">Recensement des services gratuits au Canada | par L’expérimentation à l’œuvre | Juin, 2021 | Medium</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=15ec2a61071" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Exploring Behavioural Insights At AAFC]]></title>
            <link>https://exp-works.medium.com/exploring-behavioural-insights-at-aafc-21e6ac6e7f88?source=rss-c1fc7bcd7280------2</link>
            <guid isPermaLink="false">https://medium.com/p/21e6ac6e7f88</guid>
            <category><![CDATA[experimentation]]></category>
            <category><![CDATA[behavioral-science]]></category>
            <category><![CDATA[agriculture]]></category>
            <category><![CDATA[assurance]]></category>
            <dc:creator><![CDATA[Experimentation Works]]></dc:creator>
            <pubDate>Mon, 07 Jun 2021 12:54:16 GMT</pubDate>
            <atom:updated>2021-06-16T18:08:53.340Z</atom:updated>
            <content:encoded><![CDATA[<p>At Agriculture and Agri-Food Canada (AAFC), we like to explore options. For instance, how could we apply behavioural insights to public trust policy? In 2019, we saw that the Treasury Board of Canada Secretariat’s Experimentation Works program could help us with this. This was a chance to develop departmental capacity in experimentation. Experimentation could be a way for us to better understand consumers. We could add experimentation to our existing Public Opinion Research (POR) work.</p><p><strong>AAFC’s AgriAssurance Program</strong></p><p>We want a competitive, innovative and sustainable Canadian agriculture and agri-food sector. This sector includes everything between the farmer to the consumer. It starts at the farm and reaches to global markets. All phases of producing, processing and marketing of farm, food and bio-based products are part of agriculture and agri-food. The department has a program, AgriAssurance, that supports industry in developing assurance systems. These systems aim to make how the sector produces foods more transparent. They do this by applying certifications based on standards and verification. To the consumer, these kinds of assurances usually show up as some sort of logo or claim on a food product (e.g. organic, fair trade, etc.).</p><p>When assurance systems are working well, everyone who uses assurances is aware of them, understands them, and can easily get the information they want from them. Strong assurance systems help promote transparency and accountability about methods of production. This makes agricultural processes more clear to consumers. And it helps producers’ credibility.</p><p>Citizens can vote with their wallets for the production methods they most want. Assurances welcome the consumer into the food system, encouraging this stronger relationship between buyers and sellers. For example, you can choose to buy organic-certified meat if you have strong preferences about how your meat comes to your table. The Canadian Organic Standards require that the producer meet certain animal care expectations. This is in addition to adhering to principles of ecology, health, and fairness. So, when you buy organic, that is, verifiable and credible organic claims (e.g. Canada Organic, USDA Organic, Bio Quebec or British Columbia Certified Organic — the list goes on), you know that the way the food was made meets the standard. You are assured.</p><p>Robust assurance systems are democratic, helping consumers identify how to buy in accordance with their values. And, these systems can help make the market economy more responsive. When market economies respond to the demand that consumers ask of them, they are working well.</p><p><strong>Our Challenge</strong></p><p>At AAFC, we know that industry is interested in our AgriAssurance program because we keep getting applications. We can work with applicants, industry stakeholders and provincial and territorial counterparts to improve the process and the program as needs evolve. What we don’t understand quite so well is how well the program meets consumers’ needs.</p><p>Academic literature and POR data can tell us some things about the effectiveness of assurances. This is often based on a concept called “willingness-to-pay”. This measure is a way to order products or their specific attributes by studying which attributes participants value or prefer the most. It allows a standardized format, under specific conditions. But often, willingness-to-pay studies do not account for time pressure, monetary penalties, or other constraints that exist when shopping for food in the real world. So, these studies may not answer public trust policy questions, such as understanding whether or not, and how much, Canadians have confidence that the food they are buying is being produced in the ways they want it to be (e.g., cage-free eggs, sustainably caught, Halal, etc.).</p><p>However, experimentation is challenging. Even just the idea of exploring which research methods *might* work is challenging. We need to practice experimentation, including exploring what options could be considered before deciding to go down the experimentation path. We need to do this not just to learn, but to become comfortable — with the process as well as with the outcomes.</p><p><strong>How Experimentation Could Be Applied</strong></p><p>And here is where experimentation might be useful. We can explore and try out some new ways to get information from people about assurances. But the research landscape is complex — attitudes can be explicit (i.e., self-reported) or implicit (i.e., laying below the level of conscious awareness). Behaviour can be observed or self-reported. This makes it difficult to land on which methods would be best, in trying to answer why someone does or doesn’t trust a product or a logo, and whether that influences their decision to buy it or not. Especially if they don’t even know why, they just know that they do. Or don’t. Trust is a concept that is inherently difficult to measure.</p><p>If we want to understand behaviour, then we need to know more than what someone does — we need to know why they do it. And for that, we need to observe. And ask. And adjust our questions. And dialogue. An experimentation approach helps us to explore what kinds of research methods we should try before investing in larger-scale research.</p><p>We’re looking at developing a multi-method approach that would help distill all of this confusing landscape down into some useful results. This would bring together different research methods that can each individually give us a piece of the puzzle. One method would hopefully get us some “aha” moments from candid reactions about assurance logos on food products. A second method may provide some ideas of whether one assurance feels more credible to someone than another. We’re thinking here about discrete choice models where participants pick from various pairs of products. We’re really interested in identifying if offering some information on one assurance makes any difference in awareness of unrelated assurances.</p><p>None of these methods by themselves will give us enough useful information to make any recommendations to decision-makers. But together, especially if they are combined with POR methods such as surveys and focus groups, they could be powerful and insightful.</p><p>For example, by asking participants about assurances, we could see whether they notice them at all. Or, if they noticed them but don’t know what they mean. Or, if they know what they mean but don’t have any faith in them. Then we’d have some data we could work with. We could even examine whether someone is able to notice differences between the credibility of assurances. Say, a logo that is more rigorous (backed by standards and independently verified), compared to a logo that is less independent, or is held to a lower standard, or is compared to a claim that is unsubstantiated. What if it turns out that, actually, this is one of the things that Canadians are most confused about and really want more clarity on? We’d then have a solid basis for finding ways to address that.</p><p><strong>What We Could Use Results For</strong></p><p>Insights like these could be beneficial for industry, consumers, and the entire Canadian food system. We could improve our assurance systems programming. We could look into raising awareness and answering questions about assurances. We could apply these methods to other topics, other programs, other challenges that we haven’t even thought of yet. Surely there are other things we could start doing in the food system, or stop doing, or do differently, that these methods could help identify.</p><p>***</p><p><em>Post by Rae Payette, senior policy analyst in the Public Trust and Assurance Systems Policy team in AAFC’s Strategic Policy Branch. AAFC’s EW2 project team included Rae Payette (team lead), Inge Vander Horst (senior policy analyst), Tim Rennie (policy analyst), Laura Stortz (economist), Carol Essenburg (senior communications advisor), and others.</em></p><p>Article également disponible en français ici : <a href="https://exp-oeuvre.medium.com/explorer-lapplication-de-l-introspection-comportementale-au-sein-du-minist%C3%A8re-de-l-agriculture-et-7f226a31c297">Explorer l’application de l’introspection comportementale au sein du ministère de l’Agriculture et de l’Agroalimentaire (AAC) | par L’expérimentation à l’œuvre | Juin, 2021 | Medium</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=21e6ac6e7f88" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Laying the foundations for a behavioural intervention project — Part Two]]></title>
            <link>https://exp-works.medium.com/laying-the-foundations-for-a-behavioural-intervention-project-part-two-913f5814034?source=rss-c1fc7bcd7280------2</link>
            <guid isPermaLink="false">https://medium.com/p/913f5814034</guid>
            <category><![CDATA[behavioural-science]]></category>
            <category><![CDATA[experimentation]]></category>
            <category><![CDATA[research]]></category>
            <dc:creator><![CDATA[Experimentation Works]]></dc:creator>
            <pubDate>Wed, 26 May 2021 16:19:04 GMT</pubDate>
            <atom:updated>2021-06-16T18:10:15.304Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>Laying the foundations for a behavioural intervention project — Part Two</strong></h3><p><em>In this </em><a href="https://exp-works.medium.com/laying-the-foundations-for-a-behavioural-intervention-project-part-one-ec237557b28f"><em>blog series</em></a><em>, we will focus on the first two phases of the British Columbia Behavioural Insights Group (BC BIG)’s RIDE Model for Behaviour Shift. This post, the second in the series, will cover the Research phase.</em></p><p><strong>Part 2: Research</strong></p><p>In Part one of this series, we introduced you to the Scoping phase of the British Columbia Behavioural Insights Group (BC BIG’s) RIDE Model for Behaviour Shift. In this section, we now focus on the second phase of the model: the ever-so-important Research phase.</p><figure><img alt="Image of the BC Behavioural Insights Group “RIDE Model for Behaviour Shift” used for project work. Six columns represent the six stages of the project cycle (i.e., Scoping phase, followed by the Research, Innovate, Data collection, and Evaluate phases, and ending with a decision on behalf of the client whether or not to Scale the intervention). The first two phases (i.e., Scope and Research) are highlighted." src="https://cdn-images-1.medium.com/max/1024/1*BRd-7ArfEaZQmTqnwip-ig.jpeg" /></figure><p>While the Research phase follows the Scoping phase in the RIDE Model for behaviour shift, in practice research is often conducted <em>throughout</em> the Scoping phase. Indeed, a certain degree of research can be necessary to uncover the behaviours underlying a policy challenge and to identify potential touchpoints to determine whether a behavioural insights (BI) approach is suitable. However, even once the parameters of a BI project have been scoped, important questions will still remain. The Research phase is dedicated to answering these questions, which typically revolve around gaining a better understanding of the challenge from the perspectives of the people involved. These insights are then used to inform the design of the intervention.</p><p>In this post, we will cover the methods and considerations around conducting research as part of a BI project. This material may be familiar to public servants with a background in research and citizen engagement. However, we hope that our behavioural focus here will make this of interest to everyone. Specifically, this post will cover:</p><ol><li>What is the difference between descriptive and causal research?</li><li>Why is descriptive research important?</li><li>How can we identify which behavioural barriers exist in relation to a policy challenge?</li><li>What is meant by “desk” and “field” research methods?</li></ol><p><strong>What exactly do we mean by “research”?</strong></p><p>The term “research” is very broad and can mean different things to different people depending on the context. With respect to BI, we often think about a distinction between two forms of research: descriptive and causal. As their names imply, descriptive research focuses on <em>describing</em> the behavioural challenge as it currently exists, whereas causal research aims to identify underlying <em>causes</em> of behaviours.</p><p>In the RIDE Model for Behaviour Shift, the Research phase involves descriptive research, whereas the Data and Evaluate phases involve causal research. In a BI project, causal research is typically conducted to empirically evaluate the impact of an intervention. As such, this post focuses on descriptive research approaches.</p><figure><img alt="A three-tiered hierarchy depicts how research can be either Descriptive or Causal and then how Descriptive research can be further broken down into Desk Research or Field Research. Examples of Desk Research include jurisdictional scans and behavioural science literature reviews. Examples of Field Research include focus groups and interviews." src="https://cdn-images-1.medium.com/max/1024/1*atJf5eYx8exlntla6HeZrA.jpeg" /></figure><p><strong>Why conduct descriptive research?</strong></p><p>Descriptive research helps us to better understand the <em>who</em>, <em>what</em>, <em>how</em>, <em>when</em>, and <em>where</em> of a policy challenge. It is about exploring the experiences of users of a service or system to identify how it meets, or misses, their unique needs. It can be helpful to think of descriptive research as a method for challenging assumptions and developing hypotheses. We might think we know why members of a target population are not following through with a specific behaviour. We would do research to check assumptions and develop new hypotheses. For example, we may want to know why high school graduates (target population) do not apply to universities (behaviour) in greater numbers? One current assumption is that they are concerned about the financial requirements. Research may reveal another possible explanation — perhaps they are deterred by a complicated application process. With a clearer sense of barriers and where they show up, teams are then better equipped to develop impactful solutions.</p><p>For example, the Behavioural Insights Team (BIT) in the UK was looking to solve a challenge whereby they sought to increase collections of fines imposed by the courts, considering that a large amount of money was being spent (and charged to defendants) for bailiffs to chase overdue court payments. Before designing the intervention, a researcher from BIT shadowed a bailiff for a day to conduct research with the population of interest. By visiting the homes of the letter recipients, they noticed that in many cases there was mail piling up outside. These individuals received so many pieces of mail, the request for payment was being overlooked. As a result of this user research, they tested a different communication approach: sending a text message with the recipient’s name to people who had failed to pay their fine in order to give them one final chance to pay before issuing a distress warrant for the bailiffs. The result was a significant increase in payments in response to the text, by around ten percentage points.<a href="#_ftn1">[1]</a></p><p><strong>What to look for?</strong></p><p>When thinking about drivers and barriers related to a behaviour, famed behavioural scientist and author Dan Ariely draws on the analogy of a rocket ship. Encouraging a desired behaviour is like launching a rocket into space. There are two key forces that we need to consider:</p><ol><li><strong><em>The Fuel:</em></strong> Things that move someone towards the behaviour, and</li><li><strong><em>Friction:</em></strong> Things that get in the way.</li></ol><p>If you’re looking to change a behaviour — or move the rocket ship — you’ll need to gain a sense of both forces to see what can be adjusted. Unless there are changes in friction or fuel, you tend to stick to the status quo, with the ship going nowhere.</p><figure><img alt="A rocket ship with an arrow going in forward direction (Fuel) and an arrow going in the reverse direction (Friction)." src="https://cdn-images-1.medium.com/max/1024/1*n-nPJ3lT1YEICYUdaNz2hQ.jpeg" /></figure><p>In addition to drivers and barriers, it can be valuable to use descriptive research methods to learn more about:</p><ul><li><strong><em>The decision environment:</em></strong> What is the context in which actors are making decisions or performing the behaviours that underlie the policy challenge?</li><li><strong><em>The touchpoints:</em></strong> What opportunities exist to engage with the population? (e.g., forms, letters, websites, signs, text messages, etc.).</li><li><strong><em>Sentiments and perspectives:</em></strong> How do actors describe their views and feelings related to the policy challenge? Are there any heuristics or psychological biases at play?</li></ul><p><strong>What does descriptive research look like?</strong></p><p>The research method described above (i.e., having a researcher shadow a bailiff for a day to conduct research with the population of interest) is just one example of a research method. Research methods vary depending on the context, population of interest, and available resources (finances and time). Some research activities can be conducted from one’s desk by simply reading through background materials, analyzing available administrative data, and/or consulting the internet. This cost-effective “desk” research can be contrasted with “field” research (sometimes called “user” research), which is typically more involved and can include methods like interviews, focus groups, or surveys. Although field research tends to be more costly and extensive (e.g., recruiting participants and leveraging a skilled facilitator), the insights are often worth this added effort. This research is tailored to the specific challenge or question. And because the project team is in charge of its design, they can ensure accuracy and ethical processes.</p><p>Often at BC BIG, we use a co-design model, whereby we invite our clients to come with us to interview citizens or field staff, so they can hear firsthand about the issues and challenges they face. Even if they don’t participate in the research themselves, hearing about the findings can still be impactful. Sometimes this is the first time they’ve engaged with or heard from the people they serve or the staff that deliver the services and programs they design and oversee.</p><p><strong>Desk research and literature reviews</strong></p><p>Aside from the value of referring to existing administrative data during the Research phase, academic literature is another valuable source of information during this phase. Certainly, the growing trend towards open-source publications has made published academic findings more accessible than ever. Google Scholar can be a great reference, although some articles are still posted behind paywalls. Along with the proliferation of knowledge, however, can come contradictory findings. This is where referring to meta-analyses and systematic reviews — which combine the results of multiple studies — can provide extremely helpful overviews. In addition, unpublished articles such as dissertations, white papers, and cross-jurisdictional scans can be very useful. Related reports from consulting firms, as well as any existing process maps, journey maps and other outputs/artifacts from human-centered designers or other disciplines, can also be useful in helping to understand the context. In addition, searching for relevant articles in community publications, such as <em>Behavioral Scientist</em> and <em>Behavioral Science &amp; Policy Association</em>, can also be a great source of information for the Research phase of a BI project.</p><p>We hope that this two-part series has equipped you with some practical tools to assist with the Scoping and Research phases of a BI project. Happy researching!</p><p>***</p><p><em>Post by Mikayla Ford (Methods Specialist) and Christine Kormos (Senior Behavioural Scientist), both with the BC Behavioural Insights Group (BC BIG). Housed within British Columbia’s Public Service Agency, BC BIG employs a small team of behavioural scientists, innovation methods specialists, and other professionals who, under a consultancy co-design model, collaborate with ministries and academic partners to generate and test simple solutions to policy problems. The team applies a behavioural insights approach to evidence-based policymaking that draws on evidence from the behavioural sciences (psychology, economics, and neuroscience) about how conscious deliberation interacts with nonconscious processes to influence behaviour, and uses that knowledge to design more effective policies, programs and services for citizens and client ministries.</em></p><p>Article également disponible en français ici : <a href="https://exp-oeuvre.medium.com/jeter-les-bases-dun-projet-d-intervention-comportementale-partie-2-200ec3543a31">Jeter les bases d’un projet d’intervention comportementale — Partie 2 | par L’expérimentation à l’œuvre | Mai, 2021 | Medium</a></p><p><strong>References:</strong></p><p><a href="#_ftnref1">[1]</a><a href="https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/60539/BIT_FraudErrorDebt_accessible.pdf">https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/60539/BIT_FraudErrorDebt_accessible.pdf</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=913f5814034" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Laying the foundations for a behavioural intervention project — Part One]]></title>
            <link>https://exp-works.medium.com/laying-the-foundations-for-a-behavioural-intervention-project-part-one-ec237557b28f?source=rss-c1fc7bcd7280------2</link>
            <guid isPermaLink="false">https://medium.com/p/ec237557b28f</guid>
            <category><![CDATA[experimentation]]></category>
            <category><![CDATA[behavioural-science]]></category>
            <category><![CDATA[scoping]]></category>
            <dc:creator><![CDATA[Experimentation Works]]></dc:creator>
            <pubDate>Wed, 26 May 2021 15:23:49 GMT</pubDate>
            <atom:updated>2021-06-16T18:10:50.124Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>Laying the foundations for a behavioural intervention project — Part One</strong></h3><p><em>In this </em><a href="https://exp-works.medium.com/laying-the-foundations-for-a-behavioural-intervention-project-part-two-913f5814034"><em>blog series</em></a><em>, we will focus on the first two phases of the British Columbia Behavioural Insights Group (BC BIG)’s RIDE Model for Behaviour Shift. This post, the first in the series, will cover the Scoping phase. While scoping is certainly nothing new to public servants, there are a few aspects of scoping a behavioural insights (BI) project that make it unique.</em></p><p><strong>Part 1: Scoping</strong></p><p>Behavioural Insights (BI) helps us to understand how people make decisions in everyday life. These insights can be used to design better services and programs. Although the terms BI, “nudging” and “behavioural science” are often used interchangeably, some distinctions exist among these concepts; for this reason, we will focus on “BI” in these posts.</p><p>In this post, we’ll answer a few key questions:</p><ul><li>How do we identify a behavioural challenge in the first place (i.e., what makes it different from other policy challenges?).</li><li>How do we translate a complex policy challenge into a behavioural one using behavioural mapping to identify a target group/behaviour?</li><li>What criteria can we use to select behavioural challenges that are suitable for a BI approach?</li><li>How can we use available information to begin building an “objective statement”?</li></ul><p>As public servants, we tackle all kinds of tricky challenges. Some challenges are structural, financial, or logistical. However, others are behavioural in nature, meaning that their core issue is related to cognitive biases, tendencies, heuristics, habits, etc. It can sometimes be difficult to establish whether the challenge a client ministry is facing is a behavioural one that can be addressed using BI. It is for this reason we devote a fair amount of time to exploring this question during the ‘Scoping’ phase of our projects.</p><figure><img alt="Image of the BC Behavioural Insights Group “RIDE Model for Behaviour Shift” used for project work. Six columns represent the six stages of the project cycle (i.e., Scoping phase, followed by the Research, Innovate, Data collection, and Evaluate phases, and ending with a decision on behalf of the client whether or not to Scale the intervention). The first two phases (i.e., Scope and Research) are highlighted." src="https://cdn-images-1.medium.com/max/1024/1*BRd-7ArfEaZQmTqnwip-ig.jpeg" /></figure><p><strong>Identifying a behavioural challenge</strong></p><p>It can be difficult to discern whether a challenge is truly a behavioural one. The three questions below are a good place to start. If the answer is “no” to all three, then you just might have yourself a behavioural policy challenge:</p><ol><li><em>Is the challenge solely an awareness or educational issue?</em></li></ol><p>If the issue is solely that people don’t know what the optimal behaviour is (e.g., to turn their thermostat to 19–20oC in order to conserve electricity), an awareness or educational approach in the form of a marketing campaign may be required. Although BI can help to inform these efforts, typically this is not the best use of BI tools and methods. Instead, BI works best when people are already aware of the appropriate behaviour, intend to act accordingly, but fail to do so. Research shows this happens all the time,<a href="#_ftn1">[1]</a> which is perhaps unsurprising. We experience it regularly in our day-to-day lives. For example, we may intend to conserve electricity but don’t make the necessary changes to our thermostat regularly. We may intend to exercise but don’t make it to the gym as often as we’d like. And, we may intend to save money but get distracted by purchases in the moment. Rather than improving awareness or knowledge, BI is best-suited to situations where awareness is already high but there is evidence of an intention-action gap. BI is a powerful tool for closing these gaps.</p><p>2.<em> Is the challenge solely about changing attitudes or beliefs?</em></p><p>Just like research has failed to demonstrate a reliable link between intentions and behaviour, evidence suggests that beliefs and attitudes are also poor predictors of behaviour. For example, having a belief that water conservation is important may not be sufficient to drive actual behaviour change for many people. Someone may believe in the importance of water conservation but that belief doesn’t necessarily impact the amount of time they spend in the shower or how frequently they water their lawn. It is for this reason that BI focuses on changing behaviours directly, rather than on changing attitudes or beliefs.</p><p>3.<em> Does the challenge require a “hard” approach, like a change to regulation/legislation?</em></p><p>While a BI lens can help to inform upstream development of legislation and regulation, BI methods and tools are most often leveraged further downstream where the impact of the policy actually reaches individuals. You can think of BI as being one tool in a toolbelt where other policy levers and methods also exist to support governments. BI can help to strengthen compliance with legislation or regulation but cannot be expected to replace it when consequences loom large. For example, BI methods can be used to encourage seatbelt use but laws governing their use are still needed to ensure the safety of citizens.</p><p><strong>Creating a behavioural map of a policy challenge</strong></p><p>Often, “real world questions” — even if we can deduce that they’re behavioural at their core — are still too broad or too complex to work with. In these situations, our first goal is to distill broad policy challenges down into tangible and workable behavioural ones.</p><p>For example, let’s say our goal is to improve the health of children. Of course, there are many behaviours that underlie children’s health, such as vaccination, exercise, healthy eating, etc. We can see how a broad challenge can be broken down into myriad smaller behaviours that could each be the focus of a BI project. In cases such as this, it can be helpful to develop a <em>behavioural map</em>.</p><p>Generally speaking, a behavioural map has three key elements, plotted at different levels:</p><ol><li><strong><em>The goal:</em></strong> What is the overarching objective?</li><li><strong><em>The actors:</em></strong> Who are the key people or groups whose behaviour we would hope to change to reach this policy goal?</li><li><strong><em>The behaviours:</em></strong> What do we need the actors to do to reach the goal?</li></ol><p>If the goal is to improve school children’s health, a behavioural map including parents, children in schools, and schools as the actors, could look like the following:</p><figure><img alt="A sample behavioural map for the goal of improving the health of school children depicts a three-tiered hierarchy with the goal at the top (i.e., improving the health of school children), three key actors (i.e., Parents, Children in schools, and Schools) at the second level, and several behaviours related to this achieving the goal on the third. For example, under the actor “Parents” two behaviours are included: 1) cool healthy dinners and 2) purchase low sugar products." src="https://cdn-images-1.medium.com/max/1024/1*AQe_4vr05dlc-g4lPxhYWQ.png" /></figure><p>And, of course, your actors will change depending on the nature of your goal; for instance, a behavioural map based on the overarching objective to reduce antibiotic resistance may involve completely different actors, including policymakers, clinical professionals, and the public.</p><p><strong>Identifying a target behaviour using the MIST Framework</strong></p><p>In a world of unlimited time, money, and executive support, we could try to <em>change all of the behaviours </em>that populate a behavioural map for a policy challenge. But, more often than not, there is only capacity to address one target behaviour at a time. So how, you may ask, do we choose among the many contenders? Fortunately, there are criteria we can use to help identify those behaviours that lend themselves well (and less well) to a BI project. Sometimes, we may start with a whole host of potential target behaviours and find that — after applying the following set of “MIST” criteria (first put forward by the Ontario Behavioural Insights Unit) — there is really only one potential target behaviour that fits the bill to use in a BI project.</p><p>So, what does MIST stand for?</p><ul><li><strong><em>Measurable:</em></strong> Is the behaviour already recorded as administrative data or could it be recorded easily?</li><li><strong><em>Impactful:</em></strong> Why would changing this behaviour be important for individuals and for government? How much room is there to improve? Would even a small change matter?</li><li><strong><em>Sizable:</em></strong> Is there sufficient sample size to ensure that we can set up a rigorous evaluation (i.e. typically 100s or 1000s, ensuring any differences we’re observing aren’t just happening by chance?)</li><li><strong><em>Touchpoint:</em></strong> Is there an existing or low-cost opportunity to deliver an intervention to the focal population?</li></ul><p>For instance, if we layer the MIST criteria onto the behavioural map we built in the previous section, we might conclude that encouraging high school students to choose healthy school lunches may be our best target behaviour here. This behaviour is <strong>measurable</strong> (inventory at the school cafeteria is tracked via sales records), <strong>impactful</strong> (lunches make up a significant portion of the food students consume in a day), would have a <strong>sizable sample</strong> (a study could include all of the students attending schools with cafeterias), and could have several available <strong>touchpoints</strong> (the cafeteria itself or signage within the school).</p><figure><img alt="The sample behavioural map is shown again with the Ontario’s Behavioural Insights Unit’s “MIST” criteria overlayed on top of the behaviours. For example, all four letters (M, I, S, and T) are noted on the box that depicts children choosing healthy school lunches." src="https://cdn-images-1.medium.com/max/1024/1*G0V_Y42h5U3AvocdevGvoA.jpeg" /></figure><p>Ideally, by the end of the Scoping phase we’ve outlined here, you should be able to answer the following four key questions:</p><ol><li>What behaviour are we trying to change? (behaviour)</li><li>Who is the population of interest? (actors)</li><li>How will we reach the population? (touchpoint)</li><li>How will we measure the behaviour? (data)</li></ol><p>Stay tuned for the next post in this two-part blog series, which will focus on how to identify possible barriers in the way of your goal behaviour — the “R” (or “Research”) phase of BC BIG’s RIDE model.</p><p>***</p><p><em>Post by Mikayla Ford (Methods Specialist) and Christine Kormos (Senior Behavioural Scientist), both with the BC Behavioural Insights Group (BC BIG). Housed within British Columbia’s Public Service Agency, BC BIG employs a small team of behavioural scientists, innovation methods specialists, and other professionals who, under a consultancy co-design model, collaborate with ministries and academic partners to generate and test simple solutions to policy problems. The team applies a behavioural insights approach to evidence-based policymaking that draws on evidence from the behavioural sciences (psychology, economics, and neuroscience) about how conscious deliberation interacts with nonconscious processes to influence behaviour, and uses that knowledge to design more effective policies, programs and services for citizens and client ministries.</em></p><p>Article également disponible en français ici : <a href="https://exp-oeuvre.medium.com/jeter-les-bases-dun-projet-d-intervention-comportementale-partie-1-82903f01568">Jeter les bases d’un projet d’intervention comportementale — Partie 1 | par L’expérimentation à l’œuvre | Mai, 2021 | Medium</a></p><p><strong>References:</strong></p><p><a href="#_ftnref1">[1]</a> Webb, T. L., &amp; Sheeran, P. (2006). Does changing behavioral intentions engender behavior change? A meta-analysis of the experimental evidence. <em>Psychological bulletin</em>, <em>132</em>(2), 249.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ec237557b28f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Increasing Executive Self Identification Rates]]></title>
            <link>https://exp-works.medium.com/increasing-executive-self-identification-rates-b5482a5ba315?source=rss-c1fc7bcd7280------2</link>
            <guid isPermaLink="false">https://medium.com/p/b5482a5ba315</guid>
            <category><![CDATA[forms]]></category>
            <category><![CDATA[government-of-canada]]></category>
            <category><![CDATA[inclusion]]></category>
            <category><![CDATA[identity]]></category>
            <category><![CDATA[experimentation]]></category>
            <dc:creator><![CDATA[Experimentation Works]]></dc:creator>
            <pubDate>Mon, 03 May 2021 19:57:04 GMT</pubDate>
            <atom:updated>2021-06-16T18:11:39.634Z</atom:updated>
            <content:encoded><![CDATA[<h3>What is Self-ID?</h3><p>To meet the needs of Canadians, fostering diversity and inclusion (D&amp;I) in the public service is crucial. Employers that prioritize D&amp;I are better enabled to leverage and incorporate a range of perspectives and expertise that are necessary to address complex workplace and life challenges. Under the <em>Employment Equity Act</em> (<em>EEA</em>), employers under federal jurisdiction are required to correct situations of disadvantage historically experienced by the four designated employment equity (EE) groups (women, Indigenous peoples, persons with disabilities, and members of visible minorities).</p><p>The Office of the Chief Human Resources Officer (OCHRO) is responsible for advancing employment equity in the federal public service. To fulfil this responsibility, the collection of EE data is critical to support in developing programs, policies and interventions that encourage employment equity across the public service. EE data is collected primarily through the completion of self-identification (self-ID) questionnaires by employees who identify with one or more of the EE groups.</p><h3>Problem</h3><p>Increasing diversity and fostering inclusion across the public service, including at the executive ranks, is a key priority for the Government of Canada. Self-ID data at the executive level is crucial for understanding the progress that is being made in representation, hiring, promotion, and separation rates. Diversity in the leadership ranks sets the tone for the rest of the organization and ensures that the whole organization is more inclusive and effective.</p><p><strong>Problem Definition:</strong></p><p>Not all executives (EXs) that may want to self-identify do so, due to the time and effort required to access and complete the form, or because they lack the motivation to do so. ​In behavioural science literature, this is called the <strong>action-intention gap</strong>, which is the difference between what people intend to do, and what they actually do.</p><p>To address the action-intention gap on EX Self-ID, OCHRO launched a behavioural insights experiment. This trial had two objectives:</p><ul><li>To deliver a measurable increase in the proportion of EX’s who completed the self-ID questionnaire in the Executive Talent Management System by the deadline. ​</li><li>To provide evidence-based insights into what works in encouraging self-identification for executives.</li></ul><h3>Behavioural Insights in Action</h3><p>The behavioural insights trial tested two key behavioural science principles: ​</p><ul><li><strong>Messenger Effects </strong>— people respond differently to prompts based on who they are receiving it from. ​</li><li><strong>Social Norms </strong>— individuals are more likely to take an action when they know that many of their peers are also taking that action.</li></ul><p><strong>Trial Overview</strong></p><p>OCHRO developed an email campaign to encourage executives to complete the EE questionnaire and explain the importance of providing EE data.​ The campaign targeted 67 departments/agencies in the core public administration. Departments were randomly assigned to one of five groups, and the executives in each of those groups received different emails encouraging them to Self-ID. One group was a control group which received no communication. The emails came from either the Chief Human Resources Officer or the deputy head of the executive’s department. The text of the emails also differed, with about half of them providing authoritative instructions on self-identification and the other half emphasizing the social benefits of self-identification.</p><h3>Key Insights</h3><p>The trial showed that:</p><ul><li>Providing clear and authoritative instructions to Self-ID was 2.2x as effective than focusing on social norms and benefits.</li><li>Social norms were more effective coming from the Chief Human Resources Officer (CHRO) than from deputy heads.</li><li>A message from the CHRO was more effective than from Deputy Heads<strong>.</strong> 2.1x as many EXs acted before the deadline if they got an email from the CHRO<strong>, </strong>and<strong> </strong>1.4x as many EXs took action after a message from the CHRO.</li></ul><p>***</p><p><em>Post by Haris Khan and McKenzie Krasilczuk, Research and Experimentation Team at OCHRO, Treasury Board of Canada Secretariat</em></p><p>Article également disponible en français ici : <a href="https://exp-oeuvre.medium.com/augmentation-des-taux-de-d%C3%A9claration-volontaire-des-cadres-sup%C3%A9rieurs-1c4aaf58cce4">Augmentation des taux de déclaration volontaire des cadres supérieurs | par L’expérimentation à l’œuvre | Mai, 2021 | Medium</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b5482a5ba315" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Mitigating Bias: Integrating GBA+ in the Research Process]]></title>
            <link>https://exp-works.medium.com/mitigating-bias-integrating-gba-in-the-research-process-1fb334e954bc?source=rss-c1fc7bcd7280------2</link>
            <guid isPermaLink="false">https://medium.com/p/1fb334e954bc</guid>
            <category><![CDATA[experimentation]]></category>
            <category><![CDATA[research]]></category>
            <category><![CDATA[inclusion]]></category>
            <category><![CDATA[diversity]]></category>
            <dc:creator><![CDATA[Experimentation Works]]></dc:creator>
            <pubDate>Fri, 16 Apr 2021 19:34:11 GMT</pubDate>
            <atom:updated>2021-06-16T18:12:24.631Z</atom:updated>
            <content:encoded><![CDATA[<p>When should a project team reflect on the implications of different types of bias in a research project? As Experimentation Works 2 (EW2) experts, we asked our project teams this question in a recent learning session that discussed how to mitigate bias in research.</p><p>Bias can affect any phase of the research process; for example, failing to consider the unique experiences and inequities faced by sub-groups in the population or insufficient sampling of marginalized groups, despite being members of the population of interest. Historically, bias has contributed to an underrepresentation of marginalized groups in research and “solutions” that differentially benefit (or disadvantage) varying sub-groups of the population.</p><p>There is no perfect solution for solving bias. But there are many ways an individual or project team can reflect, challenge assumptions, and design their data collection and analysis practices to avoid perpetuating bias in their research. We developed a learning event for the EW2 cohort to outline how Gender-Based Analysis Plus (GBA+) can be integrated throughout the research process and to create space for project teams to reflect on how GBA+ can be incorporated in their projects.</p><figure><img alt="This figure illustrates some of the factors which can intersect with sex and gender. Six oblong shapes of differing colors overlap and fan out. Each oblong has two identity factors written on it. The middle oblong has the word “sex” written on the left and “gender” written on the right. From the word “sex”, in a clockwise order, other identity factors are written: language, ethnicity/race, religion, age, disability, geography, culture, income, sexual orientation, and education." src="https://cdn-images-1.medium.com/max/750/1*jVtSt5DKL9A3fkXLeRdRvg.png" /><figcaption>Drawing on the insights of <a href="https://www.researchgate.net/profile/Olena-Hankivsky/publication/279293665_Intersectionality_101/links/56c35bda08ae602342508c7f/Intersectionality-101.pdf">intersectionality</a>, <a href="https://cfc-swc.gc.ca/gba-acs/index-en.html">GBA+</a> is a tool the Government of Canada uses to reduce bias in decision making by examining how aspects of identity (e.g., gender, race, ethnicity, social class, and disability), and intersections among these identities (e.g., gender X race, race X social class), influence experience with policies, programs, and initiatives. Photo credit: <a href="https://women-gender-equality.canada.ca/en/gender-based-analysis-plus/government-approach.html">Government of Canada</a></figcaption></figure><p>Our learning event focused on applying GBA+ to quantitative research because of the experimentation focus of EW2. The <a href="https://wiki.gccollab.ca/images/8/81/3._September_2020_%28EN%29_GBA%2B_in_Research_-_EN_-_FINAL.pdf">presentation portion of the learning event</a> walked participants through ways to apply GBA+ across the stages of the research process, including questions for researchers to consider at these different stages. In what follows, is a summary of these approaches and questions.</p><p><strong>Identifying the problem</strong></p><p>Before commencing a research project, those involved in the planning should consider the composition of the research team or advisory board. Given the influence of bias in research, collaboration and consultation with those who can bring intersectional perspectives to the research, because of their lived experience and/or expertise in the area, can be critical to producing meaningful research that is beneficial to the communities included in or affected by the research.</p><p>When defining the problem at the outset of a research project, Cole (2009) suggests integrating intersectionality by asking three questions:</p><p>1. <em>Who is included within this category</em>? Reflecting on who is included within a social category can help to prevent researchers from focusing on only certain, often privileged, subgroups and ensure representation that reflects the diversity of the social category.</p><p>2. <em>What role does inequality play?</em> Sub-groups of people within a social category will have different historical and current experiences of inequality. Considering this, and the social processes that contribute to this inequality, can help to shape the types of problems under examination in the research.</p><p>3. <em>Where are there similarities?</em> Looking at commonalities across different sub-groups can help researchers identify similarities in experiences that are influenced by the broader culture, social structures, and human behaviour.</p><blockquote><strong>Additional GBA+ questions to consider during problem definition:</strong></blockquote><blockquote>• Who is the research for and does it advance the needs of those under study? Will the research have unintended impacts/consequences for those not specifically targeted?</blockquote><blockquote>• Which groups should be consulted (e.g., internal and external stakeholders, subject matter experts, those with lived experience) to inform problem definition and other aspects of the research process?</blockquote><p><strong>Literature Review</strong></p><blockquote><strong>GBA+ questions to consider when conducting a literature review:</strong></blockquote><blockquote>• Does the literature review reveal gaps in the representation of the experience of different groups of women, men, and gender-diverse people?</blockquote><blockquote>• Are there biases in the literature that perpetuate stereotypes?</blockquote><blockquote>• Might the findings obtained in the literature yield different results if (other) marginalized groups were considered?</blockquote><p>A traditional literature review that incorporates keywords relevant to the subject matter may be insufficient for addressing the above GBA+ questions and for understanding the unique experiences and challenges of different sub-groups. Literature <a href="https://study.sagepub.com/sites/default/files/Phelps%20et%20al%20-%20Ch7.pdf">search strategies</a>, such as including intersectional terms through Boolean operators (e.g., subject matter AND race AND gender), and writing literature reviews that incorporate intersectional research results and/or identifies gaps in the consideration of intersectionality in the literature, help to integrate GBA+ in literature reviews.</p><p><strong>Sampling &amp; Measurement</strong></p><p>Sampling randomly from a population often does not produce a large enough sample size of low-frequency groups to perform quantitative analyses. For this reason Else-Quest &amp; Hyde (2016) recommend <a href="https://www.povertyactionlab.org/resource/randomization">stratified random sampling</a> or <a href="https://www.betterevaluation.org/en/rainbow_framework/describe/sample">purposive sampling</a>.</p><blockquote><strong>GBA+ questions related to sampling:</strong></blockquote><blockquote>• Is the sample representative of the experiences of diverse groups of people for whom the issue under study is relevant?</blockquote><blockquote>• Does the sampling approach reinforce traditional notions of representation that focus on most commonly shared experiences?</blockquote><blockquote>• Are there barriers to participation for under-represented groups (e.g. access to internet, shift work)?</blockquote><p><a href="https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3293252/#:~:text=Conceptual%20equivalence%20is%20hence%20defined,the%20same%20meaning%20across%20groups.&amp;text=It%20is%20a%20form%20of,groups%20via%20a%20particular%20instrument.">Conceptual equivalence</a> is an important consideration when it comes to measurement. It refers to whether the measure being used taps into the same construct across groups (Else-Quest &amp; Hyde, 2016). For example, a translated instrument, such as a scale appearing in a survey, may contain terms that are semantically similar across languages. However, subtle differences in a term’s connotations can systematically influence response patterns across linguistic groups. If a tool being applied in a research study is not equivalent for all sub-groups within the sample, it is not possible to decipher whether the results obtained are due to conceptual inequivalence of the tool or more substantive differences between groups. One way to ensure that a tool being applied is equivalent across different groups is by assessing <a href="https://www.psychologicalscience.org/observer/testing-for-measurement-invariance">measurement invariance</a>.</p><blockquote><strong>GBA+ questions about measurement:</strong></blockquote><blockquote>• Does the data collection strategy provide opportunity for expression of diverse experiences and perspectives?</blockquote><blockquote>• Does the methodology address the gaps identified in the literature review?</blockquote><p><strong>Analyzing &amp; Interpretation</strong></p><p>There are several quantitative analytical approaches that may be suited to exploring intersectional differences. Depending on the research question and available data (e.g., sample size, type of data), some options include factorial design, multiple regression with moderators, multi-level modeling, and moderators in meta-analysis. Regardless of the analytical approach applied, the intersectional analysis undertaken should be grounded in an explicit rationale that considers historical and current inequality experienced by the groups being compared. Without this linkage, the results lack context and meaning.</p><blockquote><strong>GBA+ questions relevant to analysis and interpretation of results:</strong></blockquote><blockquote>• Based on the results, are there implications for diverse groups of men and women and the relationships among them?</blockquote><blockquote>• What limitations are there for the interpretation of results if there was unequal sampling or limited representativeness of diverse groups?</blockquote><blockquote>• Do power and inequality contribute to the findings? If so, how?</blockquote><blockquote>• Are gender roles or other identities of subpopulations presented in absolute terms?</blockquote><p>This brings us back to the question, when should a project team reflect on the implications of different types of bias in a research project? We hope, after reflecting on the above, that it is no longer a matter of when but a matter of how and where. To help get you started, the activity sheets from the learning event are available <a href="https://wiki.gccollab.ca/images/c/c7/3._Sept._2020_%28EN%29_GBA%2B_in_Research_Handouts.pptx">here</a>. Use the questions provided for projects in the exploratory and/or experimental phase to consider how and where an intersectional lens can be applied throughout your research. We hope these questions and materials will help you explore tangible ways you can deliver research while understanding the impacts bias can play throughout your work.</p><p>***</p><p><em>Post by Sara Rubenfeld, PhD, Defence Scientist, Department of National Defence and Greg White, Project Manager with Canada’s Free Agents, currently posted at Employment and Social Development Canada</em></p><p>Article également disponible en français ici :<em> </em><a href="https://exp-oeuvre.medium.com/att%C3%A9nuation-des-biais-int%C3%A9grer-lacs-au-processus-de-recherche-6a05f7f6e6df">Atténuation des biais : Intégrer l’ACS+ au processus de recherche | by L’expérimentation à l’œuvre | Avril, 2021 | Medium</a></p><p><strong>References &amp; Resources:</strong></p><p>Better Evaluation (n.d.). <em>Sample</em>. <a href="https://www.betterevaluation.org/en/rainbow_framework/describe/sample">https://www.betterevaluation.org/en/rainbow_framework/describe/sample</a></p><p>Bowleg, L. (2008). When Black + Lesbian + Woman ≠ Black Lesbian Woman: The Methodological Challenges of Qualitative and Quantitative Intersectionality Research.<em> Sex Roles, 59,</em> 312–325.</p><p>Bowleg, L., &amp; Bauer, G. (2016). Invited reflection: Quantifying intersectionality. <em>Psychology of Women Quarterly</em>, <em>40</em>(3), 337–341.</p><p>Canada School of Public Service (2020).<em> Removing Bias and Building Trust in Your Data. </em>Retrieved from<em> </em><a href="https://www.youtube.com/watch?v=8bOFIkX213s."><em>https://</em>www.youtube.com/watch?v=8bOFIkX213s.</a></p><p>Cole, E. R. (2009). <em>Intersectionality and Research in Psychology, American Psychologist, 64</em>(3), 170–180.</p><p>Else-Quest, N. M., &amp; Hyde, J. S. (2016). Intersectionality in quantitative psychological research: II. Methods and techniques. <em>Psychology of Women Quarterly</em>, <em>40</em>(3), 319–336.</p><p>Hachey, K., Bryson, R., &amp; Davis, K. (2016). <em>Defence science research and Gender-Based Analysis+: Shaping the research landscape</em>. Director General Military Personnel Research and Analysis Scientific Poster DRDC 2016. Defence Research and Development Canada.</p><p>Hankivsky, O. (2014). <em>Intersectionality 101.</em> The Institute for Intersectionality Research &amp; Policy. <a href="https://www.researchgate.net/profile/Olena-Hankivsky/publication/279293665_Intersectionality_101/links/56c35bda08ae602342508c7f/Intersectionality-101.pdf">https://www.researchgate.net/profile/Olena-Hankivsky/publication/279293665_Intersectionality_101/links/56c35bda08ae602342508c7f/Intersectionality-101.pdf</a></p><p>Hankivsky, O., &amp; Mussell, L. (2018). Gender-based analysis plus in Canada: Problems and possibilities of integrating intersectionality. <em>Canadian Public Policy, 44(4)</em>, 303–316.</p><p>Harachi, T.W., Choi, Y., Abbott, R.D., Catalano, R.F., &amp; Bliesner, S.L. (2006). Examining equivalence of concepts and measures in diverse samples. <em>Prevention Science, 7(4)</em>, 359–368. <a href="https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3293252/#">https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3293252/#</a></p><p>Kopper, S., &amp; Sautmann, A. (n.d.). <em>Randomization</em>. Abdul Latif Jameel Poverty Action Lab (J-PAL). <a href="https://www.povertyactionlab.org/resource/randomization">https://www.povertyactionlab.org/resource/randomization</a></p><p>Lee, S.T.H. (2018, September 18). <em>Testing for measurement invariance: Does your measure mean the same thing for different participants?</em> Association for Psychological Science. <a href="https://www.psychologicalscience.org/observer/testing-for-measurement-invariance">https://www.psychologicalscience.org/observer/testing-for-measurement-invariance</a></p><p>Malone, H., Nicholl, H., &amp; Tracey, C. (2014). Awareness and minimisation of systematic bias in research. <em>British Journal of Nursing, 23(5)</em>, 279–282.</p><p>Potochnik, A. (2020, August 9). <em>Awareness of our biases is essential to good science. </em>Scientific American. <a href="https://www.scientificamerican.com/article/awareness-of-our-biases-is-essential-to-good-science/">https://www.scientificamerican.com/article/awareness-of-our-biases-is-essential-to-good-science/</a></p><p>Women and Gender Equality Canada (2017). Gender-based analysis plus course. Retrieved from <a href="https://cfc-swc.gc.ca/gba-acs/course-cours-en.html">https://cfc-swc.gc.ca/gba-acs/course-cours-en.html</a></p><p>Women and Gender Equality Canada (2017). GBA+ Research Guide. Retrieved from <a href="https://cfc-swc.gc.ca/gba-acs/guide-en.">https://cfc-swc.gc.ca/gba-acs/guide-en.</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1fb334e954bc" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Reflections on 2020 and the Last Stretch of EW2]]></title>
            <link>https://exp-works.medium.com/reflections-on-2020-and-the-last-stretch-of-ew2-e145cc911df0?source=rss-c1fc7bcd7280------2</link>
            <guid isPermaLink="false">https://medium.com/p/e145cc911df0</guid>
            <category><![CDATA[experimentation]]></category>
            <category><![CDATA[reflections]]></category>
            <category><![CDATA[covid19]]></category>
            <dc:creator><![CDATA[Experimentation Works]]></dc:creator>
            <pubDate>Thu, 11 Feb 2021 15:03:05 GMT</pubDate>
            <atom:updated>2021-06-16T18:13:24.397Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*v1Aj60z8JTfaID7R3aD8Sg.jpeg" /><figcaption>Credit: <a href="https://unsplash.com/@omgitsyeshi">Yeshi Kangrang</a></figcaption></figure><p>Happy New Year! As we move into the last stretch of Experimentation Works 2 (EW2), we thought we’d share some reflections. This is normally a good opportunity to take stock and assess what is working, what isn’t, and adjust as needed: the same principles that underpin experiments. This year, like everyone else, our project teams and all those supporting them have already had to recalibrate, adjust, and evolve on the fly due to the pandemic.</p><p><strong>Maintaining momentum amidst change</strong></p><p>So, where are we? Since May, we have connected with the full EW2 Cohort through a total of 21 virtual events. These events included monthly cohort check-ins, project presentations, and monthly learning events. We also launched a fall series focused on reviewing early modules of the experimentation course being developed by EW2 experts and our team. Our learning event topics ranged from design choice, GBA+, ethics and featured experiments (materials <a href="https://wiki.gccollab.ca/Experimentation_Works_Learning_Resources">accessible here</a>). The course material will also be available in the coming months.</p><p>Several of our projects had to withdraw due to changed priorities. That said, a full 9 are making progress, though many with changes to scope or timing. We still have over 30 experts involved in a variety of ways. These experts have been instrumental in supporting projects, hosting learning events, and helping us develop the experimentation course mentioned above.</p><p><strong>Making our materials publicly accessible</strong></p><p>We’ve also posted all EW-related materials on a new publicly-accessible <a href="https://wiki.gccollab.ca/Experimentation_Works">GCWiki page</a>. This resource provides an overview of EW as well as detailed links and resources for both Cohorts (EW1: 2018–2019; EW2: 2020–2021). Our GCWiki also includes a link to <a href="https://create.piktochart.com/output/50062981-ew2-at-a-glance-draft">our new infographic</a>, with a few screenshots below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*V6M2_MY618hvCjxtGIA6cA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-nmSsVi1WV0Oa5INIfXh6g.png" /><figcaption><em>Above: snapshot of our EW2 Cohort including the 9 featured projects. You can view the whole infographic </em><a href="https://create.piktochart.com/output/50062981-ew2-at-a-glance-draft"><em>here</em></a><em> and read more about EW2 and related projects </em><a href="https://wiki.gccollab.ca/Experimentation_Works"><em>here</em></a><em>.</em></figcaption></figure><p><strong>Pain points and challenges</strong></p><p>Running EW2 has given us a unique perspective into the experimentation journey. This includes seeing common pain points and challenges. For example, exploratory work is essential in setting the groundwork for a successful experiment. And this can take a lot more time than anticipated. Also, experimentation can sometimes be confused with public opinion research. If definitions between these two things are not clear, teams can spend more time having conversations about this than is ideal. Lastly, teams seeking to contract research and experimentation expertise would benefit from more support and examples of success in this space. This includes the need for newer research methods, such as ethnography, to be recognized and supported.</p><p>How are we supporting these challenges? EW2 was always designed to support pre-experimental research as well as experimental research. As such, our model is flexible by design. We can easily pivot our resources to support and showcase the pre-experimental work as much as the experimental. To help reduce the confusion around public opinion research, we also worked with TBS colleagues to <a href="https://www.canada.ca/en/treasury-board-secretariat/services/government-communications/public-opinion-research-government.html">update the definition of public opinion research on Canada.ca</a>. This definition now includes experimental research and what distinguishes it from public opinion research.</p><p><strong>Experimentation and Covid-19</strong></p><p>A final thought: the Covid-19 pandemic has demonstrated the value of the experimental approach. At least two of our EW2 experts have been re-deployed to support an evidence-based response to the pandemic. All 9 project teams are building skills and gaining experience that will support future research and experimental approaches in government. Together, we are learning how to collaborate across departmental boundaries and use our cross-functional diversity of skill-sets to tackle complex problems. The pandemic has also shown us a wide range of issues that will require research and evidence to address. To respond, the public service will need to continue to rapidly evolve and find new ways of working. This means there is no shortage of areas in need of rigorous testing to see what works!</p><p><strong>Keep reading and stay tuned!</strong></p><p>As we go into the final months of Experimentation Works 2, we’re also thinking about what’s next for EW and how we can keep supporting experimentation across the public service. We’ll have more to share on this soon. In the meantime, be sure to read the past and upcoming posts on individual projects. They each have their own story to share!</p><p>* * *</p><p><em>Post by the TBS Experimentation Team (Nick Chesterley, Pierre-Olivier Bédard, Sarah Chan, Alana Couvrette)</em></p><p>Article également disponible en français ici : <a href="https://exp-oeuvre.medium.com/r%C3%A9flexions-sur-2020-et-le-dernier-tron%C3%A7on-deo2-8bb9f260cf08">Réflexions sur 2020 et le dernier tronçon d’EO2 | par L’expérimentation à l’œuvre| Février, 2021 | Medium</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e145cc911df0" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>