CHAPTER FIVE

Agency Self-Evaluation

When you think about agency self-evaluation, it may be tempting to reflexively focus exclusively on data and measurement. Although this is certainly the core focus for self-evaluation, it is far too simplistic to serve the complex interests of contemporary child protection agencies. Keeping in mind that data and measurement are the core of evaluation, first consider evaluation at the most abstract level.

Evaluation of any type is based on understanding the current state of something. If you want to evaluate “X,” you must start by understanding the current state of “X.”

Suppose that you want to evaluate the investigation services of your agency. To do this, you start by understanding the current state of those services. What elements are important? How can you tell how well you are doing? For example, do workers always see the reported child victim face-to-face when doing an investigation? You can then develop a strategy to determine whether and how frequently reported child victims are seen face-to-face.

You have decided that seeing the reported child victim face-to-face is an important element in your agency’s investigation service. You have, for instance, determined that this happens 87% of the time. From this you can conclude that in terms of seeing reported child victims, the current state of investigation services is at 87%. From here, you can then move on to determine the current state for other elements that you believe are important aspects of investigation services. Through this process, you come to understand the current state of “X” investigation services.

Once you have determined the current state of whatever you choose to evaluate, there are then two possible evaluation strategies. You can determine how “X” compares with “Y” or you can determine change in “X,” if any, during some time interval. The first strategy may be thought of as “comparison evaluation” and the second may be thought of as “change evaluation.”

Most typically, comparison evaluation focuses on the state of “X” in comparison to some fixed standard. For example, you could evaluate aspects of your agency in comparison to best-practice standards such as those from the Council On Accreditation of Services for Children and Families (www.COANET.org), The Child Welfare League of America (www.CWLA.org), or those from the Public Children Services Association of Ohio (www.PCSAO.org). At a less inclusive level, you could determine the average caseload for your agency’s ongoing workers as compared to, for instance, a standard of fourteen cases per worker.

In addition to evaluating “X” in comparison to a standard, you can evaluate “X” compared to something else that may change or is changing. For example, you might be interested in how quickly children are started in the new school when there is a placement change that causes a change in the school they attend. More specifically, you may be interested in whether workers do as well with getting children in school in some school districts as compared to others. You want to look at the time interval as the school district changes. This would let you compare school districts in terms of the time interval for getting children started in a new school. You could then plan intervention strategies based on this comparative evaluation.

Comparative evaluation is always a point-in-time activity where as change evaluation is an over-time process. For example, is the COA or PCSAO standard being met today or, using the above example, what is the time interval today for children starting school for each school district in which you are interested? Alternatively, change evaluation is always interested in the direction of change and in the rate or amount of change over-time (and usually both). For example, suppose you want to evaluate how well agency social workers are doing with keeping up with paperwork. You first establish the baseline that shows that, on average, workers are submitting paperwork 12.2 days after the event for which the paperwork is required. You then add an additional case aide for each fifteen social workers, assuming that this will free social work time for paperwork. Over the next six weeks, the submission average drops to 11.9 days. Has adding case aides helped? The direction of change (less time) was in the desired direction but the rate or amount of change was very low. In fact, it was so low that adding the case aide may not have had anything significant to do with the change. Based on this, you cannot tell whether the intervention helped or not. As you work on reducing paperwork submission intervals trying various strategies, you will surely need to continuously evaluate the direction and amount of change.

From the above you can see that, although data and measurement are important, the real key to evaluation is to develop appropriate strategies that either focus on comparison or change. This chapter focuses on agency self-evaluation and presents four strategies you can consider. You will see that each strategy approaches self-evaluation from a different perspective and uses quite different techniques. You will also see that implied in the strategies is this simple point: evaluation, whatever the strategy, is only useful to the extent that it leads to action or intervention. If nothing is to be done as a result of the evaluation, there was little point in evaluating.

The first strategy discussed below identified an agency deficit in staff understanding of agency values and philosophy as well as a deficit in individual leadership knowledge and skills. The intervention partially corrected the deficits while concurrently monitoring the levels of change among staff members in relation to the deficits.

The second strategy recognized a deficit in factual knowledge about the beliefs and perceptions of staff, clients, and others in the community about the agency and initiated an ongoing process to capture good data about those beliefs and perceptions. The third strategy was initiated by the Ohio Department of Job and Family Services and recognized a deficit related to statewide data about those outcomes most valued by the state in child protection. The intervention was designed to provide county-level data that LCCS uses to understand its performance and to use as input for its continuous quality improvement processes.

The fourth strategy recognizes that LCCS is impression-rich and data-poor. The intervention incorporates activities and outcomes into the agency’s strategic plan to move toward becoming more data-driven and fact-based in relation to agency planning and management.

Whether you use these or other strategies, the point is this. No agency can successfully conduct its business in the contemporary child protection environment without a commitment to and strategies for self-evaluation. It has become a minimum, bottom-line expectation of many agency stakeholders and virtually all agency authorizers.

The next section discusses how evaluation and intervention work together in a context that may not at first seem to relate to agency self-evaluation. As you consider it further, though, you will see that agency self-evaluation is at the essence of this leadership development initiative.