Computational Policy Lab
June 19, 2018
In the state of California, thousands of people are in jail because they can’t afford their bail. Senate Bill 10 will heavily reduce the state’s use of money bail and will mandate the use of risk assessments in pretrial release decisions.
Risk assessments can allow for more defendants to be released pretrial while maintaining public safety. While we support this effort, we recommend changes to the language in SB 10 to promote the proper use of pretrial risk assessments. Specifically, SB 10 should be amended with respect to the design, deployment, and validation of pretrial risk assessments.
Pretrial risk assessments are questionnaires that gather information about a defendant’s criminal and/or social history and use this information to output a risk score. Pretrial risk assessments intend to predict what a defendant would do if released before their trial: specifically, whether they will fail to appear in court and whether they are a risk to public safety. For each question used on the risk assessment, there is a demonstrated empirical relationship between a defendant’s answer and their future risk. [Read more…]
We recommend that SB 10 address each of the following points to ensure best practices for risk assessment tools.
Issue: Some risk assessments currently in use in California produce a single score for a defendant’s risk of “any failure,” including both failing to appear in court and committing a new crime. Using a single risk score to predict multiple distinct outcomes can be misleading, since a person who is at risk of failing to appear in court has different needs from a person likely to commit a new crime if released.
Recommendation: SB 10 should prohibit risk scores that combine different outcomes, and instead require a separate score for every outcome being predicted. This will allow judges to better gauge a defendant’s risk to public safety and determine what level of pretrial supervision is appropriate. [Read more…]
Issue: SB 10 says that risk assessments must be “equally accurate” across genders, but forbids assessments from considering gender as a factor when determining risk. In practice, risk assessments sometimes overestimate the risk of female defendants. In this case, equal accuracy is best achieved by using gender-specific risk assessments.
Recommendation: SB 10 should allow risk assessments to consider gender when necessary to ensure equal accuracy across genders. [Read more…]
Issue: If judges make a decision contrary to a risk assessment’s recommendation, SB 10 appropriately requires them to document the reason for their disagreement (SB 10, Section 17 §1275(c) (2017)). However, there is concern that this requirement will not be included in the final version of the bill.
Recommendation: SB 10 should continue requiring judges to provide a written statement of reasons in any case where their release decision deviates from the recommendation of the risk assessment. This statement of reasons will protect defendants’ due process rights and could be used to help improve risk assessments over time. [Read more…]
Issue: SB 10 currently only requires counties to report aggregate statistics, such as the number of defendants detained. This information is not enough to properly test risk assessments for bias and accuracy.
Recommendation: SB 10 should mandate the collection and reporting of case-specific information, including — for each defendant — their demographics, risk assessment, and release decision. [Read more…]
Pretrial risk assessments are questionnaires that gather information about a defendant’s criminal and/or social history and use this information to output a risk score. Pretrial risk assessments intend to predict what a defendant would do if released before their trial: specifically, whether they will fail to appear in court and whether they are a risk to public safety. For each question used on the risk assessment, there is a demonstrated empirical relationship between a defendant’s answer and their future risk.
Below are two risk assessments used in California. The first predicts a defendant’s likelihood of failing to appear in court by considering factors like prior convictions and whether the person has previously failed to appear in court. The second predicts a defendant’s risk of committing a new violent crime if released, based on criminal history, age, and current offense.
Consider a hypothetical defendant who has just been arrested on a drug charge. The defendant is 25 years old, has two prior drug convictions, and has failed to appear in court pretrial twice in the past.
On the first risk assessment, the defendant would get one point for their prior convictions and four points for their history of failing to appear. This person would have a total score of five out of seven on this scale, rating them as having a high risk of failing to appear in court if released.
On the second scale, this same defendant would only get one point for their prior conviction. With a risk score of one out of seven, this person would be rated as low risk of committing a new violent crime.
|
Failure to appear (maximum total weight = 7 points) |
|
|---|---|
| Risk Factor | Weights |
| Pending charge at the time of the offense |
No
0
Yes
1
|
| Prior Conviction |
No
0
Yes
1
|
| Prior failure to appear pretrial in past 2 years |
0 failures
0
1 failure
1
2+ failures
4
|
| Prior failure to appear pretrial older than 2 years |
No
0
Yes
1
|
|
New Violent Criminal Activity (maximum total weight = 7 points) |
|
|---|---|
| Risk Factor | Weights |
| Current violent offense |
No
0
Yes
2
|
| Current violent offense & 20 years old or younger |
No
0
Yes
1
|
| Pending charge at the time of the offense |
No
0
Yes
1
|
| Prior conviction |
No
0
Yes
1
|
| Prior violent conviction |
No convictions
0
1 or 2 convictions
1
3+ convictions
2
|
Issue: Some risk assessments currently in use in California produce a single score for a defendant’s risk of “any failure,” including both failing to appear in court and committing a new crime. Using a single risk score to predict multiple distinct outcomes can be misleading, since a person who is at risk of failing to appear in court has different needs from a person likely to commit a new crime if released.
Recommendation: SB 10 should prohibit risk scores that combine different outcomes, and instead require a separate score for every outcome being predicted. This will allow judges to better gauge a defendant’s risk to public safety and determine what level of pretrial supervision is appropriate.
SB 10 requires that pretrial risk assessments measure both a defendant's probability of failing to appear and their risk to public safety. In light of these mandates, we recommend the bill require risk assessments to produce at least two risk scores: one for a defendant's risk of failing to appear in court, and another for a defendant's risk of committing a new violent or otherwise serious crime if released. Providing separate scores will give judges more specific information about the risks and needs of individual defendants.
Although it is common for existing risk assessment tools to predict the likelihood of any new crime, we believe that limiting to serious crime better aligns with SB 10's goal of assessing a defendant's risk to public safety. Further, for certain types of minor street crime, data on arrests and convictions may be poor proxies for actual criminal behavior. For example, even though rates of marijuana use are similar across races, black and Hispanic individuals are charged for marijuana use at far higher rates than whites. As a result, risk assessments that attempt to predict one's likelihood of engaging in any new criminal activity may incorrectly rate minorities as higher risk than similarly situated whites.
Issue: SB 10 says that risk assessments must be “equally accurate” across genders, but forbids assessments from considering gender as a factor when determining risk. In practice, risk assessments sometimes overestimate the risk of female defendants. In this case, equal accuracy is best achieved by using gender-specific risk assessments.
Recommendation: SB 10 should allow risk assessments to consider gender when necessary to ensure equal accuracy across genders.
SB 10 prohibits the use of gender in risk assessments, but says that assessments must be “equally accurate” across genders (SB 10, Section 38 §1318.3(b) (2017)). This language, designed to protect against discrimination, may contradict itself in practice. Risk assessments that do not include gender often overestimate female defendants’ risk of committing new crimes compared to male defendants. Our analysis of pretrial release data from Florida’s Broward County demonstrates this.
Broward County uses the COMPAS risk assessment tool, which does not include gender as a factor when calculating risk. The chart below shows that for women and men given the same risk score by the COMPAS risk assessment, men have higher rates of recidivism than women. Women with a risk score of seven, for example, reoffend about 50% of the time, whereas men with a risk score of seven reoffend more than 60% of the time.
Consequently, it is at times impossible for a risk assessment to be “equally accurate across all genders” when the use of gender is prohibited. To fix this and ensure gender equality, some risk assessments (including a tool used for sentencing in Wisconsin) use separate risk scales for men and women.
New Jersey’s Bail Reform Law allows for risk assessments to gather information about race, gender, and socio-economic status, but says that release recommendations can’t be discriminatory based on those factors. Research has shown that other demographic factors, such as race, do not need to be included in risk assessments to achieve equally accurate risk scores. These factors could continue to be excluded by the bill.
Issue: If judges make a decision contrary to a risk assessment’s recommendation, SB 10 appropriately requires them to document the reason for their disagreement (SB 10, Section 17 §1275(c) (2017)). However, there is concern that this requirement will not be included in the final version of the bill.
Recommendation: SB 10 should continue requiring judges to provide a written statement of reasons in any case where their release decision deviates from the recommendation of the risk assessment. This statement of reasons will protect defendants’ due process rights and could be used to help improve risk assessments over time.
Judges should have discretion to override pretrial risk assessments when they disagree with the assessment’s recommendations. As currently drafted, the requirement for judges to document their disagreement from risk assessments will help SB 10 achieve its goal of reducing pretrial detention and will improve evidence-based practices going forward. Some, however, are concerned that this requirement represents a check on judicial authority, and so there is worry it will be excluded from the final bill. On the contrary, such a record will promote transparency and may improve risk assessments based on judge input. This statement of reasons could provide feedback for tool creators to revise the risk assessment or create a more rigorous matrix for translating risk scores into release decisions.
Issue: SB 10 currently only requires counties to report aggregate statistics, such as the number of defendants detained. This information is not enough to properly test risk assessments for bias and accuracy.
Recommendation: SB 10 should mandate the collection and reporting of case-specific information, including — for each defendant — their demographics, risk assessment, and release decision.
Validation of risk assessments can be costly and statistically challenging, and the bill’s current reporting requirements would not provide enough data for validation by the Judicial Council or a third party. To enable proper validation, each county should be required to report, for every defendant:
If possible, counties should also collect this information about defendants between the time the bill is passed and the time it goes into effect on January 1, 2020. Collection of data in this time frame will help measure the difference made by SB 10 in each county.
