Update here (1/2).

What Happened With Tyler Technologies

First off, thanks for taking the time to read this. If you're here, you're probably already familiar with what's happened. If not, check out here.

If you're already familiar with all that or aren't interested in reading through it, here's the setup:

And I run judyrecords. Holy. Shit.

I see these around 11pm, along with several other news websites already picking up the story, immediately stop browsing reddit, and start rifling through a temp file for the closest thing that could delete the CA State Bar cases off the index.

Meeting With California State Bar. Monday Morning.

The night I learned of the issue, I'd immediately called out the CA State Bar saying that all the records accessed were publicly available (confidential & non-confidential). By Monday morning, however, the story had already hit local and national media. And I was a hacker. In what can only be described as simultaneously the most bizarre and unbizzare meeting I have ever had, by the time the Monday morning meeting arrived, everyone already knew what the problem was. Direct case access did not perform any access control check before returning case data. Holy. Fucking. Shit. It's one of those things that almost doesn't compute, honestly. A security measure so fundamental, and without which the system can't even be called secure. There's no buildup or grand reveal on the technical side of things, if you were hoping for one. You might be wondering then, what's the most reliable way to size this up? What information available would have the highest liklihood of accurately reflecting what is actually true? And I would simply suggest looking at what happened in the following days. Tyler starts taking their portals offline accross the county. I disable the search on judyrecords, and disable all Tyler Technologies cases from direct access. Silence.

The Great Retraction

Given the severity of the software defect, I'd like to think it was an entirely foregone conclusion about what the Bar's response would be after the initial meeting. The California State Bar released the following statement that same day:

Within the span of 48 hours, I had metamorphisized from an ordinary citizen to a hacker. And to an ordinary citizen, once more. The least biased reporting I've found up to this point here.

Requests, Requests, Requests

At this point, you're undoubtedly wondering about this "unique access method" described above. Any funny business going on there? The short answer, is yes. But only from the court systems themselves, actually, not from me. Broadly speaking, there are 2 kinds of public data systems:   (1) Public data systems that make their public data available and accessible.   (2) Public data systems that make their public data available and accessible, while simultaneously and intentionally errecting barriers that make simple and straightforward access difficult. Two-faced systems, if you will.

Type 1 systems   — Oklahoma Statewide Court System System   — United States Patent and Trademark Office System   — United States Trademark Trial and Appeal Inquiry System   — Clayton County, Georgia Court Records Inquiry   — Arkansas Statewide Courts System   — The Supreme Court of the United States What makes these and others Type 1 systems? Look at the URLs here, here, here, here, here, here, here, here, here, here, here, here, and here. Seriously, take a look at these URLs. Go ahead, open some new tabs, and check them out. By the way, I mean the URLs themselves, not the pages. And what do they all have in common? They all present simple and straightforward mechanisms to access data. Public records portals allowing simple and straigtforward access to public records. Great! And Type 2 systems? Well, in Type 2 systems you have garbage like this: What. The. Fuck. This is from organizations that are in the same class of entity as the Type 1 systems and URLs given above. Each of the URLs directly above, with less effort, could have been: But the programmers of these systems went out of their way to intentionally create barriers that make simple and straightforward access to public records difficult. In each of the cases above, a level of indirection was added to make the collection of public records as difficult as possible. This is a common anti-scraping technique, which is justifiable for those with proprietary data that aren't funded by taxpayer dollars, but has no place among publicly-funded public records systems. But wait, more Type 2 public records systems glory! In addition to the above scheme, you also see garbage like this:   (1) Date filed searches are arbitrarily restricted, preventing the ability to simply query all cases filed within (even a small) date range and get all the cases.   (2) Want to get an update to a case or download a public case document? Fill out a captcha first to request the case URL.   (3) Want to get all the cases of a type, filed within a certain date range? First specify at least 2 characters of the last name, and one first name character. I've just checked 10 random Odyssey Portals and not a single one allows doing a simple date filed search. Each one requires that additional information be supplied. Are you seeing a pattern here? In terms of public records access, this is intentionally antagonistic.

Records Wide Open

Tyler, as it happens, sells Type 2 public records systems. They've implemented every single tactic mentioned above to varying degrees across their portals. Honestly, you just deal with it. These systems and the people who sign off on them certainly have a special place in public records hell, but you take what you can get, and move on. I'd pulled data from about 20 of Tyler's Odyssey Portals from jurisdictions around the country. California, Georgia, Texas, and so on, marshalling through searches, farming out the captcha solving to a third-party service when needed. Another day in public records aggregation paradise. In Tyler's Odyssey Portal, cases could be requested directly, no need to jump through hoops for that part. More often than not, cases can be requested directly, but sometimes you have to do a search first, sometimes you need to fill out a captcha, etc. It depends on how far Type 2 they've made the system, but in this case, direct requests were available. So, remember that first set of garbage URLs from Type 2 systems above? The second one was actually one of Tyler's, corresponding to whatever case ID. I remember looking at the table of a portal I'd already pulled from, and thinking:

And it turns out, they were. At this point, it's just a matter of marching through case IDs, as usual. If you've followed all the way through to this point, you might've figured out what went wrong. Tyler went out of their way to implement Type 2 measures, intentionally requiring convoluted access approaches by their own design, but didn't bother with basic security. Did I ever think a direct case request could lead to private records? Not in a million years. Because that would mean the software had a defect so severe that any and all systems running that software would need to be taken offline immediately, due to its inability to perform arguably its most critical function. Well, that happened. And I downloaded from about 30 different Tyler Odyssey portals this way. The California State Bar times 30 — potentially.

It Gets Worse

There's an old Internet adage:

Tyler's software didn't know either. If you had a link, you were a fully authorized user — tail or otherwise. I pointed out early on that this isn't just a judyrecords problem. If these links are not performing access control checks, if a case is found by a docket aggregator, and then goes private and has updates, those updates that happened after the case became private are still public and subject to be downloaded. That would seem to be a large and widespread problem, but simply hasn't been noticed yet. Docket aggregators have millions of cases from this software system. Probably no less than 10 million on average. According to Tyler's own spec sheet, this software system serves more than 600 counties across 22 states — although I believe this counts their legacy Odyssey system as well, which doesn't have the same issue. Two days in, I had two asks for Tyler. My exact message:

Type 2 Collaboration And Disbelief

By Thursday of the same week I had initially met with the California State Bar, I had already shared exactly how I downloaded cases from their portals. Some by search first, then direct access. The others just by direct access. By Friday, it was great to clear the air. We discussed leveraging the same ability to identify which cases were actually viewed that the CA State Bar used. That there are far fewer cases intended to be private that have been viewed vs. the number that have been downloaded is an incredibly tempering circumstance. judyrecords had also saved all case identifiers for every case downloaded, which would allow accurate cross-referencing between systems. I've done this for every system where possible, not just Odyssey. A strong point of contention was exactly how data would flow. I wanted data to flow from Tyler to me first, and Tyler wanted the opposite. When data flows from my end first, I have no ability to know what's what, Tyler can say whatever it wants about the data, and I don't have any basis to know, evaluate, or say anything. They control the information entirely. My main concern however, was that data flowing from me first would kill their incentive to collaborate back in kind. But by the end of Friday, still in the very first week of it all, I was already sending them data up front. I did manage to require that I wouldn't be sending more than 5 sources before they had to send data back before getting more. But I gave in to the next special request, and the next, and the next. By the middle of the next week, I had provided 15 data sources up front. Then, I went out of my way to create truncated files with actual data for every direct access portal. 10k-record files that could be used for testing, and for any data source for which I haven't given them a full file for yet. But two weeks later, after significant back and forth, and despite the nature of the software defect, they are unwilling to even say they believe the access to records meant to be nonpublic was inadvertent. And I've only received one data file back of nonpublic case IDs to remove.

Where Do We Go From Here?

As far as removing cases on judyrecords retrieved from Odyssey (non-legacy) portals that were meant to be nonpublic, I only need a list of case IDs. After that, a confirmation check on a sample of the cases provided, and confirmation of the records removed. As for general cleanup, the only viable solution I see is for Tyler to establish an API on their public records portals so that docket aggregators can voluntarily synchronize their datasets to remove currently nonpublic cases. This will have the effect of handling cases that became private, but updates were received after that. After that, I seriously hope they'll consider getting rid of Type 2 designs that intentionally errect barriers that make simple and straightforward access to public records difficult.

Supplemental Q&A (last updated 7/22)

Which versions of Odyssey Portal were affected by the direct access issue? Tyler has 2 major Odyssey products that I know of. Odyssey "New" (referred to as just Odyssey Portal now) and Odyssey "Old", their legacy Odyssey system. First one looks like this, second like this. I've asked Tyler multiple times. I finally caved. And just asked: What's the complicated part? My exact words:

It's like talking to a brick wall. Is sharing the access method or case IDs security sensitive? No, absolutely not. First, access method. Tyler has already taken their public portals offline across the country. If there are any portals that have either (1) been taken offline then brought back online without fixing the issue or (2) have the issue but have not been taken offline, that would be extrordinarily incompetent. If this is the case, there are even bigger problems with Tyler. Second, case IDs or URLs aren't sensitive information in any practical sense. Only the lack of access control upon those, is what can make them insecure. I just checked Washington's Odyssey system, just for the heck of it. Both identifiers exist in the case right now. See here. Does that mean the system is insecure? No! It's entirely irrelevant. As mentioned previously, judyrecords, YouTube, and others use the same type of URLs as Tyler's Odyssey system. In fact, here's an export of internal case ID mapping to URLs for judyrecords for a currently restricted datasource — 569k cases total! CaseIdPlusUrlMapping-AccessControlDemo.zip Here's the format: These cases were blocked at the beginning of the month. (March) Am I any less secure now? No. I've implemented access control. Those cases are for my eyes only. It does not matter whether you have the case IDs or URLs. Does not fucking matter. Tyler not doing this is the exact cause of the current situation. Exact fucking cause. That Tyler thinks these IDs are security sensitive shows an unbelievable lack of understanding of security, and of access control specifically. Is Tyler Technologies a Type 2 collaborator? From Tyler: Those are the only timetables I've known about. Kinda makes you feel like you're being hung out to dry. And a company has a defect in their software so bad, that by the admission of their own actions is enough to warrant immediately taking everything offline, but you can't be given the benefit of the doubt of relying on that circumstance not being true? And if it weren't true, the code would have retrieved exactly the information intended? I don't know how you can use the word responsible, not owning up to the issue, then leaving doubt to be cast on me for this entire clusterfuck. Tyler has said that judyrecords needs to provide full data about cases up front, and that this is necessary. Is that true? That's an absolute lie. Data in this situation can flow 2 ways and achieve the exact same end result: There is absolutely nothing in the way of achieving the exact same result when Tyler sends data to judyrecords first. How does it look like when data comes from judyrecords first? How does it look like when data comes from Tyler first? As of 3/26, I've told Tyler I will no longer be providing any data sources upfront, due to their extrordinary dishonesty and incompetence in the whole situation. Only in return. Also see the questions: - Is sharing the access method or case IDs security sensitive? - Which data sources were provided to Tyler upfront? How bad was the defect, 1 to 10? Case access is the most critical point to ensure access control. Everything else is a distant second. Security would easily be the most important non-functional requirement of software handling sensitive information. Easily 10. Does Tyler Technologies have a page for this issue? Yes, this page whispered its way onto the Internet about 2 weeks ago. Interestingly, it was just updated today. (3/21) Earlier version here. How many Odyssey Portal cases meant to be private does judyrecords have? At this point, I believe this number is well into the millions. These cases have been accessible over roughly the past year. Also, this is not considering the "any dog with a link" update issue described in the section "It Gets Worse". Where do things stand with the California State Bar? While the investigation is considered ongoing, the California State Bar has stated: and Updates here. I'm a CIO of a potentially affected jurisdiction. How can I contact judyrecords? You can email me at judyrecordssite@gmail.com. I've provided information for numerous portal sites to Tyler upfront. I'm sure you couldn't care less about that, and I wouldn't either, honestly. That said, I have never had any reservations about providing data, except that it will disincentivize returning data in kind. Toward that end, the best way to move forward is just to provide a list of case IDs of non-public cases that can be checked against. After that, I'd be happy to provide all relevant data back about those cases, and remove them. Tyler has taken the position that judyrecords not providing data first has held things up somehow. I cannot tell you how incredibly frustrating and outright dishonest that is. I'd be happy to help in this situation. Basically, you need to write a query that will return all non-public cases. Writing this query may not be just a plain query with a single filter, so it may be necessary to reach out to Tyler technical support to those who have worked on similar queries already. Obviously, you don't need judyrecords data to make this happen and move things forward. Two Four Five Six jurisdictions have done so already. Which Tyler Technologies Odyssey Portals were accessed by search, then direct case requests?   — Alameda County, California - Court Records   — Brazos County, Texas - Court Records (received case IDs, courtesey processed)   — Chambers County, Texas - Court Records (received case IDs, pending)   — DeKalb County, Georgia - Court Records   — Fresno County, California - Court Records   — Glenn County, California - Court Records   — Glynn County, Georgia - Court Records   — Gwinnett County, Georgia - Court Records   — Harris County, Texas - Court Records   — Houston County, Georgia - Court Records   — Hunt County, Texas - Court Records   — Kansas - Statewide Court Records   — Kern County, California - Court Records   — Napa County, California - Court Records   — Ohio - Court Records   — Rockwall County, Texas - Court Records   — San Bernardino County, California - Court Records   — Santa Barbara County, California - Court Records   — Santa Clara County, California - Court Records   — Wichita County, Texas - Court Records (received case IDs, courtesey processed)   — Yolo County, California - Court Records Courtesey processed means that there's no basis or reason to believe that these data sources are at issue, as far as judyrecords is concerned, but that the jurisdiction still provided the list of case IDs intended to be public or nonpublic out of caution. Which Tyler Technologies Odyssey Portals were accessed by direct case requests? * Case IDs of cases meant to be nonpublic have been provided * Case IDs of cases meant to be nonpublic have not been provided Current progress as of 5/15: Odyssey Portal cases removed: 1,390,336   — * California State Bar - Court Records       — 322,525 cases removed (Mar 2)   — * Forsyth County, Georgia - Superior & State Court Records       — 259,473 cases removed (Mar 9)   — * Bell County, Texas - Court Records       — 286,200 cases removed (Mar 30)   — * San Mateo County, California - Court Records       — 18,691 cases removed (Apr 5)   — * San Diego County, California - Court Records       — 124,943 cases removed (Apr 13)   — * Santa Cruz County, California - Court Records       — 30,381 cases removed (Apr 13)   — * Tehama County, California - Court Records       — 27,526 cases removed (Apr 13)   — * Bexar County, Texas - Court Records       — 105,136 cases removed (Apr 13)   — * Sutter County, California - Court Records       — 946 cases removed (Apr 17)   — * Yuba County, California - Court Records       — 1,136 cases removed (Apr 17)   — * Mendocino County, California - Court Records       — 20,859 cases removed (Apr 17)   — * Butte County, California - Court Records       — 1,193 cases removed (Apr 21)   — * Stanislaus County, California - Court Records       — 6,505 cases removed (Apr 21)   — * Kings County, California - Court Records       — 9,146 cases removed (Apr 21)   — * Calaveras County, California - Court Records       — 5,513 cases removed (Apr 27)   — * Washington - Statewide Court Records       — 12,151 cases removed (Apr 27)   — * Sonoma County, California - Court Records       — 409 cases removed (Apr 29)   — * Merced County, California - Court Records       — 151,790 cases removed (Apr 29)   — * Vermont - Statewide Court Records       — 3,913 cases removed (Jul 20)   — * Maine - Statewide Court Records       — 1,900 cases removed (Jul 21)   — * Chatham County, Georgia - Court Records (contacted Jul 20th)   — * Dallas County, Texas - Court Records (received case IDs, in progress)   — * Ector County, Texas - Court Records (contacted Jul 20th)   — * Kane County, Illinois - Court Records (contacted by)   — * Lowndes County, Georgia - Court Records (contacted Jul 20th)   — * Lubbock County, Texas - Court Records (contacted Jul 20th)   — * Muscogee County, Georgia - Court Records (contacted Jul 20th)   — * New Hampshire - Statewide Court Records (contacted Jul 20th)   — * Potter County, Texas - Court Records (contacted Jul 21st)   — * Rhode Island - Statewide Court Records (contacted Jul 21st)   — * Shelby County, Tennessee - Criminal Court Records (contacted Jul 20th)   — * Spalding County, Georgia - Court Records (contacted by)   — * St Tammany, Louisiana - Court Records (received case IDs, in progress)   — * Tazewell County, Illinois - Court Records (contacted Jul 21st) Which data sources were provided to Tyler upfront? Complete record listings were provided to Tyler upfront for the following data sources on the dates noted:   — Butte County, California - Court Records (provided Mar 4th)   — Calaveras County, California - Court Records (provided Mar 4th)   — Kings County, California - Court Records (provided Mar 4th)   — Mendocino County, California - Court Records (provided Mar 4th)   — Merced County, California - Court Records (provided Mar 4th)   — Muscogee County, California - Court Records (provided Mar 4th)   — San Diego County, California - Court Records (provided Mar 4th)   — San Mateo County, California - Court Records (provided Mar 4th)   — Santa Cruz County, California - Court Records (provided Mar 4th)   — Sonoma County, California - Court Records (provided Mar 4th)   — Stanislaus County, California - Court Records (provided Mar 4th)   — Sutter County, California - Court Records (provided Mar 4th)   — Tehama County, California - Court Records (provided Mar 4th)   — Bexar County, Texas - Court Records (provided Mar 4th)   — Forsyth County, Georgia - Superior & State Court Records (provided Mar 5th)   — Dallas County, Texas - Court Records (provided Mar 9th) Data sources returned by Tyler as of Apr 1st:   — Forsyth County, Georgia - Superior & State Court Records (returned Mar 9th) According to Tyler's 4/1 update here: Considering I sent Tyler 16 15 data sources upfront nearly a month ago, while they've returned only 1, the above statement is extraordinarily dishonest. And previous updates: Additionally, my understanding now is that Tyler likely represented to California courts that I had provided data for only 3 California courts, when in fact I had actually provided all 13 California direct access data sources on March 4th. * 7/18 - Muscogee is a county in Georgia and was inadvertently included in the original list. My jurisdiction was not in the first list. Does that mean nonpublic cases weren't exposed? In general, no. However, these are low risk at least as far as judyrecords is concerned. If judyrecords has any issues with these data sources, then Tyler's entire product was access control defective, not just the most important part (i.e., case access). As mentioned in the first question, it is my current understanding/assumption that any non classic Odyssey Portal had the issue of no authorization check for direct access. Depending on what the counts look like when data is provided, whether or the extent to which this is the case will become more apparent. Additionally, this is not considering the "any dog with a link" update issue described in the section "It Gets Worse". How long has Tyler known about which jurisdictions were accessed which way by judyrecords? I provided Tyler this information on March 4th, before the end of the first discussion I had with them. How long has Tyler known about the "any dog with a link" problem? See the section "It Gets Worse" for details. I sent that message to the CA State Bar to forward to Tyler on March 2nd, who had been in communication with Tyler up to that point, whereas I had not. Will judyrecords be providing any more data upfront? No. Only in return at this point. Any other form of collaboration is unacceptable. Tyler's software had an extrordinaliy bad defect, we've got a huge fucking mess, and I've gone far beyond what was agreed. 3x more to be exact. See "Type 2 Collaboration And Disbelief" for details. Yet Tyler's position has been to continue to demand more without returning data in kind. At some point, you have to put on the brakes. Tyler: You fucked up not implementing basic security, and you need to own it. I'm not going to do anything that will hamper your incentive to collaborate back at this point, and that means not providing data upfront, any longer. Only in return. Why? I have to do this because that's the only way I have a fighting chance of keeping you accountable and maximally ensuring I can get the data right on my side. Regarding #2, Odyssey Portals (non classic) that were accessed by direct case requests only would be the scope of cases at issue as far as judyrecords is concerned. Also see "Tyler has said that judyrecords needs to provide full data about cases up front, and that this is necessary. Is that true?". Were any of the unprotected case links indexed by Google from Odyssey Portals? Yes. The number still indexed by Google as of 4/4 in fact is around a quarter million — even after Tyler changed the case URLs for all their sites. See here. The number for Idaho alone is sitting around 100k right now. See here. Even sealed cases — unprotected and publicly available in Google's index from Idaho's own Tyler portal. Google suggests the phrase "felony expungement", seemingly related to the contents of the search. According to Google, cases from Idaho's portal have been indexed as far back as 2015. This is consistent with the Internet Archive's capture of the Odyssey Portal at that link here. Has Google been keeping up to date on the sealed cases of Odyssey Portal clients? Well, yes. From Google's cache here. Sealed cases receiving updates on Google months, even years, after the fact. Does that mean sealed cases have been publicly available from other jurisdictions on Google, like Idaho? Also yes. The number of these unprotected case links still lingering in Google's index as of today, even though it's still almost a quarter million, is likely to be a small subset compared to what's been available over previous years. Where was the mapping of case ID to URL provided? The mapping was provided as part of the Odyssey search API case results object. An actual example: