iFixit recently posted more than 13,000 service and maintenance manuals for medical devices from hundreds of manufacturers. The effort, CEO Kyle Wiens told Marketplace Tech, took “close to half of my team” off normal work for two months . With no advertising, most of the server, development, and staffing costs were out-of-pocket. It was, he told the show with a laugh, “maybe one of the stupidest business ideas I’ve ever had.”
The medical database was also a novel effort, combining crowdsourced file sharing, an all-remote coordinated team effort, and unique technical challenges. I interviewed some key iFixit staff and outside volunteers about how the project came together, in the hopes it might provide some lessons and inspiration for similar efforts.
How It Started
Kyle Wiens, CEO: We were seeing stories about a ventilator shortage in Italy. Then stories about how some people were 3-D printing replacement valves. How can I help? My first thought was, let’s build guides, create ventilator service manuals. We started with 2-3 Cal-Poly (California Polytechnic State University) classes working on them.
We quickly realized that this pandemic was also going to need anaesthesia machines. That CPAP, BiPAP machines were in play. What is the real problem, then? Ventilator service guides are kinda useful, but it’s not every day you’re fixing ventilators. It’s all kinds of machines in hospitals.
So, let’s do that.
Jeff Suovanen, senior technical writer, medical project supervisor: Our goal was simply to be helpful right now, in this crisis, for the people fixing the equipment that keeps people alive. And when we realized many of them were losing valuable time on a scavenger hunt for service information, we thought, this seems like an area where we can make an impact.
Jared Wilson, biomedical technician, CTO & co-founder, Insight HTM: I’ve been doing (biomed work) for 15 years. Every device I work on, there’s always a service manual. You learn pretty quickly to hold onto them. Every time I went to a training, every time coworkers went to a training, we’d save them. You don’t know you need it until you need it, and you can’t wait for a (manufacturer) to get back to you. … I’m also really good at finding things online. I’ll find a manual, then (browse the site) and get more, that kind of thing.
… When you work in a large healthcare organization, the amount they pay for the training is just astounding. I spent 6 weeks in Wisconsin, in a half-dozen GE “schools” for devices, and (my former employer) spent well over $50,000 on it. This is just unsustainable for any organization, let alone a smaller medical site. … Now, as a business owner, my clients need access to these documents. It’s built into my work.
Wiens: We wanted to make sure that we were doing something useful. I didn’t want to add busywork to anybody’s agenda. … We called doctors and biomeds and asked, is this needed? Universally, they told us, something like this is needed.
Gay Gordon-Byrne, executive director, The Repair Association, project volunteer: I think it’s really helpful, for a project like this, that you didn’t know what it was really going to be like when you started.
Wiens: We posted on biomed forums, the big one being /r/BMET, asking for help. Well, we tried to work with that subreddit, but the moderator had gone silent, for something like two years, with no replacement. A splinter sub-reddit had appeared, which was confusing. So our first task, weirdly, was begging Reddit to get this subreddit a moderator.
Biomedical technicians started sending in links to large collections of PDF manuals for medical devices—among other files.
Wiens: These folders, they’ve been around. Maybe one person had an idea for organizing, but the next person gets it and adds a folder called “Misc” with a bunch more manuals in there. Some are just jumbled with a lot of unrelated files. This is something biomeds told us they struggled with: curating a personal library of manuals, then having every other biomed have to duplicate that effort.
Eventually, the motherlode appeared: 300,000 files totaling 220 GB, shared via Google Drive.
Wiens: Great, I thought. Hit “Download.” Wait for an hour, Drive errors out. There are dozens and dozens of subfolders, so you can’t do it piecemeal. I try setting up Google File Stream, running it overnight, but the computer just chokes, and there are bandwidth issues. So I set it up on the fiber connection at the office … (Wiens goes on to explain how Google Drive’s desktop client uses a different file system, SQLite, on a Mac than its native Fuse, and how developers at iFixit built a way to grab the files on a Linux system).
After receiving the files, Wiens pulled two site engineers from iFixit to de-duplicate and filter the files in the Motherlode and other file collections. What was left was 47,848 seemingly relevant PDFs. A handful of iFixit staffers, nearly all of them working from home now, started sorting and loosely categorizing.
Among them was Kristen Gismondi. Normally, Gismondi helps manage iFixit’s university technical writing projects. Most of these courses had been suspended by the COVID-19 crisis. She served as a volunteer coordinator for the medical device project.
Gismondi: The day I started on the project, it was 10-ish people sorting through PDFs. (My manager) told me to make this my priority. The thought was, all of us working on this, with some help from a few others, a couple weeks, we’d have this figured out.
A week or two later, working straight through, we’d made about a 5 percent dent. It was not encouraging.
Wiens: We expanded it to 20 iFixit-ers pretty quickly. I told the department heads, “Okay, everyone on your staff takes 2 minutes per PDF, they can do about 30 an hour.” … We realized pretty quickly that people couldn’t work like that. We needed to do this project faster than our staff could accomplish on their own.
Enter the Librarians
Gismondi: Kyle sets up this email blast, asking people to commit 20 hours to our project. It’s a huge response, people think this is a great project to be doing. We invite everyone who wants to help to a Zoom call. … Before we even really get started, Kyle’s Zoom account maxes out at 100 people.
Wiens: I have my card out, and I’m ready to give Zoom money, but you can’t increase the limit in the middle of a call. Why is that?
Gismondi: We get a ton of email from people, “I couldn’t get in!” Luckily, we recorded it … I was really pleased to see people on our same wavelength. People really treated it like it was a matter of life or death, having things organized in a way that anyone could make sense of quickly. It was kind of cool to see people with the same mindset, like we have when we’re working with devices.
The Real Work
The bulk of the biomedical project effort was to rename, and sort into hierarchical folders, the thousands of device manuals received as PDFs. Some had file names that tell you nothing about the device. Some devices had multiple types of manuals (service, assembly, user), often for model names with very small differences. Some devices could be used for more than one purpose.
Because iFixit is a wiki, every device must have a globally unique name. People searching for specific device manuals should be able to find them quickly, without having to check multiple options.
Wiens: We needed software to organize the files, and we knew Google Drive wasn’t going to work. … The maker of FileRun was nice enough to donate an unlimited license for this project. It’s an easy way for a whole bunch of people to collaborate on a whole bunch of files.
Because we had it installed on our own server, we got faster transfer speeds, we had revision history and access control, and we could sync to our dev and live servers how we wanted. … I can’t recommend it enough for projects with over 200 librarians.
Gismondi: It was scary and intimidating at first. Some of these devices and manufacturers, you’ve never heard of them before. It’s not like electronics, where, okay, this is a laptop, and at least you can start from there.
Gordon-Byrne: Like doing a jigsaw puzzle, you start with doing the obvious stuff first. Then, you get into it, and the huge variety of equipment is overwhelming. … I’ve categorized a lot of equipment in data centers, where, even with 1,500 products, you have maybe 150, 200 manufacturers, and the uses are pretty distinct. This was much, much harder.
Gismondi: Things you would think were obvious, like having the manufacturer name on the front page? Even if that was the case, a lot of these manufacturers acquire each other. So you’d have different PDFs for the same device. It took some Googling to figure out even the simple things.
Justin Ogihara, EDU at iFixit and project volunteer: Google didn’t really help, since some of the companies either didn’t exist or were absorbed into a smaller branch of a larger overarching company.
Suovanen: You had to draw on your own librarian skills, on-the-spot research, and context and clues within each file. And thankfully we had a biomed acting as final referee. We wanted one canonical procedure for organizing everything—but it’s difficult when people have different perspectives on how things should be organized, and they can be equally valid.
Wilson: I just had to do my best. But, at the end of the day? A different biomed might have structured it completely differently than me, based on their experiences, their work.
Suovanen: One librarian asked, “What’s the difference between a technical manual versus a service manual, or an operator versus a user manual? Some devices have all four.” And I had to sheepishly admit that there’s no standardization on the contents of those documents. It’s up to each individual manufacturer. This is why we needed help; you can’t automate a process where almost every file requires a judgment call.
Gordon-Byrne: I was amazed by how many laser devices there are. Lasers and ultrasound. You go in they’re generally used for a few things, but there are tons and tons of ultrasound applications. … Those categories were tricky.
Gismondi: Maybe the weirdest device I came across was a “surgical slush machine.” You hear those words in your head, and … It actually generates ice for organ transplants. But then, it has a quiet operation, so it’s the “Hush Slush Surgical System.”
Ogihara: My god, there were a bunch of different types of patient monitors. … All types of patient monitors were under one category (in the original folders). The thing is, patient monitors can vary from age groups to functions. … Fetal, vital signs, electrocardiogram, ultrasound, etc. I would’ve thought patient monitors were universal, but … boy, was I wrong, after sorting through the files.
Heading Toward a Finish
During one weekly video session, a librarian asked if iFixit would run optical character recognition (OCR) on the PDFs it received. That started the ball rolling on a process that would ultimately max out a 100-core server (on $10,000 of time donated by Amazon), upgrade the PDF uploading experience on the site, and utilize the engineering team to write a PDF ingestion script for when the site went live.
Gordon-Byrne: Kyle kind of fed (progress reports) to us in chunks—”We’re such and such percentage through! We have this many left. Oh, and by the way, the newest file we received has 600 files inside.” … It’s to his credit, that he’s a clever manipulator of time and space.
Suovanen: At times we had to remind ourselves how urgent the project was because you could feel like you were on a treadmill—we kept receiving more and more files. But everybody knew we were working on something people needed right now. It was always more work than we thought. … Our initial target launch date was too optimistic, but after another ten days we regrouped and got it under control.
Gismondi: We all just kinda made this big push, with a drop-dead deadline: This the date we’re going to publish.
One of the challenges with this project is, tech writers and librarians, it’s a tough combination. … We’re all kind of perfectionists, in our way. The mantra of the project was to get it done fast, let everyone fix it later. I think we all realized that we were getting hung up on things, that it was the major hurdle slowing us down.
The last weekend before the manuals were published, testing the sync between the FileRun and iFixit’s servers revealed dozens of small issues. Umlauts in file names caused failures. Files had the same name, but were capitalized differently. Suovanen worked with the volunteers, checking every folder and solving as many lingering problem categories as possible.
Eventually, after running a 13-thread import process, the manuals were up. Now it was time to spread the word.
iFixit announces its medical database on May 19, 2020. The release is covered in The Verge, WIRED, CNET, the datahoarders subreddit, and, maybe most importantly, 24×7, an industry publication for biomedical technicians. The EFF writes about why the project is legal and important.
Gismondi: I feel like it really brought us closer to our teammates at iFixit. And meeting these amazing volunteers, who loved organizing all of this stuff, despite how hard it could be—I’m so glad so many enjoyed it. When it finally published, I was tired, I was goofy and slap-happy from the work, and so were others. But we understood the gravity of this project.
Wilson: It was a lot of fun. It was a lot of work, and it was fun working with the team on it. …. I’m that weird kind of person, where a big problem involving device category structures gets me excited. But what was really cool was how much people appreciated what we did. We made it searchable, with all the correct keywords associated with each device. That’s going to mean something to someone in the field.
Suovanen: My initial fear was that even though we wanted to be helpful, we were rookies in this device space and so we’d muck it up. But the reaction from all the biomeds we heard from was, “You guys are doing exactly what we need, this is amazing, how can we help?”
Gordon-Byrne: I’m glad the project went up, but it should never have needed to exist. We’ve included a medical component in the Right to Repair movement from the beginning, and that’s on purpose.
With what’s happening right now, hospitals have to be able to fix their equipment, without relying on OEMs (original equipment manufacturers), because the OEMs can’t do it. The OEMs can’t get the parts they need, mostly from China. And even if they could, they’re not sending people for trainings, meetings, or on-site repair. Medical device manuals should be out there… the laws exist already, but there’s no enforcement.
Wilson: Across the nation, we have biomeds reinventing the wheel, over and over and over, just to create the access they need to do their jobs. … If your hospital can pay for a device that helps someone, you should be able to work on it. That’s what this is about.