Downtime in 2002 verses 2013

On November 13, 2002, the network core at Beth Israel Deaconess failed due to a complex series of events and the hospital lost access to all applications.   Clinicians had no email, no lab results, no PACS images, and no order entry.    All centrally stored files were unavailable.   The revenue cycle could not flow.   For 2 days, the hospital of 2002 became the hospital of 1972.  Much has been written about this incident including a CIO Magazine article and a Harvard Business School case.

On July 25, 2013, a storage virtualization appliance at BIDMC failed in a manner which gave us Hobson's choice  - do nothing and risk potential data loss; or intervene and create slowness/downtime.   Since data loss was not an option, we chose slowness.  Here's the email I sent to all staff on the morning of July 25.

"Last evening, the vendor of the storage components that support Home directories (H:) and Shared drives (S:) recommended that we run a re-indexing maintenance task in order avoid potential data corruption. They anticipated this task could be run in the middle of the night and would not impact our users.   They were mistaken.

The indexing continues to run and must run to completion to protect H: and S: drive data.  While it is running, access to H: and S: will be slow, but also selected clinical web applications such as Provider Order Entry, webOMR, Peri-operative Information System, and the ED Dashboard will be slow.  Our engineers are monitoring the clinical web applications minute to minute and making adjustments to ensure they are as functional as possible.   We are also investigating options to separate clinical web applications from the storage systems which are causing the slowness.

All available IS resources are focused on resolving this as soon as possible.  We ask that all staff and clinical services affected by the interruption utilize downtime procedures  until the issue is resolved.  We apologize for the disruption this issue has caused to patients, providers, and staff."

2002 and 2013 were very different experiences.   Here's a brief analysis:

1.  Although 2002 was an enterprise downtime of all applications, there was an expectation and understanding that failure happens.   The early 2000's were still early in the history of the web.   There was no cloud, no app-enabled smartphones, and no universal adoption of social networking. Technology was not massively redundant.  Planned downtime still occurred on nights and weekends.

In 2013, there is a sense that IT is like heat, power, and light - always there and assumed to be high performing.   Any downtime is unacceptable as emphasized by the typical emails I received from clinicians:

"My patients are still coming on time and expect the high quality care they normally receive. They also want it in a timely manner.  Telling them the computer system is down is not an acceptable answer to them.   Having an electronic health care record is vital but when we as physicians rely on it and when it is not available, it leads to gaps in care."

"Any idea how long we will be down? I am at the point where I may cancel my office for the rest of the day as I cannot provide adequate care without access to electronic records."

In 2013, we've become dependent on technology and any downtime procedures seem insufficient.

2. The burden of regulation is much different in 2013.  Meaningful Use, the Affordable Care Act, ICD10, the HIPAA Omnibus rule, and the Physician's Quality Reporting System did not exist in 2002.   There is a sense now that clinicians cannot get through each day unless every tool  and process, especially IT related, is working perfectly.

Add downtime/slowness and the camel's back is broken.

3.   Society, in general, has more anxiety and less optimism.    Competition for scarce resources  translates into less flexibility, impatience, and lack of a long-term perspective.

4.  The failure modes of technology in 2013 are more subtle and are harder to anticipate.

In 2002, networking was simple.  Servers were physical.  Storage was physical.  Today, networks are multi-layered.  Servers are virtual.  Storage is virtual. More moving parts and more complexity lead to more capabilities but when failure occurs, it takes a multi-disciplinary team to diagnose and treat it.

5.  Users are more savvy.   Here's another email:

"Although I was profoundly impacted by today's events as a PCP trying to see 21 patients, I understand how difficult it is to balance all that goes into making a decision with a vendor on hardware/software maintenance. However, I was responsible for this for a large private group on very sophisticated IT, and I would urge you to consider doing future maintenance and upgrade projects starting on Friday nights, so as to have as little impact as possible on ambulatory patient care."

My experience with last week's event will shape the way I think about future communications for any IT related issues.    Expectations are higher, tolerance is lower, and clinician stress is overwhelming.    No data was lost, no patient harm occurred, and the entire event lasted a few hours, not a few days.     However, it will take months of perfection to regain the trust of my stakeholders.

It's been 10 years since we had to use downtime procedures.   We'll continue to reduce single points of failure and remove complexity, reducing the potential for downtime.   As a clinician I know that reliability, security, and usability are critical.   As a CIO I know how hard this is to deliver every day.

Building Unity Farm - Scenes of Summer

Unity Farm is at the peak of Summer.  Everything is in bloom, the forest is bursting with wildlife, and all our outdoor activities are in full swing as we finish creation of our growing areas before we retreat inside for 6 months of winter.   Here are a few photos with the scenes of Summer at Unity Farm.

1.  Afternoon thunderstorms pop up during the hot and humid weather.    They skies are filled with billowing clouds that dwarf the barn and paddocks.

2.  The animals cling to their barn fans, run through the sprinklers and enjoy an afternoon snack of chilled romaine lettuce to keep cool

3.   In the stream, Muskrat Susie and Muskrat Sam whirl and twirl among the reeds.  (I know the song is awful)

4.  Mom and Dad proudly walk our country lane with their new fawn

5.  Guinea fowl build nests of 20-30 eggs in the deepest part of our fern forests

6.  The Great Pyrenees enjoy playing in the shade with their new ball toy

7.  The bees are storing honey for the winter.   Here's a closeup of the queen from one of our 8 hives

8.  The orchard grass has gone to seed and needs mowing .   I maintain the orchard with a push mower and a trimmer for more delicate edge work.   Here's a view of the mowing in progress and the finished result.

9.  Ground hogs (also know as Woodchucks) nibble at the grass in the meadow.

10. Garter snakes sun themselves on the rocks in the garden

The Era of Epic

In the Boston marketplace, Partners Healthcare is is replacing 30 years of self developed software with Epic.   Boston Medical Center is replacing Eclipsys (Allscripts) with Epic.   Lahey Clinic is replacing Meditech/Allscripts with Epic.  Cambridge Health Alliance and Atrius already run Epic.   Rumors abound that others are in Eastern Massachusetts are considering Epic.  Why has Epic gained such momentum over the past few years?   Watching the implementations around me, here are a few observations

1.  Epic sells software, but more importantly it has perfected a methodology to gain clinician buy in to adopt a single configuration of a single product.   Although there are a few clinician CIOs, most IT senior management teams have difficulty motivating clinicians to standardize work.  Epic's project methodology establishes the governance, the processes, and the staffing to accomplish what many administrations cannot do on their own.

2.  Epic eases the burden of demand management.   Every day, clinicians ask me for innovations because they know our self-built, cloud hosted, mobile friendly core clinical systems are limited only by our imagination.   Further, they know that we integrate department specific niche applications very well, so best of breed or best of suite is still a possibility. Demand for automation is infinite but supply is always limited.   My governance committees balance requests with scope, time, and resources.   It takes a great deal of effort and political capital.   With Epic, demand is more easily managed by noting that desired features and functions depend on Epic's release schedule.   It's not under IT control.  

3.  It's a safe bet for Meaningful Use Stage 2.   Epic has a strong track record of providing products and the change management required to help hospital and professionals achieve meaningful use.  There's no meaningful use certification or meaningful use related product functionality risk.

4.  No one got fired by buying Epic.   At the moment, buying Epic is the popular thing to do.   Just as the axiom of purchasing agents made IBM a safe selection,   the brand awareness of Epic has made it a safe choice for hospital senior management.   It does rely on 1990's era client server technology delivered via terminal services that require significant staffing to support, but purchasers overlook this fact because Epic is seen in some markets as a competitive advantage to attract and retain doctors.

5.  Most significantly, the industry pendulum has swung from best of breed/deep clinical functionality to the need for integration.   Certainly Epic has many features and overall is a good product.   It has few competitors, although Meditech and Cerner may provide a lower total cost of ownership which can be a deciding factor for some customers.   There are niche products that provide superior features for a department or specific workflow.   However,  many hospital senior managers see that Accountable Care/global capitated risk depends upon maintaining continuous wellness not  treating episodic illness, so a fully integrated record for all aspects of a patient care at all sites seems desirable.  In my experience, hospitals are now willing to give up functionality so that they can achieve the integration they believe is needed for care management and population health.

Beth Israel Deaconess builds and buys systems. I continue to believe that clinicians building core components of EHRs for clinicians using a cloud-hosted, thin client, mobile friendly, highly interoperable approach offers lower cost, faster innovation, and strategic advantage to BIDMC.  We may be the last shop in healthcare building our own software and it's one of those unique aspects of our culture that makes BIDMC so appealing.

The next few years will be interesting to watch.   Will a competitor to Epic emerge with agile, cloud hosted, thin client features such as Athenahealth?   Will Epic's total cost of ownership become an issue for struggling hospitals?   Will the fact that Epic uses Visual Basic and has been slow to adopt mobile and web-based approaches provide to be a liability?

Or alternatively, will BIDMC and Children's hospital be the last academic medical centers in Eastern Massachusetts that have not replaced their entire application suite with Epic?   There's a famous scene at the end of the classic film Invasion of the Body Snatchers, which depicts the last holdout from the alien invasion becoming a pod person himself.  At times, in the era of Epic, I feel that screams to join the Epic bandwagon are directed at me.

Image Exchange

Last week, the Clinical Operations Workgroup of the HIT Standards Committee held its third hearing on image exchange, seeking testimony from Hamid Tabatabaie, CEO of LifeImage and Michael Baglio, CTO of LifeImage.

He made several important points
1.  We should think of image exchange as having two major categories - local and long distance.    DICOM works well for modality to PACS connectivity within an enterprise (local).   DICOM was never designed for internet-based cross organizational image sharing.   DICOM images tend to be large, including a vast amount of metadata with every image object in a study.    DICOM was also never designed to work well with the kind of firewalls, load balancers, and network security appliances we have today.

2.  Two image exchange architectures have been used in the marketplace to date, which Hamid called "iTunes" and "Napster",  classifications first suggested by Dr. Keith Dreyer.

iTunes refers to the centralization of images into a single repository or what a appears to be single repository - it may actually be a clearinghouse to other image stores, but the user never knows that.   Query/response transactions against this central repository can be straightforward, using standards such as Blue Button Plus/Direct for share, access, download.

Napster refers to a decentralized, federated model in which images are not placed in a single repository -    an index of images and their location is all that is centralized.   Typically, query/response is done with standards such as XDS-i.   XDS itself was never designed for image exchange and is incomplete.  It can be challenging to search for a single exam on a known date of a known modality type.

3. Current standards do not include any privacy metadata and security is not built in.  Future standards should enable applications to restrict image flows based on consent/patient preferences.

4.  We need a web friendly method for visualization that does not require the download of a proprietary viewer, one that is often operating system specific.   Consumers should be able to view thumbnails of images on a smartphone, tablet, or the device of their choosing without special software.   If the full DICOM object is needed (patient mediated provider to provider image exchange), download and transmission should be enabled using standards such as REST, OAUTH2/OpenID, and secure email.

5.  Hamid made a forward looking statement that should be carefully considered as we plan the lifecycle of existing Radiology Information Systems (RIS) and Picture Archiving and Communication Systems (PACS) systems.   He is seeing EHR features expand to cover many aspects of RIS workflow.   If scheduling, image viewing, report construction with templates/front end voice recognition, and easy exchange of reports with clinicians are supported by the EHR, maybe radiologists (some of which want to qualify for meaningful use payments) will start using increasingly capable EHRs instead of RIS.   Vendor neutral archives (VNA) which store images of all "-ologies"  and enable easy search and retrieval may replace PACS.   Imagine 5 to 10 years from now when RIS/PACS no longer exists and is replaced by EHR, HIE,  and VNA.   Interesting possibility.


Great testimony.    In the past when I've suggested DICOM is not ideal for internet-based multi-organizational exchange, I've been criticized.   In the past when I've suggested that DICOM has issues of vendor-specific proprietary metadata extensions, cumbersome viewers, and heavy payloads, I've been challenged.   It's refreshing to hear from someone doing the hard work of high volume image sharing that current standards not ideal.  We need new approaches to move payloads efficiently on the internet, view images via web-browsers, facilitate easy searching, support security, and enable multiple provider/patient/group sharing use cases.

Building Unity Farm - The Barn Swallows of Unity Farm

One of the side effects of creating the orchard at Unity farm was opening about 2 acres of airspace adjacent to our pasture.  This has attracted  many new species of birds which now dance and dive in the clearing between our woodland and marsh.   Since adding the orchard we've seen a significant increase in our barn swallow population.  Throughout the day, at least 5 adults dash at high speed around their own open air playground, eating mosquitos and enjoying their social community of birds.

Our barn has two sliding doors which open to the male and female paddocks.  The swallows retreat to the barn for shelter at night and during the rain.   Recently a mating pair built a nest on top of a porcelain light socket.   We use LED lightbulbs in the barn, so the socket does not get hot.   The nest is a delicate combination of mud, sticks, great pyrenees fur, and feathers from our chickens/guinea fowl, pictured above.

This week, 4 swallow babies are fledging.  Two are pictured below balanced on a barn door rail.   Watching the parents feed them for the past several weeks, I've read a great deal about swallows and can now answer some lifelong questions I've had since 1975, when King Arthur first had a dialog about swallows with the guardian of the bridge.

1.  Are the swallows of Unity Farm considered European or African?

European swallows are migratory and are widespread throughout the Northern Hemisphere.   African swallows are non-migratory and are typically found in in Botswana, Republic of the Congo, Democratic Republic of the Congo, Gabon, Lesotho, Malawi, Namibia, South Africa, Zambia, and Zimbabwe.    Thus, if asked, you can definitively answer that the barn swallows of Unity Farm are European.

2.  Much of the day the adult swallows gather insects to feed to their young.  I've watched them busily carrying food and nesting materials as the cruise through the paddocks and into the barn.     When they're done feeding their babies and reinforcing the nest, they fly above the paddock turning and twisting at high speed before deftly returning to the barn.   Watching them makes me wonder - just how fast do they fly when unencumbered i.e. What is the airspeed velocity of an unladen swallow (European)?

Luckily much research has been done on this topic.


A 54-year survey of 26,285 European Swallows captured and released by the Avian Demography Unit of the University of Capetown finds that the average adult European swallow has a wing length of 12.2 cm and a body mass of 20.3 grams.

European Swallow flies at cruising speed with a frequency of roughly 15 beats per second, and an amplitude of roughly 22 cm.   However, some other researchers have measured a lower frequency of 7-9 beats per second among some swallows.

Because wing beat frequency and wing amplitude both scale with body mass and flight kinematic data is available,  we can  estimate airspeed (U).

Graham K. Taylor et al. show that as a rule of thumb, the speed of a flying animal is roughly 3 times frequency times amplitude (U ≈ 3fA).

Based on wing beats per minute, body mass, and amplitude, the answer for our swallows appears to be 24 miles per hour (11 meters per second).   So if on your quest to visit Unity farm, you are asked the airspeed velocity of an unladen swallow (European), you know the answer!

The July HIT Standards Committee

The July HIT Standards Committee included a robust discussion of Benefits and Formulary standards, a brief overview of our work on image exchange thus far, preparations for the July 23 HITSC Implementation Workgroup and HITPC Meaningful Use and Certification and Adoption Workgroup joint hearing on Implementation and Usability, an overview of ONC S&I Framework activities, and a discussion of the CMS electronic submission of medical documentation planning.

We started the meeting with a presentation from Kim Nolen and John Klimek reviewing the final recommendations for enhanced formulary and benefit standards.   While short term incremental improvements are important, the committee was more focused on the Stage 3 trajectory for formulary and benefits transactions.   Ultimately, the committee made 5 recommendations

1. We endorse the adoption of RxNorm as the preferred medication substance vocabulary in formulary and benefits transactions
2. We support standardizing content on NCPDP Formulary and Benefits version 3.0 to simplify current batch formulary import implementations in the short term but prefer the certification criteria for stage 3 focus on the real time transactions described in recommendation #5
3. We recommend that batch formulary transport standards move from FTP to Direct/XDR to align better with existing Meaningful Use transport standards
4. We recommend that patient matching to pharmacy benefits utilize PCN/BIN/Group Number to more accurately reflect the benefits of the patient's plan.
5. ONC should facilitate development of national standards for real time lookup of patient specific drug/dose benefits by prescribers at the point of care including estimated patient out of pocket cost at the time of the prescription order.

Also, I provided this image exchange update.  We will continue our hearings with more expert testimony this Friday.

Next we heard from Liz Johnson and Cris Ross about their planned July 23 implementation and usability hearing.   Their group has prepared great questions for its four panels - Eligible Professionals, Eligible Hospitals, Health Information Exchange/Interoperability, and Usability

Doug Fridsma provided a comprehensive update about the S&I framework activities, identifying all the current and planned deliverables for the many groups working on these important future looking projects.   Especially interesting was the data accress framework project which incorporates many of the previous point solution projects like QueryHealth and targeted query ("pull" of medical records from multiple sources).

Jodi Daniel offered this ONC Policy and Program Update focusing on the Health IT Patient Safety Action & 
Surveillance Plan.

Finally Melanie Combs-Dyer from CMS and Bob Dieterle presented the roadmap for fully electronic closed loop CMS medical documentation submission to support medical review by Medicare Administrative Contractors (MACs), Medical Review (MR) Departments,
Comprehensive Error Rate Testing Contractor (CERT), Payment Error Rate Measurement Contractor (PERM), and Medicare Recovery Auditors (formerly called RACs).   The digital signature provisions which provide provenance and integrity protections generated the most discussions and these will be reviewed by the HITSC Privacy and Security Workgroup and Clinical Operations Workgroup.

In my introductory remarks to the meeting, I stressed the importance of focusing our standards work on those items which will support the hard work of hospitals and professionals to achieve all phases of meaningful use, ICD-10, Accountable Care, and compliance/regulatory mandates.   Formulary support, image sharing, "pull" based health information exchange,  ensuring are EHRs are safe, and supporting Medicare review workflows definite fall into that category.

As the meeting closed, we thanked MacKenzie Robertson for her work as our FACA facilitator.   She will be replaced by Michelle Consolazio.

The Healthcare IT Applications of Google Glass

Last week I had the opportunity to test Google Glass.  

It's basically an Android smartphone (without the cellular transmitter) capable of running Android apps, built into a pair of glasses.  The small prism "screen" displays video at half HD resolution.  The sound features use bone conduction, so only the wearer can hear audio output.   It has a motion sensitive accelerometer for gestural commands.    It has a microphone to support voice commands.   The right temple is a touch pad.  It has WiFi and Bluetooth.   Battery power lasts about a day per charge.

Of course, there have been parodies of the user experience but I believe that clinicians can successfully use Google Glass to improve quality, safety, and efficiency in a manner that is less bothersome to the patients than a clinician staring at a keyboard.

Here are few examples

1.  Meaningful Use Stage 2 for Hospitals - Electronic Medication Admission Records must include the use of "assistive technology" to ensure the right dose of the right medication is given via the right route to the right patient at the right time.   Today, many hospitals unit dose bar code every medication - a painful process.   Imagine instead that a nurse puts on a pair of glasses, walks in the room and wi-fi geolocation shows the nurse a picture of the patient in the room who should be receiving medications.  Then, pictures of the medications will be shown one at a time.  The temple touch user interface could be used to scroll through medication pictures and even indicate that they were administered.

2.  Clinical documentation - All of us are trying hard to document the clinical encounter using templates, macros, voice recognition, natural language processing and clinical documentation improvement tools.     However, our documentation models may misalign with the ways patients communicate and doctors conceptualize medical information per Ross Koppel's excellent JAMIA article.  Maybe the best clinical documentation is real time video of the patient encounter, captured from the vantage point of the clinician's Google Glass.   Every audio/visual cue that the clinician sees and hears will be faithfully recorded.

3.  Emergency Department Dashboards - Emergency physicians work in a high stress, fast paced environment and must be able to quickly access information, filtering relevant information and making evidence-based decisions.    Imagine that a clinician enters the room of a patient - instead of reaching for a keyboard or even an iPad, the clinician looks at the patient.   In "tricorder" like fashion, vital signs, triage details, and nursing documentation appear in the Google Glass.   Touching the temple brings up lab and radiology results.  An entire ED Dashboard is easily reduced to visual cues in Google Glass.    At BIDMC, we hope to pilot such an application this year.

4.  Decision Support - All clinicians involved in resuscitation know the stress of memorizing all the ACLS "code" algorithms.   Imagine that a clinician responding to a cardiac arrest uses Google glass to retrieve the appropriate decision support for the patient in question and visually sees a decision tree that incorporates optimal doses of medications, the EKG of the patient, and vital signs.  

5.  Alerts and Reminders  -  Clinicians are very busy people.   They have to manage communications from email, phone calls, patients on their schedule, patients who need to be seen emergently, and data flowing from numerous clinical systems.   They key to surviving the day is to transform data into information, knowledge and wisdom.   Imagine that Google Glass displays those events and issues which are most critical, requiring action today (alerts) and those issues which are generally good for the wellness of the patient (reminders).    Having the benefits of alerts and reminders enables a clinician to get done what is most important.

Just as the iPad has become the chosen form factor for clinicians today, I can definitely see a day when computing devices are more integrated into the clothing or body of the clinician.    My experience with Google Glass helps me understand why Apple just hired the CEO of Yves Saint Laurent to work on special projects.  

Ten years ago, no one could imagine a world in which everyone walked around carrying a smartphone.   Although Google Glass may make the wearer appear a bit Borg-like, it's highly likely that computing built into the items we wear will seem entirely normal soon.

I will report back on our Google Glass experiments as they unfold.

Building Unity Farm - The Heat of Summer

In 2013, Boston had a wet spring and is having a very hot, humid Summer.   The 90 degree temps with
90% humidity typical of August arrived in June and July.   Last weekend, the discomfort index rose to 107F as temps soared to 93 with nearly 100% humidity for 5 successive days.

 Alpaca and Llama live in the Andes and are accustomed to dry, cold weather.  Great Pyrenees live in the alps and prefer cold snowy conditions.   Our chickens are optimized for the cold winters of the Northeast.  How do we care for a group of animals which would be happier in the Arctic than the tropics when the temperature feels like 107F?

 I wrote about water management last week and the various ways we have of keeping our paddocks and pastures moist.   Those same low flow rotors keep the alpacas cool in Summer.  Just as children run through sprinklers and gushing fire hydrants to cool off in Summer, the alpaca flock to flowing water, rolling in it, sitting on it and reveling in the cool of a water soaked belly.   Once they're drenched they roll in fine dusty soil to create a cooling layer of mud on their fiber.

Last year we mounted large barn fans in each stall and the animals compete for prime airflow spots.

We have 5 gallon buckets of fresh water hanging through the barn and paddock areas.

The chicken coop has a shaded 20x20 outdoor run, a large barn fan, two water supplies and windows to keep air circulating.

The dogs are a bit tougher.   We gently spread water on their heads and faces, but Great Pyrenees are not drawn to swimming.   Instead they dig wallows (pictured above) in the shadiest areas of the paddock.   The soil around the irrigation rotors is soft and moist.  They dogs dig foot deep dog sized holes and curl up in them to stay cool.   They have morning wallows and afternoon wallows, changing location as the sun tracks through the sky and alters shade patterns.

The bees work hard to keep the hives cool.   This week we added "supers" - a second story to the hives which provides an "attic" of insulation over the primary hive.   We keep a rock filled basin topped off with water nearby so they can access hydration.  Worker bees sit at the entrance to the hive and keep the entire colony cool by flapping their wings in unison, creating a natural fan.

The mushroom farm is in the shade house underneath a grove of pines, out of the heat of direct sunlight.

 For the humans, we recently pressure washed the 18 year old air conditioning compressors and replaced all the filters in the air handlers, so we're cooling the bedrooms as efficiently as we can.   In future years, we'll likely replace the air conditioners with heat pump technology.  Today's most advanced heat pumps match the efficiency of geothermal at a fraction of the cost.

This is our second heat wave of 2013.   So far, all the animals of Unity Farm have enough options to stay cool enough, if not perfectly comfortable.   I look forward to the cooler weather forecasted for next week.   Now that we've lived at Unity for every season, we're ready for whatever nature may bring.

The Innovator's Challenge

I've written many blog posts about our efforts along the path to ICD-10 that will enhance our inpatient clinical documentation.  We're hard at work planning the improvements we think are foundational to support care coordination, compliance, and quality measurement goals.

It's very challenging to create tools which simultaneously enable rapid, accurate, and complete clinical documentation.   We've deferred radical redesign of inpatient documentation for several years awaiting the alignment of technology, policy, and urgency to create the perfect storm for innovation.

BIDMC has had many firsts - early personal health record adoption, first in the country attestation for meaningful use, innovation in the use of web-based provider order entry, rapid adoption of iPads, and one of the first vendor neutral image archives.

Sometimes we're a leader and sometimes we're a follower.  Deciding which to be is the innovator's challenge.

BIDMC decided to ignore the entire client/server era in the mid 1990's.   As others were creating Visual Basic, Filemaker Pro, Delphi, and Access front ends to applications, we continued the use of roll and scroll terminal emulators.  When the web appeared, we jumped in with both feet and moved all our clinician facing applications to thin client, cloud hosted, web-service architectures in 1998.  That approach has served us well.   It still feels modern in 2013.

Recently we completed the implementation of a next generation electronic medical administration record (EMAR) using iPhones, iPads, and an Amazon-like shopping cart motif for choosing medications.   In the past, other organizations were first with EMAR designs, but they had to use computers on wheels and cumbersome user interfaces because the technology was not quite ready for a more streamlined approach.

We feel the same way about clinical documentation.   Offering clinicians an enhanced word processor does not result in orderly, complete, and readable documentation.   On the other hand, forcing structured input of every clinical observation may yield high quality data but usability will be poor.   We're working with 4 different companies to create next generation documentation tools that we think will benefit inpatient documentation the same way that waiting for the iPhone/iPad benefited EMAR.

Characteristics of this new approach include

*Natural Language Processing - the ability to prospectively or retrospectively identify key concepts in unstructured text
*Clinical documentation improvement - the ability to pop up templates just in time that offer structured input in the middle of unstructured text i.e. laterally and specific bone names for fractures
*Vocabulary crosswalks - linkage between problem lists, documentation, and billing diagnoses based on mapping SNOMED-CT to ICD-9,ICD-10, and CPT.
*Metadata markup - near real time SNOMED-CT markup of unstructured data so that structured clinical concepts are embedded within typed or dictated documents
*Computer assisted coding - suggesting ICD-10 codes to clinicians or coders based on the markup in current notes combined with structured data extracted from past notes

Also, we've considered social documentation  (group authoring) and patient generated data.

As with many IT innovations our stakeholders will feel that we lag existing commercial products while we're in the midst of developing these new ideas.   However, once we go live with the finished product, incorporating cutting edge built and bought technologies, no one will remember the days of the hybrid medical record that today includes many electronic features but paper-based progress notes.

Although the paperless hospital is about as realistic as the paperless bathroom, we will substantially reduce paper on our inpatient units in the next 18 months.  As a CIO, I look forward to the day when we've closed the last gap in our self built systems compared to commercial EHRs so that our users can revel in the innovation rather than describing the greener grass available elsewhere.   Luckily, we're as good as our latest go live and we're confident now is the time to implement advanced approaches to clinical documentation.  Tolerating impatience until technology, policy, and urgency align is what makes an innovator successful.

What Matters

I've been at Beth Israel Deaconess Medical Center for 17 years this week.   I'm sometimes asked why BIDMC has been and will continue to be my long term career home.

 The answer is simple - it's a foundation for what matters.

1.  Colleagues matter
Loyalty to my staff is the number one reason I stay at BIDMC.  Together we've shared the network outage of 2002, security challenges, first in the country meaningful use attestation, hundreds of innovative application go lives, and the creation of a world class cloud computing infrastructure.    The average tenure of IS people at BIDMC is 17 years.   Many have been here over 30 years.   Turnover is never more than 10% per year across all IS divisions.

2.  Mentoring matters
The real world experience of operating large scale applications and infrastructure at BIDMC enables me to share lessons learned with students and professionals all over the world.  Whether I'm doing a Harvard Business School case study, helping a government in Asia, or empowering young investigators by connecting them to collaborators, it is my experience at BIDMC building and buying technology that gives me a broad base of successes and failures to share.

3.  Patients and Providers matter
Creating technology for technology's sake is not as impactful as using technologies to achieve policy goals.  BIDMC is a learning laboratory with 250,000 active patients, 3000 doctors, and 2 million patient records.   Every day we can iteratively improve quality, safety, and efficiency by listening to our stakeholders and testing new technologies in production environments.

4.  Innovation matters
BIDMC has a unique blend of built and bought technologies that enable us to control our own destiny.   If a new meaningful use idea needs to be piloted, a new technology investigated, or a new workflow trialed, we can move with agility, often without dependency on vendors.   When a vendor wants to accelerate innovation by testing new technologies we can be a development partner.  Many commercial infrastructure and application products had their start at BIDMC.

5.  Culture matters
For 30 years, BIDMC has had an impatience with the status quo.   The complaints we hear from our stakeholders often relate to problems that other organizations have not yet thought about.   There is never time to rest on our laurels.  At times it seems that memories of our successes fade fast, but the culture of impatience ensures we get rapid adoption of whatever new features we introduce.

While on the plane back from Osaka on Friday, I spoke with a gentleman who has worked in many companies throughout  his career.   At this point he's decided that he needs a company of the right size, right leadership, and right structure to empower problem solving - he has no tolerance for people and organizations that impede progress.    For me, BIDMC has all the characteristics which are foundational to a satisfying career.

As I reflect on my time in Japan, my most influential moments were those I spent teaching, talking with colleagues, listening to others' experiences, connecting people for collaboration, and sharing meals.    BIDMC provides me a base of operations that enables these international experiences, national committee membership, and regional cross institutional cooperation.

Life is complex, budgets are limited, and people are diverse.   Our careers will have their frustrations when there is competition for resources, ever-increasing regulatory pressure, and accelerating change.   However, if you have  great colleagues, remarkable students, a learning lab, a capacity to innovate, and a supportive culture, you have all the ingredients you need for a career home where you can make a difference.  That's what matters.  

Building Unity Farm - Water management

I was recently asked how we manage irrigation at Unity Farm during the peak of Summer heat.

Although we have a stream running through the farm, it's part of a protected wetland, so we do not use it for irrigation.

Our water source is a 300 foot deep well that consistently produces 7 gallons per minute all year long.   Sherborn was founded in 1652 and was initially called Bogostow ("Bog's town") because of the many streams, ponds, and wetlands.   Our well has the benefit of being at near a drainage swale an nature keeps the aquifer seasonally recharged.

However, irrigating acres of orchards, paddocks, and produce as well as meeting the water needs of the 3 generations living at Unity Farm via a 7 gallons per minute well takes careful planning.

Here's what we do:

In 2012, we replaced the 18 year old well pump, wiring, and control system to give us the best infrastructure possible.   We also replaced the pressure tank.   The water is so chemically pure that it does not need a filter, a settling tank, or treatment of any kind. We were able to retire all the water treatment equipment that was initially installed after the well was drilled.   We placed digital flow gauges on the internal and external water mains so we can measure our water use and rapidly identify any unexpected variations such a broken pipe or leaking valve.

As we designed our animal and crop areas, we incorporated over 50 zones of irrigation controlled by two Hunter irrigation controllers, that enable us to selectively apply drip irrigation or low volume rotors to each area based on rainfall, plant maturity, and seasonal conditions.

For example, 7 gallons per minute can supply 420 one gallon per hour drip heads.   For each blueberry zone, we ran 400 feet of drip hose containing built-in one gallon per hour heads spaced every foot.   With two heads per plant and twice weekly 30 minute waterings, each plant receives 2 gallons per week precisely targeted to the roots.  We have 7 of these zones, running on Wednesday and Sunday nights between midnight and 3:30am.

Our pastures and paddocks have 23 zones of 4 low flow 1 gallon per minute rotors.   We run these for 10 minutes between midnight and 4am on Tuesday, Thursday, Saturday.

We tend to be generous with water to new plantings and parsimonious with water to well established plants.

We also have two rain sensors that automatically shutdown all our watering systems whenever more than half an inch of precipitation falls within a 24 hour period.

In summary we have

Crop zones
23 low volume rotor zones for pasture
7 drip lines for berries

House and Barn zones
8 low volume rotor zones for paddocks
4 drip lines for foundation plants around the house
11 low volume rotors zones for landscaping

We've also placed seven 75 foot hoses around the property for selective hand watering, cleaning, and animal care

From 1986-1993 we ran a vineyard on an arid mountaintop in Northern California and developed all the skills needed to carefully manage an agricultural operation from a well with limited capacity.

We've had record heat this season.  With targeted use of drip, rotators, and hand watering, everything is thriving on Unity farm.

Dispatch from Osaka

This week I'm in Osaka keynoting the IEEE Engineering in Medicine and Biology Society annual meeting.

Dr. Hiroyuki Yoshihara, my colleague from Kyoto University, hosted a pre-conference workshop on the state of EHRs throughout the world.   For years I've said that the challenges of EHR adoption and healthcare information exchange are the same in every country.    This workshop confirmed my impressions.   Speakers from Japan, New Zealand, and Brazil each spoke about issues such as:

Do we centralize data from multiple EHRs into a single repository for care coordination, patient engagement, and research?  Or do we create federated options with just in time data sharing from distributed EHRs?   The answer is that every locality approaches the problem based on requirements and policy concerns.   There is no one right answer.   Care coordination works well with distributed approaches (pushing summaries, pulling summaries, viewing external records) but analytics for population health and quality measurement benefit from data aggregation and normalization.

Do we accelerate change via top down government programs or bottom up industry driven initiatives?   The consensus was that top down, standardized, regulated approaches may temporarily slow innovation but they are effective in promulgating widespread adoption.   In the US, we've tripled adoption of EHRs since HITECH.  

Do we issue a national patient identifier (required or voluntary) to improve the accuracy of healthcare data aggregation?  Every country has different policy concerns and varying cultural tolerance of government mandated programs.   However, there was widespread agreement that an identifier for healthcare, required or voluntary, would make healthcare information exchange and analytics easier.   A curated national provider directory is also desirable.

How do we ensure data is comparable across multiple EHR systems?   Should we adopt detailed clinical models such as the work of the CIMI effort?  There was widespread agreement that vocabularies and clinical models are foundational to semantic interoperability and data aggregation.

Finally, how do we protect privacy?  Issues of consent, metadata, and de-identification are important throughout the world and approaches are highly variable based on culture, privacy laws, and regulations.   There is no single right answer but all agree that patient mediated exchange in which the patient is steward of their own information addresses many of the concerns.

My keynote today will focus on the grand challenges facing all engineers, informaticians, and policymaking professionals.  The five topics above - ubiquitous healthcare information exchange, universal adoption of standardized EHRs, accurately identifying patients/providers, creating foundational semantic interoperability tools, and protecting privacy are important worldwide and I look forward to solving them in our generation.

The FY14 BIDMC Information Technology Budget

Beth Israel Deaconess Medical Center runs on an October 1 to September 30 fiscal year, so Summer is
always a time of capital and operating budget decision making.

We've finished our FY14 capital budget submission and requested $10.7 million to fund all the needed infrastructure/applications supporting 3000 doctors, 22000 users, and a growing 1.2 billion dollar clinical enterprise.     This breaks down into $7.5 million for general operations, $3 million for security/regulatory initiatives and $210,000 (plus a $1.3 million FY13 carryover) for ICD10.

We've finished our FY14 operating budget submission and requested a base budget which is flat compared to FY13.   However, we have added two special operating budget requests - $2.6 million for security/regulatory initiatives ($1.3 million in new FTEs, $1.3 million in purchased services),  and $4 million in ICD-10 related costs.     Our total IS expense will be about 2 percent of the operating budget of the entire organization.

It is interesting to note that like many CIOs, more and more of my budget is directed to government mandated initiatives.  I've published the graphic above previously which shows what a healthcare CIO must do 2010-2015 -  Meaningful Use, ICD-10, HIPAA 5010, Physicians Quality Reporting Initiative, Value Based Purchasing, and Affordable Care Act/ACO formation all while ensuring appropriate protections are put in place to comply with the new HIPAA Omnibus Rule and increased Office of Civil Rights enforcement.   Also, there is an uptick of audits of all kinds motivated by federal, state, and internal risk reduction mandates.

My modest budgets are not a sign of unwillingness of the organization to allocate resources to IT - they are a function of the budget realities facing all healthcare organizations.

The work that we do is a function of time, scope, and resources.    As I reflect on the external pressures all CIOs face in 2014, change is accelerating, reducing time to complete projects.   Scope is increasing as healthcare moves from fee for service to global captitated risk.    Resources are shrinking with increasing pressure to reduce reimbursement, merge/acquire/affiliate, and overall reduce total medical expense associated with the care we give.

We all need to be very careful that CIOs and their staff do not approach the breaking point as the requirements for time, scope, and resources (FTEs, capital, operating) no longer align supply and demand. As I discussed in a previous blog post, we're moving forward diligently to inform the Board and senior management about the cost of every new project and deal.