Abstract: 

At UC Davis Health, telemedicine is used to connect our pediatric specialists to acutely ill children presenting to rural hospitals throughout northern California. Our team's efforts in the Department of Pediatrics have been centered on improving the technical reliability of telemedicine systems, the functionality and consistency of technology equipment, and network connections used during critical and sometimes lifesaving consultations. In addition, our team has worked to improve the clinical workflows of the telemedicine network, which connects to 40 different hospitals. 

Key interventions included standardized staff training across all hospitals in the network, routine test calls to evaluate audio, video, camera, microphone, and network quality, and training sessions provided to hospital staff to help increase their familiarity with the telemedicine carts. We developed call procedures and worked to troubleshoot any recurring technical and operational issues. These efforts are aimed to increase proficiency with the telemedicine carts, reduce technical errors, and ensure consistent connections for successful emergency connections across all sites. 

From these interventions, telemedicine workflows have become more consistent across all hospitals. With improved equipment functionality, increased telemedicine reliability and staff proficiency, our work has been crucial to supporting more effective emergency telemedicine consultations for critically ill children in northern California. 

Introduction: 

The COVID-19 pandemic had an explosive impact on the utilization of telemedicine as a method of providing healthcare services across the United States. By addressing geographic barriers, virtual patient care has enabled patients to continue receiving comprehensive consultations from their providers, regardless of location. This profound impact has transformed how health professionals provide care, leading to telemedicine’s continued widespread use, even years following the pandemic. 

At the national level, surveillance data from the Centers for Disease Control and Prevention indicated a rapid and substantial increase in telehealth utilization during the COVID-19 pandemic in the United States. An analysis of usage data from the largest telehealth providers in the country found that telehealth encounters increased by 50% in early 2020, with an overall 154% increase between March 2019 and March 2020 (CDC, 2020). Additional national studies indicate that there has been a statistically significant increase in telehealth visits in all specialties from 2019 to 2021. In a large study from the University of Arkansas for Medical Sciences Medical Center, from 2019 to 2020, the percentage of telemedicine encounters increased by 89%, which is an increase from 51,589 to 97,461 telemedicine visits. From 2020 to 2021, there was an additional 21% increase from 97,461 to 117,730 visits (Cengil et al., 2024). 

This surge highlights how telemedicine usage is now a vital part of providing modern day healthcare. It allows for less restrictive access to medical services, helping to overcome inconveniences of in person encounters. Additionally, not all health encounters need to be in person. Telemedicine models provide a new option for healthcare for these types of encounters. Studies have demonstrated that telemedicine has helped improve access to healthcare for underserved populations, reduce the waiting time for appointments, and increase flexibility in scheduling between patients and providers. However, telemedicine is not short of challenges. There are limitations in access to the internet, clinical and physical medical examinations, and different degrees of patient and provider familiarity with telehealth procedures and technology (Harju et al., 2022; Grossman et al., 2020). 

As telemedicine use has expanded nationally, it is critical to improve its operations and reliability to maintain quality healthcare. At UC Davis Health, our team's efforts in the Pediatric department have been centered on a particular use of telemedicine, used by pediatric critical care physicians to provide emergency telemedicine consultations to acutely ill children presenting to regional emergency departments. Our focus has been on improving the technical reliability of telehealth systems, the functionality and consistency of technological equipment, and network connections used during virtual visits. In addition to technical reliability, our team has been working to improve the operational reliability of the UC Davis Health pediatric telemedicine network, which connects UC Davis Children’s Hospital specialists with partner hospitals across California. 

Sufficient training of hospital staff is needed across all sites to ensure they are capable of consistently and effectively using telehealth. By improving both technical and operational reliability, we are working towards strengthening the efficiency, reliability, and clinical significance of telemedicine consultations. Increasing these factors helps to reduce the likelihood of failed and lost connections, delays in patient care, and faulty communication. 

Despite the rapid growth of telemedicine use since the COVID-19 pandemic, few studies have thoroughly analyzed the efforts to improve the reliability and impact of telemedicine used in this pediatric critical care telemedicine model of care. Current research highlights the implementation of telemedicine technology rather than the reliability, failure rates, and outcomes of this telemedicine model of care. This has left a major gap in the assessment of reliability of these encounters and patient outcomes. The goal of this study is to evaluate the reliability and clinical outcomes of pediatric emergency telemedicine encounters for a “low volume; high impact” model of telehealth care at UC Davis Health. Findings from this research will help guide future medical practice decisions and recommendations for telehealth use in pediatric healthcare. It is necessary to examine whether structured quality improvement efforts targeting telemedicine reliability lead to measurable improvements in healthcare performance and execution. How do quality improvement efforts focused on telemedicine reliability, including telemedicine training of healthcare teams, translate into measurable improvements in technical and operational failure rates, clinical practices, and pediatric patient outcomes? 

Methods:

To evaluate the reliability and clinical outcomes of pediatric telehealth, we carried out a series of quantitative operational assessments, telemedicine connection and equipment testing, and training sessions, which are described in the following section. This project is a part of an ongoing Quality Improvement research study in the Research Experience in Surgery Program at the UC Davis Medical Center’s Department of Surgery. It focuses on improving the operational and technical reliability of telemedicine systems during pediatric consultations from the UC Davis Children’s Hospital to its partner hospitals across California. The study includes many regional and community hospitals across California with a total of 40 carts across 20 sites. 

Routine test calls were conducted by trained research assistants starting in September 2024. Test calls are performed to evaluate audio, video, camera, microphone, and network quality, as well as confirming proper equipment function, and identification of any technical issues prior to patient-provider encounters. In addition, training sessions are provided to hospital staff to help increase their familiarity with telemedicine carts, virtual call procedures, and to troubleshoot any recurring issues. This leads to increased proficiency with telehealth carts, reduces errors, and ensures consistent connections across pediatric sites. Research assistants were trained technically and operationally to be able to assist different hospital staff through the telemedicine workflow operations. During this study, no patient level information was collected during this phase of the study. 

Routine test calls between trained research assistants and hospital staff were conducted to assess the functionality of telehealth equipment and the hospital staff’s proficiency at operations. During each test call, research assistants walk hospital staff through each step to ensure proper operation of the technology. After successfully connecting, research assistants assess video resolution, camera movement and zoom, audio level, functionality of the microphone, and network stability. All details of each cart’s performance were documented, with issues noted and sent directly to IT teams (tickets submitted). Through the documentation of test calls and issues, we are able to identify patterns in equipment reliability, identify common technical issues between sites, and identify problematic sites, all while coordinating with IT services to fix persistent issues. 

To improve operational reliability, structured training sessions were provided across hospital sites. Each site was provided with an instruction sheet for each cart, and our training focused on ensuring that hospital staff is consistently able to follow connection procedures and operate telemedicine carts. The participation and proficiency of staff was documented to monitor progress and mark sites requiring more support. 

All test call results, interventions, and staff training are logged. Our documentation includes the status of each cart’s functionality, the success rate of each cart, common technical and operational issues, and recommendations for future training. Our logs are utilized to understand site-specific reliability, assessing compliance, and providing data to highlight sites that face consistent issues. 

Findings from routine tests were regularly delivered to Telemedicine Information Technology (IT) teams at UC Davis. IT Teams were able to help identify both hardware and software issues, repair technology with issues, and coordinate further efforts on issues that remain unresolved. Biweekly IT meetings help the team communicate, highlight important updates, and stay on track with the progress of each site. 

To evaluate the telemedicine performance across hospitals, data collected from routine test calls were systematically organized and assessed over time. To allow for longitudinal assessment of telemedicine reliability, operational performance, and hospital level compliance, the outcomes of test calls were aggregated by month. Several graphs and models were used to analyze the patterns in the outcomes of test calls, trends in operational reliability, trends in technology, and to evaluate the consistency of performance throughout hospital sites. These analyses were designed to assess the overall reliability of sites, their changes over time, and finally, variability across sites and hospitals. Test calls were not performed during October 2025 and December 2025, as a result these months were not included in the dataset. 

Telehealth test outcomes were illustrated by month using a bar graph that depicts changes in cart performance over time (Figure 1). Every month, all routine test calls conducted were categorized into one of four outcome groups: (1) successful test calls, (2) test calls that had technical difficulties, (3) test calls that required additional training of staff, and (4) sites that were not available to test. The total number of test calls conducted each month was used as the denominator for this model. The counts of each outcome category served as the numerator for this model. Percentages of total monthly test calls allowed for standardized comparison throughout test months. This model allowed the assessment of trends in telemedicine reliability and operational performance over time. This highlights the changes in the frequency of successful connections, technical issues, and training that is needed across each site.  

Figure1: Telehealth Test Call Data by Month

Monthly telehealth test call outcomes were further modeled using a line graph to illustrate changes in the trends of performance over time. We created four total categories: (1) successful test calls, (2) test calls with technical difficulties, (3) test calls requiring additional training, (4) and sites unavailable for testing. Percentages were calculated with consistent comparison across time, using the number of test calls conducted each month as the denominator. This was done to show temporal trends in telemedicine reliability by showing how the proportion of each outcome category shifts over time throughout several months (Figure 2).

Figure 2: Telemedicine Test Call Data Report per Month

Next, monthly technology issues were compared between UC Davis Telemedicine carts, and non-UC Davis Telemedicine carts using a bar graph. For each month, all reported technical issues that were identified during routine test calls were analyzed and categorized either as a UC Davis Cart or a non-UC Davis Cart. The total number of technical issues in each category was aggregated and displayed to show direct comparison between UC Davis equipment and non-UC Davis equipment over time. This analysis was used to analyze the relative patterns of operational efficiency and technical reliability between UC Davis and non-UC Davis carts (Figure 3).

Figure 3: Monthly Technological Issues Reported from UC Davis vs Site Carts

The frequency of successful test calls was further evaluated using a line graph to assess changes over time and compare the performance between UC Davis and non-UC Davis carts. For each month, test call outcomes were analyzed to find which calls were successfully completed. Successful calls were categorized by cart location as either UC Davis or non-UC Davis carts. Success rates per month were calculated by dividing the number of successful test calls in each category by the total number of test calls that were conducted during that month. These values were converted into percentages and plotted over time to show the comparison of success trends between UC Davis carts and non-UC Davis carts. This allowed for the evaluation of changes in telemedicine reliability across all the sites (Figure 4).

Figure 4: Number of Successful Test Calls Per Month; UC Davis vs. Site Carts

Monthly compliance was assessed to evaluate the extent to which each hospital successfully completed routine test calls across each cart. For each month, compliance was defined at the hospital level. A hospital is considered compliant if every telemedicine cart at that site achieved at least one successful test call during the month. To calculate monthly compliance, the total number of hospitals contacted during each month was recorded while identifying the number of hospitals that met the full criteria for compliance. Compliance rates were determined by dividing the number of fully compliant hospitals by the total number of hospitals that were contacted that month. This allowed for the standardized comparison of hospital level compliance over time. This elucidates the consistency of telemedicine reliability across sites with multiple carts (Figure 5).

Figure 5: Monthly Compliance by Percent

This work was conducted with the goal of quality improvement, without involving patient identifiable data and patient information. All procedures performed in this study follow the UC Davis Health Privacy Policies and Institutional guidelines for operational improvement projects. 

Results Overview of Telemedicine Testing and the Analytical Approach 

Routine test calls were conducted across pediatric telemedicine sites starting in September 2024. The test call data was aggregated each month and analyzed at both the cart and hospital levels. This was done to evaluate the technical reliability, operational performance, and consistency of calls across all sites. A series of models, including bar graphs and line graphs were created to illustrate the distributions of outcomes, show trends over time, highlight technology issues per each cart, and inform hospital level-compliance. These analyses together provide a longitudinal overview of the telemedicine performance of sites while identifying patterns in reliability and operational consistency over time. 

Monthly Telehealth Test Call Outcomes

Telehealth test call outcomes were summarized by month using models to illustrate the change in performance over time (Figure 1). Each routine test call was placed into one of four categories of outcome groups: (1) Successful test calls, (2) test calls that had technical difficulties, (3) test calls that required additional training of staff, and (4) sites that were not available for testing. 

Greater variability was found in earlier months of testing. This variability includes greater proportions of technical issues and unavailable sites for test calls. As the testing continued over time, the proportion of successful test calls increased throughout the year. At the same time, the proportion of calls affected by technical problems and site unavailability decreased. Furthermore, as some months showed an increase in the number of calls requiring additional training of staff, these increases occurred alongside increased testing volume and greater participation of staff. Overall, the distribution of outcome shifted towards higher proportions of successful test calls as the study progressed. 

Temporal Trends in Test Call Outcomes 

Monthly test call outcomes were analyzed using a line graph to examine the changes in the proportions of outcomes over time. The x-axis represented dates, and the y-axis represented the percentage of calls in each outcome category. Separate lines were used to represent successful test calls, technical issues, sites requiring training, and sites unavailable for testing. 

This visualization emphasized temporal trends in the reliability of telemedicine across the time of the study (Figure 2). Early in the testing period, there were greater fluctuations in outcome categories, specifically in technical difficulties and the unavailability of sites. Over time, these proportions became more stable, showing a consistent increase in the proportion of successful test calls. The proportion of test calls requiring training showed increases, illustrating continuing staff engagement. This line graph highlights changes in the proportions of outcomes over time, showing progression and a shift towards more reliable telemedicine performance.

Technology Issues by Cart Location 

Monthly issues were compared between UC Davis telemedicine carts and other non-UC Davis carts using a bar graph. For each month, all technical issues were identified during test calls, were reviewed, and were categorized based on the location of the cart. The total number of technical problems of UC Davis carts and other carts was aggregated monthly to compare results over time. During the early months of testing, partner sites had greater reported technological issues compared to UC Davis carts. These issues included network issues, malfunctioning of video and audio, connectivity issues, and challenges in managing equipment. The frequency of technical issues reported for non-UC Davis cars decreased over time. Later on, the number of technical issues at non-UC Davis sites reached similar levels to UC Davis sites. The UC Davis carts illustrated relatively consistent technical performance throughout the study, with less variability in reported issues. These findings highlight the changing patterns in the technical reliability of different carts and sites over time. 

Successful Test Calls by Cart Location 

he frequency of successful test calls was evaluated further using a line graph that compared the performance trends between UC Davis carts and non-UC Davis carts. For each month, test call results were analyzed to identify successful connections. Each successful test call was categorized by the cart’s location, and monthly success rates were calculated, converted into percentages, and plotted over time.

Earlier months of test calls showed lower success rates for UC Davis sites and non-UC Davis Sites. As testing continued, success rates increased for both cart types. With increased testing volume and staff training at non-UC Davis sites, they saw a greater proportion of successful test calls at a midpoint of the study. Later on, UC Davis Carts showed higher success rates. Due to a decrease in test calls in the month of August, a temporary decrease in success rates was observed. Overall, the success rates for both types of carts increased over time, illustrating improved reliability of telemedicine across locations. 

Monthly Hospital Wide Compliance 

Hospital level compliance was analyzed monthly to evaluate the consistency of routine telemedicine testing across single cart sites and multi cart sites. At the hospital level, a hospital is considered compliant if every telemedicine cart at that site had at least one successful test call during that month. The total number of hospitals contacted and the number of hospitals meeting the criteria of full compliance was recorded each month. Compliance rates were calculated by dividing the number of fully compliant hospitals by the total number of hospitals that were contacted that month. 

Earlier in the study, compliance rates were lower. As routine test calls became more consistent and standardized, hospital level compliance increased throughout subsequent months. Compliance rates increase gradually over time. Ultimately, 100% compliance across all hospitals was reached in August 2025. This shift illustrates that increasing the frequency of telemedicine testing improves the consistency of telemedicine performance at the hospital level. 

Summary of Reliability Trends 

The analysis of all results demonstrated progressive improvement in technical reliability, operational performance, and hospital level compliance. Improvements were observed and analyzed across several different outcome measures over time. Reduced variability in results, increased rates of success, decreased reported technical issues, and greater compliance rates illustrate increased telemedicine reliability across sites. These findings provide a complete overview of the trends in telemedicine performance over time. 

Discussion: 

This quality improvement study assessed structured methods to improve the operation and reliability of telemedicine, specifically through routine testing of technology and training of healthcare staff. This was translated into measurable shifts in technical and operational performance across all sites. Using multiple analytic approaches, results showed progressive improvements in successful connections, decreased technical failures, and increased hospital level compliance over time. Together, these findings suggest that specific operational interventions may play a key role in improving telemedicine reliability in high impact pediatric healthcare.

A key finding of this study was the consistent increase in successful test calls across sites over time. Earlier testing demonstrated increased variability with higher rates of technical difficulties, and unavailable sites. As routine test calls and contact with sites proceeded and increased, these challenges gradually decreased while performance stabilized. The trends found from several run charts indicate improvement at the system level. Causality cannot be established in this observational quality improvement, however, the consistency of improvement across various outcomes and sites supports the finding that operational interventions are meaningfully associated with improved reliability of telemedicine.

A plausible explanation for these improvements is increased healthcare staff engagement resulting from regular and consistent outreach to hospitals. Routine test calls could have potentially acted as reminders that reinforce expectations about the readiness of carts, maintenance of equipment, and efficient operational management. Frequent contact to sites likely encouraged providers to keep carts on, accessible, and updated, helping to decrease failures due to operational oversights that are preventable. Additionally, consistent repeated exposure to standardized testing procedures likely increased familiarity with telemedicine workflows.

Training also likely contributed to improvements in operational reliability. Training of staff focused on the standardization of connection procedures, functionality of equipment, and the trouble shooting of common issues. Throughout time, the increased familiarity with the workflow of telemedicine carts likely reduced error caused by users, and increased hospital staff confidence in the utilization of carts. Few intermittent increases observed in outcomes where training was needed reflects continued staff onboarding, learning, and training. The identification of sites that need training allowed targeted support per site, instead of troubleshooting during live patient visits.

Comparing UC Davis carts and non-UC Davis carts provided further insight into reliability across all sites. Early on, partner sites had greater rates of reported technical issues compared to UC Davis carts. This difference likely reflects differences in the sites’ infrastructure, technical support on site, and staff experience with telemedicine systems. Over time, the frequency of technical issues of non-UC Davis carts neared the same levels as UC Davis carts. This convergence suggests that through repeated testing, documentation of calls, and collaboration with Information Technology services, disparities in telemedicine reliability can be reduced throughout different hospital sites.

Throughout the study period, hospital level compliance increased gradually, finally reaching 100% compliance. Defining compliance at the hospital level allowed for a strict measure of operational consistency. Improvements in compliance suggest that hospitals became more capable of maintaining consistent readiness across several carts, even with an increase in the volume of testing. Readiness across all carts is critical in pediatric telemedicine to ensure timely and efficient access to care.

This study focused primarily on technical and operational reliability, which both have important implications for pediatric clinical care. As failed connections, delayed consults, and disruptions in communication can compromise efficient patient care, reducing the likelihood of technical failures and operational issues in telemedicine is critical.

The results also emphasize the impotence of routine testing and outreach as a strategy to improve connection outcomes and quality. Outreach calls can be implemented throughout various healthcare settings without requiring major processes. The consistent improvement observed over time illustrates that interventions like these may provide effective benefits over time. Maintaining consistent operational readiness is critical for clinical telemedicine.

It is critical to consider the limitations of this study when interpreting its findings. As a quality improvement study, the analyses demonstrate associations, and not causal relationships. Measuring confounding factors influenced the improvements that were observed over time. These include changes in staff, variation in seasons, and greater familiarity with telehealth technology after the COVID-19 pandemic. Run charges are helpful to visualize trends, but they do not control for external variables or provide formal statistical inference. Furthermore, patient level outcome data is absent in this study. The study does not directly assess whether these reliability improvements can be applied to improved clinical outcomes. Finally, because this study was conducted over one group of select hospitals across northern California, generalizability to other models or institutions. Future quality improvement initiatives can build on this work but incorporating further study designs to better highlight the impact of outreach interventions.

Overall, this study illustrates that structured technological and operational interventions including routine test calls, staff training, and consistent outreach, are associated with measurable improvements in telemedicine reliability. These improvements are seen in technical reliability, operational performance, and hospital level compliance. These results show the significance of proactive quality improvement methods to help improve telemedicine systems. They highlight the importance of reliability as part of the foundation of effective telehealth care.

References 

1. Cengil, A. B., Eksioglu, S., Eksioglu, B., Eswaran, H., Hayes, C. J., & Bogulski, C. A. (2024). Statistical analysis of telehealth use and pre- and postpandemic insurance coverage in selected health care specialties in a large health care system in Arkansas: Comparative cross-sectional study. Journal of Medical Internet Research, 26, e49190. https://doi.org/10.2196/49190 
2. Grossman, Z., Chodick, G., Reingold, S. M., Chapnick, G., & Ashkenazi, S. (2020). The future of telemedicine visits after COVID-19: Perceptions of primary care pediatricians. Israel Journal of Health Policy Research, 9(1), 53. https://doi.org/10.1186/s13584-020-00414-0 
3. Harju, A., & Neufeld, J. (2022). Telehealth utilization during the COVID-19 pandemic: A preliminary selective review. Telemedicine Reports, 3(1), 38–47. https://doi.org/10.1089/tmr.2021.0040 
4. Koonin, L. M., Hoots, B., Tsang, C. A., Leroy, Z., Farris, K., Jolly, B. T., Antall, P., McCabe, B., Zelis, C. B. R., Tong, I., & Harris, A. M. (2020). Trends in the use of telehealth during the emergence of the COVID-19 pandemic — United States, January–March 2020. Morbidity and Mortality Weekly Report, 69(43), 1595–1599. https://doi.org/10.15585/mmwr.mm6943a3