Codonics – Final Co-op Report – Spring 2012

During my co-op at Codonics, I have been heavily involved in implementing wireless and networking support for Safe Label System, a medical device used in Operating Rooms to help reduce the number of medical mistakes in syringe labeling.

Responsibilities

            During my second co-op at Codonics, my responsibilities were focused on software development; creating new features that were required in order stabilize and market the Safe Label System. Sometimes I was given a very specific set of requested features, and asked to implement them exactly as specified. Other times, I was given room to prototype possible solutions to common requests, and design a solution that would meet customer needs in a safe and effective way. My role as a software engineer began to extend into product development, where I was designing new features without specific requirements, trying to find ways in which I could improve the existing system.

            Near the end of my second co-op I began to help setup the SLS System at various hospitals across the United States. I worked with Massachusetts General Hospital, Porter Medical Center, Northbay Medical Center, and St. Vincent Charity Medical Center to install the Codonics Safe Label System at their sites. In all cases, I was responsible for diagnosing any problems during the setup, including wireless connectivity, network setup, and email alerts. I had to think on my feet when issues arose during these installs, because I had no way of changing any code if a defect was found. I was responsible for ensuring the success of the technical part of the installation. In some case, I was flown to the site, or participated in conference calls with IT Administrators.

Skills

            My co-op at Codonics has helped to hone my skills in programming, as well as understand advanced concepts like multi-threaded applications and wireless networking protocols. By writing the networking code for the SLS in Java, I have gained a lot of knowledge of how wireless networks work, can be configured, and optimized.  I have spent many hours testing wireless devices, building drivers, and setting up various server configurations to test various types of wireless networks. By designing the networking code in Java, I became familiar with how the underlying Linux OS works with networking, and since then I have began thinking of how it could be improved. I even started my own open source project that communicates with the Linux kernel at a very low level in order to get information about existing network connections, and relay that information to Java through a Java Native Interface.

During the times that I was on site at various hospitals, I improved my communication skills by talking with nurses, doctors, and IT administrations about their needs and existing network infrastructure. This allowed me to improve my communication skills with non-engineers, who may not understand some of the technical lingo used by engineers. The ability to communicate effectively is not a skill that is easily learned in a classroom environment, so my experience at Codonics has been very valuable.

Reflections

My co-op at Codonics has helped me to develop many skills related to Computer and Software Engineering. I have spent much of time developing new software features that have helped to fuel sales of a medical product that has the capability to save lives in hospitals across the world. There is a lot more to being a software developer than just writing code. It’s important to go out in the real world to meet your customers; getting to know the customer will help you define new features for your product that will continue to make it relevant in your industry.

Continual improvement of the Safe Label System will benefit not only the customer, but the nurses and patients as well. Establishing relationships with customers helps to gain their feedback about what features work well, and which ones need reconsidered. It’s a continual feedback process that develops and matures the product over many cycles. The software must be designed in a way that is atheistically pleasing, easy to use, and efficient. Sometimes decision revolving around the UI can take several weeks or even months, but it is important to choose the best option for customers.

My co-op has allowed me to apply many courses to real-world applications. My Advanced Game Design course gave me a deeper understanding of the Java language, which improved my ability to write code for the Safe Label System. I have become very proficient in Java, and I hope to continue that skill into creating mobile applications for Android. Additionally, my Operating Systems course helped to me to understand the complications that can arise in multi-threaded applications. This knowledge helped me to write performance enhancement and fix bugs relating to concurrency.

I am excited to be returning back to school to complete my Bachelors degree. My current plans are to continue one extra semester to obtain a Masters degree as well. My co-op at Codonics has helped give me an idea of what software engineering is like, and I will be excited to obtain a full-time position in software when I have completed my schoolwork. My co-op has shown to me that I am well adept in a field that I enjoy very much. I look forward to the completion of my studies at Case Western, and I am confident I will be ready to work as a software engineer upon graduation.

Setting the nofile limit in upstart

While working on a Java project that executes multiple sub processes, I found it necessary to increase to the file descriptor limit. By default, processes spawned in Ubuntu had a limit of 1024 open files, but changing that limit via ulimit or by /etc/security/limits.conf had no effect. I discovered that changes in /etc/security/limits.conf (and subsequently /etc/pam.d/common-session ) are only applied to process spawned by a login shell. Programs that launch on startup via “upstart” do not get these limits applied to them. Thankfully, upstart provides the limit stanza that allows you to modify some of these parameters, including the maximum number of open files. To see the limits on process, grab its PID, and cat /proc/<<PID>>/limits

The following is an example of increase the maximum number of open files to 4096 for a given upstart job:

limit nofile 4096 4096

You can checkout the full set of limit options at: http://upstart.ubuntu.com/wiki/Stanzas#limit

Note: Some of these limits (if not all) are transferred to any children that are forked by the parent

Check out upstart at: http://upstart.ubuntu.com/

A warning about Runtime.getRuntime.exec() : You may be leaking File Descriptors

Normally Java’s online documentation is very informational, but this not one of those times. Recently, I had run into an issue where my application was issuing: java.io.IOException: error=24, Too many open files.

If you are making a lot of use of the Java’s Runtime.getRuntime.exec() or ProcessBuilder, then its very possible you are leaking File descriptors. Unix systems have a limit to how many files can be opened at any time by a given process. On Ubuntu 10.04, that limit is 1024 files. You can find out your system’s max limit per process by issuing: ulimit -n

Every time you execute a process using Java’s Runtime, STDIN, STDOUT, and STDERR are piped to Java as an Output or Input Stream. For each stream, there exists a pair of file descriptors. When the process has terminated, these streams are closed (in general) at the process side, closing on of the two file descriptors for that stream. If your Java program continues on, and you don’t use those streams in Java, then you are leaking the File descriptors on the Java side of the pipe (assuming your not specifically closing them or destroying the process). Normally, these will be closed when Garbage Collecting occurs, but you shouldn’t leave it up to the Garbage Collector to decide.

After you use ProcessBuilder or Runtime.getRuntime.exec(), you can do one of two things:

  1. Use a handler to the Process that was executed (say: Process p), and destroy it (Note: This also terminates the process if its running).  p.destroy();
  2. Close all the input streams if you haven’t already:
1
2
3
4
Process p = Runtime.getRuntime().exec( <command></command> );
p.getInputStream().close();
p.getOutputStream().close();
p.getErrorStream().close();

Of course, if you used one of the above streams within say: a BufferedReader, you just need to close the buffered reader, and the underlying streams will be closed automatically, Once these streams are closed, their file descriptors are removed. If you want to take a look at a list of opens files: lsof

If you can get the PID of your Java process, you can filter out most of results from lsof by doing: lsof -p 1234

Getting a count is also easy:

lsof | wc -l

lsof -p 1234 | wc -l

Forking the JVM

java.io.IOException: error=12, Cannot allocate memory

While working on an embedded system with tight memory constraints, I discovered an inefficiency with the way Java executes processes. For this embedded system, it was sometimes necessary to execute bash scripts, which can be done by using Java’s ProcessBuilder. In general, when you execute a process, you must first fork() and then exec(). Forking creates a child process by duplicating the current process. Then, you call exec() to change the “process image” to a new “process image”, essentially executing different code within the child process. In general, this is how you create new processes to execute other programs / scripts. When attempting to execute these smaller processes from Java, I began receiving “java.io.IOException: error=12, Cannot allocate memory”.

 

The problem lies within fork().

 

An embedded system has a limited amount of memory; take 512MB for example. When we want to fork a new process, we have to copy the ENTIRE Java JVM… well… almost. What we really are doing is requesting the same amount of memory the JVM been allocated. So we want to execute a new process, and our JVM is taking up 350MB of memory, then our fork will request 350MB of memory. But wait! We only have 512MB! We may not have enough memory to fork our current UI, but all we want to do is execute a command, say… “ls”. The “ls” command doesn’t require 350MB, but if there isn’t at least 350MB, we may not be able to even fork a new child process. Now, the important thing to remember is when we fork, we are just requesting 350MB of memory; we are not going to use it necessarily. If we were to fork(), and the immediately call exec(), we wouldn’t use all of the 350MB.

1. A Second JVM

Our first attempt at a solution may involve using a second JVM. This whole mess is a result of our main program being too large in memory to exist twice. We could create a Java process, whose sole purpose is to execute our tasks. Such a JVM would be much smaller than our main program, and therefore fork() would have a better chance at get enough  memory allocated to spawn a child process. This solution is really only a workaround, because we did not eliminate the problem. What if our memory eventually gets so full, that our smaller JVM is unable to fork() because there just isn’t enough to allocate? What if our second JVM suddenly crashes? A second JVM is a potential solution, but not the only solution. If you’re interested, you can find an example of this idea at: http://www.assembla.com/spaces/forkbuddy/wiki

2. Custom JNI bindings for posix_spawn()

This whole mess began because of fork(), and how it copies one process to another in order to create the child process. Under the covers, Java’s ProcessBuilder uses fork/exec, but those are not the only UNIX commands that can be used to spawn processes. posix_spawn() is a different command that allows you to create a new child process, using a specified process image. There is no copying of the parent process. I guess Sun or Oracle never got around to using this implementation, which would certainly solve our dilemma. The only way to implement this in Java is to create a custom JNI binding. The Java Native Interface (JNI) is a way of calling native code (C/C++) from the Java Virtual Machine. You can find a similar implementation at https://github.com/bxm156/java_posix_spawn. Except instead of using posix_spawn(), we implement it ourselves. It turns out that not all Linux distributions implement posix_spawn() using vfork(), so its best just to write our own vfork() and exec() implementation. Instead of allocating memory for the new child process, vfork() allows the child process to share the same memory as the parent. There are both benefits and risks to using vfork(), but for our case, where we want to launch some other type of process, vfork() can be executed safely. Since we call exec() right after vfork(), we don’t need to worry about the parent process’s memory getting modified. If we were to vfork(), and then do something else besides exec(), we would risk modifying our parent process.

3. Over-commit

Our last option is the easiest fix, but comes at a cost. Every time we fork, the OS determines if enough memory can be allocated to copy our process to a child process. We can turn that safety-check off, and let the OS allow us to fork a new child process, even if there is not enough memory available to be allocated to make a copy of the parent. For our purposes, our scripts that we want to execute are always going to be much smaller than a 350MB JVM. So we probably will be fine. By enabling over-commit, we solve the problem, but there is a cost.

When you allocate memory in C, malloc() returns either a pointer to the memory address, or null if no memory could be allocated. If the programs on your system were written correctly (and by correctly, I mean with some sanity checking), they should attempt to die gracefully if they receive null from malloc(). When that happens, it means the system’s memory completely used up. When over-commit is enabled however, malloc() will NEVER return null. In fact, it will always return some pointer to a memory address, even if that address is in use (or maybe it doesn’t even exist).  At that point, the OS will release the dreaded OOM killer.Which will began terminate programs as it sees fit, in order to free up the memory. In general, the way in which it picks its victims is at random. Andries Brouwer came up with this analogy:

An aircraft company discovered that it was cheaper to fly its planes with less fuel on board. The planes would be lighter and use less fuel and money was saved. On rare occasions however the amount of fuel was insufficient, and the plane would crash. This problem was solved by the engineers of the company by the development of a special OOF (out-of-fuel) mechanism. In emergency cases a passenger was selected and thrown out of the plane. (When necessary, the procedure was repeated.) A large body of theory was developed and many publications were devoted to the problem of properly selecting the victim to be ejected. Should the victim be chosen at random? Or should one choose the heaviest person? Or the oldest? Should passengers pay in order not to be ejected, so that the victim would be the poorest on board? And if for example the heaviest person was chosen, should there be a special exception in case that was the pilot? Should first class passengers be exempted? Now that the OOF mechanism existed, it would be activated every now and then, and eject passengers even when there was no fuel shortage. The engineers are still studying precisely how this malfunction is caused.

To enable over-commit temporarily:

echo 1 > /proc/sys/vm/overcommit_memory

For a more permeant solution, you will need to edit /etc/sysctl.conf and add the following:

vm.overcommit_memory = 1

Now just restart your system for the change to take effect.

Codonics Final Co-op Report for Fall 2011

Codonics

During my co-op at Codonics, I have been heavily involved in implementing wireless and networking support for the Safe Label System, a medical device used in Operating Rooms to help reduce the number of medical mistakes in syringe labeling.

 

Responsibilities

At the beginning of my co-op at Codonics, my responsibilities were focused on testing software. I used a system called TestTrack to record bugs and related information. During development, developers updated these TestTrack tickets, and it was my job to verify the bugs were fixed.  In addition, I wrote up Test Reports, which detailed tests that I preformed to verify some type of functionality as defined in the functional specifications. One of my tests involved the creation of a webserver that would display barcodes on a web page. This webpage was displayed on an iPod Touch, and placed directly underneath a scanner. My test helped to verify the long-term functionality of the Safe Label System by printing over 1,000 labels automatically.

My largest responsibility at Codonics by far has been to oversee the development of the Safe Label System’s networking capabilities. I spent a few weeks testing multiple wireless adapters, and determining what software was needed to utilize them. Then, I began to write the software that would allow users to connect to wireless networks on the device. As the requirements changed, I have been updating the networking code. Recently, I have been implementing support for WPA/WPA2 Enterprise networks, such as EAP-TLS and PEAPv0-MSCHAPv2. These networks use certificates to encrypt and authenticate connections, which adds complexity to the software. I had to consider the best way to store this information, without letting it get into the hands of unauthorized users. Since these types of networks are used in hospitals, it is important that the software correctly handles such network authentication protocols.

 

Skills

Codonics has helped me to further supplement skills that I would not have received through classes at Case Western Reserve University. I gained experience with writing test procedures using LaTeX, a typesetting language used to create professional looking documents. I also learned how to write scripts using AutoIt, in order to automate many of the test procedures. Codonics also introduced me to JUnit tests, which are small pieces of code that test production code in order to look for regressions in functionality. These test are useful when multiple programmers are working on a project, or when a large refactoring of logic or code occurs.

I have used many skills that I learned in class while at my co-op with Codonics. My Advanced Game Design class helped me further familiarize myself with Java, a programming language used extensively during my co-op. I used the knowledge in my Operating Systems class to help develop the wireless networking code, as well as the restricted shell environment used in the Safe Label System. Operating Systems class helped to further my knowledge of concurrent processes and thread-safety, which I incorporated into my networking code. If multiple processes or threads attempted to manage a single wireless device, the adapter could end up getting configured improperly. In order to prevent such an occurrence, I implemented the use of a single thread pool that executes task synchronously, based on a given priority. Operating Systems, as well as Compilers, increased my knowledge of programming languages such as C and C++. These two languages are used in creating the restricted shell environment, which only allows authorized users to execute pre-specified commands, while blocking the use of others. There are cases in which we want to allow the user to run a pre-defined script, which could execute commands not normally allowed. In that case, the restricted environment is able to break out of the restrictions in order to use those specific commands, as well as access specific files and directories that are not normally permissible.

 

Reflections

My co-op at Codonics has helped me develop real-world experience in developing software as part of a team. I spent a majority of time at the beginning of my co-op doing testing, and it allowed me to understand how the products functioned in respect to the end user.  Before my co-op at Codonics, I never realized how much effort goes into testing, especially for medical products. For example, there are many test procedures that need to be executed on every release candidate to ensure there have been no regressions. New test procedures are created from test reports, which are sometimes planned out before the functionality has even been implemented. As I became familiar with the products, I began writing my own tests. My first test report covered printing 1,000 labels on an SLS, to ensure there were no memory leaks. In order to run the test, I created a webserver that generated barcode images, and loaded it onto an iPod Touch. The plan worked successfully, and soon my SLS unit was printing barcodes non-stop, automatically. My test interested many of my coworkers and supervisors, who stopped by to see the test in action. Before my barcode server, tests were done manually by testers, which consumed valuable time. In addition, testers were only able to test a limited number of drug vials that were around during the testing. My barcode scanner was able to test every drug known to the SLS, because it generated the barcodes automatically from a list of drugs in the SLS system. I hope that my code helps to shorten the time it takes to test the Safe Label System, and give testers more time to investigate other functional aspects of the device.

As I moved to more of a developer role, I began to learn the importance of writing unit tests. I was given a book on JUnit, and from there began writing and updating unit tests as I worked on the SLS project. Unit tests are small pieces of code that test the functionality of production code. These tests are very valuable when multiple developers are working on a project, and help to detect regressions quickly. There are some downsides however, because time must be taken to update the unit tests when the logic has changed. During my co-op I wrote the networking system for both wired and wireless network interfaces, and was forced to refactor my code as the input specifications changed. Unit testing helped to ensure the overall functionality of my design did not regress after refactoring the logic, and served to increase my confidence in my code changes.

I learned the important of writing software to technical specifications, which allows multiple programmers to work on various parts of the code at separate times. When code is written as agreed upon, it makes for an easier integration when developers begin to merge their code together. At first, such specifications can seem tedious, especially when they change as the software matures, but these specifications help to ensure the software behaves as expected, and can also serve as an outline that can be used by the Testing department to ensure a thorough examination of the software.

While working on Codonics, I was given the opportunity to do some business travel, an opportunity rarely given to co-op students. My first time traveling as a Codonics employee was to Anesthesiology 2011 in Chicago. While working at the tradeshow, I learned about various vendors in the medical industry, and how the Safe Label System positioned itself among them. During off the clock hours, I relaxed with fellow employees and learned more about the company, employees, and culture.

My second trip representing Codonics was to Boston, and it was my first time flying by myself. It was a very new experience for me, because I was responsible for taking along a Safe Label System to demonstrate the new networking capabilities at Massachusetts General Hospital. Once in Boston, I worked with MGH staff in order to verify our software was able to connect to their EAP-TLS network. I caught a glimpse of how a hospital’s IT staff was structured, and how different people in the chain were able to help push agendas along. We discussed networking requirements, user interface preferences, and likely use cases for how the Safe Label System would be networked in the operating rooms. I was awe-struck at how large the hospital complex was, which impacted my view of how important networking support would be for our next release of software. If a hospital buys hundreds of units, and places them in various buildings, it would be a challenge to update and verify their all working properly.  Networking support will give administrators control of all the units from one computer, with the ability to push updates out to all the systems with a push of a button. In addition, network administrators will be able to monitor the status of the units, and receive email notifications if something unexpected occurs, such as an unknown drug being scanned. The software I developed will help to ease the introduction of the Safe Label System into large hospitals, and reduce medical mistakes by improperly labeled drugs. Being given the chance to travel with Codonics has been a great privilege that has helped me to see other parts of the company that I would not have normally be exposed to.

I have been very happy with my co-o p at Codonics, and the experience has helped to boost my confidence in my programming skills. Codonics has helped me grow as a computer engineer, and I hope to continue learning more during my second co-op with them.

Privacy Policy