Ameloblastoma 04: Follow-up

Nov 23rd, 2020 -- All clear on the year and half follow-up. Our best thanksgiving ever. 

We were somewhat nervous since we missed the one year follow-up due to COVID-19.

Ameloblastoma 03 (Drafted on 06/20/2019): Surgery and Recovery

We are home safely

The Surgery

We arrived and settled in on June 3rd, one day before the pre-op appointments. The trip was event-less, but there was the great suspense of what to expect.

The pre-op appointments, including a medical grade CT, pre-anesthesia appointment, and talking to the surgeon, went smoothly. The process made us feel "yep, we are doing this". The day after, we arrived at the hospital a little after 6 AM for the 8:30 AM surgery. Being first time in a hospital for surgery, we began to learn the terms such as prep, in ops, recovery, recovery II. We were with Elvis for quite a while in the prep room (with his gown on and even the IV started), until shortly before 8, when he was wheeled into the operations room, and we ushered back to the lounge.

The monitors in the lounge show the status of each patient (of course without any personally identifiable information). The staff give regular updates like "family of who, the procedure has begun..." The surgery was scheduled to finish at 10 AM or so, at that time we were informed everything was going fine, just took a little longer. The surgeon came out around 10:10'ish, and told us all went well.

Since Elvis was taking more time to wake up from anesthesia, we decided to have a quick lunch in the cafeteria. Around 2 PM, we were led to see Elvis in Recovery II -- swollen, ice pack around his jaws, gauze with blood in his mouth, and subdued. A few minutes later, the IV was done, and Elvis wanted to go pee. Called the nurse to take the IV out, and we helped him to the bathroom, then changed clothes, and were discharged. Of course, the nurse went through the discharge notes.

While we pulled the car to the front door, an staff wheeled Elvis out to the car.

The 4th day after surgery was a turning point -- nothing got worse and most symptoms were subsiding. On the June 17th visit, Elvis was pretty much back to normal. The stitches were taken out, and Elvis was re-affirmed to continue 4 more weeks of soft diet.

We had very good experience with the hospital. All the staff were super friendly, and the surgeon and his team are phenomenal.

Diagnosis

We were told that it would take 24 hours after the surgery to know the biopsy results. As you may remember, the exact type of Ameloblastoma is unknown until the whole tumor is removed, so the pathologist can examine the whole thing. We decided not to call ahead to find out. I think both good and bad news can wait, and we should focus on taking care of Elvis. Hui was more afraid of any possible bad news.

When we visited the doctor on June 10th for the follow-up, we were tremendously relieved to learn the final diagnosis: Unicystic Ameloblastoma -- the sub-type with least likelihood of recurrence. While nothing is guaranteed, the diagnosis is the best we have hoped for. The conservative treatment seems to be all that is needed.

Looking forward

Elvis should be back to normal diet and activities by end of July, as long as he follows the proper directions and avoids any accidental injury. Then 6 months later we will have another follow up, and most likely annually afterwards.

Emotions

Hui commented that the trip felt shorter returning home. It makes sense. When we went there, we didn't know what to expect, of anything. After the surgery, the weight of the biopsy results grabbed the central stage in our mind (though it never left our thought before). But after  knowing the diagnosis, and seeing Elvis recovering very smoothly, we traveled back light.

While in the Doctor's office on the 17th, we realized it was the same room we were in, 3 months (plus 1 day) ago for the initial consultation. At that time, we were so much worried, confused by the wide possibilities of unknowns and ramifications. The doctor's experience, rigorous approach, and conviction in long-term cure of the patient with minimum interruption, gave us confidence, hope and clarity. 3 months later, we are ready to call it done. It is a great relief.

Thank YOU for your help

Many times we feel so blessed, for the help we are getting from all of you.

• Many of you pray for Elvis, and our family
• Your thoughtful inquires of how things are going
• Cards and messages
• Kind thoughts and blessings
• The medical professionals. Though we must decide a single doctor at the end, all the doctors were very sincere, patient, in answering our questions during lengthy consultation sessions; and helped us learn the disease, the treatment options, and finally narrowing down an approach.  They paved the stones along the path for us. We are very grateful.
• Most importantly, we are very thankful for the surgeon's phenomenal expertise and professional integrity.

The After Note

The journey over the past 4 months does teach us something. Life throws all sort of things to you, often least expected. I keep on reminding ourselves, "life is not defined by these incidents, but by how we handle them." Like Hui said "treasure every minute, every second with the family." We hope our story can give you more hope for any struggle you may be in. And, we root for the best for you.

Many thanks.

Ameloblastoma 02 (Drafted on 04/04/2019): Search for Cure

First, thank you all for your support.

So what have we been up to over the past 6 or 7 weeks? We have been seeing specialists, both in the St. Louis area and the region, in the search for the best treatment.

What Experts to See?

Since Ameloblastoma is a rare disease, finding doctors experienced in the treatment takes some work. My employer has contract with Consumer Medical, which I engaged early. A nurse was assigned to our case (they call the "ally"), who actually would research on the disease, consult with their doctors, and come up with a list of providers. They provided some research articles electronically, and sent us physical copies of the articles, which are much more friendly to my eyes. The doctor list is a good starting point. Our regular dentist has been very eager to help. Of course, there is always google.

Then I thought, the insurance company must know, and answer the question like "what are the top 5 institutions treating this disease?" It turned out the insurance company can't answer the apparently simple question.

As I started calling the doctors, some of them just don't treat the disease. But I learned to ask the question "who would you recommend?" As the list got groomed, we began to make appointments.

Expert Opinions

The vast majority of the doctors we saw have been very good. Every time we walk out the office, we felt we learned more about the disease and treatments. No matter how much I read or google, that is not the same as talking to a medical professional. We felt very lucky and are very grateful for getting help from these experts. One of the confusion is what kind of specialist to see, there are the OMFS (oral maxillofacial surgeon) and ENT (otolaryngologist). It took some phone calls and a few visits for us to know that often the OMFS focuses on the treatment of the tumor, and an ENT will focus on reconstruction.

Of course, we developed the list of questions, and adapted them for the doctor visits. We also took notes, to compare and put in the computer for later reference.

The experts have different opinions based on experience, believes and assessment of the situation. Initially we were somewhat baffled by the diverse views, but later got better understanding.

• The most important factor driving the treatment approach is diagnosis from the biopsy. The biopsy report is ambiguously stated "this can be considered, ..., but can't rule out ..that". We learned the ambiguity is due to the biopsy being a part, not the whole tumor.  Therefore the discomforting realization -- the tumor is at least as severe as what the pathologist saw.
• Based on the ambiguous diagnosis from incomplete information, the doctor had to make a decision and recommendation.
• In addition, specifics can differ for the same general approach. For example, there is the bone grafting using hip vs. fibular bone vs cadaver bone vs BMP.

At the end, there is no consensus. And, we arrived at the point -- ok, now how do we decide?

The Decision

Our goal was to make a decision by April 1st, an arbitrary date since by then we would have visited all the doctors. It took us a few more days, so we can noodle on the options. One helpful tool is the comparison chart on a whiteboard at the family room.

• 3 rows: Unicystic Ameloblastoma, Solid Ameloblastoma, and Questions
• One column for each treatment
• For each treatment, we note down the advantage (with green sticky notes), risk (red) and possible mitigation (yellow), and simple facts (white).

The idea is to identity a treatment with the least amount of risk that can't be mitigated. 

The key lesson we learned during this process is the importance of seeking multiple opinions, since this is not a common simple disease with singular treatment. Part of the process is also seeking additional pathologist reviews (they call it "look at the slides"), since different pathologist can have different conclusions.

Emotions

We are all much calm now. Part of the reason is, most doctors confirmed there is no need to rush, and we can do the surgery in late May or early June. The time allowed Elvis to finish high school, and take the summer for treatment and recovery.

Also comforting to us are the helps from friends. The encouraging words and wisdom help us more than you know.

Next Steps

The surgery is scheduled for early June, so Elvis can recover during the summer, and be ready for college in the fall.

Ameloblastoma 01 (Drafted on 02/23/2019): Discovery

Our son, Elvis, was diagnosed of ameloblastoma on the left mandible (lower jaw) on 02/04/2019. He was 3 months past his 18th birthday when we found this out. A dark cloud looms above us, in his senior year of high school, and the college years ahead.

Why Blogging?

I pretty quickly came to the decision to blog our journey. To start, this is my way to reflect,  organize my thoughts, and leave a written record for our future reference. Our family lives a pretty private and quiet lifestyle, and are not exhibitionists by a long stretch. Later during research, I found we have benefited tremendously from other people's blogs. I can't image us going through the dark tunnel without learning from others. We want to contribute back to the community with these records, in any tiny way possible. Lastly, we want to keep our friends informed, of what is happening with our family. If you ask, "what can I help?"

• Please keep Elvis in your prayers, for the blessing to protect him through this difficult journey, and instill in him with wisdom and strength.

What is Ameloblastoma?

A rare benign tumor inflicting about 0.6 person/million each year (in comparison, 44 cases/million of cancer cases are diagnosed). It is locally aggressive, and has a high rate of recurrence if initial treatment is not thorough.

• It is a tumor, which means it continues to grow if left alone.
• It is benign, so it's not cancer. We feel so lucky it is not malignant. The rarity though, makes it more challenging to find doctors experienced in treating the disease.
• It is locally aggressive, and can cause a lot of damage (even life threatening) if not treated timely and thoroughly

Treatment comes down two basic approaches for Elvis:

• Conservative treatment is basically removing the tumor itself, has high recurrence rate anywhere between 50-90%. Recurrence can also turn malignant, and invading to other organs. Another problem with recurrence is, the tumor could be much more pervasive (and hard to treat) when detected since it may be more in the soft tissues instead bone with clear border. For these reasons, conservative treatment is typically conducted for young patients who are actively growing, and small tumors.
• The standard treatment involves resecting a portion of the jaw bone (anywhere between 5-10 cm), including the tumor and a safety margin (tooth 18, 19, and most likely 20 will be removed, plus about equal length of jaw bone from 17 backwards), followed by/in tandem with, reconstructive surgery. The reconstructive surgery involves taking bone from a donor site (hip or leg) and graft to the jaw. Once the grafted bone has healed and grown to the right volume, dental implants will be followed to make up for the removed teeth. 

What Have We Been Doing Since?

Research

The first thing I started was to google it, anything we could find. By the end of that week, I felt I was pretty much exhausted with research, and didn't think any more research would give us better clarity and direction. It was clear to me that, this is the biggest challenge our family has ever encountered (maybe even ever in the future). The research both educates us, but also helps us decide the treatment plan.

Reading Blogs

One part of the research is to figure out the typical patient experience. The doctors are very much focused on the medical aspect. We need testimonials of patient experience. By googling, we have found other patients, and their treatment lasting from months to years, with one year being the typical in the absence of complications. The variance primarily depends on complications post surgery, which is driven by the scale of the tumor and complexity of treatment, as well as the inherent uncertainty of such invasive surgeries. We took several blogs and began to summarize the extent of the tumor, the treatment method, what complications, duration of recovery, what went well for them, and tips.  We learn from others so we can be best prepared. We found the following blog entries very helpful

• Sandra: Taking My LifeBack: My BATTLE With AMELOBLASTOMA 
• Tina: Renewing Strength- My Journey with Ameloblastoma
• Laura: Amelo... what?
• Bailey’s Story
• Atxameloguy: Ameloblastoma: My Experience

The Revelation

Over the first 2 weeks, we also formulated the following principles going forward.

• We will not hide anything from Elvis, no matter how ugly the reality will be. He is pretty much an adult now, though a young one. It will not be fair for him not knowing the full extent of his health. Also, it is unlikely for him to make the most effective effort without adequately informed and educated.
• On the other hand, we will not overwhelm him. He needs to primarily focus on his school work, and other things he enjoys (like talking to friends, having fun together, running school clubs, video games...). Life must go on, despite of the tumor.
• We started the daily ritual at the dinner table of discussing what we have been doing that day, progress we are making, and plans. It is our daily standup.
• We would be absolutely open to each other of different opinions and views, and work through them. We felt our different views are the strength of our family, and this is the time we need all hands on deck. 
• We MUST enjoy  life, and continue doing the things we like, with more zest since our time is more squeezed. We will not  let ameloblastoma taking any more joy from our lives than absolutely necessary.

The Emotions

I never expected the emotions would be so strong. I didn't remember I was so emotionally moved when I lost my dad at 15. Later I lost 2 brothers (our siblings have always been very close) during the years, and I mourned the loss internally, not with tears. 

• Especially during the first week, my emotion was so raw. Everytime I talked with somebody, I would choke up. At one point, I just burst into tears.  I also broke into tears a few times at home. At the lunar new year, we continued our tradition of having dumplings. It took me several attempts to finish the sentence "you and mom means the world to me, and I hope we can have many, many, new years day with dumplings." The typical mundane days are the best.
• Sometimes, a tiny thing would trigger the emotion, such as saying a quiet prayer. I am not a particularly religious person, but feel more of the urge to pray these days, for the protection. By end of the 2nd week, I was finally able to sit down and play the piano (I started learning last September). There is one particular beautiful tune that is my favorite, and the beauty brought tears to my eyes. 
• I didn't know and still don't know why I was so emotional. I am not ashamed or embarrassed (though totally not characteristic of me) of the tears, but "not knowing why" frustrates me. I guess one of the reason is, it is very hard, as a father, to see his son going through the difficult journey. The aggressive treatment involves major surgery during a 1-2 year period with high risks. The first surgery itself will be a 10-12 hour process.  One friend at work said well "other people don't get it. Not that they don't try or want to, but because they are not where you are". I also knew I worried how the surgery and recovery and possible complications may affect his upcoming college life. I would give everything I have if a miracle can turn his tumor off -- but that option doesn't exist. This burden is too heavy and it wouldn't be good for me, for any long duration of time. 
• By late 2nd week, the emotion has quieted down a little bit, as we sorted out a plan of actions, and kept on working through them. 

Looking Forward

In the coming weeks, we will continue to work on the treatment planning and preparation. Future entries of this blog series will be brief, since we can't afford me taking a few hours writing blogs often. All entries will be tagged "ameloblastoma".

Please, keep Elvis in your prayers, please.

Lessons from a Piano Stand

We need a place for the keyboard. We found this stand while looking, liked it so much, and decided to make one our own. We of course adjusted the dimensions to count for difference in geometry (the person, the keyboard, ...). This is the final product.

We borrowed from our earlier experience in building our deck, by mentally going through each step first (actually multiple times). For example, we realized that when assembling the sides of the stand, the two horizontal pieces need to go in after the diagonal pieces, so there is no need to tighten them, just enough as a place holder.
But we still came out with many "unexpected" things. The most ironic is the wrong assumption of proper sitting posture. As we do with computers,  we assumed the upper arms down along the side of the body AND legs halfway under the keyboard, with the upper leg generally horizontal. This lead to the conundrum of  inadequate vertical space for the table top and keyboard.  After realizing the upper arm should be slightly in front of the body, and the knees (not legs) to be just under the keyboard, we found the bench can be raised for more comfortable arm positions.

• But that was after everything was made and finished. So for now, we use a pillow to make up the difference. This is annoying, since we cut the bench legs by about an inch during one of the checkpoints. However, we are very lucky the mistake is on the bench, not the stand itself, which will be much harder to compensate. Not all errors are equal.

And there are more things we could have done differently ...

Where to start? 

We normally start with the item(s) most prominent in our minds, which is the stand in this case. That is a mistake, since any errors we made had more significant impact (appearance, cost or effort to correct). Instead we should have started with the bench, so we could apply the lessons to the stand and made it better.  We did learn the lesson onward, and started with the bench during staining and finishing. Similarly, we worked on the less conspicuous sides first.

Check Points

After the parts for the stand were made, and BEFORE final assembly, we propped them together and put the piano on it to try. It became obvious that there wasn't enough room for my legs. So we raised the stand by 1.5", and lowered the bench by about an inch. This is both good and bad. On one hand, we are happy to have raised the stand so there is adequate leg room. On the other hand, there was no need to cut the bench legs (misguided by wrong assumptions).

Warped Legs

The legs came out crooked when the pieces are fully assembled. Since the left/right support are doweled and glued to the two horizontal pieces in the back, there was limited time to identify the issue and come up with any resolution. Then it dawned on us that we did find the legs were somewhat warped, but didn't realize it would matter.

• Later with the bench legs, we paid more attention to the blind holes making sure they are straight. When the legs still came out to be warped, we were more mentally prepared, and were able to force them to be straight.
• At the end of the project, we concluded that all lumber (natural and manufactured) are warped, PERIOD. So being prepared for warped surfaces is an essential element of woodworking. We could make perfectly angled blind holes for the dowel binding, but that alone would never produce un-warped structures.

Standards We can Trust

How do we determine that the table/bench legs are on the same plane? The workshop floor is a poor measure. We tried another table (looked shiny and felt smooth), which turned out to be warped as well. We finally settled with the straight edges of two levels -- as we can confirm the edges are straight, and two straight lines make a perfect plane.

Wish Knew it Earlier

We diligently sanded the outside surfaces to they felt smooth. It wasn't until staining we realized that the insides of the legs should have been sanded as well, so they would look better after staining. But it was too late, and may not be noticeable to the untrained eyes. So that is one corner we can be cavalier about.

On Dowel Joints

We opted for dowel joints (vs pocket hole joinery). Initially we eye balled when drilling, and found out the holes were absolutely not of the same depth (making it more difficult to cut dowel pins). Later we learned to mark the depth with duct tape on the drill bit. We also realized the importance to make the holes perpendicular to the suffice so the dowel pegs are easier to get into the holes and the assembled parts are less warped. This picture shows all the pieces to drill a dowel hole (see the duct tape on the drill bit?). Honing on specific skills helps us getting better.

Know Thy Tools

A family friend kindly showed us how to calibrate the table saw (we didn't even know such a a thing to even google). Exhilarated with our perfectly calibrated table saw, we planned to do most the cutting thence. It turned out the table saw doesn't work well to cut long pieces, so we ended up using the circular saw for most of the cutting.  After learning how to make a guide, the circular saw cuts came out perfectly every time.

Closing Thoughts

The final product actually came out better than we anticipated, and nothing we could buy at similar (or much higher) price point would come out as good. The effort is very worthwhile.

Of course, we will do things somewhat differently the second time. On projects of larger scale, a prototype would be very worthwhile. In the real life of making one time objects, we consoled ourselves with the epiphany that all artists make mistakes. Stupid ones either ignore the mistakes or get upset. Good artists do a better job to hide the defects, and great ones make the defects shine. We aspire to be nothing short of great artists.

Home Automation - Part 2

This post is to make our program in Part 1 a 12 factor app. It is therefore a digression from home automation, with more focus on software architecture and design.

December 2020 Update:

Things have changed after this article was published in February 2017.

• Weatherunderground ceased to be an option, so we switched to Openweather
• mlab has merged to Mongo proper, so we needed to do a DB migration. That was painless.
• ubidots changed the API endpoint, which we had to adjust our code.

Generally speaking, the system has worked fine. None of the dependencies changes required significant code change. 

A recap of what we have from Part 1:

• Node.js scripts running on a Raspberry pi (model B). If the Pi goes, the program goes.
• All parameters hard coded, including URLs, sampling frequency, username, password,..
• The output written to standout, and redirected to a log file, which is later parsed for analysis.
• A cron job to kick off the measurement script, and another cron job (different frequency) to post to ubidots for dashboarding.

We try to use free online services as much as possible, without hard dependency, so they can be easily swapped out.

Journey to the 12 factor app

1. One revision controlled code base: This is achieved after moving the code to gitlab. We decided to use gitlab (instead of github) since gitlab allows private repos. There is nothing secret, but the program is also of little interest to other people.
2. Explicitly declare and isolate dependencies: We just use package.json to declare dependencies, and use 'npm shrinkwrap' for runtime isolation. With shrinkwrap, we have a set of deterministic runtime packages that will be exactly the same as development. 
3. Store config in the environment: We store all configuration information in environmental variables. This approach definitely is not without risks, as an env dump will reveal all the credentials.
4. Backing services as attached resources: Backing services locator/credentials are stored in the config. These include Mongodb (storing the data. we can also use MariaDB or other mySQL variants. mLab offers MongoDB for free, an offer hard to beat), Redis (distributed cache & locking), ubidots (dashboarding), particle API endpoint (for checking the sensor readings), Wunderground API endpoint (for local weather).
5. Build, Release, Run: The app can run by itself at command line. We also packaged the app into a self contained Docker image. By packaging the configs into the container, we have a ready to run  release.
6. Stateless and share nothing processes: Stateless processes with no local dependencies, no sticky sessions, no assumption of # of instances, and no knowledge which process is running where.
7. Port binding: Listening port is defined through environmental variable with local default. Express is used as a dependency to provide the HTTP service.
8. Concurrency through process model: While not necessary, we did refactor the app to allow concurrency.
1. Use distributed locks (redlock) so only a single processes is allowed to take measurements or posting to the dashboard. If the process dies, another process will acquire the lock when the lock expires.
2. Use redis pub/sub messaging to make measurement results available for posting, asynchronously. Part of the reason for doing so, is to separate the work of taking measurement from posting to the dashboard or serving HTTP requests. When a new measurement is taken, the result is published to a redis pub/sub channel, and all processes will get the new data, so they can serve HTTP requests. This method has its scalability limitations inherent to redis.
3. Now we can comfortably/confidently running as many instances of the app, without worrying about over-using the backing services (and charged for the overage).
9. Disposability: The processes can be started/stopped easily, and it is crash-safe. If a process owning the lock(s) crashes, another process will automatically acquire the lock and perform the work.
10. Dev/Prod parity: Since we use the same types of backing services (like redis for pub/sub), the environments match "mostly".  It turned out that their version also matters.
11. Logs as event streams: We log all outputs to stdout. Concurrency makes logging a first level priority. When there are more than 3 or 4 instances of the application running, it is a must that the log is accessible in a central place for monitoring or troubleshooting for distributed, parallel programs. The Synology DS station turns out to be very handy as a syslog server, with visual dashboard automatically refreshed. 
12.  Admin tasks as one-off processes: We ended up needing this for data import.

Design Considerations

Take measurements at defined intervals

The asynchronous nature of JavaScript is both a blessing and a curse. To carry out a periodic task, both setInternal() and recursive setTimeout() will do. If a task may take a long time, recursive setTimeout()has the benefit of preventing the tasks overlapping each other. But both functions return immediately since they are asynchronous. So if the next functional block is dependent on the tasks, they need to be wrapped into the callback. Asynchronous programming takes new thinking for most people coming from imperative programming backgrounds.

Another side effect of asynchronous programming is, the main program exits after the asynchronous calls, even before the callback logic is executed. Therefore, we need a mechanism to keep the main program from exiting, without consuming resources (like an infinite loop).  This is where Express comes in handy. In addition to providing a web interface and REST API endpoint, it also provides an efficient loop to keep the main program.

Parallelism

One concern of multiple parallel processes is race conditions -- the processes stepping upon each other leading to data corruption. Another concern is a wasting of resources -- there is no benefit for the temperature being measured 20 times within a second. The solution is using a distributed locking mechanism, so each process will acquire the lock before taking on the given action. The lock must have an expiration time, so a crashed process doesn't hold the lock indefinitely. 

With parallelism, we also need to consider the granularity of computing. Does a single lock cover all the functions the program can perform? If so, only one process is active, and all other others are in stand-by by waiting for the lock. We would normally prefer to break the main logic into smaller units (or micro services), so they can be executed by other instances. With micro services, the execution orchestration becomes an issue of greater scale. Messaging and locks used in inter-process communications (IPC) and multi-threaded programming, now need to function across multiple machines, or the wide area network.

Runtime Environment

Once we have achieved the goal of making our application a 12factor app, the execution becomes more predictable, since we have a clearly defined and managed contract with the execution environment. We experimented with the following mechanisms.

• Init: We still use the Raspberry Pi for the ongoing runtime. But instead of cron, we use /etc/inittab as the process monitor to keep the process running. We wrote a "run" script to define the environment variables. 
• Docker: We also tested by creating a docker image, and launched like 5 instances. This is extremely straightforward thanks to the refactoring. All we need to do is to define the environmental variables in the Dockerfile, and use -p to expose the http port to the host.
• Cloud Foundry: We initially attempted bringing the docker image into Cloud Foundry, but found that CF expects the docker image from the Docker Hub, or a trusted Docker Registry. A private registry does not work. Then it turns out that using the buildpack is very straightforward,  4-step process. 
1. Download the code with "git clone ..". 
2. Push the app to Cloud Foundry with cf push my-app-name -c "node index.js"
3. Generate the depoloyment manifest with cf create-app-manifest app_name -o manifest.yml.
4. Edit manifest.yml to add the ENV variables do a "cf push" to update the app.

At one time, we attempted to launch 10+1 instances of the app (1 in init, 5 in Cloud Foundry, and 5 in Docker), but one keeps on failing. It turns out we exceeded our connection limit on the free redis service.

Summary

Refactoring the application, as small and trivial it is, is a very good exercise. Through the process, we gained more confidence that we can bring the application up and running in multiple but consistent ways, as proclaimed by the 12factor principles.

In a loosely coupled world, coordinated through locking and messaging, things are fine most of the time, but not as tight as "never miss a beat." For example, the lock expiration time MUST be longer than the time it takes for a task to complete. If the process owning a lock crashes while executing the task, that task execution will be missed, until the next period when it will be re-tried. Missing a "beat" is very acceptable in taking the attic temperature, but totally unacceptable in mission critical situations. Designing mission critical systems in a loosely coupled environment is
very different from a system that works the most of time.

There are other concerns to be battled with beyond code. For example, how to automate the testing for a distributed system, where pre-existing conditions are needed, such as a lock to be in a certain state?

Lastly are security concerns. For example, redis is very convenient but default security is lacking. As long as I know the password, I can connect to the cache, inspecting cached K/V pairs, and listening on pub/sub channels. Another example is the ENV configurations. If they be packed into containers, the containers need to be secured. In case of CloudFoundry, the admin console can examine all the ENVs with the cf env command.

Home Automation - Part 1

The Assertion: Ever since we had the new roof on (June 2015), I feel our 2nd floor is hotter.

Background: A new roof was installed in June 2015 on our typical two story house. Before that we had a gable mounted attic fan at the east end of the house. The fan has a thermostat that turns on the fan when the attic temperature goes above 105F. The fan has been working since early 2000'ish (2002-03?).

With the new roof, the contractor urged us to include the ridge vent so the roof would have adequate ventilation to honor the manufacturer's warranty. With the ridge vent, the gables were blocked off, and the fan disconnected.

Purpose:

• How hot is the attic, really? How is the attic temperature impacted other factors (weather, outside temperature, etc)
• Would and how much the attic be cooler if we turn on the attic fan again?

Part 1: Baseline

This is to answer the question -- how hot is the attic, and what are the factors?

Solution Components: To achieve the purpose, we need

• Some kind of sensor (temperature at least)
• Automated data capture so we don't have to climb to the attic every 5 minutes to read it.
• Environment data, particularly outside weather data.
• A way to analyse and visualize the data.

For the sensor, we decided to use the DHT22 which measures both temperature and humidity. We passed the DHT11, which can only go up to 50C (the attic is hotter than that). Another nice thing of the DHT22 is it is digital, we don't need to worry about the calibration.

For the automation, we could use the Pi (already have the model B, and v2). But the Pi's are not WIFI enabled, and are physically located a distance away from the attic. That lead us to the Particle Photon, a $20, WIFI enabled controller, with the Particle cloud enabling firmware development, OTA firmware management, and application integration.

The package looks like this, with a proto board to make some thing more permanent if we decide so. 

The Make: We mounted the photon to a breadboard, plugged in a USB cable, and were ready to play with the firmware. After a few tries, we found out D0 just couldn't work, so we decided on D1. Actually we have a "prototype station", and whole thing looks more permanent.

The Coding: There are two parts, coding the photon (creating the firmware), and integration.

For the firmware, we used the DHT_sampling.ino with minor modifications.Basically it runs a loop of nFrequency seconds, read the data, and make them available either as individual datum or as a json object. In Particle Build, you can compile the code, and flash the photon with one click. This works really well.

For the integration, we used the Particle JavaScript SDK. We also grabbed the weather data from weather underground.


The Unexpected: Then we migrated the whole thing to the attic. By end of the first day, there was mounting concern that the thing may get too hot, and create a fire hazard. Decision was made to move the unit to the 2nd floor living area, and only leave the sensor in the attic. Luckily we have a pool of thermostat cable handy. We found the move was judicious, since the breadboard came off the plywood simply in one day. The glue turned didn't survive 100+ degree heat.

The Result: Once we have all the data, we loaded it to Tableau Public, It is a great software FREE, with some limitations. From July 20th to July 30th, it is a stretch of relatively hot summer days. Play with the dashboard filters, and draw your conclusion.

My Conclusion: 

• Staring at sunrise (7 AM'ish), the temperature starts to rise. The attic temperature rises up to 42% higher than the outside temperature. 
• On average, the attic goes to 130F. On extreme days, it can be as high as 140F.
• The Attic/Outside temperature ratio:
• Varies by time of the day, obviously.
• Does NOT vary much by wind speed.
• Influenced by outside weather conditions. On July 22nd, there was rain from 7 to 8 AM, causing the outside temperature failure to rise. The rain caused the attic temperature to dip as well.

Part 2:  Turn on the Fan

Would the Gable Mounted Attic Fan Lower the Attic Temperature, and by how much?

Using everything we have built so far, the only other thing we need to do is to turn on the attic fan. Before that, we need to remove the cardbox that was used to seal off the gable.

We decided to do this around 7-8 AM on Saturday, July 30th, when the attic was around a comfortable 70F or so.

• The morning on Saturday (7/30) was relatively cool, with cloud covering. The afternoon became sunny, but not exactly hot.
• On Sunday 7/31st, the question was raised "are you sure the fan is working"? The only way would be to climb up and take a look. We did so around 1 PM when it was hot (so the thermostat would definitely turn the fan on). It was NOT moving. 
• A trip to Home Depot became necessary. A new fan was installed between 4:30 PM - 5:30 PM on the same day.
• Monday 8/1st, the morning was cool and cloudy. The PM was partly cloudy, warm but not very hot.
• Tuesday 8/2nd, Wednesday 8/3rd are both cloudy/rainy.
• Thursday 8/4th will be sunny and hot and will be a good test.
• Other hot/warm/clear days: 8/9-11

The Result: Using the same dashboard as in Part 1, this is what we see. Filter out the data between 7/31-8/3 to remove some of the bias due to weather, and early dysfunction of the fan. We can study the data on 7/30-7/31 with a limited view of the attic atmosphere when the gable is open without the fan turned on.

Post Note: Curious of the current temperature in the attic? The following is a 24 hour view updated every few minutes.

 
04/28/2019:  Recently, the local weather information (local TemperatureF and local Humidity) is missing. It turned out, wunderground has stopped the free API service. So we switched to Openseathermap