Skip to content

James Carr - Rants and Musings of an Agile Developer
Syndicate content
Rants and Musings of an Agile Developer
Updated: 6 weeks 2 days ago

Connecting AWS CloudWatch Alarms to Pager Duty with Terraform

Sat, 07/09/2016 - 01:42

I have a small item to share from the trenches today: connecting AWS Cloudwatch alarms to Pager Duty using Terraform. No back story here… just a simple “here’s how you connect these two up.”


Categories: Blogs

Signing Bonuses: Indentured Servitude of the Engineering World

Fri, 06/17/2016 - 04:36

Recently I was having lunch with several friends, two of who worked for a certain Fortune 500 company. These two in particular were discussing a lot of negative situations at their current company and I asked the obvious question… “Why don’t you just leave?” I mean right now in my opinion it is definitely a software engineer’s market. Off hand I know of at least a dozen companies with the number of open positions in the double digits that they’re desperately seeking to fill. And while I know this is anecdotal, my inbox get a ridiculous amount of emails per day from recruiters who aren’t just asking me if I’d be interested, they’re practically begging me to please forward them resumes of anyone I know who might be a good fit. So with such booming opportunities, why put up with shitty behavior of an employer?

The answers I got were interesting. While one guy had the common excuse I see where they believe their employer, despite being crummy, is the only well paying opportunity available the other gentlemen told me something I was rather surprised to hear. When he joined he was given a $10,000 signing bonus. He happily used this to pay for a bunch of of home improvements. But this wasn’t just free money, it came with some strings attached. I might have the details muddy but from what I heard the sum had to be returned in full (even including the amount that got taxed out of the original $10,000) if the employee terminates employment with the employer within 2 years. Basically what it comes down to is he has to pay $10,000 to leave and even if someone hadn’t spent the money they still need to scrap up the amount taxed on it in order to leave. What!?

I’ve heard of signing bonuses all the time and a lot of companies offer some pretty sweet stock options to prospective employees with the typical constraints (e.g. they take 3 years to vest fully) but this is the first time I heard of a signing bonus that has to be returned if the employee doesn’t complete two full years of employment. With so many of the expenses people have from mortgages to car payments to child care scraping up that amount is no small task and might even be impossible. As with my friend here he’s pretty much stuck with an employer he doesn’t like for the next year because there’s no other option available to him aside from taking out a loan to pay the signing bonus back. That’s no signing bonus… that’s indentured servitude.

The point to really ponder when you’re mulling over several prospective job offers is to understand that anything that is a bonus for signing on is just that, a bonus. Never take a job based on any signing bonus they offer. Stock options are great but don’t accept them in lieu of the full market value for your abilities. And if any company comes along with a signing bonus of a lump sum be sure you read the fine print. If it comes with a constraint where you must pay it back run straight for the hills and make sure you let them know that you won’t work for indentured servitude. Don’t sell yourself short! There’s tons of software engineering jobs out there and a ton of them that don’t come with such ridiculous, almost outright vindictive strings attached!

Categories: Blogs

Elasticsearch Throttling Indexing

Thu, 06/09/2016 - 00:25

Recent adventures in Zapierland had me in a somewhat scary predicament. Starting at some point last week I’d get an alert that the Elasticsearch cluster we run Graylog against had hit very high CPU, load and memory usage. I was confused… we had more than enough horsepower to handle queries and normally the cluster only uses about 10% of CPU on each node. I paused message processing (that’s a nice feature of graylog there, it just sticks the messages into a local journal) and began looking around. Nothing seemed off. Nothing stuck out in the logs. I noticed that the CPU and load eventually dropped, shrugged my shoulders and resumed message processing. Maybe a rogue query? Who knows.

A couple days later it happened again. Pause, resume. Everything began operating normally. While I had some hunches I had no solid evidence as to the cause and assumed that maybe it was time to expand the cluster. Since it wasn’t a fire (we just use graylog for internal log searching of API requests) I added a card to our trello board to just go ahead and kill two birds with one stone by upgrading Graylog and ES and ensuring we up the cluster size in the rebuilt cluster. I can handle the pause/unpause cycle in the short term.

As it continued happening daily, I got very curious. It only happened during peak load time and when there were a lot of staff members using it. And it always became 100% fine with a pause/unpause messaging cycle. I looked around some more and realized that we should tune the logging level a bit and see if anything stood out the next time it hit.

Sure enough, when our engineering team did a quick support blitz it happened again. This time the Elasticsearch logs were filled to the brim with entries like the two below.

[2016-06-06 18:20:34,880][INFO ][index.engine ] [ip-10-0-0-2.ec2.internal] [graylog2_761][0] now throttling indexing: numMergesInFlight=5, maxNumMerges=4
[2016-06-06 18:20:40,804][INFO ][index.engine ] [ip-10-0-0-2.ec2.internal] [graylog2_761][0] stop throttling indexing: numMergesInFlight=3, maxNumMerges=4

Well that’s a bit curious. Off to google. After a little searching I came across this entry in the elasticsearch reference.

Segment merging is computationally expensive, and can eat up a lot of disk I/O. Merges are scheduled to operate in the background because they can take a long time to finish, especially large segments. This is normally fine, because the rate of large segment merges is relatively rare.

But sometimes merging falls behind the ingestion rate. If this happens, Elasticsearch will automatically throttle indexing requests to a single thread. This prevents a segment explosion problem, in which hundreds of segments are generated before they can be merged. Elasticsearch will log INFO-level messages stating now throttling indexing when it detects merging falling behind indexing.

Elasticsearch defaults here are conservative: you don’t want search performance to be impacted by background merging. But sometimes (especially on SSD, or logging scenarios), the throttle limit is too low.

A key point following that statement is the revelation of what the default rate is.

The default is 20 MB/s, which is a good setting for spinning disks. If you have SSDs, you might consider increasing this to 100–200 MB/s.

Well this seems promising because we DO use SSD. From the documentation I issued a curl call to up the rate.

PUT /_cluster/settings
    "persistent" : {
        "" : "100mb"

With these settings in place I had a few hours until the next support blitz. Suffice to say it has been several days and we haven’t had to pause message processing yet. This is definitely an important setting to ensure you configure correctly when you tune the defaults on your cluster.

Categories: Blogs

A Dream Deferred

Thu, 05/26/2016 - 10:29

This is a hard blog post to write. I’ve actually pondered for some time whether I should write this down and publish it or just bottle it up and deal with it like we did with a similar situation two and a half years ago. I’m not one to shame companies, stir the pot up and make a scene. But I still feel some seething anger about what I perceived as discrimination then and I feel just as angry about this new situation I’m writing about today. After some reflection I’ve decided to simply tell this story and my hope is that, while venting, perhaps find some positive resolution. You may say I’m twisting events or that I’m over-reacting. You may put my wife’s character and capabilities into question. It is what is and sometimes I think it’s worth it to just share a story. To put today’s story into context I think it is best to start from the beginning, talk about the experience I bottled up and then describe today’s scenario. I’ll draw my own conclusion while you may draw your own.

My wife is a very ambitious woman. Born in a refugee camp by two parents who met after losing their loved ones under the Khmer Rouge regime she was born a fighter. She survived a bout of lung cancer when she was only two years old and to this day has only one lung. She graduated college with a Bachelor’s in Mathematic and in Computer Science. When we worked together I was easily attracted to her ambition, intelligence and genuine concern for others. She always gives a twenty dollar bill to the homeless on the street. When she encounters a veteran or military personnel in uniform in public she always thanks them for their service. But I always loved how great she was to work with… we had so much fun attending conferences together and even gave a joint presentation on Behavior Driven Development with Scala at Strange Loop back in 2010.

Where I’m going with this is my wife has very specific career goals. Sure it can be fun to be an engineer but at one point or another one has a natural desire to move up. The employer we both worked at in the past unfortunately had a very flat structure with little upward momentum. So she began hunting and landed an engineering team lead position at another local company. She spent her days mentoring junior developers on practices like test driven development, object oriented design, patterns, etc. She also provided career guidance and even sent one of her developers to management training and put him in charge of some projects to give him experience. As a side story, he went on to become a team lead himself who I hear has been doing a pretty good job. Suffice to say, she really enjoyed this position and often said she was “living the dream.”

Conflict and Discord

Looking back I think sometimes the source of the conflict between Than and her manager was the fact that Than’s ambition is very aggressive and she’s not beyond doing what it takes to get things done. I think an amusing story is when her team had five different product owners say “this is a top priority from our CEO” in different meetings, she got back to her desk, picked up the phone and called the company CEO directly and chatted with him about it and how each five priorities can’t be the top and they need to be prioritized. He really appreciated her bluntness and worked with her to re-prioritize a bit. Her manager gave her a comment that the next time she should go through her.

It probably also didn’t help that her team was very strong willed and unafraid to go against Cargo Cult practices and shake up the status quo. But I always heard good things independently from people on her team. They really looked up to her and respected how much she stood up for them. She never backed down and was always willing to fight for her team when she had to. She never got a bad review and members of the executive team really liked her. Her manager once confided in her that sometimes she felt somewhat intimidated by Than because of her knowledge, experience and aggressiveness.

Something Strange Comes This Way

Than’s team kept growing and at one point was the largest team. I think they had a good ten or twelve people and I know at least two or three had specifically requested a transfer to be on her team. However as other teams began to work on new projects members started being “borrowed” off her team. It began downsizing. Six. Four. Two. This started becoming a little stressful to her. What began as an innocuous “Brian has great knowledge of the LMD subsystem so we’ll loan him out to this project to rewrite LMD” had trickled to no team projects… just support tickets. My wife sensed something was up… perhaps a downsize? She told her manager in her one on one that if there was no reason for her team to exist and they were keeping it place just to let her keep her position they should just disband it. And her manager did with a sigh of relief.

I keep looking back on this incident. Out of six or so teams the only team to be downsized and eventually disbanded was the one with a strong, female, dark skinned Asian. None of the other Caucasian males (who I might add had a lot less experience than my wife does) had this situation closing in on them. However I think I could have accepted if they had used this opportunity to perhaps promote my wife to a position better suited for her. Perhaps the Agile Coach position to transform the department in the same way she had transformed her team? They were looking to add that position and even the CTO mentioned that he thought she’d be a good fit.

In what will go down as perhaps one of the puzzling “promotions” I’ve ever witnessed, she was moved to a Business Analyst position. While this position was dressed as a promotion (after all, her salary remained the same, right?) the truth is that she’d be on the same level as her previous direct report; a business analyst that she successfully trained. I still find this highly puzzling, perhaps her manager thought that she’d be a good fit because she did so well training her business analyst? Her days were pretty much filled with nothing but doing business analyst type tasks and continuing to work the support queue her team used to own (but all by herself).

It should come as no surprise that this was a miserable state of affairs and after about two or three months she turned in her resignation letter. Like I said Than is very ambitious and is not content to what is a dead end. And we could both see that this recent “promotion” was nothing more than a dead end. We moved on. She found an agile coaching position that was located two hours east in St. Louis and gladly took it as an opportunity to move forward in her career.

Over that year I kept feeling anger but remained cool. “Cooler heads always prevail.” is the motto I’ve always lived by. But I cannot help but keep looking back and wondering. Was this what discrimination looks like? She never had a bad review. Everyone on her team thought she was fantastic. Yet at the end of the day she… a dark skinned Asian female who had clawed her way up from poverty, the daughter of elderly refugees… was the only team lead to face a demotion. I was so mad. I wanted to breath smoke. I wanted to grab someone and yell “She deserves better than that. For all her hard work, this just is not how you should treat someone!”

But every time I reflect on the situation, it is a permanent gray area. Maybe it was discrimination. Maybe it was just dumb luck. If we cut it another way, perhaps a team had to be cut and it just had to be hers? Why should another team lead have to suffer just because they were born white and male? There’s also the nagging feeling that of all the team leads, Than as the one who came into conflict the most with her manager. Anyhow this whole situation was topped off with Than’s mother passing away and, three weeks after the funeral, her manager’s response to her resignation letter was “I knew this was coming sooner or later.”

Oh well. As Frank Underwood would say, I don’t cry over what happened in kindergarten.

Cooler Heads Prevail

Than loved her new job. She felt like she was making a difference again. As an engineering coach she put together classes to teach TDD, patterns and all that good stuff. She worked with various teams across multiple data centers to organize their work into backlogs and remove road blocks. She got lots of praise from upper management. Her career was moving again. She even showed me a letter, written by someone who had attended her classes, state that he was able to take what he learned in her class and land a different job with his salary doubled. It was really touching to see how much of a difference she was making in other’s lives.

We did have a difficulty of her living in St. Louis during the week and being home only on the weekend but we worked through it. Richard lived with her in St. Louis while Caitlynn and I spent time together working on her speech issues. We cherished our time together as much as possible on the weekends and, when school schedule would allow it, Caitlynn and I would pack up and drive down to St. Louis and I’d work from T-Rex or a random coffeeshop and meet up with her afterwards. Sure it was hard but we made it work.

Unfortunately the contract took the turn that some contracts just do. The consulting company was bought out by a larger company. The contract’s future was put into doubt. If she remained she might be looking at having to consult on site somewhere very far away. Just to give an idea of where she might wind up, the other coach she worked with was sent to Minnesota. After some clever accounting (more on that one day) we made it so that she could simply take the summer off to rebalance and figure out what she wanted to do next. Before we did this though she did take an interview at another company. As she walked out to her car the recruiter called and told her that they were very impressed. The client would love to build a team around her. When could she start? We discussed it at length and in the end she decided it’d be best to take the summer off so she could refocus. A small sabbatical of sorts.

Summer Revelations

That summer Than found out an interesting revelation that really wasn’t quite a revelation given her entire life to this point. She was just way too ambitious to be a stay at home mom. Now this isn’t meant in any shape to be an attack on those who do chose to stay at home. But she just couldn’t tolerate not bringing income into the household. I kept trying to convince her to perhaps work on a startup idea or something but she just couldn’t focus being in a completely work free situation for the first time in her life. Her parents were too elderly to work and when she turned 18 she was the sole provider of her household. She moved them out of Section Eight housing and into a trailer working at Target while attending college. And after she landed her first job out of college she moved them into their first new construction home at the age of 23. Being without a job and not being the breadwinner of the family was a completely alien experience for her.

At the end of the summer she decided to re-enter the field. As luck would have it, around this time the same recruiter who had tried to hire her before the summer started reached out. “Are you still available?” They still hadn’t filled that position and still felt that Than was the perfect fit. I drove down to St. Louis with her and worked on my laptop in a Starbucks while she met with the recruiter and Director of Engineering. They spoke for about an hour and were all smiles and excitement. As we left Than offered to buy me a nice bottle of scotch to celebrate, she had accepted their offer and was going to head up a Quality Engineering team at this new client. They’d build a team around her and after a trial period convert her to a full time manager.

Life Goes On

The remainder of the year was somewhat uneventful. She loved her new job and her new co-workers. With the salary she was now commanding she decided to simply drive back and forth (a total commute of four hours) every day because moving her career forward was worth it. We did look at houses in St. Louis but in the end just really loved our home and our daughter loved her school and friends. So we stayed put and she continued commuting. The engineering team had some growing pains here and there but they worked though them. Her director became VP of engineering. We attended dinners and bourbon tastings with the product owners she worked with. Life was good.

After some time, her contract had reached the point where she was able to convert over to full time. HR pushed really hard to get her to convert but she kept holding out. She wasn’t sure if she wanted to put roots down or not. We discussed it and in the end she decided well, it has been a nice place to work. It pays well. And they do have that sweet $10,000 signing bonus (repayable if you leave earlier than two years) that could definitely help with our deck restoration project. Plus it was a true management position that she could grow in. After much deliberation she signed the employment offer and returned it. This is where things took a quite distinct turn for the worse.

A New Director

Only a week after she converted to a full employee she received the news. A certain rockstar programmer that her employer had been trying to recruit for some time had finally accepted an offer. He’d take on the VP of Engineering’s previous director role. She was a bit uneasy about this because her VP had told someone else that if he kept up the good work he might transition him into this role and this seemed like a broken promise. She was also a little displeased that the team was 100% uninvolved with his hiring process but after meeting with him for lunch she walked away feeling that this guy was pretty alright.

By the new director’s second day he told Than that he didn’t really see a need for her team and would be disbanding it. She’d convert to an engineer and he told her, straight up, that he thinks she’ll do good being in engineering. When she told him this would be a step backwards quite a bit since the past 4 years she’s been grooming herself for management he told her “Well, if you do good and prove yourself there might be a path forward for some kind of promotion in the future.” When she pulled up in the driveway that evening I met her in the driveway and saw she still had tears in her eyes. She felt cheated. Lied to. To take a management position and have the rug pulled out from under her only three weeks later. Eleven years of engineering experience, four years leading teams and coaching, and to be told in the end that she needs to prove herself. What more does one have to prove?

Again I have a gnawing feeling. I want to push it out of my head, I want to make the excuses. “Flat organization structures are all the rage these days,” I think. Maybe he’s just doing this because that’s his philosophy. Then more circumstantial evidence builds up. None of the other managers are being demoted and my wife feels like she’s being picked on. She’s texted multiple teammates asking if she’s being petty by pushing back against her demotion. They’ve all told her not at all and even added that they’re surprised she’s not fighting back harder.

Are we coming face to face with discrimination again? Am I imagining things? I want to believe that her director is just oblivious to what he’s doing to her. It’s easy to meet someone, so far removed from their engineering origins, and come to a conclusion that they’re not that smart. That they have something to prove. He doesn’t know us. He doesn’t know my wife’s past. He doesn’t know that she co-founded a technical user group with me 6 years ago that is still going strong. He doesn’t know she was a judge at startup weekend that had it’s first place winner actually land $5.5 million in VC funding today. He doesn’t know that she’s very respected amongst so many people in our field.

Earlier today I read a very long blog post about discrimination another woman faced at a company and I began seeing some familiar narratives. Both her director and VP are projecting the perception that she’s the one causing problems. That somehow complaining about being demoted only three weeks after signing on is being “petty”. She’s being singled out as the one who is stirring the pot. Today they made the announcement that she’s transitioning to engineering in two weeks and she went to the bathroom and just cried. She feels so defeated and even asked me if she’s really that bad? Should she just exit this field altogether?

In a lot of ways I’m reminded of a poem by Langston Hughes that I first heard during my freshman year of college that really put the hook in me.

What happens to a dream deferred?

Does it dry up
like a raisin in the sun?
Or fester like a sore—
And then run?
Does it stink like rotten meat?
Or crust and sugar over—
like a syrupy sweet?

Maybe it just sags
like a heavy load.

Or does it explode?

The Road Forward

Sometimes our dreams and career goals have their ups and downs. Sometimes we take two steps forward and then have to take three back. Like I told my wife while calming her, only 4 years ago I found myself exiting a one-on-one with my manager and taking a long walk down the road wondering if maybe I should pursue a different career. Maybe I’m somehow just not cut for this field. Thankfully I lucked out because after that moment a friend’s startup really started to take off and they reached out asking me if I’d like to be their first engineering hire. If I hadn’t been in such a low point of my career I likely would have said no and bypassed what has been the best 4 years of my career.

Back to my wife’s situation, to say my anger has been simmering is an understatement. Yeah I lodge a few angry tweets here and there as I’m wont to do. But being negative really doesn’t solve much. Twitter is full of people who I can look at a long list of complaints of injustice and that’s just all they focus on day in and day out. Sure simmering in anger might feel good but it’s just not constructive in the long run.

I advised her that the situation definitely is toxic and it’s time to move on. But I just wish there was a way to make her get past the feeling of worthlessness. It sticks for awhile. It stuck two years ago and it’s sticking now. Everywhere I look in our industry we talk about diversity. “How can we attract more women to this field?” we ask ourselves curiously.

Like I said, perhaps this is all circumstantial. Perhaps you’ll say my wife just isn’t good at what she does. Maybe you’ll say like her director told her that she needs to prove herself. Maybe you’ll say she’s being petty. I’ve come to my conclusion and my conclusion is sexism. Discrimination. Call me a liar and social justice warrior of sorts if it makes you sleep better at night. I’ve simply chosen to call a spade a spade and will call it out.

We won’t take this lying down though and we won’t go silently into the night. We’re survivors. We’re a team and we don’t stop fighting. I’ll be dedicating the next several days to finding her something even better than what she thought she had with this lousy position. You might think less of me for sitting down and airing all of this but so be it. I stand up for those I love. I also can’t help but think that perhaps just distilling all my feelings about this situation into words will be more constructive than silently bottling it up like last time. I don’t want my daughter to ever have to face situations where it really is debatable if one is facing a demotion for anything other than performance. I’m not terribly worried though… my wife and I are both fighters. I am one hundred percent confident that she’ll come out on top.

These situations have left me with a final thought. In a field that has so many problems attracting and retaining diversity it is hard to imagine why when one is faced with so many set backs that leaving the industry is anything else but a logical conclusion. I don’t mean to be yet another voice beating a dead horse but it’s really time we started doing better.

Categories: Blogs